cancer is a genetic disease which is driven by the somatic evolution of cells  , where driver mutations for cancer increase the reproductive rate of cells through different mechanisms , e.g. evading growth suppressors , sustaining proliferative signaling , or resisting cell death  .tumors are initiated by some genetic event which increases the reproductive rate of previously normal cells .the evolution of cancer ( malignant tumor ) is a multi - step process where cells need to receive several mutations subsequently  .this phase of tumor progression is characterized by the uncontrolled growth of cells  .the requirement to accumulate multiple mutations over time explains the increased risk of cancer with age .there are several mathematical models to explain tumor progression and the age incidence of cancer  .the models have also provided quantitative insights in the evolution of resistance to cancer therapy  .the models for tumor progression are multi - type branching processes which represent an exponentially growing heterogeneous population of cells , where the key parameters for the process are : ( i ) the fitness landscape of the cells ( which determine the reproductive rate ) , ( ii ) the mutation rate ( which determines the accumulation of driver mutations ) , and ( iii ) the average cell division time ( or the generation time for new cells ) .the fitness landscapes allow the analysis of the effects of interdependent ( driver ) mutations on the evolution of cancer .  in this work , we present a very flexible tool ( namely , ttp , tool for tumor progression ) to study the dynamics of tumor progression .the input to our tool could be any fitness landscape , mutation rate , and cell division time , and the tool generates the growth dynamics and all relevant statistics ( such as the expected tumor detection time , or the expected appearance time of surviving mutants , etc ) .our stochastic computer simulation is an efficient simulation of a multi - type branching process under all possible fitness landscapes , driver mutation rates , and cell division times .our tool provides a quantitative framework to study the dynamics of tumor progression in different stages of tumor growth .currently , the data to understand the effects of complex fitness landscapes can only be obtained from patients or animals suffering the disease . with our tool , playing with the parameters , once the real - world data is reproduced , the computer simulations can provide many simulation examples that would aid to understand these complex effects .moreover , once the correct mathematical models for specific types of cancer are identified where the simulations match the real - world data , verification tools for probabilistic systems can be used to further analyze and understand the tumor progression process ( such an approach has been followed in  for the verification of biological models ) . in this direction , results of specific fitness landscapes of our tool have already been used in a biological application paper  . while we present our tool for the discrete - time process ( which provides a good approximation of the continuous - time process ) , results of our tool for the special case of a uniform fitness landscape in the continuous - time process have also been shown to have excellent agreement with the real - world data for the time to treatment failure for colorectal cancer  .tumor progression is modeled as a discrete - time branching process ( galton - watson process ) . at each time step , a cell can either divide or die .the phenotype of a cancerous cell determines its division probability and is encoded as a bit string of length four ( i.e.  ) .the death probability follows from as .if a cell divides , one of the two daughter cells can receive an additional mutation ( i.e. , a bit flips from wildtype to the mutated type ) with probability in one of the wildtype positions ( e.g. , cells of phenotype can receive an additional mutation only at positions two and four ; cells of phenotype can not receive any additional mutations ) .the branching process is initiated by a single cell of phenotype  ( resident cell ) .the resident cells are wildtype at all four positions and have a strictly positive growth rate ( i.e. , ) .  * fitness landscapes . *our tool provides two predefined fitness landscapes for driver mutations in tumor progression : ( 1 ) multiplicative fitness landscape ( mfl ) and ( 2 ) path fitness landscape ( pfl ) .additionally , the user can also define its own general fitness landscape ( gfl ) .a fitness landscape defines the birth probability for all possible phenotypes . following the convention of the standard modeling approaches , we let be the birth probability of the resident cells ( i.e. , cells of phenotype  ) .the growth coefficient indicates the selective advantage provided by an additional mutation at position  in the phenotype ._ multiplicative fitness landscape ._ in the mfl a mutation at position of the phenotype of a cell results in a multiplication of its birth probability by .specifically , the birth probability of a cell with phenotype is given by where if the -th position of is 0 ; otherwise .hence , each additional mutation can be weighted differently and provides a predefined effect , , , or on the birth probability of a cell .additional mutations can also be costly ( or neutral ) which can be modeled by a negative ( or ) .if the fitness landscape reduces to the model studied by bozic et al . , which we call emfl ( equal multiplicative fitness landscape ) and is also predefined in our tool . _ path fitness landscape ._ the pfl defines a certain path on which additional mutations need to occur to increase the birth probability of a cell .the predefined path can be , and again the growth coefficients determine the multiplicative effect of the new mutation on the birth probability ( see appendix for more details ) .mutations not on this path are deleterious for the growth rate of a cell and its birth probability is set to  .the parameter  ( ) specifies the disadvantage for cells of all phenotypes which do not belong to the given path ._ general fitness landscapes . _our tool allows to input any fitness landscape as follows : for for , our tool can take as input the value of . in this way , any fitness landscape can be a parameter to the tool.  * density limitation .* in some situations , a tumor needs to overcome current geometric or metabolic constraints ( e.g. when the tumor needs to develop blood vessels to provide enough oxygen and nutrients for further growth ) .such growth limitations are modeled by a density limit ( carrying capacity ) for various phenotypes .hence , the cells of a phenotype  grow first exponentially but eventually reach a steady state around a given carrying capacity . only cells with another phenotype ( additional mutation ) can overcome the density limit .logistic growth is modeled with variable growth coefficients where is the current number of cells of phenotype  in the tumor . in this model , initially ( ) , however , if is on the order of , becomes approximately zero ( details are given in the appendix ) .our tool provides an efficient implementation of a very general tumor progression model . essentially , the tool implements the above defined branching processes to simulate the dynamics of tumor growth and to obtain statistics about the expected tumor detection time and the appearance of additional driver mutations during different stages of disease progression .ttp can be downloaded from here : http://pub.ist.ac.at/ttp .  for an efficient processing of the discrete - time branching process , the stochastic simulation samples from a multinomial distribution for each phenotype at each time step .the sample returns the number of cells which divided with and without mutation and the number of cells which died in the current generation ( see the appendix for details ) . from the samples for each phenotypethe program calculates the phenotype distribution in the next generation .hence , the program needs to store only the number of cells of each phenotype during the simulation .this efficient implementation of the branching process allows the tool to simulate many patients within a second and to obtain very good statistical results in a reasonable time frame .  .in panel c the cells grow according to the pfl . in paneld we show statistical results for the probability density of tumor detection when cells grow according to different fitness landscapes .parameter values : growth coefficients , , , , , and , mutation rate , cell division time  days , tumor detection size cells . ,title="fig:",scaledwidth=49.0% ] . in panel cthe cells grow according to the pfl . in paneld we show statistical results for the probability density of tumor detection when cells grow according to different fitness landscapes .parameter values : growth coefficients , , , , , and , mutation rate , cell division time  days , tumor detection size cells . , title="fig:",scaledwidth=49.0% ] . in panel cthe cells grow according to the pfl . in paneld we show statistical results for the probability density of tumor detection when cells grow according to different fitness landscapes .parameter values : growth coefficients , , , , , and , mutation rate , cell division time  days , tumor detection size cells . ,title="fig:",scaledwidth=49.0% ] . in panel cthe cells grow according to the pfl . in paneld we show statistical results for the probability density of tumor detection when cells grow according to different fitness landscapes .parameter values : growth coefficients , , , , , and , mutation rate , cell division time  days , tumor detection size cells . , title="fig:",scaledwidth=49.0% ]  * modes .* the tool can run in the following two modes : individual or statistics . in the _ individual mode_ the tool produces the growth dynamics of one tumor in a patient ( see panels a , b , and c in fig . [fig : examples ] ) .furthermore , both the growth dynamics and the phenotype distribution of the tumor are depicted graphically . in the _ statistics mode _ the tool produces the probability distribution for the detection time of the tumor ( see panel d in fig .[ fig : examples ] ) both graphically and quantitatively .additionally , the tool calculates for all phenotypes the appearance times of the first surviving lineage , the existence probability , and the average number of cells at detection time .* features .* ttp provides an intuitive graphical user interface to enter the parameters of the model and shows plots of the dynamics during tumor progression , the phenotype distribution , or the probability density of tumor detection .these plots can also be saved as files in various image formats .furthermore , the tool can create data files ( tab - separated values ) of the tumor growth history and the probability distribution of tumor detection for any set of input parameters ( details on the format are given in the appendix ).  * input parameters . * in both modes , the tool takes the following input parameters : ( i ) growth coefficients , , , , and ( and in the case of pfl ) , ( ii ) mutation rate , ( iii ) cell generation time , ( iv ) fitness landscape ( mfl , pfl , emfl , or gfl with the birth probability for each phenotype ) , and optional ( v ) density limits for some phenotypes . in the individual mode , additionally , the user needs to provide the number of generations which have to be simulated . in the statistics mode ,the additional parameters are : the tumor detection size and the number of patients ( tumors which survive the initial stochastic fluctuations ) which have to be simulated.  * experimental results . * in panels a , b , and c of fig . [fig : examples ] we show examples of the growth dynamics during tumor progression .although we used exactly the same parameters in panels a and b , we observe that the time from tumor initiation until detection can be very different . in panel dwe show the probability density of tumor detection under various fitness landscapes .further experimental results are given in the appendix .* case studies .* several results of these models have shown excellent agreement with different aspects of real - world data . in , results for the expected tumor size at detection time using a emfl fit the reported polyp sizes of the patients very well .similarly , using a continuous - time branching process and a uniform fitness landscape , results for the expected time to the relapse of a tumor after start of treatment agree thoroughly with the observed times in patients .  * future work . * in some ongoing work , we also investigate mathematical models for tumor dynamics occurring during cancer treatment modeled by a continuous - time branching process .thus an interesting extension of our tool would be to model treatment as well .another interesting direction is to model the seeding of metastasis during tumor progression and hence simulate a `` full '' patient rather than the primary tumor alone .once faithful models of the evolution of cancer have been identified , verification tools such as prism  and theoretical results such as  might contribute to the understanding of these processes .  ** this work is supported by the erc start grant ( 279307 : graph games ) , the fwf nfn grant ( no s11407-n23 , rise ) , the fwf grant ( no p 23499-n23 ) , a microsoft faculty fellow award , the foundational questions in evolutionary biology initiative of the john templeton foundation , and the nsf / nih joint program in mathematical biology ( nih grant r01gm078986 ) .  10  vogelstein , b. , kinzler , k.w . : nature medicine * 10*(8 ) ( 2004 ) 789799  hanahan , d. , weinberg , r.a . : cell * 144*(5 ) ( 2011 ) 646674  jones , s. , chen , w.d . , parmigiani , g. , diehl , f. , beerenwinkel , n. , antal , t. , traulsen , a. , nowak , m.a . , siegel , c. , velculescu , v.e . , kinzler , k.w . ,vogelstein , b. , willis , j. , markowitz , s.d .: pnas * 105*(11 ) ( 2008 ) 42834288  nowak , m.a .: . the belknap press of harvard university press , cambridge , ma ( 2006 )  komarova , n.l . ,sengupta , a. , nowak , m.a . : .journal of theoretical biology * 223*(4 ) ( 2003 ) 433450  iwasa , y. , michor , f. , nowak , m.a . :. genetics * 166*(3 ) ( 2004 ) 15711579  nowak , m.a . ,michor , f. , komarova , n.l . ,iwasa , y. : .pnas * 101*(29 ) ( 2004 ) 1063510638  diaz , l.a . , williams , r.t . , wu , j. , kinde , i. , hecht , j.r . ,berlin , j. , allen , b. , bozic , i. , reiter , j.g . , nowak , m.a ., kinzler , k.w . , oliner , k.s . , vogelstein , b. : .486*(7404 ) ( 2012 ) 537540  bozic , i. , antal , t. , ohtsuki , h. , carter , h. , kim , d. , chen , s. , karchin , r. , kinzler , k.w . ,vogelstein , b. , nowak , m.a . : .pnas * 107*(43 ) ( 2010 ) 1854518550  sadot , a. , fisher , j. , barak , d. , admanit , y. , stern , m. , hubbard , e. , harel , d. : toward verified biological models . computational biology and bioinformatics , ieee / acm transactions on * 5*(2 ) ( 2008 ) 223234  reiter , j.g . , bozic , i. , allen , b. , chatterjee , k. , nowak , m.a . : .evolutionary applications * 6*(1 ) ( 2012 ) 3445  haccou , p. , jagers , p. , vatutin , v.a . : .cambridge university press ( 2005 )  kerbel , r.s . :carcinogenesis * 21*(3 ) ( march 2000 ) 50515  hinton , a. , kwiatkowska , m.z . , norman , g. , parker , d. : prism : a tool for automatic verification of probabilistic systems . in : tacas( 2006 ) 441444  etessami , k. , stewart , a. , yannakakis , m. : polynomial time algorithms for multi - type branching processesand stochastic context - free grammars . in : stoc .( 2012 ) 579588ttp is available for download at http://pub.ist.ac.at/ttp .the tool is implemented in java and runs on all operating systems which run a java virtual machine ( jvm ) of version 1.7 or above ( ` www.java.com/getjava ` ) .all the necessary libraries are included in the jar - file .our tool supports various features in two running modes . in the individual mode ,ttp simulates the tumor growth dynamics for a given number of generations .plots of the growth dynamics over time and the current phenotype distribution are produced simultaneously .( both plots can be saved in a png - file or svg - file . ) the full growth history for all cell types can also be stored in a data - file ( format is described in section [ app : impl ] ) .  in the statistics mode , ttp simulates the given number of patients with the same parameters and simultaneously shows the probability density of tumor detection for a given detection size ( cells correspond to a tumor volume of approximately ) .the average tumor detection time and the average fraction of resident cells at detection are also shown during the simulations . after all patients have been simulated , the existence probability at detection , the average number of cells , and the average appearance year of the first surviving cell for all phenotypes are calculated and shown in a new window .in addition the tool shows the number of detected and died tumors per year in a separate window .all these data is stored in a data - file ( format is described in section [ app : impl ] ) .ttp is written in java and makes use of several other libraries .the tool requires the java runtime environment ( jre ) of version 1.7 or above . to start ttp double - click on ` ttp.jar ` or on the command line type : ` java -jar ttp.jar ` .( make sure that you have the permission to execute ` ttp.jar ` . on mac os invokingthe tool from the command line can overcome the security restrictions . )* model implementation . *the core component of the tool is the efficient implementation of the discrete - time branching process . following bozic et al . , the number of cells of phenotype in the next generation is calculated by sampling from the multinomial distribution = \frac{x_i(t ) ! } { y_1 !! y_3 ! } [ b_i ( 1-u)^{y_1 } d_i^{y_2 } ( b_i u)^{y_3}]\ ] ] where , , and the number of cells which give birth to an identical daughter cell is denoted by , the number of cells which die is denoted by .the number of cells which divide with an additional mutation is given by and the number of cells mutated from phenotype to is given by .in general , one can define a mutation matrix to encode the probabilities that a cell of phenotype mutates to a cell of phenotype . in our case this matrix is defined by the sequential accumulation of mutations .a cell of some phenotype can receive an additional mutation only on the positions in its bit - string encoding which are wildtype ( i.e. only bit flips from 0 to 1 are allowed ) .mutations on all allowed positions are equally likely . back mutations are not considered .  _ fitness landscapes. _ our tool supports four fitness landscapes for additional driver and passenger mutations : ( i ) mfl , ( ii ) emfl , ( iii ) pfl , and ( iv ) gfl .in principal , driver mutations increase the birth rate of a cell whereas passenger mutations have no effect on the cell s birth rate . in the tables[ tbl : mfl ] and [ tbl : pfl ] we present the complete definition of the mfl and the pfl , respectively .( the definition of emfl and gfl have been given in section [ sec : model ] . )

Abstract: in this work we present a flexible tool for tumor progression , which simulates the evolutionary dynamics of cancer . tumor progression implements a multi - type branching process where the key parameters are the fitness landscape , the mutation rate , and the average time of cell division . the fitness of a cancer cell depends on the mutations it has accumulated . the input to our tool could be any fitness landscape , mutation rate , and cell division time , and the tool produces the growth dynamics and all relevant statistics .


as we are approaching the turn of the century we wish to review very briefly the status of nuclear physics as it has evolved in the course of our explorations of nature during this period .nuclear physics was born over 100 years ago with the discovery of radioactivity by becquerel and followed by the work of the curies in identifying the sources of the new radiations . the discovery of electrons by j.j .thomson occurred approximately at the same time and the identification of one of the radiations in radioactivity ( beta rays ) as electrons , pointed to a close connection in this new realm of nature .the nucleus as a small heavy center of the atom , was only deduced in 1911 by rutherford from the large angles into which energetic alpha particles from a radioactive source were scattered when incident on a foil .the simplest nucleus was evidently that of the hydrogen atom , the proton and hence thought to be elementary " along with the electron .the existence of a nucleus was a crucial feature of the bohr atom and thus central to the development of quantum mechanics . until the discovery of the neutron in 1932 it was believed that the nucleus was made of electrons and protons .the nucleus of protons and neutrons , as we know it , can be said to date from that time .the first studies of nuclear reactions were carried out with alpha particles from radioactive decay and it was quickly realized that to pursue such studies required more intense sources of charged particles .several classes of accelerators , to provide such energetic particles , were invented and developed for the study of the nucleus in the 1930 s , from the electrostatic generators of high voltage by cockroft and walton and van de graaff , to the cyclotron of lawrence , to betatrons and their derivatives .techniques for detecting particles also started in this period from the early scintillating screens of rutherford to the counters and cloud chambers of the 1920 s and 30 s .the early work on nuclear reactions quickly established the size of the nucleus and thus the range of nuclear forces .the electron spectrum of beta decay led pauli to the supposition that there had to be an additional particle involved , the neutrino , and fermi subsequently formulated the correct theory of the beta decay process .these early developments of nuclear physics provided a rich case study for the application of the new quantum theory .one of the surprising features of the nucleus was the fact that nuclear forces seemed to have a very short range . in 1935yukawa postulated exchange forces and a meson " as the carrier of the nuclear force that keeps the neutrons and protons bound inside the nucleus .such a particle , of mass between the electron and proton that was thought to be the meson , was found in cosmic rays in 1937 , but it was realized that its nuclear interactions were too weak .we now know that this was the muon and the pion was not discovered until 1946 .the 1930 s also saw the first realization by bethe of how nuclear reactions fuel the sun in converting hydrogen into the light elements .progress in nuclear physics was rapid after this start and the last 50 years have seen explosive growth in our understanding of the nucleus , its constituents and its forces . the internal symmetry between n - p ( neutron - proton ) , n - n and p - p forces , now known as charge independence , was already postulated in the 1930 s and its implications explored by heisenberg , by wigner and others .this led to the concept of isospin that has had a crucial role , both in particle and nuclear physics .the realization in the late 1940 s by m. goeppert - mayer and j.h.d .jensen that the structure of the nucleus can be understood in terms of a shell model , as in atomic physics , came as a great surprise because of the high density of the nucleus and the strength of nuclear forces , but the validity of this description has been verified by many phenomena .its theoretical basis came later .although heavy nuclei could be built up from nucleons , it was realized that there must be simpler degrees freedom and descriptions .there were many such models , including a model of an almost degenerate fermi gas of nucleons , and a liquid drop model .the dynamics of fission , discovered in 1939 by hahn and meitner , could be understood on the basis of such models .the large quadrupole deformation of some classes of nuclei led to a further understanding of new degrees of freedom , and the dynamical collective model of bohr and mottelson came out of this realization .these degrees of freedom , primarily collective rotations of non - spherical nuclei as well as vibrations , lead to a simplified description , particularly , of deformed nuclei .when combined with the shell model in a deformed potential , this work led to a unified model of the nucleus. an important degree of freedom in nuclei was found to be that of pairing " , the correlations between pairs of nucleons coupled to zero spin , and the theoretical understanding has a close analogy with the bcs theory of superconductivity involving the pairing of electrons .more recently , a very successful model has emerged , algebraic models of dynamic symmetries " named the interacting boson model . herethe degrees of freedom are those of a boson , primarily of zero spin. the unified model and the mean - field theories also permitted descriptions of excited states and transition rates .they also led to an understanding of giant resonances seen in the excitation of a nucleus , be they monopole ( breathing mode ) , dipole , or higher multipoles .  in parallel withthe vast improvement in our understanding of the structure of nuclei was work on nuclear reactions .early work showed pronounced resonances in nuclear reactions , particularly the absorption of slow neutrons , indicating long - lived intermediate states .this implied many degrees of freedom and apparently a complicated process .considerable simplification resulted from descriptions in which the description of the compound system through which a reaction proceeds is described in terms of the _ average _ properties of the resonances , without the detailed consideration of their individual properties .reactions can be considered in two limits , one proceeding through long - lived intermediate resonances whose decay is independent of their mode of formation the simple expression for a resonance was given by breit and wigner in the 1930 s , followed by the detailed treatment in reaction formalisms , particularly of wigner in the 1940 s and 50 s .the other limit developed in the 1960 s is that of direct , one step reactions , proceeding in a time comparable to the passage of the projectile through the nuclear volume . in this limit , the reaction may be described by an effective optical " potential with a real and imaginary part . here the interaction can be described as a perturbation in a quasi - elastic scattering process with the incident wave modified by the interaction but continuing coherently in the outgoing channel .formalisms for the descriptions of more complicated reactions between the two extremes were filled in more slowly .the 1950 s and 60 s also saw major new developments in beta decay , the realm of the weak " interactions .the ideas of lee and yang that parity need not be conserved in beta decay was quickly followed by the work of c.s .wu showing that , indeed , parity conservation was not a valid symmetry in these processes .the study of nuclear beta decay has laid the foundations of the standard model .the 1970 s and 80 s saw the consolidation of the description of the nucleus with improved experimental techniques supporting the theoretical framework .the use of computers aided this endeavor greatly .the use of heavy ion beams ( accelerated heavy nuclei ) was expanded and new features of nuclei were investigated . the use of electron beams for mapping of nuclear charge distributions , pioneered by hofstadter ,was extended in precision .distributions of charge and magnetization densities and transition probabilities were mapped out and this field was established as a powerful quantitative source of information about nuclei . more generally , detection techniques evolved considerably in resolution and in the capacity to handle complex information .new accelerators using superconductivity were developed , both linear accelerators with superconducting rf cavities and cyclotrons with superconducting magnets . also , the realization of the pion s role in the nucleus lead to the construction of meson factories " , where intense beams of pions were produced for studies of nuclear properties .strange mesons produced at high - energy accelerators were used to produce hypernuclei " , a new class of nuclei in which a long - lived baryon with a strange " quark is bound along with neutrons and protons .coming into the current decade , nuclear physics continues to address the state of hadronic matter , which increasingly includes the structure of hadrons as well as the larger many - body aspects of nuclei .the field of nuclear physics at the end of the century encompasses a number of areas and in this article we will attempt to discuss briefly a few of the current thrusts and cite a few review articles that provide more details .the hadrons are the simplest entities of strongly interacting matter that can exist by themselves .their properties are well established experimentally , but the way they are constituted out of quarks and gluons is still not well understood .recent experimental results have shown that the spin of the nucleons , for instance , is not as simple as it seemed a few years ago , but has contributions from the polarization of the qcd sea " , from gluons and from the possible angular momentum of quarks and gluons .how protons and neutrons the most stable hadrons interact with each other to form simple nuclei has seen substantial progress .evidence is now quite conclusive that simple two - body forces are insufficient to explain the properties of the simple nuclei and that many - body forces are important .the understanding of the structure of nuclei in terms of the shell model and the various collective rotations , vibrations and excitations that nuclei undergo has advanced in several directions .in particular , new detection techniques have helped unravel bands of states that correspond to shapes that are deformed from spherical symmetry much more drastically than previously observed suggesting a region of stability with 2:1 ( major to minor ) axis ratios .these states appear to be rather pure and hardly mix at all with the more spherical normal states .other advances in experimental capabilities have allowed the approach of the limits of nuclear stability , the so - called drip line .one of the aspects of qcd that is not satisfactorily understood is the concept of _ confinement _ , the fact that the constituents of hadrons can never appear isolated by themselves . at very high densities this confinementis expected to break down to the extent that quarks can travel freely within the region of high energy density .this is presumably a state that the universe passed through shortly after the big bang .there will soon be a new tool for investigating the state of matter at that time : a large collider of heavy ( e.g. , au ) ions is being constructed in which similar energy densities should be reached . a key symmetry that is broken in normal qcd , that of _ chiral symmetry _ , may well be restored in this regime of energy density . both the experimental undertakings at this facility and the theoretical interpretations are a major challenge for the field in the coming decade .  the crucial role of nuclear physics in fueling the stars has been recognized since the early work of bethe , who showed how stars are powered by fusion reactions , and later of fowler and coworkers who developed the understanding of the processes responsible for the formation of elements .specific nuclear properties play key roles in the big bang , in the energy production in our sun and in other stars , and in all the nucleosynthetic processes of the universe .this intimate relationship is beautifully illustrated by the fact that the properties of the lightest neutrinos have been enormously clarified by the theoretical interpretation of experiments that searched for the nuclear reactions that these neutrinos induce on earth .  finally , not only has the field depended critically on developing a large variety of experimental and theoretical techniques , but these techniques have in turn served society in a number of ways the prevalence of nuclear medicine being a prominent example .the smallest entities of accessible strongly - interacting matter in the world are hadrons , either baryons that are aggregates of three quarks or mesons that are made from quark - antiquark pairs .the most stable baryons , the protons and neutrons , are the major constituents of atomic nuclei , and the lightest meson is the pion .understanding the structure of hadrons and how the properties of these particles arise from qcd is a major interest of nuclear physics .this interest follows two paths .one concerns the properties of families of hadrons as they exist freely , to accurately characterize the members of the rich hadron spectrum in mass and decay properties and reflect the structure that arises from qcd .the other is to understand how these properties change when the hadrons are immersed in nuclei or nuclear matter .   among the mesons ,the lightest and most important one is certainly the pion .thus , it is no accident that its properties , production , and interactions with nucleons and nuclei have received considerable attention in the past and again at the present time .  because quantum chromodynamics ( qcd ) is the underlying theory of hadronic interactions ,there have been many models built on one aspect or another of the theory .a particularly important symmetry of qcd , which is almost preserved at low energies , is chiral invariance or the symmetry between left- and right - handedness .this symmetry is incorporated into the most recent treatments of few - body problems , with the use of low - energy effective theories , such as chiral perturbation theory , first introduced by weinberg many years ago. here an effective lagrangian is constructed which incorporates all terms allowed by the symmetries of qcd . in qcd with massless up ( u ) , down ( d ) and strange ( s ) quarks , the theory satisfies chiral invariance .this leads to both conserved vector and axial vector currents and to parity doublets .since the axial current is not conserved in nature and parity doublets are not observed , one assumes that spontaneous symmetry breaking leads to the eight goldstone ( almost massless ) pseudoscalar bosons .the finite quark masses also break the symmetry somewhat , and this leads to the non - vanishing pion and other light pseudoscalar meson masses .the approximate chiral invariance is incorporated in all low energy effective theories .chiral perturbation theory is a low - energy theory with a systematic expansion around the chiral limit in powers of and , where is a quantum chromodynamics ( qcd ) scale , of the order of 1 gev .because it is an effective theory , it needs to be renormalized at each order of the expansion .one introduces an effective operator in terms of the pion field .the most general lagrangian density is then unique .when expanded in terms of and momenta/ , the free pion lagrangian is obtained to lowest order and pion - pion scattering is found at the next order . although the agreement with experiment is quite good , even at this order , it is improved by continuing the expansion through the inclusion of higher order terms.  recently the photoproduction of pions has received considerable attention because it is a test of chiral perturbation theory . at threshold the production mechanism is dominated by the electric dipole amplitude which is given by gauge invariance. in chiral perturbation theory the production amplitude is independent of the pion mass and the pion - nucleon coupling constant . at this order ,the photoproduction from protons and neutrons vanishes .but at the next order , an expansion in terms of and gives a finite value which agrees quite well with recent experiments. .higher order calculations do even better .we compare theory and experiment in table i.  .magnitudes of the amplitude for photoproduction of pions in units of to lowest ( n = 1 ) and higher ( n = 2 and 4 ) order in a chiral perturbation theory expansion compared to experiment. [ cols="<,>,>,>,>",options="header " , ]   while we know that nucleons and mesons are composed of quarks and gluons , the transition from a description of nuclei in terms of nucleons and mesons to one in terms of quarks and gluons is still not understood . nordo we fully understand the structure of the nucleons and mesons .progress has been made by solving qcd numerically on a finite lattice rather than in continuous space - time .  in an effective theory the gluon degrees of freedomdo not appear explicitly ; in some models they are incorporated in the dressing of the quarks , which then are called constituent quarks .the constituent quarks ( up and down ) have masses close to one third of the mass of the nucleon and thus have small binding energies .there are models of the nucleon made up of such quarks , often treated non - relativistically and bound by harmonic or other simple forces .these models are amazingly successful in predicting the ratio of the proton to neutron magnetic moments .however , there have been a number of authors who have pointed out that this result is not very model dependent .most effective theories ( e.g. , chiral perturbation theory ) use almost massless current quarks .one of the original motivations for the study of nuclear structure was to gain an understanding of the strong interaction . following this interest in the structure of hadronic matter, nuclear physicists have become more and more interested in a quantitative understanding of the structure of hadrons .qcd provides the framework for such understanding .for instance , qcd - based constituent quark models not only can reproduce accurately the masses of mesons with heavy quarks , but can also account for the main features of the masses and electromagnetic decays of baryons with light quarks .however , significant problems remain . in particular ,some signs and magnitudes of strong decays of the higher mass nucleon resonances are poorly understood , possibly because of the lack of a proper treatment of chiral symmetry .the lack of clear experimental evidence for particles that would correspond to excitations of the gluonic field or hybrid " states of baryons that involve involve gluonic excitations is an another outstanding puzzle .  in the last few years deep inelastic polarized electron scattering on polarized h and d targets have provided insights into the spin structure of the nucleon and have taught us that we do not yet fully understand it . for a given quark species ( u , d , or s ) the fraction of the nucleon s spin that is carried by quark spins is defined as    with arrows indicating spins parallel ( ) and antiparallel ( ) to the proton s spin and is the weak isovector axial coupling constant .the nonrelativistic naive " constituent quark model predicts    and   eq .( 5 ) turns out to be far from the truth .neutron beta decay yields and the deep inelastic scattering experiments show that only about 30% of the spin of the proton comes from the quarks .it is found that     the experimental result came as a surprise and was called the proton spin puzzle " ; it implies that the major fraction of the proton spin comes from the gluons and possibly from angular momentum .however , is generally evaluated in the infinite momentum frame and the nonrelativistic quark model is for a nucleon at rest . in terms of nuclear physics ,the angular momentum could come from pions coupled to quarks .another surprise was the relatively large ( ) contribution to the spin from strange sea quarks .again , in principle , this can be understood in terms of low energy nuclear physics via the dissociation of protons into strange baryons and mesons such as and .antiquarks play a role because gluons can split into pairs and because quarks may couple to mesons , particularly pions . to the extent that the fock space contains pions, one can understand the excess of over in a proton since a one meson decomposition gives or , with a . indeed, there is evidence for an excess of over in the proton from inclusive hadronic reactions with lepton pair production ( drell - yan processes ) and from tests of the gottfried sum rule " , which follows from the assumption of a flavorless sea of light quarks ( ) .the nucleon - nucleon force is basic to understanding nuclei and thus has been of great interest for many decades .broadly speaking , potential representations of the force are either purely phenomenological , or based on meson - exchange but with the parameters determined phenomenologically .all models have a one - pion exchange character at long range , which gives rise to a spin - spin central potential and a tensor term .the correctness of pion exchange and its dominance at large distances is clear from the nucleon - nucleon phase shifts at large angular momenta and from various properties of the deuteron ground state , such as the non - zero quadrupole moment .indeed , the strong tensor component of the pion - exchange force is a unique feature that makes the solution of nuclear many - body problems particularly challenging .some potential models represent the shorter range interaction by heavy - meson exchanges .the reid and the urbana - argonne potentials are examples of more phenomenological models , while the nijmegen , paris , and bonn potentials are based more on meson exchange., in the last few years significant progress has been made in obtaining high - precision fits to the elastic scattering data .the argonne " , the cd bonn " , and several nijmegen models all fit these data within the experimental accuracy .these modern potentials , coupled with recent advances in nuclear many - body theory and in the capacity of computers now makes it possible to understand the stability , structure , and reactions of light nuclei at a microscopic level .three and four - nucleon systems are studied accurately in both bound and scattering states by faddeev and hyperspherical harmonic methods . for nuclei with up to 7 nucleons ,quantum monte carlo methods have been applied successfully .ground states , stable against breakup into subclusters , are determined for and , and their binding and excited state spectra agree reasonably with the data . in thiswork three - nucleon forces are required ; their strength is adjusted to reproduce the binding energy of and to give a reasonable saturation density for nuclear matter.  chiral perturbative theories and other effective theories have also been applied to the nucleon - nucleon problem .the s - state scattering lengths and effective ranges as well as the phase shifts up to momenta of 300 mev / c can be fit very well , as is illustrated in fig . 1 , despite the difficulty of the anomalously large scattering lengths in the state and the states .their effective potential includes a contact interaction and pion exchange .related techniques with effective chiral theories have been used by others. it is interesting that chiral perturbation theory allows one to show that 3-body forces are smaller than two - body ones by the ratio , where p is a nucleon momentum and , as before , is a qcd scale ; for example , if the two - body potential has an average strength of 20 mev , then the three - body one would have a strength of about 1 mev .the theory can also be applied to charge independence- and charge symmetry - breaking forces and also gives a strength ranking for these forces ; e.g. , one can show that the charge symmetry - breaking force is weaker than the charge - independence breaking one .very sensitive tests of the nucleon - nucleon interaction in bound states are precise measurements of the radial distribution of nucleons in nuclei for comparison with _ ab initio _calculations. in many cases electron scattering can measure these distributions directly . however , in the case of the deuteron which has unit spin , the orbital angular momentum 0 and 2 contributions can be separated using polarized electron - deuteron scattering in the quantity . the current information on this quantity is illustrated in fig .a measurement that should provide such information to momentum transfer , or about 0.15 fm in the distances in the radial distribution of the deuteron , is among the early experiments at cebaf .while these measurements demonstrate the wide validity of the hadronic description of the deuteron , at the shortest distance scales the nucleon substructure does become important .to date one nuclear reaction , the photodisintegration of the deuteron at large transverse momenta , into a proton and a neutron , shows the behavior expected of coherently transferring the energy of the incident photon to the six constituent quarks in the deuteron .indeed , this reaction seems to show counting rule " behavior at considerably lower energies than was expected , suggesting a wider validity for quark descriptions .the successes in understanding the structure of nuclei are represented by models such as the shell model and the collective model .the characterizations of the finite many - body system with complex forces between the constituents is necessarily approximate .the use of the hartree approximation with simplified potentials representing the nucleon - nucleon interaction , or even a field theory with scalar and vector mesons ( quantum hadrodynamics " ) leads to a mean field in which the nucleons move .the theory can be extended to hartree - fock and to include deviations from the mean field ; it can be compared to experiment .for instance , it does well in reproducing the charge density determined by electron scattering as is seen in fig .3 .  electrons are a great tool for precision studies of nuclei : their interaction is sufficiently weak that perturbation theory can be used , they cause little distortion of the system , and their wavelengths can be made sufficiently short to study both nuclei and nucleons in detail . electron scattering beautifully shows the single - particle structure of nuclei in a mean - field description by measuring the properties of individual shell - model orbitals and also by exploring the limitations of this description in exploring the correlation effects that arise from the short - range part of the nucleon - nucleon interaction. this is clearly seen in the proton knockout reactions illustrated in fig .the consequences of these correlations are manifold .they substantially renormalize the single - particle mean field orbitals and appear to be the source of high - momentum nucleons that are important in sub - threshold " production of mesons and other particles ( e.g. anti - protons ) .such correlations can have an effect on the mean - free paths of nucleons and other hadrons in nuclei and on mechanisms responsible for pion absorption in the nuclear medium .but we are only at the beginning of being able to separate the effects of short - range correlations from other many - body effects .another perspective of the single - particle structure in nuclei with many nucleons comes from an entirely different dimension .while the successes of the single - particle description in heavy nuclei are remarkable , the description is tested almost entirely for the valence " orbitals and not for the deeply bound states .such tests become possible by introducing a different baryon into the nucleus that can settle into the lowest state without violating the exclusion principle .the most suitable baryon , because of its relative stability , is the , and the structure of the deeply - bound states of the so - called hypernuclei " beautifully confirm the single - particle description with the mean field modified to account for the differences in the -nucleon interaction.  major advances have been made recently in exploring the structure of nuclei under extreme , limiting conditions at very high angular momentum , in approaching the limits of nuclear binding , and in temperature and energy density ( discussed in section v ) .we discuss the first two in the section below , as examples of recent developments in the field .major advances in recent decades have come primarily from novel and substantially improved experimental techniques coupled with new theoretical understanding .in particular , accelerator developments have expanded enormously the available beams . with beams of heavy nuclei accelerated as projectilesit has become possible to bring large amounts of angular momentum into nuclei. for instance , with a 200 mev beam incident on a target of one forms a compound nucleus at high excitation energy , that first rapidly decays ( in sec ) by emitting particles ( neutrons ) , and then remains highly excited in a bound system , usually with high angular momentum . in a rotating reference frame, the excitation energy is not very high it has to be measured with respect to the lowest energy that the nucleus can have at this angular momentum ( the so - called yrast line " ) . under the influence of centrifugal forcesthe lowest configurations in the nucleus can be quite different from those at low angular momentum .the shell structure of nuclei becomes rearranged under the centrifugal effects of rotation and new pockets of relative stability , in the potential energy surface of the nucleus , may develop as a function of quadrupole deformation .this then results in new classes of nuclear states , with high deformation , in which many of the nucleons have microscopic quantum numbers that are different from those of the ground state .the exploration of nuclear structure at high angular momentum has shown that a few percent of the time fusion reactions populate states that decay by electromagnetic cascades which show characteristic rotational bands of remarkable simplicity as is shown in fig .the energy spacings in these bands correspond to nuclei with much higher deformations ( 2:1 axis ratios ) than those in the normal deformed bands that had been the basis of the collective model of bohr and mottelson ( typically 1.3:1 axis ratios ) .the exploration of the properties of these _ super deformed " _ bands has uncovered a great deal of structural information .the precise energies in these including the microscopic details of the small deviations from the pattern expected of perfect rotors are reproduced with surprising accuracy in several nearby nuclei , giving rise to the _ identical band _phenomenon as is illustrated in fig .it seems that these microscopic signatures carry over from one nucleus to another , without change , in a way that has not been seen elsewhere in nuclear structure and is not fully understood .one of the most interesting features of superdeformed bands , mentioned above , is the fact that even when these states are well above the yrast line , they hardly mix at all into the higher density of states with more normal " deformations .there appear to be two distinct classes of states corresponding to two minima in the potential energy surface , with different deformations .the superdeformed states are closely related to a class of states that appeared in the 1960s in the study of delayed fission of very heavy nuclei .the mixing between these states leading to fission and the ordinary states is similarly inhibited .the discovery and study of these phenomena in nuclei at high - angular momentum have become possible through major advances in the detection and precision energy measurement of gamma rays with high - resolution germanium diodes .the size of the detectors , their anti - coincidence shields to suppress the compton - scattering background have now been developed to the point where complete spheres of detectors are used to search for multiple coincident gamma rays from a cascade .this instrumental advance culminated in detectors such as gammasphere in the u.s . andeuroball in europe. the new experimental information , in turn , has lead to major advances in the theoretical insights into the structure of nuclei at the limits of large centrifugal stress .  experimental work with these new instrumentshas also lead to other new discoveries .one of the most interesting of these are bands of states connected by the emission of a sequence of electromagnetic quanta whose energies increase in small smooth increments , very much as those for rotational states. however , unlike the electric quadrupole radiations that are the earmark of transitions within rotational bands , the gamma - ray transitions in these sequences are magnetic dipole in character .this came as a dramatic surprise .the plausible explanation for these magnetic transitions is that there are two , rather stable , configurations , one for protons and one for neutrons , each of large angular momentum .the sequence of states with increasing angular momentum then correspond to states where the two configurations simply are re - oriented to be more nearly parallel and thus give higher total angular momentum , as the closing blades of shears .this explains the large magnetic dipole transitions , but the smooth dependence to the sequence of energies is still very much a puzzle .the phenomenon suggests that some cooperative , collective features are present , though theoretical understanding is still not complete .   the 1950 - 1980 period saw the systematic exploration of the structure of nuclei in and near the valley of nuclear stability through a variety of techniques .the limits of nuclear binding , where for a fixed number of protons no more neutrons can be bound , ( or for a fixed number of neutrons no more protons ) , the so - called drip lines " were largely unknown , except for the lightest nuclei .the drip lines are of interest because nuclear properties might change , especially near the neutron drip line .they are also of particular interest in various stellar processes where , in a hot environment , a sequence of captures takes place rapidly . with recent advancesthe exploration of these limits has started in the 1990 s .one result along the drip lines is the observation of proton radioactivity , where the nuclei are literally dripping protons because their binding is insufficient but the coulomb barrier retards their emission . at present , these have been identified in a number of elemental isotopic sequences , from co to bi , and the structure of these nuclei at the proton drip line is beginning to be explored.  another result is in the limit of neutron binding where much less is known , because this regime is much more difficult to reach in laboratory experiments. since neutrons are are not subject to the coulomb force , loosely bound neutrons result in long tails in the density distributions that fall off exponentially but with the exponent decreasing with binding .thus the neutrons will reach far beyond the proton distributions in very neutron - rich nuclei .such a separation between neutrons and protons , might in turn cause some qualitative changes in nuclear properties .for instance , it has been suggested that the spin - orbit term may be substantially reduced in such nuclei , and this would cause a change in shell structure that would be very interesting to observe experimentally .such a change could also have serious consequences on the rapid neutron capture , r - process , in explosive stellar nucleosynthesis .the best current example of a nucleus with diffuse neutron excess is in a very light nucleus with three protons and eight neutrons . hereexperiments show clearly how the last two , very loosely bound , neutrons form a diffuse tail , a halo " around the protons .thus the interaction radius of this nucleus is substantially larger than that of other li isotopes as is shown in fig .7 and the structure of , what would be the electric dipole giant resonance in other nuclei , is substantially different here , as is the momentum distribution. the further exploration of very neutron - rich nuclei , beyond the very light ones , requires major new advances in experimental techniques and facilities .another limit is being explored in the size a nucleus can reach in its total mass , or nucleon number . hereingenious improvements in experimental techniques permit the production of heavier and heavier new isotopes and elements beginning to approach the region where calculations predict that , because of the stabilizing effects of shell structure , a new island of relatively stable _ superheavy " _nuclei should occur .a few nuclei of the new element with z = 112 have been produced near one such possible island with z = 114 , but further from another suggestion with z = 126 .the results may in fact indicate a relatively stable bridge leading to a more stable island .very heavy atoms are also of interest because of their atomic structure , to explore the rapidly increasing effects from the large coulomb field in approaching the limits of vacuum polarization as well as the increasing relativistic effects.   a crucial assumption in most of our many - body descriptions of nuclei is that nucleons and other hadrons do not change in the nuclear medium .deep inelastic scattering of electrons on nuclei gave evidence that nucleons and their quark structures are altered somewhat when they are placed in nuclei . this is called the emc effect after the group that first discovered it .their finding has caused interest in studies of these changes both experimentally and theoretically and has raised important issues , more generally , about how the properties of hadrons change in the hadronic medium of a nucleus .the changes can be investigated at the quark level in high momentum - transfer reactions and at the hadron level in both electron scattering and heavy - ion collisions . using electrons , the elastic form factors of the proton and neutron inside nucleihave been compared to those of free protons and neutrons in deuterium .intriguing differences in the ratio of the magnetic to electric properties of the proton in light nuclei have been observed , and this is an active area of investigations .on the other hand , there is clear evidence for changes in the effective nucleon - nucleon force in the nuclear medium .the challenge is to distinguish normal " many - body effects from changes in the hadronic substructure .for instance , how do virtual pions from pion exchange manifest themselves in a change of the sea antiquark distribution of nucleons in nuclei ?this is being examined at the hadronic level in looking for pions knocked out by electrons , in drell - yan processes that are directly sensitive to antiquarks , and in looking for pionic modes excited in proton scattering .if the structure and properties of the mesons change in the nuclear medium , there may be profound implications for the effective internucleon forces .these changes involve not only pions but also vector mesons .theorists have suggested that the masses of the latter should decrease , but there remains controversy about this suggestion .the width of the resonance is also likely to be affected .these alterations can be sought by searching for the leptonic decays of the vector mesons produced in heavy - ion collisions and studying their leptonic decays within the nucleus .such experiments are being undertaken .many of the changes of hadrons are not large , but the methodology seems to be at hand to seek them out and the consequences could have a profound impact throughout nuclear physics .when a colorless hadron is produced in a high momentum transfer reaction , the hadron is small at birth . due to color screening of the quark components of this small hadron ,its mean free path in the nucleus is large and its final state interactions small . this phenomenon is called color transparency, and should be visible in quasielastic ( e , ep ) and ( p , pp ) reactions on a nuclear target . at presentthe evidence for nuclear transparency is ambivalent and further experiments are required to tie the effect down .nuclear physics plays a key role in the processes that take place in the universe , from the big bang on to the energy production in stars .the synthesis of the chemical elements in the universe is the result of the various nuclear reactions that take place in different stellar environments .the big bang produced mostly protons and some of the lightest elements .when stars are formed from these remnants of the big bang , their matter is heated as the star contracts under gravity . in the hot star ,nuclei run through cycles of nuclear reactions and hydrogen is gradually converted into helium , as in our sun .somewhat hotter stars will form carbon , and as the carbon cycle described by bethe becomes important more hydrogen is converted into helium .further heating will cause captures of protons and alpha particles beyond the carbon cycle . under appropriate conditions these reactions will produce elements up to about mass 56 . to get further in mass , neutron capture is essential and this can happen slowly in the s - process " or explosively in the cataclysmic r - process " .all the elements heavier than iron in our world were produced in such stellar environments .below we mention only a few of the key developments in our understanding of the recent past following the major insights in this field by bethe , fowler , and others in identifying how nuclear processes determine the evolution of matter in the universe , in describing energy production in the sun and in explaining the formation of the elements from the big bang through the stages of stellar evolution .one of the intriguing developments of the past decades has been the study of neutrinos from the sun .other than the heat radiated , the neutrinos are the one accessible observable product from the chain of nuclear processes that take place in the interior . it was noted early that the number of neutrinos detected on earth was too small , given that the energy output of the sun is known accurately and thus the number of nuclear reactions leading to neutrinos is also known .the pioneering experiments of davis and colleagues with a chlorine detector in the homestake mine in south dakota were sensitive primarily to the highest energy neutrinos , those from the decay of .the observed neutrino flux was about a third of that expected , and in spite of extensive measurements and remeasurements of the nuclear parameters relevant to the solar processes and to the detection scheme , the discrepancy remains today as is shown in fig .more recent experiments with gallium as the detecting material , in europe and russia are sensitive to much lower energy neutrinos .they also find substantially fewer neutrinos than expected , as do experiments in which the high - energy neutrinos are detected more directly in large water detectors in japan .the combined impact of these very different experiments has already been profound .the deficiency in the number of electron neutrinos might have its origins in the possibility that neutrinos have finite mass and that they oscillate between the originally emitted electron neutrinos and neutrinos of other flavors , with this oscillation enhanced by their passage through the dense matter of the sun .whether this is indeed the case will be tested in experiments to be carried out in the coming years.   among the most spectacular events in the universe are the supernova explosions that occur under the right circumstances during the evolution of sufficiently massive stars .the dynamics of such an explosion involves a complex interplay between nuclear properties and gravity the weak interaction . supernovae play by far the dominant role in the synthesis of heavy elements .the decay in the light curve of a supernova , shown in fig .9 , is governed predominantly by the decay of  the progenitor of , the most tightly bound nucleus and the most abundant element constituting the earth .the enormous flux of neutrinos from supernovae was evidenced by the dramatic detection of a neutrino pulse from the supernova sn1987a .recent theoretical work has shown that the shock front of neutrinos interacting with nuclear matter plays a major role in the dynamics of supernova explosions .a typical neutron star has about 1.4 solar masses compressed in a sphere of 10 km radius .more than 500 neutron stars have been detected in our galaxy , most as radio pulsars and some at optical and x - ray wavelengths .a dozen neutron stars are found in close binary pairs , which has allowed their individual masses to be measured very accurately through orbital analysis .binary neutron star coalescence events may be the source of observed extra - galactic gamma - ray bursts , and our best hope for directly detecting gravitational radiation .the structure of a neutron star is a consequence of the interplay between all interactions : the strong nuclear force , electroweak interactions , and gravity , with significant corrections from general relativity .the surface is metallic iron , which is the most stable form of matter at zero temperature and pressure .underneath is a lattice of nuclei that grow progressively larger , more neutron - rich , and more tightly packed as the density increases with depth , since it is energetically favorable to capture electrons on protons to reduce the kinetic energy of the electron gas . at about matter density , neutrons start to leak out of the nuclei , forming a low - density neutron superfluid in the intervening space .as the density increases further , around , various unusual shapes of nuclei may occur , with transitions from spheres to rods to sheets to tubes to bubbles . eventually at a density around , or normal nuclear matter density ,the nuclei dissolve into a uniform fluid which is over 90% neutrons , 5 - 10% protons and an equal number of electrons to preserve electrical neutrality , all in beta - equilibrium . at successively higher densities muons , and perhaps pions and/or kaonsmay be present . at high enough densities ,quark matter will probably appear , perhaps initially as bubbles in the nucleon fluid .progress in characterizing the nucleon - nucleon interaction has been key to our increased understanding of dense nucleon matter and consequently of neutron star structure .if neutrons were noninteracting , the maximum neutron star mass stable against gravitational collapse to a black hole would be 0.7 solar masses .however , many neutron stars have been observed with masses 1.3 - 1.6 times the solar mass , so the role of nuclear forces is clearly important .recent work has set the minimum upper limit on neutron star mass at 2.9 solar masses , helping to further refine the observational boundary between neutron stars and black holes .observations of quasi - periodic oscillations in binary x - ray sources may also soon lead to limits on neutron star radii , which will give an even tighter constraint on the dense matter equation of state .the nuclei of the chemical elements are formed in the very hot environment inside stars . at higher stellar temperatures , these processesoccur sufficiently fast that the nuclei involved are themselves short - lived .new tools are being developed in nuclear physics that will help meet the challenge to determine properties of these short - lived nuclei , including the cross sections that are likely to be most important in astrophysical contexts .in addition , there are some expectations that the general outlines of nuclear structure may change near the limits of binding , where the path of explosive nucleosynthesis takes place .the enormous improvements in the observational techniques of astronomy and astrophysics inevitably will require better quantitative understanding of the nuclear processes that yield energy in the universe and form an intriguing interface with nuclear physics .one of the areas of intense interest is associated with the physics at very high densities , where the description of matter in terms of quarks and gluons contained in individual hadrons must break down .calculations based on qcd suggest that when high densities and temperatures are reached in a volume large compared to that of a typical hadron , a transition to a state of matter will occur where the quarks are no longer confined to their individual hadrons but can move freely within the larger volume .  a schematic phase diagram is shown in fig .10 illustrating this transition and its relationship to the evolution of the universe following the big bang . to explore this state ,experiments have been carried out , first at the bevalac at berkeley , then at increasingly higher energies at the ags in brookhaven and the sps at cern , and soon at a new collider , the rhic ( relativistic heavy ion collider ) facility at brookhaven that is to come into operation shortly . since the confinement of quarks in hadrons is one of the key features of the strong - interaction world , this deconfinement , to a volume larger than that of a hadron , is of course of intense interest to physicists .a key question is how this deconfinement might be observed in an unambiguous fashion .the collisions between two heavy nuclei at ultrarelativistic energies produce many thousands of fragments .their detection , identification , and characterization is a formidable challenge to experimentalists .the first question is whether the large densities in energy and baryon number are indeed obtained in such collisions whether their kinetic energy is absorbed or , whether the two nuclei primarily just pass through each other .this may be deduced from the measurements of the transverse momentum carried by the products of the interaction .such `` stopping '' studies at the ags , at laboratory energies up to 14 gev per nucleon show clearly that the kinetic energy is absorbed , and that high temperature and high baryon density are indeed created .this remains true in work at cern up to laboratory energies of 200 gev per nucleon .as yet , there is no unequivocal evidence in these data for a phase transition .the energies at rhic will be an order of magnitude higher and it is expected that this will lead to the formation of a high - temperature low baryon - density system well past the expected transition point .  how the properties of this transitory state of matter can be deduced from the data is the subject of intense discussions between experimenters and theorists .some of the possible signatures " discussed and explored in current experiments are the suppression of the production of mesons due to screening of the charm - anticharm interaction by the surrounding quarks and gluons , an increase in energetic leptons in radiation from the early stages of the system , the modification of the width and decay modes of specific mesons , such as the and , enhancement in the production of strangeness , the attenuation of high energy jets , etc .a change in the state of matter should first show up in a clear correlation between a number of such signatures .another possibility at high energy density is that the intrinsic chiral symmetry of qcd may be restored .this would show up through modifications in the masses of mesons .possibly , non - statistical fluctuations in the distributions of pions , might signal the production of a so - called disoriented chiral condensate " .the major point is not the specific scenarios in such a complex environment , but that these energy densities will bring an unprecedented new regime of matter under experimental scrutiny where our current pictures will necessarily have to undergo radical changes. .most of the tests of the standard model of the strong and electroweak interactions at nuclear energies have involved semi - leptonic interactions , particularly electrons and nucleons , and rare decay modes , primarily of kaons . herewe describe some of the semi - leptonic tests .the study of superallowed fermi beta decays ( parent nucleus of spin / parity daughter ) in the same family of isospin ( isospin multiplet ) permit an important test of the standard model to be carried out via the unitarity of the matrix that describes the weak interaction connecting the various quarks : the cabibbo - kobayashi - maskawa ( ckm ) matrix .the matrix element connecting up and down quarks is by far the largest one in the unitarity of   here and connect the up quark with the strange and bottom quarks , respectively .the precise measurements of superallowed transitions together with radiative corrections and removal of charge dependent nuclear effects allow one to determine to better than .in addition , these measurements , shown in fig .11 demonstrate that cvc holds to .a straightforward analysis of the experiments , including a recent experiment , gives .together with the measurements of and , one then obtains .there remain uncertainties , particularly in the charge - dependent nuclear effects ; it has been proposed that these corrections can be approximated by a smooth z - dependence . in that case .however , better calculations of this charge - dependence remains to be carried out .the decay is expected in the standard model and has been seen in several nuclei ( , , and ) with half lives of about , consistent with the standard model. searches for the no - neutrino decay mode , important for determining whether s are massive and of the majorana ( neutrino and antineutrino are identical ) type , have been continuously improved . the present lower limit on the half life is for .improved experiments hope to find this mode of double beta decay .the finding of strangeness in the nucleon led to considerable experimental and theoretical activity to understand it .recently , in an ongoing experiment ( sample ) at mit , the weak interaction of the electron and proton is being used to investigate strangeness in the nucleon . as in all parity violation experiments , it is the interference of the weak interaction with the electromagnetic one that is being detected by searching for a parity - odd signal such as , where is the incident momentum of the electron and is its polarization .the presence of strangeness in the nucleon can modify the momentum dependence of the form factors , but it can also add two new unconstrained ones , a vector magnetic form factor and an axial ( isoscalar ) form factor .the results to date are inconclusive , but do not indicate any strangeness ( within large errors ) , as shown in fig .further experiments are planned .  other precision parity - violating ( pv ) studies of the weak interactions of electrons and nucleihave been carried out with atoms . despite their being at lower momenta , where the pv effects are smaller , , these experiments have reached the incredible precision of in the parity - violating asymmetry . at the present timetheoretical errors are at the level of . at this level of precisionthe atomic experiments provide meaningful tests of the standard model .the dominant weak interaction term is where the lower case is the axial current of the electron and is the vector current , which is coherent over the nucleus .the effective charge , the weak equivalent of the electrical charge in this case , is where is the weinberg angle , . is large for heavy atoms .the measurement on cs , a one - valence- electron atom , at the level , gives . this limits deviations from the standard model .the term , where is the weak vector current of the electron and is the nuclear axial current , is much smaller than because for the electron and only a single nucleon contributes to , the nuclear spin .thus , the asymmetry is reduced by .the atomic measurements of this term make use of the hyperfine structure , which is due to the nuclear spin .this term has not yet been detected because it is hidden by the stronger nuclear anapole moment , a toroidal axial current of the nucleus coupling to photons .that is , this is a parity - violating coupling of the photon to the nucleus .the recent measurement in atomic cs at the 1/2 level has discovered the nuclear anapole moment , the first anapole moment observed for such a microscopic system .the experiment is of particular interest because it is sensitive to the weak inter - nucleon force caused by neutral currents .this force has not been found in pure nuclear experiments and the upper limit found there ( in ) is at least a factor of three below that deduced from the anapole measurement , and also from a theoretical quark model .it remains to be seen whether this is an experimental or theoretical problem .the nonleptonic weak interaction is of interest because it is the least well understood one due to the strong interactions of all particles involved .the initial knowledge came from the weak decays of strange mesons and baryons .even here , there remains the problem of fully understanding the ratio of parity - non - conserving to parity - conserving amplitudes in the decays of hyperons . due to the change in flavor ,only charged currents contribute to these decays in the electroweak theory . with the advent of precision nuclear experiments it became possible to study the weak interactions of nucleons by means of parity - violating asymmetries with polarized beams .these experiments do not probe the standard model as much as as our understanding of the structure and weak forces of the nucleons . the asymmetry comes about from the interference of the weak and strong forces .experiments in scattering and in light nuclei have provided the most reliable information. the weak neutral currents have yet to be seen in non - leptonic weak interactions .they are the primary source of the isospin - changing = 1 interaction , which arises from exchange with one weak ( ) and one strong pion- coupling to nucleons . since there is only an upper limit on the asymmetry in , which is determined by this mechanism, neutral current effects have not yet appeared .further measurements are planned for , e.g. , in low energy polarized neutron capture by hydrogen , e.g. , .   despite the finding of cp violation in 1964 , over 30 years ago , we still do not have any definitive theory of time reversal noninvariance .this is not due to a lack of effort .the only system where cp violation ( and by implication t violation ) has been found is in the and system . to date , the most sensitive searches for time - reversal - invariance breaking are those for an atomic electric dipole moment of a neutron or atomic  ; these tests are sensitive to simultaneous violations of parity and time reversal invariance .no finite time - reversal violation effect has yet been seen , but the upper limits keep decreasing and have already ruled out some models of cp violation .the development of nuclear physics , particularly in the latter part of the century , has required major investments in facilities for acceleration of the required probing beams and the detection systems required for experiments . while the scale of these has not been the same as those of particle physics , they have been sufficient to require planning and priority choices , carried out primarily through the nuclear science advisory committee , with broad participation by the scientific community .  out of these recommendationsnew facilities have emerged : cebaf and rhic are the two major examples , but a number of smaller facilities have been constructed as part of this planning process ; also substantial u.s .participations in international collaborations have occurred . at the same time , this planning process has also phased out major facilities in order to make new initiatives feasible under a constrained budget .the study of the structure of nuclei , of hadronic matter , started less than 100 years ago .enormous advances have been made in this time . but much work remains .even the simplest building blocks of hadronic matter that we have in our world , the proton , neutron , or pion are structures that are incompletely understood .so are the interactions between them , the quantitative features of the forces that hold nuclei together . the properties of nucleias observed experimentally are understood in the framework of approximate models , but the more fundamental reasons for the validity of many of these models are not well understood nor can we reliably extrapolate these properties to limiting conditions , whether in the limits of stability , or the limiting energy densities of matter .the concerted efforts of nuclear physicists , theorists and experimentalists is needed to pursue these areas of knowledge into the 21st century .the authors would like to thank their colleagues g. bertsch , d. geesaman , w. haxton , r. janssens , g. miller , and r. wiringa for helpful discussions and advice in connection with the preparation of this article .1 the phase shift for the channel . the dot - dash curve is a one parameter fit in chiral perturbation theory at lowest order .the dotted and dashed curves are fits at the next order in the expansion ; the dashed one corresponds to fitting the phase shift between , whereas the dotted one is fit to the scattering length and effective range .the solid line corresponds to the phase shift obtained from a partial wave analysis carried out by the nijmegen group.  fig .2 the results of polarization measurements from electron scattering from the deuteron .the quantity from a variety of experiments is shown by different symbols , with predictions of different theoretical models of the nucleon - nucleon interactions drawn by lines .new measurements from cebaf should help to better distinguish between the models .3 the points show the difference in charge densities between and and show the oscillations due to the radial nodes in the wavefunction of the last proton . the line is a hartree - fock calculation of the same difference .  fig. 4 demonstration of high - momentum components in the nuclear wave function .transition densities from electron scattering knocking out a proton from the single - particle states in the doubly - magic nucleus .the inset shows the spectrum of hole states .the solid line represents the transition density calculated using a mean field for the protons , the dot - dashed curve includes the effects of short - range , high - momentum correlations .  fig .5 a super - deformed " rotational band in showing the gamma - ray transitions between members of the band .the constant increments in gamma - ray energies are characteristic of a band that follows the symmetry of a very good quantum - mechanical rotor .the variation in intensity of these gamma rays reflects the angular momentum distribution in the population of the band in the fusion - evaporation reaction that was used .the peaks that are not indicated by arrows correspond to states of lower energy of normal " deformation , populated after the decay out of the super - deformed band.  fig . 6 similarities between super - deformed bands in nuclei in the same vicinity . the quantity plotted is the mean percentage difference in between different bands . about sixty bands within the mass 150 region and fifty bands within the mass 190 regionare compared .note that in both regions there is a large excess of pairs of bands for which this quantity differs from zero by less than 2% these are the identical bands " others differ by up to 20% .this identical reproduction of bands in different nuclei is not yet fully understood .interaction radii of li ( 3 protons ) and be ( 4 protons ) nuclei with carbon derived directly from total cross section measurements .the line represents a smooth dependence of the radius.  fig .8 the solar neutrino signal expected with three different detection methods that are sensitive to different neutrino energies .the light bars indicate the expected neutrino yield , in appropriate units , from the various sources in the solar cycle .the solid bars represent the observed signals from the corresponding experiments .the shaded areas represent the uncertainties .9 the logarithm of the light intensity from the supernova sn1987a as a function of time after the supernova explosion .most of the intensity for the first two years comes from and its daughter .the calculated contributions from other radioactive nuclei is also shown .10 a qualitative phase diagram for hadronic matter showing the transition to a deconfined quark - gluon state at high temperature and high density .the paths followed in the early universe and in the interior of neutron stars is also indicated . is the density of normal nuclei .collisions in the relativistic heavy - ion collider rhic will explore this phase diagram in detail .11 beta - decay transition probabilities ( ft values ) for nine superallowed fermi -decays and the best least - squares one - parameter fit , plotted as a function of the proton number of the final nucleus.  fig .12 results for the parity - violating asymmetry measured in the sample experiment in the 1995 and 1996 running periods .the hatched region is the asymmetry band ( due to axial radiative corrections ) for .

Abstract: nuclear physics is the branch of physics that deals with the properties and structure of matter on the hadronic level . in this article we review briefly the history of this field , which has a major role in the development of our understanding of nature . we then proceed to give an outline of a current perspective of the field and of some of the issues that are now on its frontiers .


real world problems usually demand continuous state or action spaces , and one of the challenges for reinforcement learning is to deal with such continuous domains . in many problemsthere is a natural metric on the state space such that close states exhibit similar behavior .often such similarities can be formalized as lipschitz or more generally hlder continuity of reward and transition functions .the simplest continuous reinforcement learning problem is the 1-dimensional continuum - armed bandit , where the learner has to choose arms from a bounded interval .bounds on the regret with respect to an optimal policy under the assumption that the reward function is hlder continuous have been given in .the proposed algorithms apply the ucb algorithm  to a discretization of the problem .that way , the regret suffered by the algorithm consists of the loss by aggregation ( which can be bounded using hlder continuity ) plus the regret the algorithm incurs in the discretized setting .more recently , algorithms that adapt the used discretization ( making it finer in more promising regions ) have been proposed and analyzed  .  while the continuous bandit case has been investigated in detail , in the general case of continuous state markov decision processes ( mdps ) a lot of workis confined to rather particular settings , primarily with respect to the considered transition model . in the simplest case ,the transition function is considered to be deterministic as in  , and mistake bounds for the respective discounted setting have been derived in  .another common assumption is that transition functions are linear functions of state and action plus some noise . for such settings sample complexity boundshave been given in  , while bounds for the regret after steps are shown in  .however , there is also some research considering more general transition dynamics under the assumption that close states behave similarly , as will be considered here . while most of this work is purely experimental , there are also some contributions with theoretical guarantees .thus , considers pac - learning for continuous reinforcement learning in metric state spaces , when generative sampling is possible .the proposed algorithm is a generalization of the e algorithm  to continuous domains .a respective adaptive discretization approach is suggested in  .the pac - like bounds derived there however depend on the ( random ) behavior of the proposed algorithm .  herewe suggest a learning algorithm for undiscounted reinforcement learning in continuous state space .the proposed algorithm is in the tradition of algorithms like ucrl2  in that it implements the `` optimism in the face of uncertainty '' maxim , here combined with state aggregation .thus , the algorithm does not need a generative model or access to `` resets : '' learning is done online , that is , in a single continual session of interactions between the environment and the learning policy .for our algorithm we derive regret bounds of for mdps with -dimensional state space and hlder - continuous rewards and transition probabilities with parameter .these bounds also straightforwardly generalize to dimension where the regret is bounded by .thus , in particular , if rewards and transition probabilities are lipschitz , the regret is bounded by in dimension and in dimension 1 .we also present an accompanying lower bound of . as far as we know , these are the first regret bounds for a general undiscounted continuous reinforcement learning setting .we consider the following setting . given is a markov decision process ( mdp ) with state space ^d ] with mean .the transition probability distribution in state under action is denoted by .we will make the natural assumption that rewards and transition probabilities are similar in close states .more precisely , we assume that rewards and transition probabilities are _ hlder continuous_.  [ ass : rew ] there are such that for any two states and all actions ,   [ ass : prob ] there are such that for any two states and all actions ,   for the sake of simplicity we will assume that and in assumptions [ ass : rew ] and [ ass : prob ] are the same .we also assume existence of an optimal policy which gives optimal average reward on independent of the initial state .a sufficient condition for state - independent optimal reward is geometric convergence of to an invariant probability measure .this is a natural condition which e.g. holds for any communicating finite state mdp. it also ensures ( cf .chapter 10 of ) that the poisson equation holds for the optimal policy .in general , under suitable technical conditions ( like geometric convergence to an invariant probability measure ) the _ poisson equation _ relates the rewards and transition probabilities under any measurable policy to its average reward  and the _ bias _ function of . intuitively , the bias is the difference in accumulated rewards when starting in a different state .formally , the bias is defined by the poisson equation and the normalizing equation ( cf .e.g.  ) .the following result follows from the bias definition and assumptions [ ass : rew ] and [ ass : prob ] ( together with results from chapter 10 of  ) .[ ass : mix ] under assumptions [ ass : rew ] and [ ass : prob ] , the bias of the optimal policy is bounded .consequently , it makes sense to define the _ bias span _ of a continuous state mdp  satisfying assumptions [ ass : rew ] and [ ass : prob ] to be . note that since by definition of the bias , the bias function  is upper bounded by  .we are interested in algorithms which can compete with the optimal policy and measure their performance by the _ regret _( after steps ) defined as , where is the random reward obtained by the algorithm at step . indeed , within steps no _ canonical _ or even _ bias optimal _ optimal policy ( cf. chapter 10 of  ) can obtain higher accumulated reward than .our algorithm uccrl , shown in detail in figure [ alg : col ] , implements the `` optimism in the face of uncertainty maxim '' just like ucrl2  or regal  .  state space  ] ,  ] ,  ] , actions , rewards and transition probabilities satisfying assumptions [ ass : rew ] and [ ass : prob ] , and bias span upper bounded by .then with probability , the regret of uccrl ( run with input parameters and ) after steps is upper bounded by therefore , setting gives regret upper bounded by with no known upper bound on the bias span , guessing by one still obtains an upper bound on the regret of .  intuitively , the second term in the regret bound of is the discretization error , while the first term corresponds to the regret on the discretized mdp . a detailed proof of theorem [ thm ] can be found in section [ sec : proof ] below .[ rem : d ] the general -dimensional case can be handled as described for dimension 1 , with the only difference being that the discretization now has states , so that one has instead of in the first term of .then choosing bounds the regret by .if the horizon is unknown then the doubling trick ( executing the algorithm in rounds guessing and setting the confidence parameter to ) gives the same bounds .[ r : uh ] the uccrl algorithm receives ( bounds on ) the hlder parameters as as inputs . if these parameters are not known , then one can still obtain sublinear regret bounds albeit with worse dependence on .specifically , we can use the model - selection technique introduced in . to do this , fix a certain number of values for the constants and  ; each of these values will be considered as a model .the model selection consists in running uccrl with each of these parameter values for a certain period of time steps ( _ exploration _ ) .then one selects the model with the highest reward and uses it for a period of time steps ( _ exploitation _ ) , while checking that its average reward stays within ( [ eq : thm ] ) of what was obtained in the exploitation phase .if the average reward does not pass this test , then the model with the second - best average reward is selected , and so on .then one switches to exploration with longer periods , etc .since there are no guarantees on the behavior of uccrl when the hlder parameters are wrong , none of the models can be discarded at any stage .optimizing over the parameters and as done in  , and increasing the number of considered parameter values , one can obtain regret bounds of , or in the lipschitz case .for details see .since in this model - selection process is used in a `` black - box '' fashion , the exploration is rather wasteful , and thus we think that this bound is suboptimal .recently , the results of  have been improved  , and it seems that similar analysis gives improved regret bounds for the case of unknown hlder parameters as well .the following is a complementing lower bound on the regret for continuous state reinforcement learning .[ thm : lower ] for any and any reinforcement learning algorithm there is a continuous state reinforcement learning problem with actions and bias span satisfying assumption [ ass : rew ] such that the algorithm suffers regret of .consider the following reinforcement learning problem with state space  ] can be written as . sincethe are assumed to be in the same interval , it follows that - r(s , a)| < l n^{-\alpha} ] .together with and this shows that with probability at least for all state - action pairs  this shows that the true mdp is contained in the set of plausible mdps at step with probability at least , just as in lemma 17 of  . the argument that with probability at least then can be taken without any changes from section 4.2 of .now for episodes with , by the optimistic choice of and in  we can bound   any term is bounded according to and , as we assume that , so that summarizing states in the same interval  since , setting to be the length of episode we have      & & + \ ;2l n^{-\alpha } \tau_k + \sqrt { 14\log\left ( \tfrac{2 n a t}{\delta }  \right ) }      \sum_{j=1}^n \sum_{a\in { { \mathcal{a } } } } \frac { v_k(i_j , a)}{\sqrt{\max\{1,n_k(i_j , a)\ } } } \,.\quad  \label{eq : poisson3}\end{aligned}\ ] ]  we continue analyzing the first term on the right hand side of . by the poisson equation  for on , denoting the respective bias by we can write      & = &  \sum_{s } v_k(s,{\tilde\pi_{k}}(s ) )  \big ( \int_{{\mathcal{s}}}{\tilde{p}}_k(ds'|s,{\tilde\pi_{k}}(s ) ) \cdot { \tilde{\lambda}}_k(s ' ) - { \tilde{\lambda}}_k(s ) \big ) \nonumber \\\nonumber \\      & = &  \sum_{s } v_k(s,{\tilde\pi_{k}}(s ) )  \big ( \int_{{{\mathcal{s } } } } p(ds'|s,{\tilde\pi_{k}}(s ) ) \cdot { \tilde{\lambda}}_k(s ' ) - { \tilde{\lambda}}_k(s ) \big ) \label{eq : neu1 } \\      & & + \sum_{s } v_k(s,{\tilde\pi_{k}}(s ) ) \sum_{j=1}^n\int_{i_j } \big ( { \tilde{p}}_k(ds'|s,{\tilde\pi_{k}}(s ) ) -p(ds'|s,{\tilde\pi_{k}}(s ) ) \big ) \cdot { \tilde{\lambda}}_k(s ' ) .\label{eq : neu2}\end{aligned}\ ] ]   now can be bounded by and , because we assume . hence , since by definition of the algorithm bounds the bias function , the term in  is bounded by while for the term in    let be the index of the episode time step belongs to .then the sequence is a sequence of martingale differences so that azuma - hoeffding inequality shows ( cf .section 4.3.2 and in particular eq .( 18 ) in  ) that after summing over all episodes we have where the second term comes from an upper bound on the number of episodes , which can be derived analogously to appendix c.2 of .to conclude , we sum over all the episodes with , using , , and .this yields that with probability at least  analogously to section 4.3.3 and appendix c.3 of , one can show that and we get from after some simplifications that with probability  finally , evaluating by summing over all episodes , by and we have with probability an upper bound on the regret of a union bound over all possible values of and further simplifications as in appendix c.4 of finish the proof .we think that a generalization of our results to continuous action space should not pose any major problems . in order to improve over the given bounds, it may be promising to investigate more sophisticated discretization patterns .the assumption of hlder continuity is an obvious , yet not the only possible assumption one can make about the transition probabilities and reward functions .a more general problem is to assume a set of functions , find a way to measure the `` size '' of , and derive regret bounds depending on this size of .the authors would like to thank the three anonymous reviewers for their helpful suggestions and rmi munos for useful discussion which helped to improve the bounds .this research was funded by the ministry of higher education and research , nord - pas - de - calais regional council and feder ( contrat de projets etat region cper 2007 - 2013 ) , anr projects explo - ra ( anr-08-cosi-004 ) , lampada ( anr-09-emer-007 ) and coadapt , and by the european community s fp7 program under grant agreements n ( pascal2 ) and n ( complacs ) . the first author is currently funded by the austrian science fund ( fwf ) : j 3259-n13 .sbastien bubeck , rmi munos , gilles stoltz , and csaba szepesv ' ari .online optimization of -armed bandits . in _ advances in neural information processing systems 22 , nips 2009 _ , pages 201208 , 2010 .

Abstract: we derive sublinear regret bounds for undiscounted reinforcement learning in continuous state space . the proposed algorithm combines state aggregation with the use of upper confidence bounds for implementing optimism in the face of uncertainty . beside the existence of an optimal policy which satisfies the poisson equation , the only assumptions made are hlder continuity of rewards and transition probabilities