recently people feel data in various fields due to developments in information and communication technology ( ict ) .this circumstance enables us to collect , store , and analyse data on our socio - economic environment in order to interpret our society and to pursue evidence - based management. one can collect and accumulate large amounts of socio - economic data on human activities , and analyse and visualise them .vast amounts of data collected from socio - economic systems have made new types of commercial services and research fields emerging . according to m.f .goodchild  , every human is able to act as an intelligent sensor , and in that sense , the earth s surface is currently occupied by more than six billion sensors .we can extract information from an amount of data , construct knowledge from lots of information , and hopefully establish wisdom from several pieces of knowledge .specifically , researchers in the fields of sociology , economics , informatics , and physics are focusing on these frontiers and have launched data - centric social sciences in order to understand the complexity of socio - economic systems .  however , since our society which is the sum total of both internal and external states of individuals is several orders of magnitude more complicated than each individual , it seems difficult to image how we could manage to capture the real totality of its states with the cooperation of many agencies .the nature of this problem is referred to recognised as `` complexity '' , which is a new research field into knowledge on groups of people , organisations , communities , and the economy actually behave in the real world .information theory describes the number of possible states of a system as its `` complexity '' .complexity may be estimated with shannon information entropy or thermodynamic entropy .for example , researchers have used a methodology to characterise the structure of networks with information - theoretic entropy  . according to a study by dehmer and mowshowitz  ,the concept of graph entropy was first proposed in the 1950s to measure structural complexity .rashevsky  , trucco  , and mowshowitz  were the first researchers to define and investigate the entropy of graphs .several graph invariants , such as the number of vertices , vertex degree sequence , and extended degree sequence , have been used in the construction of entropy - based measures .wilhelm an hollunder considered the normalised weight of the flux between two nodes as the probability of a symbol in the transmitter signal that corresponds to the sum of all influxes to / effluxes from a given node  .sato also considered information - theoretic measures for a bipartite graph , and inferred economic situations using the network entropy of relative frequencies among group populations  .bianconi looks at the entropy measure of network ensembles under several constraints  .  according to heinz von foerster  ,complexity is not a property which observed systems possess rather , it is to be perceived by observing systems .he asks us about it through the following question : _ are the states of order and disorder states of affairs that have been discovered , or are their states of affairs that are invented ? _ if states of order and disorder are discovered , then complexity is a property of the observed systems .if invented then it pertains to the observing systems .foerster s definition of complexity proposes that the relative degree of order and disorder is determined by the degrees of freedom of an observed system and an observing system .one of the most significant reasons that we recognise complexity in observed systems is because of the finiteness of periods when and abilities with which we are able to observe the systems , and the limitations of our memory and a priori knowledge of them that lead to our bounded rationality , or nescience .  from the data - centric perspective , complexityis referred to as a ratio of an amount of data to human cognitive capacity .how many megabytes of memory the human brain has ?von neumann proposed that human memory is estimated as 100 exabits from all the neural impulses conducted in the brain during a lifetime  .another method is to estimate the total number of synapses , and then presume that each synapse can hold a few bits .estimates of the number of synapses have been made in the range from to , with corresponding estimates of memory capacity .landauer investigated how much people remember at bell communications research  .the remarkable result of this work was that human beings remembered very nearly two bits per second under all the experimental conditions ( visual , verbal , musical ) .therefore , the 35-year accumulation of human beings memory estiamtes of 0.2 to 1.4 gigabits if the loss of memory are assumed .if the amount of data is greater than this estimates of memory , then all the data can not be memorised by each human being . therefore, the amounts of data may determine the complexity in the sense of the data accessiblity .this may provide a kind of definition of big - data in data - centric sciences .clearly , we also need to carefully consider methods of collecting , storing , handling , and analysing a vast amount of data in computer systems .  in this article ,i present exemplar studies of observation and data analysis with a large amount of data in several socio - economic systems .i would like to call such studies `` econoinformatics '' .econoinformatics needs powerful computer systems to visualise and quantify the behaviour of human beings , adapt to their changing environments , and evolve over time .furthermore , we should mention the problem of data linkage in data - centric studies .data linkage is referred as to a method of linking data from different sources with the same elements .if we can synthesise data from different sources , we may find new insights from the synthesised data .the comprehensive study is further useful to obtain new findings on our environment .the literature review on data - centric social sciences is provided in section [ sec : literature ] of this article . in section[ sec : exemplar ] exemplar studies on econoinformatics are shown . the japanese stock exchange , the foreign exchange market , hotel booking data , and flight booking data are analysed . in section [ sec : discussion ] , a method of synthesising data from different sources , and a method of econoinformatics , are discussed .section [ sec : conclusion ] is devoted to conclusions .recently , several researchers in a wide spectrum of fields have paid a remarkable amount of attention to massive amounts of comprehensive data . for example , search engines of web services need massive data about hyperlink connections among web pages , and electronic commerce systems need to cover information of various kinds of products . due to the development of ict ,the advanced information society has already emerged globally and it has gradually made our world smaller and smaller .the term `` information explosion '' has been coined to describe this situation and has been realised  .this term refers to a situation in which the total amount of information created by individuals exceeds the individuals information processing capability .studies with vast amounts of socio - economic data have several branches . here, five kinds of recent studies ( financial market data , demographic data , traffic flow data , pos data , and e - commerce data ) are surveyed for the purpose of finding ways of coping with the complexity of human societies .a large amount of data on financial markets is available because the electronic matching systems of financial markets are spreading all over the world due to the development of ict .contemporary trading is done through electronic platforms , and settlement operations are done through electronic clearing systems .financial market data can be collected through a direct application programming interface ( api ) or through the historical data centres of data providers .applications of statistical mechanics to finance , by means of statistical physics , agent - based modelling , and network analysis , have progressed during the last decade  .the launch of the e - stat database by the japanese government  provides us with new technological means for a data - based understanding of our country . in principle, everyone can understand the state of our country from demographic data .furthermore , real - time demographic data are also available since the technologies to collect human activities via personal mobile phones have been established  . in the near future , we will be able to visualise real - time demographics , both comprehensively and circumstantially .recently , several car navigation companies have launched autonomous sensory navigation services in japan . as a result, these companies can collect real - time car traffic data via each car navigation terminal . by collecting data from many cars , one can find roads and points where traffic jams are occurring . without constructing new infrastructure to collect traffic states ,real - time traffic data can be accumulated due to the development of integrated transport systems ( its ) .based on such data , comprehensive analyses of traffic flows can be conducted in order to cope with traffic jams  .recent developments in traffic measurement technologies have been driving the theoretical development of traffic control and modelling  .pos is an abbreviation for `` point - of - sales '' , and all department stores and supermarkets have introduced this kind of system in order to ring up purchases at cash registers . as a result , retail sales can be managed in real - time , and data - centric operations can be done .on the basis of these massive amounts of data , new marketing methods have been developed .the statistical properties of expenditure in a single shopping trip show a power - law distribution  .a comprehensive analysis of retail sales is one of the most promising directions to be followed in order to bridge the gap between microeconomics and macroeconomics .web - based commerce systems enable us to purchase everything , from books to electronic equipment , via websites .the details of consumers and goods can be stored on the data - base engine of each website .if we can use such data , then we may , in principle , capture real - time demand and supply of all items which are traded via websites .analysing massive amounts of data on items which are sold via web commerce systems is expected to open a window to new economic theory and service engineering  .data on hotel booking opportunities  , international flight booking opportunities  , and price comparison sites  has also been studied .the property that these studies seem to have in common is their ability to overcome the complexity in socio - economic systems , by using massive amounts of data and vast computations .copious amounts of data on human activities are collected by means of ict , and vast amounts of computation for such data are conducted for the purposes of searching , matching , visualising , and extracting .large - scale data on socio - economic systems may make it possible to understand our society from a holistic point of view .to do so , we need to discuss the possibilities of comprehensive analysis and data linkage . in this section ,four types of exemplar studies on socio - economic systems are shown .these examples include studies on the japanese stock exchange , the foreign exchange market , hotel bookings through an e - commerce platform , flight bookings through an e - commerce platform . in order to archive a comprehensive point of view ,both the japanese and international economic situations are measured .   during the last two decades, it is said that the main problem faced by japan is a low productivity growth rate . despite these circumstances ,the japanese economy recovered eventually , with the aid of the global growth from 2004 to 2007 .however , the latest global financial crisis strongly affected many countries , including japan , after the midquarter of 2008 . at that time , we observed that japanese stock prices dropped sharply due to financial turmoil of 2008 - 2009 .furthermore , huge earthquakes hit japan on 11 march 2011 , and the japanese economy suffered from a large number of social losses .so far , we have explained the affairs by the japan explanatory narrative .the question remains , however , as to how we can describe our macroeconomics situations in a more quantitative fashion .macroeconomic situations strongly influence money flows at all levels of society .stocks at each sector are traded by investors and traders every minute through the stock exchange market. moreover , stock prices are so sensitive to money flows , that stock prices in all sectors reflect demand - supply gaps of the money by economic actors .therefore , they are expected to be useful for detecting changes in macroeconomic situations .  in this study, we hope to provide some insights on the problem of the quantification of japanese macroeconomic situations , through a comprehensive analysis of stock prices traded in the tokyo stock exchange . in the context of economics and finance ,there are various methods available for segmenting highly nonstationary financial time series into stationary segments , called regimes or trends . following the pioneering works of goldfeld and quandt , there is an enormous body of literature on detecting structural breaks or change points separating stationary segments .recently , a recursive entropic scheme to segment financial time series was proposed  .  in this study ,a recursive segmentation procedure is applied to an analysis of security prices of 1,413 japanese firms listed on the first section of the tokyo stock exchange .the number of segments in quintiles in terms of variance is computed in order to detect change points of money flows of the japanese security market .let and be daily opening and closing prices of -th stock at day . and are denoted as the total number of stock and the observation length .the daily log - return ( opening to closing ) time series is computed as and .  according to the seminal work by mantegna and stanley , the log - return time series of stock prices are modelled by lvy distributions .superstatistics suggested that a mixture of gaussian distributions with -distributions in terms of variance gives a lvy distribution .therefore , we assume that each segment is sampled from a gaussian distribution with different mean and variance .it is assumed that the log - return time series in segment follows a stationary gaussian distribution with mean and variance .  to find the unknown segment boundaries separating segment and , the recursive segmentation scheme introduced by bernaola - galvn et al  is employed . in this segmentation scheme ,the likelihood of the total time series is compared with the likelihood of the two segmented time series .suppose that there are observations .let be a gaussian distribution : , \label{eq : gauss}\ ] ] assuming that the observations should be segmented at , and that the observations on the left hand side are sampled from , and that those on the right hand side are from , we define likelihood functions : furthermore , we define the logarithmic difference between and as , inserting eq .( [ eq : gauss ] ) into eq .( [ eq : def - delta ] ) , we have ,   in general , if a random varialbe is given and its distribution admits a probability density function , then the expected value of ( if exists ) can be calculated as = \int_{-\infty}^{\infty}a f(a)\mbox{d}a,\ ] ] where is a probability density function of .this implies that the arithmetic mean of random variables is approximated as for a sufficiently large value of .  by using this approximation , for observations , we obtain and is rewritten as ,   can be used as an indicator to separate the observations into two parts . an adequate way to separate the observationsis that a segmentation is conducted at where takes the maximum value .namely , an adequate segmentation should be done at , if is less than a threshold value , then the segmentation should be terminated .the hierarchical segmentation procedure is also applied to the time series .after segmentation , we also apply this procedure for each segment recursively . in order to stop the segmentation procedurewe assume that the minimum value .if is less than , then we do not apply the segmentation procedure any more .this is used as the stopping condition of the recursive segmentation procedure .1,413 companies listed on the first section of tokyo stock exchange are selected for empirical analysis .the duration is 4 january , 2000 to 30 january , 2012 .these companies maintain during the observation period .the recursive segmentation procedure was applied to 1,413 security prices .the segmentation analysis of daily log - return time series for ending prices was conducted . throughout the investigation fixed as 10 .[ fig : count ] shows the number of starting dates of segments for 1,413 log - return time series .the number of segments increase at june 2001 , april 2004 , february 2006 , and 2007 to 2009 .these seem to correspond to regimes or change points in the japanese economy .specifically , during the last global financial crisis the number of segments tends to increase ( about 260 segments can be found at this period ) .furthermore , after 11 march , 2011 , the great east japan earthquake , the number of segments steeply increased larger than during the last global financial crisis .    the number of segments belonging to the same quintile of variances is counted .compute order statistics of variance in all the segments of stock .next , each segment of stock is labelled , depending on variance which belongs to the quintile .the number of segments which have the same labels is counted on each day .[ fig : quintile2 ] shows the number of segments belonging to each quintile on every day .the number of first quintile segments shows stability of economic affairs , and the number of fifth quintile segments indicates instability of economic affairs .it is found that from 2003 to 2007 , the japanese economy was in a stable regime . from the end of 2007 an unstable regime was observed .specifically , during september 2008 , when we experienced the lehman shock , the number of fifth quintile regimes steeply increased .this implies that the money flows of the japanese economy became unstable just after the lehman shock . from march 2009 ,the money flow eventually recovered , and the number of unstable regimes decreased while the number of stable segments eventually increased . from 11 march to 10 april , 2011 the number of unstable regimes steeply increased due to the great east japan earthquake . however , the number of stable regimes rapidly increased thereafter , and japanese macroeconomic affairs have been eventually recovered .     the foreign exchange market is the largest financial market in the world .the foreign exchange market is organised by international banks and funds through brokerage platforms .since the currency exchange strongly reflects and is related to international economies , we can predict global economic conditions from the foreign exchange market .the analysis is conducted using high - resolution data ( icap ebs data mine level 1.0 ) collected using the icap ebs platform .the data includes currency pairs , quotation prices , and transaction prices with a one - second resolution . in the exchangeable currency pairs , consisting of 41 currencies and 11 precious metals , 93 kinds of currency pairs are included during the period from june 2008 to june 2012 .there are about 520 million records in the observation period .[ fig : network ] shows exchangeable currency pairs .usd and eur have a lot of links to other currencies .aud , cnh , chf , jpy , and gbp have five to eight links .others have only one or two links to major currencies .suppose one can observe quotations / transactions about the -th currency , and the -th currency and count the arrival of quotations or occurrence of transactions for each currency pair on a brokerage system with an interval of .this activity is defined as the number of quotations / transactions which market participants enter into the electronic brokerage system per .we define as the quotation activities between the -th currency and -th currency ( ) in \quad ( t=1,2,\ldots , t)$ ] on the -th observation period . in this analysis, we adopt the definition that the activities should be counted in symmetrical manner and satisfy no self - dealing condition .then , the density of quotations between the -th currency and -th currency can be estimated as , obviously , it has probabilistic properties , such that , , , and .  under the assumption that the attention of market participants to the exchangeable currency pairs can be estimated as the centrality of currency pairs , can be empirically estimated by using quotation / transaction frequencies calculated from high - resolution data without knowledge of the network structure of market participants . moreover ,relative occurrence rates of the -th currency on the -th observation period are defined as , where it also has probabilistic properties , such that , and . sinceboth and may be regarded as fingerprints representing the market states on the observation period , their shape may describe market states at . to capture the difference of shapes between and , jensen - shannon divergence  is employed . where and are , respectively , denoted as the shannon entropies defined as , from these definitions , we can confirm that they have the following properties :   the similarity of market states between two observation periods is computed for each week .the jensen - shannon divergence is employed in order to compute similarities since it has resistance characteristic for zero probabilities .this means that the shannon entropy does not diverge to infinity because of , which is proven in [ sec : zero - log - zero ].  figs .[ fig : dp ] and [ fig : dd ] show similarities calculated from of quotations ( a ) , of quotations ( b ) and similarities obtained from of transactions ( a ) , and of transactions ( b ) , respectively .( a ) ( b )  ( a ) ( b )  from these figures , there are regimes where shares are equivalent .macroeconomic situations may correspond to these regimes .for example , there are three regimes for the period from january 2008 to june 2009 as shown in fig .[ fig : dp ] .these regimes are related to the last global financial crisis .furthermore , we can confirm the euro shock which happened march to may 2010 .we can capture both the past and present circumstances of the international economy from the high - resolution data , and may predict or infer the future affair .it is also found that there is an extreme regime in 14 to 21 march , 2011 .this event was observed with the sudden increase of jpy in relation to other currencies ( specifically usd and eur ) .this chaos was triggered by the devastation caused by the great east earthquake of japan and the subsequent tsunami .recent technological developments enable us to purchase various kinds of items and services via e - commerce systems .the emergence of internet applications has had an unprecedented impact on our ability to purchase goods and services . from the data available about items and services on e - commerce platforms, we may expect that utilities of agents in socio - economic systems are directly estimated .it is found that it is becoming more popular to make reservations of hotels via the internet .when we use a hotel booking site , we notice that we sometimes find preferable room opportunities . in other words ,hotel vacancies seem to be random .we further know that both the date and place of stay are important factors to determine the availability of room opportunities .hence , room availability depends on the calendar ( weekdays , weekends , and holidays ) and regions .this availability of hotel rooms may indicate future migration trends of travellers .therefore , it is worth considering the accumulation of comprehensive data of hotel availability in order to detect inter - migration within countries .  here, we give a brief explanation of a method for collecting data on hotel availability . in this study, we used a web api in order to collect the data from a japanese hotel booking site named _jalan_. the data is provided by the jalan web service  .jalan is one of the most popular hotel reservation services in japan .the api is an interface code set designed for the purpose of simplifying the development of application programs .the jalan web service provides interfaces for both hotel managers and customers .the mechanism of jalan is as follows : hotel managers can enter information on room vacancies at their hotels via an web interface .the consumers can book rooms from available opportunities via the jalan web site .third parties can even built their web services with jalan data by using the web api .we collected all available opportunities which appeared on jalan web regarding one - night room vacancies for two adults .the data were sampled from the jalan web service daily .the data on room opportunities collected through the jalan web api are stored as comma separated value ( csv ) files .  in the data set , there exist over 100,000 room opportunities at over 14,000 hotels . in table[ tab : data ] , we show contents included in the data set .each plan contains sampled date , stay date , regional sequential number , hotel identification number , hotel name , postal address , url of the hotel website , geographical position , plan name , and rate .since the data contains regional information , it is possible for us to analyse regional dependence of hotel rates . throughout the investigation , we regard the number of recorded opportunities ( plan ) as a proxy variable of the number of available room stocks ..the data format of room opportunities .[ cols= " < " , ]   we can confirm that house damage was serious in sanrikukaigan , sendai , matsushima , shiogama , ishinomaki , kesennuma , soma , koriyama , iwaki , and futaba .the greatest number of completely - destroyed houses is 33,661 in ishinomaki and kesennuma .the second is 21,789 in sendai .the third is 18,098 in sanrikukaigan .the greatest number of partially - destroyed houses is 37,522 in sendai .the second is 17,614 in iwaki and futaba .the third is 12,185 in koriyama .in fact , in places where the ratio is greater than 1 , the number of destroyed houses is not significant , as shown in figure [ tab : district ] .we confirmed that the ratio may measure the degree of damage to economic activity in the travel industry .however , it is not confirmed that there was significant physical damage to houses in oushu , hiraizumi , ichinoseki , fukushima , and nihonmatsu , even having a ratio less than 1 .it may be thought that hotels in oushu , hiraizumi , and ichinoseki were used by workers and evacuated victims of the disaster .decreases of available hotels in fukushima and nihonmatsu may be related to accidents in fukushima daiich nuclear power plant .the number of victims evacuated from the disaster in each prefecture , according to an official announcement by the japanese cabinet office on 3 june 2011 , is shown in table [ tab : victims ] . in the case of fukushima prefecture , 17,874 people were evacuated to hotels at that time .ab - road  is a japanese internet travel booking site .about 14,000 flights are available on this site every day .this booking site serves as a web api for both travel agencies and customers . on the one hand, travel agencies can register their flight opportunities on the site via the internet . on the other hand, consumers can search for and book flights that they want to purchase from all the registered flights via the website .third parties can even build web services with the data provided by the web api .i collected information regarding available flight tickets using the ab - road web service every day and stored it as comma - separated ( csv ) files .this data set contains the flight tickets that a person would be able to use to depart from one of the airports in japan .each flight also contains the date when the data was sampled , departure date , departure airport , arrival airport , type of class ( economy , business , and first ) , name of air carrier , and price ( the fuel surcharge and tax are excluded ) .the data period is from 29 july , 2010 to 14 december , 2011 . for technical reasons , data on several dates is missing ( 8 november , 2010 , 10 april , 2011 , from 14 to 25 april , 2011 ) .we use data of opportunities where flights would depart from some airports in japan 4 weeks later .the supply - demand situation of air travels may be related to the number of opportunities .the price of flight tickets between departure and arrival airports is also dependent on their distance .fundamentally , i investigates current situations of international air flights departing from japanese airports . in the dataset, there exist about 14,000 kinds of flight opportunities for about 78 airline companies  every day .the total number of available flights from a city in japan to a city in a foreign country was totalled from the data throughout the entire sampled period .figure [ fig : flightnumber ] shows the total number of available flights per day . from observing this data , we made three observations :  \(1 ) there exists weekly seasonality for the total number of available flight tickets . the demand of flight tickets is higher on sundays and mondays than on other days .\(2 ) the number of flight tickets strongly depends on the japanese calendar .namely , summer holidays influence the reservation activities of consumers .for example , during the golden week holidays ( from 1 to 5 may , 2011 ) and the holidays in the spring season ( around 20 march , 2011 ) , total availability shows steep decreases .\(3 ) since several airline companies update their flight schedule every april and october , ticket availability drastically drops at that time .recently , the airport network has been studied by several researchers  . according to the study by guimer and amaral , the world - wide airport network has properties of a small - world network . the degree and betweenness centrality distributions exhibit power - law decay .in fact , the most connected cities ( largest degree ) are typically not the most central cities ( largest betweenness centrality ) .airports with high betweenness tend to play a more important role in keeping networks connected than those with high degree , because a passenger can travel from a departure airport to a destination with a short path .the geodesic distance between departure and arrival airports may give a good approximation of the actual flight distance of passengers .the geodesic distance is measured by vincenty s formulae .let , , and be the geographical latitude and longitude of two points , respectively , and . under the assumption that the earth is a sphere , the distance is given by , where represents the earth s radius ( [ km ] ) .it is possible to analyse the geodesic dependence of ticket prices with this data .figure [ fig : flight ] shows the relation between the price of economy - class flight tickets and the geodesic distance from the departure airport to the destination .the distance of each flight ticket is computed from the geographical latitude and longitude of the departure and arrival airports using equation ( 1 ) .[ fig : flight ] represents the relationship on 4 august , 2010 ( high demand season ) and on 16 october , 2010 ( low demand season ) .short - distance corresponds to flights to asian cities ( 1,000 km to 3,000 km ) , middle - distance to cities in europe and north america ( 8,000 km to 10,000 km ) , and long - distance to cities in south america ( 15,000 km to 20,000 km ) . during high demand season, it is found that various kinds of flights appear for both short - distance and long - distance flights , but , during low demand season , there are few long - distance flights .( a ) ( b )it was shown from looking at these exemplar studies that there are two kinds of axes with which to best represent our society .the studies of the japanese stock exchange market and the foreign exchange market show that different stocks or currencies are quoted or traded concurrently , and that they are mutually related .namely , time is one of important key factors to link data observed in different places .investigation of both the hotel and international flights showed that geographical information is also a key factor to link data from different layers .since events in the world happen in massively concurrent , time and locations are the most important elements to link results obtained from different data sources .furthermore , we may synthesise data from different data sources with a time and location , and construct a large - scale database .i also want to say that these studies are of econoinformatics .econoinformatics includes several research topics such as mathematical concepts , algorithms , computer systems , databases , modelling , and so forth .mathematical concepts may help us to develop automated algorithms to extract knowledge on socio - economic systems .the algorithm and computation systems help us to collect , store and analyse large amounts of data .it is also crucial to construct synthesised data sets from uncensored data obtained from different data sources .furthermore , mathematical models will become a guide for us to understand observed systems connecting physical mechanisms with observations .figure [ fig : system ] shows a conceptual illustration of a research platform .this consists of data crawlers , cluster computers , database servers , management servers , file servers , and consoles .the data crawlers collect socio - economic data from e - commerce platforms in the internet .the data is stored in the file servers . after verifying the data ,the data is synthesised in the database servers . to construct the synthesised database ,the data is stored with the time and/or locations .the management servers handle this procedure of collecting , verifying , and storing data automatically . in the cluster computers ,the socio - economic data is computed and analysed .researchers use this system from the consoles .the concept of complexity was discussed from several points of view .complexity is determined by the relative degrees of freedoms of observing systems and of observed systems .recent exemplar studies on data - centric socio - economic systems were reviewed and presented .the japanese stock exchange , foreign exchange market , domestic hotel bookings , and international flight bookings were analysed from a comprehensive point of view .econoinformatics can be established from data - centric studies based on large amounts of data on socio - economic systems .such data can be obtained from our society . to overcome the complexity of socio - economic data, we need to develop mathematical concepts , algorithms , rich computer systems , and synthesised databases . to construct synthesised data on socio - economic systems ,spatial - temporal axes should be employed .various kinds of data collected from different paths may be synthesised with time and locations .rich synthesised databases representing aspects of our society may help us to obtain deeper insights into our own socio - economic systems .the author is thankful to prof .thomas lux , prof .burda zdzisaw , prof .janusz hoyst , and prof .dirk helbing for stimulating discussions , and to ms .youko miura ( ab - road ) , mr .kotaro sasaki ( jalan ) and mr .daichi tanaka ( jalan ) , mr .hiroshi yoshimura ( jalan ) for providing useful information on air travel and the domestic hotel industry .this work was partially supported by the grant - in - aid for young scientists ( b ) ( # 23760074 ) by the japanese society of promotion of science ( jsps ) .99 goodchild m f 2007 _ citizens as voluntary sensors : spatial data infrastructure in the world of web 2.0 _ _ international journal of spatial data infrastructures research _ * 2 * 24 eu fp-8 fet flagship : futurict http://www.futurict.eu dehmer m and mowshowitz a 2011 _ a history of graph entropy measures_ _ information sciences _* 181 * 57wilhelm t and hollunder j 2007 _ information theoretic description of networks _ _ physica _ a * 385 * 385 rashevsky n 1955 _ life , information theory , and topology _ _ bull . math . biophys . _* 17 * 229 trucco e 1956 _ a note on the information content of graphs _ _ bull . math .biophys . _ * 18 * 129 mowshowitz a 1968 _ entropy and the complexity of graphs : i. an index of the relative complexity of a graph _ _ bull . math . biophys . _ * 30 * 175 sato a h 2010 _ comprehensive analysis of information transmission among agents : similarity and heterogeneity of collective behaviour _ _ agent - based in economic and social systems vi : post - proceedings of the aescs international workshop 2009 , agent - based social systems 8 _ ed s .- h .chen et al .( tokyo : springer ) pp 117 .g 2009 _ entropy of network ensembles _ _ phys ._ e * 79 * 036114 anand k and bianconi g 2009 _ entropy measures for networks : toward an information theory of complex topologies _ _ phys ._ e * 80 * 045102 foerster v h 1984 _ observing systems _( california : intersystems publications ) von neumann j 1958 _ the computer and the brain _ ( new haven / london : yale univesity press ) landauer t k 1986 _ how much do people remember ? some estimates of the quantity of learned information in long - term memory __ cognitive science _ * 10 * 477 new it infrastructure for the information - explosion era ( july 2005march 2011 ) url : http://www.infoplosion.nii.ac.jp/info-plosion/ctr.php/m/indexeng/a/index/ korth h f 1997 database research faces the information explosion communications of the acm * 40 * 139 takayasu h 2002 _ the advent of econophysics _ ( tokyo : springer ) mantegna r n and stanley h e 2000 _ an introduction to econophysics : correlations and complexity in finance _ ( cambridge : cambridge university press ) sornette d 2003 _ why stock markets crash : critical events in complex financial systems _ ( princeton : princeton university press ) sato a h and hoyst j a 2008 _ characteristic periodicities of collective behavior at the foreign exchange market _b * 62 * 373 portal site of official statistics of japan by national statistics center : http://www.e-stat.go.jp gonzlez m c , hidalgo c a and barabsi a l 2008 _ understanding individual human mobility patterns _ _ nature _ * 453 * 479 antoniou c , ben - akiva m and koutsopoulos h n 2006 _ dynamic traffic demand prediction using conventional and emerging data sources _ _ iee proc .syst . _ * 153 * 97 ed helbing d 2007 _ managing complexity : insights , concepts , applications _( berlin : springer ) mizuno t , toriyama m , terano t and takayasu m 2008 _ pareto law of the expenditure of a person in convenience stores _ _ physica _ a * 387 * 3931 lambiotte r and ausloos m 2005 _ endo - vs .exogenous shocks and relaxation rates in book and music sales _ _ physica _ a * 362 * 485 deschtres f and sornette d 2005 _ dynamics of book sales : endogenous versus exogenous shocks in complex networks _ _ phys ._ e * 72 * 016112 sato a h 2012 _ patterns of regional travel behavior : an analysis of japanese hotel reservation data _ _ int .rev . of finan .* 23 * 55 sato a h 2012 _ japanese international air travel : the relationship between flight ticket price and geodesic distance _ _ proc .2012 ieee world congress on computational intelligence ( brisbane ) _ pp 2821 - 2826 mizuno t and watanabe t 2010 _ a statistical analysis of product prices in online marketb * 76 * 501 goldfeld s m and quandt r e 1973 _ a markov model for switching regressions _ _ j. econometrics _ * 1 * 3 cheong s a , fornia r p , lee g h t , kok j l , yim w s , xu d y and zhang y 2012 _ the japanese economy in crises a time series segmentation study _ _ economics e - journal _ * 6 * 2012 - 5 , www.economics-ejournal.org bernaola - galvn p , romn - roldn r and oliver j l 1996 _ compositional segmentation and long - range fractal correlations in dna sequences _ _ phys_ e * 53 * 5181 lin j 1991 _ divergence measures base on the shannon entropy _ _ ieee trans .theor . _ * 37 * 145 the data is downloaded from jalan web service : http://www.jalan.net the data is downloaded from a web page of national research institute for earth science and disaster prevention : http://www.j-risq.bosai.go.jp/ndis/ ( 31 aug 2011 ) .the data is downloaded from ab - road web service : http://www.ab-road.net guimer r and amarall a n 2004 _ modeling the world - wide airport network _ _ eur .phys . j. _b * 38 * 381 guida m and maria f 2007 _ topology of the italian airport network : a scale - free small - world network with a fractal structure ? _ _ chaos , solitons and fractals _ * 31 * 527 bagler g 2008 _ analysis of the airport network of india as a complex weighted network _ _ physica _ a * 387 * 2972let us consider putting one has by using the taylor expansion of , one obtains therefore , we gets

Abstract: our society has been computerised and globalised due to emergence and spread of information and communication technology ( ict ) . this enables us to investigate our own socio - economic systems based on large amounts of data on human activities . in this article , methods of treating complexity arising from a vast amount of data , and linking data from different sources , are discussed . furthermore , several examples are given of studies into the applications of econoinformatics for the japanese stock exchange , foreign exchange markets , domestic hotel booking data and international flight booking data are shown . it is the main message that spatio - temporal information is a key element to synthesise data from different data sources .


the earth s magnetic field is generated by motions of an electrically conducting fluid in the outer core , the bulk being liquid iron .the processes include magnetic induction and are subsumed under the term ` geodynamo ' .it is generally accepted that the fluid motions in the outer core , which are most important for maintaining the geodynamo , are driven by convection , i.e. by thermal and compositional buoyancy forces .there are in general three ways to study the dynamics of the outer core .the first builds on theoretical considerations like force balances and thermodynamics ( e.g. * ? ? ?secondly , it is possible to model the whole system numerically on the basis of the fundamental physical equations . finally , laboratory experiments analogue to the processes proposed for the earth s core can help to determine certain aspects of the dynamics . in this paper ,we focus on the second approach .+ an important part of the increase in knowledge about core dynamics in the last two decades came from numerical simulations of the dynamo process . starting from the first successful 3d mhd self - sustained dynamo models of and , numerical dynamo simulations have been able to reproduce various features of the geomagnetic field such as field morphology , secular variations and polarity reversals . the problem , however , remains how to apply results from numerical simulations to the earth .+ a major challenge is the discrepancy between numerical models and the core in terms of the non - dimensional parameters defined in table [ tab : nondim_no ] .specifically , numerical dynamos have far too slow rotation ( ekman number too large ) , are less turbulent ( rayleigh number too small ) and excessively viscous relative to their electrical conductivity ( magnetic prandtl number too large ) compared to the core .this gap can not be bridged easily due to the enormous computational power required to resolve all relevant time and length scales .+ one way of using present - day numerical dynamo simulations to estimate quantities that are relevant to earth s core ( e.g. heat flux , flow velocity , magnetic field strength ) is to extract scaling laws between these quantities and other characteristic parameters from the data . assuming that the relevant processes in the core are the same as in our simulations , we may extrapolate the results to the parameter regime of the core and in that way gain insight into the processes in earth s core .+ this has been done for various quantities .important results were the diffusivity - free scalings of heat transport , flow velocity and magnetic field strength and simple scalings for the magnetic dissipation time .the question arises , however , how complex a model needs to be in order to do justice to the data .+ we address the classical problem of model selection , where a model is defined in terms of a number of parameters . on the one hand ,the paradigm of occam prefers a model that is less complex over another that is more complex ( when both fit the data equally well ) , generally meaning that the former model contains the fewest parameters of all models .but what is often not recognised , and is equally important , is that models with fewer parameters can have greater predictive power than more complicated models .physical theories are not only validated by their fit to existing data , but even more by their performance in predicting new data .a few words are in order to motivate why this phenomenon is true . + we imagine a noisy dataset with points and fit it with parameters ; we begin by taking to achieve a perfect fit to our data . because of noise, this model is extremely complex , containing high frequency oscillations ( in the case of a function fitted to points distributed in ) .imagine now receiving a new datum .the parameter model will have almost no predictive power for this new datum , since it has fitted all of the noise in the dataset from which it was derived .indeed , a far simpler model , with will have far greater predictive power .we use this principle by implementing a procedure called ` leave - one - out cross validation ' , where we systematically omit one of the data points and hold it in reserve as a test point , against which different models can test their predictive power . in this way we evaluate the predictive power of models , and find models based on an optimal number of parameters that have the most predictive power . +the format of the paper is as follows : in section [ sec : dynamo_dataset ] , we present the database used in our analysis . in section[ sec : cross_validation ] , we illustrate the method of cross - validation with a toy problem , before going on to apply it to the dynamo problem at hand .subsequently , we analyse the scaling laws for heat transport , flow velocity and magnetic field strength using diffusivity - free parameters ( section [ sec : diff_free_scal ] ) and traditional non - dimensional numbers ( section [ sec : trad_scal ] ) . section [ sec : mag_dissipation_core ] is concerned with the scaling of magnetic dissipation time as well as the application of the scalings to the core .in the numerical dynamo simulations used in this study , convection is driven by a fixed superadiabatic temperature contrast between inner and outer boundaries of a rotating spherical shell .moreover the boussinesq approximation is used , i.e. density variations enter the equations only through a buoyancy term in the momentum equation .the standard set of equations consists of five equations describing conservation of momentum ( navier - stokes equation ) , magnetic induction , the transport of temperature and the solenoidal nature of the magnetic field and the velocity field ( cf .eq . [ eqn : ns_eq]-[eqn : solenoidal_u ] ) . + these equations can be non - dimensionalised by introducing four independent control parameters .their choice is not unique .we follow and use the shell thickness of the outer core , the inverse rotation rate , the temperature difference , and the quantity as fundamental scales for length , time , temperature and magnetic field , respectively ; is the outer core radius , the inner core radius , density and magnetic permeability .this leads to the following set of non - dimensional equations for magnetic field , fluid velocity and temperature :   where is the unit vector in the direction of the rotation axis . in these equations, gravity is assumed to vary proportional to the radius , being the value of gravity at the outer boundary ; volumetric heating is neglected and is the non - hydrostatic pressure .the four non - dimensional parameters governing equations [ eqn : ns_eq]-[eqn : solenoidal_u ] are defined in table [ tab : nondim_no ] .    only considering models that lie in the ` earth - like triangle ' for magnetic field morphology in figure [ fig : c116_rm_eketa_triangle ] ( criteria of* ) , the dynamo dataset is reduced from 116 to 61 models .table [ tab : earthlike_diffless_scalings ] shows the scaling laws that in this case are preferred by loocv under the assumption of equal errors in .although the dataset is reduced by almost half , the resulting laws only differ in their exponents ( up to ) , but not in the parameters included ( cf .table [ tab : diffless_scalings ] ) .christensen , u. r. & aubert , j. , 2006 .scaling properties of convection - driven dynamos in rotating spherical shells and application to planetary magnetic fields , _ geophysical journal international _ , * 166*(1 ) , 97114 .de koker , n. , steinle - neumann , g. , & vlcek , v. , 2012 .electrical resistivity and thermal conductivity of liquid fe alloys at high p and t , and heat flux in earth s core , _ p natl acad sci usa _ , * 109*(11 ) , 40704073 .

Abstract: earth s magnetic field is generated by processes in the electrically conducting , liquid outer core , subsumed under the term ` geodynamo ' . in the last decades , great effort has been put into the numerical simulation of core dynamics following from the magnetohydrodynamic ( mhd ) equations . however , the numerical simulations are far from earth s core in terms of several control parameters . different scaling analyses found simple scaling laws for quantities like heat transport , flow velocity , magnetic field strength and magnetic dissipation time . + we use an extensive dataset of 116 numerical dynamo models compiled by christensen and co - workers to analyse these scalings from a rigorous model selection point of view . our method of choice is leave - one - out cross - validation which rates models according to their predictive abilities . in contrast to earlier results , we find that diffusive processes are not negligible for the flow velocity and magnetic field strength in the numerical dynamos . also the scaling of the magnetic dissipation time turns out to be more complex than previously suggested . assuming that the processes relevant in the numerical models are the same as in earth s core , we use this scaling to estimate an ohmic dissipation of 3 - 8 tw for the core . this appears to be consistent with recent high cmb heat flux scenarios .  2


teachers of introductory courses on computational linguistics are often faced with the challenge of setting up a practical programming component for student assignments and projects .this is a difficult task because different computational linguistics domains require a variety of different data structures and functions , and because a diverse range of topics may need to be included in the syllabus .a widespread practice is to employ multiple programming languages , where each language provides native data structures and functions that are a good fit for the task at hand .for example , a course might use prolog for parsing , perl for corpus processing , and a finite - state toolkit for morphological analysis . by relying on the built - in features of various languages, the teacher avoids having to develop a lot of software infrastructure .an unfortunate consequence is that a significant part of such courses must be devoted to teaching programming languages .further , many interesting projects span a variety of domains , and would require that multiple languages be bridged .for example , a student project that involved syntactic parsing of corpus data from a morphologically rich language might involve all three of the languages mentioned above : perl for string processing ; a finite state toolkit for morphological analysis ; and prolog for parsing .it is clear that these considerable overheads and shortcomings warrant a fresh approach .apart from the practical component , computational linguistics courses may also depend on software for in - class demonstrations .this context calls for highly interactive graphical user interfaces , making it possible to view program state ( e.g. the chart of a chart parser ) , observe program execution step - by - step ( e.g. execution of a finite - state machine ) , and even make minor modifications to programs in response to `` what if '' questions from the class . because of these difficulties it is common to avoid live demonstrations , and keep classes for theoretical presentations only .apart from being dull , this approach leaves students to solve important practical problems on their own , or to deal with them less efficiently in office hours .  in this paperwe introduce a new approach to the above challenges , a streamlined and flexible way of organizing the practical component of an introductory computational linguistics course .we describe nltk , the natural language toolkit , which we have developed in conjunction with a course we have taught at the university of pennsylvania .the natural language toolkit is available under an open source license from .nltk runs on all platforms supported by python , including windows , os x , linux , and unix .the most basic step in setting up a practical component is choosing a suitable programming language .a number of considerations influenced our choice .first , the language must have a shallow learning curve , so that novice programmers get immediate rewards for their efforts .second , the language must support rapid prototyping and a short develop / test cycle ; an obligatory compilation step is a serious detraction .third , the code should be self - documenting , with a transparent syntax and semantics .fourth , it should be easy to write structured programs , ideally object - oriented but without the burden associated with languages like c++ .finally , the language must have an easy - to - use graphics library to support the development of graphical user interfaces .  in surveying the available languages, we believe that python offers an especially good fit to the above requirements .python is an object - oriented scripting language developed by guido van rossum and available on all platforms ( www.python.org ) .python offers a shallow learning curve ; it was designed to be easily learnt by children . as an interpreted language , python is suitable for rapid prototyping .python code is exceptionally readable , and it has been praised as `` executable pseudocode . ''python is an object - oriented language , but not punitively so , and it is easy to encapsulate data and methods inside python classes .finally , python has an interface to the tk graphics toolkit , and writing graphical interfaces is straightforward .several criteria were considered in the design and implementation of the toolkit .these design criteria are listed in the order of their importance .it was also important to decide what goals the toolkit would _ not _ attempt to accomplish ; we therefore include an explicit set of non - requirements , which the toolkit is not expected to satisfy .[ [ ease - of - use . ] ] _ ease of use . _+ + + + + + + + + + + + + +  the primary purpose of the toolkit is to allow students to concentrate on building natural language processing ( nlp ) systems .the more time students must spend learning to use the toolkit , the less useful it is .  [ [ consistency . ] ] _ consistency . _+ + + + + + + + + + + + + +  the toolkit should use consistent data structures and interfaces .  [[ extensibility . ] ] _ extensibility . _ + + + + + + + + + + + + + + + +  the toolkit should easily accommodate new components , whether those components replicate or extend the toolkit s existing functionality .the toolkit should be structured in such a way that it is obvious where new extensions would fit into the toolkit s infrastructure .  [[ documentation . ] ] _ documentation . _ + + + + + + + + + + + + + + + +  the toolkit , its data structures , and its implementation all need to be carefully and thoroughly documented .all nomenclature must be carefully chosen and consistently used .[ [ simplicity . ] ] _ simplicity . _+ + + + + + + + + + + + +  the toolkit should structure the complexities of building nlp systems , not hide them .therefore , each class defined by the toolkit should be simple enough that a student could implement it by the time they finish an introductory course in computational linguistics .[ [ modularity . ] ] _ modularity . _+ + + + + + + + + + + + +  the interaction between different components of the toolkit should be kept to a minimum , using simple , well - defined interfaces . in particular, it should be possible to complete individual projects using small parts of the toolkit , without worrying about how they interact with the rest of the toolkit .this allows students to learn how to use the toolkit incrementally throughout a course .modularity also makes it easier to change and extend the toolkit .[ [ comprehensiveness . ] ] _ comprehensiveness . _ + + + + + + + + + + + + + + + + + + + +  the toolkit is not intended to provide a comprehensive set of tools .indeed , there should be a wide variety of ways in which students can extend the toolkit .[ [ efficiency . ] ] _ efficiency . _ + + + + + + + + + + + + +  the toolkit does not need to be highly optimized for runtime performance. however , it should be efficient enough that students can use their nlp systems to perform real tasks .[ [ cleverness . ] ] _ cleverness ._ + + + + + + + + + + + + +  clear designs and implementations are far preferable to ingenious yet indecipherable ones .the toolkit is implemented as a collection of independent _ modules _ , each of which defines a specific data structure or task .a set of core modules defines basic data types and processing systems that are used throughout the toolkit .the ` token ` module provides basic classes for processing individual elements of text , such as words or sentences .the ` tree ` module defines data structures for representing tree structures over text , such as syntax trees and morphological trees .the ` probability ` module implements classes that encode frequency distributions and probability distributions , including a variety of statistical smoothing techniques .the remaining modules define data structures and interfaces for performing specific nlp tasks. this list of modules will grow over time , as we add new tasks and algorithms to the toolkit .the ` parser ` module defines a high - level interface for producing trees that represent the structures of texts .the ` chunkparser ` module defines a sub - interface for parsers that identify non - overlapping linguistic groups ( such as base noun phrases ) in unrestricted text .four modules provide implementations for these abstract interfaces .the ` srparser ` module implements a simple shift - reduce parser .the ` chartparser ` module defines a flexible parser that uses a _ chart _ to record hypotheses about syntactic constituents .the ` pcfgparser ` module provides a variety of different parsers for probabilistic grammars . and the `rechunkparser ` module defines a transformational regular - expression based implementation of the chunk parser interface .the ` tagger ` module defines a standard interface for augmenting each token of a text with supplementary information , such as its part of speech or its wordnet synset tag ; and provides several different implementations for this interface .the ` fsa ` module defines a data type for encoding finite state automata ; and an interface for creating automata from regular expressions .debugging time is an important factor in the toolkit s ease of use . to reduce the amount of timestudents must spend debugging their code , we provide a type checking module , which can be used to ensure that functions are given valid arguments .the type checking module is used by all of the basic data types and processing classes .  since type checkingis done explicitly , it can slow the toolkit down .however , when efficiency is an issue , type checking can be easily turned off ; and with type checking is disabled , there is no performance penalty .visualization modules define graphical interfaces for viewing and manipulating data structures , and graphical tools for experimenting with nlp tasks .the ` draw.tree ` module provides a simple graphical interface for displaying tree structures .the ` draw.tree_edit ` module provides an interface for building and modifying tree structures .the ` draw.plot_graph ` module can be used to graph mathematical functions .the ` draw.fsa ` module provides a graphical tool for displaying and simulating finite state automata .the ` draw.chart ` module provides an interactive graphical tool for experimenting with chart parsers .the visualization modules provide interfaces for interaction and experimentation ; they do not directly implement nlp data structures or tasks .simplicity of implementation is therefore less of an issue for the visualization modules than it is for the rest of the toolkit .the ` classifier ` module defines a standard interface for classifying texts into categories .this interface is currently implemented by two modules .the ` classifier.naivebayes ` module defines a text classifier based on the naive bayes assumption .the ` classifier.maxent ` module defines the maximum entropy model for text classification , and implements two algorithms for training the model : generalized iterative scaling and improved iterative scaling .the ` classifier.feature ` module provides a standard encoding for the information that is used to make decisions for a particular classification task .this standard encoding allows students to experiment with the differences between different text classification algorithms , using identical feature sets .the ` classifier.featureselection ` module defines a standard interface for choosing which features are relevant for a particular classification task .good feature selection can significantly improve classification performance .the toolkit is accompanied by extensive documentation that explains the toolkit , and describes how to use and extend it .this documentation is divided into three primary categories :  [ [ tutorials ] ] _ tutorials _ + + + + + + + + + + +  teach students how to use the toolkit , in the context of performing specific tasks .each tutorial focuses on a single domain , such as tagging , probabilistic systems , or text classification .the tutorials include a high - level discussion that explains and motivates the domain , followed by a detailed walk - through that uses examples to show how nltk can be used to perform specific tasks .[ [ reference - documentation ] ] _ reference documentation _ + + + + + + + + + + + + + + + + + + + + + + + + +  provides precise definitions for every module , interface , class , method , function , and variable in the toolkit .it is automatically extracted from docstring comments in the python source code , using epydoc .[ [ technical - reports ] ] _ technical reports _ + ++ + + + + + + + + + + + + + + + +  explain and justify the toolkit s design and implementation .they are used by the developers of the toolkit to guide and document the toolkit s construction .students can also consult these reports if they would like further information about how the toolkit is designed , and why it is designed that way .nltk can be used to create student assignments of varying difficulty and scope . in the simplest assignments , students experiment with an existing module .the wide variety of existing modules provide many opportunities for creating these simple assignments .once students become more familiar with the toolkit , they can be asked to make minor changes or extensions to an existing module .a more challenging task is to develop a new module .here , nltk provides some useful starting points : predefined interfaces and data structures , and existing modules that implement the same interface .   as an example of a moderately difficult assignment, we asked students to construct a chunk parser that correctly identifies base noun phrase chunks in a given text , by defining a cascade of transformational chunking rules .the nltk ` rechunkparser ` module provides a variety of regular - expression based rule types , which the students can instantiate to construct complete rules . for example , ` chunkrule(<nn.*> ) ` builds chunks from sequences of consecutive nouns ; ` chinkrule(<vb.> ) ` excises verbs from existing chunks ; ` splitrule(<nn> , <dt> ) ` splits any existing chunk that contains a singular noun followed by determiner into two pieces ; and ` mergerule(<jj> , <jj> ) ` combines two adjacent chunks where the first chunk ends and the second chunk starts with adjectives .the chunking tutorial motivates chunk parsing , describes each rule type , and provides all the necessary code for the assignment .the provided code is responsible for loading the chunked , part - of - speech tagged text using an existing tokenizer , creating an unchunked version of the text , applying the chunk rules to the unchunked text , and scoring the result .students focus on the nlp task only providing a rule set with the best coverage .  in the remainder of this sectionwe reproduce some of the cascades created by the students .the first example illustrates a combination of several rule types :  cascade = [ chunkrule(
<nn.*><vb.><nn.*> ) , chunkrule(
<vb.><nn.*> ) , chunkrule(<.*> ) , unchunkrule(<in|vb.*|cc|md|rb.*> ) , unchunkrule(``<,|\\.|``| '' > '' ) , mergerule(<nn.*|dt|jj.*|cd> , <nn.*|dt|jj.*|cd> ) , splitrule(<nn.*> ,
 ) ]  the next example illustrates a brute - force statistical approach .the student calculated how often each part - of - speech tag was included in a noun phrase .they then constructed chunks from any sequence of tags that occurred in a noun phrase more than 50% of the time .cascade = [ chunkrule(<\\$|cd|dt|ex|pdt |prp.*|wp.*|\\#|fw   in the third example , the student constructed a single chunk containing the entire text , and then excised all elements that did not belong .cascade = [ chunkrule(<.*>+ ) chinkrule(<vb.*|in|cc|r.*|md|wrb|to|.|,>+ ) ]   nltk provides graphical tools that can be used in class demonstrations to help explain basic nlp concepts and algorithms .these interactive tools can be used to display relevant data structures and to show the step - by - step execution of algorithms .both data structures and control flow can be easily modified during the demonstration , in response to questions from the class .since these graphical tools are included with the toolkit , they can also be used by students .this allows students to experiment at home with the algorithms that they have seen presented in class .the chart parsing tool is an example of a graphical tool provided by nltk .this tool can be used to explain the basic concepts behind chart parsing , and to show how the algorithm works .chart parsing is a flexible parsing algorithm that uses a data structure called a _chart _ to record hypotheses about syntactic constituents .each hypothesis is represented by a single _ edge _ on the chart . a set of _ rules _determine when new edges can be added to the chart .this set of rules controls the overall behavior of the parser ( e.g. , whether it parses top - down or bottom - up ) .the chart parsing tool demonstrates the process of parsing a single sentence , with a given grammar and lexicon .its display is divided into three sections : the bottom section displays the chart ; the middle section displays the sentence ; and the top section displays the partial syntax tree corresponding to the selected edge .buttons along the bottom of the window are used to control the execution of the algorithm .the main display window for the chart parsing tool is shown in figure [ fig : chartparse ] .  ' '' ''  this tool can be used to explain several different aspects of chart parsing .first , it can be used to explain the basic chart data structure , and to show how edges can represent hypotheses about syntactic constituents .it can then be used to demonstrate and explain the individual rules that the chart parser uses to create new edges .finally , it can be used to show how these individual rules combine to find a complete parse for a given sentence .  to reduce the overhead of setting up demonstrations during lecture ,the user can define a list of preset charts .the tool can then be reset to any one of these charts at any time .the chart parsing tool allows for flexible control of the parsing algorithm . at each step of the algorithm ,the user can select which rule or strategy they wish to apply .this allows the user to experiment with mixing different strategies ( e.g. , top - down and bottom - up ) .the user can exercise fine - grained control over the algorithm by selecting which edge they wish to apply a rule to .this flexibility allows lecturers to use the tool to respond to a wide variety of questions ; and allows students to experiment with different variations on the chart parsing algorithm .nltk provides students with a flexible framework for advanced projects .typical projects involve the development of entirely new functionality for a previously unsupported nlp task , or the development of a complete system out of existing and new modules .the toolkit s broad coverage allows students to explore a wide variety of topics .in our introductory computational linguistics course , topics for student projects included text generation , word sense disambiguation , collocation analysis , and morphological analysis .nltk eliminates the tedious infrastructure - building that is typically associated with advanced student projects by providing students with the basic data structures , tools , and interfaces that they need .this allows the students to concentrate on the problems that interest them.  the collaborative , open - source nature of the toolkit can provide students with a sense that their projects are meaningful contributions , and not just exercises .several of the students in our course have expressed interest in incorporating their projects into the toolkit .finally , many of the modules included in the toolkit provide students with good examples of what projects should look like , with well thought - out interfaces , clean code structure , and thorough documentation .the probabilistic parsing module was created as a class project for a statistical nlp course .the toolkit provided the basic data types and interfaces for parsing .the project extended these , adding a new probabilistic parsing interface , and using subclasses to create a probabilistic version of the context free grammar data structure .these new components were used in conjunction with several existing components , such as the chart data structure , to define two implementations of the probabilistic parsing interface .finally , a tutorial was written that explained the basic motivations and concepts behind probabilistic parsing , and described the new interfaces , data structures , and parsers .we used nltk as a basis for the assignments and student projects in cis-530 , an introductory computational linguistics class taught at the university of pennsylvania .cis-530 is a graduate level class , although some advanced undergraduates were also enrolled .most students had a background in either computer science or linguistics ( and occasionally both ) .students were required to complete five assignments , two exams , and a final project .all class materials are available from the course website .the experience of using nltk was very positive , both for us and for the students .the students liked the fact that they could do interesting projects from the outset .they also liked being able to run everything on their computer at home .the students found the extensive documentation very helpful for learning to use the toolkit .they found the interfaces defined by nltk intuitive , and appreciated the ease with which they could combine different components to create complete nlp systems .we did encounter a few difficulties during the semester .one problem was finding large clean corpora that the students could use for their assignments .several of the students needed assistance finding suitable corpora for their final projects .another issue was the fact that we were actively developing nltk during the semester ; some modules were only completed one or two weeks before the students used them . as a result , students who worked at home needed to download new versions of the toolkit several times throughout the semester .luckily , python has extensive support for installation scripts , which made these upgrades simple .the students encountered a couple of bugs in the toolkit , but none were serious , and all were quickly corrected .the computational component of computational linguistics courses takes many forms . in this sectionwe briefly review a selection of approaches , classified according to the ( original ) target audience .* linguistics students . *various books introduce programming or computing to linguists .these are elementary on the computational side , providing a gentle introduction to students having no prior experience in computer science .examples of such books are : _ using computers in linguistics _ , and _ programming for linguistics : java technology for language researchers _ .  * grammar developers . * infrastructure for grammar development has a long history in unification - based ( or constraint - based ) grammar frameworks , from dcg to hpsg .recent work includes .a concurrent development has been the finite state toolkits , such as the xerox toolkit .this work has found widespread pedagogical application .* other researchers and developers . * a variety of toolkits have been created for research or r&d purposes .examples include the _ cmu - cambridge statistical language modeling toolkit _ , the _ emu speech database system _ , the _ general architecture for text engineering _ , the _ maxent package for maximum entropy models _ , and the _ annotation graph toolkit _ .although not originally motivated by pedagogical needs , all of these toolkits have pedagogical applications and many have already been used in teaching .nltk provides a simple , extensible , uniform framework for assignments , projects , and class demonstrations .it is well documented , easy to learn , and simple to use .we hope that nltk will allow computational linguistics classes to include more hands - on experience with using and building nlp components and systems .nltk is unique in its combination of three factors .first , it was deliberately designed as courseware and gives pedagogical goals primary status .second , its target audience consists of both linguists and computer scientists , and it is accessible and challenging at many levels of prior computational skill . finally , it is based on an object - oriented scripting language supporting rapid prototyping and literate programming .we plan to continue extending the breadth of materials covered by the toolkit .we are currently working on nltk modules for hidden markov models , language modeling , and tree adjoining grammars .we also plan to increase the number of algorithms implemented by some existing modules , such as the text classification module .finding suitable corpora is a prerequisite for many student assignments and projects .we are therefore putting together a collection of corpora containing data appropriate for every module defined by the toolkit .nltk is an open source project , and we welcome any contributions . readers who are interested in contributing to nltk , or who have suggestions for improvements , are encouraged to contact the authors .we are indebted to our students for feedback on the toolkit , and to anonymous reviewers , jee bang , and the workshop organizers for comments on an earlier version of this paper .we are grateful to mitch marcus and the department of computer and information science at the university of pennsylvania for sponsoring the work reported here .jason baldridge , john dowding , and susana early .leo : an architecture for sharing resources for unification - based grammars . in _ proceedings of the third language resources and evaluationconference_. paris : european language resources association .+ http://www.iccs.informatics.ed.ac.uk/~jmb/leo-lrec.ps.gz .kalina bontcheva , hamish cunningham , valentin tablan , diana maynard , and oana hamza .2002 . using gate as an environment for teaching nlp . in _ proceedings of the acl workshop on effective tools and methodologies for teaching nlp and cl_. somerset , nj : association for computational linguistics .philip r. clarkson and ronald rosenfeld . 1997 .statistical language modeling using the cmu - cambridge toolkit . in _ proceedings of the 5th european conference on speech communication and technology ( eurospeech 97)_. http://svr-www.eng.cam.ac.uk/~prc14/eurospeech97.ps .kazuaki maeda , steven bird , xiaoyi ma , and haejoong lee .2002 . creating annotation tools with the annotation graph toolkit . in _ proceedings of the third international conference on language resources andevaluation_. http://arxiv.org/abs/cs/0204005 .

Abstract: nltk , the natural language toolkit , is a suite of open source program modules , tutorials and problem sets , providing ready - to - use computational linguistics courseware . nltk covers symbolic and statistical natural language processing , and is interfaced to annotated corpora . students augment and replace existing components , learn structured programming by example , and manipulate sophisticated models from the outset .


relativity theory  provides a remarkable example of the construction of a nontrivial time scale on the basis of a symmetry principle ; namely , the proper time of a moving body from the requirement of invariance of the laws of nature for different observers .this example , in which symmetry lays at the very foundation of the physical theory , may suggest to look for other contexts in which symmetries could help in defining useful notions of time .non - stationary time series are very frequently encountered in fields like meteorology  , seismology  , physiology  , economy and finance  .the non - stationary mark , limits the applicability of statistical methods in the quest for the process characterizing the series , and may hinder a correct detection of the possible power - law scalings of the signals and of the associated long - range correlations  .depending on the quality of the stochastic process , different strategies can be applied to simplify the signals and to recover stationarity properties  .for instance , various detrending procedures have been proposed  , although in the general case their success is not always guaranteed  .other approaches  are instead based on a reassessment of the time stamp in terms of which the series is recorded . in particular , in the case of seismology it has been recently advanced  that the distribution of intercurrence times between earthquakes becomes scale - invariant under a proper redefinition of time ; the latter renders intercurrencies stationary on the basis of the omori s aftershock law  .following inspiration provided by the example of relativity , within the context of time series affected by cyclic non - stationary behavior  we show here that one can promote the fulfillment of a scaling symmetry to become an operative criterion for the definition of a novel time .as one should expect , the symmetry enforcement leads to a distinct simplification of the analysis .our focus is on finance , a field in which the very definition of time constitutes a long - standing problem , while the application of symmetry principles lacks a strong tradition .a clock properly adjusted to financial activity is not easy to identify .financial markets have to cope with human daily routines throughout the world and bursts and doldrums occur at various time scales . in addition , during nights , weekends , and festivities , transactions stop in most cases . these interruptions ,at the end of which assets prices turn out to have changed anyhow , together with the recurrent patterns in the activity ( seasonalities ) , immediately signal the inadequacy of `` natural '' physical time as the appropriate one in terms of which to describe the stochastic evolution of financial markets . besides physical time, different alternatives have been studied  , including trading time , volume time , and tick - by - tick time .a time definition ( theta - time ) has also been put forward with the specific intent of getting rid of the seasonalities in financial time series  .theta - time is designed to record the progress of market activity through the increase of the volatility as measured by the average absolute return .for the latter a power - law behavior is assumed as a function of physical time , but no further attempt to discuss and possibly simplify the scaling properties of the return probability density function ( pdf ) is made . in many of the referred proposals , inactive periods are cut from the elapsing time  , and the duration to be ascribed to overnight and similar returns remains an open question .in general , there is also little focus on the requirement that increments over intervals of equal span should be identically distributed in order to make sliding - window sampling applicable for the construction of their empirical pdfs .here we show that a solution to these problems is naturally suggested by the existence of an approximate symmetry which appears when the pdfs increments over different intervals are compared .indeed , from days to several weeks , the empirical financial returns pdfs are found to be approximately scale - invariant , in a sense familiar from the physics of critical phenomena  . defining financial returns over the time span (here is an integer number of days ) as the log - difference of the asset price value , , this means that the empirical pdfs of over different s can be collapsed onto each other in force of the scaling law where is called the hurst exponent  , and , which is not gaussian , is a scaling function .it turns out that is close to for assets of developed markets  . the symmetry property expressed by eq.  has been recently assumed as a modeling basis for the stochastic dynamics of financial indexes  . combining scaling with ideas of fine graining inspired by the renormalization group approach of statistical mechanics  ,these models reproduce many stylized facts exposed by interday time series analysis .most important here , the strict validity of eq .  would establish an univoque correspondence between returns pdfs and time intervals durations s .however , eq .  definitely does not hold at the intraday time scales , where non - stationarities affect pdfs for returns defined over intervals with equal physical time duration  , and can be very different from in selected time windows  . even in the interday domain multiscaling effects  and a slow crossover to gaussianity  a full realization of the collapses implied by eq .( [ eq_scaling ] ) .  in view of the difficulties involved in the definition of time in finance , specifying the returns pdfs in terms of a single, appropriate parameter remains a basic goal worth pursuing . herewe show that if defined with respect to a suitable function of physical time which we call financial scaling time ( fst ) returns pdfs become identical for intervals of the same fst duration .moreover , when rescaled through an hurst exponent equal to , satisfactory collapses for pdfs related to different fst spans are exhibited .our results hold within a window ranging from few minutes to several days in physical time , hence bridging the intraday and interday regimes .our basic ansatz is that , once expressed in terms of fst , eq .turns into an exact scaling law with : where represents the fst duration of an interval and is a suitable scaling function .the fst construction exploits the fact that , in spite of the manifest non - stationarities at the intraday level  , the process of return aggregation appears compatible with the assumption of one - day ciclostationarity  . in other words ,the evolution of the aggregate return from the opening during each market day can be regarded as a realization of the same stochastic process  , and empirical averages can be taken over correspondent time windows within different days .the fst construction proceeds as follows : on the physical time axis we first operate a partition into intervals consistent with the one - day periodicity of the process of return formation .intervals are arbitrary , except for the fact that their extrema should not fall within the periods of market closure , and that their duration should be above a lower cutoff ( typically of the order of a few minutes , as we discuss below ) .it is convenient , but not necessary , to set the morning opening of the market as the lower extremum of the first partition interval , and to let the last interval of each day coincide with the full overnight ( or weekend ) closure . in the exampletreated below , we further choose to assign the same physical time span to all remaining intraday intervals .intervals exceeding the duration of one day are then constructed as the union of a suitable number of intraday intervals pertaining to different days .our purpose is to associate an appropriate fst - duration to each interval with extrema belonging to the partition .of course , if we call and the fst durations of any two contiguous intervals , we require the span of their union to satisfy the measure property this allows us to consistently map the physical - time partition onto the -axis .so , a -axis partition possesses a -axis partition image , and vice versa .eq .  together with eq.  immediately impose an important constraint .calling and the returns over the contiguous intervals of duration and , respectively , we necessarily have =(\delta\tau_1+\delta\tau_2)\int\mathrm{d}x\,g(x)\,x^2 ] , or , in the presence of a general hurst exponent like in eq . , =(\delta \tau)^{qh } \,\int\mathrm{d}x\,g(x)\,|x|^q .\label{eq_moments}\ ] ] one of the consequence of eq .is that a log - log plot of the - moment  ] the number of minutes characterizing ( for instance , =10 ] where is a sliding - window empirical average with respect to , performed over all available couples in the dataset .an analogous definition follows for in fst .

Abstract: a symmetry - guided time redefinition may enhance and simplify analyses of historical series displaying recurrent patterns . enforcing a simple - scaling symmetry with hurst exponent 1/2 and the requirement of increments stationarity , we identify a time - definition protocol in the financial case . the novel time scale , constructed through a systematic application of the kolmogorov - smirnov criterion to extensive data of the sp500 index , lays a bridge between the regime of minutes and that of several days in physical time . it allows us to quantify the duration of periods in which the market is inactive , like amid nights , and to optimally exploit the statistical information contained in the series . the overall strategy leads to a significant reduction of multiscaling features , once the moments of the return probability density function are analyzed versus the novel time .