by integrating the ability of sensing physical world and the privilege in availing communication capabilities , internet of things ( iot ) enables close interactions between humans and machines .iot generally consists of numerous end iot devices for sensing and action , intermediate nodes with wired connectivity for data relaying , and application servers in the cloud for data controlling and analysis .typically , iot devices can communicate with each other with minimal human intervention and build an autonomous and complex network .as the boundary between machines and humans gets blurry , adversaries in the cyberspace can threaten human users safety and privacy in the physical world .obviously , the growing popularity of devices with rich wireless communication capabilities has made iot attractive to digital viruses and malicious contents .consequently , in recent years the security issues in iot has been an ever - increasing concern  .  from an adversary s perspective , the unique features of iot facilitate the exploits of devices as well as the propagation of iot malware .these features include constrained resources , heterogeneous links , and vulnerable usability , which are discussed as follows .  comparing with the intermediate nodes located at the end side of the infrastructure with wired connectivity , iot devices designed to perform simple sensing and actuation operations have limited computation and communication capabilities . in this case , the algorithm and mechanism applied on iot devices are relatively simple . as a result, the attacker can spend much less resource to break in iot devices , rendering them the targets of malicious users .for example , due to the overhead of certificate management and public - key cryptography , many existing iot devices fail to support state - of - the - art secure communication protocols ( e.g. , ssl / tls ) .therefore , the adversary can eavesdrop on sensitive sensor data and even manipulate data without being detected .another example is that iot devices often have limited entropy sources , which results in weak cryptographic keys that can be predicted by the attacker .moreover , since most iot devices run on embedded linux os , the attacker can easily create iot malware by recompiling existing linux malware for other instruction set architectures .  in order to support different kinds of iot applications ,iot devices are often equipped with heterogeneous communication and computation capabilities for the purpose of seamless operations .however , the heterogeneity and potentially vast amount of iot devices facilitate the fabrication of identity and hiding of malware . moreover , as shown in fig .[ fig_iotarch ] , compromised iot devices might disseminate malware via heterogeneous communication links as described below .  ** infrastructure links .* iot malware can propagate using infrastructure - based communication technologies , such as gsm / gprs / umts / lte and wlan , via intermediate nodes , such as access point ( ap ) , base station ( bs ) , or gateway .in particular , iot malware inherits the threats caused by computer malware .similar to computer malware , most iot malware families today scan the ip address space for vulnerable victims and spread via the internet . due to the widespread use of weak login credentials and the fact that many iot devices are internet - accessible , some botnets have allegedly harvested more than one million of infected iot devices . ** device - to - device links .* iot malware could exploit the proximity - based wireless media such as ble , wi - fi direct , and nfc to infect the devices in the vicinity  . in this case ,iot malware is stored and forwarded by taking advantages of mobility and ubiquity .for example , colin oflynn in black hat usa 2016 as well as ronen and shamir  discussed the possibility of light bulb worm , which allows a reprogrammed bulb to re - flash nearby bulbs .security is only as strong as its weakest link , and the weakest link , in many cases , is the humans who implement , operate , and use the system . for example , a proven secure cryptographic primitive , if implemented or used incorrectly , can still be circumvented .moreover , users may choose to ignore or even bypass a security mechanism if it prevents ( e.g. , due to slow performance , badly designed user interface , and unclear instructions ) the users from doing what they meant to do .since iot devices often lack convenient input and output interfaces , the original security features might be bypassed by the non - professional it users , thereby increasing the possibility and risk of human errors and facilitating the spreading of malware .    obviously , the software updates and patching are necessary to prevent the iot devices from being compromised .a single software flaw will make a tremendous range of iot devices vulnerable to attacks since software components are reused in different devices .however , without a friendly interface to get alerted about security updates , most users forget to update software installed in iot devices and leave them out - of - date .in addition , without basic programming knowledge and security awareness , users might be unwilling to perform manual - download - and - install approach for the software update . as a result , it is critical to design a reasonable solution to prevent the occurrence of the large - scale malware propagation among trillion of unpatched , insecure , and even compromised iot devices .  instead of patching resource - constrained and ui - unfriendly compromised iot devices directly , this article introduces a more feasible solution , where operators could only patch or recover iot devices via infrastructure , i.e. , securing the intermediate nodes . in this case , the patched ap , bs or gateway could stop the malware propagation by patching via infrastructure links .the concept of leveraging intermediate nodes to improve iot security has appeared in the recent commercial product f - secure sense . however , its main purpose is to block malicious websites and iot botnet masters instead of considering securing important infrastructure links between iot devices and intermediate nodes . on the other hand ,the idea behind iot sentinel  is similar to our solution , where the type of iot devices are identified by intermediate nodes , and the communications of vulnerable iot devices are constrained by enabling enforcement of rules .different from our solution , software - defined network ( sdn ) is exploited in iot sentinel for network flow isolation and for prevention of malware propagation .  with limited efforts and resources , the operator might not be able to patch all intermediate nodes but only a portion of them .one naive method is to simply patch those intermediate nodes in a random order .however , a smarter approach is to protect the most important node first , as suggested by the framework of network robustness analysis  .this article proposes a traffic - aware patching scheme , where the operator patches the intermediate nodes sequentially in a descending importance order .in particular , an intermediate node who could contact with a large number of iot devices will be protected first . moreover , such volume - based patching approach is effective to the current infamous ddos attacks launched by iot bots .  by leveraging a real - world trace datasets containing communication history overdevice - to - device and infrastructure links , we conduct an extensive experiment to demonstrate the effect on constraining malware propagation via infrastructure links . to the best of authors knowledge, this article is the first work discussing the control of malware propagation from the perspective of infection paths , which could avail the damage estimation caused by the malware and improve the development of attack detection methods for iot networks .iot devices are an attractive attack target for cybercriminals : iot devices often employ weak security measures , and their compromise can lead to privacy breaches and safety threats in the real world .the insecurity of existing iot devices has been highlighted repeatedly by security researchers and practitioners .recently , several malware families were found to target vulnerable iot devices ( e.g. , routers , ip cameras , and cctvs ) and form botnets for ddos .it is estimated that some iot botnets comprise more than one million of infected devices , and thus can generate high - volume ddos traffic even without amplification .for example , in september 2016 , an iot botnet called mirai crippled a website with 620 gbps of attack traffic , which is almost twice as much as the biggest ddos attack witnessed in 2015 . later in october 2016 ,the same botnet attacked the dyn dns service provider , taking down a large portion of websites in the north america , including github , twitter , netflix , etc . at defcon 2016 , security researchers showed a proof - of - concept iot ransomware that demands ransom for a hacked smart thermostat , which will be set to a high temperature without a timely payment .as attackers are finding creative ways to monetize infected iot devices , it is inevitable to see an increase of new iot malware families that are more destructive and contagious than ever .iot malware can propagate via infrastructure links and/or device - to - device links .we discuss both cases in this section .many of the iot malware families today propagate via infrastructure links , particularly the internet . moreover , they share a common infection and spreading pattern : the attacker harvests new vulnerable iot devices through address space scanning .this scanning can be performed by external servers , such as the c&c servers , or by the compromised devices .the attacker targets telnet- or ssh - accessible devices that use default or weak login credentials and thus can easily obtain root access permission by brute - force password cracking .once the attacker gets the shell of the hacked device , malware payload will be downloaded and installed .iotpot  , an iot honeypot project , observed at least four iot malware families that can propagate via telnet .in addition to cracking weak passwords , some malware also exploits software vulnerabilities .for example , cctv - targeting radiation malware exploits shellshock and some known cctv vulnerabilities to spread from device to device .malware can also propagate in proximity via device - to - device links in additional to infrastructure links .cabir and commwarror are examples of mobile worms that spread via bluetooth and infect mobile phones running symbian os .although we have not witnessed device - to - device iot malware in the wild , it is theoretically possible .for example , researchers pointed out the possibility of light bulb worms that spread to nearby bulbs via zigbee  and worms that infect wearable trackers and then spread to others by bluetooth .moreover , since proximity - based wireless interfaces are often always - on and users have no control to disable them , it would be difficult to contain malware propagation given the large attack surface .regardless of how malware propagates , the risk of self - replicating iot malware is amplified by unpatched iots .patching vulnerable iot devices nevertheless remains extremely expensive and far from successful in practice . in 2015 ,charlie miller and chris valasek demonstrated remote exploitation of a jeep , which forced chrysler to recall and patch 1.4 million vehicles .cui and stolfo  discovered more than 540,000 publicly accessible devices using default root passwords an old yet persisting vulnerability since the invention of password - based authentication .worse yet , the problems encountered when patching computers and mobile phones ( e.g. , privacy , legacy devices , and lack of incentives ) will linger and even exacerbate when attempting to patch iot devices .the topic of modeling malware / virus spreading has been investigated in a traditional scenario where computers or laptops are not connected to the internet .since the spread of epidemics among people is similar to the spread of malware over networks , the current literature adopts the idea from epidemiological models to build the models for malware on the assumption of homogeneous infection path  . in the mobile environment, malware can propagate via intermittently connected networks by taking advantage of opportunistic encounters .et al . _ study spreading patterns of mobile phone viruses which may traverse through multimedia messaging services ( mms ) or bluetooth by simulations .cheng and chen  further models malware propagation in generalized social networks consisting of delocalized and localized links .  from the discussion of the previous section , we understand that in practice patching the compromised iot devices is difficult to be achieved .consequently , the current formulation of malware propagation and the control model  can not be applied directly in the iot field .typically , in one of the most famous susceptible - infection - recover ( sir ) model , the malware is assumed to be detected and repaired at each node , which reflects the transition from `` infected '' state to `` recovered '' state . regarding the iot device who detects the malware , instead of directly patching it, it is more feasible to patch on the infrastructure side to prevent further spreading of malware . in this case ,compromised iot devices located in the coverage area of the patched intermediate nodes are controlled , that is , malware can not be propagated via patched intermediate nodes . as a result, the infrastructure links can be regarded as `` recovered '' while the compromised iot device remains `` infected '' using the terminology of sir model .the observation that malware control in iot environment can be cast as a `` link recovery '' problem instead of a `` node recovery '' problem motivates a different development of modeling and formulation .this section proposes patching schemes for iot environment , where we can only control infrastructure links but not the compromised nodes themselves .the patching scheme consists of several phases . in the _ detecting phase _ , infrastructure leverages traditional ids or firewall to identify the existence of malware or compromised node .once a malicious code is found to be propagated from the compromised iot devices , _ patching phase _ starts to analyze the malware and patches the intermediate nodes according to patching sequence to prevent the large - scale propagation of malware . in practice ,intermediate nodes are capable of performing resource intensive tasks and thus can support over the air ( ota ) update mechanisms . in the _ patching phase _ , such ota mechanisms allow the administrator to remotely install required update on the intermediate nodes , thereby ensuring timely mitigation of compromised nodes .in addition , since intermediate nodes are significantly fewer than iot devices , the administrator can also manually patch legacy intermediate nodes that do not support ota update .[ fig_propagation ] describes an example of how a compromised device propagates malware in iot environment with patched and unpatched intermediate nodes . for the devices located in the coverage area of the patched intermediate nodes ,two possible operations will be executed .  ** compromised devices * can distribute malware via device - to - device links but not infrastructure links . as shown in step 1 of fig .[ fig_propagation ] , the compromised device propagates malware to devices _ b _ and _ c _ in the vicinity .however , in step 2 of fig .[ fig_propagation ] , device _b _ can not propagate malware via infrastructure link since the malware is blocked at the patched bs .* * normal devices * can only be compromised via device - to - device links since the malware propagated from infrastructure will be identified and blocked by the patched intermediate nodes .for example , in step 3 of fig .[ fig_propagation ] , device _propagates malware from bs 2 to bs 1 , however , the patched bs 1 will not relay the malware to any device in its coverage area .  for the devices located in the coverage area of the unpatched intermediate nodes , there are no means to prevent malware propagation .for example , in step 2 of fig .[ fig_propagation ] , device _c _ under unpatched bs 3 could infect device _d _ controlled by unpatched bs 2 via infrastructure links .moreover , device _a _ moving from patched bs 1 to unpatched bs 2 could propagate malware via device - to - device links freely .collect traffic information for each intermediate node sort intermediate nodes according to the importance metric in descending traffic order patch top   algorithm [ alg_traffic ] describes the detailed steps in _ patching phase_. with limited resources and efforts , the operator could provide a fixed amount of patches on the intermediate nodes ( e.g. , percentage ) . to alleviate the propagation from the infrastructure links ,the most important intermediate nodes will be chosen for patching .it is similar to the idea of protecting the most important node to maintain network robustness  . as a result, we introduce the traffic monitoring duration ( see lines 1 - 2 in algorithm [ alg_traffic ] ) for evaluating the importance of intermediate nodes . from the monitored results, the proposed traffic - aware patching scheme sorts the intermediate nodes in descending order according to the traffic volumes ( see lines 5 - 6 in algorithm [ alg_traffic ] ) , and the top intermediate nodes are patched ( see line 7 in algorithm [ alg_traffic ] ) .obvious , the proposed volume - based patching is effective to the attack which generates a large number of traffic volume , e.g. , ddos attacks .the patched intermediate nodes could prevent the redirection of malicious traffic introduced by the ddos attack launched by the iot botnets .in this section , we implement the proposed traffic - aware patching scheme and compare its performance with a randomized patching scheme on real - life traffic traces collected from a mobile social network consisting of 59 users ( devices ) and 1751 aps  . in this network , each user can communicate with other users through two types of links : ( 1 ) an infrastructure link via ( possibly multiple ) aps , and ( 2 ) a direct device - to - device link to users within transmission range .these two types of links among users are similar to the illustration of mobile iot in fig .[ fig_iotarch ] . as we mentioned in previous sections , in this experiment infrastructure linkscan be made secure via patching , whereas direct device - to - device links are vulnerable to potential security threats .following the vulnerability analysis of transmissive attacks in  , we simulate the propagation dynamics of self - replication malicious codes by first randomly selecting a user in the network as the initially compromised device .then , using the actual traces of communication patterns provided by the dataset  , each infected device can compromise its contact through an infrastructure link with probability , and can compromise its contact through a direct device - to - device link with probability . specifically ,if one of the aps in the communication path between one infected device to its contact has been successfully patched , then the malware propagation is in vain due to enhanced security .  for traffic - aware patching, we are interested in investigating the trade - offs between the time spent in analyzing traffic volumes ( i.e. , the traffic monitoring duration ) and the time instance to patch aps ( i.e. , the patch time ) .as described in the previous section , given a fixed amount of patches , the proposed traffic - aware patching scheme sorts the aps in descending order according to the traffic volumes in the traffic monitoring duration , and provides patches to the top aps .intuitively , longer traffic monitoring duration better specifies the important aps in communicating devices .however , longer traffic monitoring duration also leads to more exploits in security vulnerabilities due to later patch time . as a result , given a fixed amount of patches, we aim to study the non - trivial optimal patch time that collects sufficient traffic information for patching while minimizing the security risks .       and .the results are averaged over 500 trials.,width=307 ]    fig .[ fig_degree ] shows the fraction of compromised users with respect to different patch time and patched aps under the traffic - aware patching scheme . to demonstrate the effectiveness of the proposed traffic - aware patching scheme , fig .[ fig_degree_2 ] further compares the difference of compromised users between the no - patching scheme and the traffic - aware scheme .it can be observed that the best patching strategy that leads to a maximal decrease in the number of compromised users compared with the no - patching scheme is to monitor the traffics for 40 seconds and then provide patches to all aps .note that 100% patched aps ( i.e. , securing all infrastructure links ) with patch time 0 may not be the optimal patch strategy since the malicious codes are still able to propagate through direct device - to - decide links . to further understand the effect of traffic - aware patching , for a given fraction of patched aps , fig .[ fig_optimal ] shows the optimal patch time that leads to the lowest total number of compromised users .we observe that if one is able to patch more aps , then late patch time can have better performance , which suggests that traffic volumes are indeed important information for patching .  for fair comparison, we also compare the performance of traffic - aware patching with random patching .random patching provides immediate patches ( i.e. , has patch time ) and randomly selects a fraction of aps to patch .[ fig_difference ] shows the difference between the fraction of compromised users under random patching to that of traffic - aware patching , where larger positive values imply traffic - aware patching is more effective in securing the network , and vice versa .we observe that traffic - aware patching is significantly better than random patching in the regime of few patched aps ( e.g. , below 30% ) .moreover , given a fixed fraction of patched aps , for traffic - aware patching , there is at least one patch time that leads to either better or identical performance compared with random patching , which suggests the robustness and reliability of the proposed patching scheme . even in the regime of many patched aps ( e.g. , above 90% ) ,the performance of traffic - aware patching is still superior to random patching , which suggests the importance of patching aps of high traffic volumes for enhanced security .here we discuss several ongoing challenges and open research questions related to iot malware propagation and patching .* * transfer learning for optimal patch time . *  + in the experiments, we find that the patch time is crucial to preventing malware propagation . how to design and simulate realistic testbeds to assist determining the optimal patch time and to enable transfer learning for defending real - life unknown security threats are ongoing challenges .* * predictive malware propagation models for mobile iot .*  + in this article , we have addressed patching issues in mobile iot as link recovery instead of node recovery , where the latter has been extensively studied in traditional wireless networking scenarios . how to establish effective mathematical models for predicting malware propagation dynamics in mobile iot that take into account the traffic - aware and random patching schemes are new research challenges .* * various importance metrics for intermediate nodes .* the proposed scheme simply applies traffic volume as the metric to determine the importance of intermediate nodes and the patching sequence .it can be regarded as protecting the entire network by patching a relatively small fraction of intermediate nodes with the highest `` degree '' metric .the operator could consider more information about intermediate nodes , such as the topology of intermediate nodes , in order to design a more effective importance metric for determining the patching sequence .for example , the `` betweenness '' metric could be leveraged , which is defined as the fraction of all shortest paths passing through the node among all shortest paths between each node pair in the network . ** patching via path - based traffic patterns . *the proposed traffic - aware patching scheme only considers the one - hop traffic information in terms of the traffic volume from iot devices to intermediate nodes .the patching scheme could benefit from the knowledge beyond one - hop information , such as the path - based end - to - end traffic patterns. however , path - based traffic patterns are relatively difficult to be collected or acquired compared to the one - hop traffic information . * * how to achieve ( virtually ) patch ?* iot devices often lack friendly user interfaces and are left unattended after installation . as a consequence , users have trouble knowing whether a device is hacked , and even they do , they may find it challenging to _ manually patch _ the device : they need to retrieve updated firmware online , access the hacked device , install the firmware , etc .thus , _ automatic patching _ is needed to secure iot at scale .+ one promising direction is for iot devices to support firmware over the air ( fota ) , as most pcs and mobile phones do nowadays .however , an efficient and secure fota for iot remains an open challenge due to the heterogeneity of iot networks .for example , transport security and code signing are required to ensure the authenticity of the updated firmware .the iot gateway might help reduce the overhead by caching and offloading the security check .moreover , the human factors need to be taken into consideration as well . as in the pc and mobile phone worlds , forcing software update without explicit user consent can be disastrous .it can even be life - threatening if the update happens at a wrong time ( e.g. , updating the vehicle while driving ) .this article considers the security threats incurred by the heterogeneous links of iot and designs a novel patching scheme to alleviate malware propagation . instead of the impractical solution of directly patching compromised iot devices, we propose to patch important intermediate nodes based on the traffic volumes to prevent major security exploits and to avoid catastrophic malware propagation . with the proposed traffic - aware patching scheme , malware propagationis restricted to direct device - to - device connection , and therefore the damage of malware propagation can be significantly reduced .we conduct experiments in iot environment to demonstrate the effectiveness of the proposed traffic - aware patching scheme , and we also discuss some ongoing research challenges and open research questions related to iot patching .  the proposed traffic - aware patching scheme and the experimental results bring new insights to iot security . for instance , the infeasibility of direct patching on iot devices calls for new iot malware models and security assessment approaches . the experimental results can assist in developing new attack detection techniques and patching strategies for preventing malware propagation . obviously ,the resource - constrained , user - unfriendly , and heterogeneous features of iot devices hinder the security design and development for iot .however , the experimental results indicate a promising method to secure the entire iot system by patching intermediate nodes . in summary , we provide two guidelines for how to consider cyber security when designing iot systems accordingly :  * the consideration of intermediate nodes that bridge the gap between resource - constrained iot devices and powerful iot application servers is necessary when designing cyber security for iot . by shifting computation - consuming , security related functionalities ( e.g. , flow identification , filtering , and isolation ) to intermediate nodes, they can play the roles of the onsite guards .in particular , the flexibility and reconfigurability of intermediate nodes could easily introduce patches and updates to mitigate the iot malware propagation or attacks in a timely manner . * the future cyber security solution for iot should take into the consideration that adversaries might leverage iot devices with unpatched vulnerabilities to propagate malware via device - to - device links .in other words , the security mechanisms developed for iot shall coexist with insecure , unpatched legacy iot devices with uncontrolled device - to - device channels .a notification mechanism is suggested to help users identify the iot devices at risk and further deny possible device - to - device connections .this work was supported in part by taiwan information security center ( twisc ) , academia sinica , and ministry of science and technology , taiwan , under the grant most 104 - 2923-e-011 - 006-my2 , 105 - 2221-e-002 - 146-my2 , and 105 - 2218-e-001 - 001 .j. granjal , e. monteiro , and j. silva , `` security for the internet of things : a survey of existing protocols and open research issues , '' _ ieee commun .surveys tuts ._ , vol .12941312 , jul .2015 .m. miettinen , s. marchal , i. hafeez , n. asokan , a .- r .sadeghi , and s. tarkoma , `` iot sentinel : automated device - type identification for security enforcement in iot , '' _ corr _ , vol .abs/1611.04880v2 , 2016 .

Abstract: the various types of communication technologies and mobility features in internet of things ( iot ) on the one hand enable fruitful and attractive applications , but on the other hand facilitates malware propagation , thereby raising new challenges on handling iot - empowered malware for cyber security . comparing with the malware propagation control scheme in traditional wireless networks where nodes can be directly repaired and secured , in iot , compromised end devices are difficult to be patched . alternatively , blocking malware via patching intermediate nodes turns out to be a more feasible and practical solution . specifically , patching intermediate nodes can effectively prevent the proliferation of malware propagation by securing infrastructure links and limiting malware propagation to local device - to - device dissemination . this article proposes a novel traffic - aware patching scheme to select important intermediate nodes to patch , which applies to the iot system with limited patching resources and response time constraint . experiments on real - world trace datasets in iot networks are conducted to demonstrate the advantage of the proposed traffic - aware patching scheme in alleviating malware propagation .   heterogeneous links , iot malware , patching


successful deployment of robotic assistants in our society requires these systems to deal with high complexity and wide variability of their surroundings to perform typical everyday tasks robustly and without sacrificing safety .consequently , there exists a pressing need to furnish these robotic systems not only with discrete high - level reasoning ( e.g. , task planning , diagnostic reasoning ) and continuous low - level reasoning ( e.g. , trajectory planning , deadline and stability enforcement ) capabilities , but also their tight integration resulting in hybrid planning .  motivated by the importance of hybrid planning , recently there have been some studies on integrating discrete task planning and continuous motion planning .these studies can be grouped into two , where integration is done at the search level or at the representation level .for instance , take advantage of a forward - search task planner to incrementally build a task plan , while checking its kinematic / geometric feasibility at each step by a motion planner ; all these approaches use different methods to utilize the information from the task - level to guide and narrow the search in the configuration space . by this way, the task planner helps focus the search process during motion planning .each one of these approaches presents a specialized combination of task and motion planning at the search level , and does not consider a general interface between task and motion planning .  on the other hand , integrate task and motion planning by considering a general interface between them , using `` external predicates / functions '' , which are predicates / functions that are computed by an external mechanism , e.g. , by a c++ program .the idea is to use external predicates / functions in the representation of actions , e.g. , for checking the feasibility of a primitive action by a motion planner .so , instead of guiding the task planner at the _ search level _ by manipulating its search algorithm directly , the motion planner guides the task planner at the _ representation level _ by means of external predicates / functions . apply this approach in the action description language  using the causal reasoner  ; applies it in answer set programming ( asp )  using the asp solver  ; extends the planning domain description language pddl to support external predicates / functions ( called semantic attachments ) and modifies the planner ff accordingly .in these approaches , integration of task and motion planning is achieved at various levels .for instance , do not delegate all sorts of feasibility checks to external predicates as in , but implements only some of the feasibility checks ( e.g. , checking collisions of robots with each other and with other objects , but not collisions of objects with each other ) as external predicates and use these external predicates in action descriptions to guide task planning . for a tighter integration ,feasibility of task plans is checked by a dynamic simulator ; in case of infeasible plans , the planning problem is modified with respect to the causes of infeasibilities , and the task planner is asked to find another plan .  in this paper ,our goal is to better understand how much of integration between high - level reasoning and continuous low - level reasoning is useful , and for what sort of robotic applications .for that , we consider integration at the representation level , since this approach allows a modular integration via an interface , external predicates / functions , which provides some flexibility of embedding continuous low - level reasoning into high - level reasoning at various levels .such a flexible framework allowing a modular integration is important for a systematic analysis of levels of integration .we identify four distinct strategies to integrate a set of continuous feasibility checks into high - level reasoning , grouped into two : _ directly integrating _ low - level checks into high - level reasoning while a feasible plan is being generated , and generating candidate plans and then _ post - checking _ the feasibility of these candidate solutions with respect to the low - level checks . for direct integration we investigate two methods of integration : ( i ) low - level checks are done for all possible cases in advance and then this information is used during plan generation , ( ii ) low - level checks are done when they are needed during the search for a plan . for post - checkingwe look at two methods of integration : ( iii ) all plans are computed and then infeasible ones are filtered , ( iv ) by means of replanning , after finding a plan , low - level checks identify whether it is infeasible or not ; if it is infeasible , a new plan is computed considering the results of previous low - level checks .we consider these four methods of integration , as well as some of their combinations ; for instance , some geometric reasoning can be integrated within search as needed , whereas some temporal reasoning is utilized only after a plan is computed in a replanning loop . considering each method andsome of their combinations provide us different levels of integration .to investigate the usefulness of these levels of integration at representation level , we consider 1 ) the expressive formalism of hex programs for describing actions and the efficient hexsolver to compute plans , and 2 ) the expressive formalism of asp programs for describing actions and the efficient asp solver to compute plans . unlike the formalisms and solvers used in other approaches  , that study integration at representation level , hex  and  allow external predicates / functions to take relations ( e.g. , a fluent describing locations of all objects ) as input without having to explicitly enumerate the objects in the domain .other formalisms and solvers allow external predicates / functions to take a limited number of objects and/or object variables as input only , and thus they do not allow embedding all continuous feasibility checks in the action descriptions . in that sense , the use of hexprograms with , along with the asp programs with enriches the extent of our experiments .we perform experiments on planning problems in a robotic manipulation domain ( like in  ) and in a legged locomotion domain ( like in  ) .robotic manipulation domain involves 3d collision checks and inverse kinematics , whereas legged locomotion involves stability and reachability checks .we analyze the usefulness of levels of integration in these domains , both from the point of view of computational efficiency ( in time and space ) and from the point of view of plan quality relative to its feasibility .assume that we have a task planning problem instance ( consisting of an initial state , goal conditions , and action descriptions ) in a robotics domain , represented in some logic - based formalism .a history of a plan from the given initial state to a goal state computed for consists of a sequence of transitions between states : . a _low - level continuous reasoning module _ gets as input , a part of a plan history computed for and returns whether this part of the plan history is feasible or not with respect to some geometric , dynamic or temporal reasoning .for example , if the position of a robot at step is represented as and the robot s action of moving to another location at step is represented as , then a motion planner could be used to verify feasibility of the movement  . if duration of this action is represented as well , e.g. , as , then the low - level module can find an estimate of the duration of this movement relative to the trajectory computed by a motion planner , and it can determine the feasibility of the movement by comparing this estimate with .let denote a low - level reasoning module that can be used for the feasibility checks of plans for a planning problem instance .we consider four different methods of utilizing for computing feasible plans for , grouped into two : _ directly integrating _reasoning into , and _ post - checking _candidate solutions of using .  for_ directly integrating _ low - level reasoning into plan generation , we propose the following two levels of integration :  * * pre * _ precomputation _ we perform all possible feasibility checks of that can be required by , in advance . for each failed check , we identify actions that cause the failure , and then add a constraint to the action descriptions in ensuring that these actions do not occur in a plan computed for .we then try to find a plan for the augmented planning problem instance .clearly , every plan obtained with this method satisfies all low - level checks . * * int * _ interleaved computation _ we do not precompute but we interleave low - level checks with high - level reasoning in the search of a plan : for each action considered during the search , the necessary low - level checks are immediately performed to find out whether including this action will lead to an infeasible plan .an action is included in the plan only if it is feasible .the results of feasibility checks of actions can be stored not to consider infeasible actions repeatedly in the search of a plan .plans generated by interleaved computation satisfy all low - level checks .let us denote by and the low - level checks directly integrated into plan generation , with respect to preand int , respectively .alternatively , we can integrate low - level checks with , by means of _ post - checking _candidate solutions of relative to .we propose the following two methods to perform post - checks on solution candidates :  * * filt * _ filtering _ : we generate all plan candidates for . for each low - level check in , we check feasibility of each plan candidate and discard all infeasible candidates . * * repl * _ replanning _ : we generate a plan candidate for . for each low - level check in , we check feasibility of the plan candidate . whenever a low - level check fails , we identify the actions that cause the failure , and then add a constraint to ensuring that these actions do not occur in a plan computed for .we generate a plan candidate for the updated planning problem instance and do the feasibility checks .we continue with generation of plan candidates and low - level checks until we find a feasible plan , or find out that such a feasible plan does not exist .let us denote by the low - level checks done after plan generation , with respect to filtor repl .  figure [ fig : flows ] shows the hybrid planning framework we use in this paper to compare different levels of integration , and combinations thereof , on robotics planning scenarios . in particular , fig .[ fig : flows ] depicts computational components : precomputation extends the problem instance using a low - level reasoning module , planning integrates a low - level reasoning module into its search for a plan candidate for the problem instance generated by precomputation .postcheck uses a low - level module to verify solution candidates ( using filtor repl ) and to potentially add constraints to the input of planning .  in our systematic analysis of levels of integration, we do consider this hybrid framework by disabling some of its components .for instance , to analyze the usefulness of pre , we disable the other integrations ( i.e. , ) ; to analyze the usefulness of a combination of preand filt , we disable other integrations ( i.e. , ) .we investigate the usefulness of levels of integration as described above , considering two orthogonal properties : solution quality and planning efficiency .we quantify these properties as follows .( pre ) precomputation ; ( hex )  [ cols="^ " , ]   ; ( post ) postcheck ; ; ; ;  ( pre.south ) node[left ] ( hex.north ) ; ( hex.south ) node[left ] candidate solution ( post.north ) ; ( post.south ) node[left , near end ] feasible solution ( ) ;  ( ) -| ( pre.north ) ; at ( ) ;  ( ) + + ( 0,-1em ) + +( 6em,0 ) + + ( 0,8.7em ) + + ( -6em,0 ) ( ) ; at ( ) constraints ; at ( ) ;  * solution quality . *   if some low - level module is not integrated into the planning process , some plan candidates will be infeasible due to failed low - level checks of .we quantify solution quality by measuring the number of feasible and infeasible plan candidates generated by the search for a plan .this way we obtain a measure that shows how _ relevant _ a given low - level check is for plan feasibility . note that with the filtapproach an infeasible plan candidate simply causes a new plan to be generated , while with replan infeasible plan candidate causes computation of additional constraints , and a restart of the plan search .tightly connected to the number of feasible and infeasible solution candidates is the number of low - level checks that is performed until finding the first feasible plan , and until finding all feasible plans .* planning efficiency . *   we quantify planning efficiency by measuring the time required to obtain the first feasible plan , and the time to enumerate all feasible plans .( note that this includes proving that no further plan exists . )independent from the number of low - level checks , the duration of these external computations can dominate the overall planning cost , or it can be negligible .therefore we measure not only the number of computations of low - level modules but also the time spent in these computations .for our empirical evaluation we use the robotic manipulation and the legged locomotion domains . both require hybrid planning .we next give an overview of the domains , their characteristics , and scenarios we used .* robotic manipulation . *   we consider a cooperative robotic manipulation problem , as in  , where two robots arrange elongated objects in a space that contains obstacles .the manipulated objects can only be carried cooperatively by both robots , objects must not collide with each other or the environment , similarly robots must not collide with each other .a large part of collision checks between objects can already be realized in the high - level representation , however certain checks require usage of geometric models .collision - freeness between robots for particular collaborative actions can only be determined using low - level geometric reasoning and is not represented in the world model .therefore we use two low - level reasoning components to check collision - freeness : the checks collisions between the robotic arms , and collisions between an object and its environment .we experiment with 10 instances ( over a grid ) that require plans of upto 20 ( average 9.2 ) steps , and involving up to 58 ( average 25.1 ) actions .* legged locomotion .*   in the legged locomotion domain , a robot with high degrees of freedom must find a plan for placing its legs and moving its center of mass ( cm ) in order to move from one location to another one .for the purpose of studying integration of geometric reasoning with high - level task planning , we created a planning formulation for a four - legged robot that moves on a grid .some grid locations are occupied and must not be used by the robot .starting from a given initial configuration , the robot must reach a specified goal location where all legs are in contact with the ground.  as legged robots have high degrees of freedom , legged locomotion planning deals with planning in a high - dimensional space .we use a planning problem that is of similar complexity as has been investigated in climbing  and walking  robots .we also require a feasibility check of leg placement actions .we allow concurrent actions , i.e. , moving the center of mass while detaching a leg from the ground , if this does not cause the robot to lose its balance .we use a low - level reasoning component that determines whether the robot is in a balanced stable equilibrium ( ) , given its leg positions and the position of its cm .we realize this check by computing the support polygon of legs that are currently connected to the ground , and by checking if cm is within that polygon . for these checks we use the boost::geometry library to compute a convex hull of all leg positions , and then check whether cm is located within that convex hull .a second low - level module determines if leg positions are realistic wrt .the position of cm , i.e. , if every leg can reach the position where it is supposed to touch to the ground .this check ( ) is realized as a distance computation between coordinates of legs and cm .the domains we experiment with exhibit various differences in their characteristics , and such a variety allows us to get practically more relevant results .the most important differences between these two domains are as follows .* complexity of low - level reasoning . *in legged locomotion we use a c++ geometric library to perform basic geometric operations which are sufficient for computing check results .  in robotic manipulation ,object collision checking on 3d models of objects and environments , and requires inverse kinematics to determine the joint configuration of each robot reaching a certain point before performing collision checks between arms .hence , in legged locomotion , each low - level check requires less time and memory than in robotic manipulation .  * information relevant for low - level reasoning . *   in legged locomotion, we consider problem instances over a grid . a check over two coordinates , therefore there are possible .the balance check a totally different situation : we have an input of four leg coordinates and one cm coordinate , therefore , there are possible .such a large number of checks makes precomputation infeasible .  in robotic manipulation ,both low - level checks are over coordinate pairs on a grid ; therefore , there are low - level checks .  based on the number of low - level checks , precomputation forlegged locomotion seems feasible for only one of the two low - level modules ( ) , while for robotic manipulation we can apply precomputation for both low - level computations .indeed , precomputation for legged locomotion can be done in less than 1 second , and for robotic manipulation in 238 seconds .we applied different integration methods to 20 legged locomotion and 10 robotic manipulation instances of varying size and difficulty .tables [ tbleff ] and [ tblqual ] present results for  * * first * : obtaining the first feasible plan , and for * * all * : obtaining all ( maximum 10000 ) feasible plans .  in our experiments, we use a timeout of 2 hours ( 7200 seconds ) after which we stop computation and take measurements until that moment .we also limit the number of enumerated plans to 10000 plans .the measurements for enumerating up to 10000 plans reveal information about solution quality and provides a more complete picture of the behavior of each method : one method might find a feasible solutions very fast by chance , whereas finding many or all solutions fast by chance is unlikely .  * time measurements . *table [ tbleff ] shows measurements about planning efficiency and time spent in low - level reasoning .firstly , it is clear that preand int the direct integration methods outperform filtand repl the post - checking methods : for robotic manipulation , only preand intare able to enumerate all solutions within the given time limit ; for legged locomotion , only intand the pre / intcombination enumerates all solutions .  comparing the times required by preand int , we see that preis more efficient for robotic manipulation ( 888 sec vs 1007 sec on average ) , which is mainly due to efficient precomputation ( see below ) .even though preperforms better than int , it spends more time in low - level reasoning , hence high - level reasoning is faster there ; we can explain this by a more constrained search space ( low - level check results constraint the search ) .after preand int , the next best choice is repl : it finds solutions to 8 out of 10 instances in robotic manipulation , and it finds solutions to all instances for legged locomotion , whereas filthas the same number of timeouts in the manipulation domain and 5 timeouts for legged locomotion .in addition to that , we can see that replspends little time in low - level checks compared to other approaches .this is because replperforms many restarts of the high - level planner which causes it to spend a disproportionate amount of time in high - level planning .nevertheless , replshows its robustness by finding solutions to all but 2 instances .  finally , filtfails to find solutions for 7 instances in total which clearly makes it the worst - performing method .the time results for robotic manipulation suggest that filtmay be a bit faster than repl ; this may be an effect of some easy instances in that domain where replanning spends more time by reinitialization , than filtspends by iterating over many similar infeasible solutions .therefore , even in that domain , we would not suggest to use filt , as it might by chance , as low - level reasoning can not give feedback to high - level reasoning fail to find a feasible solution for a long time .* effort of low - level reasoning . *   in robotic manipulation , while attempting to enumerate all solutions , filtperforms only 724 low - level checks compared to 29282 checks of pre . similarly , in legged locomotion , filtperforms 35888 checks and fails to enumerate all solutions for 8 of 20 instances , while intenumerates all solutions while performing more ( 171109 ) low - level checks .note that these numbers ( the last column of table [ tbleff ] ) indicate _ distinct _ low - level reasoning tasks as we cache low - level check results .these numbers show that filtencounters a small fraction of the low - level checks that are needed to verify all solutions in int .caching in fact allows filtto verify much more actions than int(numbers not shown ) , however the number of _ distinct _ checks ( numbers shown ) is higher in int .we conclude that inttraverses the solution space much more efficiently .in legged locomotion , low - level checks depend on a large part of the candidate plan , so caching is not as effective as in robotic manipulation .this , together with the fact that in filtthe high - level is not guided by low - level checks , causes the filtapproach to spend more time in low - level reasoning than other approaches .note that , to obtain a reasonable comparison between preand the other approaches , we include times and counts of precomputed low - level checks in table [ tbleff ] ( which explains the large values for low - level computations in these rows ) .  * solution quality . *   methods preand intdonot generate infeasible solution candidates , as they use all low - level checks already in search .if we compare the number of infeasible solution candidates of filtand replin robotic manipulation , we observe that filtgenerates mainly infeasible solution candidates compared to the number of feasible solutions ( 11787 vs 622 ) while replcreates only 38 infeasible candidates while enumerating 621 feasible plans .in legged locomotion , the results for filtare similar , however replperforms a bit worse than in robotic manipulation with 250 infeasible candidates compared to 68 feasible solutions . a possible reason for this difference could be the same reason why preis not feasible in that domain : there is a large amount of possible inputs to to the other low - level checks we used . due to the large input space ,each failed constrains the search space only by a small amount , so replproduces more infeasible solutions than in robotic manipulation .* memory usage . *   we measured peak memory usage over the whole runtime of each instance .interleaved computation with the solver ( columns with int ) requires an average of around 2000 mb , the maximum stays below 4000 mb . for non - interleaved computations , and were connected with low - level checks using scripts .these approaches require around 400 mb of memory with a maximum below 1000 mb .* combination of prewith other methods . *as shown in the legged locomotion experiments , precan be combined with other approaches . in our experimentswe observe that adding preincreases efficiency .  however , preadds a fixed cost to solving because it precomputes many points . depending on efficiency of low - level computations ,even if there are few possible input combinations to low - level checks precomputation might be infeasible .dedicated precomputation methods can be more efficient than just checking for each possible input combination , e.g. , by saving on motion planner initialization . in our experiments we created such dedicated precomputation methods : for robotic manipulation dedicated precomputation takes 238 seconds in total , calling individual checks requires 1361 seconds in total . without dedicated efficient precomputation , preperforms worse than int .our experiments suggest the following conclusions . if robust and highly complex reasoning is required , and if this reasoning is done frequently ( so that performance gains will become relevant ) then using full interleaved reasoning ( int ) is the only good option . inthas the best performance with respect to run times , and it can enumerate most solutions compared to other approaches .the reason is that intuses only those low - level checks which are necessary ( they are computed on demand ) and therefore does not overload the solver with redundant information ( as predoes ) .furthermore , intconsiders failed checks in the search process and thereby never picks an action where it is known that the action will violate a low - level check .this is similar as in the replapproach , but much more efficient as the integration is much tighter compared to repl .however , the performance of intcomes at a price :  it requires more memory , and  it requires a solver that allows for interacting with the search process in a tight way , usually through an api that has to be used in a sophisticated way to be efficient .if reasoning operates on a manageable amount of inputs , such that precomputation is a feasible option , then preis a good choice . in our robotic manipulation experiments , preoutperforms all other methods , which is partially due to our using a dedicated efficient precomputation tool .in legged locomotion , combining prewith other methods also increased efficiency .the filtapproach performs the worst , because nothing guides the search into the direction of a feasible solution ; filtis not robust and enumerates many infeasible solutions .  if both preand intare not possible then replshould be used; this approach does not have the same performance as intand pre , however it is a very robust approach as it is guided by its wrong choices we can think of the constraints that are added for failed low - level checks as the approach ` learning from its mistakes ' .the benchmark results for legged locomotion clearly show the robustness of replcompared to filt : the former finds solutions for all problems , the latter only for 15 out of 20 instances .  a possible improvement to replcould be to let it enumerate a certain amount of solutions to gather more constraints , then add all these constraints and restart the search .this is a hybrid approach between filtand repl . selecting the right moment to abort enumeration and restartthe solver is crucial to the performance of such a hybrid approach , and we consider this a worthwhile subject for future investigations .this work is partially supported by tubitak grant 111e116 .peter schller is supported by tubitak 2216 research fellowship .gravot , f. , cambon , s. , alami , r. : planner that deals with intricate symbolic and geometric problems . in : robotics researchthe eleventh international symposium .volume 15 of springer tracts in advanced robotics .springer ( 2005 ) 100110  hauser , k. , latombe , j.c .: integrating task and prm motion planning : dealing with many infeasible motion planning queries . in : workshop on bridging the gap between task and motion planning at icaps .( 2009 )        erdem , e. , haspalamutgil , k. , palaz , c. , patoglu , v. , uras , t. : combining high - level causal reasoning with low - level geometric reasoning and motion planning for robotic manipulation . in : international conference on robotics and automation .( 2011 ) 45754581            bretl , t. , lall , s. , latombe , j.c . , rock , s.m . : multi - step motion planning for free - climbing robots . in : algorithmic foundations of robotics vi .volume 17 of springer tracts in advanced robotics . , springer ( 2005 ) 5974  bretl , t. , rock , s.m . ,latombe , j.c ., kennedy , b. , aghazarian , h. : free - climbing with a multi - use robot . in jr ., m.h.a . ,khatib , o. , eds . : iser .volume 21 of springer tracts in advanced robotics . , springer ( 2004 ) 449458

Abstract: we provide a systematic analysis of levels of integration between discrete high - level reasoning and continuous low - level reasoning to address hybrid planning problems in robotics . we identify four distinct strategies for such an integration : ( i ) low - level checks are done for all possible cases in advance and then this information is used during plan generation , ( ii ) low - level checks are done exactly when they are needed during the search for a plan , ( iii ) first all plans are computed and then infeasible ones are filtered , and ( iv ) by means of replanning , after finding a plan , low - level checks identify whether it is infeasible or not ; if it is infeasible , a new plan is computed considering the results of previous low - level checks . we perform experiments on hybrid planning problems in robotic manipulation and legged locomotion domains considering these four methods of integration , as well as some of their combinations . we analyze the usefulness of levels of integration in these domains , both from the point of view of computational efficiency ( in time and space ) and from the point of view of plan quality relative to its feasibility . we discuss advantages and disadvantages of each strategy in the light of experimental results and provide some guidelines on choosing proper strategies for a given domain .   [ firstpage ]  task planning , geometric reasoning , answer set programming .


aster models [ ] are a partial generalization of generalized linear models ( glm ) that allow different components of the response vector to have different families ( some bernoulli , some poisson , some zero - truncated poisson , some normal ) and also to be dependent , the dependence being specified by a simple graphical model . because of the way they incorporate dependence among components of the response , aster models are not glm nor like other regression models with which statisticians are familiar , but they are special cases of graphical models and of exponential families .although aster models can be used whenever their assumptions hold [ for which see ] , they were designed particularly for life history analysis of plants and animals , which aims to model total lifetime reproductive output ( observed darwinian fitness ) , a random variable that fits no familiar distribution , often having a large atom at zero ( individuals that died before producing offspring ) as well as multiple modes .aster models can fit such data adequately by using data on other components of fitness ( survival in each year , number of flowers in each year , number of seeds in each year , and number of seeds that germinate in each year for a plant and similar sorts of data for other organisms ) and modeling all these data jointly .it often turns out that , although the marginal distribution of total lifetime reproductive output is intractable , the conditional distribution of each component of the response vector given some other component is tractable ( e.g. , number of seeds per flower is poisson ) .biologists had recognized for decades that no statistical methods before aster allowed valid statistical analysis of life history data [ ] , so aster models are becoming widely used .here we extend aster models to allow for random effects .our applications illustrate three areas where random effects models are traditional .first , when one categorical predictor is nested within another , the effects for the nested predictor are commonly treated as random , especially when they are nuisance parameters .this is seen in both of our analyses of local adaptation .second , when levels of a categorical predictor ( such as years ) are not interesting in themselves but only as representatives of a larger population , the corresponding effects are commonly treated as random .this is seen in one of our analyses of local adaptation .third , , a paper that was the forerunner of all random effects models , introduced the idea of random effects representing the cumulative effects of many genes . to obtain evolutionary predictions from life history analysis ,random effects models are necessary .this is seen in our analysis of genetic variance for fitness .( mapping the genes that contribute to variation in fitness is not feasible ; the number of them is so large , and many are individually of such small effect , that it is unrealistic to generate a sufficiently large study population to detect an informative subset of them [ ] . if there were only a few genes for fitness , then sequencing and `` machine learning '' would help , but there is no sparsity here . ) as with glm with random effects ( generalized linear mixed models , glmm ) , aster models with random effects have analytically intractable likelihoods necessitating the use of monte carlo , numerical integration or approximate likelihood .markov chain monte carlo likelihood inference has a rich literature [ ] , but we have avoided it because it is very computationally intensive and also very difficult for ordinary users to do correctly .ordinary monte carlo [ ] has also been used , but is also very computationally intensive . numerical integration [ ]is useful when there is only one variance component but not otherwise [ , section 7.2 ] , and we have avoided this too .approximate integrated likelihood ( ail ) is based on the idea that if the complete data log likelihood were quadratic in the random effects , then the random effects could be integrated out analytically , and if the complete data log likelihood is only close to quadratic in the random effects , then this is a reasonable approximation , usually referred to as laplace approximation [ ] . for sufficiently large sample sizes and sufficiently few random effects ,the log likelihood is asymptotically expected to be approximately quadratic [ , chapter 6 ; ] , so this approximation may work well .we use a second approximation , also introduced by , that is likewise an assumption that the log likelihood is close to quadratic in the random effects .if the complete data log likelihood were exactly quadratic in the random effects , then all derivatives higher than second would be zero , and we assume this . since the ail already involves second derivatives with respect to the random effects of the complete data log likelihood , second derivatives of the log ail would involve fourth derivatives of the complete data log likelihood and would be computationally intractable .this approximation allows us to compute approximate second derivatives of the log ail and hence approximate observed fisher information .although we are particularly interested in aster models , our theory works for any exponential family model .the log likelihood can be written where is the canonical statistic vector , is the canonical parameter vector , and the cumulant function satisfies where denotes the vector of first partial derivatives and denotes the matrix of second partial derivatives .  we assume a canonical affine submodel with random effects determined by where is a known vector , and are known matrices , is a normal random vector with mean vector zero and variance matrix .the vector is called the _ offset vector _ and the matrices and are called the _ model matrices _ for fixed and random effects , respectively , in the terminology of the r function ` glm ` .we assume the matrix is diagonal , so the random effects are independent random variables .the diagonal components of are called _ variance components_.  the unknown parameter vectors are and , where is the vector of variance components .thus , is a function of , although this is not indicated by the notation .typically each variance component corresponds to many random effects , so each component of occurs multiple times as a diagonal element of .  in order to agree with the optimization literature , we prefer to minimize rather than maximize .thus , we use minus log likelihoods . minusthe complete data log likelihood is in case none of the variance components are zero .we deal with the case of zero variance components in sections [ secroot ] and [ seczero ] .let denote the result of minimizing ( [ eqfoo ] ) considered as a function of for fixed and .since minus the log likelihood of an exponential family is a convex function [ , theorem 9.1 ] and the middle term on the right - hand side of ( [ eqfoo ] ) is a strictly convex function , it follows that ( [ eqfoo ] ) considered as a function of for fixed and is a strictly convex function. moreover , this function has bounded level sets , because the first term on the right - hand side of ( [ eqfoo ] ) is bounded below [ , theorems 4 and 6 ] and the second term has bounded level sets .it follows that there is a unique global minimizer [ , theorems 1.9 and 2.6 ] .thus , is well defined for all values of and .we define minus the log ail to be \\[-8pt ] & & { } + \tfrac{1}{2 } \log \det \bigl [ z^t w\bigl(a + m \alpha+ zb^*\bigr ) z d + i \bigr],\nonumber\end{aligned}\ ] ] where denotes the identity matrix of the appropriate dimension , where is a function of and and is a function of , although this is not indicated by the notation .our equation ( [ eqlog - pickle ] ) is the negation of equation ( 5 ) in , who introduced the terminology _ penalized quasi - likelihood _ ( pql ) for this approach .minimizing ( [ eqlog - pickle ] ) gives approximate maximum likelihood estimates of and , and differentiating ( [ eqlog - pickle ] ) twice gives an approximate observed fisher information matrix .however , ( [ eqlog - pickle ] ) is not easy to differentiate because is already the second derivative matrix of the cumulant function , so second derivatives of ( [ eqlog - pickle ] ) involve fourth derivatives of the cumulant function . for aster modelsthere are no published formulas for derivatives higher than second of the aster model cumulant function and the software [ the r package ` aster ` , ] does not compute them .the derivatives do , of course , exist because every cumulant function of a regular exponential family is infinitely differentiable at every point of the canonical parameter space [ , theorem 8.1 ] .thus , we ignore derivatives higher than second , which is equivalent to assuming is constant or that and are quadratic .this leads to the following idea . rather than basing inference on ( [ eqlog - pickle ] ) , we actually use ,\hspace*{-28pt}\ ] ] where is a constant matrix ( not a function of and ) .this makes sense for any choice of that is symmetric and positive semidefinite , but we will choose that are close to , where and are the joint minimizers of ( [ eqlog - pickle ] ) and .note that ( [ eqlog - pickle - too ] ) is a redefinition of .hereafter we will no longer use the definition ( [ eqlog - pickle ] ) .  introduce ,\hspace*{-22pt}\ ] ] where , as the left - hand side says , , and are all free variables and , as usual , is a function of .since the terms that contain are the same in both ( [ eqfoo ] ) and ( [ eqkey ] ) , can also be defined as the result of minimizing ( [ eqkey ] ) considered as a function of for fixed and .thus , ( [ eqlog - pickle - too ] ) is a profile of ( [ eqkey ] ) and is the joint minimizer of ( [ eqkey ] ) .we now switch notation for partial derivatives , using subscripts to indicate derivatives , explained in more detail in section 1.6 of the accompanying technical report [ ]. then second derivatives of ( [ eqlog - pickle - too ] ) can be written using the implicit function theorem and the fact that minimizes ( [ eqkey ] ) as a particularly simple and symmetric form [ for a detailed derivation see sections 1.7 and 1.8 of ] .if we combine all the parameters in one vector and write instead of , we have this form is familiar from the conditional variance formula for normal distributions ; if is the partitioned variance matrix of a partitioned normal random vector with components and , then the variance matrix of the conditional distribution of given is assuming that is nondegenerate [ , theorem 2.5.1 ] .moreover , if the conditional distribution is degenerate , that is , if there exists a nonrandom vector such that , then almost surely , assuming and have mean zero [ also by , theorem 2.5.1 ] , and the joint distribution of and is also degenerate .thus , we conclude that if the ( joint ) hessian matrix of is nonsingular , then so is the ( joint ) hessian matrix of given by ( [ eqpsi - psi ] ) .the second derivatives of we need for the second derivatives of are ^{- 1 } z^t { \hat}{w } z e_j \\ & & \hspace*{29.7pt}{}\times\bigl [ z^t { \hat}{w } z d + i \bigr]^{- 1 } z^t { \hat}{w } z e_k \bigr),\end{aligned}\ ] ] where [ for a detailed derivation see section 1.8 of ] . in our use of the implicit function theorem we needed to be invertible . from the explicit form given abovewe see that it is actually positive definite , because is positive semidefinite by ( [ eqw ] ) .it is part of the folklore of random effects models that introducing square roots of variance components avoids issues with zero variance components and with constrained optimization .introduce new parameters by and new random effects by , where is diagonal and . then the objective function ( [ eqkey ] ) becomes .\ ] ] there are now no constraints ( the are allowed to be negative ) and ( [ eqkey - rooted ] ) is a continuous function of all variables ( there is no discontinuity when ) .we find this change - of - parameter useful and use it to avoid constrained optimization [ r package ` aster ` , ] .however , it also causes problems .first , it introduces spurious zeros of the first derivative of ( [ eqkey - rooted ] ) that are not stationary points of ( [ eqkey ] ) .in fact , the partial of ( [ eqkey - rooted ] ) with respect to is always zero when by symmetry .thus , first derivatives of ( [ eqkey - rooted ] ) can not be used to test whether the minimum occurs when some variance component is zero .since the issue of whether a variance component is zero is often of scientific interest , this is very problematic .we solve this problem by looking at first derivatives of ( [ eqlog - pickle - too ] ) on the original parameter scale ( section [ seczero ] below ) and using the theory of constrained optimization .second , the formula ( [ eqpsi - psi ] ) for observed fisher information , although guaranteed to be positive definite if infinite precision arithmetic is used , is not so guaranteed if it is evaluated by the usual computer arithmetic ( with 16 decimal digit precision ) .we found that the analog of ( [ eqpsi - psi ] ) after the change of parameter from to was even more computationally unstable .thus , although ( [ eqkey - rooted ] ) is useful for finding approximate maximum likelihood estimates , we find it problematic for calculating approximate observed fisher information or for determining whether approximate maximum likelihood estimates of variance components are zero .in order to determine whether the minimizer of ( [ eqkey ] ) occurs on the boundary of the parameter space where some variance component is zero , we need to use the theory of constrained optimization . unfortunately , we can not use the karush kuhn tucker theory [ , section 9.1 ; , section 12.2 ] , which is familiar to some statisticians , because the constraint set is not determined by smooth inequality constraints .more advanced nonsmooth analysis [ ] does handle our problem , but is unfamiliar to most statisticians .fortunately , for our analysis we can use a simplification of the latter theory based on the notion of directional derivatives .the technical report [ ] uses the full theory from , but the results are the same as those stated here in terms of directional derivatives .the formula ( [ eqkey ] ) makes sense when all variance components are positive ( so is invertible ) .otherwise , it does not . as is common in nonsmooth analysis [ , section 1a ], we define the objective function to have the value off of the constraint set .since can never minimize the objective function , this incorporates the constraints in the objective function . on the boundary of the constraint set ( where some variance components are zero and the corresponding random effects are also zero ) we extend the objective function by lower semicontinuity .  since all but the middle term on the right - hand side of ( [ eqkey ] ) are actually defined on some neighborhood of each point of the constraint set and differentiable at each point of the constraint set , we only need to deal with the middle term .define let denote the variance of , and let denote the number of random effects . then ( [ eqkey ] ) can be rewritten \\[-8pt ] & & { } + \frac{1}{2 } \log \det \bigl [ z^t { \hat}{w } z d + i \bigr],\nonumber\end{aligned}\ ] ] where is given by ( [ eqh - lsc ] ) , provided all of the components of nonnegative .the proviso is necessary because the third term on the right - hand side is not defined for all values of , only those such that the argument of the determinant is a positive definite matrix .hence , we must separately define whenever any component of is negative .a necessary condition for a local minimum of a smooth function is that the first derivative is zero ( fermat s rule ) .this works at points in the interior of the constraint set where ( [ eqkey - lsc ] ) is differentiable .it does not work at points on the boundary .there we need what rockafellar and wets [ ( ) , theorem 10.1 ] call _ fermat s rule , generalized : _ a necessary condition for a local minimum is that all directional derivatives are nonnegative .  for any extended- real - valued function on , the directional derivative of at the point in the direction is defined by at a point where is differentiable , we have , and the notion of directional derivatives gives no information that can not be obtained from partial derivatives .it is only on the boundary where we need directional derivatives .  in the interior of the constraint set , where this function is smooth , ordinary calculus gives where the notation on the left - hand side means the directional derivative of at the point in the direction . on the boundary of the constraint set , which consists of the single point ,the directional derivatives are given by    this theory tells us nothing we did not already know about points in the interior of the constraint set . the only way we can have for all vectors is if .it is only at points on the boundary of the constraint set , where directional derivatives are the key .  even on the boundary , the conclusions of the theory about components of the state that are not on the boundary agree with what we already knew . at a local minimumwe have and \\[-8pt ] p_{b_i}(\alpha , b , \nu ) & = & 0,\qquad \mbox{ such that }\nonumber\end{aligned}\ ] ] [ , section 1.10.4 , gives details ] .thus , assuming that we are at a point ) where ( [ eqdescent - alpha ] ) and ( [ eqdescent - other - smooth ] ) hold , and we do assume this throughout the rest of this section , actually involves only and such that and .define \ ] ] [ the part of ( [ eqkey - lsc ] ) consisting of the smooth terms ] .then \\[-8pt ] & & \qquad = \sum _ { j \in j } \biggl [ v_j \bar{p}_{\nu_j } ( \alpha , b , \nu ) + \sum_{i \in k^{- 1}(j ) } \bigl ( u_i \bar{p}_{b_i}(\alpha , b , \nu ) + h(u_i , v_j ) \bigr ) \biggr],\nonumber\end{aligned}\ ] ] where is the set of such that , where denotes the set of such that , and where is defined by ( [ eqh - lsc ] ) . to check that we are at a local minimum, we need to show that ( [ eqdescent - other - nonsmooth ] ) is nonnegative for all vectors and .conversely , to verify that we are not at a local minimum , we need to find one pair of vectors and such that ( [ eqdescent - other - nonsmooth ] ) is negative .such a pair we call a _ descent direction_. since fermat s rule generalized is a necessary but not sufficient condition ( like the ordinary fermat s rule ) , the check that we are at a local minimum is not definitive , but the check that we are not is .if a descent direction is found , then moving in that direction away from the current value of will decrease the objective function ( [ eqkey - lsc ] ) .so how do we find a descent direction ?we want to minimize ( [ eqdescent - other - nonsmooth ] ) considered as a function of and for fixed , and .we can consider the terms of ( [ eqdescent - other - nonsmooth ] ) for each separately .if the minimum of over all vectors and is nonnegative , then the minimum is zero , because ( [ eqdescent - other - nonsmooth - j ] ) has the value zero when and .thus , we can ignore this in calculating the descent direction .since we are only interested in finding a descent direction , the length of the direction vector does not matter .thus , we can do a constrained minimization of ( [ eqdescent - other - nonsmooth - j ] ) , constraining to lie in a ball .this is found by the well - known karush kuhn tucker theory of constrained optimization [ , section 9.1 ; , section 12.2 ] to be the minimum of the lagrangian function where is the lagrange multiplier , which would have to be adjusted if we were interested in constraining to lie in a particular ball .since we do not care about the length of , we can use any .we have replaced by because we know that if we are finding an actual descent direction , then we will have .now the minimum occurs where these are zero .setting the first equal to zero and solving for gives plugging this back into the second gives and we seek zeros of this .the right - hand is clearly an increasing function of , so it is negative somewhere only if it is negative when where it has the value so that gives us a test for a descent direction : we have a descent direction if and only if ( [ eqdescent - test ] ) is negative .conversely , we appear to have if ( [ eqdescent - test ] ) is nonnegative .we illustrate the use of this work with three examples , beginning with a study of the invasive california wild radish ( _ raphanus sativus _ ) described by . for each individual , three response variables are observed , connected by the following graphical model : with being an indicator of whether any flowers were produced , being the count of the number of flowers produced , being the count of the number of fruits produced , the unconditional distribution of being bernoulli , the conditional distribution of given being zero - truncated poisson , and the conditional distribution of given being poisson .( the combination of a bernoulli arrow followed by a zero - truncated poisson arrow gives a combined zero - inflated poisson distribution , that is , the unconditional distribution of is zero - inflated poisson . )these data are found in the data set ` radish ` in the r package ` aster ` .they come from a designed experiment started with seeds collected from three large wild populations of northern , coastal california wild radish and three populations of southern , inland california wild radish .thus , we have populations nested within region .plants were grown at two experimental sites , one northern , coastal california field site located at point reyes national seashore and one southern , inland site located at the university of california riverside agricultural experiment station .thus , we have blocks nested within site .the issue of main scientific interest is the interaction of region and site , which is indicative of local adaptation when the pattern of mean values shows that each population has higher fitness in its home environment than in other environments .testing significance of this interaction is complicated by the nesting of populations within region and blocks within site and the goal of scientists to account for variation due to these nested factors in evaluating effects of the higher factors .the best surrogate of fitness in these data is the number of fruits produced .thus , we form the `` interaction '' with the indicator of this component and all scientifically interesting predictors [ see section 5 of or section 4 of ] .  the traditional way to deal with a situation like this is to treat the population effects as random ( within region ) and the block effects as random ( within site ) .when we fit this model [ see the technical report for details ] , we obtained positive and statistically significantly greater than zero estimates of both variance components and an estimate 0.499 with standard error 0.012 for the fixed effect that is the scientifically important site - region interaction parameter . did not do a random effects aster analysis because it had not yet been invented . nevertheless , the conclusions from their fixed effect aster analysis hold up .the main conclusion of interest is that there is evidence of local adaptation .this is indicated by the statistical significance of the fixed effect for region - site interaction together with the pattern of mean values for the different populations in the two sites , showing that populations growing near to their sampling locations had higher fitness than in the other location as found by .the fact that random effects analysis and fixed effects analysis agree qualitatively on this one example does not , of course , imply that they would agree on all examples . in these datathe region - site interaction is very large and almost any sensible statistical analysis would show it .when the interaction is not so large , the analysis done will make a difference .the analysis reported above is based on the approximations derived in the theory section .we are using the log approximate integrated likelihood and its hessian matrix to do likelihood inference .but what if its approximations are not valid ?section 6 of does a parametric bootstrap of this analysis .it turns out that a 95% confidence interval for the parameter of interest ( the region - site interaction ) does not change much , but other aspects of the parametric bootstrap are interesting . sampling distributions of the estimates of the variance components ( as simulated by the parametric bootstrap )turn out to be highly nonnormal , and these estimators have bias that is a significant fraction of their standard errors .we use data on the slender wild oat ( _ avena barbata _ ) described by and contained in the data set ` oats ` in the r contributed package ` aster ` . for each individual , two response variablesare observed , connected by the following graphical model : with being an indicator of survival and being the count of the number of spikelets ( compound flowers ) produced , the unconditional distribution of being bernoulli , and the conditional distribution of given being zero - truncated poisson .these data come from a designed experiment started with seeds collected in the 1980s in northern california of the xeric ( found in drier regions ) and mesic ( found in less dry regions ) ecotypes .the variable ` gen ` is the ecotype ( `` ` x ` '' or `` ` m ` '' ) .the variable ` fam ` is the accession ( nested within ` gen ` ) .the variable ` site ` is the site .the variable ` year ` is the year ( 2003 to 2007 ) .the experimental sites were at the sierra foothills research and extension center ( ) , which is northeast of sacramento on the east side of the central valley of california , and at the hopland research and extension center ( ) , which is in the california coastal ranges north of san francisco .hopland receives 30% more rainfall and has a less severe summer drought than the sierra foothills .the best surrogate of fitness in these data is the number of spikelets produced .thus , we form the `` interaction '' with the indicator of this component and all scientifically interesting predictors .  in the previous analysis [ ] a linear mixed modelwas used , despite the response being highly nonnormal , because no better tool was available .here we reanalyze these data using the same random effects structure in an aster model .  [cols="<,^",options="header " , ]   we have only three fixed effects parameters because there are only two levels of ` site ` and two levels of ` gen ` .there are five variance components , one for each row of the table having random type .all variance components are estimated to be significantly different from zero except for the ` fam ` random effect , which is estimated to be exactly zero .the results of the reanalysis agree qualitatively with the original analysis .local adaptation , which would have been shown by a statistically significant site - ecotype ( ) interaction , was not found in either analysis ( for this interaction , the aster random effects analysis obtained the point estimate 0.091 and standard error 0.143 ) .moreover , the pattern of mean values was not consistent with local adaptation . found that the mesic ecotype had higher fitness ( survived and reproduced better ) in all environments .this means that even if the site - ecotype interaction had been statistically significant , it would not have indicated local adaptation .we use data on the partridge pea ( _ chamaecrista fasciculata _ ) described by etterson ( ) and and contained in the data set ` chamae3 ` in the r contributed package ` aster ` . _c. fasciculata _ grows in the great plains of north america from southern minnesota to mexico .three focal populations were sampled in the following locations :  1 .kellog - weaver dunes , wabasha county , minnesota ; 2 .konza prairie , riley county , kansas ; 3 .pontotoc ridge , pontotoc county , oklahoma .these sites are progressively more arid from north to south and also differ in other characteristics .seed pods were collected from 200 plants in each of these three natural populations . from these ,plants were grown and crosses were done ; parent plants are indicated by the variables ` sire ` and ` dam ` in the data set .the resulting seeds were germinated and established as seedlings in the greenhouse and then planted using a randomized block design [ ] in three field sites :  1 .robert s. kerr environmental research center , ada , oklahoma ; 2 .konza prairie research natural area , manhatten , kansas ; 3 .university of minnesota , st .paul minnesota .the oklahoma field site was 30 km northwest of the oklahoma natural population ; the kansas field site was 5 km from the kansas natural population ; the minnesota field site was 110 km northwest of the minnesota natural population .for each individual , two response variables are observed , connected by the following graphical model : with being an indicator of whether any fruits were produced , being the count of the number of fruits produced , the unconditional distribution of being bernoulli , and the conditional distribution of given being zero - truncated poisson .  we here consider a subset of data previously analyzed by nonaster methods by etterson ( ) and and by aster without allowing for random ( genetic ) effects by .though seed counts were also observed , the complexity of the seed count data makes analysis difficult [ ] , so it does not serve as a good example .thus , here we analyze only the pod number data , which does have straightforward aster analysis and serves as a better example , even though this makes our reanalysis not really comparable with the analysis in which does use the seed counts . to aid design of future experiments , , page e43 ,explain two alternative experimental designs that permit straightforward aster analysis ( including random effects aster models ) . used one of these designs .individuals descended from all three natural populations were planted in all three field sites , so these data can address local adaptation and previous analyses [ , discussion ] did find local adaptation .but local adaptation is not the main point of interest for our analysis here .instead we investigate sire and dam effects , which we treat as random effects , as did the previous conventional quantitative genetics analysis [ ] .we focus on sire effects because in this experimental design sire effects are expected to correspond closely to pure breeding values ( additive genetic effects ) but dam effects confound additive with maternal and dominance effects .because the biology that leads to fitness may differ at different sites and in different populations , we did nine separate analyses , one for each population - site combination .we found that the sire variance components for the minnesota and oklahoma natural population are not close to statistically significant at the minnesota field site .all the other sire variance components are at least borderline statistically significant .our analysis produces not only estimates of the variance components but also estimates of the random effects ( these are the penalized quasi - likelihood estimates described in the theory sections ) . as a matter of purely statistical interest, we examined the gaussianity of the random effects .they seemed to be normal ( or at least not statistically significantly nonnormal by a shapiro wilk test ) .we conjectured that this apparent normality was due to the estimation procedure , but this turned out not to be the case , since when we redid penalized quasi - likelihood estimates of the random effects using much smaller penalties than the maximum likelihood penalty , the random effects still seemed normal .  for interpretability ,biologists want random effects mapped to the mean value parameter scale ( rather than the canonical parameter scale where they originally are ) . to illustrate this , we mapped the sire effects for two population - site pairs to the mean value parameter scale , setting the dam effects to be zero ( the middling value ) and setting the block effect to be block 1 ( each site was divided into blocks ) .figure [ figone ] shows these plots ; for details of how they were done see section 8.6 of .this figure was made using the default smoothing parameter selection of the r function ` density ` .the apparent non - gaussianity is not statistically significant [ shapiro wilk test , , section 8.6 ] .thus , the aster model can include random effects for parents and permit quantitative genetic inference for fitness variation .our methods are founded on two approximations taken from .our technical innovations are that we provide derivatives of the log approximate integrated likelihood ( section [ seclail ] ) and the test ( [ eqdescent - test ] ) for when variance components are zero , which is based on the theory of constrained optimization .our methods work well when there are multiple variance components .two examples had two variance components and one had five variance components .however , problems arise when there are thousands of random effects , and especially when there is one random effect per individual .since quantitative genetics traditionally does have individual random effects as well as parental random effects , our methods are not fully comparable to traditional quantitative genetics .past experience [ ] with examples taken from the literature shows that log integrated likelihoods are often far from quadratic , even when no approximations are done , in which case asymptotics based on fisher information are inaccurate .thus , we recommend the parametric bootstrap here , as we do whenever there is doubt about the validity of asymptotics for parametric inference .we illustrate the parametric bootstrap for one of our examples .this need for doing a parametric bootstrap is another reason for preferring computationally efficient methods . in particular , it is incredibly time consuming to bootstrap monte carlo calculations if monte carlo run lengths are long enough for accurate calculation . introduced yet another approximation that is supposed to be analogous to restricted maximum likelihood ( reml ) , but we did not use this .first , the analogy to reml is weak , and this method has no provable mathematical properties .second , we do not see how this method extends to general exponential family models , such as aster models .third , even in conventional linear mixed models , reml does not seem to be appropriate when the parameters of interest are fixed effects , which is often the case in biology and is the case in some of our examples .

Abstract: random effects are implemented for aster models using two approximations taken from breslow and clayton [ _ j . amer . statist . assoc . _ * 88 * ( 1993 ) 925 ] . random effects are analytically integrated out of the laplace approximation to the complete data log likelihood , giving a closed - form expression for an approximate missing data log likelihood . third and higher derivatives of the complete data log likelihood with respect to the random effects are ignored , giving a closed - form expression for second derivatives of the approximate missing data log likelihood , hence approximate observed fisher information . this method is applicable to any exponential family random effects model . it is implemented in the cran package ` aster ` ( r core team [ r : a language and environment for statistical computing ( ) r foundation for statistical computing ] , geyer [ r package aster ( 2012 ) http://cran.r-project.org/package=aster ] ) . applications are analyses of local adaptation in the invasive california wild radish ( _ raphanus sativus _ ) and the slender wild oat ( _ avena barbata _ ) and of additive genetic variance for fitness in the partridge pea ( _ chamaecrista fasciculata _ ) .  ,  ,  , +