the internet and its main application the web have been growing continuously in recent years .just in the last two years the web content has doubled in size from billion to over billion pages , according to results from google and yahoo .unfortunately , such growth has been accompanied by a parallel increase of nefarious activities , as reported by  .the web represents a very appealing platform for various types of electronic fraud , of which phishing and typosquatting are two examples ._ phishing _ is the well - known activity aimed at eliciting sensitive information e.g. , usernames , passwords and credit card details from unsuspecting users .it typically starts with a user being directed to a fake web site with the look - and - feel of a legitimate , familiar and/or well - known web site .consequences of phishing can range from denial - of - service to full - blown identity theft , followed by real financial losses . only in 2007 ,more than 3 billion u.s .dollars were lost to such attacks  ._ typosquatting _ is the practice of registering domain names that are typographical errors ( or minor spelling variations ) of well - known web site addresses ( target domains ) .it is often related to domain parking services and advertisement syndication , i.e. instructing browsers to fetch advertisements from a server and blending them with content of the web site that the user intends to visit .in addition to displaying unexpected pages , typo - domains often display malicious , offensive and unwanted content , install malware ( ) and certain typo - domains of children - oriented web sites redirect users to adult content .worse yet , typo - domains of financial web sites can serve as natural platforms for _ passive _ phishing attacks .recent studies have assessed the popularity of both types of malicious activities .for example , in fall 2008 , mcafee alert labs found more than domains typosquatting on just the top web sites  . also , according to the anti - phishing working group , the number of reported phishing attacks between april 2006 and april 2007 exceeded  .nevertheless , the problem remains far from solved and continues to be a race between the web - fraudsters and the security community .our goal is to counter web - fraud by detecting domains hosting such activities .our approach is inspired by several recent discussions and results within the security community .security researchers and practitioners increasingly advocate transition to https for all web transactions , similar to that from telnet to ssh .eminent examples of such discussions can be found in  and  .moreover , a recent study  has pointed out the emergence of phishing attacks abusing ssl certificates .the main goal is to avoid raising the suspision of users by masquerading as legitimate `` secure '' site .this leads us to the following questions :  1 .how prevalent is https on the internet ? 2.  how _ different _ are ssl certificates used by web - fraudsters from those of legitimate domains ? 3 .can we use the information in the ssl certificates to identify web - fraud activities such as phishing and typosquatting , without compromising user privacy ?  0this paper makes several contributions , by answering the above questions : * first * , we present a novel technique to detect web - fraud domains using https .it relies on a classifier that analyzes certificates of such domains .we validate the classifier by training and testing it over data collected from the internet .the classifier achieves a detection accuracy of 80% and , in some cases , as high as 95% , with few false positives .our classifier is orthogonal to existing mitigation techniques and can be integrated with other methods that do not rely on https . *second * , to justify using https , we measure the overall prevalence of https in popular and randomly sampled internet domains .we then consider the popularity of https in the context of web - fraud by studying its use in phishing and typosquatting activities .we then analyze , for the first time , all fields of ssl certificates .we identify useful features and patterns that represent possible symptoms of web - fraud ; they guide the design of our detection technique .finally , our findings serve as an additional motivation for increasing the use of https , not only to guarantee confidentiality and authenticity , but also to combat web - fraud .  * contributions . *this paper makes several contributions .first , we measure the overall prevalence of https in popular and randomly sampled internet domains .we then consider the popularity of https in the context of web - fraud by studying its use in phishing and typosquatting activities .we then analyze , for the first time , all fields of ssl certificates and we identify useful features and patterns that represent possible symptoms of web - fraud . based on the measurement findings , we propose a novel technique to identify web - fraud domains that use https , based on a classifier that analyzes certificates of web domains .we validate the classifier by training and testing it over data collected from the internet .the classifier achieves a detection accuracy over 80% and , in some cases , as high as 95% .our classifier is orthogonal to prior mitigation techniques and can be integrated with other methods that do not rely on https to face a complete range of malicious domains .also the integration with pre - existing solutions could improve their efficiency in the context of potentially critical applications enforcing https .note that the classifier only relies on data in the ssl certificate and not any other private user information .finally , our findings serve as an additional motivation for increasing the use of https , not only to guarantee confidentiality and authenticity , but also to help combat web - fraud .  * paper organization . * the rest of the paper is organized as follows .section [ sec : x509 ] introduces a brief overview of x.509 certificates .section [ sec : experiments ] presents the rationale and details of our measurements and analysis . in section [ sec : classifier ] , we describe the details of a classifier that detects malicious domains , based on information obtained from ssl certificates .section [ sec : discussion ] discusses the implications of our findings and the limitations of our study .section [ sec : related ] contains an overview of related work .finally , we conclude our paper in section [ sec : conclusion ].  0   * x.509 certificate format*,scaledwidth=45.0% ]the term _ x.509 certificate _ usually refers to an ietf s pkix certificate and crl profile of the x.509 v3 certificate standard , as specified in rfc 5280 . in this paperwe are concerned with the public key certificate portion of x.509 . in the rest of the paper, we refer to the public key certificate format of x.509 as a _ certificate _ , or an _ ssl / https certificate_.  according to x.509 , a certification authority ( ca ) issues a certificate binding a public key to an x.500 distinguished name ( or to an alternative name , e.g. , an e - mail address or a dns - entry ) .web - browsers such as internet explorer , mozilla / firefox , chrome and safari come with pre - installed trusted root certificates .browser software makers determine which cas are trusted third parties for the browsers users .the root certificates can be manually removed or disabled , however , users rarely do so .the general structure of an x.509 v3 certificate is shown in figure [ fig : x509format ] . as discussed later in section [ sec : experiments ] , we analyze _ all fields _ for certificates of both legitimate and malicious domains .we first describe our data sets and how we collect them .we then present our analysis and show how these results guide the design of a classifier that detects web - fraud domains .our measurement data was collected mainly during september 2009 .it includes three types of domain sets : _ legitimate , phishing and typosquatting_. each domain is probed twice .we probe each domain for web existence ( by sending an http request ) and for https existence ( by sending an https request ) .if a domain responds to https , we harvest its ssl certificate .below , we describe the data sets and the state of http / https utilization in them , also summarized in table [ table : data set - desc ] .* legitimate and popular domain data sets . *the following data sets are considered for measurements and analysis of popular and legitimate domains : + * * : most popular domains according to  + * * : random samples of domain zone file , collected from verisign . + * * : random samples of domain zone file , collected from verisign . is a subsidiary of amazon.com known for its web - browser toolbar and for its web site that reports traffic ranking of internet web sites .the company does not provide information on the number of users using its toolbar , but it claims several millions .we use ranking as our measure of popularity of legitimate domains .we also randomly sample and to ensure an unbiased data - set . in the rest of the paper ,we use , , and to denote three respective data sets containing the domains or certificates harvested as described above .* we find that 34% of domains use https ; 26% in and 16% in * .the fact that 34% of domains respond to https indicates a healthy degree of https usage among popular internet domains .our explanation for lower percentage in is that , perhaps because domains are historically non - commercial , most of them do not require user interaction and do not involve sensitive data , in contrast with commercial - minded domains . 0the observed -percent https utilization represents an * increase * of about over the set probed 6 months earlier , and over 1 year ago . as a result , it is expectable that this percentage may continue increasing in the near future .financial institutions on the internet from forbes magazine  .we find that that use https , as opposed to 1 year ago . ]* phishing data set . * we collected domains considered to be hosting phishing scams .they were obtained from the phishtank web site as soon as they are listed .we note that reported urls in phishtank are verified by several users and , as a result , are malicious with high probability .we consider this data set as a baseline for phishing domains .we find that a significant percentage  of these phishing web sites employ https .this represents a 5% increase over earlier measurements conducted in october 2008 .a similar trend has also been independently observed in a recent study by symantec .furthermore , we discovered a number of phishing domains using https ( % ) for which we can not obtain corresponding certificates , due to various ssl errors ( e.g. , illegal certificate size ) .* typosquatting data set .* to collect ssl certificates of typosquatting domains we first identified the typo domains in our and data sets by using google s typo correction service  to identify possible typos in domain names .this results in typo domains. however , such domains might be benign domains that _ accidentally _ resemble typos of well - known domains .we identified the typosquatting domains in this set by detecting the parked domains among these typo domains using the machine - learning - based classifier proposed in .we discovered that out of are parked domains .we consider these names as the data set of typosquatting domains .  asshown in table [ table : data set - desc ] , the result is that  domains use https .they represent our ssl certificate corpus . for convenience , when we refer to /  set , we mean the set of / domains that have ssl certificates .* overlapping of sets .* there is negligible overlapping among data sets .the size of the domain intersection between  and is 81 and the size of domain intersection between and  is 17 .there is no overlapping between and both and .the typosquatting set is a subset of the union of and sets , but it only represents a very small ratio ( 1% of the union of and sets ) . surprisingly , there are four phishing domains that also belong to set , but the ratio of phishing domains in set is extremely small ( 0.005% of set ) .thus , popular domains could ( in very rare cases ) host scam / phishing web sites .* take away . *based on the analysis of https usage in our data sets , we attempt to answer the first question posed in section  [ sec : introduction ] and assess how prevalent is the usage of https today , we find that a significant percentage of popular and legitimate domains already use https , and so do a relevant percentage of phishing domains . as a result , we proceed by analyzing the differences between the certificates of legitimate and fraudulent web sites .   the goal of this analysis is to guide the design of our detection method , the classifier .one side - benefit of the analysis is that it reveals the differences between certificates used by fraudulent and legitimate / popular domains . in total, we identified 15 distinct certificate features listed in table [ tbl : features ] .most features map to actual certificate fields , e.g. , f1 and f2 .others are computed from fields in the certificate but are not directly reflected in the fields , e.g. , certificate validation failure ( f5 ) or jaccard distance between host name and common name in the subject field ( f15 ) .note that some features match to boolean values , whereas , others are integers , reals or strings . finally , we believe that most interesting results correspond to : f1 ( md5 ) , f3 ( self - signed ) , f4 ( expired ) , f5 ( verification failed ) , f6 ( common certificate ) and f15 ( host - common name distance ) .* analysis of certificate boolean features .* features f1f8 have boolean values , e.g. , f1 ( md5 ) is true if the signature algorithm used in the certificate is `` md5withrsaencryption '' .the analysis and distributions of these features ( as shown in table [ tbl : boolean - features - analysis ] ) reveal interesting and unexpected issues and differences between legitimate and malicious domains .19% of certificates ( 24% and 22 % of  and , respectively ) use `` md5withrsaencryption '' .this feature has a much higher ratio ( 35% ) in but only 26% in .the higher ratio for suggests a possible correlation between phishing domain certificates and `` md5withrsaencryption '' signature algorithm .we were surprised that such a high percentage of legitimate domains is using `` md5withrsaencryption '' as the signature algorithm despite the well - known fact that rogue certificates can be constructed using md5 ( , ).   indicates whether the subject fields have some bogus values ( e.g. , `` st = somestate '' , `` o = someorganization '' , `` cn = localhost '' , ... ) .11% of certificates ( 7% and 8% of and , respectively ) satisfy this feature .this percentage is much higher ( 20% ) in ( 29% in ) .this might indicate that web - fraudsters fill subject values with bogus data or leave default values when generating certificates .28% of  certificates are self - signed ( 24% and 27% of and , respectively ) .we did not expect such a high percentage among legitimate domains .an interesting open question is to investigate the reason(s ) for all these certificates being self - signed ; however , this is out of the scope of our present work .the percentages of self - signed certificates in and are 36% and 53% , respectively .this is expected , since miscreants running such domains avoid leaving a trail by obtaining a certificate from a ca , which requires some documentation and payment . despite high ratios in legitimate sets, there is a significant variance between the ratios in the legitimate sets and ( 8 - 12% ) .this variance is even larger between legitimate sets and ( 25 - 29% ) .another alarming result is the fraction of expired certificates , 21% of ( 25% and 22% of and ) as well as 26% and 43% of and , respectively .such high percentages of expired certificates are less surprising for and than for legitimate domains .there is a non - negligible difference in the ratios between and ( 5% ) .we use openssl 0.9.8k to validate certificates and find that 29% of certificates ( 40% and 37% of and ) fail verification .also , 36% and 70% of and certificates fail verification . the percentages in and  sets are higher than in maybe because these are random domains and mostly unpopular thus , having a valid certificate is not essential to these domains . 0we note that these results include those for f3 and f4 : certificates failing validation are those that satisfy f3 , f4 or both .an impressive 70% of certificates in fail validation , even though 53% are self - signed .f6 indicates whether the certificate is also used for multiple domains in our data sets .33% of certificates in are duplicated ( 60% and 64% of and , respectively ) .the percentage goes up to 72% for and 95% for .f7 indicates whether a certificate serial number is used in another certificate in our data sets .38% of certificates have common serial numbers ( 33% were entirely duplicated ) .also , 64% and 68% of and as well as 75% and 96% in and satisfy f7 .certificates in and have higher duplication ratios than legitimate data sets ( especially ) suggesting that f7 , perhaps combined with other features , could be helpful in identifying phishing and typosquatting domains . 0as discussed earlier , the intersection between domains in our data sets is very small ; thus , the common certificate percentage is not polluted by accidental duplication .it seems intuitive that malicious domains would not acquire certificates valid for long periods of time .thus , we analyze the percentages of certificates with validity periods exceeding 3 years. has 17% ( 14% and 17% of and , respectively ) , while has 13% and  19% .we present further details on the exact duration of validity periods in the analysis of non - boolean features in section [ non - boolean - analysis ] .as expected , the percentage of certificates satisfying f8 is smaller than that for . whereas , percentages for and are close .+ * analysis of certificate non - boolean features . *[ non - boolean - analysis ] f9f11 are related to the certificate issuer : common name , organization name and country .f12 is the certificate subject s country .some of the common - names are popular in but not in .for example , `` equifax '' is the most popular common - name for ( 16% ) , whereas it is only in .( the same holds for the common - name `` utn '' ) .also , the percentage of certificates without common names ( `` justnone '' ) is larger in ( 16% ) than in ( 4% ) .we make similar observations for organization name .for example , `` verisign '' represents 10% of and 2.5% in . also , `` someorganization '' accounts for 11% in and 7% in .such observations alone are not enough to discriminate phishing from popular / legitimate domains , but they can be combined with others for more effective discrimination .it is also interesting to see that a non - trivial percentage of phishing domains still obtain certificates from legitimate cas .the usa ( us ) and south africa ( za ) are responsible for around 50% and 20% ( respectively ) of the certificates for both legitimate / popular and malicious domains .all other countries account only for 30% .this is expected since several major cas are located us and za .the granularity of countries is too coarse to derive useful information and the field can be easily filled with bogus values .+ 0  features f13 to f15 have integer / real values corresponding to : length of certificate validity period ( in days ) , length of serial number , and jaccard distance between the hostname and the common name of the subject .results of the analysis are shown using cumulative distribution function ( cdf ) plots in figures [ fig : validity - serial - cdf ] , and [ fig : name - cdf ] .figure [ fig : alexa - phish - com - net - cert - year - days - cdf ] compares the validity period cdf for and legitimate sets .it shows similarities among the distributions : in , 32% are valid for 365 days and 10% for 730 , as opposed to 32% for days and 17% for in .has a gradual increase in validity periods over 731 , whereas has a small jump around ( 3% ) .the longest validity period is days in and in .figure [ fig : alexa - typo - com - net - cert - year - days - cdf ] compares with the legitimate sets .the figure clearly shows that typosquatting certificates are valid for smaller number of days than those in legitimate sets .this difference in distributions is larger than that of .a genuine ssl certificate issued by a reputable ca is more likely to contain a long serial number , which assures uniqueness .the cdf for serial number length for all data sets in figure [ fig : alexa - phish - com - net - cert - serial - length - cdf ] shows that certificates tend to have shorter serial number length than legitimate certificates , especially , those in . around 40% in have a serial number of length 39 ( 18.5% of 13 , 11% of 15 , 11% , 10% of 9 , 10% of 23 and 6% of 11 ) .the distribution is different for , where 29% have length of 39 ( 22% of 13 , 16% of 9 , 11% of 23 , 10% of 11 and 7% of 15 ) .figure [ fig : alexa - typo - com - net - cert - serial - length - cdf ] shows that typosquatting certificates tend to have shorter serial number than certificates in legitimate sets .also , the difference between typosquatting and legitimate sets distributions is larger than that between and legitimate sets .we expect the common name in the subject field to be very similar to the hostname in legitimate certificates .we intuitively expect that the difference between common name and hostname in malicious domain certificates is greater than that in legitimate certificates , since malicious domains may not use complying ssl certificates .this feature captures the jaccard distance between hostname and common - name .the jaccard distance measures the closeness between two strings ( a and b ) and is defined as j(a , b ) = ( a and b)/(a or b ) .j(a , b ) is basically the size of the intersection between set a and b divided by the size of their union .thus , the larger the jaccard distance between two strings , the more similar they are .figure [ fig : alexa - phish - com - net - cert - jdist - cdf ] compares the cdf of jaccard distance of  and other legitimate sets .as expected , phishing certificates tend to have smaller jaccard distance than those in .30% in have a jaccard distance of 0 ( 5% of 1 , 4% of 0.7 and several 3 - 4% jumps from 0.6 until 0.9 ) . for43% have a jaccard distance of 0 ( 4% a distance of 1 and the main jumps exist between 0 and 0.5 as opposed to 0.6 to 0.9 in ) .the distribution of ( ) shows a trend different from .we observe that ( ) certificates have smaller jaccard distance than .for example ,90% ( 93% ) of certificates in ( ) have jaccard distance . whereas , in have a jaccard distance of . this might be due to the fact that domains in and sets are random domains and mostly unpopular ( the intersections with alexa s top 100k domains are negligible ) .thus , having a complying ssl certificate is not essential to these domains .figure [ fig : alexa - typo - com - net - cert - jdist - cdf ] compares the jaccard distance cdf of typosquatting with legitimate sets .it clearly shows that typosquatting certificates tend to have smaller jaccard distance .* summary of certificate feature analysis . *the most important observations from our analysis are :  1 .distributions of malicious sets are significantly different from those of legitimate sets for several features .2 .  around 20% of legitimate popular domains are still using the signature algorithm md5withrsaencryption despite its clear insecurity .3 .  a significant percentage ( ) of legitimate domain certificates are expired and/or self - signed .several user studies ( e.g. and ) show that an overwhelming percentage ( up to 70 - 80% in some cases ) of users ignore browser warning messages .this may explain the unexpected high percentages of expired and self - signed certificates that we find in the results of all our data sets .users simply ignore those warnings and this gives no incentive for organizations to update and pay attention to their certificates .4 .  duplicate certificate percentages are very high in phishing domains , which points to the possibility that phishers use certificates only to display the familiar lock icon in the browser , hoping that users will not pay attention .5 .  for some features ,the difference in distributions between malicious and legitimate sets is small .however , these features turn out to be useful in building a classifier ( see section [ sec : classifier ] ) . 6 .  for most features, the difference in distributions between and malicious sets is larger than that between / and malicious sets .  0the analysis in section [ analysis - ssl ] showed that several features have distributions in legitimate data sets that are significantly different from those in malicious sets .even though some show significant differences , relying on a single feature to identify malicious domains will yield a high rate of false positives .one approach , is to simply use a set of heuristics .for example , if f1-f8 are all true , the given domain is malicious with overwhelming probability .however , this might necessitate performing an exhaustive search over many combinations of heuristics , in order to find the optimal one .another possibility is to take advantage of machine - learning algorithms and let them find the best combination of features .machine learning algorithms are well suited for classification problems .we use several machine - learning - based classification algorithms and select the best performer . specifically , we consider the following algorithms : random forest , support vector machines , decision tree , decision table , nearest neighbor and neural networks . in addition , we explore two optimization techniques for decision trees : bagging and boosting .algorithm descriptions are out of the scope ; we refer to for relevant background information .we use precision - recall performance metrics to summarize the performance of a classifier .these metrics consists of the following ratios : positive recall , positive precision , negative recall , and negative precision .we use the term `` positive '' set to denote malicious domains ( phishing or typosquatting ) and `` negative '' set to refer to legitimate domains .the positive ( negative ) recall value is the fraction of positive ( negative ) domains that are correctly identified by the classifier .positive ( negative ) precision is the actual fraction of positive ( negative ) domains in the set identified by the classifier .we evaluate the performance of the classifier using the ten - fold cross validation method . in it, we randomly divide the data set into 10 sets of equal size , perform 10 different training / testing steps where each step consists of training a classifier on 9 sets and then testing it on the remaining set .we take the average of all results as the final performance result of the classifier .we use most of the features discussed in section [ analysis - ssl ] to build two binary classifiers : ( 1 ) a phishing classifier and ( 2 ) a typosquatting classifier .the phishing / typosquatting classifier distinguishes non - phishing / non - typosquatting domains from phishing / typosquatting domains using _ only _ information from ssl certificates .we tried different combinations of features in the classifier and the following yielded the best results : ( f1 ) md5 , ( f2 ) bogus subject , ( f3 ) self - signed ,( f4 ) expired , ( f6 ) common certificate , ( f7 ) common serial number , ( f8 ) validity period ( f9 ) issuer common name , ( f10 ) issuer organization name , ( f11 ) issuer country , ( f12 ) subject country ,( f14 ) serial number length and ( f15 ) host - common name distance ..performance of classifiers - data set consists of : ( a)420 phishing domain certificates and ( b)420 popular domain certificates ( ) [ cols="^,^,^,^,^ " , ]based on the measurements presented in the previous sections , we find that a significant percentage of well - known domains _ already _ use https ( alongside http ) , thus it is possible to harvest their certificates for classifying them , without requiring any modifications from the domains side .furthermore , the non - trivial portion of phishing web sites using https highlights the need to analyze and correlate the information provided in their server certificates .  * limitations . *we acknowledge that our data sets should be extended through longer observation periods .furthermore , additional data sets of legitimate domains need to be taken into consideration , e.g. popular web sites from dns logs in different organizations and countries .data sets of typosquatting domains can be strengthened by additional and more effective name variations . also , we acknowledge that our phishing(typosquatting ) classifier may incur false positives when actually deployed. however , this is a common problem to many machine - learning - based mitigation solutions ( spam filtering and intrusion detections based on machine - learning algorithms ) and the number of false positives can be minimized by training the classifier on larger and more comprehensive data sets ( and continuing to do so for new samples ) .our classifier does not provide a complete standalone solution to the phishing(typosquatting ) threat since many domains do not have https .instead , integrated with pre - existing solutions ( e.g. ,  ) , it improves their effectiveness in the context of potentially critical applications enforcing https and it contributes to face a complete range of malicious domains .  finally , we note that our classifier does not identify all kinds of typosquatting domains .for instance , typo - domains displaying unexpected pages and ads might have no incentive to use https , since they do not attempt to elicit private information from users .however , in the case of a further increase of https usage , our technique will be as a consequence increasing its effectiveness .generic typosquatting detection remains at the moment an interesting open issue that has not found effective solutions so far .* using information in certificates . * our results show significant differences between certificates of legitimate domains and those of malicious domains . not only is this information alone sufficient to detect fraudulent activities as we have shown , but it is also a useful component in assessing a web site s degree of trustworthiness , thus improving prior metrics , such as  .our method should be integrated with other techniques to improve the effectiveness of detecting malicious domains .  * keeping state of encountered certificates .* we deliberately chose to conduct our measurements as general as possible , without relying on user navigation history or on user specific training data .these components are fundamental for most current mitigation techniques  .moreover , we believe that keeping track of navigation history is detrimental to user privacy. however , our work yields effective detection by analyzing certain coarse - grained information extracted from server certificates and not specific to a user s navigation patterns .this is not privacy compromising as keeping fine - grained navigation history .  0 for example ,suppose that a user misspells the domain name of her bank and ends up in a passive phishing web site .it is quite likely that , in the phishing site s certificate , several fields would be encountered for the first time . according to our measurements ,this could be interpreted as an indicator of fraud . as future work, we plan to consider an enhanced version of our classifier assuming that it is possible to keep track of such privacy - insensitive information .* how malicious domains will adapt .* web fraud detection is an arms race and web - fraudsters are diligent and quickly adapt to new security mechanisms and studies that threaten their business .we hope that this work will raise the bar and make it more difficult for web - fraudsters to deceive users .if web - browsers use our classifier and start analyzing ssl certificate fields in more detail , we expect there will be two scenarios for web - fraudsters to adapt : ( 1 ) to get legitimate certificates from cas and leave a paper trail pointing to `` some person or organization '' which is connected to such an activity , ( 2 ) to craft certificates that have similar values like those that are the most common in those of legitimate domains .some fields / features will be easy to forge with legitimate values ( e.g. , country of issuer , country of subject , subject common and organization name , validity period , signature algorithm , serial number ... etc ) but for some this will not be possible ( issuer name , signature ... etc ) because otherwise the verification of the certificate will fail . in either casethe effectiveness of web - fraud will be reduced .the authors in conducted a measurement study to measure the cryptographic strength of more than 19,000 public servers running ssl / tls .the study reported that many ssl sites still supported the weak ssl 2.0 protocol . also , most of the probed servers supported des , which is vulnerable to exhaustive search . some of the sites used rsa - based authentication with only 512-bit key size , which is insufficient .nevertheless , the authors showed encouraging measurement results such as the use of aes as default option for most of the servers that did not support aes .also , a trend toward using stronger cryptographic function has been observed over two years of probing , despite a slow improvement . in ,the authors performed a six - month measurement study on the aftermath of the discovered vulnerability in openssl in may 2008 , in order to measure how fast the hosts recovered from the bug and changed their weak keys into strong ones . through the probing of thousands of servers, they showed that the replacement of the weak keys was slow .also , the speed of recovery was shown to be correlated to different ssl certificate characteristics such as : ca type , expiry time , and key size .the article in presented a profiling study of a set of ssl certificates .probing around 9,754 ssl servers and collecting 8,081 ssl certificates , it found that around 30% of responding servers had weak security ( small key size , supporting only ssl 2.0 , ... ) , 10% of them were already expired and 3% were self - signed .netcraft conducts a monthly survey to measure the certificate validity of internet servers .recently , the study showed that 25% of the sites had their certificates self - signed and less than half had their certificates signed by valid ca .symantec has observed an increase in the number of urls abusing ssl certificates . only in the period between may and june 2009 ,26% of all the ssl certificate attacks have been performed by fraudulent domains using ssl certificates .although our measurement study conducts a profiling of ssl certificates , our purpose is different from the ones above .we analyze the certificates to show how malicious certificates are different from benign ones and to leverage this difference in designing a mitigation technique .the importance and danger of web - fraud ( such as phishing and typosquatting ) has been recognized in numerous prior publications , studies and industry reports mainly due to the tremendous financial losses that it causes .one notable study is which analyzed the infrastructure used for hosting and supporting internet scams , including phishing .it used an opportunistic measurement technique that mined email messages in real - time , followed the embedded link structure , and automatically clustered destination web sites using image shingling . in ,a machine learning based methodology was proposed for detecting phishing email .the methodology was based on a classifier that detected phishing with 96% accuracy and false negative rate of 0.1% .our work differs since it does not rely on phishing emails which are sometimes hard to identify . an anti - phishing browser extension ( antiphish ) was given in .it kept track of sensitive information and warned the user whenever the user tried to enter sensitive information into untrusted web sites .our classifier can be easily integrated with antiphish . however , antiphish compromised user privacy by keeping state of sensitive data .other anti - phishing proposals relied on trusted devices , such as a user s cell phone in . in , the authors tackled the problem of detecting malicious web sites by only analyzing their urls using machine - learning statistical techniques on the lexical and host - based features of their urls .the proposed solution achieved a prediction accuracy around 95% .other studies measured the extent of typosquatting and suggested mitigation techniques .wang , ed al . showed that many typosquatting domains were active and parked with a few parking services , which served ads on them .similarly , showed that , for nearly 57% of original urls considered , over 35% of all possible url variations existed on the internet .surprisingly , over 99% of such similarly - named web sites were considered phony . devised a methodology for identifying ads - portals and parked domains and found out that around 25% of ( two - level ) .com and .net domains were ads - portals and around 40% of those were typosquatting .mcafee also studied the typosquatting problem in  .a set of 1.9 million single - error typos was generated and suspected typosquatting domains were discovered . alarmingly , the study also found that typosquatters targeted children - oriented domains .finally , mcafee added to its extension site advisor  some capabilities for identifying typosquatters .user studies analyzing the effectiveness of browser warning messages indicated that an overwhelming percentage ( up to 70 - 80% in some cases ) of users ignored them . this observation confirmed by recent research results( e.g. , and ) might explain the unexpected high percentage of expired and self signed certificates that we found in the results of all our data sets .in , the authors proposed a solution to simplify the process ( for the users ) of authenticating the servers public keys ( or ssl certificates ) by deploying a set of semi - trusted collection of network servers that continuously probed the servers and collected their public keys ( or ssl certificates ) . when the user was exposed to a new public key ( ssl certificate ), he referred to these semi - trusted servers to verify the authenticity of public keys ( ssl certificates ) .we conclude that , to the best of our knowledge , no prior work has explored the web - fraud problem in the context of https and proposed analyzing server - side ssl certificates in more detail .our work yields a detailed analysis of ssl certificates from different domain families and a classifier that detects web - fraud domains based on their certificates .in this paper , we study the prevalence of https usage in popular and legitimate domains as well as in the context of web - fraud , i.e. , phishing and typosquatting . to the best of our knowledge ,this is the first effort to analyze information in ssl certificates to profile domains and assess their degree of trustworthiness .we design and build a machine - learning - based classifier that identifies fraudulent domains using https based solely on their ssl certificates , thus also preserving user privacy .our work can be integrated with pre - exisiting detection techniques to improve their effectiveness in the context of potentially critical applications enforcing https .finally , we believe that our results may serve as a motivating factor to increase the use of https on the web , showing that besides its intended benefits of confidentiality and authenticity , the use of https can help identifying web - fraud domains .

Abstract: web - fraud is one of the most unpleasant features of today s internet . two eminent examples of web - fraudulent activities are phishing and typosquatting . their effects range from relatively benign ( such as unwanted or unexpected ads ) to downright sinister ( especially , when typosquatting is combined with phishing ) . this paper presents a novel technique to detect web - fraud domains that utilize https . to achieve this , we conduct the first comprehensive study of ssl certificates for legitimate and popular domains , as opposed to those used for web - fraud . drawing from extensive measurements , we build a classifier that detects malicious domains with high accuracy . we validate our methodology with different data sets collected from the internet . our prototype is orthogonal to existing mitigation techniques and can be integrated with other available solutions . our work shows that , besides its intended benefits of confidentiality and authenticity , the use of https can help mitigate web - fraud .


bepicolombo is an european space agency mission to be launched in 2014 , with the goal of an in - depth exploration of the planet mercury ; it has been identified as one of the most challenging long - term planetary projects .only two nasa missions had mercury as target in the past , the mariner 10 , which flew by three times in 1974 - 5 and messenger , which carried out its flybys on january and october 2008 , september 2009 before it starts its year - long orbiter phase in march 2011 .the bepicolombo mission is composed by two spacecraft to be put in orbit around mercury .the radioscience experiment is one of the on board experiments , which would coordinate a gravimetry , a rotation and a relativity experiment , using a very accurate range and range rate tracking .these measurements will be performed by a full 5-way link ( ) to the mercury orbiter ; by exploiting the frequency dependence of the refraction index , the differences between the doppler measurements ( done in ka and x band ) and the delay give information on the plasma content along the radiowave path . in this way most of the measurements errors introduced can be removed , improving of about two orders of magnitude with respect to the past technologies .the accuracies that can be achieved are cm in range and cm / s in range rate .how we compute these observables ?for example , a first approximation of the range could be given by the formula which models a very simple geometrical situation ( see figure [ fig : range ] ) .the vector is the mercurycentric position of the orbiter , the vector is the position of the center of mass of mercury ( m ) in a reference system with origin at the solar system barycenter ( ssb ) , the vector is the position of the earth - moon center of mass in the same reference system , is the vector from the earth - moon barycenter ( emb ) to the center of mass of the earth ( e ) , the vector is the position of the reference point of the ground antenna with respect to the center of mass of the earth .    using ( [ eq : new ] ) means to model the space as flat arena ( is an euclidean distance ) and the time as absolute parameter .this is obviously not possible because it is clear that , beyond some threshold of accuracy , these quantities have to be formulated within the framework of einstein s theory of gravity ( general relativity theory , grt ) .moreover we have to take into account the different times at which the events have to be computed : the transmission of the signal at the transmit time ( ) , the signal at the mercury orbiter at the time of bounce ( ) and the reception of the signal at the receive time ( ) .  formula ( [ eq : new ] ) could be a good starting point to construct a correct relativistic formulation ; with the word `` correct '' we do not mean all the possible relativistic effects , but the effects that can be measured by the experiment .this paper deals with the corrections to apply to this formula to obtain a consistent relativistic model for the computations of the observables and the practical implementation of such computations .  in section [ sec : refsys ] we discuss the relativistic four - dimensional reference systems used and the transformations adopted to make the sums in ( [ eq : new ] ) consistent ; according to , with `` reference system '' we mean a purely mathematical construction , while a `` reference frame '' is a some physical realization of a reference system .the relativistic contribution to the time delay due to the sun s gravitational field , the shapiro effect , is described in section [ sec : shap ] .section [ sec : lt ] deals with the theoretical procedure to compute the light - time ( range ) and the doppler shift ( range rate ) . in section[ sec : num ] we discuss the practical implementation of the algorithms showing how we eliminate rounding - off problems .the equations of motion for the planets mercury and earth , including all the relativistic effects ( and potential violations of grt ) required to the accuracy of the bepicolombo radioscience experiment have already been discussed in , thus this paper concentrates on the computation of the observables .the five vectors involved in formula ( [ eq : new ] ) have to be computed at their own time , the epoch of different events : e.g. , , and are computed at both the antenna _ transmit time _ and the _ receive time _ of the signal . and are computed at the _ bounce time _ ( when the signal has arrived to the orbiter and is sent back , with correction for the delay of the transponder ) . to be able to perform the vector sums and differences , these vectors have to be converted to a common space - time reference system , the only possible choice being some realization of the bcrs ( barycentric celestial referece system ) .we adopt for now a realization of the bcrs that we call ssb ( solar system barycentric ) reference frame and in which the time is tdb ( barycentric dynamic time ) ; other possible choices , such as a tcb ( barycentric celestial time ) , only can differ by linear scaling .the tdb choice of the ssb timescale entails also the appropriate linear scaling of space - coordinates and planetary masses as described for instance in or .the vectors , , and are already in ssb as provided by numerical integration and external ephemerides ; thus the vectors and have to be converted to ssb from the geocentric and mercurycentric systems , respectively .of course the conversion of reference system implies also the conversion of the time coordinate .there are three different time coordinates to be considered .the currently published planetary ephemerides are provided in tdb .the observations are based on averages of clock and frequency measurements on the earth surface : this defines another time coordinate called tt ( terrestrial time ) .thus for each observation the times of transmission and receiving need to be converted from tt to tdb to find the corresponding positions of the planets , e.g. , the earth and the moon , by combining information from the precomputed ephemerides and the output of the numerical integration for mercury and the earth - moon barycenter .this time conversion step is necessary for the accurate processing of each set of interplanetary tracking data ; the main term in the difference tt - tdb is periodic , with period 1 year and amplitude s , while there is essentially no linear trend , as a result of a suitable definition of the tdb .the equation of motion of a mercurycentric orbiter can be approximated , to the required level of accuracy , by a newtonian equation provided the independent variable is the proper time of mercury .thus , for the bepicolombo radioscience experiment , it is necessary to define a new time coordinate tdm ( mercury dynamic time ) , as described in , containing terms of 1-pn order depending mostly upon the distance from the sun and velocity of mercury .  from now on we shall call the quantities related to the ssb frame `` tdb - compatible '' , the quantities related to the geocentric frame `` tt - compatible '' , and the quantities related to the mercurycentric frame `` tdm - compatible '' , in accordance with the paper , and label them tb , tt and tm , respectively .the differential equation giving the local time as a function of the ssb time , which we are currently assuming to be tdb , is the following : \,\ , , \label{eq : dert}\ ] ] where is the gravitational potential ( the list of contributing bodies depends upon the accuracy required : in our implementation we use sun , mercury to neptune , moon ) at the planet center and is the ssb velocity of the same planet .the constant term is used to perform the conventional rescaling motivated by removal of secular terms , e.g. , for the earth we use .the space - time transformations we have to perform involve essentially the position of the antenna and the position of the orbiter .the geocentric coordinates of the antenna should be transformed into tdb - compatible coordinates ; the transformation is expressed by the formula where is the gravitational potential at the geocenter ( excluding the earth mass ) , is a scaling factor given as definition , supposed to be a good approximation for removing secular terms from the transformation and is the barycentric velocity of the earth .the next formula contains the effect on the velocities of the time coordinate change , which should be consistently used together with the coordinate change : \cdot \left[{\frac{d t}{d t}}\right]\,\,.\ ] ] note that the previous formula contains the factor ( expressed by eq .( [ eq : dert ] ) ) that deals with time transformation : is the local time for earth , that is tt , and is the corresponding tdb time .the mercurycentric coordinates of the orbiter should be transformed into tdb - compatible coordinates through the formula where is the gravitational potential at the center of mass of mercury ( excluding the mercury mass ) and could be used to remove the secular term in the time transformation ( thus defining a tm scale , implying a rescaling of the mass of mercury ) .we believe this is not necessary : the secular drift of tdm with respect to other time scales is significant , see figure 5 in paper , but a simple iterative scheme is very efficient in providing the inverse time transformation .thus we set , assuming the reference frame is tdm - compatible . as for the antennawe have a formula expressing the velocity transformation that contains the derivative of time for mercury , that is tdm , with respect to time , that is tdb : \cdot  \left[{\frac{d t}{d t } } \right]\,\ , .\ ] ]  in all the formulas for these coordinate changes we have neglected the terms of the ssb acceleration of the planet center ( ) , because they contain beside the additional small parameter ( distance from planet center)/(planet distance to the sun ) , which is of the order of even for a mercury orbiter .[ cols="^ " , ]   the shapiro correction for the computation of the range rate is \,\,,\ ] ] where the time derivative is with respect to a tdb time .this formula is almost never found in the literature and has not been much used in the processing of the past radioscience experiments , such as in , because the observable range rate is typically computed as difference of ranges divided by time ; however , for reasons explained in section [ sec : num ] , this formula is now necessary .since radar measurements are usually referred to the receive time the observables are seen as functions of this time , and the computation sequence works backward in time : starting from , the bounce time is computed iteratively , and , using this information the transmit time is computed .  the vectors and are obtained integrating the post - newtonian equations of motions .the vectors are obtained by integrating the orbit in the mercurycentric tdm - compatible frame .the vector is obtained from a standard iers model of earth rotation , given accurate station coordinates , and from lunar ephemerides ( ) .  in the following subsectionswe shall describe the procedure to compute the range ( section [ sec : range ] ) and the range rate ( section [ sec : ranger ] ) .once the five vectors are available at the appropriate times and in a consistent ssb system , there are two different light - times , the up - leg for the signal from the antenna to the orbiter , and the down - leg for the return signal .they are defined implicitly by the distances down - leg and up - leg  respectively , with somewhat different shapiro effects .then and are the two portions of the light - time , in the time attached to the ssb , that is tdb ; this provides the computation of . then these times are to be converted back in the time system applicable at the receiving station , where the time measurement is performed , which is tt ( or some other form of local time , such as the standard utc ) . is already available in the local time scale , from the original measurement , while needs to be converted back from tdb to tt .the difference between these two tt times is , from which we can conventionally define .note that the difference in tt is significantly different from in tdb , by an amount of the order of s , while the sensitivity of the bepicolombo radioscience experiment is of the order of s , thus these conversions change the computed observable in a significant way , see figure [ fig : difft ] .    the practical method for solving and in eqs . ( [ eq : rdo ] ) and ( [ eq : rup ] ) is as follows .since the measurement is labeled with the receive time , the iterative procedure needs to start from eq .( [ eq : rdo ] ) by computing the states and at epoch , then selecting a rough guess for the bounce time ( e.g. , ) .then the states and are computed at and a successive guess is given by ( [ eq : rdo ] ) .this is repeated computing , and so on until convergence , that is , until is smaller than the required accuracy .this fixed point iteration to solve the implicit equation for is convergent because the motion of the satellite and of mercury , in the time , is a small fraction of the total difference vector . after accepting the last value of start with the states and at and with a rough guess for the transmit time ( e.g. , ) .then and are computed at epoch and is given by eq .( [ eq : rup ] ) , and the same procedure is iterated to convergence , that is to achieve a small enough .this double iterative procedure to compute range is consistent with what has been used for a long time in planetary radar , as described in the paper .we conventionally define .   after the two iterations providing at convergence and are complete , we can proceed to compute the range rate .we rewrite the expression for the euclidean range ( down - leg and up - leg ) as a scalar product : \, \cdot \ , \left [ { \bf x}_{\rm ms}(t_b ) -{\bf x}_{\rm   ea}(t_r ) \right ] \,\ , , \\r^2_{up}(t_r ) & = & \left [ { \bf x}_{\rm   ms}(t_b)-{\bf x}_{\rm ea}(t_t)\right ] \cdot \left [ { \bf x}_{\rm   ms}(t_b ) -{\bf x}_{\rm ea}(t_t)\right ] \,\ , , \end{aligned}\ ] ] where and .the light - time equation contains also the shapiro terms , thus the range rate observable contains also additive terms and , with significant effects ( a few cm / s during superior conjunctions ) .since the equations giving and are still ( [ eq : rdo ] ) and ( [ eq : rup ] ) , in computing the time derivatives , we need to take into account that and , with non - unit derivatives .computing the derivative with respect to the receive time , and using the dot notation to stand for , we obtain : \,\ , , \\\dot r_{up}(t_r ) & = & \hat { \bf r}_{up}\,\bigg [ { \bf \dot x}_{\rm  ms}(t_b)\ , \left ( 1-\frac{\dot r_{do}(t_r)+\dot s_{do}}{c } \right ) -  \nonumber\end{aligned}\ ] ] \,\ , , \ ] ] where   however , the contribution of the time derivatives of the shapiro effect to the and corrective factors is small , of the order of , which is marginally significant for the bepicolombo radioscience experiment .we conventionally define .these equations are compatible with , taking into account that they use a single iteration .since the time derivatives of the shapiro effects contain and , the equations ( [ eq : rdodot ] ) and ( [ eq : rupdot ] ) are implicit , thus we can again use a fixed point iteration .it is also possible to use a very good approximation which solves explicitly for and then for , neglecting the very small contribution of shapiro terms : \;\left[1 + \frac{{\bf \dot x}_{\rm ms}(t_b)\cdot \hat{\bf r}_{do}}{c } \right]^{-1 } \,\ , , \ ] ] where the right hand side is weakly dependent upon only through , thus a moderately accurate approximation could be used in the computation of , followed by a single iteration . for the other leg \\  & & \quad \quad \ , \left [  1-\frac{{\bf \dot x}_{\rm ea}(t_t ) \cdot\hat{\bf r}_{up}}{c }  \right]^{-1 } \,\ , .\end{aligned}\ ] ]  all the above computations are in ssb with tdb ; however , the frequency measurements , at both and , are done on earth , that is with a time which is tt .this introduces a change in the measured frequencies at both ends , and because this change is not the same ( the earth having moved by about of its orbit ) there is a correction needed to be performed .the quantity we are measuring is essentially the derivative of with respect to , but this in two different time systems : for readability , we use for tt , for tdb where the derivatives of the time coordinate changes are the same as the right hand sides of the differential equation giving as a function of in the first factor and the inverse of the same for the last factor . however , the accuracy required is such that the main term with the gravitational mass of the sun and the position of the sun is enough :   \;{\frac{d t_t}{d t_r } } \nonumber \\  & & \left[1-\frac{\mu_0}{|{\bf x}_{\rme}(t_r)-{\bf x}_0(t_r)|\,c^2}-  \frac{|{\bf \dot x}_{\rm e}(t_r)|^2}{2\,c^2}\right]^{-1}\ .\label{eq : dertimeconv }  \end{aligned}\ ] ] note that we do not need the constant term discussed above because it cancels in the first and last terms in the right hand sides of eq .( [ eq : dertimeconv ] ) .the correction in the above formula is required for consistency , but in fact the correction has an order of magnitude of cm / s and is negligible for the sensitivity of the bepicolombo radioscience experiment ( figure [ fig : difft ] ) .the computation of the observables , as presented in the previous section , is already complex , but still the list of subtle technicalities is not complete .a problem well known in radioscience is that , for top accuracy , the range rate measurement can not be the instantaneous value .in fact , the measurement is not instantaneous : an accurate measure of a doppler effect requires to fit the difference of phase between carrier waves , the one generated at the station and the one returned from space , accumulated over some _ integration time _ , typically between and s. thus the observable is really a difference of ranges or , equivalently , an averaged value of range rate over the integration interval   in order to understand the computational difficulty we need to take also into account the orders of magnitude . as said in the introduction , for state of the art tracking systems , such as those using a multi - frequency link in the x and ka bands , the accuracy of the range measurements can be cm and the one of range - rate cm / s ( over an integration time of s ) .let us take an integration time s , which is adequate for measuring the gravity field of mercury ; in fact if the orbital period is s , the harmonics of order have periods as short as s. the accuracy over s of the range rate measurement can be , by gaussian statistics , cm / s , and the required accuracy in the computation of the difference is cm .the distances can be as large as cm , thus the relative accuracy in the difference needs to be .this implies that _ rounding off _ is a problem with current computers , with relative rounding off error of ( figure [ fig : rr3 ] ) ; extended precision is supported in software , but it has many limitations .the practical consequences are that the computer program processing the tracking observables , at this level of precision and over interplanetary distances , needs to be a mixture of ordinary and extended precision variables .any imperfection may result in `` banding '' , that is residuals showing a discrete set of values , implying that some information corresponding to the real accuracy of the measurements has been lost in the digital processing .   of the harmonic coefficient : the range rate computed as range difference divided by the integration time of 30 s , eq .( [ eq : rrate_mode3 ] ) , is obscured by the rounding off.,width=264 ]   of the harmonic coefficient : the range rate computed as an integral , eq . ( [ eq : rrate_mode4 ] ) , is smooth ; the difference is marginally significant with respect to the measurement accuracy.,width=264 ]  as an alternative , the use of a quadrature formula for the integral in eq .( [ eq : rrate_mode4 ] ) can provide a numerically more stable result , because the s / n of the range rate measurement is .figure [ fig : rr4 ] shows that a very small model change , generating a range rate signal micron / s over one pass , can be computed smoothly by using a 7 nodes gauss quadrature formula .by combining the results of the previous paper ( ) , and of this one , we have completed the task of showing that it is possible to build a consistent relativistic model of the dynamics and of the observations for a mercury orbiter tracked from the earth , at a level of accuracy and self - consistency compatible with the very demanding requirements of the bepicolombo radioscience experiment .in particular , in this paper we have given the algorithm definitions for the computation of the observables range and range rate , including the reference system effects and the shapiro effect .we have shown which computation can be performed explicitly and which ones need to be obtained from an iterative procedure .we have also shown how to push these computations , when implemented in a realistic computer with rounding - off , to the needed accuracy level , even without the cumbersome usage of quadruple precision .the list of `` relativistic corrections '' , assuming we can distinguish their effects separately , is long , and we have shown that many subtle effects are relevant to the required accuracy . however , in the end what is required is just to be fully consistent with a post - newtonian formulation to some order , to be adjusted when necessary .interestingly , the high accuracy of bepicolombo radio system may require implementation of the second post - newtonian effects in range .the results of the research presented in this paper , as well as in the previous one ( ) have been performed within the scope of the contract asi/2007/i/082/06/0 with the italian space agency .bepicolombo is a scientific space mission of the science directorate of the european space agency .the work of dv was partially supported by the czech grant agency ( grant 202/09/0772 ) and the research program msm0021620860 of the czech ministry of education .ashby , n. , bertotti , b. : second - order corrections to time delay and deflection of light passing near a massive object .american astronomical society , dda meeting # 39 ( 2008 ) bertotti , b. , iess , l. , tortora , p. : a test of general relativity using radio links with the cassini spacecraft .nature , 425 , 374 - 376 ( 2003 ) damour , t. , soffel , m. , hu , c. : general - relativistic celestial mechanics .theory of satellite motion .d , 49 , 618 - 635 ( 1994 ) iess , l. , boscagli , g. : advanced radio science instrumentation for the mission bepicolombo to mercury .sp . sci . , 49 , 1597 - 1608 ( 2001 ) klioner , s.a . , zschocke , s. : gaia - ca - tn - lo - sk-002 - 1 report ( 2007 ) klioner , s.a . : relativistic scaling of astronomical quantities and the system of astronomical units .478 , 951958 ( 2008 ) klioner , s.a . ,capitaine , n. , folkner , w. , guinot , b. , huang , t. y. , kopeikin , s. , petit , g. , pitjeva , e. , seidelmann , p. k. , soffel , m. : units of relativistic time scales and associated quantities . in : klioner , s. , seidelmann ,p.k . , soffel , m. ( eds . ) relativity in fundamental astronomy : dynamics , reference frames , and data analysis , cambridge university press , in press milani , a. , vokrouhlick , d. , villani , d. , bonanno , c. , rossi , a. : testing general relativity with the bepicolombo radio science experiment .d , 66 ( 2002 ) milani a. , gronchi g.f .: theory of orbit determination .cambridge university press ( 2009 ) milani , a. , tommei , g. , vokrouhlick , d. , latorre , e. , cical , s. : relativistic models for the bepicolombo radioscience experiment . in : klioner , s. , seidelmann , p.k . ,soffel , m. ( eds . ) relativity in fundamental astronomy : dynamics , reference frames , and data analysis , cambridge university press , in press moyer , t.d .: formulation for observed and computed values of deep space network data types for navigation .wiley - interscience ( 2003 ) shapiro , i.i . : fourth test of general relativity . phys . rev ., 13 , 789 - 791 ( 1964 ) soffel , m. , klioner , s.a . , petit , g. , kopeikin , s.m . ,bretagnon , p. , brumberg , v.a . ,capitaine , n. , damour , t. , fukushima , t. , guinot , b. , huang , t .- y . ,lindegren , l. , ma , c. , nordtvedt , k. , ries , j.c . ,seidelmann , p.k ., vokrouhlick , d. , will , c.m ., xu , c. : the iau 200 resolutions for astrometry , celestial mechanics , and metrology in the relativistic framework : explanatory supplement . astron .j. , 126 , 2687 - 2706 ( 2003 ) teyssandier , p. , le poncin - lafitte , c. : general post - minkowskian expansion of time transfer functions . class .quantum grav ., 25 , 145020 ( 2008 ) will , c.m . :theory and experiment in gravitational physics .cambridge university press ( 1993 ) yeomans , d. k. , chodas , p. w. , keesey , m. s. , ostro , s. j. , chandler , j. f. , shapiro , i. i. : asteroid and comet orbits using radar data .j. , 103 , 303 - 317 ( 1992 )

Abstract: the radioscience experiment is one of the on board experiment of the mercury esa mission bepicolombo that will be launched in 2014 . the goals of the experiment are to determine the gravity field of mercury and its rotation state , to determine the orbit of mercury , to constrain the possible theories of gravitation ( for example by determining the post - newtonian ( pn ) parameters ) , to provide the spacecraft position for geodesy experiments and to contribute to planetary ephemerides improvement . this is possible thanks to a new technology which allows to reach great accuracies in the observables range and range rate ; it is well known that a similar level of accuracy requires studying a suitable model taking into account numerous relativistic effects . in this paper we deal with the modelling of the space - time coordinate transformations needed for the light - time computations and the numerical methods adopted to avoid rounding - off errors in such computations .


the problem of recovering a signal from quadratic measurements is known as phase retrieval .a typical case with many applications , for instance in optics or crystallography , is when the measurements correspond to the ( squared ) magnitude of the fourier transform of the signal . here , we consider the more general setting of arbitrary quadratic measurements , , , while focusing on the case where the signal is assumed to be sparse , i.e. , with few nonzero entries in . as in compressive sensing , which deals with the recovery of sparse signals from linear measurements , the sparsity prior reduces the number of measurements required to recover the signal .[ [ related - work . ] ] related work .+ + + + + + + + + + + + +  seminal works on phase retrieval did not consider the sparsity prior .though these methods were able to incorporate prior information on the support , they were not designed to estimate the support with limited information on its size .more recently , matrix lifting techniques were developed in for phase retrieval and more particularly for the case of sparse signals in .the basic idea is to apply a change of variable resulting in linearized measurement equations with a rank-1 constraint on the new matrix variable .then , the rank-1 constraint is relaxed to the problem of minimizing the rank of the matrix , which is further relaxed to the minimization of the nuclear norm . in these methods ,the sparsity prior is typically incorporated as the minimization of an -norm , which induces a trade - off between the satisfaction of the rank-1 constraint and the sparsity of the solution .other methods focusing on sparsity are typically iterative , like fienup - type methods using alternate projections or the gespar method implementing a local search strategy with a bi - directional greedy algorithm .these iterative methods usually come without recovery guarantees .finally , note that sparse phase retrieval also enters the more general framework of nonlinear compressed sensing , as investigated in for analytic functions computing the measurements , in for quasi - linear functions and in for polynomials .[ [ contribution . ] ] contribution .+ + + + + + + + + + + + + +  we propose a convex approach to the sparse phase retrieval problem .this approach relies on two main steps : the linearization of the constraints inducing a group - sparse structure on the variables and a convex relaxation of the group - sparse optimization problem enforcing this structure .more precisely , the linearization is based on the veronese map lifting the signal to a higher dimensional space .this map is invariant to a global sign change and preserves sparsity in the sense that the lifted signal has a group - sparsity similar to the sparsity of the original signal .thus , the proposed method amounts to estimating a group - sparse signal satisfying linear constraints , from which the original signal can be recovered .this estimation relies on a convex relaxation of the group - sparse problem based on a sum of norms , or block -norm .thus , while methods based on matrix lifting lead to semi - definite programming problems , the proposed approach yields a more amenable second - order cone programming formulation . this formulation is also easily extended to deal with noisy measurements .in addition to these algorithmic benefits , we derive exact recovery conditions in the noiseless case and stable recovery guarantees in the presence of noise .note that the approach taken here is similar in spirit to the one derived in for the more general problem of finding sparse solutions of polynomial systems of equations .however , the analysis in is limited to the real case and does not apply to polynomials without linear terms as the ones found in phase retrieval .[ [ paper - organization . ] ] paper organization .+ + + + + + + + + + + + + + + + + + +  for the sake of clarity , we first detail in sect . [sec : real ] the proposed method in the real case before extending it in sect .[ sec : complex ] to the complex case .the effect of noise and stability results are discussed in sect .[ sec : noise ] .the case of real signals measured via complex vectors is considered in sect .[ sec : complexreal ] which also deals with the invariance of the measurements to circular shifts and reflections .finally , section [ sec : exp ] tests the proposed methods in numerical experiments .[ [ notations . ] ] notations .+ + + + + + + + + +  matrices are written with bold uppercase letters and vectors in bold lowercase letters , except for the column of a matrix .the notation denotes the element at the row and column of a matrix . and denote the real and imaginary parts of a complex number , vector or matrix , and the imaginary unit .the superscripts and denote the transpose and conjugate transpose , respectively , i.e. , . denotes the -norm in , while denotes the norm in induced by the inner product as .the -pseudo - norm of a real or complex vector of dimension is defined as and denotes the number of nonzero components .we also define the -pseudo - norm of a vector - valued sequence as .we write the sparse phase retrieval problem as where are the measurements and . due to symmetry , solutions to are defined up to their sign and the goal is to obtain an estimate , for in the solution set of . in particular , we are interested in the case where  has a unique pair of solutions , while conditions ensuring such a uniqueness are discussed in .the proposed method relies on two subsequent relaxations .while the first one linearizes the constraints , the second one convexifies the objective function .let the veronese map of degree 2 , , be defined by ^t , \ ] ] and the subscript denote the index of its component equal to , i.e. , this notation is symmetric , i.e. , and denote the same index , and will be used throughtout the paper to index the components of vectors of or .the constraints of the phase retrieval problem  can be rewritten as with and .let be an -binary matrix such that are given by , , , where is the kronecker delta and is an index as in . ] , i.e. , is the vector of entries corresponding to the monomials in including .then , we have and the objective function in  can be written as thus , can be reformulated as the nonlinear group - sparse optimization problem   note that , but that for , .thus , the problem can be posed as the one of recovering the value of , from which can be inferred up to its sign .  to estimate , we relax to wherethe variables in estimating the components of are not constrained to be interdepent monomials of base variables , but the last constraints in  nonetheless ensure that the s estimating the s are positive .problem  is a ( linear ) group - sparse optimization problem with highly overlapping groups . while groupwise - greedy algorithms , such as the one proposed in , can be applied , their analysis is not available for the case of overlapping groups. therefore , here , we consider the convex relaxation approach which aims at solving via the following surrogate formulation : where we introduced the diagonal matrix of precompensating weights , and which can be solved efficiently by off - the - shelf second - order cone programming ( socp ) solvers .then , we easily obtain an estimate of from the estimate of as , where the inverse mapping is defined as ^t},\quad  \mbox{if } i > 0 \mbox { and } \displaystyle{\frac{v_{ji}^2 } { v_{ii } } = v_{jj}},\ \forall j\in\{1,\dots , n\ } \\   { \boldsymbol{0}},\ \mbox{otherwise } ,   \end{cases}\ ] ] where in the above , the first nonzero entry ( of index ) is assumed to be positive to fix the signs and make injective .this definition also ensures that and that since , for , .we now turn to theoretical guarantees offered by the proposed approach .first , the following theorem provides the rationale for tackling the sparse phase retrieval problem via the group - sparse optimization formulation .[ thm : grouppoly ] if the solution to is unique and yields such that , , then is the unique pair of solutions of .assume there is an satisfying the constraints of and at least as sparse as .then , and , by using  and , which contradicts the fact that is the unique solution to unless .but since , we have for some , which implies .therefore , by using lemma [ lem : nudiffreal ] in appendix [ sec : lemmas ] with the assumption , there can not be such an .the following results regarding the convex formulation  are based on the notion of mutual coherence .[ def : mu ] the _ mutual coherence _ of a matrix \in{\mathbb{r}}^{n\times m} ] .next , we relax this formulation by substituting for , which yields where the last constraints ensure that the s estimating the modulus of the base variables are positive real numbers .the following theorem shows that this relaxation can be used as a proxy to solve the original problem .[ thm : grouppolycomplex ] if the solution to is unique and yields such that , , then is the unique set of solutions of .assume there is an satisfying the constraints of and at least as sparse as . then , and , by using  and lemma [ lem : sparsitynuinv ] , which contradicts the fact that is the unique solution to unless .but since , we have and thus for some , which implies . therefore , by using lemma [ lem : nudiff ] with the assumption , there can not be such an .we now introduce a second level of relaxation by replacing the -pseudo norm by a block- norm .this leads to a convex relaxation in the form of a socp : where and with a diagonal matrix of precompensating weights given by and .[ thm : groupsparsecomplex ] let be such that , , and  ] and ^t ] and ^t ] . in plain words , the transformation in  first shifts the vector such that the first component is the one with maximal magnitude .then , it applies a centered reflection to the shifted , named , only if the sum of squares over the first entries of ( without the first one ) is smaller than the one over the last ones .if this is the case , the result is shifted again to recover the first component of with maximal magnitude .finally , the veronese map is applied to to give .invariance of to circular shifts and reflections is implied by the invariance of given in the proposition below ( proof in appendix [ sec : proofprop ] ) .[ prop : varphi ] for all such that , the following statements hold for the transformation defined in :  1 . is idempotent , i.e. , ;   and , for all additionally satisfying with ,  1 . is shift - invariant , i.e. , ; 2 . is reflection - invariant , i.e. , .note that statements 2 and 3 imply that is invariant to any combination of shifts and reflections .proposition [ prop : varphi ] also shows the idempotence of , which of course does not transfer to directly but which is however very useful .indeed , this allows us to test if a candidate , supposed to be the result of applied to some vector , is consistent with the definition of as , which can be checked via simple inequalities .since these inequalities only involve the ( squared ) magnitude of the entries in the vector , they can also be easily embedded as linear constraints in to compute an estimate that is consistent with the transformation .this yields the convex program where estimates .the advantage of using is that it has a single solution independently of the number of shifted / reflected solutions to , in the sense of the next theorem .[ thm : uniquereflect ] if problem  has a unique set of shifted / reflected solutions , and if , then there is exactly one vector with in the feasible set of .since all are solutions to , they satisfy the first constraint in  as . by proposition [ prop : varphi ], is constant over the set and equal to one of the vectors from this set , say , for which we have in particular .then , and thus satisfies all the constraints of  and is a feasible point .now take an with . then , either or for some .this implies .since satisfies the constraints in , we have and thus , which shows that violates the second constraint and is not a feasible point of .  [[ illustrative - example . ]] illustrative example .+ + + + + + + + + + + + + + + + + + + + +  consider the vectors ( and are not related to the notations of ) ^t,\quad       { \boldsymbol{x}}_2 = [ 4 , 0 , 0 , 1 , 2 , 3 ] ^t,\quad       { \boldsymbol{x}}_3 = [ 0 , 0 , 4 , 3 , 2 , 1]^t,\ ] ] ^t,\quad         { \boldsymbol{x}}_5 = [ 4 , 3 , 2 , 1 , 0 , 0]^t,\ ] ] which are all obtained by shifts and reflections of the same vector .they all lead to the same . indeed, the transformation first applies the required shift and reflect operations to map to which has a first component with the largest magnitude and the largest half - sum over its first entries .then , computes the veronese map of .also note that all the are feasible with respect to , but only satisfies the additional constraints implementing shift and reflection invariance in .therefore , we can obtain as the unique solution to .when the measurements correspond to the squared magnitude of the fourier transform of the signal , i.e. , its ( squared ) power spectrum , there is another issue beside shift / reflection - invariances . even with the correct support , the linear system , where is the submatrix of with the corresponding columns , is under - determined .more precisely , for all and all , , which implies and .thus , even when limited to the support , has similar columns and is rank - defficient .therefore , in this case the proposed approach can not exactly recover and is used to estimate the support .indeed , knowing the correct support can be useful for other methods dedicated to the classical phase retrieval problem .though this case seems unfavorable , if the number of measurements satisfies , the autocorreleation of ( with zero padding ) , can be computed via the inverse fourier transform of and thus is available .this information can be included as linear constraints on the components of estimating the cross products to drive the solution towards a satisfactory one .in addition , following , this can be used to restrict the support of the solution as follows .first , note that with , the measurements are not invariant to circular shifts of but of with zero - padding .thus , we can not assume , .however , we can fix . then, for all base variable index , thus , the corresponding variables in can be set to zero , i.e. , .moreover , let be the minimum of the s satisfying the condition in , then , which can be favored by removing the corresponding term in the cost function .combining all the information on the solution in a convex program leads to the final formulation   a noise - tolerant version of  is obtained by replacing the constraint by .this modification also applies to  and .for the experiments , we consider two variants of the proposed approach : the convex method described in details in the previous sections and the greedy method for solving the group - sparse optimization problems  and .implementation details for these two methods are as described in  , with slight modifications to handle complex variables .in particular , the convex method uses the iterative reweighting of adapted to the group - sparse setting to enhance the sparsity of the solution .the greedy method starts with an empty support and , at each iteration , adds the group of variables in corresponding to the base variable that results in the best approximation of .  for complex signals , , we measure the relative error corresponding to the normalized distance between the estimate and the set , i.e. , .exact recovery is detected when this error is smaller than .[ [ exact - recovery - in - the - noiseless - case . ] ] exact recovery in the noiseless case .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  we estimate the probability of exact recovery at various sparsity levels , , in the complex case where and .for each sparsity level , the probability is estimated as the percentage of successful trials over a monte carlo experiment with 100 trials . in each trial , complex vectors are drawn from a zero - mean gaussian distribution of unit variance to measure a random signal with nonzero entries at random locations whose values are drawn from a zero - mean gaussian distribution of unit variance .results shown in fig . [ fig : exact ] indicate that the proposed approach can exactly recover sufficiently sparse signals with high probability .similar experiments are performed to evaluate the influence of the number of measurements on the probability of exact recovery .results reported in the right plot of fig . [ fig : exact ] show that , with a sparsity level of , the convex method requires more measurements for perfect recovery than the greedy strategy. however , measurements are already sufficient to exactly recover in all trials .  ,title="fig : " ] , title="fig : " ]  [ [ stable - recovery - in - the - noisy - case . ] ] stable recovery in the noisy case .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  we now consider the noisy case where , , and .over 100 trials , figure [ fig : noise ] reports the mean relative error and the rate of successful support recovery for and .these results show that sufficiently sparse signals can be accurately estimated from noisy quadratic measurements and that the probability of support recovery in this case follows a curve similar to the one of the probability of exact recovery in the noiseless case .this means that the methods are robust to noise regarding the recovery of the correct support ., title="fig : " ] , title="fig : " ]  [ [ support - estimation - from - the - power - spectrum . ] ] support estimation from the power spectrum .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  we now test if the convex method is robust to invariances , such as the ones discussed in sect .[ sec : fourier ] , when estimating the support .in particular , we start with a setting in which , and , , are the rows of the -point fourier matrix . in this case , the autocorrelation can not be computed and the convex formulation  is used to estimate the support .then , we perform similar experiments but with oversampling ( ) , thus allowing for the computation of the autocorrelation and the use of .results shown in fig . [ fig : fourier ] indicate that by using  we can recover the correct support for sufficiently sparse signals without oversampling , i.e. , with as many measurements as unknowns , while using more information extracted from the autocorrelation of the signal only slightly helps to recover larger supports .( plain line ) and when using  for ( dashed line ) versus the sparsity level .[ fig : fourier ] ]the paper proposed a new approach to phase retrieval of sparse signals .this approach is based on a group - sparse optimization formulation of the problem with linearized constraints .exact and stable recovery results were shown for a convex relaxation of this formulation both in the real and complex case .invariances to circular shifts and reflections that are common in phase retrieval problems were also discussed and a practical technique was given to prevent these from breaking the sparsity of the solution .future work will focus on deriving theoretical guarantees for the invariance issue .another direction of research concerns the analysis of greedy algorithms for the group - sparse optimization problem , which proved as valuable as the convex relaxations in experiments for measurements without invariance . how to deal with invariances in these methods will also be investigated .10  e. j cands .compressive sampling . in _proceedings oh the international congress of mathematicians : invited lectures _ ,pages 14331452 , 2006 .e. j. cands , y. c. eldar , t. strohmer , and v. voroninski .phase retrieval via matrix completion ., 6(1):199225 , 2013 .e. j. cands , t. strohmer , and v. voroninski .phaselift : exact and stable signal recovery from magnitude measurements via convex programming . , 66(8):12411274 , 2013 .e. j. cands , m. b. wakin , and s. p. boyd . ., 14(5):877905 , 2008 .  d. l. donoho .compressed sensing ., 52(4):12891306 , 2006 .d. l. donoho , m. elad , and v. n. temlyakov . stable recovery of sparse overcomplete representations in the presence of noise ., 52(1):618 , 2006 .m. ehler , m. fornasier , and j. sigl .quasi - linear compressed sensing .technical report , 2013 .http://www-m15.ma.tum.de/foswiki/pub/m15/allgemeines/publicationsen/greedy_21.pdf .j. fienup. phase retrieval algorithms : a comparison ., 21(15):27582769 , 1982 .r. gerchberg and w. saxton . a practical algorithm for the determination of phase from image and diffraction plane pictures . , 35:237246 , 1972 .r. gonsalves .phase retrieval from modulus data ., 66(9):961964 , 1976 .r. w. harrison .phase problem in crystallography ., 10(5):10461055 , 1993 .k. jaganathan , s. oymak , and b. hassibi .recovery of sparse 1-d signals from the magnitudes of their fourier transform . in _ieee international symposium on information theory ( isit ) _ , pages 14731477 , 2012 .d. kohler and l. mandel .source reconstruction from the modulus of the correlation function : a practical approach to the phase problem of optical coherence theory ., 63(2):126134 , 1973 .f. lauer and h. ohlsson . finding sparse solutions of systems of polynomial equations via group - sparsity optimization . ,arxiv:1311.5871 , 2013 .s. mukherjee and c. s. seelamantula .an iterative algorithm for phase retrieval with sparsity constraints : application to frequency domain optical coherence tomography . in _ieee international conference on acoustics , speech and signal processing ( icassp ) _ , pages 553556 , 2012 .h. ohlsson and y. c. eldar . on conditions for uniqueness in sparse phase retrieval . ,abs/1308.5447 , 2013 .h. ohlsson , a. y. yang , r. dong , and s. sastry .nonlinear basis pursuit . , 2013 .h. ohlsson , a. y. yang , r. dong , m. verhaegen , and s. sastry .quadratic basis pursuit . , 2013 .j. ranieri , a. chebira , y. m. lu , and m. vetterli .phase retrieval for sparse signals : uniqueness conditions ., abs/1308.3058 , 2013 .y. shechtman , a. beck , and y. c. eldar . : efficient phase retrieval of sparse signals . , 2013 .y. shechtman , y. c. eldar , a. szameit , and m. segev .sparsity based sub - wavelength imaging with partially incoherent light via quadratic compressed sensing . , 19(16):1480714822 , 2011 .i. waldspurger , a. daspremont , and s. mallat .phase recovery , maxcut and complex semidefinite programming . , 2012 .a. walther .the question of phase retrieval in optics ., 10(1):4149 , 1963 .[ lem : nudiffreal ] let and be defined as in section [ sec : real ] . for all such that and , if for some , then .assume that and let .then , , we have where the last equality follows from the definition of . therefore ,if for some , we obtain .[ lem : bounddelta2 ] let  ] .we rewrite the assumption as where ^t \in { \mathbb{r}}^{2m}$ ] .then , we can use lemma [ lem : bounddelta2 ] to bound the entries in as with a diagonal matrix of entries , and these inequalities lead to where   [ lem : deltacomplexreal ] for all such that , the inequality where is a diagonal matrix of entries .this is a direct consequence of lemma [ lem : bounddelta2 ] applied to and .the vector is the unique solution to if the inequality holds for all satisfying .the inequality above can be rewritten as where . by the triangle inequality , , this condition is met if or   by defining as the set of indexes corresponding to nonzero columns of , lemma [ lem : bounddelta2 ] yields due to the fact that , we also have which then leads to introducing this result in gives the condition finally , given that , this yields or , after rearranging the terms , which can be rewritten as in the statement of the theorem .the vector is the unique solution to if the inequality holds for all such that , which implies the constraint on .the inequality above can be rewritten as where . by the triangle inequality , with ,this condition is met if   by defining as the set of indexes corresponding to nonzero columns of , lemma [ lem : deltacomplexreal ] yields due to the fact that , we also have which then leads to introducing this result in gives the condition finally , given that , this yields , for , which can be rewritten as in the statement of the theorem .   to prove statement 1 , note that the operation is equivalent to ^t,\ ] ] where the reflection is centered on .thus , ^t\ ] ] and where the inequality holds whenever this operation is performed in  ( i.e. , if this is not the case , then ) .therefore , for all , we obtain a vector such that .since and , this implies that , i.e. , .       therefore , and applies a reflection to if and only if it does not apply one to ( the inequalities above are strict by assumption ) .thus , in the case is reflected by , we have and while in the case is not reflected by , we have and

Abstract: this paper deals with sparse phase retrieval , i.e. , the problem of estimating a vector from quadratic measurements under the assumption that few components are nonzero . in particular , we consider the problem of finding the sparsest vector consistent with the measurements and reformulate it as a group - sparse optimization problem with linear constraints . then , we analyze the convex relaxation of the latter based on the minimization of a block -norm and show various exact recovery and stability results in the real and complex cases . invariance to circular shifts and reflections are also discussed for real vectors measured via complex matrices .