Using Large-Scale Anomaly Detection on Code to Improve Kotlin Compiler

Abstract: In this work, we apply anomaly detection to source code and bytecode to facilitate the development of a programming language and its compiler. We define anomaly as a code fragment that is different from typical code written in a particular programming language. Identifying such code fragments is beneficial to both language developers and end users, since anomalies may indicate potential issues with the compiler or with runtime performance. Moreover, anomalies could correspond to problems in language design. For this study, we choose Kotlin as the target programming language. We outline and discuss approaches to obtaining vector representations of source code and bytecode and to the detection of anomalies across vectorized code snippets. The paper presents a method that aims to detect two types of anomalies: syntax tree anomalies and so-called compiler-induced anomalies that arise only in the compiled bytecode. We describe several experiments that employ different combinations of vectorization and anomaly detection techniques and discuss types of detected anomalies and their usefulness for language developers. We demonstrate that the extracted anomalies and the underlying extraction technique provide additional value for language development.

\section{Introduction}

Anomaly detection techniques~\cite{hodge2004survey} have been successfully applied to a variety of practical tasks in many areas.
These techniques help to detect cyberattacks~\cite{richardson2018anomaly, stolfo2005anomaly}, identify pathologies in medical images~\cite{baur2018deep}, and detect traces of fraudulent activities in financial data~\cite{Ahmed2016}.

In software engineering, anomaly detection is widely applied to finding bugs~\cite{S1}, security issues~\cite{feng2003anomaly}, architectural design flaws~\cite{Oizumi2015}, workflow errors~\cite{execution-anomaly-detection}, synchronization errors in concurrent programs~\cite{Taylor1980}, and other anomalous patterns in code or other software artifacts. Definitions of an anomaly vary from study to study and are imposed by the exact task in each case.

In this study, we propose a new area of application for anomaly detection --- finding issues in programming language compilers. We define \emph{code anomalies} as fragments of code that are not typical within the community or an ecosystem of a given programming language, or machine code that is uncharacteristic within the range of output produced by the compiler. Such code could be useful to both users and developers of the language: for example, anomalous, yet actually existing, code snippets can highlight flaws in the language design or indicate problems in the performance of programs, problems in code generation, compiler optimizations, type inference, or data flow analysis, which turns them into a valuable material for compiler tests.

The task of identifying code anomalies at a scale of a language ecosystem consists in the following steps:
    (1) retrieve large corpora of source code and compiled machine code that are representative of the whole range of conventional coding practices and compiler output, respectively;
    (2) transform source code and machine code into a vectorized form that is digestible by anomaly detection algorithms;
    (3) run anomaly detection across vectorized data;
    (4) process the output of anomaly detection algorithms to identify, interpret, and classify meaningful anomalies.

Established industry standard languages, such as Java and C++, are not the most feasible targets for detecting anomalies at the scale of a language ecosystem. While large amounts of open source code in these languages are publicly available, mainstream compilers are very well-tested and stable, which makes it hard to identify unknown compiler issues through anomaly detection.
At the same time, it is essential that the target language has a significant and diverse community of users.

Considering these arguments, we choose Kotlin~\cite{kotlin} and its ecosystem as a target for this study.
Since the language is relatively young, its compiler might still contain bugs and performance issues, which makes
anomaly detection more likely to provide actionable insights for the language development team.
At the same time, Kotlin is one of the most rapidly developing languages with an actively growing community and a significant ecosystem of diverse open source projects~\cite{kotlin-growth}. Finally, the development team of Kotlin is easy to reach via a public issue tracker~\cite{kotlin-yt}, which makes it easy to communicate potential findings.

The contribution of this paper is threefold:
\begin{itemize}
    \item A method for finding code anomalies for a chosen programming language that is based on vectorizing a large code corpus with code embedding techniques and then applying anomaly detection algorithms on this vector data.
    \item A set of tools implementing the proposed method as well as the dataset containing more than 4 million unique Kotlin functions collected from GitHub and the bytecode for a part of them (more than 41,000 compiled classes).
    \item The evaluation of the proposed method on the collected dataset, which resulted in discovering several dozens of code fragments that were considered useful by the Kotlin compiler team and that were included into the Kotlin compiler test infrastructure.

\end{itemize}

The rest of the paper is organized as follows. In Section~\ref{related-work}, we discuss existing studies and tools that apply the anomaly detection techniques to programs. Section~\ref{overview} provides an overview of methods and techniques that are potentially applicable to finding code anomalies. Section~\ref{approach} presents our proposed approach to anomaly detection, and describes the collected dataset as well as steps of the processing pipeline. Section~\ref{evaluation} outlines our approach to evaluation of viability of the proposed method, as well as significance and practical value of the detected anomalies. Section~\ref{threats} discusses possible threats to validity of our study. Section~\ref{discussion} presents the obtained results and describes the most interesting types of discovered anomalies. Section~\ref{conclusion} concludes the paper by summarizing our results and providing possible directions for future work.

\section{Related work}\label{related-work}

Several papers and tools exist that are aiming to search for anomalies in programs. All of them introduce their own definitions of code anomalies, so their goals, methods and results also differ.

The GrouMiner tool~\cite{S2} was developed to detect anomalous patterns in object interaction in Java programs. The approach is based on modeling object interaction with a directed acyclic graph, the nodes of which are constructor and methods calls as well as fields references, while the edges represent dependencies between them. The tool performs a static code analysis: the source code is parsed into an abstract syntax tree and an object usage graph is built. Graph-based anomaly detection methods are used to detect unusual method calls and other atypical areas of the control flow graph.

A somewhat similar idea is presented in~\cite{S3}: the authors propose mining usage models for all objects from source code as sequences of their method calls. If an abnormal usage pattern emerges in some code fragment, it is treated as an anomaly and a defect candidate. This approach is also based on static code analysis and employs graph-based anomaly detection techniques.

Undoubtedly, object interaction anomalies are important, but they represent only one possible anomaly type, and there could be many others, for example, atypical usage of some language constructs (not involving object interaction of any kind).

The DIDUCE tool~\cite{S1} for Java programs is based on dynamic code analysis: it runs a program and stores values for each expression found in it. It tries to induce invariants for these expressions, starting from the strictest ones and weakening them as new values are encountered. When these rules are violated, meaning that some expression gets a value that significantly differs from all previous values of this expression, this is reported as an anomaly. Papers~\cite{sekar2001fast} and~\cite{feng2003anomaly} also use dynamic code analysis but collect and analyze traces of system calls.

Several papers~\cite{Oizumi2015, Macia2012} define code anomalies as ``code smells'' --- specific code patterns indicating possible architectural flaws. An example of such pattern is a Feature Envy smell that arises when a method interacts with other classes more than with methods and fields of its own class. Finding such code fragments in a project usually helps to improve its design.

All these approaches are helpful if one is trying to find logical errors or architectural issues in programs and therefore are targeting programming language users, not its developers. It's also worth noting that dynamic analysis algorithms don't seem like a good fit for our task since we are looking at a potentially very large code base, and running all this code does not seem feasible. Projects might have all kinds of strange dependencies, and it would require a lot of human effort to understand how all of them are supposed to be compiled and run.

In the domain of programming language development, code anomalies, thanks to being atypical code, while having been actually implemented by someone, can be used as a source of data for fuzz testing techniques, which have proven to be valuable for finding tricky issues in compilers~\cite{yang2011finding}.

Finally, in 2018 we published a preliminary discussion of the problem statement for this study~\cite{bryksin2018}.

\section{Detection of code anomalies}\label{overview}

\subsection{Anomaly detection techniques}\label{anomaly-detection}

In data mining, anomalies are defined as deviations of the observed behavior from the expected behavior and are divided into three types~\cite{Chandola}:

\begin{enumerate}
\item \textit{Point anomalies} occur when a single data instance is considered abnormal compared to other data.
\item \textit{Contextual anomalies} occur when a single data object is anomalous in a specific context but not otherwise.
\item \textit{Collective anomalies} occur when linked objects are observed against other objects as an anomaly.
\end{enumerate}

In this study, we focus solely on point anomalies, but other anomaly types are also worth investigating and might lead to interesting results.

Our task implies that a dataset is unlabeled and we have no examples of code anomalies, which leads us to unsupervised anomaly detection methods~\cite{hodge2004survey}, such as Local Outlier Factor~\cite{lof} and Isolation Forest~\cite{if, if-anomaly-detection}. Several clustering methods~\cite{maimon2009introduction} could be useful for our task. Clustering methods such as DBSCAN~\cite{dbscan}, ROCK~\cite{rock}, or SNN~\cite{snn} do not require each object to be a part of a cluster, and we can see out-of-cluster objects as anomalies. One-class SVM~\cite{ocsvm} is a classification algorithm, but it assumes the presence of only one class, so the method could be used to detect outliers in unlabeled data.

Neural autoencoder~\cite{dau2014anomaly} can also serve as an anomaly detection method. Autoencoder is tasked with reconstruction of a data point through an intermediate representation. After decoder is trained on a set of datapoints, we can identify anomalous points by measuring reconstruction loss, with the assumption that the loss will be higher for outliers.

Statistical methods~\cite{prasad2013statistical} for anomaly detection measure the compliance of data to a specific probability distribution. The degree of anomalous behavior is the magnitude of the object's deviation from the distribution. For some methods in this group, an initial assumption about the distribution of data points is required, others may calculate a likely distribution that can generate the observed dataset.

\subsection{Source code embedding}

All of the anomaly detection algorithms, like most machine learning techniques, require embedding of analyzed objects in numeric vector space. To do so, first of all, we need to determine the level of structural units that will undergo analysis. It could vary from individual tokens and functions to files or even entire projects. Individual tokens and lines of code are highly dependent on their context and will not capture any significant anomalies. Functions are good candidates because most of them contain at least several lines of code that might form an anomaly and are isolated enough to represent a single operation within a class. Classes are also fitting for finding anomalies, especially in inheritance, properties use, class type parameters, etc. Using files as structural units would only allow us to detect anomalies in top-level constructions, of which there are not many and which rarely form anomalous code. Though analyzing entire projects as structural units of code may reveal some general patterns, these patterns will vary from one subject area to another and may not be informative.

Second, we need to define the level of code representation. It could be the source code itself (as text or token sets) or representations at different compilation stages. In case of Kotlin, we have access to following representations: a syntax tree; an intermediate representation for a specific compiler back-end (IR); JVM bytecode, LLVM bitcode or JavaScript code generated by the compiler. Different representations require different embedding techniques.

We can treat source code as text, which will allow the use of various natural language processing (NLP) features, such as bag-of-words or N-grams~\cite{Wang-ngrams-bug-detection, Hsiao-ngrams-js}. These features tend to capture the semantics of functions and variables names but tend to ignore the program's overall structure. NLP techniques could also be applied to bytecode since bytecode representation has a similar linear structure.

Representing a code fragment as a syntax tree introduces a wide range of embedding techniques divided into explicit and implicit approaches. Explicit approaches construct the vector from software metrics values~\cite{metrics-review, caliskan2015anonymizing} capturing lexical, syntactical, architectural and other properties. The resulting vectors are quite easy to build and comprehend: observing these values may provide a clear understanding of why this particular code fragment is considered to be anomalous. However, it is always a challenge to choose which metrics to include (for example, \cite{Varela-source-code-metrics} describes almost 300 metrics), especially when we are trying to search for anomalies of an unknown nature. Implicit approaches use features such as N-grams of token types, syntax trees hashing~\cite{jiang2007deckard, chilowicz2009syntax}, syntax trees encoding~\cite{peng2015building}, the latent vector of a neural network autoencoder~\cite{Tufano-DL-bytecode-and-AST} and other distributed code representations~\cite{code2vec-overview}. These vectors usually lack in interpretability but have proven themselves to be capable of capturing complex code properties, including semantic dependencies.

Described methods are also applicable to other available representations of Kotlin source code mentioned before: a list of LLVM bytcode instructions, a JavaScript syntax tree, and an intermediate representation (IR).

\section{The proposed method}\label{approach}
In this study, we perform a search for code anomalies in Kotlin programs that are written specifically for the JVM, since it comprises the majority of code written in Kotlin. We analyze these programs in the form of syntax trees and sequences of bytecode instructions (other representations mentioned above are available only for JavaScript and LLVM platforms).

Our goal is to detect two types of anomalies: syntax tree anomalies and compiler-induced anomalies. A \textit{syntax tree anomaly} is a code fragment that is written in some way that is not typical for the programming language community. It could have abnormal complexity, be composed of sophisticated code constructs or in any other way differs from the rest of the code written in this language.

A \textit{compiler-induced anomaly} is a code fragment that is not an anomaly in the syntax tree form but is an anomaly in the bytecode or vice versa. We choose to call them compiler-induced because their anomalous nature is revealed only after their bytecode is obtained and analyzed. We should note, though, that this wording does not imply any negative connotation: for instance, if the compiler does a good optimizing job and turns anomalous source code into non-anomalous bytecode, this is still an unexpected behavior and, therefore, an anomaly.

\begin{figure*}[h]
\centering
\includegraphics[width=\textwidth, height=4cm]{pipeline.png}
\caption{Overview of the proposed method for extraction and assessment of anomalies} \label{pipeline}
\end{figure*}

\subsection{Dataset}
\subsubsection{Source code and syntax trees}
To collect a large dataset, we cloned GitHub repositories that were created before March 2018, stated Kotlin as their main language, and were not forks of some other projects. That resulted in 47,751 repositories containing 932,548 source files with 4,044,790 unique functions. The collected source code was transformed into syntax trees using the parser module of the Kotlin compiler.

For the syntax trees code representation, we decided to use functions as code units. This allows us to simplify further analysis of detected anomalies, whether it is an expert assessment or a performance test. A function is an isolated unit of code, which is convenient if an anomalous fragment is to be analyzed by a compiler test because we only need to load the classes used in the function and to provide the input data it requires. Also, it will be easier for experts to analyze the function's code due to its isolation.

\subsubsection{Bytecode}
To detect compiler-induced anomalies we require both syntax tree and bytecode representations. GitHub supports publication of the project's builds packages which allows the direct collection of bytecode. Even though the dataset we collect in this way is relatively small, that kind of extraction approach is highly convenient because it allows us to collect the bytecode produced by its developers instead of figuring out the correct environment for each project and building them ourselves. We managed to obtain 41,226 compiled class files, which we then transformed into lists of JVM instructions.

Anomaly detection at functions level for bytecode representation is not as efficient as in the case of syntax trees because many Kotlin syntax constructions create additional code in bytecode representation. For example, lambdas (anonymous functions) are transformed into dummy functions. If an algorithm finds a bytecode anomaly related to such dummy function, it will be difficult to match a set of bytecode instructions to the source code. This directed us to choosing a class as a unit of evaluation for bytecode representation.

\subsection{Anomaly detection pipeline}
To detect and evaluate code anomalies, we propose the pipeline shown in Figure~\ref{pipeline}. It takes source code and project builds that are collected from GitHub as an input and converts them into syntax trees and JVM instructions lists using the dataset extraction module. Features are then extracted and anomaly detection techniques are applied. On the post-processing step, detected anomalies are classified according to their type. Finally, classes of anomalies are presented for expert evaluation. Then the pipeline's steps are adjusted for the next iteration with provided feedback.

The rest of this section discusses each step of anomaly detection in more detail. We describe assessment of individual anomalies in section \ref{sec:assessment}.

\subsubsection{Feature extraction}
We use software metrics and N-grams extraction to embed the syntax tree representation of the code. For the bytecode representation, only implicit feature extraction via N-grams is used. The bytecode has a linear structure and is rarely analyzed by humans, and fewer metrics are known for it.

Software metrics are divided into 4 groups:
\begin{itemize}
    \item general code metrics: the number of lines of code, the number of nodes and the height of the syntax tree, etc.
    \item structural metrics: nesting depth, cyclomatic complexity, number of branches in ``when'' expression, etc.
    \item external metrics of Kotlin functions: the formal arguments number, type parameters, annotations, the presence of a suspend modifier, etc.
    \item the number of particular language elements: expressions, operators, keywords, function calls, string patterns, etc.
\end{itemize}
Each function is encoded by a vector that contains the values of these metrics.

The N-grams extraction is performed as follows: the algorithm traverses a syntax tree and generates all unigrams, bigrams, ..., N-grams from connected nodes and adds a counter with value 1 for a new N-gram or increments the counter for an existing N-gram. For efficiency reasons, only parent-child relations are used to build N-grams, since considering all other relations (e.g., between sibling nodes) results in a vast number of N-grams, which will hinder the algorithm performance.

Both approaches result in a set of feature vectors. In the case of explicit representation, it is a k-dimensional vector where k is the number of metrics used. In the case of N-grams representation, it is a sparse vector where values represent counters of all N-grams met.

For compiler-induced anomalies detection, we have to process bytecode instructions sequences. N-grams are extracted incrementally from the sequence using a fixed size window. Within this window the extraction module generates all possible unigrams, bigrams, ..., N-grams, then moves the window one instruction ahead and repeats the procedure. Figure~\ref{bytecode_to_ngrams} presents an example of such extraction for $N=3$.

\begin{figure}[h]
\centering
\includegraphics[width=1.0\columnwidth]{n-grams.png}
\caption{Extraction of N-grams from sequences of bytecode instructions}\label{bytecode_to_ngrams}
\end{figure}

\subsubsection{Anomaly detection}\label{anomaly-detection-step}

In the first phase of this study, to assess general viability of the idea of anomaly detection in syntax trees, we settled on relatively simple algorithms for anomaly detection --- Local Outlier Factor~\cite{lof} and Isolation Forest~\cite{if}. We chose these algorithms because they are relatively easy to implement and trial, compared to more complex approaches.

Local Outlier Factor is based on the idea of calculating an anomaly score for each object based on its distance to k nearest neighbors in the dataset. More precisely, the local density of a data point is considered: a coefficient of the point's reachability to its k nearest neighbors. The anomaly score of a point is calculated as the ratio of local density to the mean local density in the region of a point.

Isolation Forest employs the principle of random forest to separate outlier objects from the rest of the data. Each isolating tree is constructed in the following way: at the current node, the algorithm randomly selects a feature and a value to split the data into child nodes. The process continues until all elements are separated from each other. The normality measure for an object is introduced as the average length from the root to the object's position in all isolation trees. Thus, the earlier the object is separated from the sample by isolation trees, the smaller its measure of normality will be. Figure~\ref{if_architecture} shows an example of isolation trees. Anomalous objects are marked in black, while ``normal'' objects are marked in gray.

\begin{figure}[h]
\centering
\includegraphics[width=0.9\columnwidth]{isolation_forest.png}
\caption{Detecting anomalies with Isolation Forest}\label{if_architecture}
\end{figure}

Aforementioned algorithms are more efficient if the analyzed objects are represented by vectors of low dimensionality. When working with high-dimensional data, calculations might require too much time, hence it is necessary to fine-tune the algorithm parameters. For instance, a good choice of parameters for the Isolation Forest can sufficiently reduce processing time and memory usage without any noticeable changes in detection accuracy~\cite{if-anomaly-detection}.

A popular approach to anomaly detection in high-dimensional datasets is a neural autoencoder~\cite{autoencoder-1}, a popular type of neural networks. An autoencoder isn't so resource-intensive comparing to the Local Outlier Factor and Isolation Forest, which is crucial when we analyze the N-grams representation dataset since each feature vector contains thousands of values.

Figure~\ref{autencoder_architecture} shows a basic autoencoder architecture that consists of input and output layers of the same size and a hidden layer that is significantly smaller. During the learning process, autoencoder trains to return the same vector as it receives as input. One of the results of this training is that we then have hidden layer values that comprise a vector representation (or embedding) of the input data. Another result is that if the trained autoencoder fails to reconstruct a data point, this data point may be an outlier to the dataset. Therefore, we can use the recovery error, i.e. the distance between input and output vectors as an anomaly score.

We propose the following as a way to detect compiler-induced anomalies: first, we calculate autoencoder anomaly scores both for syntax tree and bytecode representations of the same code fragment. If these scores differ by more than a specified threshold, we consider this fragment to be a compiler-induced anomaly.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\columnwidth]{autoencoder.png}
\caption{Architecture of a neural autoencoder}\label{autencoder_architecture}
\end{figure}

\subsubsection{Anomalies classification}\label{anomaly-classification}

We grouped anomalies into classes for further evaluation. Each anomaly was manually labeled with a set of tags. We performed classification manually to get the most reliable result possible.

\section{Evaluation}\label{evaluation}
To assess the viability of our approach to identification of anomalies, we conduct an evaluation of significance and practical value of anomalies extracted from the dataset. In this section, we describe the process of building the evaluation set (Section \ref{sec:eval-set}), our approach to assessment of importance of individual anomalies (Section \ref{sec:assessment}), and the results of this assessment (Section \ref{sec:assess-results}).

\subsection{Building the evaluation set}\label{sec:eval-set}
In this section we describe the process of building the set of anomalies for expert assessment.
\subsubsection{Extracting anomalies from explicit source code representations}
As the first step, to build explicit representation vectors for syntax trees in the dataset, we calculated 51 software metrics for each code fragment (49 quantitative and 2 binary). We scaled the quantitative attributes to the mean of 0 and the variance of 1, and reduced dimensionality of metric vectors from 51 to 20 via Principal Component Analysis. The number of principal components was chosen manually as a compromise between the time of training the models and the persistence of the explained variance (0.8).

We ran Local Outlier Factor and Isolation Forest outlier detection algorithms (Section~\ref{anomaly-detection-step}) against the vectorized snippets. Both of these algorithms have multiple hyperparameters. The first notable parameter is a contamination parameter, influencing the fraction of code fragments that should be marked as potential anomalies. Since the aim of the study is to find a set of anomalies that could be reviewed by experts, we chose the contamination parameter so that the classifier marked 0.01\% of the dataset, or approximately 400 out of 4 million collected functions. Specific parameter values were 0.0001 for Isolation Forest and 0.001 for Local Outlier Factor.

Other notable parameters of the outlier detection algorithms are the number of neighbors ($n\_neighbors$) for the Local Outlier Factor, which we set to 20, and the number of trees built ($n\_estimators$) for the Isolation Forest, which we set to 200. We settled on these values after several rounds of experiments and manual assessment of the resulting anomalies. As a result, these outlier detection algorithms have extracted 322 unique anomalies that we considered worthy of the attention of the Kotlin compiler team, from explicit representations of code snippets in the dataset.

\subsubsection{Extracting anomalies from implicit representations}
To build implicit representation vectors for both syntax trees and bytecode, we extracted unigrams, bigrams, and trigrams from individual instruction sets. The extraction yielded 1,708,022 unique N-grams for syntax trees and 110,835 unique N-grams for bytecode. Then we filtered rare and frequent N-grams, since they contain less information about the object, and ended up with 15982 N-grams for syntax trees and 4560 for bytecode.

Further, we trained an autoencoder to reconstruct N-gram vectors and optimized hyperparameters of the autoencoder by a heuristic search, taking into account limitations on memory use. Resulting parameters for the autoencoder are: number of epochs --- 5, minibatch size --- 1024, compression rates --- 0.25, 0.5, and 0.75.

To measure the \emph{anomaly score} of individual objects, we calculated the Euclidean distance between the input and the output vectors of the autoencoder. We considered anomalies all vectors of syntax trees with anomaly score exceeding 3 root mean square distances. For the compiler-induced anomalies, the threshold difference between the syntax tree and bytecode anomaly scores was set to 0.8. These values were also heuristically evaluated over multiple experiments to yield a reasonable number of anomalies. Using three autoencoder networks with different compression rates, we have extracted 191 unique syntax tree anomalies and 54 unique compiler-induced anomalies.

\subsubsection{Final processing of the evaluation set}
Using both explicit representations of source code and implicit representations of source code and bytecode, we obtained a total of 375 unique anomalies. After a quick manual filtering to remove non-interpretable items, we presented 145 remaining anomalies to Kotlin language experts for assessment of their usefulness. We describe the motivation behind expert assessment and its technique in the next section.

\subsection{Assessment of anomalies}\label{sec:assessment}
\subsubsection{Choice of assessment technique}
Code anomalies could be evaluated in several ways: by measuring the performance of the compiler on the anomalous code, by measuring runtime performance of a compiled program that contains anomalous code, or through expert assessment.

However, there are several important concerns that make performance measurements difficult to use and interpret.
We can only measure influence of syntax tree anomalies on the performance of the compiler's parser. It is worth noting that a parser is usually one of the simplest parts of a compiler, and performance problems in a compiler usually arise during other compilation stages: type inference, resolving, or code generation. Thus, running a performance evaluation of only a parser would be a shallow evaluation technique. At other compilation stages, performance of the compiler is highly dependent on the environment (e.g. the speed of type inference depends on the type parameters signature of the function that is called from an anomalous fragment). Moreover, a performance problem might be observed in one environment and not observed in another, and going through the complete variety of environments is an almost impossible task.

As for compiler-induced anomalies, even though we know the initial source code and the produced bytecode, it is problematic to evaluate the performance of the compiler. Data and computation that are related to a specific anomaly may be ordered differently within compiler's internal structure, which would make it very difficult to pinpoint the specific contribution of the anomalous code to the overall performance. However, it would be possible to evaluate performance of the program runtime, because we already have a corresponding bytecode to run.

A performance measurement would be relevant for assessment of the anomaly types that cause performance problems, but other anomaly types exist as well: for example, anomalies that highlight problems in the programming language design. Such problems can only be detected by an expert who deeply understands the concepts of the programming language and the internal structure of the compiler.
\subsubsection{Expert assessment}

To cover all the anomaly types, we decided to settle on expert evaluation for assessment of the anomalies. We have invited the development team of the Kotlin compiler to serve as experts in our study. The developers of the compiler have the most comprehensive understanding of the compiler's internals, and thus can be capable of attributing individual code constructs to concrete problems. Such problems include compiler performance, program runtime performance, and language design. Two people agreed to volunteer their services as experts.

We asked the experts to evaluate each anomaly, using a five-point scale from 1 to 5. Evaluation criteria addressed three aspects: (1) whether the code fragment is typical structurally, (2) whether the code fragment can become a valuable compiler test, and (3) whether the code fragment can cause compiler performance issues. The experts were asked to select the highest rank that will hold a true statement from the following list:

\begin{itemize}
    \item \textit{Rank 1}: very typical code; could not be used as a compiler test of any kind; does not cause performance issues;
    \item \textit{Rank 2}: code with typical architecture patterns; highly unlikely to be used as a compiler test; highly unlikely to cause performance issues;
    \item \textit{Rank 3}: mostly typical code with rare use of distinctive features; could be used as a compiler test, but fragments similar to this are already used in tests; Unlikely to cause performance issues;
    \item \textit{Rank 4}: the code comprises atypical combinations of language features; could become a valuable compiler test, fragments similar to this are already used in tests;  likely to cause performance issues;
    \item \textit{Rank 5}: highly atypical code structurally; will be a unique compiler test;  most likely will cause a performance issue.
\end{itemize}

The experts evaluated anomalies together, discussing each anomaly to conclusion and providing a collective ranking decision.

We also asked the experts to rank each of the anomaly types that we obtained on the post-processing step (Section~\ref{anomaly-classification}) to assess the perceived importance of individual types.

This ranking ensures that in future experiments we are able to tune our pipeline to produce more anomalies that are of interest to the experts.

\subsection{Assessment results}\label{sec:assess-results}

Figure~\ref{experts_evaluation_syntax_tree_anomalies} presents the results of expert importance assessment for syntax tree anomalies. 38 out of 91 source code anomalies selected for evaluation were considered to be important, i.e., helpful, by the experts.

Table~\ref{experts_classes_evaluation} presents assessment results for the anomaly types. The \textit{Rank} column contains the expert's evaluation rank, and the \textit{Size} column shows the number of detected anomalies of this type.

\begin{figure}[h]
\includegraphics[scale=0.2]{experts_evaluation_syntax_tree_anomalies}
\caption{Expert evaluation of importance for the discovered syntax tree anomalies}
\label{experts_evaluation_syntax_tree_anomalies}
\end{figure}

Bytecode anomalies (54 out of 145) correspond to the properties of the compiler, rather than source code. It makes such anomalies difficult to classify, hence we did not assign class labels to them. Figure~\ref{experts_evaluation_conditional_anomalies} presents the results of expert evaluations for compiler-induced anomalies. 31 out of 54 anomalies were considered helpful.

\begin{table}
\captionof{table}{Results of expert assessment of anomaly types, sorted by importance}
  \label{experts_classes_evaluation}
  \begin{tabular}{ l l l l }
    \hline
    \textbf{\#} & \textbf{Anomaly type} & \textbf{Size} & \textbf{Rank} \\ \hline
    1 & Delegates & 1 & 5 \\
    2 & Type arguments & 8 & 5 \\
    3 & ``When'' expression & 35 & 5 \\

    4 & Annotations & 2 & 4 \\
    5 & Call chains & 5 & 4 \\
    6 & Enumerations in ``when'' & 2 & 4 \\
    7 & ``If'' expressions & 15 & 4 \\
    8 & Nested calls & 3 & 4 \\
    9 & Similar call expressions & 88 & 4 \\
    10 & Strange code constructs & 3 & 4 \\

    11 & Assignments & 57 & 3 \\
    12 & Large methods & 2 & 3 \\
    13 & Code hierarchy & 32 & 3 \\
    14 & Function parameters & 17 & 3 \\
    15 & Multiline strings & 27 & 3 \\
    16 & ``Try-catch'' expressions & 1 & 3 \\

    17 & Arrays or maps & 32 & 2 \\
    18 & Class references & 2 & 2 \\
    19 & Concatenations & 13 & 2 \\
    20 & Lambdas & 2 & 2 \\
    21 & String literals & 2 & 2 \\
    22 & Logical expressions & 6 & 2 \\
    23 & Complex loops & 8 & 2 \\
    24 & Similar code fragments & 2 & 2 \\
    25 & ``Throw'' expressions & 2 & 2 \\

    26 & Assertions & 1 & 1 \\
    27 & Empty string literals & 2 & 1 \\
    28 & Local variables & 2 & 1 \\
    29 & Nested functions & 2 & 1 \\
    30 & Type casts & 2 & 1 \\ \hline
  \end{tabular}
\end{table}

\begin{figure}
\includegraphics[scale=0.2]{experts_evaluation_conditional_anomalies}
\caption{Expert evaluation of importance for the discovered compiler-induced anomalies}
\label{experts_evaluation_conditional_anomalies}
\end{figure}

Based on the results of expert assessment, we have compared the assessment scores for syntax tree anomalies obtained from two experiments: the first one that used explicit vectors of metrics values and Local Outlier Factor/Isolation Forest (the explicit experiment) as well as the second one that used implicit representations based on N-grams obtained with an autoencoder (the implicit experiment). The comparison shows that anomalies from both experiments were rated high, and both approaches are suitable for code anomalies detection. On the one hand, the use of explicit code metrics simplifies the analysis of the resulting anomalies, and the set of these metrics can be easily extended to search for new anomaly types. On the other hand, implicit representations of code are able to capture properties that could be rather difficult, if possible, to describe with explicit code metrics.

\subsection{Data availability}
The data that supports the findings of this study is openly available.\footnote{The dataset used in this study: \url{https://zenodo.org/record/3733794}} The created tools are also available on GitHub.\footnote{The tools supporting the proposed methods: \url{https://github.com/JetBrains-Research/kotlin-code-anomaly}}

\section{Threats to validity}\label{threats}

Our study is subject to several threats to validity and generalizability.

\textbf{External validity}. Our dataset was comprised of only open source projects available on GitHub. This limits our ability to claim that our list of discovered anomalies is exhaustive, as code of proprietary projects might contain other anomalies.

Our anomaly detection pipeline was targeted at the Kotlin ecosystem from the very start of the project. We were able to obtain our results thanks to a combination of several unique factors, notably including quick growth of the Kotlin open source community, our ability to get in contact with the compiler development team, and slightly lower maturity of the compiler, compared to other more established languages and ecosystems. Considering these unique factors, we should acknowledge that obtaining similar results for other programming languages would require building a potentially sophisticated method from scratch.

\textbf{Construct validity}. Our resulting list of anomalies is ultimately influenced by our choices regarding the set of metrics, methods for anomaly detection, and parameters of the algorithms.

\textbf{Bias of experts}. Our ranking of importance for individual anomaly types is based on estimates provided by two experts. Such quantitative estimates based on expert opinion are intrinsically subjective~\cite{kitchenham1996evaluating}. Moreover, the role of individual biases in such a small group is particularly high. Finally, our experts are the developers of Kotlin, who are deeply involved with the compiler internals; thus, their subjective perception of importance might differ from the broader group of interested public --- for example, from professional Kotlin developers.

\smallskip
We believe that these threats, while worth noting, do not invalidate our results or diminish their value. We do not claim that our methodology is universal or applicable to other languages. Instead, with this study we unravel a new scope for application of anomaly detection techniques in software engineering, and demonstrate that these techniques help to provide actionable insights to language developers.

\begin{figure*}

\includegraphics[width=\textwidth]{images/anomaly_example_forth_compiler.png}
\caption{An example of atypical, yet valid, code from a Forth implementation written in Kotlin.}
\label{anomaly_forth_compiler}
\end{figure*}

\section{Discussion}\label{discussion}
\subsection{Syntax tree anomalies}
Syntax tree anomalies are code fragments that are not typical in the Kotlin community. They are either very rare combinations of code constructs, certain combinations of code constructs repeated a large number of times, or functions with atypical characteristics, like very large object hierarchy or extensive domain-specific constructs~(Figure \ref{anomaly_forth_compiler}).

The detected syntax tree anomalies were divided into three categories:
\begin{enumerate}
\item \textbf{Anomalies that correspond to language design issues}. Such anomalies were noted and archived by the Kotlin compiler team for further discussion and research. Later, such discussions could lead, for example, to introduction of a new construct to the language, deprecation of constructs use in certain cases, extension of the language standard library, refinement of the reference documentation or the language specification. Thus, the detected anomalies will be able to play a role in the development of the programming language.
\item \textbf{Anomalies that correspond to cases when some part of the compiler might work incorrectly or slowly}. The experts considered such anomalies as valuable compiler tests, and accepted to include them in the compiler's testing infrastructure. They can be used as performance tests for the compilation stages of type inference and resolving, as type inference tests for correctness, or as tests for generated bytecode. For example, the experts noted that some anomalies highlight various corner cases of the type inference as well as data flow analysis and can be very useful during refactorings in these compiler modules. An example of such an anomaly is presented in Figure~\ref{anomaly_example_2_source}. This function can act as a good test for the performance and correctness of the compiler type inference module, since processing it involves non-trivial analysis and type inference.
\item \textbf{Anomalies that correspond to potential performance issues of the program's runtime environment}. In such cases, there may also be some issues with the compiler. For example, non-optimal code generation, failures, or lack of some optimizations. Such code fragments can also be used as compiler tests; more precisely, as performance tests for the compiled programs.
\end{enumerate}

\subsection{Compiler-induced anomalies}
Compiler-induced anomalies correspond to cases when a complex or atypical bytecode was generated by a simple or ``normal'' syntax tree (or vice versa, a non-anomalous bytecode was produced from an anomalous syntax tree).

The detected compiler-induced anomalies were caused by the following issues:
\begin{enumerate}
\item \textbf{Non-optimal code generation cases in the compiler that manifested in some rather exotic code examples}. In such cases, the anomaly can be used as a test for bytecode generation.
\item \textbf{Rather complex functions were inlined in code fragments that were considered abnormal}. Such cases have nothing to do with compiler problems, but could be very useful for developers of libraries and frameworks, because their API is available to a potentially large number of programmers. Detection of function inlining problems could also help regular developers (i.e. users of the language) to fix performance issues in their projects. Figure~\ref{conditional_anomaly_example_1} presents an example of such an anomaly containing problematic function inlining: the framework developers wrote a large and complex function to bind properties with configuration data, which was marked ``inline''; therefore, its bytecode was copied to all places where it was called. In this example, the ``bind'' function is only called 9 times, but the bytecode is already very large and complex. This code fragment was not considered a syntax tree anomaly, but its bytecode was marked as anomalous, which helped to catch this code fragment.
\end{enumerate}
Among the suspicious code fragments there were several cases when some of the compiler optimizations worked well, and complex syntax trees were minimized to form a rather simple bytecode. Surely, such cases are not issues of any kind, but they can nevertheless be useful to compiler developers: for example, they can learn from the successful optimizations and further improve them.

\section{Conclusion}\label{conclusion}

In this paper we present two kinds of experiments to detect different types of code anomalies (syntax tree anomalies and compiler-induced anomalies), using different approaches both at the code vectorization and at the anomaly detection stages. As a result, 91 syntax tree anomalies and 54 compiler-induced anomalies were presented for assessment of importance to the Kotlin compiler developers. According to the results of the assessment, 38 syntax tree anomalies and 31 compiler-induced anomalies were considered useful (they got rank values 4 and 5). Some of these anomalies were added into the compiler testing infrastructure as performance and correctness tests for the compiler front-end and back-end; several anomalies were postponed for further discussion on possible language design issues. Based on these results and further discussions with the Kotlin compiler developers, we believe that the detected anomalies are useful and valuable for the language development, and our proposed approach proved itself successful in detection of various code anomalies.

We outline several directions for future research:
\begin{enumerate}
\item The post-processing step of the pipeline could be automated (at least to some extent) to provide tools for classification and labeling of the found anomalies. This would allow us to focus more on finding new and more interesting types of anomalies.
\item The bytecode dataset could be increased (for example, by creating tools that automatically build projects from GitHub). At the moment, only a small part of the repositories publish builds of their projects in the ``Releases'' GitHub section. Automatic build tools would provide us with bytecode for all projects with syntactically and semantically correct code. With such dataset at hand, we could identify much more compiler-induced anomalies.
\item Compiler-induced anomalies could be used to evaluate the impact of new language features on the compilation process. This could be implemented as tests that track anomaly scores of some code examples during the compiler development process. If the anomaly scores change too much after a specific change in the compiler code, then such changes should be analyzed in detail and it should be understood what led to such an effect.
\end{enumerate}

\begin{figure*}
\includegraphics[scale=0.55]{anomaly_example_2_source}
\caption{An example of a syntax tree anomaly used as a test for performance and correctness of type inference in the compiler}
\label{anomaly_example_2_source}
\end{figure*}

\begin{figure*}
\includegraphics[scale=0.17]{conditional_anomaly_example_1.png}
\caption{An example of a compiler-induced anomaly highlighting an issue with function inlining}
\label{conditional_anomaly_example_1}
\end{figure*}

\clearpage

\bibliographystyle{ACM-Reference-Format}

Software Engineering (cs.SE); Machine Learning (cs.LG); Programming Languages (cs.PL)


Interstellar Dust Emission as a CMBR Foreground

Abstract: This chapter discusses Galactic dust and how its thermal emission confuses CMBR measurements. Interstellar dust grains are composed of many differing species, and observational evidence has only begun to disentagle their composition and sizes. Fortunately, their far-IR/millimeter emission is less complex. We describe how a very simple two-species model can describe the emission at 200 < nu < 3000 GHz to high precision. At lower frequencies, other non-thermal processes may dominate the emission from dust.

\section{Introduction}

The different sources of radiation in the night sky from the near-infrared
through radio are shown in Figure \ref{fig_nightsky}.  These are to be
compared with the CMBR at a temperature of $2.73{\rm ~K}$, and the fluctuations
in the CMBR which have an amplitude lower by $\sim 10^{-5}$ (dashed line).
On the Wien tail of the CMBR, thermal emission from Galactic dust is
the main source of confusion.  This emission is now well-understood,
thanks to the {\it COBE}/FIRAS experiment (Fixsen {\it et al.}\ 1996).
We can predict the thermal emission at all frequencies $\nu \la 3000 {\rm ~GHz}$
to a precision of $\sim 10\%$ (Finkbeiner, Schlegel \& Davis 1999).
This work builds upon the maps of dust column density produced
by Schlegel, Finkbeiner, \& Davis (1998; hereafter SFD98).

At frequencies below $\sim 100{\rm ~GHz}$, there are more possible sources of
confusion with the CMBR.  Galactic dust may be one such source if it is
rotating supra-thermally, as has been proposed by Draine \& Lazarian (1998).
Other possible sources are synchrotron emission,
free-free emission from ionized hydrogen, and radio emission from
extragalactic sources.  We briefly discuss all as possible sources for
anomalous emission seen at $\nu \la 100{\rm ~GHz}$ by CMBR experiments.

\begin{figure}
\epsfxsize=5.0in\epsfbox{finkfig1.eps}
\caption{Contributions to the infrared/millimeter/radio sky above the
Earth's atmosphere}
\label{fig_nightsky}
\end{figure}

\subsection{Observational Evidence}

Dark clouds are apparent to any observer who looks at the Milky Way
from a dark site.  These dark regions are especially conspicuous in
the Southern hemisphere where one sees the utter darkness of the Coal
Sack and dark bands across Centaurus and Sagittarius.
William Herschel regarded these regions as curious voids in the
distribution of stars.  In the early 20th century, detailed studies
by Trumpler convincingly demonstrated that interstellar dust was responsible.
Starlight passing through a dusty medium is subjected to both absorption
and scattering.
The combination of these effects is termed extinction, denoted $A(V)$
for the magnitudes of visual-band extinction.  Both processes are
wavelength-dependent, resulting in an apparent reddening of the light.
Reddening is usually parameterized by its differential extinction
between $B$- and $V$-band, $E(B-V) \equiv A(B) - A(V) $.

Advances in infrared astronomy since 1980 have dramatically improved
our understanding of interstellar dust.  Dust scatters some light in
the optical, as can be seen in the Pleiades star cluster.
In our solar system, a thick disk of dust in the zodiacal plane
scatters some sunlight.  However, more energy is absorbed and then re-radiated
in the infrared.  The zodiacal dust re-radiates most of its energy at
$\lambda \ga 10 \micron$.
The energetics of this zodiacal dust would be uninteresting except that
it dominates the sky (above the atmosphere) at wavelengths $\sim 25\micron$.
This zodiacal emission has been extensively modeled by Kelsall {\it et al.}\ 1998.
Interstellar dust is heated to a lower temperature, radiating its energy
at $\lambda \ga 100\micron$.  Approximately $20\%$ of the energy from
starlight in our Galaxy is re-processed this way by interstellar dust.

\subsubsection{Dust Composition}

The diffuse ISM is known to contain many different types of molecules
and dust grains whose optical properties are dissimilar.  Identifications
for some of these constituents have been made from UV absorption features
and mid-IR emission features.  The ubiquitous $2175\AA$ bump (in absorption
from background stars) is most widely attributed to small ($<100\AA$)
graphite particles ({\it c.f.}, Draine 1989).  Carbon in another form, that of
polycyclic aromatic hydrocarbons (PAHs), is probably responsible for
strong emission features at $3.3$, $6.2$, $7.7$, $8.6$ and $11.3\micron$
({\it e.g.}, Allamandola, Tielens, \& Barker 1985; Desert {\it et al.}\ 1990).
A review by Henning \& Salama (1998) contains the latest
information on carbon-based grains.
Other mid-IR emission features are attributed to silicates and troilites.
For a coherent and only somewhat dated review, see Whittet (1992).

\subsubsection{Grain Sizes}

Along with the composition of dust grains, their size distribution
also impacts their extinction and emission properties.
The UV/optical extinction, $A(\lambda)$, is neatly described by a
one-parameter family of curves (Cardelli, Clayton, \& Mathis 1989).
That one parameter is usually chosen as an observable ratio,
\begin{equation} R_V = {A(V) \over A(V) - A(B)} . \end{equation}
Clayton \& Cardelli (1988) have measured the extinction curve for
76 lines-of-sight in our Galaxy.  Most diffuse clouds favor an $R_V=3.1$
extinction curve.  However, there are variations in $R_V$ from $2.6$ to
$5.5$, where the larger values correspond to some dense cloud environments.
Changes in grain size distribution is presumed responsible for these
variations.  On average, interstellar dust grains are thought to follow a
power law distribution of grain sizes $dn/da \propto a^{-3.5}$
(Mathis, Rumpl, \& Nordsieck 1977)
from about $0.0005\micron$ (the size at which absorption of a single photon
can sublimate mass away from the grain) to $0.25\micron$.
The large-size cutoff at $0.25\micron$ is necessary to be consistent
with the measurements of $R_V$ (Kim, Martin \& Hendry 1994).
Larger grains would be relatively gray absorbers, increasing the value
of $R_V$ beyond that observed.  So $R_V$ provides crucial information on
the grain size distribution, and may be found in the future to correlate with
some aspect of the far-IR emission from dust.

\subsubsection{Polarization Effects}

Besides attenuating starlight, dust grains are known to polarize that
light.  To date, polarization of optical light has been measured
towards $\sim 10,000$ stars (Mathewson \& Ford 1970; Korhonen \& Reiz
1986), some high above the Galactic plane (Berdyugin \& Teerikorpi
1997).  Regions of higher polarization are correlated with regions of
higher extinction ({\it e.g.}, Larson, Whittet, \& Hough 1996).  One expects
polarization only if the dust grains are non-spherical and have a
preferential axis of alignment.  For this reason, polarization studies
have been used to argue for coherent, large-scale magnetic fields in
the Galaxy ({\it e.g.}, Heiles 1996).  The magnetic moments of the dust grains
will preferentially align with the ambient magnetic field.
As these aligned grains rotate, they emit polarized radiation
that may interfere with attempts to measure the polarization of the CMBR
anisotropy.

\subsubsection{Dust Distribution}

The morphology of the interstellar dust is quite complex, consisting
of dark clouds and long filamentary structures arching across the sky.
The diffuse, wispy parts of this emission are known as cirrus, to
differentiate them from more compact features (often associated with
individual objects).  Not surprisingly, the dust
emission is concentrated in the Galactic plane, but significant cirrus
is observed in every part of the sky at $100\micron$
(Low {\it et al.}\ 1984).  SFD98 have computed the power spectrum of this dust
at latitudes $|b| > 45^\circ$, finding a reasonable fit to
$P(k)\propto k^{-2.5}$.  However, as this cirrus is in coherent structures, a
1-D power spectrum is not statistically sufficient to describe its distribution.
This morphology is another obstacle to CMBR measurements since it is not at
all reminiscent of gaussian random-fields that CMBR pundits usually model.
However, this phase coherence provides avenues that
may prove helpful in determining the quality of a dust foreground
model as applied to real microwave data.  For example, we can easily
measure the 3-point function of the Galactic dust.  If the CMBR is
presumed to be gaussian, a non-zero 3-point function in a
foreground-subtracted CMBR map will measure the level of residual
contamination (Spergel 1998).

\subsection{Far-IR Emission}

In spite of the expected melange of dust grains, it was originally
expected that in the far-IR/submillimeter bands, all dust would have
similar optical properties.  For example, Draine \& Lee (1984)
predicted $\nu^2$ emissivity for both silicate and graphite grains
whereby the emission is described by
\begin{equation} I_\nu = B_\nu(T) \nu^2 . \end{equation}
The theoretical motivation for this emissivity is based on the fact that the
response of a damped harmonic oscillator to a driving force varies as
$\nu^2$ in the low frequency limit (far below resonance).  Even though
microwave frequencies are below all of the phonon modes in the
smallest crystals, this is not true for grains bigger than $\sim
0.1\micron$, and these bigger grains are expected to dominate the
emission.  More recent laboratory measurements (see Pollack {\it et al.}\
1994 for a review) suggest that this is an oversimplication, with
different species of grains having differing emissivity laws in the
far-IR.  In particular, glassy materials (e.g. amorphous silicates) in
some cases radiate more efficiently than their crystalline
counterparts, and are more closely characterized by a $\nu^{1.5}$
emissivity (Agladze {\it et al.}\ 1996).  As we show below, these silicates
probably dominate the millimeter/microwave emission relevant to CMBR
studies.

Fortunately, complete knowledge of the size and composition of interstellar
dust grains is not necessary to predict their impact upon CMBR studies.
Since the long wavelength emissivity of a grain scales as its size $a$
times the surface area, or volume $a^3$, the larger grains dominate
the submillimeter emission (Draine \& Anderson 1985)
The very small grains (VSGs) are transiently heated to higher temperatures
and dominate the dust emission at $\lambda \la 100\micron$, but do not
contribute significantly to submillimeter emission.
The opacities of dust grains are exceedingly complicated in the UV/optical
where they absorb energy, and slight changes in composition can significantly
alter them.  However, their far-IR opacities are not nearly as sensitive.
Both laboratory and astronomical measurements are well-fit at
$\lambda \ga 1000 \micron$ ($\nu \la 300 {\rm ~GHz}$) using power-law
emissivities with indices in the range from $1$ to $3$
(see Pollack {\it et al.}\ 1994).

\section{Microwave Foregrounds}

Interstellar dust is only one of the many foregrounds that complicate
measurement of the CMBR anisotropy.  An analysis of the relative
importance of the various foregrounds is given by Tegmark \&
Efstathiou (1996).  In this section, we provide a brief overveiw of
the most important foregrounds.

At low frequencies, the dominant diffuse Galactic emission is due to
synchrotron radiation produced by relativistic electrons interacting
with the Galactic magnetic field.  A map at 408 MHz by Haslam {\it et al.}\
(1981) is the only full-sky synchrotron template in existence.
Reich \& Reich (1986) cover the northern sky in the $1420 {\rm ~MHz}$
continuum, and are complemented by the new Rhodes/HartRAO survey at
$2326 {\rm ~MHz}$ in the south (Jonas, Baart, \& Nicolson 1998).  Where these
maps all overlap, they allow examination of the synchrotron radiation
at 3 frequencies -- and indicate significant spectral index variation
across the sky.  The synchrotron radiation is usually characterized by
a spectral index $\beta=-2.7$ in units of brightness temperature, or
$\beta=-0.7$ in units of flux ({\it e.g.}, MJy/sr).  The typical
high-latitude brightness temperature at $20 {\rm ~GHz}$ predicted from these
data is approximately $0.5 - 1 {\rm ~mK}$, rising to no more than $20 {\rm ~mK}$ in the
Galactic plane.  Extrapolating the power law to $100 {\rm ~GHz}$, the
synchrotron emission would contribute at most 10 $\mu K$ to the
emission at high latitude.  However, the index $\beta$ is seen to
steepen to $-2.9$ or less even at low frequencies, due to a cutoff in
the electron energy distribution (Jonas 1998). Therefore,
synchrotron emission is unlikely to be an important emission component
at high latitude at $\nu \ga 30 {\rm ~GHz}$.

Another diffuse component that is not well constrained is free-free
emission from ionized gas.  Except in H\,{\scriptsize II}\ regions, free-free is lost
in the signal of synchrotron emission at low frequencies ({\it e.g.}, at 408 MHz).
Although there is known to be ionized gas at high latitude from
H$\alpha$\ emission (Haffner, Reynolds, \& Tufte 1998), so far no
convincing correlation has been seen between H$\alpha$\ data and $\sim
20 {\rm ~GHz}$ data.  As more data of both types become available, we expect
to see a correlation.  Another possible emitter of free-free is the
Local Hot Bubble (LHB) of ionized gas in which the Sun and many other
stars reside.  A map of this bubble was produced using the absorption
of soft X-rays observed by ROSAT (Snowden {\it et al.}\ 1998).  Because the gas in
the LHB is very hot, the anticipated H$\alpha$\ emission per microwave
emission is very low, so this will not appear in the H$\alpha$\
template.

In addition to these diffuse components, a wealth of point sources are
apparent at radio and far-IR frequencies.  Several hundred thousand
radio sources have been observed in the PMN (Griffith \& Wright 1993)
and NVSS (Condon {\it et al.}\ 1998) surveys.  At $60\micron$, SFD98
identified $\sim 20,000$ extragalactic sources and unresolved Galactic
sources.  These sources will contribute a great deal of non-gaussian
contamination to the CMBR.  The brightest ones are very easy to
identify and remove.  Preliminary work has shown that removing the
brightest sources will be sufficient for power-spectrum analyses with
the MAP experiment (Tegmark \& de Oliveira-Costa 1998).  Point source
decrements due to the Sunyaev-Zeldovich effect will have also been
seen, and their impact on CMBR observations has been noted (Aghanim
{\it et al.}\ 1997).

As we have emphasized,
the dust emission described in the next section is not the only
diffuse foreground, but it is the brightest at high frequencies
($\nu \ga 100{\rm ~GHz}$).  However, it is important for another reason.  The
excess dust-correlated emission observed in the DMR $31.5$ and $53 {\rm ~GHz}$
channels (Kogut {\it et al.}\ 1996) and later by other experiments
(e.g. Leitch {\it et al.}\ 1997; de Oliveira-Costa {\it et al.}\ 1998) is still
unexplained.  It may be caused by dust-correlated free-free emission.
On the other hand, it may be the result of rapidly rotating dust
grains (Draine \& Lazarian 1998).  Whatever this component is, it appears to
correlate fairly well with dust, making the determination of a good
dust template more important than thermal (vibrational) dust emission
alone would indicate.

\section{Single Component Dust Model}

A simple but naive prediction for millimeter/microwave dust emission
can be made from our previous work.  SFD98 extensively studied the
emission from dust in the regime $100\micron < \lambda < 240\micron$.
Assuming a $\nu^2$ emissivity model, the temperature of the dust was
mapped with a resolution of $1.3^\circ$ from the DIRBE
$100\micron$/$240\micron$ emission ratio.  The $100\micron$ emission
of the dust was mapped with a resolution of $6.1\arcmin$
by utilizing small-scale information from the {\it IRAS}\ Sky Survey Atlas
(Wheelock {\it et al.}\ 1994).
Emission at lower (millimeter/microwave) frequencies can be predicted by
extrapolating the $100\micron$ flux using this temperature fit.
For each line-of-sight in the maps, the emission at frequency $\nu$ can
be expressed as
\BEL{equ_extrap1}
    I_\nu = K^{-1}_{100}(\alpha,T) I_{100}
    {\nu^\alpha B_\nu(T) \over \nu_0^\alpha B_{\nu_0}(T) }
\end{equation}
where $B_\nu(T)$ is the Planck function at temperature $T$,
$I_{100}$ is the DIRBE-calibrated $100\micron$ map, $K_{100}(\alpha,T)$
is the color correction factor for the DIRBE $100\micron$ filter when
observing a $\nu^\alpha$ emissivity profile, and $\nu_0=3000{\rm ~GHz}$ is
the reference frequency corresponding to $100\micron$.

The extrapolated, millimeter emission from dust as predicted from
SFD98 can be compared directly to the FIRAS measurements.
We synthesize a broadband FIRAS $500 {\rm ~GHz}$ channel (to increase S/N)
and show that the correlation with predictions from equation \ref{equ_extrap1}
is very good (Figure \ref{fig_firas_scatter}c).
For comparison, we plot the correlation
with (a) H\,{\scriptsize I}\ column density, and (b) DIRBE $100\micron$ flux.
Both of these show a scatter that is 3.7 times worse than with SFD,
demonstrating that millimeter emission from dust is neither simply related
to the H\,{\scriptsize I}\ column density, nor is the dust at one temperature everywhere
on the sky.

\begin{figure}
\epsfxsize=5.5in\epsfbox{finkfig2.eps}
\caption{Comparison of the FIRAS data at $500{\rm ~GHz}$ with three predictors.}
\label{fig_firas_scatter}
\end{figure}

Our regression line in Figure \ref{fig_firas_scatter} differs significantly
from unity, indicating that a $\nu^2$ emissivity is incorrect for the dust.
At lower frequencies, the slope departs even more strongly from unity.
This is an indication that a $\nu^2$ emissivity is incorrect for the dust,
as can be seen from the mean spectrum of large regions of the sky
(Figure \ref{fig_3temp}: since the DMR is a differential instrument,
we have plotted the difference between bright and faint regions of the sky.)

\begin{figure}
\epsfxsize=4.5in\epsfbox{finkfig3.eps}
\caption{Difference spectra from the DMR ($\nu < 100{\rm ~GHz}$) and
DMR ($\nu > 200{\rm ~GHz}$) instruments.  These spectra divide the sky into
three regions based upon temperature, and are renormalized to
$I_{100\micron} = 1{\rm ~MJy}/{\rm sr}$. }
\label{fig_3temp}
\end{figure}

The SFD prediction can be made using other emissivity profiles by
modifying the exponent, $\alpha$, in equation \ref{equ_extrap1}.
An $\alpha=1.5$ emissivity profile results in a better fit
at low frequencies, but ruins the fit at high frequencies.
In fact, there is \emph{no} power-law emissivity model that adequately
fits the FIRAS data.

\section{Multi-Component Dust Model}

Motivated by observations of many different grain species in the ISM,
we have formulated a general description of a multi-component ISM
suitable for microwave predictions (Finkbeiner {\it et al.}\ 1999).  We
capitalize upon the uncertain degeneracies between components to
construct a multi-component model with very few degrees of freedom.
As we discussed, any one component of dust might be expected to
exhibit power-law emissivity in the range $100 \la \nu \la 1000 {\rm ~GHz}$.
The complicated behavior of their optical opacities is unimportant.
The only relevant optical parameter is $\kapstar$, the effective
cross-section to the interstellar radiation field (ISRF): \begin{equation}
\kapstar_k = { \int_0^{\infty} \kappa_k(\nu) E(\nu) d\nu \over
\int_0^{\infty} E(\nu) d\nu } \end{equation} where $E(\nu)$ represents the
angle-averaged intensity in the ISRF.

We construct a multi-component model of the ISM, analogous
to equation \ref{equ_extrap1}, by summing the emission of these components:
\BEL{equ_multi}
   I_{\nu} =
   { \sum_k{ f_k \epsilon_k(\nu) B_\nu(T_{k})}
    \over \sum_k{ f_k \epsilon_k(\nu_0) B_{\nu_0}(T_{k})
        K_{100}(\alpha_k,T_{k})} } I_{100}
\end{equation}
where $f_k$ is the mass fraction of the $k$-th component, $T_{k}$ is
the temperature, $K_{100}$ is the DIRBE
color-correction factor and $I_{p,100}$ is the SFD98
$100\micron$ flux.
The mass fractions are forced to sum to unity, {\it i.e.}\ $ \sum_k f_k = 1 $.
Each emissivity is a power-law with index $\alpha_k$.
The observationally relevant parameter is the ratio of far-IR emissivity
to the effective optical opacity, $q = \epsilon(\nu_0)/\kapstar$.

Each dust component is therefore described by three global parameters
($f_k$, $q_k$, $\alpha_k$) and one parameter
that varies with position on the sky, {\it e.g.}, $T_1(\vec{x}$).
The temperatures of all dust species are coupled by demanding that they
all respond to the same ISRF (for more details, see Finkbeiner {\it et al.}\ 1999).
Whether or not the actual components of the dust physically correspond to these
components, equation \ref{equ_multi} can be thought of as a phenomenological
``expansion set'' for describing the composite dust spectrum.

Remarkably, we find that only two components of dust are required to
describe the spectrum everywhere on the sky as seen by FIRAS.
As a first guess, we tested a two-component model designed to replicate
the spectrum in Pollack {\it et al.}\ 1994.
The prescription in their paper for their best-fit broken
power law corresponds in our model to $\alpha_1=1.5, \alpha_2 = 2.6,
f_1=0.25, $ and $q_1/q_2=0.61$.  This results in a considerably better
fit without fitting any new free parameters.
Between $800$ and $1800 {\rm ~GHz}$, where the FIRAS signal is very good, the
model matches the data to approximately 1\% everywhere.
At lower frequencies, the largest deviation is $10\%$.

Allowing $f_1$ and $q_1/q_2$ to float provides even better fits.
Our best fit is achieved with two components with power-law emissivities
$\alpha_1 = 1.68$, $\alpha_2 = 2.78$, where the former component
dominates the emission at $\nu \la 500 {\rm ~GHz}$.
We have fit 71\% of the sky as observed by FIRAS (4378 pixels)
at 123 frequencies, for a total of $\sim 540,000$ data value comparisons.
We also effectively fit for an unconstrained zero-point value at each
frequency to avoid having the solution influenced by uncertainties in
the cosmic infrared background.
The resulting reduced $\chi_\nu^2$ is $2.3$/DOF, as compared to $\sim 30$ (!)
for the best single-component model.  Results for a suite of models,
including our best fit 2-component model, are given in Finkbeiner {\it et al.}\ (1999)
along with tests of their robustness.

\begin{figure}
\epsfysize=7.0in\epsfbox{finkfig4.eps}
\caption{Correlation of different templates with the Leitch {\it et al.}\ 1997
CMBR data.  A $\csc|b|$ model or 21-cm map (from Leiden-Dwingeloo)
are poor predictors for microwave dust emission.}
\label{fig_leitch}
\end{figure}

\section{Discussion}

The thermal emission from Galactic dust can be very successfully
predicted at millimeter/microwave frequencies using a two-component
composition model with temperature varying on the sky.  We interpret
the best fit as a amorphous silicate-like component ($\nu^{1.7}$
emissivity, $\langle T \rangle \approx 9.5{\rm ~K}$) and a carbonate-like component
($\nu^{2.8}$ emissivity, $\langle T \rangle \approx 16{\rm ~K}$).  The FIRAS data are
also reasonably well fit using two $\nu^2$ emissivity components, if one of
these components is very cold ($\langle T \rangle \approx 5{\rm ~K}$) as proposed by
Reach {\it et al.}\ (1995).  We find no evidence for a warm component ($\nu^1$
emissivity, $\langle T \rangle \approx 29{\rm ~K}$) as proposed by Lagache {\it et al.}\ 1999.

A full range of Galactic dust models are discussed in Finkbeiner
{\it et al.}\ 1999.  Numerical predictions for thermal emission will be made
available at our web page,
\begin{quote} {\tt http://astro.berkeley.edu/dust} , \end{quote}
or at the FORECAST (FOREgrounds and CMB Anisotropy Scan simulation
Tools) home page,
\begin{quote} {\tt
http://cfpa.berkeley.edu/forecast}.
\end{quote}
By using the {\it IRAS}\ maps as reprocessed by SFD98, these predictions
will be provided at a resolution of $6{\farcm1}$.

It is critical that CMBR experimentalists compare their observations
with valid models for the Galactic dust emission.  A ``template approach''
is often carelessly
used to compare observations with alleged contaminants, with
the correlation amplitude indicating the level of contamination.
For example, radio maps ({\it e.g.}, the Haslam {\it et al.}\ map at $408 {\rm ~MHz}$)
are used as templates for synchrotron emission at much higher
frequencies, in spite of the well known spectral index variation.
$21{\rm ~cm}$ maps ({\it e.g.}, Leiden-Dwingeloo)
are often used as templates for microwave dust emission.
This can be a terrible template, as it ignores variations in the
dust/gas ratio and the well-measured variations in dust temperature.
This point is demonstrated in Figure \ref{fig_leitch}, where we
have correlated the Leitch {\it et al.}\ (1997) 14.5 GHz data with possible templates
for the dust emission.  The microwave data shows very little correlation
with a $21{\rm ~cm}$ template, or with the even more simple-minded (but still
occasionally used) $\csc|b|$ model for Galactic dust.
One might incorrectly conclude from this
that there is little dust contamination in this CMBR data.
In fact, when we correlate the Leitch {\it et al.}\ data with a proper prediction
at $14.5 {\rm ~GHz}$, there is evidence for significant contamination.  A
naive comparison with $100\micron$ IRAS flux also indicates a strong
correlation with dust - but with the wrong amplitude.  We find that
the dust on the $\delta=88^\circ$ ring used by Leitch {\it et al.}\ is cold
enough to lower the $100\micron$ emission per dust column by a factor
of 2.75 relative to the average high-latitude ratio.  Therefore, the
amount of 14.5 GHz emission per dust column is overstated by a factor
of $\sim3$ in comparisons such as the famous Figure 11 in Draine \&
Lazarian (1998).  We hope that our dust spectrum model will allow both a
realistic prediction of the thermal (vibrational) dust emission, and
also facilitate a meaningful comparison between the dust and the
mysterious excess 15-90 GHz emission component that correlates so well
with it.

\acknowledgments

The {\it COBE}\ datasets were developed by the NASA Goddard Space Flight
Center under the guidance of the {\it COBE}\ Science Working Group and were
provided by the NSSDC.  Dale Fixsen has been very helpful in understanding
the calibration uncertainties in the {\it COBE}/FIRAS data.


Divergence of the quasiparticle lifetime with doping and evidence for pre-formed pairs below T* in YBa_2Cu3$O_{7-δ}

Abstract: We report new time-resolved data of quasiparticle relaxation and Cooper pair recombination dynamics in YBa_2Cu_3O_{7-\delta}, measured as a function of temperature and doping \delta using femtosecond optical spectroscopy. The data show the existence of a normal state pseudo-gap for in-plane charge excitations below T* and an unusual divergence of the quasiparticle relaxation time in the superconducting state \tau_s with \delta towards optimum doping. In the underdoped state, no change in the amplitude of the induced transmission (which is proportional to the DOS at E_F), or relaxation time \tau_s is observed at T_c. From the sum rule, both observations signify that no gap opens at T_c in underdoped YBCO for \delta >0.15. T_c in this case only signifies the onset of phase coherence. The presented data thus suggest pair formation with an associated redistribution of the DOS starting at T* and the establishment of phase coherence at T_c consistent with Bose-Einstein condensation. In the optimally doped material, \delta ~ 0.1, on the other hand, both a divergence in lifetime and a change of the DOS occur at T_c, signifying the opening of a gap and the occurence of pairing takes place simultaneously.

\section{Introduction}

The low-energy electronic excitation spectrum in high-temperature
superconducting cuprates has been controversial ever since the first
experimental data were reported on the subject more than 10 years ago. The
standard measurement techniques used to obtain experimental data have been
mainly infrared reflectivity, transmission and ellipsometry together with
electronic Raman scattering. Although there was significant agreement
regarding the raw data between different groups\cite{Ginsberg}, controversy
arose when interpretation of the spectra was attempted. The single particle
spectrum on the other hand was very successfully investigated by
angle-resolved photoemission (ARPES)\cite{ARPES dispersion}. However in this
case, just as for optical spectroscopy, the complexity of the observed
multi-component spectral features necessitates the use of a model in
interpreting the data. Furthermore in all time-integrated spectroscopies the
issue of whether the spectrum shows homogeneous or inhomogeneous linewidth
cannot be easily circumvented, so particle lifetimes determined by these
techniques are necessarily ambiguous.

In this paper we describe an example of the application of time-resolved
optical spectroscopy to the direct investigation of low-energy electronic
excitation dynamics. The method by default directly gives quasiparticle
lifetimes in the case of high-temperature superconductor materials. From
such measurements as a function of temperature and doping $\delta $ in YBa$%
_{2}$Cu$_{3}$O$_{1-\delta }$ we are able to infer the occurrence of
Bose-Einstein condensation of quasiparticles at $T_{c}$ in the underdoped
state and a cross-over to BCS-like state near optimum doping.

\section{Experiments}

The success of the time-resolved spectroscopy method relies on two factors.
Firstly the technological developments of femtosecond lasers and related
technology, with the recent development of high-frequency lock-in
amplifiers. Secondly - a particularly important generic feature of the
cuprate materials - the existence of a charge-transfer resonance in the
wavelength region of 800 nm easily accessible by these lasers. The
assignment of this resonance by use of laser wavelengths ranging from 1500
to 350 nm has been particularly important for the understanding the
time-resolved measurements\cite{Stevens}. In previous work\cite{Han}$^{,}$%
\cite{tr refs}, the spectroscopy off-resonance (usually at 2 eV) showed
features which were substantially more difficult to interpret.

The laser used in the present time-resolved experiment was a Ti:Sapphire
mode-locked laser giving 150 fs pulses at 800 nm with a repetition rate of
88 MHz. The pump pulse train average power was typically between 10 and 120
mW, while the probe pulse train was typically 0.2 mW or less. The two beams
were focussed onto the sample mounted onto a copper block in an Oxford
Instruments Microstat. The pump beam was modulated at 200 kHz and the probe
detected via an amplified photodiode and a high-frequency digital lock-in
amplifier (EG\&G 7260).

For the understanding of the present experiments, it is important to
establish that the optical transitions involve excitations from the ground
state (of predominantly O character) near $E_{F}$ to unoccupied states (of
predominantly Cu orbitals) 1.5 eV above in energy. In such a resonance case,
the changes in the ground state DOS occurring with temperature, particularly
at $T_{c}$ and $T^{*}$ can thus be measured by pump-probe spectroscopy. The
evidence for the assignment given above is summarized below. From early
X-ray work of Bianconi and others\cite{Bianconi88} it emerged that the
charge carriers (holes) are located mainly on the O ions, while the Cu $d$
(upper Hubbard) band is approximately 1.5-1.8 eV\ higher in energy
(depending on the material and level of doping). Experiments on YBa$_{2}$Cu$%
_{3}$O$_{x}$ for example have shown that the charge transfer (CT) transition
between O and Cu is strongly observed in optical conductivity, optical
ellipsometry\cite{Kircher LDA} and absorption spectra\cite{PC} as well as
photoconductivity\cite{PC} at 1.8 eV with a typical width of 0.1 eV. Upon
doping it splits into two bands approximately $\sim 0.1$ eV apart at the
insulator-to metal transition\cite{Kircher LDA}$^{,}$\cite{Demsar} and
eventually at optimum doping only a peak at 1.5 eV is clearly visible. A
feature at 1.5 eV is clearly observed also in large-shift electronic Raman
scattering\cite{Cooper}with an apparent splitting observed at $\delta $ $%
\sim $ 0.6. This probably indicates that the local symmetry for the CT\
transition is lower than orthorhombic and has no center of inversion,
otherwise the 1.5 eV feature would not be seen both in {\it gerade} (Raman)
and {\it ungerade} (infrared)\ transitions.

Further confirmation that the initial states for optical excitations at 1.5
eV cross $E_{F}$ comes from thermal differential reflectance (TDR)\cite
{Little} and time-resolved (TR) optical spectroscopies\cite{Stevens}. The
former measures the change in reflectivity which occurs at $T_{c}$ as a
result of the redistribution of the DOS\ and shows distinctly a double peak
at 1.5 eV similar to the one seen in the optical conductivity and Raman
spectra. TR spectroscopy shows the same feature at 1.5 eV in the optimally
doped YBa$_{2}$Cu$_{3}$O$_{7-\delta }$ - albeit with much lower resolution -
also appearing below $T_{c}$\cite{Stevens} and is similarly thought to be
caused by a redistribution of initial states for the optical transitions at $%
E_{F}$. In this case the 4 : 1 in-plane to out-of-plane polarization ratio
also confirms this assignment. Yet more independent confirmation that the
initial states for absorption are at $E_{F}$ come from ARPES\cite{ARPES
dispersion}\ in conjunction with theoretical calculations, while the effect
of doped holes on the 1.5 eV transition was nicely demonstrated by Matsuda
et al\cite{Matsuda}.

The schematic diagram for the resonant pump and probe processes which is
based on the assignment discussed above is shown in Figure 1. The carriers
are first excited by the pump pulse from occupied states at or below $E_{F}$%
. The second, probing step involves determining - as a function of time -
the resulting changes in the density of states at $E_{F}$ $N(0),$ caused by
the photoexcited quasiparticles by measuring the change in transmission $%
{\cal T}$ of the sample with a delayed weak laser probe pulse. In the
adiabatic approximation, the golden rule gives the {\em change }in optical
absorbance of the probe pulse to be proportional to the change in the DOS at
$E_{F}$ i.e. $\delta A$ $\propto \delta N(0)$ caused by the pump pulse.
Since - as was discussed above - this is in turn proportional to $N(0)$, the
time-resolved changes in optical transmission $\delta {\cal T}/{\cal T}%
\,(=-\delta {\cal A}/{\cal A})$ can directly probe temperature- and $\delta $%
- dependence of $N(0)$. The probe pulse was polarized in the $a-b$ plane to
probe predominantly CuO-plane excitations (the signal was found previously
to be independent of pump polarization\cite{Stevens}).

\section{Experimental results}

The induced transmission $\Delta {\cal T}/{\cal T}$ following the pump pulse
typically shows a relatively fast relaxation of $\tau \sim $ 0.5 - 3 ps,
followed by a distinct long-lived component with a decay time 10 ns or more%
\cite{Stevens,Thomas}, which is most evident in optimally doped YBCO with $%
x=6.94$ and also for $x\sim 6.75.$ This long lived component is attributed
to quasiparticle relaxation in localized states and was discussed in detail
by Stevens et al\cite{Stevens}. In this paper we will only discuss the fast
component of the relaxation. The amplitude of this component was measured
{\em well below }$T_{c}$ as a function of doping $x$ in YBCO over a wide
range of doping. The results shown in Figure 2 indicate that the integrated
amplitude - which is proportional to the total number of particles excited, $%
n=$ $\int (dn/dt)dt\,$ $\propto $ $\int_{-\infty }^{5\tau }\delta {\cal T}/%
{\cal T\,}dt\,$- increases almost linearly with doping, starting at the
insulator-to-metal transition at $x=6.4.$ (Near optimum doping, the
non-integrated amplitude of the signal actually decreases, but this is not
attributed to a drop in the DOS, but rather to the fact that the relaxation
time $\tau $ starts to increase dramatically for $x$ $\rightarrow 6.9.)$

An important feature of the data is that the amplitude is strongly
temperature dependent (Figure 3). At low temperatures ($T<T_{c}$) the
amplitude is nearly temperature independent. With increasing temperature the
amplitude starts to drop and eventually vanishes, such that at room
temperature the signal is usually not visible, or very small. The
temperature $T^{*}$ at which the amplitude $\delta {\cal T}/{\cal T}$ drops
to zero - indicating a change (drop) in the DOS\ at $E_{F}$ - increases with
decreasing carrier concentration in the underdoped phase in close agreement
with the ''pseudogap'' temperatures for YBCO determined from other
experiments\cite{NMR}. Importantly, near optimum doping the amplitude
vanishes near $T_{c}$, which has already been noted previously\cite{Stevens}%
. In underdoped samples on the other hand, the amplitude shows {\em no
change }at $T_{c},$ suggesting no changes in the DOS\ take place at $T_{c}$,
which as we shall discuss below is presented as evidence for Bose-Einstein
condensation.

As already mentioned briefly above, the quasiparticle lifetime $\tau _{s}$
in the superconducting state - determined by fitting a single exponential to
the time resolved trace - shows a divergence with increasing carrier
concentration $1-\delta ,$ in the region close to optimum doping as shown in
Figure 4. In contrast, almost no change in lifetime with $\delta $ is
observed above $T_{c}$, and is $\tau _{n}=0.5\pm 0.1$ ps at 100 K over the
whole range of doping. Importantly, the lifetimes below and above $T_{c}$ in
the underdoped state are typically the same below $x\sim 6.85.$ Approaching
optimum doping, this is no longer the case, and a clear divergence of $\tau $
is observed at $T_{c},$ behavior which is well known in BCS\ superconductors%
\cite{Rothwarf and Taylor} and is thought to be due to a quasiparticle
relaxation bottleneck as $T\rightarrow T_{c}$ from below (not a
Hebel-Slichter peak).

\section{Discussion}

The systematic investigation of the amplitude and relaxation time $\tau $ of
the photoinduced optical transmission through thin film samples shows rather
well the cross-over in behavior which occurs near optimum doping. In the
underdoped state, neither the amplitude (which is related to the DOS at $%
E_{F}$) nor the lifetime $\tau $ show any effect at $T_{c}$. Instead, both
show a gradual change with temperature, the amplitude vanishing at $T^{*}$,
while the lifetime showing only very gradual change up to $T^{*}$. The
simplest way to explain the behavior in the underdoped state is to assume
that no change in amplitude or relaxation time is observed at $T_{c}$
because no change in DOS (i.e. gap) occurs at $T_{c}$. Instead - in a
typical Bose-Einstein scenario - the pairs which are formed at $T^{*}$ only
acquire phase coherence at $T_{c}$.

The onset of a macroscopically coherent condensed state occurs when the
wavefunctions of adjacent pairs overlap sufficiently for phase coherence to
be established between them. We can estimate the temperature at which the
phase coherence is established by considering when the DeBroglie wavelength $%
\lambda $ becomes comparable to the superconducting coherence length $\xi .$
Thus $k_{B}T_{c}\simeq \hbar ^{2}/(m^{*}\xi ^{2}),$ which, using measured
values of $\xi =18$ A for YBaCuO\cite{Poole} and $m^{*}=3m_{e},$ gives $%
T_{c}=91$ K, which is close to the observed $T_{c}$-s in this material. The
density of carriers at the point of condensation is given by Einstein\cite
{Einstein} as $(N/V)^{2/3}=2.612^{-2/3}m^{*}k_{B}T_{c}/2\pi \hbar ^{2}$
which, using the same value of $m^{*}$ and $T_{c}$ as above, gives $%
n_{c}=N/V\sim 10^{21}cm^{-3}$ which is close to the value estimated from
Hall data\cite{Poole}.

The data in the near-optimally doped material with $x>6.85$ are distinctly
different to the undedoped state. Both the amplitude and the relaxation time
show a dramatic change at $T_{c},$ the latter exhibiting diverging behavior
as $T$ approaches $T_{c}$ similar to BCS superconductors like Al\cite
{Aluminium divergence}. The amplitude on the other hand shows a rapid drop
to zero very close to $T_{c}$, also with a BCS-like $T$-dependence,
suggesting that a gap opens simultaneously with pair formation (and phase
coherence) as in the BCS\ case.

To conclude, time-resolved optical spectroscopy is demonstrated to be a
powerful tool for the investigation of quasiparticle dynamics in HTS. The
occurrence of BE\ condensation in underdoped YBCO - as suggested by the
absence of any observable change in the quasiparticle relaxation time $\tau $
or density of states $N(0)$ at $T_{c}$ in our experiments - is consistent
with simple theoretical expectations. Obeyance of the sum rule implies that
pairs must form above $T_{c}$ (the observed changes in the DOS\ from $\Delta
{\cal T}/{\cal T}$ suggest that this occurs at $T^{*}$). In contrast, the
optimally doped and overdoped state appears to show a divergence in the
quasiparticle lifetime with doping and a BCS-like temperature dependence of
both the intensity and lifetime is also apparent.

\section{Figure Captions}

Figure 1. A schematic diagram of the pump and probe processes in resonance
with the 1.5 eV CT\ transition in YBaCuO. The carriers relax rapidly to a
quasi-steady state between the pump and the probe pulses.

Figure 2. The amplitude of the induced transmission (squares)\ increases
with doping, following $T_{c}$. The data for $T_{c}$ are from Conder et al%
\cite{Conder Tc}.

Figure 3. The temperature dependence of the amplitude of the induced
transmission for a number of different samples. The $T_{c}$-s are shown in
the plot. The temperature at which $\Delta T/T$ drops to zero is close to
the pseudogap temperature $T^{*}$.

Figure 4. The relaxation time measured as a function of doping in the normal
state at 100 K (top panel) and at 20 K (lower panel). A clear divergence of $%
\tau _{s}^{20K}$ is observed near optimum doping.

\section{Acknowledgments}

We wish to acknowledge V.V.Kabanov for useful discussions and T.Mertelj for
his comments regarding the present manuscript.

Superconductivity (cond-mat.supr-con); Strongly Correlated Electrons (cond-mat.str-el)