A Markerless Deep Learning-based 6 Degrees of Freedom PoseEstimation for with Mobile Robots using RGB Data

Abstract: Augmented Reality has been subject to various integration efforts within industries due to its ability to enhance human machine interaction and understanding. Neural networks have achieved remarkable results in areas of computer vision, which bear great potential to assist and facilitate an enhanced Augmented Reality experience. However, most neural networks are computationally intensive and demand huge processing power thus, are not suitable for deployment on Augmented Reality devices. In this work we propose a method to deploy state of the art neural networks for real time 3D object localization on augmented reality devices. As a result, we provide a more automated method of calibrating the AR devices with mobile robotic systems. To accelerate the calibration process and enhance user experience, we focus on fast 2D detection approaches which are extracting the 3D pose of the object fast and accurately by using only 2D input. The results are implemented into an Augmented Reality application for intuitive robot control and sensor data visualization. For the 6D annotation of 2D images, we developed an annotation tool, which is, to our knowledge, the first open source tool to be available. We achieve feasible results which are generally applicable to any AR device thus making this work promising for further research in combining high demanding neural networks with Internet of Things devices.

\section{INTRODUCTION}

The need of an environmental perception and understanding is essential for tasks such as autonomous driving, Augmented Reality (AR) and mobile robotics.
Mobile robots are one of the main systems to profit from the recent progress in computer vision research. Their popularity within industries has increased due to their flexibility and the variety of use cases they can operate in. Tasks such as provision of components, transportation, commissioning or the work in hazardous environments are increasingly being executed by such robots \cite{smartfactory}, \cite{mobilerob}. However, operation and understanding of mobile robots is still a privilege to experts \cite{mobilerob2} because they are more complex and thus harder to operate and understand.
On this account, Augmented Reality (AR) has gained much attention in research due to the high potential and ability to enhance efficiency in human robot collaboration and interaction which had been proved by various scientific publications \cite{hashimoto}.
AR has the potential to aid the user with help of spatial information and the combination with intuitive interaction technology, e.g. gestures or voice commands \cite{arsystems}. Our previous work focused on AR-based enhancements in user understanding for robotics. In \cite{lambrecht} we simplify robot programming with the help of visualizing spatial information and intuitive gesture commands. In  \cite{kaestner} we developed an AR-based application to control the robot with gestures and visualize its navigation data, like robot sensors, path planing information and environment maps. Other work used AR for enhanced visualization of robot data or for multi modal teleoperation \cite{ar1}, \cite{armultimodal}.
The key aspect to facilitate AR, is the initial calibration between AR device and the components within the environment. For our use case, this means that the AR device has to be spatially aligned with the mobile robot in order to visualize data properly. State of the art approaches are relying on marker detection which proves to be accurate but unhandy. Additionally, markers cannot be deployed everywhere especially not in complex dynamic environments for use cases like autonomous driving. Furthermore, occlusion could impact the performance drastically as was proven in our previous work \cite{kaestner2}. For these reasons, neural networks are increasingly being subject of research efforts.
Recently published work focused on the usage of 3D data like depth sensors for better accuracy. However, within AR applications, 3D data is still too unhandy to use because it require additional preprocessing steps, which slows down the overall application pipeline \cite{kaestner1}. Our previous work explored the possibilities of a 3D sensor integration into an AR head mounted device \cite{kaestner2}. Although we could show the feasibility and the localization was accurate, the processing time was very slow which drops user experience. RGB data on the other hand, dont require the complex preprocessing steps which reduces the computation time and enhance user experience. Furthermore, RGB data is most ubiquitous within AR devices like headsets or smartphones thus, our method can cover a wider range of devices. RGB data also contains color information which is helpful for classification scenarios. On this account, we
explore the possibilities of an integration of neural networks
that can localize the robots 6 Degrees of Freedom (DoF) pose solely based on 2D
data input. We compared two different state of the art neural networks and propose an distributed architecture to integrate those into our AR use case.
Our results can be adapted for every other AR system which opens many possibilities for further research.
The main contributions of this work are following:
\begin{itemize}
	\item Proposal of a distributed architecture to deploy neural networks (NNs) for markerless pose estimation on AR devices
	\item Evaluation of different neural network models in terms of feasibility for deployment within AR applications
	\item Development of an open source 6 DoF annotation tool for 2D data
\end{itemize}
The paper is structured as follows. Sec. II will give an overview of related work. Sec. III will present the conceptional design of our approach while sec. IV will describe the the training process and implementation. Sec. V will demonstrate the results with a discussion in Sec. VII. Finally, Sec. VIII will give a conclusion.

\section{RELATED WORK}
Due to the high potential of AR, it has been considered for integration into various industrial use cases.The essential step to facilitate AR is the calibration between the AR device and environmental components e.g. machines or robots.
\subsection{AR calibration methods}
Manual calibration using external tools or special equipment is the most straightforward method and requires the user to manually align 2D objects to objects in the real environment which position is known to the calibration system.
 Azuma and Bishop \cite{azuma1994improving} proposed an AR camera pose estimation by manually aligning fiducials like virtual squares to specific locations in the environment.
 Similar work from Oishi et al. \cite{oishi1996methods} require the user to align calibration patterns to different physical objects which positions are known beforehand. However, manual calibration bear disadvantages in a continuously automated environment due to the necessity to have a human specialist to do the calibration. The external tools themselves can be expensive or are case specific.\\ Recent research focus on automated calibration methods rather than manual ones. Most ubiquitous are marker based approaches. Libraries and toolkits like \textit{Aruco} or \textit{Vuforia} make it possible to deploy self created fiducial markers for pose estimation. The marker is detected and tracked using computer vision and image processing. Especially when working with static robots, marker based approaches are widely used due to the simple setup and competitive accuracy. Several work including Aoki et al. \cite{aoki2018self} or \cite{chu2018helping} et al. relied on a marker based calibration between robot and AR device.
However, Baratoff et al. \cite{baratoff2002interactive} stated, that the instrumentation of the real environment, by placing additional tools like markers, sensors or cameras is a main bottleneck for complex use cases especially in navigation that are working with dynamically changing environments. Furthermore, deploying additional instruments require more time to setup and are not able to react to changes within the environment. Especially for industrial scenarios, this is the main concern \cite{baratoff2002interactive}. Another factor that drops performance in dynamically changing environments is occlusion of markers which was noted in our previous work \cite{kaestner2}. \\
On this account, marker-less approaches have became a major focus \cite{yan2017application}. This includes model-based approaches which use foreknown 3D models to localize objects and generate the pose estimation for the AR device. This is done using computer vision and frame processing to automatically match the known 3D models on the object from the incoming frame. Subsequently the pose is estimated. Works by Bleser et al. \cite{bleser2006online} were able to establish a competitive approach using CAD models and achieving high accuracy.
The main bottleneck is, that these approaches are very resource intensive due to the processing of each frame.
In addition, these methods are highly dependent on the CAD models, which are not always available. Furthermore, the approach becomes less flexible, which is crucial in dynamic environments for tasks such as autonomous navigation of mobile robots. \\
 With the recent advances in machine learning algorithms, especially deep learning, and the higher computational capabilities of computers, networks can be trained to recognize 3D objects.
An overview of current research is provided by Yan et al. \cite{yan2017application}.
Recent publications by \cite{qi2019deep}, \cite{shi2019pointrcnn} have focused on learning directly from 3D data. However, 3D data often require preprocessing steps to be reliably deployable within an AR device. Furthermore it do  not contain color chanels which could be important for classification. On that account, we focus on 2D approaches to achieve real time performance.
Alhaija et al. \cite{alhaija2018augmented} worked with neural networks for object detection and segmentation for AR-supported navigation. The research closest to our method is proposed by Garon and Lalonde \cite{garon2017deep} which used convolutional neural networks (CNNs) to detect 3D objects. We extend this approach principle by applying more recent neural networks based on \textit{YOLOv3} and integrate them into an AR robotic application. Therefore we choose two state of the art neural networks for our use case, SSPE and BetaPose, which are described in the following.
\subsubsection{SSPE}
\textit{SSPE} is heavily inspired by \textit{YOLO} . Thus, its a fast method for 3D pose estimation. \textit{YOLO} was modified to deliver 8 corners of a 3D bounding box together with its center instead of 4 points, like in the original implementation. After the 2D projections are estimated, the Perspective-n-Point (PnP) algorithm is used to calculate the rotation and translation of the object. \textit{SSPE} can manage up to 50 FPS and does not need any refinement. The accuracy on the \textit{LINEMOD} dataset was 55.95 percent which is competitive for an approach without refinement steps.
\subsubsection{Betapose}
Figure 4.2 shows the \textit{Betaposes} pipeline. It consists of 2 NNs. The first one is
YOLO which detects where the object is on the image. If \textit{YOLO} detects
the object, it cuts it out and gives the cutout image to the Key Point Detector
(KPD) network which marks the 2D position of the Key Points (KPs) it was trained to localize.
Finally, a PnP algorithm is used to recover the 3D pose according to the 2D-3D
relationship of the KPs. Just like \textit{SSPE}, \textit{Betapose} does not
rely on a time-consuming post-processing procedure. The method
can achieve competitive accuracy without any refinement
after pose prediction. \textit{Betapose} could produce up to 25 FPS.

\subsection{Distributed Architecture}
Combining neural networks with AR require an efficient architecture to ensure proper visualization for high user experience. That is because most of currently available AR devices are computationally restricted thus, an outsourcing of demanding services and functions has to be taken into consideration.
Distributed systems for AR devices are increasingly being researched by
the community. Ren et al. \cite{ren2019edge} and Schneider et al. \cite{schneider2017augmented} both presented different
edge computing architectures for AR use cases. They both put emphasis
on the latency and power consumption connected with the distributed nature
of the system. However, the most extensive and well documented research was presented by Ha et al. \cite{ha2014towards}, which is a system for wearable cognitive assistance on the network edge.
It consists of 2 hardware devices - the wearable AR-enabled device (client) and the server also called \textit{Cloudlet}. A \textit{Cloudlet} is a computer located on the network edge. It can also be described as a "data center in a box" which is one hop away from the AR-enabled device.
Within that server the user can deploy own services which are also called proxies. These proxies can cover a wide range of application ranging from face recognition, motion classifiers or object detectors.
The client is sending different sensory data that it wants to be evaluated to the server. The implemented proxies will evaluate them and returns assimilated data back to the client. One of the advantages of Gabriel is that it is easily extensible with new and multiple services/proxies. It offers tight latency bounds on computationally heavy calculations. Furthermore, it also takes into consideration the limited battery and computational capacity of
the AR-enabled devices, which is why it is most suitable for our use case.

\section{CONCEPTUAL DESIGN}
In this chapter, we present the conceptual design of our work. The modules and approaches used are based on the research done in the related work section.

\subsection{AR Device Calibration}
To facilitate AR, the AR device has to be aligned with its environment. Our method assists in robotic environment hence the AR device has to be properly aligned with the robot entities.
For our use case, both entities, AR device and robot are fully mobile which means that a continuous localization of the robot has to be ensured.
For the AR application first developed in our previous work \cite{kaestner}, we proposed a two-stage calibration between AR headset and robot. First, the \textit{Hololens} is calibrated with the robot using \textit{Aruco} markers. Second, the spatial anchor functionality of the \textit{Hololens} was used for calibration of the robot with the robot map by placing an spatial anchor at the position of the detected marker. The spatial anchor capability refers to the internal SLAM of the \textit{Hololens} to scan the room and memorize its location within the room.
The robot map is the general reference map from which all sensor data and information are displayed. Thus, calibration between the AR device and the robot internal map ($T_{AR-Map}$) is the overall goal to visualize data properly within the AR headset. The robot position ($T_{Robot-Map}$) is always known by the map through the robots \textit{SLAM} packages (e.g. \textit{Adaptive Monte Carlo}). Hence, to achieve a transformation between AR device and ROS map, we have to achieve a transformation between AR device and robot ($T_{AR-Robot}$).
This is shown in equation \ref{eq1}.

\begin{align}
T_{AR-Map} = T_{AR-Robot} \cdot T_{Robot-Map}
\label{eq1}
\end{align}

With this work, the transformation between AR device and robot $T_{AR-Robot}$ is generated with a neural network-based robot pose estimation thus replacing the marker-based approach from our previous work. Based on that localization, the spatial anchor capability can be set and continuously track the robots position. This concept is illustrated in Fig. \ref{conceptcoord}.

\begin{figure}[!h]
	\centering
	\includegraphics[width=3.3in]{newconc1}
	\caption{Conception of Coordinate System Alignment}
	\label{conceptcoord}
\end{figure}

\subsection{Neural Network Selection}
Our AR calibration has to provide a fast localization ob the robot results in order to speed up the spatial alignment process and ensure an enhanced user experience. The neural network should therefore perform fast and accurate results at the same time. Our previous \cite{kaestner2} work explored the performance of 3D data and although we showed promising results and the overall feasibility, the approach required up to 40 seconds of preprocessing time which drops the user experience tremendously. RGB images have the potential to perform faster and are more established with countless state of the art approaches achieving competitive results. Thus, our research will only consider these kind of neural networks that takes RGB data as input. We are inspired by the release of state of the art neural networks working only with 2D information to obtain the 3D position in a fast manner by using single shot object detectors like \textit{YOLO} \cite{redmon2016you}.  Hence, we choose \textit{Betapose} by Zhao et al. \cite{zhao2018estimating}  and \textit{SSPE} by Tekin et al.  \cite{tekin2018real} for our use case as they provide fast and accurate 6 DoF pose estimation using only RGB data. As input, we will take the AR device video-stream for training and evaluation of the neural networks.

\subsection{Distributed Architecture}
Due to the limited hardware capabilities of the \textit{Microsoft HoloLens}, we
propose a distributed application with two main components - the client (the AR-enabled
device) and the server (a more powerful computer located on the network
edge near the AR-enabled device). The computationally intensive neural network operations have to be outsourced to the server site, whereas visualizations of the output are computed on client site. For our use case we build 2 separate services/proxies using the \textit{Gabriel} framework which are described in the implementation chapter in more detail.

\section{IMPLEMENTATION}
We propose a distributed architecture using the open source framework \textit{Gabriel} which allows the user to build servers to run computationally intensive operations like neural networks on it and deploy the results in edge devices. These servers are also called proxies. The overall workflow is depicted in Fig. \ref{im2}. We implemented 2 proxies for the 2 different neural networks.  The implementation for \textit{SSPE} was based on the open source \textit{Github} repository provided by the authors. For our use case, we modified the evaluation code so that it would only take in a single image to evaluate and not a whole
dataset. Additionally to the estimated 3D bounding box projection this proxy
also projects the output of the \textit{SSPE} NN - that is the centroid and the 8 estimated
corners of the bounding box.
The implementation of \textit{Betapose} was more complex compared to \textit{SSPE} since \textit{Betapose} contains two NNs one
feeding it’s result to the next one. The system had to be parallelized for maximum
efficiency by using 3 threads - one for the main program
accepting and responding to HTTP requests, one for the \textit{YOLO} NN and
one for the KPD NN. Additionally to the estimated 3D bounding box projection
our implemented proxy also projects the output of the \textit{YOLO} NN and the KPD NN onto the
\textit{Cloudlet} AR device for enhanced understanding. For the \textit{YOLO} NN that is the 2D bounding box around the
object. For the KPD NN that is the 2D position of each KP.
The two proxies are can be run separately as well as in parallel e.g. for performance comparison.

\subsection*{Hardware Setup}
The hardware setup contains a mobile robot, a powerful Linux computer with Tesla K80 GPU acting as the server and a head mounted AR device - the \textit{Microsoft Hololens}. We are working with a \textit{Kuka Mobile Youbot} running ROS \textit{Hydro} on a \textit{Linux} version 12.04 distribution. As a development environment for the \textit{Hololens}, we use a windows notebook with \textit{Unity3D 2019} installed on which the application was developed. The videostream extraction is done in \textit{CSharp} using the \textit{Mediastream API} provided by \textit{Microsoft} using \textit{Visual Studio 2019} as development environment.
All entities are connected to the same network and communicate a \textit{WLAN} connection.

\begin{figure}[h!]
	\centering
	\includegraphics[width=3.4in]{impl3}
	\caption{Overall workflow of the application}
	\label{im2}
\end{figure}

\subsection*{Dataset generation}
For training the neural networks, we created a dataset consisting of real images as well as artificially created ones, to ensure a large dataset for high performance. The process of dataset generation is illustrated in Fig. \ref{data3}. To generate the artificial dataset, we used the game engine \textit{Unreal Engine} and a 3D robot model. We used 18 pre existing environments and placed the robot at different angles as depicted in Fig. \ref{data3} to ensure variety. The annotation of the artificial data was done with a plugin tool from \textit{Unreal Engine 4} called \textit{NDDS} which annotates the 6D pose of the object within the 2D images.

\begin{figure}[h!]
	\centering
	\includegraphics[width=3.4in]{dat5}
	\caption{Real and artificial dataset generation pipeline}
	\label{data3}
\end{figure}
The real dataset was captured with an \textit{Intel Real Sense} camera as well as with the \textit{Microsoft Hololens} camera stream.
Additionally, we used data augmentation like flipping, rotation and scaling to enlarge the real dataset. For the annotation of the real data, we developed a new tool \footnote[2]{http://annotate.photo} which required no installation and was as easy to use as possible. This tool can annotate 2D images with the 6 DoF pose using a 3D model.
The user can import a 3D model as well as the 2D dataset into the web application. Afterwards, the user has to define a minimum of 3 points and set the camera intrinsics. Our tool has a build in PnP algorithm to calculate the 6D pose out of these information. Additionally, the pose can be manually be refined. The annotated dataset is exported as a JSON file which is containing the annotations for the whole dataset. The tool was used, to annotate the real data captured with the cameras.
In total, our dataset consists of \textbf{13,069} images with 6994 artificially created images and 6075 real images. Originally, we collected 1223 real images which we augmented using rotation, flipping, scaling and changing the contrast to simulate bad lightning conditions.
Fig. \ref{model2} depicts the used 3D model of the robot as well as the real robot. It must be noted, that both are not exactly equal and differ in small details. E.g. the arm of the model do not have claws and the overall appearance looks more static compared to the real robot. This is why we considered both, artificially created and real data for our training.

 \begin{figure}[h!]
	\centering
	\includegraphics[width=3.4in]{res2}
	\caption{Results from the neural networks - Near. \textit{Left, from SSPE. Right, from Betapose. The user is 0.5 - 1m away from the robot. Both NN can detect the robot and its 6D pose accurately.}}
	\label{res2}
\end{figure}

\subsection{Training process}
For the training a GPU server with \textit{Tesla K80}  with 12 GB GDDR5 memory, 4vCPUs and 15GB RAM was used.
The network was trained for 3700 epochs (where a epoch means one pass through the whole
data set), using batches of size 8 and the Adam optimizer for minimizing the loss function.
During the second phase, the KPD neural network is retrained using the
weights produced by the first phase and using Pose-guided Proposal Generator
(PGPG) (also called DPG in the \textit{GitHub} repository of \textit{Betapose}) during training.
This PGPG is done to deal with the bias in bounding box detection which would
lead to an error in keypoint localization. It basically means that the dataset is
augmented. Both phases took around 24 hours to complete in total
Training the first phase took around 20 iterations and
around 1 hour to complete. Training the second phase was slower and took
around 135 iterations and around 20 hours to reach peak accuracy.

\section{Evaluation}
This chapter presents the results of the implemented approaches as well as a discussion of these.
Fig. \ref{res1} and Fig. \ref{res2} illustrates the results of \textit{Betapose} and \textit{SSPE}. For better understanding, we visualized the output of several stages of the approaches. For Betapose, we visualized the \textit{YOLO} output, the NN output and the bounding box after PnP is done. Furthermore, the KPs are visualized. For \textit{SSPE} we visualized NN output and BB after PnP is done. It can be seen that both methods providing accurate 3D localization of the robot. This is achieved in real time with the video stream of our AR device. Overall \textit{SSPE} is faster achieving 170ms to evaluate the image compared to 500ms for \textit{Betapose}.
\begin{figure}[]
	\centering
	\includegraphics[width=3.4in]{res11}
	\caption{Results from neural networks - Far. \textit{Left, from SSPE. Right, from Beta Pose. In the left picture, the user is 10 meters away from the object. While Betapose can only detect the object, SSPE can still localize its 6D pose.}}
	\label{res1}
\end{figure}
We compared the application with results from our previous work \cite{kaestner2} where we considered a 3D object localization based solely on depth sensor data (point-clouds) as well as our very first approach - the marker detection using \textit{Vuforia} markers with a marker size of 28x28 cm. The comparison is listed in Table \ref{tt}.
All three methods are compared in terms of accuracy, computation time and the maximum distance, where an accuracy of 50 can be still achieved.

\begin{table}[h!]
	\caption{Comparison of implemented approaches}
	\begin{tabular}{lllll}
		\hline
		Metric   & VoteNet(3D) & BetaPose & SSPE   & Marker   \\ \hline
		Accuracy & 95,5        & 93,2         & 92.74      & 91,67    \\
		Computing Time    & 6.2s        & 0.5s         & 0.17s      & 0.72s    \\
		Distance & $\sim$4m    & $\sim$8m    & $\sim$10 m & $\sim$2m \\ \hline
	\end{tabular}

	\label{tt}

\end{table}

 The accuracy metric is also called ADD metric in many papers and is an 3D intersection over union. We compared the predictions of the neural networks with our annotated ground truth. The computation time includes all steps - data acquisition, the transmission to the neural network server, the processing and visualization of results on the AR device. The distance was calculated by exploring the space where the accuracy of the approach reach a minimum of 50. The test were done on the \textit{Microsoft Hololens}. All values are average means of 20 different measurements each. It can be observed that the 3D method is the most accurate with 95.5$\%$ in accuracy whereas our recently implemented approach with \textit{SSPE} is by far the fastest approach taking only 0.27s to overall localize the object and display the observings within the AR application. This is taking into consideration the time to send the video-stream from the AR device to the server which takes a total of 0.1s with our proxy implementation. The slowest approach is unsurprisingly the 3D method due to the huge preprocessing steps needed. The extraction of 3D point-clouds from the AR device and processing took 3 seconds while transmitting the data to the server took another 2-3 seconds which results in a long time. Our new neural network-based approach is even faster than the marker-based approach which takes on average 1.32 seconds to detect the robot. This is taking into consideration, that at some positions where the marker is occluded, this approach performed worse which decreased the overall performance even though this approach performed fast at positions where the marker is clearly visible. Another important aspect is that the marker-based approach performed bad when the user was far away and was only working well for a distance up to 2m away from the marker. Whereas the 2D based approach could still provide accurate results with up to 8m as is seen in Fig. \ref{res1}.

\section{CONCLUSION}
We proposed a pipeline to deploy state of the art neural networks for robot pose estimation on capacity limited AR device - the \textit{Microsoft Hololens}. Therefore we made use of a distributed approach with an efficient client server communication. We showed the feasibility of our distributed approach and could replace the classical marker detection approach thus making an AR integration more intuitive and less tedious. We compared two different state of the art neural networks and concluded that \textit{SSPE} is the most suitable network for real time deployment. we obtained remarkable results and compared them with the results of our previous work. Our neural network based object localization outperformed the previously used marker-based approach both in terms of accuracy as well as speed. The 3D based point cloud approach was outperformed in terms of speed. Even though its less accurate then 3D approach, the 3D approach still takes too much time to be considered as competitive for application in industries. The results of this papers are promising for further research in the area of combining neural networks into IoT devices like AR headsets. In future, sensor fusion approaches should be considered to obtain more accurate results.
Furthermore we developed an annotation tool for 6D pose labeling of 2D images which is to our knowledge the first open source tool available online.

\section{Appendix}
All the code and demo materials are publicly available on \textit{Github} - https://github.com/lee4138/6d-pose-estimation-with-ml-in-ar. The annotation tool is online under the domain http://annotate.photo.
\addtolength{\textheight}{-2cm}

\bibliographystyle{IEEEtran}

Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG); Robotics (cs.RO)


Spherical agglomeration of superconducting and normal microparticles with and without applied electric field

Abstract: It was reported by R. Tao and coworkers that in the presence of a strong electric field superconducting microparticles assemble into balls of macroscopic dimensions. Such a finding has potentially important implications for the understanding of the fundamental physics of superconductors. However, we report here the results of experimental studies showing that (i) ball formation also occurs in the absence of an applied electric field, (ii) the phenomenon also occurs at temperatures above the superconducting transition temperature, and (iii) it can also occur for non-superconducting materials. Possible origins of the phenomenon are discussed.

\section{introduction}
It has been reported by R. Tao and coworkers that superconducting microparticles in a strong electric field assemble
into compact balls  of macroscopic dimensions (diameter fraction of a mm) \cite{tao,tao2,tao3,tao4}.  These balls  are robust and survive collisions with the electrodes at high speeds.
Tao et al attributed the ball formation  to a new kind of ``surface tension'' of superconducting
particles in the presence of a strong electric field \cite{tao}. They found  that this phenomenon occurs for both high $T_c$ cuprate superconductors\cite{tao}
and for low $T_c$ conventional superconductors\cite{tao2,tao4}. Furthermore they reported that the balls become fragile and break apart easily both when the
electric field is turned off and/or when the temperature is raised above the superconducting transition temperature.

If  valid, these findings would have enormous  implications for the understanding of the fundamental physics of superconductors. The conventional theory of superconductivity\cite{tinkham} predicts
that the response of superconductors to applied electrostatic fields is the same as that of normal metals\cite{bs}: electric fields should be  screened within a distance on the order of Angstroms
(Thomas-Fermi length) of the surface, and thus would not be expected\cite{tao}  to give rise to phenomena  such as the spherical aggregation
reported by Tao. One of us has proposed\cite{taoeffect}  that the phenomenon reported by Tao
 is evidence of the inadequacy of the conventional theory of superconductivity\cite{bcsvalidity}, and is
a consequence of the same physics that leads to the Meissner effect within the unconventional theory of
``hole superconductivity''\cite{holesc}, namely expulsion of negative charge from the interior of the superconductor to the surface. Quite generally, if this aggregation indeed took place only
in the superconducting state and only under a strong applied electric field, it would call for an explanation beyond the generally accepted physics of superconductors.

 However, we report in this paper that the  spherical aggregation  is a more widespread phenomenon than previously reported. We find that
  it can occur both in the presence and in the absence of an electric field, and both  in the  superconducting state as well as above the
  superconducting transition temperature, and also in non-superconducting materials. These findings undermine the claims that the Tao experiments cast doubt on the conventional theory of
  superconductivity\cite{bcsvalidity} and are evidence in favor of an alternative theory\cite{taoeffect}. While the phenomenon appears to be strongest in materials that are high temperature
  superconductors, we have not found clear evidence that it is stronger in the superconducting versus the nonsuperconducting state.

 \section{agglomeration in an electric field}

 We started by attempting to reproduce Tao's experimental results. In addition to Tao's papers, one unpublished report exists that independently supported some of
  Tao's findings\cite{taoconfirm}.

 We used
$BSCCO$ and  $YBa_2Cu_3O_{7-\delta}$   disks  provided by Colorado Superconductor. The $BSCCO$ disks were reportedly a mixture of the 2223 and the 2212 phases, with critical temperature above $100 K$. Both types of disks exhibited a strong Meissner effect at liquid nitrogen
temperatures.
We ground the disks with a Dremel (952 aluminum oxide grinding stone) and sorted particles by sizes using sieves. Figure 1 shows typical shapes of particles obtained for various sizes.
We used only the resulting microparticles that
passed through a $32 \mu m$ sieve.   In another set of experiments, we used $Bi_2Sr_2CaCu_2O_{8+x}$,  $x=0.15-0.20)$
powder provided by
Sigma-Aldrich, hereafter termed `$\sigma-$powder', with nominal grain size $\leq 5 \mu m$.

    \begin{figure}
 \resizebox{7.5cm}{!}{\includegraphics[width=8cm]{figure1n.eps}}
 \caption {Typical microparticles of various sizes obtained by grinding superconducting disks and passing through sieves. The range of diameters of the four panels are $53\mu m$ to $75\mu m$,
 $32\mu m$ to $53\mu m$, $25\mu m$ to $32\mu m$,  and less than $32 \mu m$ for the upper left, upper right, lower left and lower right panels.
 Note the irregular shapes of the particles.}
 \label{chain}
 \end{figure}

 We constructed a cell measuring $15mm \times 70mm$ horizontally and $13mm$ vertically   from plastic plates of thickness $1 mm$, and attached thin ($<1mm$) metal plates  either on the inside or outside walls. This formed a parallel plate capacitor with electrodes separated by $12mm$ to $16mm$ depending on the configuration. We used both $Al$ and stainless steel metal plates and observed the same behavior,
 indicating that magnetism was not playing a role. The metal plates  were connected to a DC voltage source that could deliver up to 20kV, i.e. maximum electric field
 inside the capacitor approximately $1700 V/mm$. This is well above the `lower critical field'   $E_{c1}$ required for ball formation reported by Tao of order $500-800 V/mm$\cite{tao,tao2,tao3,tao4}.

  We poured liquid nitrogen into the capacitor,
added $BSCCO$ or $YBCO$ powder (typically between $60 mg$ and $150 mg$) and observed the behavior when the voltage source was connected to the capacitor plates by
recording  with a high speed camera that yielded 1000 frames per second.

When we put both metal plates on the inside of the plastic plates we got interesting action starting at a few thousand volts, as described below. When both metal plates were outside
of the plastic plates to form insulating electrodes we found no action whatsoever, even for voltages up to 20kV. When one of the metal plates was on the inside and one on the outside we found
some very limited action. We conclude that the effect is $not$ driven by a large electric field, as postulated by Tao\cite{tao,tao2,tao3,tao4},
since it  requires $charge$ being able to
circulate from the conducting electrodes to the superconducting microparticles.

   \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure2n.eps}}
 \caption { Balls bouncing up and down between electrodes, frames shown are consecutive separated by 10 ms. The vertical size of the figure corresponds
 to 1cm in real life. Each ball is composed of several thousand micrograins}
 \label{chain}
 \end{figure}

The action we found in the capacitor can be summarized as follows. On increasing the voltage, at several thousand volts, powder starts to move. Sometimes it suddenly flies to the electrodes. Then, balls form and start bouncing between the electrodes, as described by Tao. In addition, as described by Tao, in all runs clinging of powder to the electrodes occurs. The voltage where these processes occurs is variable, depending for example on the amount of powder and its distribution between the electrodes, it was typically in the range between 6kV and 14kV.  While Tao usually found only one
ball in his $4mm \times 5mm$ chamber, in our setup we typically found a large number of balls of varying size.
Often, the balls alligned in a linear fashion and bounced back and forth against each other and against the electrodes.  Figure \ref{chain} shows as an
example a row of 8  balls in five snapshots separated by 10 frames each, i.e 10 ms (the camera records 1000 frames per second),
in a run where the potential difference between top and bottom electrodes was 9kV.
The distance between top and bottom electrodes (black horizontal lines) is 1 cm, so these balls range from approximately 0.45 mm to 0.70 mm in diameter.
Note that a ball of 0.5 mm diameter would be composed of at least 4,000   microparticles  in close proximity.
The balls
are bouncing up and down and there is no apparent deformation when they bounce.  Here they move between collisions at a speed of approximately $6 cm/s$,
 and  the acceleration involved in a collision is
of order  $10 m/s^2$. This linear arrangement was stable for approximately 2s, corresponding to each particle traveling the equivalent of 12 times
the distance between the electrodes and undergoing about 200 collisions with the neighboring particles. At the end of that period a ninth particle squeezed into the row and the same arrangement continued until shedding from the layer
coating the upper electrode occurred.

The fact that ball formation only occurred when the electrodes were conducting indicates that the phenomenon is not due to the presence of an electric field
as argued by Tao, rather it appears
to be $facilitated$ by the  presence of   free charge. We hypothesize two processes: in the first process,
microparticles pick up positive and negative charges from the electrodes, after which they are repelled from the electrodes and
return to the bulk of the chamber. Microparticles of opposite sign attract each other due to the long-range  Coulomb force. When they come in contact, they share their charge. One might
expect that they would then separate, the fact that they stay together indicates that   another force  sets in that keeps them together.
In the second process, ball formation occurs at the electrodes themselves, where a thick layer of micrograins accumulates, with micrograins in close proximity to each other;
occasionally, pieces of the layer detach themselves in the form of balls that return to the bulk of the chamber. Again, the balls are held together by a force that acts
between micrograins in close proximity.

  \section{agglomeration in the absence of an electric field}
   We next   tested the possibility of sphere formation in the absence of applied  electric field and electric charges,
   by gently shaking a container with
superconducting powder horizontally in liquid $N_2$.
    We poured 60 mg of $BSCCO$ or $YBCO$ powder in a container of diameter  $80mm$ containing approximately $5mm$ height of $N_2$, and
   kept the container cold by submerging it in a container of larger diameter with substantially higher levels of $N_2$. In all cases the micrograins in the powders
   had diameter smaller than $32 \mu m$. Indeed, it was found after fine-tuning the procedure that it was very easy to make balls of
the same sizes as produced in the capacitor. Figure \ref{nofield} shows one example. After formed, the balls retain their integrity upon strong  shaking of the container. When the agitation became sufficiently rapid and erratic the spheres would break apart back into powder. The balls could be reformed by returning to regular shaking.  In addition, we
verified that placing these balls inside the capacitor and applying a voltage between the plates leads to the same motion as described earlier for the case where
balls were created in the capacitor when voltage was applied.
 \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure3n.eps}}
 \caption {$BSCCO$ balls formed by shaking a container with micrograins, no electric field applied. The scale is the same
 as in Figure 1. The two images are separated by 30 ms and the motion is due to motion of the nitrogen caused
 by shaking.}
 \label{nofield}
 \end{figure}

   \section{temperature and material dependence}
   Having established that ball formation in high temperature superconductors is independent of the presence or absence of an applied electric field,
   we proceeded to investigate the temperature dependence of the effect and to test the behavior of various different materials. Tao et al had reported that both in high temperature superconductors,
   in low temperature elemental superconductors, and in magnesium diboride ($MgB_2$), balls don't form above $T_c$ and that balls that formed below
   $T_c$ become fragile and are easily destroyed as soon as the temperature rises above $T_c$.
   In the following, we will present results for various materials in liquid $N_2$ (temperature $77.4K$) and liquid $Ar$ (temperature $87.3K$), as well as in
   water at room temperature and in ethanol and methanol at temperatures in the range  $150-180 K$.

   \subsection{Materials used}
   We used the following materials:

   (1) $\sigma-$powder was pressed into a pellet of density approximately $3.5 g/cm^3$. It was heated to $850 ^oC$ and kept at that temperature in
   air for $40$ hours, then cooled slowly to room temperature in another 40 hours. After this process the pellet did not exhibit a Meissner effect either in $N_2$ or in $Ar$. In addition,
   its transition temperature  was measured resistively and found to have onset at $T=73^oK$, confirming the observed absence of Meissner effects. The pellet was then ground and the resulting powder passed through a $32 \mu m$ sieve. The resulting powder collected in a dry tube and gently shaken had
   density $1.6 g /cm^3$.

(2) A $YBCO$ disk from Colorado Superconductors was broken into 3 pieces. One piece of the disk  was left intact.
This piece exhibited a strong Meissner effect in $N_2$ and a weaker Meissner effect in $Ar$.
Another piece   was brought to   $800^oC$ for 90 minutes and then quenched to $77K$ in liquid $N_2$. According to the literature \cite{schuller,hariharan},
this procedure removes oxygen and leads to an underdoped sample, with $\delta \sim 0.65$ and $T_c$ less than $30K$. We did not measure the $T_c$ of this
sample but verified that it exhibited no Meissner effect in $N_2$. The sample was conducting at room temperature.
We ground and sieved these samples the same way as described for sample (1).
The third piece was heated to $1050 ^oC$ for $120$ minutes and then cooled suddenly to $77K$. The resulting sample was now insulating, indicating
that more oxygen had been removed by this process, as expected, and of course exhibited no Meisser effect. Also this sample turned to a green color. We ground this sample the usual way, which turned out
to be considerably more difficult than in the other cases as this sample was considerably harder.

(3) $MgB_2$ powder obtained from
Alfa Aesar, with transition temperature expected to be $39K$. Again the powder was passed through a $32\mu m$ sieve.

(4) $Pb$ powder, ground from a bulk piece and passed through a $32\mu m$ sieve.

(5) $Zn$ powder passed through a $32\mu m$ sieve.

   \subsection{Results}

We found that ball formation occurs both below and above $T_c$, and that balls that formed below $T_c$ do not become fragile above $T_c$.
   Indeed, balls formed at low temperature remain robust even when they are brought to �room temperature.

 Figure  \ref{sigma4} shows ball formation for the annealed $\sigma$-powder (see (1) in previous subsection) in $N_2$ and $Ar$, both in the capacitor with an electric field
 (upper panels)  and obtained
 by shaking without an electric field (lower panels). Since the critical temperature for this
 sample was $73K$, all cases correspond to temperatures higher than $T_c$. It can be seen that the balls look very similar in all cases.

     \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure4n.eps}}
 \caption { $BSCCO$ $\sigma-$powder annealed at $850^oC$ inside the capacitor with voltage 9kV (upper panels) and obtained by shaking in the absence
 of electric field (lower panels). }
 \label{sigma4}
 \end{figure}

Figure \ref{ybcocooked}  shows results for $YBCO$ heat treated  to $800^oC$ (left panels) (see (2) in previous subsection for description of heat treatment)
and untreated (right panels)
 in the capacitor in $N_2$. Initially (upper panels) we brought the voltage between capacitor plates to
9kV. The untreated powder (which exhibits a Meissner effect) forms balls under this voltage, while the heat treated powder did not. When we then raised the voltage
to 14kV, the heated powder also formed balls  as seen in figure \ref{ybcocooked}.
There was also powder clinging to the electrodes both below and above $T_c$, as the right and left sides of figure   \ref{ybcocooked} illustrate.

 \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure5n.eps}}
 \caption {$YBCO$ heat treated  to $800^oC$  (left panels) and untreated (right panels)  in capacitor  in liquid nitrogen.
 Voltage between capacitor plates is $9kV$ in upper panels and $14 kV$ in lower panels.}
 \label{ybcocooked}
 \end{figure}

We also investigated ball formation under shaking in these samples, both in $N_2$ and $Ar$, and show representative images in Figure \ref{ybcocooked2}. Again no discernable difference
is seen among the four different cases.

 \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure6n.eps}}
 \caption {$YBCO$ balls obtained by shaking in container with nitrogen and argon, no electric field. Upper panels: sample heat treateded to $800^oC$; lower panels:
 untreated sample.}
 \label{ybcocooked2}
 \end{figure}

Next we turned to the conventional  temperature superconductors $MgB_2$ ($T_c=39K$), $Pb$ ($T_c=7.2K$) and $Zn$ ($T_c=0.875 K$), samples (3), (4)  and (5). It was found that balls still formed
under shaking, however
the agglomeration process took several more minutes and the resulting balls for $Pb$ and $Zn$ were considerably smaller than in the other cases, of diameter less than $0.3 mm$.
We show a snapshot of the balls formed in Fig. \ref{pbzn}. We also show in Fig. \ref{pbzn} balls obtained from insulating $YBCO$ (green phase) (see sect. IV.A. (2)).

  \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure7n.eps}}
 \caption {Balls obtained by shaking powders in liquid nitrogen: $MgB_2$, $Pb$, $Zn$ and underdoped $YBCO$ (green phase).}
 \label{pbzn}
 \end{figure}

 We also attempted to form balls by the same shaking procedure in  water at room temperature. No tendency to agglomeration was found despite repeated attempts.
 Figure  \ref{water} shows examples of snapshots obtained during the shaking process for $BSCCO$ micrograins (size less than $32 \mu m$ as always) obtained from grinding
 the $BSCCO$ disk with transition temperature above $100K$. No tendency to agglomeration whatsoever was found despite repeated attempts, as figure  \ref{water} illustrates.
 We repeated these experiments in ethanol and methanol close to their solidification temperature (159 K and 175 K) and found behavior similar to that shown in figure  \ref{water}, with no
 tendency to ball formation.

   \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure8n.eps}}
 \caption {Shaking of $BSCCO$ micrograins in water at room temperature does not result in agglomeration and ball formation, the powder remains highly dispersed.
 The four panels show snapshots separated by several minutes of shaking each.}
 \label{water}
 \end{figure}

    \section{nature of the balls formed}

    The balls formed by shaking or in the capacitor remain stable when the liquid evaporates and the container heats to room temperature, provided the heating proceeds
    slowly enough to prevent violent boiling. In this section we discuss some properties of the balls obtained for the case of high temperature superconductors ($BSCCO$ and $YBCO$).

       \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure9n.eps}}
 \caption {Two examples of balls formed by shaking $BSCCO$ in nitrogen. The smaller balls (top figure) formed first after about two minutes of shaking, the bottom figure
 showing bigger balls is a snapshot taken after approximately two more minutes of shaking. The right side of the figures shows histograms of the ball sizes for both cases.}
 \label{size}
 \end{figure}

    In a typical run we obtain a distribution of ball sizes. Figure  \ref{size} shows an example. The maximum ball size found was of order 2 mm diameter, both in the cases where
    balls were formed by shaking as well as in the capacitor.
       Figure  \ref{largeball} shows a typical large ball, of diameter 1.5 mm, surrounded by microparticles of diameters between $25 \mu m$ and $32 \mu m$.
    All microparticles used in forming this and other balls passed through a $32 \mu m$ sieve, as discussed in section II, hence none are larger than those shown in Fig. 10.
    However most of the microparticles forming the balls are substantially smaller than those shown in Fig. 10 surrounding the ball, as illustrated in Fig.  \ref{decomp} where we
 show a ball of 1 mm diameter in various stages of
    decomposition.

    It appears that the presence of small microparticles is essential to the process of ball formation. We attempted to form balls composed of microparticles of large sizes only, and found
    that balls didn't form when the minimum size of the microparticles was $32 \mu m$ or larger. We did succeed after shaking gently for several minutes to form balls
    with microparticles restricted to the range $25 \mu m$ to $32 \mu m$. Figure  \ref{2balls} shows examples of such balls obtained. Note that the surface of a ball here is rougher
    than that of  the larger ball shown in figure  \ref{largeball}, where the powder contained also a large number of microparticles smaller than $25 \mu m$. In addition to being
    more difficult to form, the balls formed
    when the micrograin size was restricted to larger than $25 \mu m$ were found to be substantially more fragile than those that included smaller micrograins.

        \begin{figure}
 \resizebox{7.5cm}{!}{\includegraphics[width=9cm]{figure10n.eps}}
 \caption { A typical $BSCCO$
  large ball of diameter approximately $1.5 mm$ formed by microparticles smaller than $32 \mu$, surrounded by microparticles  that passed through a $32 \mu m$ sieve but did not pass through a $25 \mu m$ sieve.
 The microparticles forming the ball are of this size and  smaller.}
 \label{largeball}
 \end{figure}

        \begin{figure}
 \resizebox{7.5cm}{!}{\includegraphics[width=9cm]{figure11n.eps}}
 \caption {A typical $BSCCO$ ball of diameter approximately $1 mm$ in varius stages of decomposition. }
 \label{decomp}
 \end{figure}

    We measured the density of the balls at room temperature for $BSCCO$ in various runs, and found that it ranged between  $1.2 g/cm^3$ and $1.6 g/cm^3$.
    In contrast, the dry powder of microparticles of diameter $< 32\mu m$ used in forming the balls,
    when put in a glass tube and gently shaken, reached a  density  between  $1.8 g/cm^3$ and $2 g/cm^3$, and the disks from where the powders were obtained had density
    between $3.3 g/cm^3$ and $3.6 g/cm^3$. This indicates  that the balls are quite   loosely assembled and  have significant voids between micrograins. There was no apparent difference in the density of balls formed by shaking and in the capacitor.

Nevertheless, the balls are remarkably stable, both in liquid nitrogen and in air at room temperature.
    We found that at room temperature balls that formed either by shaking or in the capacitor  could be dropped from a height of    at least $10cm$ without breaking. The
    measured  terminal velocity was of order
     $80 cm/s$. In the capacitor, we found that balls bouncing back at forth at speeds of order $10 cm/s$ were stable, and in the shaking experiments balls moving at
    that speed were also stable.

             \begin{figure}
 \resizebox{7.5cm}{!}{\includegraphics[width=9cm]{figure12n.eps}}
 \caption { Two balls formed from agglomeration of particles of sizes between $25 \mu m$ and $32 \mu m$ only. }
 \label{2balls}
 \end{figure}

 \section{non-superconducting materials}
 In the previous sections we have found that superconducting material microparticles aggregate into spheres in liquid nitrogen and argon
 both in the presence and in the absence of electric fields, as
 well as below and above their superconducting transition temperatures. The question then arises, can spherical aggregation also occur with nonsuperconducting particles?

 The answer is yes. Figure  \ref{nonsc} shows results for 5 different substances, as well as for superconducting $BSCCO$, obtained
 following the same  shaking procedure
  described earlier. For all materials the size of microparticles used was considerably smaller than $32 \mu m$.  Of these materials we found that only for
 $Al_2O_3$ there seemed to be no tendency for spherical aggregation.
For  the other materials aggregation was found and the spheres obtained were of
 diameter up to $\sim 1 mm$.While it took different amounts of time to get aggregation,   the balls formed
 appeared to be very similar to those found in superconducting materials, as Figure \ref{nonsc}  illustrates.

            \begin{figure}
 \resizebox{7.5cm}{!}{\includegraphics[width=9cm]{figure13n.eps}}
 \caption { Spherical aggregation of various substances  obtained by shaking of microparticles in liquid nitrogen. }
 \label{nonsc}
 \end{figure}

    \section{the physics of spherical agglomeration}

Spherical agglomeration of both superconducting and non-superconducting particles has been previously reported \cite{tao,stock}, however  the explanations for the agglomeration \cite{tao,farnand} are incapable of explaining the phenomenon presently shown. Prior explanation of the aggregation of superconducting microparticles argued that the presence of an external electric field created a new type of surface tension in superconducting particles which caused them to aggregate into spheres. However, we have shown in this paper
  that these same microparticles will aggregate even in the absence of an electric field, which discredits this argument. Agglomeration of non-superconducting microparticles has previously been attributed to bridging, which requires the use of two fluids as well as the microparticles. In the experiment here only $N_2$ (or $Ar$)  is present in liquid form and it has been shown that water from the atmosphere cannot act as a bridging agent significantly below it's melting point \cite{smith} and the nitrogen was kept from boiling so there was also no gas-liquid surface to facilitate bridging. Agglomeration on non-superconducting microparticles has been reported in the absence of a bridging agent for sub-micron particles \cite{ford}, and
  attributed   to attractive Van Der Waals forces between the particles.
The aggregation reported here of particles considerably larger  than sub-micron cannot be explained by these traditional methods indicating a new process that produces spherical agglomeration in a wide variety of microparticles at liquid nitrogen and argon temperatures.

            \begin{figure}
 \resizebox{8.5cm}{!}{\includegraphics[width=9cm]{figure14n.eps}}
 \caption { Schematic depiction of charged planes of neighboring microparticles and surrounding electric field lines
 for (a) microparticles far apart and (b) microparticles closer together. }
 \label{forces}
 \end{figure}

 We conjecture that the forces giving rise to agglomeration in the cases considered here are predominantly electrostatic. The agglomeration appeared to be strongest
 for materials that are also high temperature superconductors. These materials (including $MgB_2$) have as a common feature the presence of $planes$ with a large
 charge imbalance. The high $T_c$ cuprate materials have negatively charged $CuO_2$ planes, with two excess electrons per unit cell, and compensating
 positively charged planes in-between. $MgB_2$ has negatively charged $B^-$ planes and positively charged $Mg^{++}$ planes. For an irregularly shaped
 microparticles there are likely to be sections where either a negatively or a positively charged plane is closest to the surface, and as a consequence
   electric field lines going from the negatively to the positively charged surface of a microparticle. It would then be energetically advantageous to join
 the electric field lines of neighboring microparticles as  in the arrangement shown in   figure \ref{forces}  and bring the microparticles close together.
 This physics is related to the fact that quite generally a crystal will have different work functions for surfaces parallel to different crystal planes, and hence an electric
  potential difference
 between different surfaces and electric fields around them. We conjecture that the layered high temperature superconducting materials have larger differences between
 workfunctions for different crystal planes than other materials.

 As a rough estimate of the energetics involved, consider two surfaces of neighboring microparticles facing each other, of area $A=(10\mu m)^2$ at a distance
 (determined by the surface roughness) of $d=1\mu m$, with a potential difference of $5V$. The electrostatic energy associated with this is $U=CV^2/2\sim 10^{-14}J$, with
 $C=\epsilon_0A/d$. If these are grains of density $\rho=3g/cm^3$ and volume $V=(10\mu m)^3$ moving at speed $v=10cm/s$, their kinetic energy
 is $K=1/2\rho V v ^2\sim 10^{-14}J$, i.e. of the same order. This is consistent with the fact that we observe the balls to be stable in liquid nitrogen for speeds of that order and become unstable
 for larger speeds. For microparticles of larger volume and mass, the kinetic energy increases proportionally to the volume  while the potential energy gained in joining
 microparticles will increase not faster than the surface area, hence   the speed at which they break apart should become smaller, consistent
 with our observations.

   The hypothesis that the forces causing agglomeration are electrostatic is also consistent with the fact that we did not find agglomeration in water (figure \ref{water}).
 Water has negative $OH^-$ and positive $H^+$  (or $H_3O^+$) ions  in solution, and these ions should screen the electric fields near the surface of microparticles and
 inhibit the aggregation process. The microparticles themselves can also create further ionization of water in addition to the self-ionization so that no aggregation
 should occur even with a high concentration of microparticles. The same physics presumably inhibited aggregation in the solutions with  ethanol and methanol.

      \section{summary and conclusions}

In this paper we reported results showing that: (i) in a capacitor, ball formation is induced by application of a high
voltage between the plates if the electrodes
are conducting but not if they are insulating, (ii) aggregation of microparticles into
balls  occurs also  in the absence of
an applied electric field, (iii) balls formed below $T_c$ remain robust when the temperature is
raised above $T_c$, (iv) balls form  both  below and above $T_c$ in liquid $N_2$ and liquid $Ar$ for a variety of cuprate superconductors,   (v) $MgB_2$, $Pb$, and $Zn$ agglomerate into spheres at temperatures well above their respective $T_c$, (vi) insulating $YBCO$ also agglomerates into spheres in liquid nitrogen,
(vii) microparticles of several  different non-superconducting materials suspended in liquid nitrogen also aggregate into spheres of $mm$ dimensions upon shaking.

Our findings show significant differences with the findings reported by Tao et al\cite{tao,tao2,tao3,tao4}. Tao et al  reported that
(1) an electric field drives formation of superconducting  balls,  (2) the balls become fragile if the electric field is turned off, (3) there is a lower critical electric field $E_{c1}$, that is frequency dependent,
below which balls don't form,  (4) the radius of balls decreases when the electric field increases,
(5) in a large electric field, powder that clings to the electrodes below $T_c$  will fly off above $T_c$,
(6) balls of $BSCCO$ formed in nitrogen break apart when liquid argon is added to    increase   the temperature, (7) balls don't form
 above the superconducting transition temperatures,  (8) balls formed at liquid nitrogen temperature become fragile at room
temperature, (9) granular particles making the balls are closely packed, (10) the surface of the grains needs to be conducting for balls to form. Each one   of
these findings is  in
  disagreement with our findings.

Additionally, Tao et al   reported that the radius of the balls is a function of the AC frequency of the electric field,   that the radius of $MgB_2$ balls goes linearly to zero as the temperature
approaches $T_c$ from below, and that balls break up when a strong magnetic field is applied. We have not tested these findings directly but conclude in view of our other findings that they are unlikely to be correct.

In their study of ball formation in low frequency ac electric fields\cite{tao3}, Tao et al reported that the lower critical field for ball formation, $E_{c1}$, decreased strongly
as the frequency decreased, and jumped almost discontinuously  to a high value  at zero frequency (see Figs. 6, 7  and 8 of Ref. \cite{tao3}). For example,
for $NdBa_2Cu_3O_x$, $E_{c1}$ decreased smoothly from about $1200 V/mm$ for frequency $500 Hz$ to $117 V/mm$ for $10Hz$, and jumped to $760 V/mm$ for a
dc field.  Our interpretation of these observations is that the ac field causes ``shaking'' of the powder in a way similar to our shaking experiments. For higher frequency
a larger electric field is needed so that the displacement of the microparticles is sufficient to cause agglomeration, and for very low frequencies the amplitude of the
electric field required approaches zero, consistent with our observation that agglomeration occurs also in the absence of electric fields.

Our findings indicate that spherical aggregation of microparticles (both conducting and insulating) suspended in liquid nitrogen and liquid argon is a very general phenomenon.
We did not find clear evidence that ball formation   is helped
by the presence of  superconductivity in the sample as predicted theoretically\cite{taoeffect}. We did generally find that the easiest and fastest ball formation, particularly
of large balls, occurred for microparticles of high $T_c$ superconductors ($BSCCO$ and $YBCO$).
 We hypothesized that a strong  tendency to ball formation results from the existence of charged planes in the structures.
Whether or not there are subtle changes in the tendency to ball formation
depending on whether the system is in the superconducting or normal state  remains an open question.

  \acknowledgements{}
  The authors are grateful to B. Maple and his group members, particularly Ivy Lum and Colin McElroy,
   for kindly sharing laboratory supplies, pressing $\sigma-$powder into pellets,  and  measuring the resistive transition temperature of the
annealed $\sigma$ sample,   to Ali Basaran for assistance, and to the UCSD Physics Department for financial support.


The influence of the environment on bar formation

Abstract: Galaxy mergers and interactions are mechanisms which could drive the formation of bars. Therefore, we could expect that the fraction of barred galaxies increases with the local density. Here we show the first results of an extensive search for barred galaxies in different environments. We conclude that the bar fraction on bright (L>L*) field, Virgo, and Coma cluster galaxies is compatible. These results point towards an scenario where the formation and/or evolution of bars depend mostly on internal galaxy processes rather than external ones.

\section{Bar formation: nature vs. nurture}
The observational proofs about the influence of the environment on bar
formation and/or  evolution are still rare.   In \citet{aguerri09}, we
investigate  a volume-limited  sample  of 2106  disc  galaxies in  the
nearby universe to derive the bar  fraction as a function of the local
galaxy density.  The local  density was
calculated for  every sample galaxy using the  fifth nearest neighbor
method,  obtaining that  80\% of  our  galaxies were  located in  very
low-density  environments  ($\Sigma_5   <$  1  Mpc$^{-2}$),  and  20\%
(corresponding to  more than 400  galaxies) covers mostly  the typical
values  measured for  loose ($\Sigma_5  >$ 1  Mpc$^{-2}$)  and compact
galaxy groups  ($\Sigma_5 \sim$  10 Mpc$^{-2}$). We  did not  find any
difference  between the local  galaxy density  of barred  and unbarred
galaxies in our  range of densities.

To  extend this  conclusion to  higher density  environments,  we have
investigated the fraction of barred galaxies present in the two nearby
\emph{benchmark}  clusters: Virgo  (Zarattini et  al., in  prep.)  and
Coma (M\'endez-Abreu et al., in  prep.).  In both clusters, we found a
bar fraction in their bright galaxy population consistent with that in
the  field. Similar  results have  been recently  found in  cluster at
higher   redshift  by  Marinova   et  al.    (2009)  and   Barazza  et
al.  (2009). However,  the errors  in the  clusters bar  fractions are
large  and  the  results  might   be  affected  by  the  small  number
statistics. To deal with this  problem, we are undergoing an ambitious
project  to  study the  bar  fraction in  a  large  sample of  cluster
galaxies drawn from the WINGS project.

Cosmology and Nongalactic Astrophysics (astro-ph.CO)


Brain correlates of task-load and dementia elucidation with tensor machine learning using oddball BCI paradigm

Abstract: Dementia in the elderly has recently become the most usual cause of cognitive decline. The proliferation of dementia cases in aging societies creates a remarkable economic as well as medical problems in many communities worldwide. A recently published report by The World Health Organization (WHO) estimates that about 47 million people are suffering from dementia-related neurocognitive declines worldwide. The number of dementia cases is predicted by 2050 to triple, which requires the creation of an AI-based technology application to support interventions with early screening for subsequent mental wellbeing checking as well as preservation with digital-pharma (the so-called beyond a pill) therapeutical approaches. We present an attempt and exploratory results of brain signal (EEG) classification to establish digital biomarkers for dementia stage elucidation. We discuss a comparison of various machine learning approaches for automatic event-related potentials (ERPs) classification of a high and low task-load sound stimulus recognition. These ERPs are similar to those in dementia. The proposed winning method using tensor-based machine learning in a deep fully connected neural network setting is a step forward to develop AI-based approaches for a subsequent application for subjective- and mild-cognitive impairment (SCI and MCI) diagnostics.

\section{Introduction}
\label{sec:intro}
Currently, dementia is one of the most significant global challenges for healthcare and social services.
Worldwide, mainly for above 65 years old people, dementia statistics and associated costs are increasing due to enhanced longevity~\cite{lancet2017dementia}. Cabinet Office in Japan, to address the problem,
publishes annual reports on aging society~\cite{agingJPgovREPORT}. United Nations Sustainable Development Goal number three -- ``Good Health and Wellbeing" focus, for all at all ages, on healthy lives and it supports well-being.
We propose a method utilizing AI--related machine learning (ML) for automatic classification of anomalous EEG brain signals, which shall lead to digital biomarkers development for the task--load as well as dementia progress elucidation. The current dementia diagnostic methods rely on standard psychometric subjective tests, more recent behavioral assessments using cognitive behavioral therapy (CBT) methodologies  (a co--imagination technique~\cite{otake2009coimagination}, etc.) or the very recent multimodal interventions~\cite{kivipelto2018world}. We propose to use a human noninvasive brain signal monitoring (EEG) technique (namely, event-related potentials - ERPs derived from responses to natural stimuli within modern, comfortable recording~\cite{bciBOOKwolpaw,nozomuANDtomekAPSIPA2012,tomekJNM2015,tomekFRONTIERS2016}. Methods presented in this paper and developed by our lab, establish a machine learning (ML) or AI application allowing for near future a home monitoring of the mental decline (dementia) and the task--load elucidation with employment of digital biomarkers.

Recent neuroscience research findings have shown that dementia decline was related to abnormal amyloid and tau disposals that modified post-- and presynaptic neuronal mechanisms, which resulted in high neuronal calcium influxes leading to neuronal loss, increased excitability and finally modified brain rhythmic patterns~\cite{oscillopathy2006,dementiaBIOMARKER2018review}. The synaptic plasticity is necessary for various cognitive functions (e.g., memory formation, abstract thinking, learning,  etc.).
Dementia and especially Alzheimer's disease (AD) has been recently identified from contemporary neurophysiological and pathological studies as impaired synaptic plasticity~\cite{dementiaBIOMARKER2018review}.
The above mentioned age-related brain atrophy has also been named as network disconnection disease called ``an oscillopathy''~\cite{oscillopathy2006}.

Recently,  various approaches to AD and dementia diagnostics created a necessity to establish personalized therapies using not only traditional pharmacological interventions but also the necessary lifestyle alternations~\cite{lancet2017dementia} as well as the very popular cognitive training~\cite{otake2009coimagination}.
The usual pharmacological as well as the modern {\it digital--pharma} ({\it beyond a pill}) healthcare approaches call for robust biomarkers. The biomarkers need not only to elucidate the cognitive loss from resting-state-- digital brain biomarkers~\cite{tomekICASSP2006} but must allow for simple home monitoring. We evaluate the so-called work-- task--load paradigm for dementia-related brain responses' classification in natural auditory stimulus settings. In this paper we report on the evaluation of EEG  classification methods using dementia--modeling ERPs~\cite{oscillopathy2006,dementiaBIOMARKER2018review} in spatial auditory and task--load controlled brain-computer interface (BCI) experimental settings and from healthy subjects~\cite{nozomuANDtomekAPSIPA2012}.
The spatial auditory BCI paradigm developed in~\cite{nozomuANDtomekAPSIPA2012} allows for the task--load modulation by incorporating real and virtual spatial source sounds, which are easy or hard to localize spatially.
The users are instructed to focus their attention on spatial targets (resulting with P300 responses) and ignore distractors (a classical oddball paradigm)~\cite{nozomuANDtomekAPSIPA2012,tomekJNM2015,tomekFRONTIERS2016}.
Averaged results with abnormal and normal (varying task--load settings to model dementia responses~\cite{oscillopathy2006,dementiaBIOMARKER2018review}) ERPs are depicted in Figure~\ref{fig:p300}.

We propose a framework, using the AI--based neurotechnology, which employs ML together with data--driven preprocessing techniques utilizing a wavelet--synchro--squeezing--transform (WSST)~\cite{tomekJNM2015,tomekJOMS2016}, Riemmanian geometry (RG)~\cite{barachant2010riemannian} and tensor--train--layer (TT--layer)~\cite{oseledets2011tensor,cichocki2017tensor} classification methods.

From now on the paper is organized as follows. In the next section, we introduce EEG signal processing and machine learning methods. Results and conclusions with future research guidelines summarize the paper.
\begin{figure}
	\centering
  	\includegraphics[width=0.7\linewidth]{p300.png}
	\caption{Grand mean averaged ERPs from nine healthy subjects performing spatial auditory BCI oddball experiments~\cite{nozomuANDtomekAPSIPA2012}. All EEG channels, as in classical $10/20$--system~\cite{bciBOOKwolpaw}, are presented as bandpass filtered and averaged traces with 95--percentile ranges. Green traces represent easy (low task--load) ERPs to identified spatial targets with clear P300 responses~\cite{bciBOOKwolpaw,nozomuANDtomekAPSIPA2012}. The orange ERPs to the difficult virtual cases (high task--load), which model dementia decline signals~\cite{dementiaBIOMARKER2018review}.}\label{fig:p300}
\end{figure}

\section{Methods}\label{sec:methods}
The brain responses, which model EEG--derived digital biomarkers in our study, were collected from nine healthy subjects in a spatial auditory BCI former project~\cite{nozomuANDtomekAPSIPA2012} reviewed by The Ethical Committee of Faculty of Engineering, Information and Systems at the University of Tsukuba, Tsukuba, Japan.  The data collection details were described in~\cite{nozomuANDtomekAPSIPA2012}.
The currently discussed EEG brain signal post-processing project was approved by The Ethical Committee of RIKEN, Wako--shi, Japan.

At the EEG preprocessing filtering steps we utilize a wavelet--synchro--squeezing--transform (WSST) method previously published by the authors for sleep stages' and BCI commands'  classification~\cite{tomekJOMS2016,tomekJNM2015}. The WSST application follows precisely the cased discussed in~\cite{tomekJNM2015}.
At the next step of EEG processing pipeline we assume that $\mathbf{x}(t)\in\mathbb{R}^N$ is a zero--mean data sample captured from eight (denoted by $N$) EEG electrodes at discrete time $t.$
In such a case, let $\mathbf{x}_{c,i}\in\mathbb{\mathbb{}R}^{N}$ be the ERP event $i$ representing a response to a spatial auditory stimulus $c\in \{1,2\}$ of $M$ samples ($1.5$~seconds in our case).
Assuming a a zero mean, a sample covariance matrix of a single trial $\mathbf{x}_{c,i}$ belonging to a class $c$ is given by $\mathbf{C}_{c,i} = \frac{1}{M-1}\mathbf{x}_{c,i} \mathbf{x}_{c,i}^T,$ as proposed by~\cite{barachant2010riemannian}.

Assuming that a noise in EEG recordings could be modeled by multivariate Gaussian distributions, a covariance matrix characterizing ERP features could be considered as the only unique parameter for dementia stages (task--load). Features representing stimulus in segmented ERPs $c\in \{1,2\}$ are obtained as $\mathbf{x}_{c,i}$ and calculated as covariance matrices $\mathbf{C}_{c,i}$ for a subsequent input to ML, as discussed in following sections.

\subsection{Shallow Learning Classifiers}
\label{sec:shallow}

In a classifier training phase~\cite{barachant2010riemannian}, a geometric mean covariance matrix $\mathbf{\overline{C}}_{c},$ characterizing each ERP class $c,$ is computed, using all EEG channels as inputs. In order to measure a distance of a newly captured ERP to the class--characterizing mean matrix, the RG techniques are used.
A geodesic traversing two points $\mathbf{C}_i$ and $\mathbf{C}_j$ is the shortest path curve that connects them. It has a minimum length. A Riemannian distance between covariance matrices is obtained as follows,
\begin{equation}
    \delta_R = \left|\left|ln(\mathbf{C}_i^{-1}\mathbf{C}_j)\right|\right|_F = \sqrt{\sum_n[ln(w_n)]^2},
\end{equation}
where $||\cdot||_F$ denotes a Frobenius norm and $w_1,\ldots,w_n$ the eigenvalues of $\mathbf{C}_i^{-1}\mathbf{C}_j$, respectively~\cite{barachant2010riemannian}.

The geometric mean of $L$ covariance matrices characterizing an ERP class is obtained as,
\begin{equation}
    D(\mathbf{C}_1,\cdots,\mathbf{C}_l) = \arg\min_{\mathbf{C}}\sum_{l=1}^L\delta_R^2(\mathbf{C},\mathbf{C}_l).
\end{equation}
On a manifold, the geodesic, according to the RG principles~\cite{barachant2010riemannian} is calculated from,
\begin{equation}
    \Gamma(\mathbf{C}_i,\mathbf{C}_j, \tau) = \mathbf{C}_i^{\frac{1}{2}}\left(\mathbf{C}_i^{-\frac{1}{2}}\mathbf{C}_j\mathbf{C}_i^{-\frac{1}{2}}\right)^\tau\mathbf{C}_i^{\frac{1}{2}},
\end{equation}
where  $\tau\in\{0,1\}$ is a scalar.

A very frequently utilized classification approach for RG--related features has been based on a distance evaluation among ERP features and mean (class characterizing) covariance matrices~\cite{barachant2010riemannian}. A a minimum distance to mean (MDM) classifier~\cite{barachant2010riemannian} satisfies the above criterium.
The MDM approach is easy to apply as well as very generic. We compare it with previously reported by our group in~\cite{tomekNIPS2018} classical vectorized (vect) time domain derived ERP features~\cite{tomekJNM2015}; spatial filter xDAWN preprocessing~\cite{hiroshiAPSIPA2016}; tangent space (TS) mapping of RG features~\cite{barachant2010riemannian}; using linear regression (LR); regularized linear discriminant analysis (rLDA); linear and sigmoid kernel support vector machine (SVM), as well as with below to be introduced TT-layer--based (TT) methods as shown in Figure~\ref{fig:acc}.
\begin{sidewaysfigure}
	\centering
  	\includegraphics[width=\linewidth]{results.png}
	\caption{Median classification accuracy results (white dots within percentile ranges) depicted together with result distributions (a chance level of $0.5$ representing $50\%$ of binary classification cases depicted with bold gray line). }
	\label{fig:acc}
\end{sidewaysfigure}

\subsection{Tensor--train--layer--based Neural Network Classifiers}
\label{sec:tt}

Deep neural networks recently started to gain attention in EEG community especially for the end--to--end processing setups~\cite{braindecode2017}. For our approach, we are interested in compacting machine learning architectures with possible application to wearable devices with limited computing powers.
Tensor--train factorization (TTF) approaches have demonstrated an advantage of scaling multidimensional matrices to an arbitrary number of dimensions~\cite{oseledets2011tensor,cichocki2017tensor} with a subsequent possibility to reshape a fully connected neural network layer into a tensor and then factorize it~\cite{novikov2015tensorizing}. The methodology as mentioned earlier is applied to compress large weight matrices of deep neural models within an entire end--to--end training. Using a trick from~\cite{novikov2015tensorizing} it is possible to tensor--train--factorize a weight matrix $\mathbf{W}$ of a fully connected neural network as a $d-$dimensional double--indexed tensor represented as
\begin{eqnarray}
    \widehat{\mathbf{W}}\left((i_1, j_1), (i_2, j_2),\ldots,(i_d, j_d)\right) = \nonumber \\
    = \mathbf{\mathcal{G}}^*_1(i_1, j_1)\mathbf{\mathcal{G}}^*_2(i_2, j_2) \ldots\mathbf{\mathcal{G}}^*_d(i_d, j_d),
\end{eqnarray}
where $\mathbf{\mathcal{G}}^*_k \in \mathbb{R}^{m_k \times n_k \times r_{k-1} \times r_k}$ are the so--called core tensors uniquely represented  by indices $(i_k, j_k)$~\cite{novikov2015tensorizing}. It is possible to compress the weight matrix $\mathbf{W}$ of size $\prod_{k=1}^d m_k  \cdot n_k$ in a form of TT--formated low--rank tensors, approximately reconstructing the original $\mathbf{W}$, of size $\sum_{k=1}^d m_k \cdot n_k  \cdot r_{k-1} \cdot r_k.$
We consider a two--layered neural network with $64$ hidden units in each layer and replace both fully-connected layers by the TT--layers with all the TT--ranks in the network set to $4$ and activation functions to ReLU. For an output of a fully connected final layer with sotfmax activations we choose two units. In order to prevent overfitting we train and test (using ``unseen'' during the training samples) the model with $50\%/50\%$ balanced datasets.
We also evaluate a recurrent neural network with LSTM units as a classifier with TT--layers applied to gating units~\cite{yang2017tensor}. In this case the network has again $64$ input LSTM units and TT--ranks set also to $4$. A fully connected output layer with sotfmax activations also has two units in this case.

\section{Results}
\label{sec:results}

The discussed neurotechnology approach utilizing  AI/ML technique for the task--load elucidation, which has been proposed as a model for demented brain cognitive responses quantification, resulted in a particular classification increase.
We compared the information geometry principle using shallow learning and TT--layer classification methods introduced in the previous section.
A  solid, as well as statistically significant, the boost of the automatic task--load--dependent ERP classification was achieved in the presented project. The project goal has been to automatically discriminate the EEG signals modulated by two tasks--load--dependent cognitive levels, comprising of ordinary (easy) and anomalous (difficult) spatial auditory localization tasks.
We obtained results from the nine subjects--based dataset with classifiers trained using all participant class--balanced, to avoid overfitting, training sets.
Subsequently, we tested classifiers with respective testing utterances.
The results were summarized in the form of classification accuracy distribution plots in Figure~\ref{fig:acc}.

The best results among the shallow learning classifiers (see Section~\ref{sec:shallow}) with an average of $68\%,$  which has been considered already as satisfactory for BCI applications, were obtained for the tangent space (TC) mapping with the linear regression (LR), the regularized LDA (rLDA) and support vector machine (SVM) cases, comparing to the minimum distance to mean (MDM) and to the simple vectorized time domain ERP features. The differences were statistically significant in all comparisons as evaluated with non--parametric pairwise Wilcoxon method. RG approaches scored significantly better comparing to the classical vectorized (vect) and xDAWN--based results with $p \ll 0.01.$ Also the RG tangent space (TS) mapping--based results were significantly better, with  $p < 0.05$,  than RG--based MDM classifier accuracy results, respectively. The shallow learning results confirmed our previous findings reported in~\cite{tomekNIPS2018}.

On the other hand, using the TT--layer--based LSTM (see Section~\ref{sec:tt}), we have achieved median accuracy of $68\%$ for two task--load cases with significant differences $(p\ll 0.01)$ compared to all other shallow learning classifiers (see Figure~\ref{fig:acc} with accuracy comparisons), as evaluated with non--parametric Wilcoxon pairwise tests.

Finally, the results achieved with TT-layer fully connected neural network (details also in Section~\ref{sec:tt}) had a median accuracy of $78\%$ for two task--load cases with significant differences $(p\ll 0.01)$ compared to all other tested in this paper classifiers (see Figure~\ref{fig:acc}), as also calculated using Wilcoxon pairwise comparisons.

\section{Conclusions}
\label{sec:conclusions}

The presented project confirmed our hypothesis of a possible application of  TT--layer neural model--based task--load elucidation approach as tested with nine subjects. The proposed approach resulted with a satisfactory classification ($78\%$ on average) of binary task--load cases, namely the 'easy real' versus the 'difficult virtual' sound localization paradigms, as shown as averaged ERPs in Figure~\ref{fig:p300}, and the obtained classification accuracy boost results in Figure~\ref{fig:acc}.

The discussed EEG--task--load classification results offer a step forward in the establishment of novel dementia-related biomarkers for improving the lives of seniors and reducing health-related costs. The task--load responses (EPRs) were very similar to EEG modulations in dementia.

As for following steps in our project we plan to test the discussed method with elderly healthy and mild cognitive impairment (MCI), dementia, etc., subjects.
We also plan to collect more data to further train and evaluate deep learning models in order to achieve improvements in the proposed approach towards the final application in the real--world healthcare settings. Further research on the presented TT--layer neural model interpretability and a practical computational cost are also planned.

Neurons and Cognition (q-bio.NC); Machine Learning (cs.LG); Signal Processing (eess.SP)