Plea to publish less

Abstract: Recent research\cite{fort} has shown that in the 20th century there is an exponential growth of the number of published scientific papers, but new ideas has only a linear growth with time. The 19th and the first half of the 20th century saw major scientific discoveries. But from the second half of the 20th century, the number of publications far exceeds the number of impactful discoveries. Does an exponentially growing number of publications indicate an element of pathological research? Pressure to publish a large number of papers has led to the phenomena of overproduction, unnecessary fragmentations, overselling, predatory journals (pay and publish), clever plagiarism, and deliberate obfuscation of scientific results so as to sell and oversell. This is harming the healthy scientific culture of passion driven research. It is an urgent problem which needs attention of the whole of the scientific community. Some ways to mitigate these grave problems are discussed.

\section{Changing culture of the practice of science:  before 20th century.}

Science before the turn of the 20th century was practiced by individual researches mostly motivated by the urge to find some "transcendental  truths" in natural phenomena.  Some of the leading discovers made in the 16th, 17th, 18th, and the 19th century are: Overthrowing of the geocentric model of the universe and the introduction of the heliocentric model by Nicolaus Copernicus (1543);  law of inertia (1613), observational astronomy, and the scientific method of experimentation by Galileo Galilei; Kepler's three laws of the planetary motion (1619);  Many pioneering discoveries in almost all the fields of natural science and mathematics by Issac Newton (Principia 1687);  atomic theory in chemistry by John Dalton (1805); magnetic effects of electric current by Oersted (1820). Sadi Carnot (1824),  Lord Kelvin (1848), Rudolf Clausius (1850) and  others laid the foundations of classical thermodynamics; Statistical mechanics was developed by Maxwell, Boltzmann, and Gibbs; Investigations of Faraday, Maxwell, and many other investigators lead to the theory of electromagnetism (1862);  X-rays were discovered by Wilhelm Roentgen (in 1895);  Radioactivity by Henri Becquerel (in 1896); Electron by J. J. Thomson (in 1897); etc.

However, data shows that only a few thousand of papers were published\cite{gian}. Majority of the papers published had some impact. And the phenomenon of the collaborative research was very rare (most of the time, papers published were single-authored).

Majority of professors/investigators use of publish a couple of dozens of papers in their entire scientific career. Consider the example of Josiah Gibbs (1839-1903) who published less than two dozen papers in his entire career. Gibbs case is a typical case of a scientist of that era (in terms of the number of publications).

\section{Physics in 20th century}

From 1900  to 1950, some of the main discoveries made are: Energy quantum by M.  Planck (1900);  nuclear atom by Rutherford (1911); superconductivity by Onnes (1911); special and general relativity by Einstein (1905, and 1915 respectively);  crystal structures and X-ray diffraction (William and Lawrence Bragg, 1915); Quantum mechanics (M. Planck, N. Bohr, A. Einstein, W. Heisenberg, E. Schroedinger, M Born, P. Dirac and others); Raman effect (by C. V. Raman); Dirac's theory of electrons and positrons; nuclear fission and fusion; Quantum electrodynamics; etc. The later half of the 19th century and the first half of the 20th century can be said "the golden age" of physics.

From 1950  to Current times, some of the main discoveries made are : Nuclear and particle physics; solid state physics; semiconductors and transistors;  electroweak theory and the emergence of standard model; Bardeen-Cooper-Schrieffer (BCS) theory of superconductivity; parity violation; quantum Hall effects; renormalization; QCD; BEC realized; Higgs boson observed; gravitational waves observed, etc.

After WWII, there is an outpouring of the scientific activity. This is due to technological developments, new instruments of investigations, and due to the vast applicability of quantum mechanics into atomic, solid state, and nuclear physics.  This outpouring is evident from how the scientific journals evolved from pre-WWII era to post-WWII era. Take for example, the Phys. Rev.  The Phys Rev started on 1893, as a single volume. In 1958, PRL was introduced; and in 1970, PRA, PRB, PRC, and PRD were introduced.  Currently APS is publishing an array of journals: Phys. Rev. A, B, C, D. Phys. Rev. E; Physics; Phys. Rev. X; Phys. Rev. Applied; Phys. Rev. Fluids; Phys. Rev. Materials; Phys. Rev. Research; PRX Quantum etc.

In the year 1910 Phys. Rev. published articles with 1484 printed pages in total, whereas in 2019, only the Phys. Rev. B (one journal out of the whole array) published 98,398 articles in just six months (from Jan 2011 to June 2019). If each article contains roughly 10 pages, then this amounts to {\it one million} pages!

\section{What has gone wrong in current times?  (Are we going from facts to pathology?)}

There were some thousands of papers published before 1900. By 1960, two million scientific papers have been published! Current times, over two million scientific papers are being published every year! What was produced between 1650 to 1960 (over three hundred years) is now being produced every year (in terms of volume, not number of discoveries)\cite{gian}. But the number of impactful discoveries made in the last half of the 20th century to the current time is much less than the number of papers published in this time period (in comparison to that in the 19th and the first half of the 20th century). Concrete data\cite{fort} shows that there is a linear growth of ideas but there is an exponential growth of the number of papers published (figure 1).

\begin{figure}[h!]
\begin{center}
\begin{tabular}{cc}
\includegraphics[width =5.5cm]{fig1.pdf}&
\includegraphics[width =5.5cm]{fig2.pdf}\\
(a)&(b)\\
\end{tabular}
\end{center}
\caption{(a) Exponential growth of paper production.  (b) Only linear growth of ideas. For actual data and graphs, refer to\cite{fort}.}
\label{f1}
\end{figure}

This strange trend in the data raises serious questions: Are we going from facts to pathology? Is it due to scientific developments only (new fields, new subfields)? Or is it due to the ever growing number of active scientists? What is happening?

\section{Reasons (for floods of papers).}

Simplest reason one can give is that there are more number of active scientists working in very competitive environments (as compared to the situation in the 19th and early 20th century). Data shows that in 1930s there was one scientist per 10,000 people in the whole world. This number in 2013 swelled to one scientist per 700 people\cite{gian}.  This is 14 fold increase! There were 3 lakh scientists in 1935. And this number grows to 10 million in 2016. Even if the "production rate" per scientist (that is the average number of papers published per scientist) remains the same, this sheer rise in the number of active scientists could explain the larger production of the number of published papers\footnote{In old culture, majority of professors/investigators use of publish a couple of dozens of papers in their entire career. Currently, majority of professors/investigators publish over 100 papers in their entire scientific career. So average number of papers published {\it per scientist} has also increased, at least in the physical sciences. Collaborations increase the production rate further.}

But the question raised in the previous section: Why are the number of impactful discoveries made much less than the number of papers published?

It could be due to the following reasons:

\begin{enumerate}

\item There are distortions in the practice of science: Old trend: theoretical understanding of the experimental facts (Max Planck and black body radiation; Einstein and photoelectric effect etc). Passion to find some ``transcendental truths''.  New tread: More theoretical ideas and more ``proposed'' experiments. So more papers! More generalizations and more abstractions Imagine a hypothetical situation, do a calculation, produce a paper! This is due to the pressure to publish. This drastically reduce the ratio of the number of impactful discoveries to the total number of papers published.

\item Importance of numbers of papers published in jobs, getting pdf positions, getting funds etc. This put emphasis on numbers along with quality. As we discuss below, the metrics that measure quality are not foolproof.

\item Sheer scientific culture that promotes the number business (refer to footnote 1).

\item Emergence of competitive environments. This further leads to a constant pressure to build huge groups/collaborations etc. This results in overproduction. There is a trend to follow hot topics/fashionable topics/hypes\cite{eva}. Invent strategies to sell and over-sell (intense fights with editors and referees).

\item Old times: Mostly single author or couple of authors; Now majority of papers come out of  collaborations (Collaborations sometimes are required). Collaborations has the potential to increase volume, but whether it leads to an out-put of a great quality publications? Surely a point to ponder over!\cite{henri}.

\item Pressure to publish in reputed journals, rejections, and then the phenomenon of predatory journals: you pay and publish. Easy and low quality publications. Loss of impactful science. And most importantly the misuse of the concept of ``open access".

\item Core open scientific problems are limited in number. There are problems posed by ``Nature" and then there are "man-made" problems. The main scientific problems in theoretical physics can be counted: Beyond standard model physics; What is small CP violation trying to tell us? GUT: ``Wedding" of QCD with electroweak theory; Neutrino mass problem; Cuprate high temperature superconductivity; Physics of heavy fermions metals etc. Electroweak theory is renormalizable, but there is no Quantum theory of gravity which is renormalizable; Can there be a theory using which one can compute the mass of an electron? The problem of dark matter etc.

\begin{figure}[!h]
\begin{center}
\includegraphics[height=5cm]{fig3.pdf}
\caption{Sets of peripheral problems.}
\end{center}
\label{f2}
\end{figure}

To meet the high demands of competition and job issues one is forced to invent easy and peripheral problems (refer to figure (2)).  Working on peripheral problems leads to publications. It can be helpful to a student in his/her career. It can be helpful in promotions/jobs. It may be helpful in getting some awards, fellowship of academies etc. But it leads to a loss of the impactful science. It takes years to solve a hard problem, but an easy problem can be solved in lesser time. The current academic culture supports the latter one, unfortunately!

\item There are experimentally underdetermined problems. When the available input experimental parameters required to validate a theory are much less in number, then there is a large parameter space to play with. Many theoretical models/scenario are possible (as the problem remains observationally/experimentally under-determined). This may lead to burden to literature. Care must be exercised when publishing in such areas.

\item Addiction to publish: I know a person who gets some sort of irritation (mental upsetting) if he does not get a paper in a month's time.
So every month he must get a paper (low quality, high quality, through students, through collaborators, it does not matter!).
Will that large volume of papers (containing solutions of easy and man-made problems) be useful?
Will this not burden referees/editors/and students who are just entering the field?

\item Vicious cycle:

The more production of papers leads to getting more funding and further leading to bigger collaborations/groups. This in turn leads to a greater production of papers. Thus, a ``vicious cycle" starts! (refer to figure 3).

\begin{figure}[!h]
\begin{center}
\includegraphics[height=5cm]{fig4.pdf}
\caption{A ``vicious cycle".}
\end{center}
\label{f2}
\end{figure}

But there is a difference between an industrial production and scientific solution of a riddle posed by Nature. It is the duty of the funding agencies that they should not put emphasis on the quantity, and while measuring the quality of the scientific output through the standard measures, they must realize that these measures (metrics, impact factors of journals etc) are themselves are distorting the good culture (as discussed below). Actual reading of the scientific papers by an expert panel can help save the situation.

\end{enumerate}

\section{Consequences (on individuals; on sociology; and on science itself).}

Emphasis on the number of publications (or culture of producing a large number of publications) leads to

\begin{enumerate}

\item a shift of focus from central difficult problems to more peripheral and easy problems.  It is easy to solve an easy problem and more difficult to solve a difficult problem.

\item loss of cohesiveness of a given field and unnecessary fragmentations within the subfields.  Leads to difficulties for a beginner to understand and penetrate into a given field (One has to deal with a huge volume of literature, much of which may not be relevant). The human capacity for information processing is limited. There are physiological limitations. One can read a couple of papers published per day in one's special area (refer to arXive). How can one even read abstracts if hundreds of papers are being published in one's research topic? It will be difficult to keep oneself updated and unnecessary fragmentations (within sub-fields) occur.  Surely the cohesiveness of that field is under danger!

\item the phenomenon of ``re-discovering''. Literature survey takes time (lots of time when there are lots of papers to study). Easy way out: cut it! One remains ignorant about what has been published. Then the phenomenon of  ``rediscover it'' happens (most of the time unintentionally, but sometimes intentionally too, just to give a flavor of originality to one's own work)\cite{gian}.

\item fashionable/hot topics sometimes displace some important and core scientific problems. If some of the leading persons moves to a hot topic then majority of the community "jumps" to that hot topic, to ``catch-up'' in the race!  Is it good?  Will spirit of science gain from it? Many fundamental topics has been displaced by fashion! Dirac said: ``Quantum mechanics is a provisional theory.''  And many giants believed so. The problem of quantum mechanics has been displaced!  Practice of it is not in line with the market and fashion approach! The foundations of quantum mechanics is a difficult problem, working on it will not lead to quick and many publications.

There is another aspect to this issue. Consider a person very motivated and very deeply interested in the foundations of quantum mechanics. Will the current "market driven approach" support him? The answer is: "No"! The person will not get publications, and will be filtered out of the system.
Otherwise he/she has to do two types of research in parallel: One computational research which will lead to quicker publications and will support him/her in the system. And in the rest of the time he/she can work on the foundations of  quantum mechanics. But it requires lots of self discipline,  and very wise division of time and energy! Who is finally under loss? Foundations of quantum mechanics may be under loss!

\item loss of diversity; loss of free exploration; loss of independent minds; and loss of passion too!\cite{eva}.

\item unbearable pressure on editors and referees. Most researchers think that their work is excellent, and majority of them do all kinds of efforts and fights with the editorial board members to publish their results. This leads to loss of interest of editors and referees.

\item an introduction of ``selection staff'' (or professional editors) into editorial panels of reputed journals. As expert editors cannot do justice to hundreds of papers submitted per day, a new layer has been introduced in editorial boards of many journals.  This selection staff's duty is to reject! (sometimes 80-90 percent papers are rejected----arguing on the grounds of scope, and type of readership associated with the journal\cite{gian}).  Are the decisions of the selection staff always unbiased? No!  Who is responsible? overproduction!

\item plagiarism and clever plagiarism. There are uncountable cases of plagiarism. Overproduction and rush to publish has led to more and more cases of plagiarism. But thanks to recent powerful softwares that can trace out plagiarism! And hopefully, it is getting under control. However, "clever Plagiarism" is more "elusive". Stealing an already published idea and rewording it in a different form is a case of clever plagiarism\cite{gian}. Unfortunately, no software can catch it! Only moral values, and respect for your fellow researchers in the field can save your science from this evil.  Demand for a large number of publications is one of the reasons for these evil practices in science.

\end{enumerate}

\section{Phenomenon of predatory journals.}

Why are there so many predatory journals in recent times? Because there are so many papers produced that standard time-tested journals are not able to cope up with such floods of papers. They reject. They have to! This prompts paid and predatory journals, and this leads to the degradation of the quality and standards of scientific research.

\begin{figure}[!h]
\begin{center}
\includegraphics[height=6cm]{fig5.pdf}
\caption{Predatory journals and misuse of the open access.}
\end{center}
\end{figure}

Figure (4) depicts how the business of predatory journals grow. Overproduction is the root cause.

\section{Some Indices}

Surely the indices (impact factor of  a journal and h-index of a scientist) were created to improve the quality of scientific work and to improve scientific culture. But people always find loopholes and then these indices can be exploited and can be misused. Let us briefly go into these issues:

\subsection{Impact factor of a journal}
Impact factor (IF) of a journal is defined as:
\begin{equation}
IF = \frac{Total~ number ~of ~citations~ for~ all ~the ~papers ~
published ~by~ a~ journal~ in ~a~ given~ year}{Total ~number~ of ~papers ~ published ~by ~that~ journal~ in~ that~ year}.
\end{equation}

It is a good parameter for a journal, but some commercial journals (whose main aim is to get profit) find ways to manipulate it. As Gianfranco\cite{gian} argues, editors of such journals invite review articles from respected/famous scientists. This leads to citations, and higher IF. In addition, editors tend to select papers that follow fashion, and have the potential to attract audience! This distorts the good scientific culture. This sometimes badly affects journals having more respect to moral values and good scientific culture. Those suffer. It is like one TV channel (which showcases entertainment and hot news) outperforming another TV channel (which is more culturally oriented (like classical stuff)).

So, only solution is that the scientific community should not be too obsessed by the impact factor business of scientific journals.

\subsection{h-index}

Can a single number define the research work of a scientist?  The answer is no! But in typical situations it is a good indicator. Problem comes when this is made the basis of promotions, getting jobs, giving funds, etc. Only a careful study of one's scientific work by an expert can evaluate that person's scientific output honestly. But it takes a lot of time! So, panels consisting of experts consider the easier counting method.  However,  it has obvious flaws. Just for example: h-index of Albert Einstein = 44. There are so many researchers whose h-index is >44! Can we compare? Peter Higgs has h-index=11. There are so many scientists whose h-index is greater than 11! Again, can we compare? No, we cannot compare this way. People who support h-index can argue that these are special cases, not general. Yes, they are right in that. But time tested good indicators: passion; novelty, and depth of works done; diversity; curiosity etc cannot be quantified and h-index misses these. These are the good indicators and are impossible to formulate in numbers!\cite{henri}.

\section{What is good in current times?}

The concept of arXive is the novel one! One gets immediate publicity and the ownership of ideas (the problems of the "theft of ideas" in the pre-arXive era is thus solved to some extent)\cite{henri}. Information technology and communications has made access to scientific publications very easy. Old culture of going to  the library, finding the hard copy of the required journal, finding the required article in it, and then getting it photocopied is all gone. With online access, one can read the soft copy in one's laptop or desktop, or one can get it printed on a printer. In fact, if you have good Internet connectivity, you can sit anywhere in the world and do theoretical research (You need some papers, a pencil, and the Internet, thats all!). The case of experimental research is different. The author would like to make a request to the community that if you find a good/appropriate arXive paper (whether it is published or not) please don't hesitate to cite it in your work.

\section{What should be done? Collective efforts?}

As mentioned before, data shows that the number of active scientists is growing fast (In 1930s: 1 scientist per 10,000 people. In 2013: 1 scientist per 700 people). But, the number of fundamental problems remains the same! Current scientific sociology resolves it by inventing and working on peripheral problems. This will not help science to advance further.

There should be some mechanism of knowing how many scientific positions/post doc positions etc are available (average, say, over a couple of years) and how many phd students are joining the academic research (again, average over a couple of years). This kind of data can balance the job opportunity equations,  and can lessen the pressure to compete. Thus can lessen the pressure to publish.

But it will be great if new riddles are discovered.  Real progress will happen when more scientists are interested in experimental and observational science! Because, it is the experimental and observational science that can discover new scientific problems/riddles worth working on! Scientific community has to seriously think about it! Currently, the division seems very skewed. Also, there has to be very tight correlation between the experimental work and the theoretical work.

The current academic culture of pressure and competition is not making students as true scholars. Current system is producing skilled manpower! Skilled to produce papers! If there is less pressure to publish, students can go to libraries and spend hours reading there, deepening and widening their knowledge of the topics of their interest. More stimulating and motivating environment is required.

How can the excessive load on some international journals be reduced? And how can the phenomenon of predatory journals be controlled. I think due respect to one's national journals must be given. Develop a culture where publishing in a national journal is a matter of prestige. If one of your national journal has got a very bad reputation (for publishing articles which have been rejected by almost all relevant journals), then time is ripe to launch a new journal! And forget the old one. National science academies must look into these matters, and should launch new journals (run by an academic staff), and set high standards of quality, and encourage researchers to send their manuscripts for publication. During jobs, publications in national journals should be given the weightage. It is like ``we love and respect our country''  then why don't  ``we love and respect our national journals?'' This will reduce excessive load on some international journals!

There should be more conferences organized by the academic staff (not by the commercial organizers) with plenty of discussion time. Be aware of predatory conferences (their sole purpose is to get financial profit)\cite{gian}.

\section{Conclusion}
I would like to end with a quote from Gianfranco Pacchioni\cite{gian}: ``The pressure towards achieving new results is daily, and leaves little room, if any at all, to ponder the meaning of what one does.''

Therefore, it is the time to publish less, novel and complete works. Only collective and world wide efforts can solve these grave problems of the practice  of science. Not you and me alone!

\section*{Acknowledgments}

This article is inspired by "Round table on science in h-index era", 22th--24th July, ICAM Global summit, 2020.

\section{Changing culture of the practice of science:  before 20th century.}

Science before the turn of the 20th century was practiced by individual researches mostly motivated by the urge to find some "transcendental  truths" in natural phenomena.  Some of the leading discovers made in the 16th, 17th, 18th, and the 19th century are: Overthrowing of the geocentric model of the universe and the introduction of the heliocentric model by Nicolaus Copernicus (1543);  law of inertia (1613), observational astronomy, and the scientific method of experimentation by Galileo Galilei; Kepler's three laws of the planetary motion (1619);  Many pioneering discoveries in almost all the fields of natural science and mathematics by Issac Newton (Principia 1687);  atomic theory in chemistry by John Dalton (1805); magnetic effects of electric current by Oersted (1820). Sadi Carnot (1824),  Lord Kelvin (1848), Rudolf Clausius (1850) and  others laid the foundations of classical thermodynamics; Statistical mechanics was developed by Maxwell, Boltzmann, and Gibbs; Investigations of Faraday, Maxwell, and many other investigators lead to the theory of electromagnetism (1862);  X-rays were discovered by Wilhelm Roentgen (in 1895);  Radioactivity by Henri Becquerel (in 1896); Electron by J. J. Thomson (in 1897); etc.

However, data shows that only a few thousand of papers were published\cite{gian}. Majority of the papers published had some impact. And the phenomenon of the collaborative research was very rare (most of the time, papers published were single-authored).

Majority of professors/investigators use of publish a couple of dozens of papers in their entire scientific career. Consider the example of Josiah Gibbs (1839-1903) who published less than two dozen papers in his entire career. Gibbs case is a typical case of a scientist of that era (in terms of the number of publications).

\section{Physics in 20th century}

From 1900  to 1950, some of the main discoveries made are: Energy quantum by M.  Planck (1900);  nuclear atom by Rutherford (1911); superconductivity by Onnes (1911); special and general relativity by Einstein (1905, and 1915 respectively);  crystal structures and X-ray diffraction (William and Lawrence Bragg, 1915); Quantum mechanics (M. Planck, N. Bohr, A. Einstein, W. Heisenberg, E. Schroedinger, M Born, P. Dirac and others); Raman effect (by C. V. Raman); Dirac's theory of electrons and positrons; nuclear fission and fusion; Quantum electrodynamics; etc. The later half of the 19th century and the first half of the 20th century can be said "the golden age" of physics.

From 1950  to Current times, some of the main discoveries made are : Nuclear and particle physics; solid state physics; semiconductors and transistors;  electroweak theory and the emergence of standard model; Bardeen-Cooper-Schrieffer (BCS) theory of superconductivity; parity violation; quantum Hall effects; renormalization; QCD; BEC realized; Higgs boson observed; gravitational waves observed, etc.

After WWII, there is an outpouring of the scientific activity. This is due to technological developments, new instruments of investigations, and due to the vast applicability of quantum mechanics into atomic, solid state, and nuclear physics.  This outpouring is evident from how the scientific journals evolved from pre-WWII era to post-WWII era. Take for example, the Phys. Rev.  The Phys Rev started on 1893, as a single volume. In 1958, PRL was introduced; and in 1970, PRA, PRB, PRC, and PRD were introduced.  Currently APS is publishing an array of journals: Phys. Rev. A, B, C, D. Phys. Rev. E; Physics; Phys. Rev. X; Phys. Rev. Applied; Phys. Rev. Fluids; Phys. Rev. Materials; Phys. Rev. Research; PRX Quantum etc.

In the year 1910 Phys. Rev. published articles with 1484 printed pages in total, whereas in 2019, only the Phys. Rev. B (one journal out of the whole array) published 98,398 articles in just six months (from Jan 2011 to June 2019). If each article contains roughly 10 pages, then this amounts to {\it one million} pages!

\section{What has gone wrong in current times?  (Are we going from facts to pathology?)}

There were some thousands of papers published before 1900. By 1960, two million scientific papers have been published! Current times, over two million scientific papers are being published every year! What was produced between 1650 to 1960 (over three hundred years) is now being produced every year (in terms of volume, not number of discoveries)\cite{gian}. But the number of impactful discoveries made in the last half of the 20th century to the current time is much less than the number of papers published in this time period (in comparison to that in the 19th and the first half of the 20th century). Concrete data\cite{fort} shows that there is a linear growth of ideas but there is an exponential growth of the number of papers published (figure 1).

\begin{figure}[h!]
\begin{center}
\begin{tabular}{cc}
\includegraphics[width =5.5cm]{fig1.pdf}&
\includegraphics[width =5.5cm]{fig2.pdf}\\
(a)&(b)\\
\end{tabular}
\end{center}
\caption{(a) Exponential growth of paper production.  (b) Only linear growth of ideas. For actual data and graphs, refer to\cite{fort}.}
\label{f1}
\end{figure}

This strange trend in the data raises serious questions: Are we going from facts to pathology? Is it due to scientific developments only (new fields, new subfields)? Or is it due to the ever growing number of active scientists? What is happening?

\section{Reasons (for floods of papers).}

Simplest reason one can give is that there are more number of active scientists working in very competitive environments (as compared to the situation in the 19th and early 20th century). Data shows that in 1930s there was one scientist per 10,000 people in the whole world. This number in 2013 swelled to one scientist per 700 people\cite{gian}.  This is 14 fold increase! There were 3 lakh scientists in 1935. And this number grows to 10 million in 2016. Even if the "production rate" per scientist (that is the average number of papers published per scientist) remains the same, this sheer rise in the number of active scientists could explain the larger production of the number of published papers\footnote{In old culture, majority of professors/investigators use of publish a couple of dozens of papers in their entire career. Currently, majority of professors/investigators publish over 100 papers in their entire scientific career. So average number of papers published {\it per scientist} has also increased, at least in the physical sciences. Collaborations increase the production rate further.}

But the question raised in the previous section: Why are the number of impactful discoveries made much less than the number of papers published?

It could be due to the following reasons:

\begin{enumerate}

\item There are distortions in the practice of science: Old trend: theoretical understanding of the experimental facts (Max Planck and black body radiation; Einstein and photoelectric effect etc). Passion to find some ``transcendental truths''.  New tread: More theoretical ideas and more ``proposed'' experiments. So more papers! More generalizations and more abstractions Imagine a hypothetical situation, do a calculation, produce a paper! This is due to the pressure to publish. This drastically reduce the ratio of the number of impactful discoveries to the total number of papers published.

\item Importance of numbers of papers published in jobs, getting pdf positions, getting funds etc. This put emphasis on numbers along with quality. As we discuss below, the metrics that measure quality are not foolproof.

\item Sheer scientific culture that promotes the number business (refer to footnote 1).

\item Emergence of competitive environments. This further leads to a constant pressure to build huge groups/collaborations etc. This results in overproduction. There is a trend to follow hot topics/fashionable topics/hypes\cite{eva}. Invent strategies to sell and over-sell (intense fights with editors and referees).

\item Old times: Mostly single author or couple of authors; Now majority of papers come out of  collaborations (Collaborations sometimes are required). Collaborations has the potential to increase volume, but whether it leads to an out-put of a great quality publications? Surely a point to ponder over!\cite{henri}.

\item Pressure to publish in reputed journals, rejections, and then the phenomenon of predatory journals: you pay and publish. Easy and low quality publications. Loss of impactful science. And most importantly the misuse of the concept of ``open access".

\item Core open scientific problems are limited in number. There are problems posed by ``Nature" and then there are "man-made" problems. The main scientific problems in theoretical physics can be counted: Beyond standard model physics; What is small CP violation trying to tell us? GUT: ``Wedding" of QCD with electroweak theory; Neutrino mass problem; Cuprate high temperature superconductivity; Physics of heavy fermions metals etc. Electroweak theory is renormalizable, but there is no Quantum theory of gravity which is renormalizable; Can there be a theory using which one can compute the mass of an electron? The problem of dark matter etc.

\begin{figure}[!h]
\begin{center}
\includegraphics[height=5cm]{fig3.pdf}
\caption{Sets of peripheral problems.}
\end{center}
\label{f2}
\end{figure}

To meet the high demands of competition and job issues one is forced to invent easy and peripheral problems (refer to figure (2)).  Working on peripheral problems leads to publications. It can be helpful to a student in his/her career. It can be helpful in promotions/jobs. It may be helpful in getting some awards, fellowship of academies etc. But it leads to a loss of the impactful science. It takes years to solve a hard problem, but an easy problem can be solved in lesser time. The current academic culture supports the latter one, unfortunately!

\item There are experimentally underdetermined problems. When the available input experimental parameters required to validate a theory are much less in number, then there is a large parameter space to play with. Many theoretical models/scenario are possible (as the problem remains observationally/experimentally under-determined). This may lead to burden to literature. Care must be exercised when publishing in such areas.

\item Addiction to publish: I know a person who gets some sort of irritation (mental upsetting) if he does not get a paper in a month's time.
So every month he must get a paper (low quality, high quality, through students, through collaborators, it does not matter!).
Will that large volume of papers (containing solutions of easy and man-made problems) be useful?
Will this not burden referees/editors/and students who are just entering the field?

\item Vicious cycle:

The more production of papers leads to getting more funding and further leading to bigger collaborations/groups. This in turn leads to a greater production of papers. Thus, a ``vicious cycle" starts! (refer to figure 3).

\begin{figure}[!h]
\begin{center}
\includegraphics[height=5cm]{fig4.pdf}
\caption{A ``vicious cycle".}
\end{center}
\label{f2}
\end{figure}

But there is a difference between an industrial production and scientific solution of a riddle posed by Nature. It is the duty of the funding agencies that they should not put emphasis on the quantity, and while measuring the quality of the scientific output through the standard measures, they must realize that these measures (metrics, impact factors of journals etc) are themselves are distorting the good culture (as discussed below). Actual reading of the scientific papers by an expert panel can help save the situation.

\end{enumerate}

\section{Consequences (on individuals; on sociology; and on science itself).}

Emphasis on the number of publications (or culture of producing a large number of publications) leads to

\begin{enumerate}

\item a shift of focus from central difficult problems to more peripheral and easy problems.  It is easy to solve an easy problem and more difficult to solve a difficult problem.

\item loss of cohesiveness of a given field and unnecessary fragmentations within the subfields.  Leads to difficulties for a beginner to understand and penetrate into a given field (One has to deal with a huge volume of literature, much of which may not be relevant). The human capacity for information processing is limited. There are physiological limitations. One can read a couple of papers published per day in one's special area (refer to arXive). How can one even read abstracts if hundreds of papers are being published in one's research topic? It will be difficult to keep oneself updated and unnecessary fragmentations (within sub-fields) occur.  Surely the cohesiveness of that field is under danger!

\item the phenomenon of ``re-discovering''. Literature survey takes time (lots of time when there are lots of papers to study). Easy way out: cut it! One remains ignorant about what has been published. Then the phenomenon of  ``rediscover it'' happens (most of the time unintentionally, but sometimes intentionally too, just to give a flavor of originality to one's own work)\cite{gian}.

\item fashionable/hot topics sometimes displace some important and core scientific problems. If some of the leading persons moves to a hot topic then majority of the community "jumps" to that hot topic, to ``catch-up'' in the race!  Is it good?  Will spirit of science gain from it? Many fundamental topics has been displaced by fashion! Dirac said: ``Quantum mechanics is a provisional theory.''  And many giants believed so. The problem of quantum mechanics has been displaced!  Practice of it is not in line with the market and fashion approach! The foundations of quantum mechanics is a difficult problem, working on it will not lead to quick and many publications.

There is another aspect to this issue. Consider a person very motivated and very deeply interested in the foundations of quantum mechanics. Will the current "market driven approach" support him? The answer is: "No"! The person will not get publications, and will be filtered out of the system.
Otherwise he/she has to do two types of research in parallel: One computational research which will lead to quicker publications and will support him/her in the system. And in the rest of the time he/she can work on the foundations of  quantum mechanics. But it requires lots of self discipline,  and very wise division of time and energy! Who is finally under loss? Foundations of quantum mechanics may be under loss!

\item loss of diversity; loss of free exploration; loss of independent minds; and loss of passion too!\cite{eva}.

\item unbearable pressure on editors and referees. Most researchers think that their work is excellent, and majority of them do all kinds of efforts and fights with the editorial board members to publish their results. This leads to loss of interest of editors and referees.

\item an introduction of ``selection staff'' (or professional editors) into editorial panels of reputed journals. As expert editors cannot do justice to hundreds of papers submitted per day, a new layer has been introduced in editorial boards of many journals.  This selection staff's duty is to reject! (sometimes 80-90 percent papers are rejected----arguing on the grounds of scope, and type of readership associated with the journal\cite{gian}).  Are the decisions of the selection staff always unbiased? No!  Who is responsible? overproduction!

\item plagiarism and clever plagiarism. There are uncountable cases of plagiarism. Overproduction and rush to publish has led to more and more cases of plagiarism. But thanks to recent powerful softwares that can trace out plagiarism! And hopefully, it is getting under control. However, "clever Plagiarism" is more "elusive". Stealing an already published idea and rewording it in a different form is a case of clever plagiarism\cite{gian}. Unfortunately, no software can catch it! Only moral values, and respect for your fellow researchers in the field can save your science from this evil.  Demand for a large number of publications is one of the reasons for these evil practices in science.

\end{enumerate}

\section{Phenomenon of predatory journals.}

Why are there so many predatory journals in recent times? Because there are so many papers produced that standard time-tested journals are not able to cope up with such floods of papers. They reject. They have to! This prompts paid and predatory journals, and this leads to the degradation of the quality and standards of scientific research.

\begin{figure}[!h]
\begin{center}
\includegraphics[height=6cm]{fig5.pdf}
\caption{Predatory journals and misuse of the open access.}
\end{center}
\end{figure}

Figure (4) depicts how the business of predatory journals grow. Overproduction is the root cause.

\section{Some Indices}

Surely the indices (impact factor of  a journal and h-index of a scientist) were created to improve the quality of scientific work and to improve scientific culture. But people always find loopholes and then these indices can be exploited and can be misused. Let us briefly go into these issues:

\subsection{Impact factor of a journal}
Impact factor (IF) of a journal is defined as:
\begin{equation}
IF = \frac{Total~ number ~of ~citations~ for~ all ~the ~papers ~
published ~by~ a~ journal~ in ~a~ given~ year}{Total ~number~ of ~papers ~ published ~by ~that~ journal~ in~ that~ year}.
\end{equation}

It is a good parameter for a journal, but some commercial journals (whose main aim is to get profit) find ways to manipulate it. As Gianfranco\cite{gian} argues, editors of such journals invite review articles from respected/famous scientists. This leads to citations, and higher IF. In addition, editors tend to select papers that follow fashion, and have the potential to attract audience! This distorts the good scientific culture. This sometimes badly affects journals having more respect to moral values and good scientific culture. Those suffer. It is like one TV channel (which showcases entertainment and hot news) outperforming another TV channel (which is more culturally oriented (like classical stuff)).

So, only solution is that the scientific community should not be too obsessed by the impact factor business of scientific journals.

\subsection{h-index}

Can a single number define the research work of a scientist?  The answer is no! But in typical situations it is a good indicator. Problem comes when this is made the basis of promotions, getting jobs, giving funds, etc. Only a careful study of one's scientific work by an expert can evaluate that person's scientific output honestly. But it takes a lot of time! So, panels consisting of experts consider the easier counting method.  However,  it has obvious flaws. Just for example: h-index of Albert Einstein = 44. There are so many researchers whose h-index is >44! Can we compare? Peter Higgs has h-index=11. There are so many scientists whose h-index is greater than 11! Again, can we compare? No, we cannot compare this way. People who support h-index can argue that these are special cases, not general. Yes, they are right in that. But time tested good indicators: passion; novelty, and depth of works done; diversity; curiosity etc cannot be quantified and h-index misses these. These are the good indicators and are impossible to formulate in numbers!\cite{henri}.

\section{What is good in current times?}

The concept of arXive is the novel one! One gets immediate publicity and the ownership of ideas (the problems of the "theft of ideas" in the pre-arXive era is thus solved to some extent)\cite{henri}. Information technology and communications has made access to scientific publications very easy. Old culture of going to  the library, finding the hard copy of the required journal, finding the required article in it, and then getting it photocopied is all gone. With online access, one can read the soft copy in one's laptop or desktop, or one can get it printed on a printer. In fact, if you have good Internet connectivity, you can sit anywhere in the world and do theoretical research (You need some papers, a pencil, and the Internet, thats all!). The case of experimental research is different. The author would like to make a request to the community that if you find a good/appropriate arXive paper (whether it is published or not) please don't hesitate to cite it in your work.

\section{What should be done? Collective efforts?}

As mentioned before, data shows that the number of active scientists is growing fast (In 1930s: 1 scientist per 10,000 people. In 2013: 1 scientist per 700 people). But, the number of fundamental problems remains the same! Current scientific sociology resolves it by inventing and working on peripheral problems. This will not help science to advance further.

There should be some mechanism of knowing how many scientific positions/post doc positions etc are available (average, say, over a couple of years) and how many phd students are joining the academic research (again, average over a couple of years). This kind of data can balance the job opportunity equations,  and can lessen the pressure to compete. Thus can lessen the pressure to publish.

But it will be great if new riddles are discovered.  Real progress will happen when more scientists are interested in experimental and observational science! Because, it is the experimental and observational science that can discover new scientific problems/riddles worth working on! Scientific community has to seriously think about it! Currently, the division seems very skewed. Also, there has to be very tight correlation between the experimental work and the theoretical work.

The current academic culture of pressure and competition is not making students as true scholars. Current system is producing skilled manpower! Skilled to produce papers! If there is less pressure to publish, students can go to libraries and spend hours reading there, deepening and widening their knowledge of the topics of their interest. More stimulating and motivating environment is required.

How can the excessive load on some international journals be reduced? And how can the phenomenon of predatory journals be controlled. I think due respect to one's national journals must be given. Develop a culture where publishing in a national journal is a matter of prestige. If one of your national journal has got a very bad reputation (for publishing articles which have been rejected by almost all relevant journals), then time is ripe to launch a new journal! And forget the old one. National science academies must look into these matters, and should launch new journals (run by an academic staff), and set high standards of quality, and encourage researchers to send their manuscripts for publication. During jobs, publications in national journals should be given the weightage. It is like ``we love and respect our country''  then why don't  ``we love and respect our national journals?'' This will reduce excessive load on some international journals!

There should be more conferences organized by the academic staff (not by the commercial organizers) with plenty of discussion time. Be aware of predatory conferences (their sole purpose is to get financial profit)\cite{gian}.

\section{Conclusion}
I would like to end with a quote from Gianfranco Pacchioni\cite{gian}: ``The pressure towards achieving new results is daily, and leaves little room, if any at all, to ponder the meaning of what one does.''

Therefore, it is the time to publish less, novel and complete works. Only collective and world wide efforts can solve these grave problems of the practice  of science. Not you and me alone!

\section*{Acknowledgments}

This article is inspired by "Round table on science in h-index era", 22th--24th July, ICAM Global summit, 2020.

Plea to publish less


Circular Polarization in Pulsar Integrated Profiles: Updates

Abstract: We update the systematic studies of circular polarization in integrated pulse profiles by Han et al (1998). Data of circular polarization profiles are compiled. Sense reversals can occur in core or cone components, or near the intersection between components. The correlation between the sense of circular polarization and the sense of position angle variation for conal-double pulsars is confirmed with a much large database. Circular polarization of some pulsars has clear changes with frequency. Circular polarization of millisecond pulsars is marginally different from that of normal pulsars.

\section{Introduction}
\label{sect:intro}

Polarization properties of pulsars are very important for the
understanding of the geometry and emission mechanism of pulsars.
Generally, the degree of circular polarization is low. Many
pulsars show sense reversal in their circular polarization near
the middle of the pulse. The sense reversals sometimes are
associated with the orthogonal polarization modes (\cite{C78};
\cite{S84a}). In some pulsars, the circular polarization keeps the
same sense through the whole profile. Two obvious types of
circular polarization are identified by \cite{RR90}, namely:
$antisymmetric$, where the circular polarization changes sense
near the center of the profile, and $symmetric$, where the
circular polarization remains the same sense through the whole
profile. \cite{Han98} collected the published polarization
profiles and reviewed the characteristics of circular polarization
in pulsar integrated profiles, discovered a correlation between
the sense of circular polarization and the sense of position angle
(PA) variation for conal-double pulsars, and rebutted the
correlation between the sense reversal of circular polarization
near the core components and the sense of PA.

There are two possible origins of circular polarization of
pulsars: either intrinsic to the emission properties and dependent
on the emission mechanism, or generated by propagation effects.
For example, \cite{ML04} discussed possible circular polarization
induced by intrinsically relativistic effects of pulsar plasma.
\cite{melrose03} reviewed the properties of intrinsic circular
polarization and circular polarization due to cyclotron
absorption, and presented a plausible explanation of circular
polarization in terms of propagation effects in an inhomogeneous
birefringent plasma. \cite{LP99} considered that the rotation of
the magnetosphere gives rise to wave mode coupling in the
polarization-limiting region, which can result in circular
polarization in linearly polarized normal waves.

A large sample of normal pulsars and millisecond pulsars has been
observed for polarization (\cite{GL98}; \cite{S99}; \cite{W99};
\cite{W04}; \cite{H06}), especially at multiple frequencies. The
data have increased by a factor of about three over that in
\cite{Han98}. So, it is the time to update the database of pulsar
circular polarization and recheck the conclusions of that paper.

\section{Dataset}

Polarization profiles of pulsars are collected and cataloged if
the circular polarization has a good signal-to-noise ratio.
Circular polarization is defined observationally by the Stokes
parameter, $V= I_L-I_R$. The rotational sense of $V$, the
percentage (=$\langle V\rangle/S$, where $S$ is the mean total flux density), and
absolute circular polarization percentage, $\langle|V|\rangle/S$, the variation
of PA, and observation frequency are all included in Table~1 (This
is only part of Table~1. For the full Table~\ref{tb:big_tab} see
http://www.chjaa.org/2006v1n2/ for electronic version), which has
the same format as table~A1 in \cite{Han98}.

\begin{table}[h!!]

\centering \begin{minipage}{110mm} \caption{A Summary of Pulsar
Circular Polarization Observations}
\label{tb:big_tab}\end{minipage}

\scriptsize
\begin{tabular}{lllrrclclp{30mm}}
\hline\noalign{\smallskip}
 PSR J & PSR B & $V$ &\multicolumn{1}{c}$\langle|V|\rangle/S$ &  \multicolumn{1}{c}$\langle V\rangle/S$ & $\sigma$ &
 PA & \multicolumn{1}{r}{Freq.}  & Ref. & Comments \\
   $$  &       &     & (\%)      & (\%)  & (\%)     &    &   (MHz)                    &      &   \\
\hline\noalign{\smallskip}
0030$+$0454   &           &$  --      $&  &$   $&  &dec &  433 &L00  &MSP. $--$ for main comp \\
0034$-$0534   &           &$  --      $&18&$   $&  &xx  &  410 &S99  &PA swing not clear \\
0034$-$0721   & 0031$-$07   &$  +       $&10&$  5$& 1&dec &  234 &G98  & \\
              &           &$  +       $&  &$   $&  &dec &  268 &R83  &$+V$ in 2nd half \\
              &           &$  +       $&  &$   $&  &dec &  328 &S05  &$+V$ in 2nd half \\
              &           &$  -       $& 6&$ -6$& 0&inc &  410 &G98  &$-V$ in 1st half \\
              &           &$  +       $& 5&$  4$& 0&i+d &  606 &G98  &$+V$ in 2nd half \\
              &           &$  +       $&  &$   $&  &dec & 4850 &S05  &$+V$ in 2nd half \\
0040$+$5716   & 0037$+$56   &$  -       $&18&$-17$& 1&inc &  610 &G98  &s/n good\\
0045$-$7319   &           &$  -       $&27&$-17$& 7&xx  &  661 &C01  &strong cp \\
0048$+$3412   & 0045$+$33   &$  -       $&  &$   $&  &dec &  430
&W04  &PA not very clear. low linear
polarization\\\noalign{\smallskip} \hline
\end{tabular}
\end{table}

\section{Main properties of circular polarization}

\subsection{Sense
Reversal of Circular Polarization}

\subsubsection{Sense reversals associated with core components}

Using a sample of 25 pulsars, \cite{RR90} found that change of
circular polarization from left hand (positive) to right hand
(negative) is associated with decreasing PA, and that from right
hand to left hand is associated with increasing PA. Gould (1994)
and \cite{Han98} found many contrary examples, which leads
\cite{Han98} to conclude that no correlation exists between the
sense of the sign change of circular polarization and the sense of
variation of PA.

Here we use a very large sample of pulsar data and confirm the
conclusion of non-correlation. Table~\ref{tb:core_rev} lists all
pulsars with sense reversal of circular polarization in the core
component, and 19 pulsars in the first and fourth part of
Table~\ref{tb:core_rev} support the existence of the correlation,
but 20 pulsars in the second and third part do not.

\begin{table}

\centering
\begin{minipage}{135mm}
 \caption{Sense Reversals of Circular Polarization
Associated with Core Components} \label{tb:core_rev}
\end{minipage}

\scriptsize

\begin{minipage}{145mm}

\begin{minipage}[t]{70mm}
\tabcolsep 1mm

\begin{tabular}{lclcr}
\hline\noalign{\smallskip}
PSR Name     &$V=LH-RH$& PA  & \multicolumn{1}{c}{Freq.} & Ref.  \\
              &         &     &  (MHz)                    &       \\
              \hline\noalign{\smallskip}
J0454$+$5543  &  $+/-$  & dec &  610                      & G98   \\
J0809$-$4753  &  $+/-$  & dec &  1335                     & H06   \\
J1001$-$5507$^a$&  $+/-$  & dec &  1351                     & H06   \\
J1239$+$2453  &  $+/-$  & dec &  1418                     & W99   \\
J1509$+$5531  &  $+/-$  & dec &  610                      & G98   \\
J1534$-$5334  &  $+/-$  & dec &  1612                     & Ma80  \\
J1740$+$1311  &  $+/-$  & dec &  1418                     & W99   \\
J1801$-$0357$^b$&  $+/-$  & dec &  661                      & M98   \\
J1823$+$0550  &  $+/-$  & dec &  1408                     & G98   \\
J1900$-$2600  &  $+/-$  & dec &  1408                     & G98   \\
J1901$+$0331  &  $+/-$  & dec &  1418                     & W99   \\
J1903$-$0632  &  $+/-$  & dec &  610                      & G98   \\
J1946$-$2913  &  $+/-$  & dec &  1327                     & H06   \\
J2048$-$1616  &  $+/-$  & dec &  1420                     & LM88  \\
J2113$+$4644  &  $+/-$  & dec &  610                      & G98
\\[1mm]
J0826$+$2637m &  $+/-$  & inc &  1400                     & R89   \\
J1512$-$5759  &  $+/-$  & inc &  1319                     & H06   \\
J1600$-$3053$^c$ &  $+/-$  & inc &  1373                     & O04   \\
J1604$-$4909  &  $+/-$  & inc &  658                      & M98 \\
J1733$-$2228$^b$ &  $+/-$  & inc &  610                   & G98   \\
\hline\\
\end{tabular}
\end{minipage}~~
\begin{minipage}[t]{70mm}
\tabcolsep 1mm
\begin{tabular}{lclcr}
\hline\noalign{\smallskip}
PSR Name     &$V=LH-RH$& PA  & \multicolumn{1}{c}{Freq.} & Ref.  \\
              &         &     &  (MHz)                    &       \\
              \hline\noalign{\smallskip}
J1910$-$0309  &  $+/-$  & inc &  1408                     & G98   \\
J1926$+$0431  &  $+/-$  & inc &  1418                     & W99   \\
J2004$+$3137  &  $+/-$  & inc &  1400                      & R89   \\
J2006$-$0807  &  $+/-$  & inc &  1408                     & G98
\\[1mm]
J0332$+$5434  &  $-/+$  & dec &  1408                     & G98   \\
J0437$-$4715  &  $-/+$  & dec &  1512                     & N97  \\
J0944$-$1354  &  $-/+$  & dec &  409                      & LM88  \\
J1326$-$5859  &  $-/+$  & dec &  955                      & v97   \\
J1456$-$6843  &  $-/+$  & dec &  649                      & Mc78  \\
J1527$-$5552  &  $-/+$  & dec &  658                      & M98   \\
J1537$+$1155  &  $-/+$  & dec &  430                      & A96   \\
J1544$-$5308  &  $-/+$  & dec &  658                      & M98   \\
J1752$-$2806  &  $-/+$  & dec &  1408                     & G98   \\
J1852$-$2610  &  $-/+$  & dec &  434                      & M98   \\
J1909$+$0254$^b$  &  $-/+$  & dec &  610                      &
G98   \\[1mm]
J0452$-$1759  &  $-/+$  & inc &  408                      & LM88  \\
J1703$-$3241  &  $-/+$  & inc &  950                      & v97   \\
J2144$-$3933  &  $-/+$  & inc &  659                      & M98   \\
J2325$+$6316$^c$ &  $-/+$  & inc &  1642                     & G98
\\[2mm]
\hline\\
\end{tabular}
\end{minipage}
 { $^a$ New high resolution observation show decreasing PA.
Old data are confused by orthogonal polarization modes. \\
$^b$ PA not clear.\\
$^c$ Not so sure if sense reversal happens in core.\\
}
\end{minipage}

\centering

\begin{minipage}{100mm}
\fns \caption{Sense Reversals not Associated with Core Components}
\label{tb:other_rev}\end{minipage}

\setlength{\tabcolsep}{0.9mm}

\scriptsize
\begin{tabular}{lclcll}
\hline\noalign{\smallskip}
PSR Name       &$V=LH-RH$& PA  & \multicolumn{1}{c}{Freq.} & Ref. & Comments \\
              &         &     &  (MHz)                    &      &  \\
\hline\noalign{\smallskip}
J1651$-$4246  &$  +/-  $& dec & 1349        & H06  & intersection between two comp\\
J1807$-$0847  &$  +/-  $& dec & 1408        & G98  & cone\\
J1857$+$0943m &$  +/-  $& dec & 1400        & S86  & intersection between two comp\\
J1907$+$4002  &$  +/-  $& dec & 1408        & G98  & intersection between two comp?\\
J2324$-$6054  &$  +/-  $& dec & 1335        & H06  & intersection between two comp\\[1mm]
J0152$-$1637  &$  +/-  $& inc & 660         & Q95  & intersection between two comp\\
J0612$+$3721  &$  +/-  $& inc & 610         & G98  & associated with orthogonal polarization modes\\
J0653$+$8051  &$  +/-  $& inc & 610         & G98  & cone? \\
J0738$-$4042  &$  +/-  $& inc & 1351        & H06  & associated with orthogonal polarization modes \\
J0837$+$0610  &$  +/-  $& inc & 800         & S84b & leading cone  \\
J1913$-$0440  &$  +/-  $& inc & 408         & G98  & associated with orthogonal polarization modes \\
J2022$+$2854  &$  +/-  $& inc & 800         & S84b & leading cone  \\
J2053$-$7200  &$  +/-  $& inc & 658         & M98  & intersection between two comp\\
              &$  -/+  $& inc & 1440        & Q95  & intersection between two comp\\
J2145$-$0750  &$  +/-  $& inc & 610         & S99  & cone \\
J2326$+$6113  &$  +/-  $& inc & 1408        & G98  & intersection between two comp?\\
J1708$-$3426  &$  +/-  $& ??  & 1329        & H06  & intersection between two comp\\[1mm]
J0133$-$6957  &$  -/+  $& dec & 658         & M98  & intersection between two comp\\
J1041$-$1942  &$  -/+  $& dec & 1642        & G98  & leading cone  \\
J1045$-$4509  &$  -/+  $& dec & 1373        & O04  & cone? \\
J1559$-$4438  &$  -/+  $& dec & 1490        & M98  & intersection between trailing two comp\\
J1614$+$0737  &$  -/+  $& dec & 1418        & W99  & intersection between two comp\\
J1705$-$1906m &$  -/+  $& dec & 1642        & G98  & intersection between two comp\\
J1751$-$4657  &$  -/+  $& dec & 434         & M98  & leading cone  \\
J1916$+$0951  &$  -/+  $& dec & 610         & G98  & leading cone  \\
J1935$+$1616  &$  -/+  $& dec & 1408        & G98  & intersection between two comp\\[1mm]
J0255$-$5304  &$  -/+  $& inc & 1359        & H06  & second cone \\
J0502$+$4654  &$  -/+  $& inc & 1408        & G98  & intersection between two comp\\
J0601$-$0527  &$  -/+  $& inc & 408         & G98  & intersection between two comp\\
J0941$-$5244  &$  -/+  $& inc & 1319        & H06  & intersection between two comp\\
J1224$-$6407  &$  -/+  $& inc & 1319        & H06  & intersection between two leading comp  \\
J1328$-$4357  &$  -/+  $& inc & 435         & M98  & intersection between two leading comp  \\
J1921$+$2153  &$  -/+  $& inc & 1418        & W99  & cone\\
J1602$-$5100  &$  -/+  $& ??  & 950         & v97  & intersection between two leading comp  \\
\hline\noalign{\smallskip}

 \end{tabular}
\end{table}

\subsubsection{Sense reversals outside core}

Many sense reversals of circular polarization are detected outside
of the center region of the profile, thus not associated with core
components but with cone components or near the conjunction of
components. Table~\ref{tb:other_rev} lists such pulsars with sense
reversals in the other part of the pulse profile.

\subsubsection{Sense reversals associated with orthogonal polarization modes}

We checked possible association of sense reversal of circular
polarization with orthogonal polarization modes of the
polarization angle. Among 81 pulsars with sense reversals in $V$
with clear PA variation curves, about 31 show the association. For
example, the PA jumps about $90^\circ$ seen in PSRs J1900$-$2600,
J0601$-$0527 and J0437$-$4715 at almost all the observed
frequencies (\cite{M98}; \cite{GL98}; \cite{N97}), near the phase
of a sense transition of circular polarization. A few pulsars show
two sense reversals across the profile, as shown in Figure~1 for
PSR J2037$+$1942 which has sense reversals associated with the
peaks of two components. The orthogonal polarization modes occur
in the first component (\cite{W99}). The polarization curve thus
does not have a good $S-$shape.

\begin{figure}[h!!]

  \centering
\includegraphics[width=80mm]{figure1.eps}

\vspace{-3mm}

\caption{Polarization profile of PSR J2037+1942 at 1418~MHz (from
\cite{W99}). The total (higher full line), linear polarized (lower
full line) and circular polarized (dotted line) flux densities are
displayed in the upper panel. The lower panel shows the PA curve.}
  \label{fg:two_rev}
\end{figure}

\subsection{Circular Polarization of Conal-double Pulsars}

Using the polarization data of a sample of 20 conal-double pulsars
available at that time, \cite{Han98} found a strong correlation
between the sense of PA sweep and the sense of circular
polarization for conal-double pulsars, namely, a decrease of PA
accompanies with left-hand  circular polarization of conal
components, and an increase of PA with the right-hand.
Occasionally, sense reversal is observed in one cone component of
profiles.

Now, using a larger sample of 36 pulsars, the correlation is
solidly confirmed.  Table~\ref{tb:cone_double} lists all
conal-double pulsars with good measurements of circular
polarization and PA.

\begin{table

\centering

\begin{minipage}{110mm}
\caption{Conal-double Pulsars with Significant Circular
Polarization} \label{tb:cone_double}
\end{minipage}
 \scriptsize
\tabcolsep 4mm
\begin{tabular}{llccl}
\hline\noalign{\smallskip}
PSR Name & PA  &  \multicolumn{2}{c}{Sign of $V$} & Ref. \\
            &     & Comp 1 & Comp 2 &    \\
\hline\noalign{\smallskip}
J0151$-$0635& inc & $-$ & $-$ & LM88, G98      \\
J0528$+$2200& inc & $-$ & $-$ & S84a, R83      \\
J0653$+$8051& inc &$+/-$& $-$ & G98           \\
J0754$+$3231& inc & $-$ & $-$ & R89, G98       \\
J0820$-$1350& inc & $-$ & $-$ & v97, Q95, B87   \\
J0837$+$0610& inc & $-$ & $-$ & Mc78, S84a, G98   \\
            &     &$+/-$& $-$ & S84b \\
J0959$-$4809& inc & $-$ & $-$ & H06           \\
J1015$-$5719& inc & $-$ & $-$ & H06           \\
J1110$-$5637& inc &$\cdot$& $-$ & H06         \\
J1136$+$1551& inc & $-$ & $-$ & Mc78, S84a, G98  \\
J1137$-$6700& inc & $-$ & $-$ & H06           \\
J1159$-$7910?& inc &$\cdot$& $-$ & H06         \\
J1420$-$6048& inc & $-$ & $-$ & R01           \\
J1906$+$0641& inc & $-$ & $\cdot$& W99        \\
J1915$+$1606& inc &$-/+$& $-$ & C90           \\
J1921$+$2153& inc &$-/+$& $-$ & W99           \\
J1954$+$2923& inc & $-$ & $-$ & G98             \\
J2022$+$2854& inc & $-$ & $-$ & C78, S84a, W99      \\
J2046$+$1540& inc & $-$ & $-$ & G98 \\
J2053$-$7200& inc &$+/-$ &$-$ & Q95, M98 \\
            &     &$-/+$ &$-$ & Q95, H06 \\
J2124$+$1407& inc & $-$ & $-$ & W04 \\
\hline\noalign{\smallskip}
J0055$+$5117& dec & $+$ & $+$ & G98 \\
J0304$+$1932& dec & $+$ & $+$ & R83, R89, W99\\
J0631$+$1036& dec & $+$ & $+$ & Z96 \\
J1041$-$1942& dec &$-/+$& $+$ & LM88, G98 \\
J1123$-$4844& dec & $+$ & $+$ & M98 \\
J1302$-$6350& dec & $+$ &$\cdot$& MJ95 \\
J1345$-$6115& dec &$+$&$\cdot$& H06 \\
J1527$-$3931& dec & $+$ & $+$ & M98 \\
J1731$-$4744& dec & $+$ &$\cdot$& H77, Mc78, v97\\
J1751$-$4657& dec &$-/+$& $+$ & M98 \\
J1803$-$2137& dec & $+$ & $+$ & G98 \\
J1826$-$1344& dec & $+$ & $+$ & G98 \\
J2055$+$2209& dec & $+$ & $+$ & G98 \\
J2324$-$6054& dec & $+$ &$\cdot$& Q95\\
J2346$-$0609& dec &$\cdot$&$+$& M98 \\
\hline\noalign{\smallskip} \multicolumn{4}{l}{ ?~ Not so sure for
conal-double pulsar.}
\\
\end{tabular}
\end{table}

We also checked if there is any correlation between the
polarization percentage and the maximum sweep rate of PA. Ideally,
the PA should follow the $S$-shaped curve across the pulse profile
as described by the rotating vector model (\cite{RC69}). The
maximum rate of polarization sweep, which occurs when the line of
sight passes closest to the magnetic axis, is given by
$\left(\frac{d\psi}{d\phi}\right)_m
=\frac{\sin\alpha}{\sin\beta}$, where $\psi$ is the PA, $\phi$ is
the longitude, $\alpha$ is the inclination of the magnetic axis to
the rotation axis, and $\beta$ is the impact parameter given by
$\beta=\zeta-\alpha$, where $\zeta$ is the inclination of the
observer direction to the rotation axis. The value of
$\left|\frac{d\psi}{d\phi}\right|_m$ very sensitively depends on
$\left|\beta\right|$. Smaller $\left|\beta\right|$, i.e. the
magnetic axis closer to the observer direction, gives a larger
$\left|\frac{d\psi}{d\phi}\right|_m$. Figure~2 shows the
relationship between $\langle V\rangle/S$~ and
$\left(\frac{d\psi}{d\phi}\right)_m$ at 1400~MHz for conal-double
pulsars. There are only 27 pulsars in Figure~2 because some
pulsars have not been observed near 1400~MHz or the observed PA
does not have a good enough $S$-shaped curve to estimate
$\left|\frac{d\psi}{d\phi}\right|_m$. Pulsars are located in the
second and forth quadrants in Figure~2, which confirms the
correlation between the sign of PA swing and the sense of $V$.
Furthermore, we noticed that $|\langle V\rangle|/S$ tends to
decrease with $\left|\frac{d\psi}{d\phi}\right|_m$.

\begin{figure}

\vs
 \centering
\includegraphics[angle=-90,width=70mm]{figure2a.eps}
\includegraphics[angle=-90,width=70mm]{figure2b.eps}

  \caption{ The maximum polarization sweep rate
  $\left(\frac{d\psi}{d\phi}\right)_m$ is related to the fractional circular
  polarization $\langle V\rangle/S$~ at 1400~MHz. The data of $\left(\frac{d\psi}{d\phi}\right)_m$
  were taken from Gould (1994), Han et al.\ (2006), or estimated by ourselves.}
  \label{fg:slope_cp}
\end{figure}

\subsection{Circular Polarization with Frequency}

The circular polarization of some pulsars clearly changes with
frequency. von Hoensbroech \& Lesch (1999) showed three pulsars
with a trend of increasing circular polarization with frequency,
which was interpreted in terms of propagating natural wave modes
in pulsar magnetosphere.

The variation of degree of circular polarization with frequency is
very different from pulsar to pulsar.  Figure~3 shows eight good
examples: Four of the pulsars show their circular polarizations
increasing with frequency, but the other four in the latter part
of the figure show a decrease.

In some pulsars, the sign of sense reversal clearly changes with
frequency. PSR~J2053$-$7200
shows a sense reversal near the intersection of two components
from the left-hand to right-hand at low frequencies (\cite{Q95};
\cite{M98}; \cite{v97}), but from the right-hand to left-hand at
high frequencies (\cite{Q95}; \cite{H06}).

   \begin{figure}
   \centering

\includegraphics[angle=-90,width=135mm]{figure3.eps}

\begin{minipage}{130mm}
\caption{Eight pulsars with clear variation of circular
polarization with frequency.}
   \label{fg:absvfreq}\end{minipage}
   \end{figure}

\subsection{Circular Polarization in Normal Pulsars and Millisecond Pulsars}

Compared to normal pulsars, millisecond pulsars have weaker
surface magnetic fields, wider profiles, and a different profile
dependence on frequency (\cite{Kramer98}; \cite{Kramer99}). Though
their PA variations are often more complicated, most of them
appear to follow the rotating vector model. The basic radio
emission mechanism may be similar for millisecond pulsars and
normal pulsars. \cite{X98} found that the fractional absolute
circular polarization is higher for millisecond pulsars than for
normal pulsars, based on observations at 1410~MHz.

\begin{table}

\centering

\begin{minipage}{120mm}
\caption{ Circular Polarization of Millisecond Pulsars Near
1400~MHz}
  \label{tb:MSP}\end{minipage}

\tabcolsep 4mm

\scriptsize
  \begin{tabular}{lrrrrcl}
  \hline\noalign{\smallskip}
PSR Name & Period & \multicolumn{1}{c}{$\langle|V|\rangle/S$} & \multicolumn{1}{c}{$\langle V\rangle/S$}
& \multicolumn{1}{c}{Err.}& \multicolumn{1}{c}{Freq.} & Ref. \\
  & (ms) & (\%) & (\%) & (\%) & (MHz) &  \\
  \hline\noalign{\smallskip}
J0437$-$4715 &  5.76 & 11 &$  -5 $&  1 & 1512 & N97\\
J0711$-$6830 &  5.49 & 16 &$     $&    & 1405 & O04\\
J1022$+$1001 & 16.45 & 10 &$     $&    & 1405 & O04\\
J1045$-$4509 &  7.47 & 14 &$     $&    & 1373 & O04\\
J1600$-$3053 &  3.60 &  3 &$     $&    & 1373 & O04\\
J1603$-$7202 & 14.84 & 28 &$     $&    & 1405 & O04\\
J1623$-$2631 & 11.08 & 18 &$   1 $&  3 & 1331 & MH04\\
J1629$-$6902 &  6.00 & 14 &$     $&    & 1373 & O04\\
J1643$-$1224 &  4.62 & 11 &$  -1 $&  1 & 1331 & MH04\\
J1713$+$0747 &  4.57 &  3 &$     $&    & 1373 & O04\\
J1730$-$2304 &  8.12 & 17 &$     $&    & 1405 & O04\\
J1748$-$2446 & 11.56 & 14 &$     $&    & 1414 & S99\\
J1757$-$5322 &  8.86 & 19 &$     $&    & 1373 & O04\\
J1804$-$0735 & 23.10 & 13 &$ -12 $&  1 & 1408 & G98\\
J1857$+$0943 &  5.36 &  6 &$     $&    & 1373 & O04\\
J1909$-$3744 &  2.94 & 14 &$     $&    & 1373 & O04\\
J1911$-$1114 &  3.63 & 17 &$     $&    & 1373 & O04\\
J1915$+$1606 & 59.03 & 17 &$ -10 $&  3 & 1408 & G98\\
J1933$-$6210 &  3.54 &  6 &$     $&    & 1373 & O04\\
J1939$+$2134 &  1.56 &  3 &$     $&    & 1414 & S99\\
J2051$-$0827 &  4.51 & 10 &$     $&    & 1341 & O04\\
J2124$-$3358 &  4.93 &  6 &$  -1 $&  1 & 1327 & MH04\\
J2129$-$5721 &  3.73 & 30 &$     $&    & 1373 & O04\\
J2145$-$0750 & 16.05 &  5 &$     $&    & 1373 & O04\\
  \noalign{\smallskip}\hline\\
  \end{tabular}
\end{table}

Here we compiled a sample of millisecond pulsars observed near
1400~MHz as listed in Table~\ref{tb:MSP} (\cite{S99}; \cite{MH04};
\cite{O04}) and compared their circular polarization with that of
normal pulsars. The distributions of degree of circular
polarization of millisecond pulsars and normal pulsars are
marginally different, as shown in Figure~4. The Kolmogorov-Smirnov
test returned a probability of 16.49\% for the two populations
being from the same distribution.

   \begin{figure}
   \centering

\includegraphics[angle=-90,width=100mm]{figure4.eps}

\begin{minipage}{110mm}
\caption{Comparison of fractional absolute circular polarization
of normal pulsars and millisecond pulsars.}
   \label{fg:compa}\end{minipage}
   \end{figure}

\section{Discussion}

Circular polarization can be generated by several emission
mechanisms including curvature emission, coherent emission and
cyclotron absorption. \cite{Michel87} first noted that curvature
emission can explain the sense reversal of the circular
polarization. Following this model, \cite{GS90} re-examined
curvature radiation and demonstrated that circular polarization
could have a sense reversal near pulse center. \cite{Xu00}
considered coherent emission of a bunch of electrons with inverse
Compton scattering, and found that circular polarization can be
produced at low emission altitudes. On the other hand, cyclotron
absorption may also produce circular polarization (Melrose 2003).

   \begin{figure}
   \centering

\includegraphics[angle=-90,width=110mm]{figure5.eps}
\caption{IAG geometry of a pulsar.
The beam geometry for the inner annular gap model. The PA
decreases or increases with the pulse longitude as the line of
sight cuts the beam between the equator and the Northern or
Southern magnetic pole.
}
   \label{fg:gap}
   \end{figure}

Propagation effects can induce circular polarization or at least
influence circular polarization. There are two kinds of
propagation effect. One is in the pulsar magnetosphere and the
other in interstellar medium (ISM). \cite{PL00} investigated
refraction and polarization transferring in an ultra-relativistic
highly magnetized pulsar plasma. They found that circular
polarization arises out of rotation of the magnetosphere. Two main
types of circular polarization defined by \cite{RR90} can be
explained by refraction in the plasma with non-axisymmetric
density distribution and by magnetosphere rotation.
\cite{petrova01} also found that the change in the sense of
circular polarization can occur near the orthogonal transitions or
from non-orthogonality of the observed modes.

Macquart \& Melrose (2000) discussed a scintillation-induced
circular polarization in the interstellar medium due to rotation
measure gradient. The degree of circular polarization induced by
diffractive scintillation at lower frequency is more significant.
We calculated this effect due to rotation measure gradient
(\cite{v97}; \cite{HMQ99}; \cite{W04}) and found this effect is
very small (less than a few percent) except for a few pulsars at
low frequency.

The correlation between the sense of PA variation and the sense of
$V$ in conal-double pulsars may give some constraints to the
geometry and mechanism of pulsar emission. \cite{Qiao04} proposed
the inner annular gap (IAG) to explain the emission from pulsars.
For neutron stars, an IAG can be formed only for a pulsar
(\mbox{\boldmath $\Omega \cdot B<0$}), not for an antipulsar
(\mbox{\boldmath $\Omega \cdot B>0$}), and the beam is asymmetric
in shape, much larger toward the equator. According to this model,
as Figure~5 shows, conal-double pulses are more likely generated
in the region close to equator of the pulsar (\mbox{\boldmath
$\Omega \cdot B<0$}).
We also know that the PA decreases or increases with the pulse
longitude when the line of sight cuts the beam between the equator
and the Northern or Southern magnetic pole.
Based on the observed correlation in conal-double pulsars and the
properties of IAG model, the conal emission of pulsars in the
first part of Table~\ref{tb:cone_double} is produced from the
South magnetic pole and has right-hand circular polarizations,
whereas the emission of pulsars in the second part of
Table~\ref{tb:cone_double} is from the North magnetic pole and has
left-hand circular polarization.

\section{Conclusions}

Circular polarization in pulsars shows
diverse  patterns. Though sense reversal of circular polarization
often occurs in the core components, it can also happen in the
cone components or near the intersection between components. We
confirm the correlation between the sense of circular polarization
and the sense of position angle sweep for conal-double pulsars.
Circular polarization of some pulsars get stronger with frequency,
but others get weaker. The senses of circular polarization of
conal-double pulsars may be related to the different magnetic
poles.

\vs\vs\no {\bf Acknowledgements}~~ We thank Dick Manchester, Joel
Weisberg, Guojun Qiao, Kejia Li, Xiaohui Sun and Hui Men for their
helpful comments. The authors are supported by the National
Natural Science Foundation of China (Nos.\,10025313, 10521001 and
10473015).

\newpage


"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibility

Abstract: In current NLP research, large-scale language models and their abilities are widely being discussed. Some recent works have also found notable failures of these models. Often these failure examples involve complex reasoning abilities. This work focuses on a simple commonsense ability, reasoning about when an action (or its effect) is feasible. To this end, we introduce FeasibilityQA, a question-answering dataset involving binary classification (BCQ) and multi-choice multi-correct questions (MCQ) that test understanding of feasibility. We show that even state-of-the-art models such as GPT-3, GPT-2, and T5 struggle to answer the feasibility questions correctly. Specifically, on MCQ and BCQ questions, GPT-3 achieves an accuracy of just (19%, 62%) and (25%, 64%) in zero-shot and few-shot settings, respectively. We also evaluate models by providing relevant knowledge statements required to answer the question. We find that the additional knowledge leads to a 7% gain in performance, but the overall performance still remains low. These results make one wonder how much commonsense knowledge about action feasibility is encoded in state-of-the-art models and how well they can reason about it.

\section{Introduction}

Commonsense reasoning has been a key aspect of AI since its advent in the 1950s. It is closely associated with reasoning about actions and their effects, which is considered as a big challenge, especially for deep learning based AI approaches and language models \cite{lecun2022path}. While several datasets have been developed to evaluate large-scale language models, in this paper we propose a dataset focused on a simple ability related to reasoning about actions and their effects; specifically, we focus on the ability to reason if an action or its effect is feasible.

\begin{figure}[t!]
    \centering
    \includegraphics[width=7cm]{Pictures/image.png}
    \caption{Illustrating a binary classification (BCQ) and a multiple choice question (MCQ) from \textsc{F}easibility\textsc{QA}\xspace{}. The correct answer options (\textbf{False} in BCQ and \textbf{(45, 37)} in MCQ) are highlighted in \textbf{bold}.}

    \label{fig:teaser_figure}
\end{figure}

\input{Tables/app_t5_dataset_examples}

Figure \ref{fig:teaser_figure} illustrates an example of our dataset; given the information ``\textit{Sam gave 50 dollars to the shopkeeper to buy a book and the shopkeeper returned some money}'', it is not possible to compute the exact price of the book; however, it can be established that the feasible price of the book is \textit{less than} 50 since the shopkeeper returned some money.
We often come across such questions in our daily life and find it trivial to reason about them.
Therefore, in order to develop NLP systems that can reliably reason about real-world situations, it is important to evaluate their understanding of feasibility.

Recently, numerous datasets have been created that test different reasoning skills such as pronoun resolution \cite{sakaguchi2021winogrande,levesque_winograd_2012}, commonsense reasoning \cite{singh-etal-2021-com2sense}, numerical reasoning \cite{ravichander-etal-2019-equate, lin-etal-2020-birds, zhang-etal-2020-language-embeddings,mishra-etal-2022-numglue,amini-etal-2019-mathqa,creswell2022selection}, qualitative reasoning \cite{tafjord-etal-2019-quartz, tafjord2019quarel}, discrete reasoning \cite{dua-etal-2019-drop}, and temporal reasoning \cite{zhou-etal-2019-going}.
However, none of above datasets contain a sufficient number of examples that test the understanding of feasibility.

In this work, we address the above limitation and introduce \textsc{F}easibility\textsc{QA}\xspace{}, a dataset consisting of questions that require understanding of feasibility. This dataset comprises of two types of questions: binary classification (BCQ) and multi-choice multi-correct questions (MCQ). In BCQ, the task is to determine whether the question is feasible or not given a context and in MCQ, the task is to select all feasible answers to the given question. The dataset consists of $\sim$5K instances covering diverse aspects of feasibility.
Table \ref{tab:dataset_example} illustrates examples of various categories of feasibility questions.

We conduct comprehensive experiments with GPT-3 \cite{brown2020language} in zero-shot and few-shot settings and show that it struggles to correctly answer \textsc{F}easibility\textsc{QA}\xspace{} questions.
Specifically, on (MCQ, BCQ) questions, GPT-3 manages to achieve accuracy of just ($19\%$, $62\%$) and ($25\%$, $64\%$) in zero-shot and few-shot settings, respectively.
Prior work has found that explicitly providing relevant knowledge helps the model reason better and improves its performance.
We explore this aspect of reasoning by explicitly providing relevant knowledge statements and find that it leads to a small improvement in performance ($\sim$7\%). However, the overall performance still remains low.
We further investigate GPT-3's ability to reason about feasibility questions by prompting it to generate the reasoning chain.
In many cases, we find that GPT-3 successfully generates the correct reasoning chain but still fails to output the correct answer.
This analysis leads to several such interesting findings.

\section{Dataset}
For creating data instances of \textsc{F}easibility\textsc{QA}\xspace{}, we first create a context statement that describes a real life situation. Then, we write two binary classification questions and one multiple choice question conditioned on the context that tests the understanding of feasibility.
We describe our context creation and question creation procedures in \ref{subsec_context_creation} and \ref{subsec_question_creation}.

\subsection{Context Creation}
\label{subsec_context_creation}
We create context statements from real-world situations spanning over elementary physics, profit-loss scenarios, temporal comparison, quantity comparisons etc.
We divide these contexts into the following five categories:

\textbf{Attribute comparison:} This category consists of questions that test feasibility aspects involving comparison of different quantities. \textbf{Implicit numerical:} Questions in this category deal with  mathematical facts and test model's ability to use those facts in different circumstances. \textbf{Change with action} test the model's ability to perceive a change on an item or its state when an action is performed. \textbf{Change with time} have instances where model's understanding of time based events are tested. \textbf{Non-numerical} includes examples where numbers are not involved explicitly in reason about feasibility.
Table \ref{tab:dataset_example} provides examples of these categories.

\subsection{Question Creation}
\label{subsec_question_creation}
From each context, we create two binary classification and one MCQ. For these questions, the context does not provide sufficient information to \textbf{exactly} compute the answer. However, the information is sufficient to test the validity of the question (asked using the word `\textbf{could}'). We ensure that all our contexts and questions describe a realistic situation. To achieve this, we keep the range of numerical entities and units appropriate for their context. Some of these examples are shown in Table \ref{tab:dataset_example}.

\textbf{BCQ:}
For each context, we create two binary classification questions where the correct answer is \textit{True} for one and \textit{False} for the other. This provides an additional benefit of evaluating models' consistency in answering two contrasting hypotheses conditioned on the same context.

\textbf{MCQ:}
For each context, we create a multi-correct multiple choice question. Here, a context-question pair and the corresponding answer options are given and the task is to select all options that are feasible answers to the question. For a question, \textit{one or more options} could be correct. We ensure that there is also a \textit{None} option which needs to be selected when all the other options are not feasible.

\input{Tables/dataset_distrib}
\input{Tables/app_t4_number_of_answers}
\subsection{Dataset Statistics}
\textsc{F}easibility\textsc{QA}\xspace{} consists of $1536$ contexts and $4608$ context-question pairs ($3072$ BCQ and $1536$ MCQ). BCQ dataset is label balanced i.e $768$ instances for each of \textit{True} and \textit{False} labels. MCQ dataset has varying number of correct options. Table \ref{tab:dataset_distrib} shows the distribution of BCQ and MCQ questions across different categories in the dataset. Table \ref{tab:number_of_answers} shows the number of correct answers in the MCQ category. A detailed description about the dataset creation and the verification process is explained in Appendix \ref{appendixa_dataset}.

\section{Experiments}
\label{sec_experiments}
\subsection{Experimental Setup}

\paragraph{Performance Evaluation:} We evaluate the BCQ and MCQ task using accuracy metric. For \textbf{BCQ}, We first compute accuracy by counting the number of exact matches (True or False) against the Gold Label. We also perform \textbf{pairwise evaluation} where correctness on both true and false instances created from the same context is compared. This enables us to test consistency in predictions.

For \textbf{MCQ}, we consider answer to be correct if all the correct options are marked and no other options are given by the model. In one shot setting, the same example was repeated across all the examples to avoid data leakage. We have considered 3 shot for few shot setting and used same two examples as above. We also calculate True Positive, False Positive and False Negative for both BCQ and MCQ which are described in depth in Appendix \ref{appendixb_performance_results}. We use \textbf{GPT-3} in the following setting: Text-DaVinci-002, max tokens = 256, topp = 1, frequency \& presence penalty = 0.

\input{Tables/binary_results}
\input{Tables/dataset_distrib_results}
\subsection{Results}
We present the performance in Table \ref{tab:bc_results}, \ref{tab:one_shot_category_results} and \ref{tab:pairwise_results}.

\paragraph{Low accuracy in BCQ and MCQ:} Table \ref{tab:bc_results} shows the accuracy of GPT-3 for BCQ and MCQ. Lowest accuracy on BCQ is 57.51\% (one shot without knowledge) and highest is 69\% in (zero shot with knowledge). GPT-3 shows 57\% accuracy which is just above majority baseline (50\%). Lowest accuracy in MCQ is 19\% observed in zero shot without knowledge setting and highest accuracy was 37\% observed in with knowledge, few shot.

\paragraph{Marginal improvement with addition of knowledge:} Performance of the model was slightly better when the knowledge statement was also supplied with context and question. Contrary to the conventional thinking, performance of one-shot experiments (both with and without knowledge) was lower to as compared to zero shot experiments.

\paragraph{Contrasting results across different categories:} Table \ref{tab:one_shot_category_results} shows the category wise results in one-shot settings for BCQ and MCQ task in one shot setting. For BCQ, accuracy was lowest in Attribute comparison category and highest in Non-numerical category. Performance of non-numerical category improved significantly with knowledge. This can primarily attributed to the fact that non numerical category had instances where addition of knowledge could be very helpful in answer determination. Other results are present in Appendix \ref{appendixb_performance_results}. In MCQ, the performance gap between non-numerical and other categories reduces significantly. It is still the best performing category for the model but Change with Action category also produced similar results. There was not significant improvement as observed in the non numerical with addition of knowledge.

\input{Tables/pairwise_evaluation_results}

\paragraph{Pairwise evaluation:}
Recently, instance-level analysis of the evaluation data has received consdierable attention \cite{zhong-etal-2021-larger,varshney-etal-2022-ildae,rodriguez-etal-2021-evaluation}.
Inspired by this, we calculate the pairwise accuracy of examples in BCQ setting. This indicates how many instances the model was able to predict the true and false question of the same context. Table \ref{tab:pairwise_results} shows us that best performance was obtained in zero shot category and reduced significantly in 1 shot setting (25\% change in percent).

\subsection{Explanation of Poor Performance }
We try to analyze that why GPT-3 is failing in such scenarios. For doing that, we design the prompt so the model provides an explanation for its prediction. Figure \ref{fig:combined example} illustrates GPT-3's response for different scenarios. The answer \textit{"Price of the book could not be 45 dollars if the shopkeeper was given 50 dollars"} demonstrates that it did not understand the concept to book's price. To remedy this we also provide additional knowledge as shown in the figure. This too could not be understood by GPT-3. Even with knowledge, the model could not understand feasible book prices.

Many experiments have demonstrated the success of learning from instruction~\cite{mishra-etal-2022-cross, wei2021finetuned, wang2022benchmarking, sanh2021multitask, ouyang2022training, mishra-etal-2022-reframing} and chain of thought/scratchpad prompting \cite{wei2022chain, nye2021show}. The examples that we have in our dataset and exemplified in our paper are relatively trivial compared to huge word problems which are being solved using these models. The model's inability to understand simple numerical relations shows lack of cognitive ability. Further details about it can found in Appendix \ref{app_c}.

\begin{figure}[t!]
    \centering
    \includegraphics[width=7cm]{Pictures/combined_teaser3.png}
    \caption{Answers with explanations given by GPT-3 on \textsc{F}easibility\textsc{QA}\xspace{} dataset.}
    \label{fig:combined example}
\end{figure}

\section{Conclusion}

We proposed \textsc{F}easibility\textsc{QA}\xspace{}, a question-answering dataset involving binary classification and multi-choice multi-correct questions that test the understanding of feasibility.
We conducted comprehensive experiments with state-of-the-art models such as GPT-3 in multiple settings and demonstrated that models struggle to answer the feasibility questions correctly.
We also evaluated models by providing relevant knowledge statements required to answer the question and showed that the additional knowledge leads to a 7\% gain in performance, but the overall performance still remains low.
We hope that our dataset and findings will encourage further research in commonsense reasoning and contribute towards development of more reliable NLP systems making their widespread adoption in real-world applications possible.

\bibliographystyle{acl_natbib}