Variance of average of $n$ correlated random variables

Question: Reading about deep leaning, I came across the following formula.
$$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2  $$
where $X_1, \dots, X_n$ are identically distributed random variables with
pairwise correlation $\rho > 0$ and variance $\mbox{var}(X_i) = \sigma^2$.

How to derive this?
How does bootstrap aggregating alleviate the effect of overfitting, according to this formula? What is the relationsip?
Answer: By definition, we have
$$\operatorname{var}\left(\sum_{i=1}^n{X_i}\right)=\operatorname{cov}\left(\sum_{i=1}^n{X_i},\sum_{i=1}^n{X_i}\right)=\sum_{i=1}^n{\operatorname{var}(X_i)}+\sum_{i\neq j}\operatorname{cov}(X_i,X_j)$$
which is $n \operatorname{var}(X_i)+n(n-1)\operatorname{cov}(X_i,X_j)=n\sigma^2+n(n-1)\rho\sigma^2$, where $i\neq j$. Substituting this into the original equation yields the following:
$$\operatorname{var}\left(\frac{1}{n}\sum_{i=1}^nX_i\right)=\frac{1}{n^2}(n\sigma^2+n(n-1)\rho\sigma^2)=\rho\sigma^2+\frac{1-\rho}{n}\sigma^2$$
Each $X_i$ can be thought of as a single decision mechanism, call it DM, (e.g. regressor). The variance of your decision was $\sigma^2$. By using bootstrap samples and aggregating your DMs' outputs, you end up with a decision variance as above, which is strictly smaller than $\sigma^2$ when $\rho \neq 1$ and $n\neq 1$. DMs will have some degree of correlation of course, because they are trained over bootstrap samples obtained from the same base dataset; however, the correlation between them most probably won't be equal to $1$. Overfitted mechanisms in general have large variance, so by aiming to decrease the variance of your DM, you actually address the problem of overfitting implicitly.
Comment: @OmegaD There are $n^2$ pairs of $i,j$, where $n$ of them have $i=j$, and $n^2-n=n(n-1)$ of them have $i\neq j$.
Comment: Fantastic! Thank you for so much for your answer. Quick question, in the term $n var(X_i) + n(n-1) cov(X_i,X_j)$ n and n-1 come from. Sorry if it is too obvious question.
Comment: I know that $\rho\geq -1/(B-1)$ but that does not imply that $\rho>0$ which is stated in the book? Say we have $B=3$ we have that $\rho$ can take the values $-0.5$ without violating anything. How come the proof is only for $\rho\geq 0$?
Comment: @CutePoison it is assumed $\rho>0$ in the OP and my proof is based on this assumption. I'm not aware in which book it is, so if you can share the link, I can at least read and comment on it.
Comment: I struggle to see; Where is the use of the assumption that $\rho>0$  ?
Comment: My bad of refering to "the book" (I simply forgot to add it), but it is in "the elements of statistical learning"


Want my while loop to finish and output after I enter a number which is over 800 but it is giving me unhandled exceptions

Question: // While Loop & if statement
while (userInput < 800)
{
    if (userInput >0 && userInput <200)
    {
        cOne++;
        sumLengthOne = sumLengthOne + userInput;
        userInput = int.Parse(Console.ReadLine());
    }
    else if (userInput >200 && userInput <400)
    {
        cTwo++;
        sumLengthTwo = sumLengthTwo + userInput;
        userInput = int.Parse(Console.ReadLine());
    }
    else if (userInput >400 && userInput <600)
    {
        cThree++;
        sumLengthThree = sumLengthThree + userInput;
        userInput = int.Parse(Console.ReadLine());
    }
    else if (userInput >600 && userInput <800)
    {
        cFour++;
        sumLengthFour = sumLengthFour + userInput;
        userInput = int.Parse(Console.ReadLine());
    }
    else if (userInput > 800)
    {
        break;
    }
}

Console.WriteLine("| {0,-10} | {1,5} | {6,10} | {11,15} |" , "Range" , "Count" , "Sum Lengths" , "Percentage");
Console.WriteLine("==========");
Console.WriteLine("| {0,-10} | {1,5} | {6,10} | {11,15} |" , "<200" + "200 - 399" + "400 - 599");
Console.WriteLine("The Largest Bacteria Is: " + largestValue);
Console.WriteLine("The Smallest Bacteria Is: " + smallestValue);

The unwanted result is unhandled exception
"    Unhandled exception. System.FormatException: Index (zero based) must be greater than or equal to zero and less than the size of the argument list"
after the while loop I am outputting the results to console but it gives an unhandled exception instead.
Comment: "*after the while loop I am outputting the results to console*" - this code should be included in the question, as it is the part that triggers the exception
Comment: Code review: when you test for `<200` in the `if`, you don't need to test for `>=200` in the next `else if`. Right now a 200 value falls between the cracks! (same for 400, 600, 800)
Answer: Im not sure how your userInput comes into play, from input field or from looping through values. but...
When inputs like 200, 400, 600, 800 are typed, you have no matching condition. you are either above or under 800 but not === 800
you conditions should be something like this:

        while (userInput < 800)
        {
            if (userInput >0 && userInput <= 200) // below or equal too
            {
                cOne++;
                sumLengthOne = sumLengthOne + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >200 && userInput <= 400) // below or equal too
            {
                cTwo++;
                sumLengthTwo = sumLengthTwo + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >400 && userInput <= 600) // below or equal too
            {
                cThree++;
                sumLengthThree = sumLengthThree + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >600 && userInput <= 800) // below or equal too
            {
                cFour++;
                sumLengthFour = sumLengthFour + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput > 800)
            {
                break;
            }
        }

Ofcourse if you want to break 200, 400, 600 & 800 and under you would have to do it the other way around - like this:

        while (userInput < 800)
        {
            if (userInput >= 0 && userInput <200) // 0 or above
            {
                cOne++;
                sumLengthOne = sumLengthOne + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >= 200 && userInput <400) // 200 or above
            {
                cTwo++;
                sumLengthTwo = sumLengthTwo + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >= 400 && userInput <600) // 400 or above
            {
                cThree++;
                sumLengthThree = sumLengthThree + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >= 600 && userInput <800) // 600 or above
            {
                cFour++;
                sumLengthFour = sumLengthFour + userInput;
                userInput = int.Parse(Console.ReadLine());
            }
            else if (userInput >= 800) // 800 or above
            {
                break;
            }
        }
Answer: In your line
Console.WriteLine("| {0,-10} | {1,5} | {6,10} | {11,15} |" , "Range" , "Count" , "Sum Lengths" , "Percentage");

That {6,10} doesn't mean "from char 6 to 10" but "use argument #6, formatted as 10 chars wide". And you don't have an argument #6 (or #11). You meant index 2 and 3 there.
Fixed:
Console.WriteLine("| {0,-10} | {1,5} | {2,10} | {3,15} |" , "Range" , "Count" , "Sum Lengths" , "Percentage");

Tip: for the following dividing line, use this:
Console.WriteLine(new string ('=', 33));

to output 33 "=" characters.
Comment: Thanks Hans, this is very helpful.


Unable to create new simple volume. C:\Windows\inf - Not default?

Question: Windows Server 2008 R2.
When I attempt to create a new "simple volume" in Disk Management (or even Diskpart), the operations hangs and eventually end up with this error:

However, before the error above appears, I noticed that in Device Manager appears an Unknown device (under Other Devices), if I right-click on Unknown device and select Update Driver Software, and I browse to C:\Windows\inf, it actually finds and detects a Generic volume and the new "simple volume" ends up being created just fine.
It's as if Windows has lost the ability to look for drivers in C:\Windows\inf by default.
Can anyone suggest how to fix this?
Answer: The answer is here:
https://docs.microsoft.com/en-us/windows-hardware/drivers/install/where-setup-searches-for-drivers
Somehow C:\Windows\inf was missing from the DevicePath key.


Server Disk Partition Software

Question: Is there any free (reliable....yes I know..) disk partition software for Windows Server? I am checking before I do make a purchase. I just need to resize a disk.
Edit ~ I have a windows server 2003 Raid 5 with 2 partitions "C" and "D". I need to resize "C" by taking some space from "D". The "D" has 100GB extra and "C" is running out of space.
Comment: Can you elaborate a bit more about what you need to do?
Answer: The gParted liveCD is probably the best I've seen. It's free, opensource, and a fairly small download, with a nice simple interface.
Comment: +1 gParted works so well, I keep a copy in the glove compartment of my car.  and I've used that copy at least twice.  :)
Answer: Windows Server 2008's installation media allows you to use DISKPART to manipulate NTFS volumes very effectively. (It will manipulate Windows Server 2003-based volumes as well-- both OS's are using the same NTFS version, 5.1.)
I've used the "EXPAND" functionality in DISKPART fairly regularly in previous versions of Windows, but I've had to build Windows PE discs to take advantage of that functionality when resizing boot volumes. Booting the W2K8 installation CD and using DISKPART directly from there is nicer than having to make more WinPE discs.
I haven't used the new SHRINK functionality in the W2K8 DISKPART, but it looks handy.
I don't know if it's strictly "legal" to download the evaluation version and use it for this functionality, but you could definitely do it if you don't have a copy of the W2K8 installation media sitting around.
(I'd trust DISKPART and the canonical Microsoft implementation of NTFS more than I'd trust any other tools. That's no slight on the other tools, but being that NTFS is Microsoft-proprietary, I'd expect their tools to handle their filesystem better than any others.)
Edit: You're also going to get support for Microsoft software RAID volumes by using the Windows Server 2008 setup CD, versus open source tools. (I like the open source tools fine, but the Microsoft solution is compelling to me in this case.)
Answer: You can use a Linux live cd and gparted to resize partitions.  But resizing partitions aren't always possible.  So you need to know when you can do this safely or use tools that prevent you from doing anything destructive.
If you're trying to shrink the partition you need to run a disk defragger that will move all the data to the beginning of the disk before resizing the partition.  Gparted in this case will not let you resize your partition smaller than the last block of real data.  After you resize the partition windows will force a diskcheck when you boot up.
To grow a partition then with most of the free tools that I know of you have to have some space available at the end of your existing partition to expand to.  Once you grow a partition you'll need to run diskpart.exe to have it recognize the new size.
See this KB article: http://support.microsoft.com/kb/325590
It looks like you can also use the diskpart.exe utility to resize the partitions based on that KB.  But Gparted is a gui app...where diskpart is command line.  I would make sure that your data is backed up before proceeding.  No matter what tool you end up using.
Answer: I like Parted Magic it is a live linux distro that uses the gparted partition editor. Very nice if you like GUI's

Server Disk Partition Software


Resizing an image in PIL

Question: I have the following function which takes an image and then returns it in three sizes. Another function then uploads those images to Amazon S3. It seems to me like there is some redundancy in how the file is being saved -
def resize_image(image, size_as_tuple):
    """
    Example usage: resize_image(image, (100,200))
    """

    image_as_string=""
    for c in image.chunks():
        image_as_string += c

    imagefile = cStringIO.StringIO(image_as_string)
    image = Image.open(imagefile)

    if image.mode not in ("L", "RBG"):
        image = image.convert("RGB")

    filename = ''.join(random.choice(string.ascii_uppercase + string.digits) for x in range(14)) + ".jpg"
    height, width = size_as_tuple[0], size_as_tuple[1]
    image.thumbnail((height, width), Image.ANTIALIAS)

    imagefile = open(os.path.join('/tmp', filename), 'w')
    image.save(imagefile, 'JPEG')

    imagefile = open(os.path.join('/tmp', filename), 'r')
    content = File(imagefile)

    return (filename, content)

Is there a way to improve this?
Answer: You could replace:
height, width = size_as_tuple[0], size_as_tuple[1]
image.thumbnail((height, width), Image.ANTIALIAS)

with
image.thumbnail(size_as_tuple, Image.ANTIALIAS)

(especially since width and height are swapped; it should be width, height = size_as_tuple)
And you don't need the open(). image.save(os.path.join('/tmp', filename)) is enough.

Resizing an image in PIL


What makes a good mathematician?

Question: A soft question but I believe important to help get maturity in maths.
Not until I got admitted to a graduate applied math program that I started to learn math. Before that, I was a life science undergraduate that only know some entry level calculus. The more I learn math, the more I started to find it is more than the formulas , but also perspectives that would help to shape a good mathematician. In Terence Tao's real analysis book, he said that a mathematician don't quite think concepts in a way of  "object", that they more prefer to ask what a concept can be applied to, rather than what a concept is consisted of.  I think that is inspiring and indeed helped me really started to think as a mathematician.
My question is, when you are observing other people or yourself do math, what do you find makes a good mathematician ?  What kind of point of views you need to have in order to solve a particular problems in fields like linear algebra, topology, real analysis, statistics, etc?   Or, while learning maths, what kind of things I can pay attention to, or I can ask myself about, that would help me to become more to think like a mathematician?
(A bit more about what I used to think before I started learn math:  I was pretty good at physics, which is more like modeling by reduction work to me: apply relative few principles and you can start analyze stuff. When it got to maths problem, it doesn't work in that way. I found from equations to equations, though it is "equivalent" between two equations, but the information conveyed can be much different.
Also a lot of times problems are solved by a constructive way, which seems not nature to come up at first glance, but often it indeed make sense once you have a higher view. For example, if you have the idea that the rank and determinant is actually describing "how big" a matrix is, you can easily start to crack down the dimension problems in linear algebra one way or another.  But there are more subtlety in writing the inversion of matrix in adj(.)/det(.) that it is not obvious for building up an intuition of adj operator, but it is as well powerful in different application. Nevertheless, not many textbooks would tell you why we should care about adj.)
Comment: Seeing analogies between analogies :)
Comment: Trying to find out why things are true. Looking for understanding.
Comment: Unrelenting curiosity helps.
Comment: Flagged to make post CW.
Comment: I would recommend one of the things to read is "Mathematics as a Creative Art" By P R Halmos (search Google and you can find the paper).
Comment: a nocturnal / crepuscular circadian rhythm, sustained by fizzy drinks? [link](http://www.psychologytoday.com/blog/the-scientific-fundamentalist/201005/why-night-owls-are-more-intelligent-morning-larks)
Comment: Someone who can't do algebra. Algebra kills the mathematical creativity.
Answer: I second the notion of Intuiton and Instincts, though in a different context. Many hard research problems are hard simply because there is no cookie cutter method for solving them. After trudging though core curriculum like Calculus, Algebra, Analysis, PDE's, etc, you acquire a vast array of problem solving techniques albeit for specific classes of problems. Especially nowadays it takes an even vaster array of knowledge to push forward new results and ideas. On that notion, I think Freeman Dyson does a decent job of summarizing Birds and Frogs as two classes of (good) mathematicians, though I think that he (unintentionally?) makes out Birds to sound much cooler than Frogs. Personally I would have called them Birds and Foxes.
The point though is that IF a problem is tractable by current methods (eg NOT the Riemann Hypothesis), then a good mathematician should be able to provide a sketch for how to start  tackling the problem. For example, just like an assist in soccer or basketball, even a suggested reference or place to look can be worth as much as actually writing down a solution. In short, a good mathematician should be able to (statistically speaking) give the right guesses or references to a hard problem. Of course the hard work lies ahead but, where to start is half the battle.
Many problems have intuitively obvious quantities of interest in them. Many conjectures aren't just based on numerical evidence, but on some set of intuition which is consistant with current theory. For example, in probability theory you might be looking at some complicated stochastic process and a reasonable conjecture might be that after a long time the process is asymptotically uncorrelated with its initial conditions, even if its not a Markov chain. And, the first thing to do when tackling a difficult problem is to adhere to Polya's golden rule of "solve a simpler problem first." It's amazing that sometimes a problem has a very tractable simplification and then the whole problem gets solved by some modifications. It's as if adding complexity to a problem is very nonlinear and sometimes even has diminishing returns. So I would add here that a good mathematician should give great guesses for simpler problems to solve first that are related to the bigger one.
I also think that there's an element of luck involved. This part may be controversial to some people but i'll stand by it. People like Ramanujan and Erdos are very large deviations from the average mathematician and I think for this reason they are truly masters of their fields. They delved into very specific niches of mathematics and excelled there without equals. Even the most experienced mathematicians who worked their entire lives in these niches may not have been as prolific as these giants. Supposing you are already a semi talented mathematician, I really think in this case it's a bit of luck, being close to the right problem at the right time with the right intuition. With Ramanujan, he learned all his mathematics from an encyclopedia so at least to me it's not truly surprising that he had huge clairvoyance when it came to crazy series identities. On the other hand as far as i know from his biography (The Man Who Knew Infinity), he learned and struggled with complex analysis much much later in life. Erdos had an uncanny problem solving ability and somehow naturally fit into combinatorics and graph theory. So maybe I'll end this section by saying that a good mathematician, combined with a bit of luck knows his or her niches and has good abilities at spotting problems he or she can tackle.
Answer: A controversial and easy answer is "intuition". I know you don't like it, but sadly it is true. All that we know about Calculus started with Newton and Leibnitz's intuition about limits,continuity and derivatives and integrals. And for many decades Calculus stuck to be an "intuitively correct" idea, and along came Augustin-Louis Cauchy who defined it rigorously and calculus was no longer "intuitive" it soon became a widely accepted mathematical truth. (In 1734 Bishop Berkeley panned calculus that it lacked rigor and just plainly intuitive - he further said mathematicians had no business criticizing religious people given the way mathematicians reason themselves) But Cauchy made sure mathematics' greatest brain child was made "controversy-free".
Intuition is never an acceptable way to prove anything in mathematics. Ramanujam proved brilliant and shocking results and he wasn't even properly trained in mathematics. When he was 10, he mastered (self-learnt) trigonometry (all he had was S.L. Lony's book on trigonometry) and he proved various results himself (that was when he was 12). Erdos proved there is always a prime between $n$ and $2n$ in the easiest possible way (he also did prove other brilliant results). The list goes on really. We can argue that these results are not intuitive (well not completely) but the very idea of a mathematical formal system have what are called axioms which are completely intuitive. They are starting points or truths that cannot be proved or disproved just accepted in good faith. William Feller in his classical book on probability acknowledges that mathematical intuition can be developed through training
Formally, intuition is discouraged, its clear-headed thinking and originality (Cantor's work on transfinites) and creativity (Godels incompleteness theorems) that goes a long way. Further a very few of Ramanujam's equations has been found to be wrong since he mostly relied on intuition to come up with them.
Well, regarding your question of how to solve anything in topology etc. Top-down or bottom-up machines is what we all are. This the formal thing anybody can give you. Start with an axiom or known theorem and search your way through for your solution. Hopefully through practice you would be able to do this sub-consciously. A spark of creativity always triggers you into a different realm of thinking. As an example I cannot think of anything better than the travelling bee problem (there's literally not a single soul that I know who doesn't know this problem): Two trains 150km apart head towards each other (for an imminent collision) at 50km/hr. A bee starts from one train and flys, at 75km/hr, (in a straight line) to the other, the moment it reaches the other train it reverses it's diretion and continues this till it is squished to death. What is the distance that the bee travels? Now you can sum a geometric series to get the answer or multiply 75 and 3. The best part of that problem was, when it was asked to John von Neumann by the reporter he instantly gave the answer. The reporter asked, "Did you multiply?". He replied "No I summmed the series"!! That I think is a good mathematician right there.
As @PeteClark suggested there isn't any antagonistic relationship between formalism and intuition-ism. Formal reasoning is just the means by which intuitive ideas are shared or proved.
Comment: "When you deal with the unknowns it's almost always controversial."  What is "it's" here?  Saying that some mathematicians with intuition had ideas that were hard to accept is very far from disagreeing that good mathematicians have intuition.  You are also creating a strangely antagonistic relationship between intuition and formal reasoning.  Not only do good mathematicians evince both, the latter is not even a quality but an act.  Finally, it would be nice if you gave examples that were less than 100 years old: aren't we talking about what makes a good mathematician *today*?
Comment: @Vig: thanks, that's a good response.  As for leaving an answer myself: I don't know how to meaningfully respond to such a broad question in less than several pages.  Maybe some day I'll try to write such a response, but not today...
Comment: I don't understand why the claim that good mathematicians have intuition is controversial in the slightest.
Comment: When you deal with the unknowns it's almost always controversial. Take Cantor for example most of his intuitive ideas were hard to grasp and accept, and most of the time he was right. As a counter-example Bertrand Russel was certain about the completeness of mathematics before Godel brought the world down. When unknowns (like infinity and infinitesimals are involved) are involved a sound formal reasoning is preferred.
Comment: I do agree that my examples are ancient, so it would be helpful and valuable for us (students) to hear from a mathematician like yourself. Further I do agree that I have created this (non-existent) antagonistic relationship between the two, but I have made sure in my answer that both are in essence central to any mathematician (intuition with the calculus examples and formalism with neumann example) or maybe I wasn't quite clear in my exposition, in which case I will surely edit out the imperfections in the answer.
Comment: I don't think you need to go back to Newton to speak about how intuition is good. I think that every result has intuitive thinking underneath its surface although they're finally presented as formal statements, with formal proofs. I think that's they way our minds work. We work by intuition, not formality, formality doesn't appear naturally in human's minds. We don't work with the formal systems themselves, but with itnerpretations that allow us to form and intuitive view about it. I think GEB book makes a very beautiful explanation of how we do this.
Comment: I cant be the only here that realizes that the bee travels 75*3/2 km right ?
Answer: Necessary but not sufficient condition: recognizes the importance of good definitions.
Comment: and of notation
Comment: Necessary but not sufficient, sort of like this answer...

What makes a good mathematician?


Volume of Solid of revolution of some body in a ball

Question:
Calculate the Volume of Solid of revolution of $D$,which is subset of the ball $\{x^2+y^2+(z-\frac{1}{2})^2\le(\frac{1}{2})^2\}$ and is above the cone $\{z=\surd(x^2+y^2)\}$.

My question - I'm not sure, how can I find which substitution should I do/use here? How can I find which is the best?
Answer: Solid of revolution, or just the volume of the solid?
Spherical coordinates:
$\iiint dz\ dy\ dx  = \iiint \rho^2\sin\phi\ d\rho\ d\phi\ d\theta$
limits
$x = \rho\cos\theta\sin\phi\\y = \rho\sin\theta\sin\phi\\z = \rho\cos\phi$
$x^2+ y^2 + z^2-z = 0\\
\rho^2= \rho\cos\phi\\
\rho = 0, \rho = \cos \phi$
$z=\sqrt{x^2 + y^2}\\
\rho\cos\phi = |\rho\sin\phi|\\
\tan\phi = 1\\
\phi = \frac {\pi}4\\
$
$\int_0^{2\pi}\int_0^{\frac {\pi}4} \int_0^{\cos\phi} \rho^2\sin\phi\ d\rho\ d\phi\ d\theta$
Comment: $x^2 + y^2 + z^2 = z$  Substitute: $\rho^2\cos^2\theta\sin^2\phi + \rho^2\sin^2\theta\sin^2\phi + \rho^2\cos^2\phi = \rho\cos\phi$ Simplify: $\rho^2 = \rho\cos\phi$
Answer: You need to project the solid in the $xy$ plane and use cylindrical coordinates. The sphere and the cone intersect when
$$
r^2+(r-\frac{1}{2})^2 = \frac{1}{4}\quad \Leftrightarrow \quad r \in \{0,\frac{1}{2}\}
$$
So the projection in the $xy$ plane is the domain
$$
D=\{(r,\theta)\; |\; 0 \le r \le \frac{1}{2}, 0 \le \theta \le 2\pi \}
$$
It follows that the volume equals
$$
V = \iint_D \left((\frac{1}{2}+\sqrt{\frac{1}{4}-r^2})-r\right )\; rdrd\theta
$$


Destacar principais palavras de um texto em ordem de uma lista ja pronta

Question: Tenho uma base bem básica que gostaria de destacar as principais palavras de cada linha de uma lista de priorização, exemplo:
Base:
Bom dia
que dia lindo
vamos embora
vamos chutar o dia

Lista de priorização: bom dia vamos lindo
Resultado esperado:
Bom dia - Bom
que dia lindo - Dia
vamos embora - vamos
vamos chutar o dia - dia

Consegui achar apenas com uma variável e não um lista. Código:
texto = 'dia'

for lin in open(r"C:\Users\guilgig\Desktop\teste.txt"):
    if texto in lin:
        print (texto, lin)

Comment: Por quê "*Bom dia*" possui apenas "*Bom*" destacado se "*dia*" também pertence à lista de priorização?
Comment: Obrigado por editar! então seguindo a lista de priorização pois s preciso da mais importante então quando ele acha uma das palavras ja está ok, porem se houver uma forma mostrando todas as palavras tudo bem
Answer: O código que você fez já está bem próximo do que você precisa, basta você definir a lista de priorização e percorrê-la buscando as palavras na frase:
with open('teste.txt') as stream:
    for line in stream:
        for word in ['bom', 'dia', 'vamos', 'lindo']:
            if word in line:
                print(line.strip(), '-', word)
                break

Veja funcionando no Repl.it
Quanto ao with para abrir o arquivo, você pode ler em: Para que serve o with no Python?. Para micro-otimações, você poderá definir sua lista de palavras com uma tupla (tuple) ou até um conjunto (set), que, para este fim, possuem vantagens tanto em relação ao armazenamento em memória quanto em velocidade de acesso aos elementos.
Answer: Pode fazer usando dois for:
principais_palavras = "bom dia vamos lindo"

base = """Bom dia
que dia lindo
vamos embora
vamos chutar o dia"""

lista_prioritaria = principais_palavras.lower().split(" ")
linhas = base.lower().split("\n")

for linha in linhas:
  for palavra in lista_prioritaria:
    if (palavra in linha):
      print(linha + " - " + palavra)
      break

A lógica da leitura do arquivo não faz parte do algoritmo, portanto ficou de fora, mas é só adaptar ao seu caso.
Comment: Se me permite o comentário, a ideia de utilizar o `lower()` é boa caso a intenção seja desconsiderar as caixas do texto, mas o `split()` na linha para converter em uma lista de palavras é desnecessário. O operador `in` funciona também para *string*, servindo para buscar se existe a *substring*.
Comment: Quando ao `lower()`, coloquei porque presumi que a caixa deve ser desconsiderada na busca, baseando-me no exemplo dado pelo AP. Quando ao `split()`, tem toda a razão. Não sei onde estava com a cabeça. Vou simplificar. Obrigado. :)


Is a policy always deterministic in reinforcement learning?

Question: In reinforcement learning, is a policy always deterministic, or is it a probability distribution over actions (from which we sample)? If the policy is deterministic, why is not the value function, which is defined at a given state for a given policy $\pi$ as follows
$$V^{\pi}(s) = E\left[\sum_{t>0} \gamma^{t}r_t|s_0 = s, \pi\right]$$
a point output?
In the above definition, we take an expectation. What is this expectation over?
Can a policy lead to different routes?
Comment: A related question on StackOverflow: https://stackoverflow.com/q/46260775/712995
Answer: There are multiple questions here: 1. Is a policy always deterministic? 2. If the policy is deterministic then shouldn't the value also be deterministic?  3. What is the expectation over in the value function estimate? Your last question is not very clear "Can a policy lead to routes that have different current values?" but I think you mean: 4. Can a policy lead to different routes?

A policy is a function can be either deterministic or stochastic. It dictates what action to take given a particular state. The distribution $\pi(a\mid s)$
is used for a stochastic policy and a mapping function $\pi:S \rightarrow A$ is used for a deterministic policy, where $S$ is the set of possible states and $A$ is the set of possible actions.
The value function is not deterministic. The value (of a state) is the expected reward if you start at that state and continue to follow a policy. Even if the policy is deterministic the reward function and the environment might not be.
The expectation in that formula is over all the possible routes starting from state $s$. Usually, the routes or paths are decomposed into multiple steps, which are used to train value estimators. These steps can be represented by the tuple $(s,a,r,s')$ (state, action, reward, next state)
This is related to answer 2, the policy can lead to different paths (even a deterministic policy) because the environment is usually not deterministic.
Comment: A classical example is a robot that takes moves left by one step (action) but the surface is slippery (walking on ice) so it actually moves 2 steps left. In fact such environments are the norm and extensively studied.
 My example is actually as well known "toy"
 environment :https://gym.openai.com/envs/FrozenLake-v0/
Comment: can you give me an example of environment not being deterministic? as i see it, if the agent applies action $a$ to an environment in state $s$, it deterministically changes environment to $s^`$
Comment: so state $s$ and action $a$ upon it leads to a prob dist over $s^`$. i got  that right?
Comment: Yes, just like $p(a\mid s)$ is stochastic, $p(s' \mid s, a)$ is also stochastic.
Comment: just two more things: 1. $p(a|s)$ is stochastic only in stochastic policy right? 2. Can you confirm the other answer posted is wrong about what the expectation is taken over part so i can accept your answer?
Comment: 1. yes, 2. the other answer is not "wrong" but maybe less clear... the decomposed steps $(s,a,r,s')$ are usually stored and used "training examples" to learn a value function. The other answer is specific to an implementation (IMO)
Comment: Should the value function be deterministic given the state? You are taking the expectation, and V(s) itself is not a random variable, right?
Comment: @A.D just want to clarify, you can say s' or a is random, but you can't say p(a|s) is random. stochastic or random is used to describe a variable. Actually, if the environment is stable, the probability density p(a|s) itself doesn't change.
Answer: The policy can be stochastic or deterministic.  The expectation is over training examples given the conditions.  The value function is an estimate of the return, which is why it's an expectation.


ModuleNotFoundError when trying to run python script from javascript

Question: I have my index.js:
In my index.js i have some code which calls the ML model that i have trained and gets the predicted y value for the x value that i am providing. Everytime i am running the code it says pandas not recognised and if i move that import to down line it actually shows the same statement for al modules though they are installed perfectly.
const express = require('express');
const { spawn } = require('child_process');
const app = express();
const port = 3000;

app.get('/predict', (req, res) => {
  // Get the input value from the query parameter
  const inputValue = parseFloat(371);

  // Spawn the Python script with the input value as an argument
  const childPython = spawn('/usr/bin/python3', ['index.py', inputValue.toString()]);

  // Capture the Python script's stdout
  childPython.stdout.on('data', (data) => {
    // Send the prediction back as a response
    res.send(`Prediction: ${data}`);
  });

  // Handle errors
  childPython.stderr.on('data', (data) => {
    console.error(`Error: ${data.toString()}`);
    });
});

app.listen(port, () => {
  console.log(`Server running on http://localhost:${port}`);
});

Also i have my index.py file which will perform some prediction and has to return the result back to index.js:
import sys
import pandas as pd
from sklearn.tree import DecisionTreeRegressor
# Load your dataset here
data = pd.read_csv("biomass-biogas.csv")

# Get the input value from the command-line argument
input_value = float(371)

# Your model and data processing code
X = data.drop(['BIOGAS'], axis=1)
y = data['BIOGAS']
feature_names = X.columns.tolist()
X = X.values
model = DecisionTreeRegressor()
model.fit(X, y)

# Use the input value to make a prediction
prediction = model.predict([[input_value]])

# Print the prediction
print(prediction[0])

The error says: Error:     import pandas as pd
ModuleNotFoundError: No module named 'pandas'
I need to fix the above stated problem
Answer: Looks like a case of your Python environment at location /usr/bin/python3 not having the library pandas. You'll need to install pandas in that python environment first.
Do the following:

which python3 (shows the python directory your terminal will run python from)
If the path /usr/bin/python3 shows up jump to step 4, otherwise do the following
export PATH=/usr/bin/python3:$PATH (sets the PATH variable to the /usr/bin/python3. This is a temporary fix... if you wish I can elaborate on a more permanent fix.)
python3
pip install pandas

You should be all set.


The cardinality of the classic Hilbert space

Question: Question. The classic Hilbert space consists of all infinite sequences $(x_n)$ of real numbers, called points, for which the series $x_1^2+x_2^2+\cdots $ converges. Show that the classic Hilbert space contains just as many points as the real in $\mathbb R$.
Proof says $c\le \vert H\vert \le c^{\aleph_0} =(2^{\aleph_0})^{\aleph_0}=2^{\aleph_0 \aleph_0}=c. $
Why is $c\le \vert H\vert \le c^{\aleph_0}$ so trivial that the book adds no explanation to it?
Comment: It's not trivial. The book apparently assumes a knowledge of some basic results  about cardinal arithmetic.
Answer: $H = \{(x_1,x_2,x_3,\ldots) \in \mathbb R^{\aleph_0} : x_1^2+x_2^2+x_3^2+\cdots \} \subseteq \mathbb R^{\aleph_0}$ and that last set has cardinality $c^{\aleph_0}$.
As to the first inequality, every one-dimensional subspace of $H$ has cardinality $c$.


Control Visibility of Paragraph Without Fields

Question: I'm preprocessing a paragraph to add a custom form but I'm having trouble with the visibility. I don't need any fields on this paragraph.
If I just add the form to the paragraph the whole paragraph is hidden for anonymous users.
function my_module_preprocess_paragraph__my_paragraph(&$variables) {
  $paragraph = $variables['paragraph'];
  $form = \Drupal::formBuilder()->getForm('Drupal\my_module\Form\MyCustomForm');
  $variables['content']['form'] = $form;
}

However, if I add a title field to that paragraph everything shows up fine. Is there a way I can get the form to show without having to add fields?
Comment: You can use hook_field_widget_form_alter and hide the fields with #access FALSE
Answer: Implement hook_field_widget_alter
It will give control of paragraph fields.
function hook_field_widget_entity_reference_paragraphs_form_alter(&$element, &$form_state, $context) {
  if ($element['#paragraph_type'] == 'xyz') {
    $element['subform']['field_name']['#access'] = FALSE;
  }
}


How to recognize one item belong to which group when load a plist file?

Question: I use codes below to load plist file
<code>    NSDictionary* tSetDict = [NSDictionary dictionaryWithContentsOfFile:plistPath];
NSArray* prefs = [tSetDict objectForKey:@"PreferenceSpecifiers"];

// Iterate through dictionaries to find required value
for (NSDictionary* setDict in prefs){

            //.........
    }
}
</code>
I hope to know setDict belongs to which group (section).
Is it possible?
Welcome any comment
Thanks
interdev
Answer: Not sure if I understand your question correctly but I will try anyways....
Use the simple/slow iteration:
<code>for (int i=0;i < [prefs count];++i)
{
  NSDictionary *setDict = [prefs objectAtIndex:i];
  NSLog(@"got the dict from section %d", i);
}
</code>
Comment: Thanks. I mean when iPhone app 'Setting' loads setting.bundle plist file. It will load the groups(PSGroupSpecifier) one by one(in each group there are elements PSToggleSwitchSpecifier, PSSliderSpecifier,  PSTitleValueSpecifier, PSTextFieldSpecifier)
Comment: and app 'Settings' will display the groups in different sections of a UITableView.
Comment: Now, if I have got the element(PSToggleSwitchSpecifier, PSSliderSpecifier, PSTitleValueSpecifier, PSTextFieldSpecifier) how can I know it belongs to which group.
Comment: for example:PSTextFieldSpecifier
Comment: it has keys:Title, Key, DefaultValue, IsSecure, KeyboardType, AutocapitalizationType,AutocorrectionType. But there is no key to indicate the "Text Field Element"(Type:PSTextFieldSpecifier) belong to which group.
Comment: I found it look likes app "Settings" load a "Group Element"(type:PSGroupSpecifier) at first, then display all elements/items follow it in a same group until load the next "Group Element"(type:PSGroupSpecifier). The next "Group Element"(type:PSGroupSpecifier) and the elements follow it will display in the next group/section.
Comment: Sorry I got you wrong. For the future: use the comment section for shortly describing issues with the solution posted. If you need to rephrase or enhance your question, use the Edit-function of your original question.


find solutions of the pffafian differential equation

Question:
Find the solution of:

$$y(x+4)(y+z)\,dx - x(y+3z)\,dy + 2xy\,dz=0$$
We know it is integrable, but for the solution, we tried some methods and got stuck. We are studying from the book of Ian Sneddon.
Comment: Are you sure of the equation?
Comment: yes, I am. Sneddon, page 33, problem 6.
Answer: Hint:
$y(x+4)(y+z)~dx-x(y+3z)~dy+2xy~dz=0$
$\dfrac{(x+4)(y+z)~dx}{x}-\dfrac{(y+3z)~dy}{y}+2~dz=0$
which can find the clear solving route that $z\to y\to x$
Answer: Suppose your Pfaffin DE is solvable, that is, $X\cdot curl X= 0$ is satisfied for $X= (P, Q, R)$ with $P=y(x+4)(y+z)$, $Q=-x(y+3z)$, and $R=2xy$.
Let $x$ treat as constant, so $dx=0$. Thus we have $-x(y+3z)dy+2xydz=0$, or $-(y+3z)dy+2ydz=0$, which is a linear ODE in $y$ and $z$:
$$\frac{dz}{dy}-\frac{3}{2y}z=\frac{1}{2}$$ with the solution $c_1
= U(x, y, z)=zy^{-3/2}+y^{-1/2}$. Due to the method of solution of the Pfaffin DE there is $\mu$ such that $\mu Q=\frac{\partial U}{\partial y}$ and $\mu R=\frac{\partial U}{\partial z}$. Here it is not difficult to see that $\mu=\displaystyle \frac{1}{2xy^{5/2}}$. Then we compute $K=\mu P-\frac{\partial U}{\partial x}=\frac{x+4}{2x}U=(1/2+2/x)U$. This implies a linear ODE in $U$: $\frac{dU}{dx}+\frac{x+4}{2x}U=0$. Solving this we get $$ x^2e^{x/2}U(x, y, z)=x^2e^{x/2}(zy^{-3/2}+y^{-1/2}) =c_2.$$


Parameters with categorical and gamma distributions in posterior distribution

Question: I'm following a very good IPython notebook (the whole list can be found here) in which some sampling techniques are explained. However, I don't understand the use of a categorical variable in a change point model.
The posterior distribution for the model is the following:
\begin{aligned}
P( \lambda_1, \lambda_2, \tau | \mathbf{y} ) &\propto \left[\prod_{t=1851}^{\tau} \text{Poi}(y_t|\lambda_1) \prod_{t=\tau+1}^{1962} \text{Poi}(y_t|\lambda_2) \right] \text{Gamma}(\lambda_1|\alpha,\beta) \text{Gamma}(\lambda_2|\alpha, \beta) \\
&\propto \left[\prod_{t=1851}^{\tau} e^{-\lambda_1}\lambda_1^{y_t} \prod_{t=\tau+1}^{1962} e^{-\lambda_2} \lambda_2^{y_t} \right] \lambda_1^{\alpha-1} e^{-\beta\lambda_1} \lambda_2^{\alpha-1} e^{-\beta\lambda_2} \\
&\propto \lambda_1^{\sum_{t=1851}^{\tau} y_t +\alpha-1} e^{-(\beta+\tau)\lambda_1} \lambda_2^{\sum_{t=\tau+1}^{1962} y_i + \alpha-1} e^{-\beta\lambda_2}
\end{aligned}
After this, the parameters are given by:
$$\lambda_1 \sim \text{Gamma}\left(\sum_{t=1851}^{\tau} y_t +\alpha, \tau+\beta\right)$$
$$\lambda_2 \sim \text{Gamma}\left(\sum_{t=\tau+1}^{1962} y_i + \alpha, 1962-\tau+\beta\right)$$
$$\tau \sim \text{Categorical}\left( \frac{\lambda_1^{\sum_{t=1851}^{\tau} y_t +\alpha-1} e^{-(\beta+\tau)\lambda_1} \lambda_2^{\sum_{t=\tau+1}^{1962} y_i + \alpha-1} e^{-\beta\lambda_2}}{\sum_{k=1851}^{1962} \lambda_1^{\sum_{t=1851}^{\tau} y_t +\alpha-1} e^{-(\beta+\tau)\lambda_1} \lambda_2^{\sum_{t=\tau+1}^{1962} y_i + \alpha-1} e^{-\beta\lambda_2}} \right)$$
How can we obtain the distributions for these parameters? By marginalizing the posterior distribution? In the case of $\lambda_{1}$ and $\lambda_{2}$ each parameter is described by a Gamma distribution multiplied by some constant, so maybe that is the idea. In the case of $\tau,$ I can't see why we obtain a categorical distribution with that argument.
Comment: Ones made out of paper. Small laptop-like devices.  A scientific notebook for recording lab notes and such. Various [pieces of software](http://www.mackichan.com/techtalk/articles/mmaj01__1.png), or various kinds of files used by various kinds of software. The term 'notebook' might mean myriad things to a person unused to Python.
Comment: Not in the least. If someone hasn't seen Python at all, why would they have guessed "notebook" means "Python notebook" by reading your question? There was no hint in the title or the tags, or in the body text. If they click the link they might eventually work it out, but a person shouldn't have to click links to figure out what you're saying. I *know* what a Python notebook is and *I* didn't realize that was what you meant for a while.
Answer: I would like to write this as a comment but probably is too big. So i am writting it as an answer.
Now, for the $\lambda_1, \lambda_2$ the distributions are just the full conditional distributions for these parameters. These can be found from the full posterior for the first holding (in very simple words) only what depends on $\lambda_1$ that is
$$p(\lambda_1|\ldots)\propto \lambda_1^{\sum_{1851}^{\tau}y_{i}+a-1}e^{-(\beta+\tau)\lambda_{1}}.$$
i.e you find the distribution of $\lambda_1$ conditioned on all the other variables like they are known. (i cannot explain this well, i hope you get the meaning)
Same idea for $\lambda_2$.
Now for the categorical he uses as probability the normalized posterior distribution.
Taking samples from the distributions $p(\lambda_1|\ldots),p(\lambda_2|\ldots),p(\tau|\ldots)$ with a gibbs sampler it is like having sampled from the joint distribution $p(\lambda_1,\lambda_2,\tau|\ldots)$.
Comment: Thanks. When you say that we are taking only what depends on $\lambda_{1}$, do you mean that we are ignoring the rest? Because what I usually see is that in order to obtain $p(\lambda_{1}|...)$ I would need to marginalize over the other variables. Another question. Could you elaborate on why we use a categorical distribution with the normalized posterior distribution? That seems very odd.
Comment: We are ignoring them in the following meaning. Suppose (for the case of $\lambda_1$ that we know a value for $\lambda_2$ and for $\tau$ then the terms of the posterior that dont include $\lambda_1$ would just be constants. So you dont have to integrate, you just drop them out. I hope this helps you a little bit. Now for the categorical i will tell you what i understand from the context, maybe this will help you. Since it is a generalization of the Bernoulli its argument must be a probability thus the normalization. The outcome of this will determine the period, if it is the early or the late.
Comment: 1+ Oh, I see. Yes, that could be the reason, although I have to verify that the categorical distribution does behave the way it should. Thank you very much!
Comment: I already checked the categorical distribution. Yes, the idea is that we choose a likely better $tau$ according to the joint distribution, but using [this](http://stackoverflow.com/a/6432707/460147) nice trick to get a change point position. Then we iterate again using that $\tau$ to obtain an even better $\lambda_{1}$ and $\lambda_{2}$ and so on, until convergence.
Comment: exactly! you said what i was trying to say. glad you figured it out. Thank you for accepting the answer also.


Understanding the relative interior of a set

Question: Let $S\subset \mathbb{R}^n$ be a set. I work with the following definition of the relative interior
\begin{equation}
\text{relint}(S):=\{s\in S\,\,|\,\,\exists\,\delta>0: B(s,\delta)\cap \text{aff}(S) \subset S\}.
\end{equation}
My questions:

Am I correct that this is nothing else than the interior of $S$ with respect to the subspace topology on $\text{aff}(S)$? I do not immediately see from the above definition why this is the case, but it seems to me like this is true.
Is it correct that we have $\text{relint}(S)\neq \text{int}(S)\cap\text{aff}(S)$? I thought that we could choose $S$ to be a closed segment, where $\text{int}(S)$ is empty and $\text{relint}(S)$ equals the open segment.
Is it true that we always have $\text{int}(S) \subset \text{relint}(S)$? I think this holds as $B(s,\delta)\subset S$ immediately implies $B(s,\delta)\cap\text{aff}(S)\subset S$.

Thank you in advance.
Comment: What is $\text{aff}(S)$? I've never seen that notation used.
Comment: It's the affine hull of $S$.
Comment: Could you define it in the question? I feel that it's not a super standard object.
Comment: You're correct on all three counts.
Comment: @Jose27 An affine subset of $\mathbb R^n$ (also called an "affine manifold") is what you get if you take a subspace of $\mathbb R^n$ and shift it away from the origin. The affine hull of a set $S \subset \mathbb R^n$ is the smallest affine set containing $S$. In other words, it's the intersection of all the affine sets which contain $S$. The affine hull of $\{(x,y,z) \mid x^2 + y^2 =1, z=1\}$ is the plane $z = 1$, for example.
Answer:
Yes this is true: from the definition of subspace topology interior, if $x \in \text{int}_{\text{aff}(C)}( C)$ then there exists a set of the form $U \cap \text{aff}(C)$ contained in $C$ that contains $x$ for some $U$ open in $\mathbb{R}^k$. Then choosing an open ball $B$ containing $x$ and contained in $U$, we obtain $x \in B\cap \text{aff}(C) \subseteq C$ so $x \in \text{ri}(C)$. See if you can prove the other inclusion yourself.

In general this is false, as you said. Take $[0,1]\times\{0\}$ in $\mathbb{R}^2$. The relative interior is $(0,1)$ while the interior is empty, and the affine hull is the $x$-axis.

True, for the reasons you stated.
Comment: Thank you. Can you tell me where I can find the definition of the subspace topology interior?
Comment: @physicist23 Subspace topology interior is just interior but your sets must be open in the subspace topology. So if $A \subseteq B \subseteq C$ and we give $B$ the subspace topology, then $\text{int}_B(A)$ is the set of $x$ for which there exists an open set $O$ (in the $B$ subspace topology) containing $x$ and contained in $A$. Every such $O$ is of the form $U \cap B$ for $U$ open in the $C$ topology. Here is a link to a proper discussion of subspaces: http://www.math.toronto.edu/ivan/mat327/docs/notes/07-subspaces.pdf
Comment: I see! For an arbitrary topological space $(X, \mathcal{O})$ and a subset $A\subset X$, the interior of $A$ is defined by $\text{int}_X(A):=\{x\in A\,|\, \exists\, O\in\mathcal{O}: x\in O\subset A\}$. And here we choose the topological space as $\text{aff}(S)$ with the subspace topology. I guess the remaining inclusion $\text{relint}(S) \subset \text{int}_{\text{aff}(S)}(S)$ is then clear as we find a $\delta >0$ with $x\in B(x,\delta)\cap\text{aff}(S)\subset S$ and the set $B(x,\delta)\cap\text{aff}(S)$ is open in $\text{aff}(S)$ by definition of the subspace topology.
Comment: @physicist23 precisely!


Como Resetar Campos Select2 e Multiselect do Formulário

Question: Tenho um formulário de cadastro onde eu reseto os campos ao inserir um registro com o seguinte:
document.getElementById(id).reset(); //Aqui passo o id do formulário

O problema é que agora tenho um campo select2 e um multiselect que não estão sendo resetados.
<select name="funcao_id" class="form-control select2">
    ...
        <option>...</option>
    ...
</select>

<select multiple id="exames" name="exames[]" class="form-control">
    ...
        <option>...</option>
    ...
</select>

Coloquei apenas as partes relevantes do código, caso precise eu ponho algo mais.
Answer: A sintaxe para fazer reset do select2 é .select2("val", "");.
Podes usar assim:
$('select').select2(); // iniciar o select2
$('form').on('reset', () => { // acionar quando o reset acontecer
  $(this).find('select').select2("val", ""); // fazer reset do select2
});

jsFiddle: http://jsfiddle.net/a7gx0yyd/
Answer: tenta assim:
$('#multiselect').prop('selectedIndex',0);
$('#select2').prop('selectedIndex',0);

esse código vai deixar selecionado o primeiro item


Which toolbox for Belief Propagation and other inference methods in graphical models?

Question: Which open-source software (toolbox) do you think is the best for modelling graphical models (e.g. factor graphs), and doing inference on them? (the language doesn't matter)
Answer: Another is the gRain package in R.
Answer: I found LibDai very useful:
http://staff.science.uva.nl/~jmooij1/libDAI/
Also Factorie is very easy to use.
Answer: I recently had written PyGModels in Python capable of doing inference over LWF Chain Graphs:
It is quite lightweight as it depends only on standard library. However for heavy lifting, I suggest:

pgmpy
pomegranate

The development on PyGModels is very active and documentation severely lags.
However there is a stable release which had appeared in JOSS:
Eraslan, D. K. (2021) 'PyGModels: A python package for exploring probabilistic graphical models with graph theoretical structures', Journal of Open Source Software, 6(61). doi: 10.21105/joss.03115.


How do I calculate distance between GPS co-ordinates on a processor with poor floating point support?

Question: I need to calculate the distance between GPS co-ordinates to calculate distance traveled. I've tried both the Haversine and Vincenty algorithms, which work fine on my desktop PC, but when I port the code to the dsPIC, they return 0 for points that are close (within several meters) due to a lack of floating point precision and poor implementations of sin and cos.
For my use case, my points will be no more than 10 meters apart and will all fall within 10km of each other. I've tried the following algorithm and the results seem ok:
double dist(double latA, double lonA, double latB, double lonB)
{
    double latD = fabs(latA - latB) * 111.3;
    double lonD = fabs(lonA - lonB) * 111.3 * cos(latA * 3.14159265 / 180);

    return sqrt(latD*latD + lonD*lonD) * 1000;
}

Assuming the distance for every 1° is 111.3km, I use the pythagorean theorem to calculate distance. Is there any easy way to improve my algorithm? Or are there any other algorithms that don't depend on a highly accurate sin/cos?
Comment: A couple things.  (1) I would make sure that the FP routines (fabs, cos, sqrt) are taking doubles and not floats.  Could be that your doubles are being "downgraded" to floats on the calls.  With the PIC platform (assuming Microchip tools), anything is possible.  (2) Do you have the ability to step through the code on the PIC (simulator / emulator)?  I'd suggest stepping through the code on both platforms & seeing where the results (temp values, return values) start to diverge.  That's where you start looking.  Eliminate the guesswork if you can.
Comment: I'm using MikroElectronica's compiler rather than Microchip's. I'm beginning to believe their tools are just bad and I should switch.
Comment: I  hate to say it but it sounds like the wrong processor was chosen for the task.
Comment: @FigBug - if that's the company in Belgrade, I witnessed a client lose a lot of time finding & verifying 3 subtle (and egregious) bugs in their compiler.  This was 4-5 years ago.  I avoid PIC if it's my choice, but when I have to go that route, I'll definitely avoid MikroElectronica.  Good luck!
Comment: "will be no more than 10 meters apparent" -- huh? what does "apparent" mean in this context?
Comment: MikroElectronica's compiler only supports single-precision floating point (float, double, and long double are all 32-bit only).To resolve to 10 metres you need at least 7 digits of precision, and you probably need to resolve to at least a metre, so 8 digits are required.  You need a different compiler or used fixed point as I suggested.  Unfortunately your compiler does not support a 64-bit integer type either, so you will have to do a lot of work to implement this. There are a number of ARM Cortex-M3 or M4 parts available that are less expensive and have more on-chip resources than the dsPIC.
Answer: The accepted algorithm for use in Marine AIS systems (specified in IEC61193-2) is the Rhumb Line algorithm.  I have successfully implemented this on a target using Anthony Williams' fixed point maths library, which uses the CORDIC algorithm, and will I believe typically give a bout 5x performance improvement over software floating point.
However the library is C++ rather than C, which makes it easy to use due to extensive operator overloading, but is not perhaps what you are looking for.  Worth considering using C++ compilation for your C code however, just for the benefit of this library.  The problem with that of course is that Microchip's C31 compiler bizarrely does not support C++.
One caveat however is that the look-up table for the log() function is too short by one value and needs an additional element at the end with value zero.  This was confirmed by Anthony after I found it, but I do not believe that he has updated the download.
Either way, the answer is probably to use fixed point, and CORDIC.
To resolve to 1m of longitude or equatorial arc, you will need 8 digits of precision, so a single precision float will be insufficient, using double precision will slow things considerably.  Checking MikroElectronica's C User Manual reveals that the compiler only supports single precision - float, double, and long double are all 32-bit, so there is no way to achieve the accuracy you need using the built-in FP types in any case with this compiler.
If it is of any use, here is my Rhumb Line code using Anthony's library:
Header:
#if !defined cRhumbLine_INCLUDE
#define cRhumbLine_INCLUDE

#include "fixed.hpp"

//! Rhumb Line method for distance and bearing between two geodesic points
class cRhumbLine
{
public:

    //! @brief Default constructor
    //!
    //! Defines a zero length line, bearing zero
    cRhumbLine() : m_update_bearing(false), m_update_distance(false), m_distance(0), m_bearing(0) {}

    //! @brief Constructor defining a line
    //!
    //! @param startLatDeg  Start latitude in degrees, negative values are south of equator
    //! @param startLonDeg  Start longitude in degrees, negative values are west of meridian.
    //! @param endLatDeg    End latitude in degrees, negative values are south of equator
    //! @param endLonDeg    End longitude in degrees, negative values are west of meridian.
    cRhumbLine( fixed startLatDeg, fixed startLonDeg, fixed endLatDeg, fixed endLonDeg )
    {
        define( startLatDeg, startLonDeg, endLatDeg, endLonDeg ) ;
    }

    //! @brief Define a start/ent point
    //!
    //! @param startLatDeg  Start latitude in degrees, negarive values are south of equator
    //! @param startLonDeg  Start longitude in degrees, negative values are west of meridian.
    //! @param endLatDeg    End latitude in degrees, negarive values are south of equator
    //! @param endLonDeg    End longitude in degrees, negative values are west of meridian.
    void define( fixed startLatDeg, fixed startLonDeg, fixed endLatDeg, fixed endLonDeg ) ;

    //! @brief Get start-end distance in nautical miles
    //! @return Start-end distance in nautical miles.
    fixed distanceNm() { return distanceMetres() / ONE_NM_IN_METRE ; }

    //! @brief Get start-end distance in metres.
    //! @return Start-end distance in metres.
    fixed distanceMetres()  ;

    //! @brief Get start-end bearing in degreed.
    //! @return Start-end bearing in degreed (0 <= x < 360).
    fixed bearingDeg()  ;

private:
    static const int ONE_NM_IN_METRE = 1852 ;

    bool m_update_bearing ;
    bool m_update_distance ;
    fixed m_distance ;
    fixed m_bearing ;

    fixed m_delta_phi ;
    fixed m_delta_lon ;
    fixed m_delta_lat ;
    fixed m_lat1_radians ;
} ;

#endif

Body:
#include "cRhumbLine.h"

void cRhumbLine::define( fixed startLatDeg, fixed startLonDeg, fixed endLatDeg, fixed endLonDeg )
{
    fixed lat1 = startLatDeg / 180 * fixed_pi ;
    fixed lon1 = startLonDeg / 180 * fixed_pi ;
    fixed lat2 = endLatDeg / 180 * fixed_pi ;
    fixed lon2 = endLonDeg / 180 * fixed_pi ;

    m_update_bearing = true ;
    m_update_distance = true ;

    m_delta_phi = log( tan( lat2 / 2 + fixed_quarter_pi ) / tan( lat1 / 2 + fixed_quarter_pi ) ) ;
    m_delta_lat = lat1 - lat2 ;
    m_delta_lon = lon1 - lon2 ;
    m_lat1_radians = lat1 ;

    // Deal with delta_lon > 180deg, take shorter route across meridian
    if( m_delta_lon.abs() > fixed_pi )
    {
        m_delta_lon = m_delta_lon > 0 ? -(fixed_two_pi - m_delta_lon) : (fixed_two_pi + m_delta_lon) ;
    }
}

fixed cRhumbLine::distanceMetres()
{
    if( m_update_distance )
    {
        static const fixed mean_radius = 6371009 ; // in metres. Source: International Union of Geodesy and Geophysics (IUGG)

        fixed q = m_delta_phi != 0 ? m_delta_lat / m_delta_phi : cos( m_lat1_radians ) ; // Deal with lines with constant latitude

        m_distance = sqrt( ( sqr(m_delta_lat) + sqr(q) * sqr(m_delta_lon) ) ) * mean_radius ;
        m_update_distance = false ;
    }

    return m_distance ;
}

fixed cRhumbLine::bearingDeg()
{
    if( m_update_bearing )
    {
        static const fixed mean_radius = 6371009 ; // in metres. Source: International Union of Geodesy and Geophysics (IUGG)

        m_bearing = atan2( m_delta_lon, m_delta_phi ) / fixed_pi * 180 ;
        if( m_bearing == -180 )
        {
            m_bearing == 0 ;
        }
        else if( m_bearing < 0 )
        {
            m_bearing += 360 ;
        }
        m_update_bearing = false ;
    }

    return m_bearing ;
}

Comment: @Pablo: Add `0x0LL` to the end of the `log_two_power_n_reversed[]` array (it should have 36 elements).  I have also more recently found the sqrt() to be less precise that it could be for very small values - it can be improved upon.  I have replaced the sqrt() function with the one presented as the last code fragment in *[another question](http://stackoverflow.com/questions/8721022/how-to-improve-fixed-point-square-root-for-small-values)*.
Answer: Some comments:

You need to specify the range and the accuracy requirements of your computation. Range and accuracy are extremely important in determining what approach you use to calculate cosines. In addition, the pythagorean approximation you posted works well if the relative differences in latitude and longitude are small compared to the angular distance to the pole. Your pseudo-pythagorean algorithm won't work well at high latitudes if the latitudes aren't close together. (e.g. with latitude 43.001 and 43.002 it would work well, but not at 89.961 and 89.962)
Longitudes need to be calculated in view of their circularness -- Your algorithm will fail around the international date line, but that can be easily remedied by taking the longitudinal difference symmetric-modulo 360, where smod(x,m) = mod(x+m/2,m)-m/2. (e.g. -179.5 - +179.5 = -359 degrees, but if you compute smod(-359,360) you get +1 degrees.)
At design time, make good use of your PC. You have a very powerful calculator available, and you could evaluate for a large # of test points the high-precision answer and your approximation, and see how they differ, to evaluate accuracy. If you deduce a pattern in this information, you could use it to make a 2nd-order approximation to increase accuracy.

update: You state that your range/accuracy requirements are +/-60 degrees (no advantage in reducing range in one hemisphere) and 1% accuracy. A good approximation of cos(x), with x in degrees, within this range is c2(x) = 0.9942 - 1.39*10-4 * x2 = 0.9942 - (0.01179x)2; its error over this range has a maximum value of 0.006.
If you want better accuracy use a 4th degree polynomial (c4(x) = 0.999945-(0.01233015x)2+(0.007778x)4 has a maximum error over this range of less than 6x10-5, but is much more sensitive to parameter errors  and arithmetic errors) or split into multiple ranges.
Comment: My system is currently for water sports, so hopefully it won't be used near the poles. I think it's safe to say it won't be used north of 60°N or south of 40°S. I feel consumer GPS is going to limit my accuracy more than my choice of algorithm. As long as error is less than 1%, I think I'll be happy.
Comment: My other idea to improve my algorithm is use a table of radius of curvatures (http://en.wikipedia.org/wiki/Latitude#Degree_length) and then do a lookup and interpolate the radius.
Comment: @FigBug: Do yourself a favor: get the table, do a polynomial fit, and use that, rather than the table + interpolation. Table interpolation is a pain to implement, takes more memory + calculation than polynomial approximations. Its strengths are when you have a very weird wiggly function.
Answer: You may want to try to use a precomputed table for sin and cos.
It uses more memory, can trash the cache on general purpose processor (not your case) but will have as much accuracy as possible on your processor and will be quite fast.
Comment: @Adam: OP has specified neither the range nor the accuracy he needs; high accuracy high range implies large tables, but in other cases perhaps they are appropriate. (although I tend to shun tables in favor of polynomial evaluation)
Comment: He's on a dsPIC with a few K of memory. He needs about 11k entries. Looks like a table is not an option to me.
Answer: You're on a fixed point DSP (effectively) so you might want to look into fixed-point functions; they are likely to perform better.
Turns out that Microchip have a fixed-point library available: http://www.microchip.com/stellent/idcplg?IdcService=SS_GET_PAGE&nodeId=2680&dDocName=en552208 I don't know how helpful that will be - it may lack the precision you need.
And here is an example of how to do it yourself: http://www.coranac.com/2009/07/sines
Back on track - the Microchip page suggests that their compiler and library are IEEE-754 compatible for both single and double precision. Unless they're telling a half-truth and are using the half-precision (16-bit) format. If you still aren't getting the results you need I'd consider filing a bug report.


How do I produce 2 lists with one display?

Question: I'm back on this issue for me because I would to implement Responsive slideshows with thumbnails and many of those use 2  elements to display thumbs and full image.
So I know I need to edit the style output template or the row style output template but I don't know how to find values provided by views to print them inside my template.
This doesn't explain how to find variables.
I'm looking for some example and explanations to understand how to do.
So I get the style output template, and I have done as you said to print fields. I want to modify the field settings in the view.
<?php if (!empty($title)): ?>

<?php print $title; ?>

<?php endif; ?>

<?php foreach ($rows as $id => $row): ?>
  <li class="<?php print $classes[$id]; ?>">
    <?php print $row; ?>

<?php endforeach; ?>

      <?php foreach ($rows as $id => $row): ?>

    <?php print ($fields['field_galerie_image_vdl_fid_1']->content); ?>

<?php endforeach; ?>

I can't find the right targeting for my desired field, and in the  I need to exclude some part of the output.
Comment: `RWD slideshows` what's that? what does the R, W, and D stand for?
Comment: Responsive slideshow, sorry.
Answer: Under Advanced, Theming: Information, click on Information and you will see something like this:

For template file name you want to use the last suggested name, (which I underlined for you) that way it only affects this specific view and not others.
In the above example Row Style Output: you would use views-view--fields-shipping-cart--block.tpl.php
To print your fields, inside the template file you put

<?php print $fields["title"]->content; ?>

<?php print $fields["field_name"]->content; ?>

Title prints your node title, else if you want to print a field, you use field_name
Remember to flush the cache to see changes or you could click the Rescan template files button
If you're unsure of the name of a field, In view's click on your field, and go to re-write the output of the field, click on replacement patterns, whatever name replacement patterns uses is the one you need to use.
Alternatively, instead of working with template files, you could work with just replacement patterns if you like as my answer shows here
Else if you just want to add classes to your fields, you can do it with views like this:


Please help me with a single variable derivative applied question.

Question: I'm Will. I have a problem with a question.
A man 6 ft tall wants to construct a greenhouse of length L and width 18 ft against the outer wall of his house by building a sloping glass roof of slant height y from the ground to the wall, as shown in the figure below. He considers space in the greenhouse to be usable if he can stand upright without bumping his head. If the cost of building the roof is proportional to y, find the slope of the roof that minimizes the cost per square foot of usable space. Hint: Notice that this amounts to minimizing y/x.
<code>      /|
     / |
  y /  |
   /   |
  /----|
 / |   |6
--------
18    x
</code>
Hmmm please don't mind my bad drawing. This is a cross section of the greenhouse.
I have got to the position:
Let h be the height of the cross section.
h/ 18= (h- 6)/ x, using similar triangles.
So, xh= 18(h- 6).
Differentiating both sides with respect to x while h= (y^2- 18^2)^(1/2) and h'= yy'/(y^2- 18^2)^(1/2).I get:
xh'+ h= 18h',
xyy'+ y^2- 18^2= 18yy'.   (1)
At the same time, because I should minimize (y/x), I differentiate it and get (xy'- y)/x^2, and it should be 0.
So y'= y/x.
I plug y'=y/x into equation (1) and I get
x= 9y^2/(y^2- 9). (2)
Finally I'm totally stuck here. I get the solution which is only a '1' from the textbook without any explanations.
When I plug 1 as the slope into the figure, I find that x should be 12.
So, y should be 18*2^(1/2). But this is in contradiction to equation (2).
I'll be so appreciated for your help!
Comment: I fogot to tell you that this problem comes from the textbook  page 130 problem 27 which has a little start before it.
Answer: here's one way to do it.  I hope it helps you:
Take the variable 'd' as the height of the triangular cross section above the mans head, or the height minus the man's height.  The rectangular area that the man could use separates two right triangles that are proportional such that the lower triangle 'a' contains the same angles as the upper one 'b' but are different sizes.  So 'd' can be expressed as the proportion of height to base of triangle 'a' times the base of triangle 'b' like this:
$d = x (\frac{6}{18-x})$
or
$d = \frac{6x}{18-x}$
//where $[18-x]$ is the base of triangle 'a' and 'x' is the base of 'b'
Now that 'd' is in terms of 'x' you can plug this into the Pythagorean theorem for the whole triangular cross-section to get a formula for the length of the sloped wall 'y' like so:
$\sqrt{18^2 + (6 + (\frac{6x}{18-x}))^2} = y$
//where 6 + (6x/[18-x]) is the man's height plus the distance 'd'
Divide that whole equation by 6x to find the ratio of the length 'y' to that which defines the usable rectangular area.
$\sqrt {18^2 + (6 + (\frac{6x}{18-x})}$  /ALL DIVIDED BY 6X   = 'R' ratio
Finally upon differentiating this (it's very long so I used a calculator) the minimum value for the ratio is $x = 12$.  With 'x' as 12 you can clearly see the slope on triangle 'a' by computing
$18-12 = 6$ (base of triangle 'a')
man's height = 6 (height of triangle 'a')
so the slope of the beam 'y' is $6/6$ or '1'
Answer: With the hint, we wish to minimize $$\frac{y}{x},$$ given the following constraint $$y^{2} = 18^{2} + (\frac{108}{18-x})^{2}.$$
We use implicit differentiation, namely $$(\frac{y}{x})' = \frac{y'x-y}{x^{2}}$$ and $$(y^{2} = 18^{2} + (\frac{108}{18-x})^{2})'$$ gives $yy' = \frac{108^{2}}{(18-x)^{3}}.$
Now, the denominator of $\frac{y'x-y}{x^{2}}$ is positive (there must be some usable space), so the term vanishes if and only if the numerator vanishes. In other words, we require $y'= \frac{y}{x}.$ With this finding, we make a substitution such that $\frac{108^{2}}{(18-x)^{3}} = y (\frac{y}{x}) = \frac{y^{2}}{x}.$
We substitute the constraint in place of $y^{2}$ on the right hand side such that $\frac{108^{2}}{(18-x)^{3}} = \frac{18^{2} + (\frac{108}{18-x})^{2}}{x}.$ Then, we rewrite the equation as follows $\frac{x}{18-x} = 1 + (\frac{18-x}{6})^{2},$ where some algebraic steps have been omitted.
At this point, we substitute $u = 18-x,$ where $u$ denotes the horizontal length in the unusable part of the greenhouse. Namely, $\frac{18-u}{u} = 1 + (\frac{u}{6})^{2},$ which leads to the following cubic equation $u^{3} + 72u - 648 = 0.$ The only real solution is $u=6,$ which is in accord with John's previously posted solution. Therefore, the slope is equal to $1.$


Is the following proof correct?

Question: Is the following proof correct?
Let's say we find integers $x$ and $y$ such that $x^2 ≡ y^2($mod $n)$ and $n$ has at least $2$ distinct factors not equal to $0$ or $n$.
I intend to show that there is at least $50$% chance that gcd$(n, x - y)$ is a non-trivial factor of $n$.
(Assume already proven that if we have an $x$ and $y$ which satisfy the rules $x$ is not congruent to $± y($mod $n)$ and $x^2 ≡ y^2($mod $n)$, then the greatest common divisor of $(x − y, n)$ and the greatest common divisor of $(x + y, n)$ will both be non-trivial factors of $n$.)
Then $(x – y)(x + y) = 0 ($mod $n)$ and the only possibilities for any gcd$(n, x + y)$ or gcd$(n, x - y)$ are that they equal either $n$, $1$, or a non-trivial factor of $n$.
If $x ≡ y($mod$ n)$, then gcd$(n, x - y) = n$.
If $x ≡ -y($mod $n)$, then gcd$(n, x - y) = 1$ or $n$
If x ≡ y(mod n), then gcd$(n, x + y) = 1$ or $n$
If $x ≡ -y($mod$ n)$, then gcd$(n, x + y) = n$
But if $x$ is not congruent to $± y($mod $n)$, then gcd($n, x + y$) does not equal 1 or n. So it must equal one of the at least 2 non-trivial factors of n.
So when $x^2 ≡ y^2($mod$ n)$, there is at least a 50% chance that gcd(n, x + y) is a non-trivial factor of n (a factor not equal to $1$, or $n$) .
Comment: anyone able to verify this or did I miss something?
Comment: I believe you mean $1$ or $n$ in the first line.
Comment: I do mean $0$ actually, just trying to imply that $n$ itself is not $0$ and letting it be true that if $n$ has $2$ distinct factors besides $0$ and $n$, then it has these factors in addition to $1$, so the fact that $2$ such distinct factors exist in addition to $n$ and $0$ implies that they do not equal $1$. But I see how that might not be exactly clear the way I tried to phrase it..
Comment: I would suggest "and $n \ne 0$ has ... to $n$" in that case.
Answer: Let $n$ be $p^3$ a prime to the cube. Then if $x^2=y^2$ mod $n$ there is $0\%$ chance that $gcd(n,x-y)$ or $gcd(n,x+y)$ is neither $1$ nor $n$.
But $n$ has $p$ and $p^2$ which are two distinct factors of $n$.
The problem in your proof is that you always start with "if I have $x$ and $y$ such that $x^2=y^2$ and $x\neq \pm y$ then..." but it might happen that you can never find $x$ and $y$ verifying this (the condition is that $n$ has two distinct $\textit{prime}$ factors).
Actually if $n$ has exactly $r$ distinct prime factors and $gcd(a,n)=1$ then the equation $a=x^2$ mod $n$ has either $0$ solutions (if $a$ is not a square) and $2^r$ solutions if $a$ is a square. In that case having $x^2=y^2$ the probability that $x=\pm y$ is $\frac{1}{2^{r-1}}$.

Is the following proof correct?


Repeatedly split data in training (0.75) and test (0.25) for cross validation

Question: What kind of cross validation is it called when we randomly split the data into 0.75 training and 0.25 test data set. And this split is done 1000 times.
Comment: Why would you want to do it 1000 times?
Comment: I did it a number of times to get the average accuracy. So is it wrong to validate this way?
Comment: Ohh okay.  I mis-read it.  I think it sounds like the `Holdout method`, but I'm not completely sure.
Answer: This is frequently referred to as repeated training-test split, or leave-group-out cross-validation, or Monte Carlo cross-validation$^1$. Note that this should be done with more splits than regular CV (which would have e.g. 10 partitions) - so you doing it 1000 times is a good idea.
$^1$ See e.g. Applied Predictive Modeling by Kuhn and Johnson, Springer, 2013.
Comment: what is the different between this and bootstrap?
Comment: @Matemattica Just looked it up to not give wrong answers: bootstrap by definition does sampling with replacement, which makes it - in this regard - different from most of the other approaches, which usually do sampling without replacement.
Answer: Think you are referring to ShuffleSplit.
Shuffle Split perform random permutation of given data set and splits the data to train and test sets based on the chosen proportions you provide it as input.
As opposed to any other CV method - Shuffle Split never assumes that the folds it produce will be different than one another, meaning samples can appear in more than one fold.
This is really useful in cases like yours - where you may want a large number of random folds - but have a finite data set that can't be divided into a 1000 different folds, in strategies like KFold.
A link to sklearn's implementation of Shuffle Split:
http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.ShuffleSplit.html
Answer: I've also seen it under the name of repeated set validation. (The difference between hold-out and set validation in that context being that set validation results were used like cross validation results as surrogate for a final model trained on all data whereas hold out was refering to using the test results for exactly the model that was tested.


422 Unprocessable Entity Error. I tried upload Excel file to import data but it is not working

Question:
This is my code, i use vuejs and laravel. The function I am doing is both saving the Course and also importing the Excel file to get the data in the file. But when saving the Course is normal, but when it comes to running the validate excel file line, the error is not received. Even though I gave it a name and console.log came out. Can you help me fix it? Thank you so much.

Component - template tag
<form @submit.prevent="store()" @keydown="form.onKeydown($event)">

    Courses
    <select class="form-control" name="education_program_course" v-model="form.education_program_course">
        <option value="" disabled selected="">Select option</option>
        <option value="" disabled>-------</option>
        <option v-for="course in courses" :key="course.course_code" :value="course.course_code">{{ course.course_name }}</option>
    </select>

    Excel File
       <input type="file" class="form-control" id="file_data" name="file_data" ref="fileupload" @change="onFileChange">

     <button :disabled="form.busy" class="btn btn-success btn-lg mt-1" type="submit">Save</button>

</form>

Component - script tag
export default: {
    data() {
       return {
        file_data: "",
        form: new Form({
          education_program_course
      }),
    },
    methods: {
      onFileChange(e) {
           this.file_data = e.target.files[0];
      },
      store() {
            this.form.busy = true;
            let formData = new FormData();
            formData.append('file_data', this.file_data);
            this.form.post('../../api/admin/program/education', formData, {
                    headers: { 'content-type': 'multipart/form-data' }
                })
                .then(res => {
                if(this.form.successful){
                    this.$snotify.success('Sucessful!');
                    this.form.clear();
                    this.form.reset();
                }
            })
            .catch(err => {
                this.$snotify.error(err.response.data.errors.file_data[0]);
            });
        },
    }
  }
}

Controller
$data = $request->validate([
    'education_program_course' => ['required'],
    'file_data' => ['required', 'file', 'mimes:xls,xlsx']
]);
$program = new EducationProgram();
$program->education_program_course = $data['education_program_course'];
$path = $request->file($data['file_data'])->getRealPath();
Excel::import(new ProgramDetailImport, $path);

Comment: @guyg i just send file_data to formData because education_program_course is already available in this.form
Answer: You should append 'education_program_course' on your formdata.
    formData.append('education_program_course', this.form.education_program_course);

Comment: i think education_program_course is already available in this.form
Comment: check your validation error from browser network console
Comment: I checked and it says validation required error. But when I check `console.log(this.file_data)` it still shows that file data.
Answer: I have solved this problem. Change change at script tag -> store()
from:
let formData = new FormData();
formData.append('file_data', this.file_data);

to:
if(document.getElementById("file_data").files[0]){
   this.form.file_data = document.getElementById("file_data").files[0];
}

Adding file_date in form: new Form and remove formData in this.form.post.
Comment: Can you clarify your explanation? What do you mean by "Change change at script tag -> store()`? That doesn't seem to be reflected in your actual suggestion. Am I misunderstanding? If so, others might as well.


Login with different console Firebase (different users)?

Question: it's possible to create an android application(related to Firebase) which provides to every user the ability
to connect to his own firebase , without sharing his information with other users( not the collaboration method ,generally, the app is
about to track object : the user connect to app , get the object information and store it in his account and  than track it) .. so any solution for the login process ?
Comment: Do you mean to say having a custom object inside firebase user Object?
Comment: @erluxman  no not that the problem , all i need is to find a way that the application could connect to different users console( for example mine,  yours ,and other users  )  independently ( the issue is that the app is related to a project which is under a console one person (the owner) ) , okay ?
Answer: Which backend resources your Android code connects to is determined by the FirebaseApp instance. The default app is usually initialized from the google-services.json that you add to your project, so can't be dynamically changed from within the code.
But you can also initialize a FirebaseApp instance from within your Android code. All you need is the pertinent information from the Firebase project of the user, which would otherwise be read from their google-services.json. You'll want to build a FirebaseOptions object in that case, which contains things like the Application ID, Project ID, Database URL and a few more values.
Once you have a FirebaseApp, you can then get the other Firebase services from that. It's quite similar to my answer on accessing multiple databases in a single app from a few years ago. From there:

FirebaseOptions options = new FirebaseOptions.Builder()
        .setApiKey("AI...j0")
        .setApplicationId("1:5...e0")
        .setDatabaseUrl("https://myapp.firebaseio.com")
        .build();
FirebaseApp secondApp = FirebaseApp.initializeApp(getApplicationContext(), options, "second app");
FirebaseDatabase secondDatabase = FirebaseDatabase.getInstance(secondApp);
secondDatabase.getReference().setValue(ServerValue.TIMESTAMP);


A group isomorphic to $\mathbb{Z}_2\times\mathbb{Z}_2\times\mathbb{Z}_3$

Question: I am taking a first course on group theory.
I understand why $\mathbb{Z}_2\times\mathbb{Z}_3 \cong \mathbb{Z}_6$.
How can I use this fact to show that
$\mathbb{Z}_2\times\mathbb{Z}_2\times\mathbb{Z}_3 \cong \mathbb{Z}_2\times\mathbb{Z}_6$ ?
More in general, it looks like (but I am not sure if this is true) that given $\mathbb{G}_1\times\mathbb{G}_2\times\mathbb{G}_3$, I am allowed to take the right most "pair" (i.e. $\mathbb{G}_2\times\mathbb{G}_3$) and replace it with a group isomorphic to it. A sort of associativity rule for the external direct product.
Comment: Yes, your intuition is correct.  You can prove this pretty easily:  $G \times (L \times M) \cong G \times L \times M \cong (G \times L) \times M$.
Comment: [here](https://proofwiki.org/wiki/Associativity_of_Group_Direct_Product) is a proof of associativity.
Comment: This is (surprisingly) not a duplicate.
Answer: The isomorphisms are given by:
$$\begin{align}
\varphi: G\times (H\times K)&\to G\times H\times K,\\
(g, (h,k))&\mapsto (g,h,k);\\
\psi:G\times H\times K&\to (G\times H)\times K,\\
(g,h,k)&\mapsto ((g,h),k);\\
\theta:(G\times H)\times K&\to G\times (H\times K),\\
((g,h),k)&\mapsto (g,(h,k)).
\end{align}$$
Note that the composition of isomorphisms is an isomorphism.
Direct products are commutative via $\rho:(g,h)\mapsto (h,g)$.
Given $\pi: M\cong L$, we have
$$P\times M\cong P\times L$$
via $\pi':(p, m)\mapsto (p, \pi(m))$.
Comment: A common mistake is to think the analogous statement to the last one is true for quotients.
Comment: As to the first part,  this is almost comical.    But that's true,  it's all you need to do.   This may be called "canonical".   But that notion still confuses me.


Do not output small decimal numbers in scientific / exponential notation

Question: Some numbers in my dataset are very small, e.g. 0.00008.
When I write out to CSV, the value in the text itself is 8E-05.
I need the value to be written out to CSV in decimal notation.
Can this be achieved without having to specify the format of every single field in the CsvMap?
Answer: It turns out one can specify a format per-type on the writer Configuration property.
e.g.
using (var writer = new CsvWriter(new StreamWriter(exportCsv, append)))
{
    writer.Configuration.HasHeaderRecord = !append;
    writer.Configuration.RegisterClassMap<GeoVulcanCollarCsvMap>();
    writer.Configuration.TypeConverterOptionsCache.GetOptions<double?>().Formats = new[] { "0.######################" };
    writer.WriteRecords(list);
}


Primality test of quadratic polynomials

Question: There are some quadratic polynomials like $n^2+1$ that there exist infinitely many integers $n$ such that their value is either prime or the product of two primes (if I am right!).
I wanted to know if there are any special efficient primality test methods for these kinds of polynomials. The cases that I'm curious about are: $n^2+1$ and $2n^2-1$.
Answer: If you have a look at any collection of primality tests (say, on Wikipedia) you'll find that some of them rely on the factorization of $m-1$ to test/prove the primality of $m$. Well, those tests should have a leg up for numbers of the form you are asking about.
Comment: Thanks, I briefly looked at them. Proth's test does not help me I think, but Lucas test is suitable for $n^2+1$. Can you provide me a link that shows a efficient method for $2n^2-1$?
Comment: Not offhand, but if $m=2n^2-1$ then $m-1=2(n+1)(n-1)$ so you get a lot of factoring.
Answer: For the case $n^2 + 1$, for special values of $n$ it is very quick to use Pollards $\rho - 1$ algorithm. In particular, if you can guarantee that $n$ is B-powersmooth for some reasonably sized B.
I happen to have written up a brief expostion about this algorithm on my blog, in a TeXed-but-in-a-way-incompatible-to-anything-else format.
Analogously, the $2n^2-1$ case will be quickly factorable for special values of $n$ using Pollards $\rho + 1$ or William's $p + 1$.
But that's all I see right off.
Comment: Thanks @mixedmath, Good hint! Is there any general idea for testing $an^2+bn+c$ or we should examine them case by case?

Primality test of quadratic polynomials


likelihood vs confidence in layman terms

Question: I already referred this post post and post. Don't mark it as duplicate.
I am working on a binary classification problem using logistic regression. Loan default or not.
I have a requirement where I am told to predict the outcome classes for our unseen records and also report the confidence of predictions generated.
I am novice data scientist. So, am not really sure what is the difference between likelihood and confidence?
If I run logistic regression, I get likelihood measure like 70% probability of belonging to class 1 and 30% of probability of belonging to class 0.
My questions are as follows
a) Does likelihood and confidence mean the same? Is there any simple explanation that ordinary layman like me can understand.
b) Is there any tutorial that you can suggest which has on how to report confidence of predictions? When I use scikit-learn for logistic regression, I don't know how can I report confidence for the predictions?
c) Any idea on how can we generate the interval? In scikit-learn logistic regression tutorials that I find online, I only see probability/likelihood of an instance becoming label 1 or label 0. Can you guide me on how can we generate interval?
Comment: 4) This seems like a perfect situation to [predict tendency, not membership.](https://www.fharrell.com/post/class-damage/) Making a [hard classification](https://www.fharrell.com/post/classification/) requires knowledge of what's at stake. What do you lose when someone you predicted as safe winds up defaulting? What do you lose when someone safe is falsely regarded as risky and not approved for a loan? Those will influence how you use the probability output of your logistic regression. You don't have to use the software default of $0.5$ as your decision threshold.
Comment: It's typical to chat in the chat after you continue the discussion in chat. Let's migrate there.
Comment: 1) You're asking about the "likelihood" of an event, but that does not appear to be part of your requirements. Did your boss ask for a "likelihood" of an event? // 2) Consider asking your boss for clarification on what exactly is needed. // 3) Do you have an explicit requirement to give a confidence interval? I can think of a way to report the results without giving a confidence interval, yet I would answer the questions about predicted class membership and confidence about loan default.
Comment: Hi @Dave - Yes, I wanted to compute/predict the likelihood and the confidence of my predictions. For ex: I can say that person A has a 90% likelihood of his loan being approved. But how confident am I about the inference/conclusion that we make from the model results. That's what they wish to know. Likelihood and confidence
Comment: 3) Yes, I do have an explicit requirement to report results with confidence (interval) etc. Can direct me to any resources that involves likelihood determination as well as confidence of the results?
Comment: A confidence interval is different from a colloquial "confidence" in the model.
Comment: Okay, is there any way to find the confidence of the model output?
Comment: That would be "model validation".
Comment: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/133327/discussion-between-the-great-and-dave).
Comment: If I do cross validation, I get average accuracy across k folds. So, does it qualify as confidence in the model?
Answer: My guess is that Bayesian (Logistic) Regression is what you're looking for.
In a regular Logistic Regression you find the (model) parameters $\beta$ that maximise the likelihood of your data (see Maximum Likelihood estimation). In a binary classification you use those parameters to calculate the probability $p$ of a given (test) sample $X_{test}$ of belonging to a positive class.
In Bayesian regression, on the other hand, you're not trying to find single best parameters $\beta$ that fit your data. Instead, you're trying to find the distribution of $\beta$s given your prior belief about the parameters (e.g. you can assume that they're coming from a normal distribution centered at $0$ with some standard deviation $\sigma$*) and using your data to update your prior belief, arriving at a posterior distribution of $\beta$s. You can use the means of those posterior distributions to do predictions - this should give you results similar (or equivalent**) to predictions done using estimates from a normal logistic regression. However, having an entire distribution of $\beta$s (as opposed to a having a single value) gives you an idea about the variability of those estimates. A posterior distribution with high variance indicates that your estimate of $\beta$ has lots of variability, i.e. given a slightly different training dataset, the MLE estimate of $\beta$ can change a lot. You can use the standard deviation of the posterior distribution to calculate the confidence interval for the distribution of $\beta$. Now instead of reporting $\beta = 0.1$ you can say that e.g. 95% confidence interval for $\beta$ is e.g. $[0.05, 0.15]$. I believe that this is what your requirements refer to as confidence.
Remark You can perform predictions using both $\beta$s corresponding to the lower and upper bound of the confidence interval and this will give you two distinct predictions).
Here is a Python example of Bayesian Logistic Regression performed using PyMC3 package.
* What consitutes a good prior is a topic for a separate debate. Gaussian distribution centered at 0 is a good default choice. It also has a regularization effect as it forces the parameters to be close to 0.
** Logistic Regression with uniform prior will give results equivalent to MLE estimates.
Comment: @TheGreat It is on you to define what confidence means. From what you wrote in your comments to me, it seems like you are interested in model validation to show a colloquial "confidence" that the model is a good one, rather than formal confidence intervals.
Comment: thanks for the help. upvoted
Comment: The confidence (interval) in this case would refer to $\beta$s, not the predictions themselves (at least not directly). Regular logistic regression gives you only one (MLE) estimate of $\beta$. Using this $\beta$ you can make a prediction, e.g. say that for $X_{test}$ it says that the probability of class 1 is 60%. In Bayesian Logistic Regression you get a range of $\beta$s (corresponding to the 95% confidence interval). If you use those two extreme $\beta$s to do predictions on the same test sample you'll get a range of probabilities of belonging to the class 1 (e.g. 55% - 65%).
Comment: @TheGreat I don't know, although that would be my guess. Or it could be simply the difference between the $P(X = 1)$ and $P(X = 0)$, i.e. how _confident_ your model predictions are for a given sample (i.e. 51%/49% is not confident, 99%/1% is confident).
Comment: Note: As others have said you are incorrect in labeling this as a classification problem.  The Bayesian approach is most suited for communicating uncertainties.  Short of that, computing 0.95 confidence intervals for predicted probabilities for selected covariate settings will be helpful.  This uses standard logistic regression methods, with a normal approximation for the predicted log odds of the outcome.
Comment: the machine always predicts "yes" with a probability between 0 and 1. So, is that called as our confidence score?
Comment: I found that from here - https://medium.com/voice-tech-global/machine-learning-confidence-scores-all-you-need-to-know-as-a-conversation-designer-8babd39caae7
Comment: If Likelihood is same as confidence score of our predictions, then that is what I require. But is it same or different?
Comment: If you have time, can help me with this related question - https://stats.stackexchange.com/questions/560895/how-to-handle-hierarchical-categorical-variables-retain-or-drop
Comment: Yes, understand your point on CI. But does confidence Interval and confidence mean the same?


Attach text to center of image only using grid template

Question: I have a background image which has on top another div containing some elements.
I success to add the white-container on top of the background div using grid-template-area.
I try to do the same thing only using grid and not position. But text hello world doesn't go on top of love icon using grid area love-svg.
I think perhaps because img tag has no closed tag, but can't find a way to solve this problem.
HTML

Hello world

SCSS
.container {
    display: grid;
    justify-items: center;
    grid-template-areas: "content-wrapper";

    &>* {
        grid-area: content-wrapper;
    }

    img {
        width: 100%;
    }

    .love-section
    {
        display: grid;
        background-color: white;
        top: 40px;
        position: relative;
    }
    .love-ico {
        width: 100px;
        grid-template-areas: "love-svg";
    }

    .hello {
        color: red;
        grid-area: love-svg;
    }
}

In blue this is what I want to achieve. Please note that I want the text to be on top of the love logo but allow it to be bigger than it.
Answer: if you want to use grid to do that, you'll need to play with the align-self, and justify-self for the grid-item.
Now there are several ways to do that with grid, the way you are using it is not the simplest.
I'm proposing just one here:

html,
body {
  margin: 0;
  padding: 0;
}

#grid {
  display: grid;
  grid-template-rows: repeat(7, 1fr);
  grid-template-columns: repeat(7, 1fr);
  gap: 0;
  width: 100vw;
  min-height: 100vh;
}

#div1 {
  grid-area: 1 / 1 / 8 / 8;
}

#div1 img,
#div2 img {
  width: 100%;
  height: 100%
}

#div2 {
  grid-area: 3 / 3 / 6 / 6;
}

#div3 {
  grid-area: 4 / 4 / 5 / 5;
  align-self: center;
  justify-self: center;
}

Hello world

Now I don't know if it's mandatory for you to use grid, otherwise the old technic of relative, absolute works too:

html,
body {
  margin: 0;
  padding: 0;
}

#container {
  position: relative;
  background-image: url("https://picsum.photos/300/200");
  width: 100vw;
  height: 100vh;
}

#container img {
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

#container h2 {
  position: absolute;
  top: 50%;
  left: 50%;
  transform: translate(-50%, -50%);
}

Hello world