Why PCA of data by means of SVD of the data?

Question: This question is about an efficient way to compute principal components.

Many texts on linear PCA advocate using singular-value decomposition of the casewise data. That is, if we have data $\bf X$ and want to replace the variables (its columns) by principal components, we do SVD: $\bf X=USV'$, singular values (sq. roots of the eigenvalues) occupying the main diagonal of $\bf S$, right eigenvectors $\bf V$ are the orthogonal rotation matrix of axes-variables into axes-components, left eigenvectors $\bf U$ are like $\bf V$, only for cases. We can then compute component values as $ \bf C=XV=US$.
Another way to do PCA of variables is via decomposition of $\bf R=X'X$ square matrix (i.e. $\bf R$ can be correlations or covariances etc., between the variables ). The decomposition may be eigen-decomposition or singular-value decomposition: with square symmetric positive semidefinite matrix, they will give the same result $\bf R=VLV'$ with eigenvalues as the diagonal of $\bf L$, and $\bf V$ as described earlier. Component values will be $\bf C=XV$.

Now, my question: if data $\bf X$ is a big matrix, and number of cases is (which is often a case) much greater than the number of variables, then way (1) is expected to be much slower than way (2), because way (1) applies a quite expensive algorithm (such as SVD) to a big matrix; it computes and stores huge matrix $\bf U$ which we really doesn't need in our case (the PCA of variables). If so, then why so many texbooks seem to advocate or just mention only way (1)? Maybe it is efficient and I'm missing something?
Comment: Generally we are interested in only the few principal components which explain most of the variance. It is possible to do a reduced SVD; for example if $X$ is of dimension $N \times p$ where $p << N$ then `R`'s `svd` function will compute only the first $p$ left and right singular vectors by default.
Comment: @M.Berk: $p$ however, is the same in both approaches: they yield equivalent results (equal up to sign changes). Also, e.g. R calculates $\mathbf C$ only if asked for.
Comment: Do you have an references for way (1)? I'm only aware of PCA being implemented via SVD on the covariance matrix (i.e. way 2), as this avoids some numerical problems and abviously scales with the dimensionality, not the data set size. Way (1) I would call SVD, not PCA at all. I've only seen it in a pure SVD context, where one would not perform a complete decomposition actually.
Comment: @Anony-Mousse, Just one to mention, `Joliffe, Principal component analysis, 2nd ed.` Actually, Joliffe describes both ways, but in the core chapter on PCA he says about just way 1, as far as I can remember.
Comment: @Anony-Mousse, Way 1 for me is important from theoretical point because it clearly shows how PCA is directly related to simple _correspondence analysis_.
Comment: I agree that it is interesting from a theoretical point of view. It may well be that (1) has the nicer formal properties, while (2) has the better computational efficiency.
Answer: SVD is slower but is often considered to be the preferred method because of its higher numerical accuracy.
As you state in the question, principal component analysis (PCA) can be carried out either by SVD of the centered data matrix $\mathbf X$ (see this Q&A thread for more details) or by the eigen-decomposition of the covariance matrix $\frac{1}{n-1}\mathbf X^\top \mathbf X$ (or, alternatively, $\mathbf{XX}^\top$ if $n\ll p$, see here for more details).
Here is what is written in the MATLAB's pca() function help:

Principal component algorithm that pca uses to perform the principal component analysis [...]:
'svd' -- Default. Singular value decomposition (SVD) of X.
'eig' -- Eigenvalue decomposition (EIG) of the covariance matrix. The EIG algorithm is faster than SVD when the number of observations, $n$, exceeds the number of variables, $p$, but is less accurate because the condition number of the covariance is the square of the condition number of X.

The last sentence highlights the crucial speed-accuracy trade-off that is at play here.
You are right to observe that eigendecomposition of covariance matrix is usually faster than SVD of the data matrix. Here is a short benchmark in Matlab with a random $1000\times 100$ data matrix:
X = randn([1000 100]);

tic; svd(X); toc         %// Elapsed time is 0.004075 seconds.
tic; svd(X'); toc        %// Elapsed time is 0.011194 seconds.
tic; eig(X'*X); toc      %// Elapsed time is 0.001620 seconds.
tic; eig(X*X'); toc;     %// Elapsed time is 0.126723 seconds.

The fastest way in this case is via the covariance matrix (third row). Of course if $n \ll p$ (instead of vice versa) then it will be the slowest way, but in that case using the Gram matrix $\mathbf{XX}^\top$ (fourth row) will be the fastest way instead. SVD of the data matrix itself will be slower either way.
However, it will be more accurate because multiplying $\mathbf X$ with itself can lead to a numerical accuracy loss. Here is an example, adapted  from @J.M.'s answer to Why SVD on $X$ is preferred to eigendecomposition of $XX^⊤$ in PCA on Math.SE.
Consider a data matrix $$\mathbf X = \begin{pmatrix}1&1&1\\\epsilon & 0 & 0\\ 0 & \epsilon & 0 \\ 0 & 0 & \epsilon\end{pmatrix},$$ sometimes called Läuchli matrix (and let us omit centering for this example). Its squared singular values are $3+\epsilon^2$, $\epsilon^2$, and $\epsilon^2$. Taking $\epsilon = 10^{-5}$, we can use SVD and EIG to compute these values:
eps = 1e-5;
X = [1 1 1; eye(3)*eps];
display(['Squared sing. values of X: ' num2str(sort(svd(X),'descend').^2')])
display(['Eigenvalues of X''*X:       ' num2str(sort(eig(X'*X),'descend')')])

obtaining identical results:
Squared sing. values of X: 3       1e-10       1e-10
Eigenvalues of X'*X:       3       1e-10       1e-10

But taking now $\epsilon = 10^{-10}$ we can observe how SVD still performs well but EIG breaks down:
Squared sing. values of X: 3       1e-20       1e-20
Eigenvalues of X'*X:       3           0 -3.3307e-16

What happens here, is that the very computation of covariance matrix squares the condition number of $\mathbf X$, so especially in case when $\mathbf X$ has some nearly collinear columns (i.e. some very small singular values), first computing covariance matrix and then computing its eigendecomposition will result in loss of precision compared to direct SVD.
I should add that one is often happy to ignore this potential [tiny] loss of precision and rather use the faster method.
Comment: Also for the SVD you can write (or just use) the algorithm for $\mathbf X^T$ which has the opposite runtime behaviour (because the decomposition has symmetry wrt. transposing $\mathbf X$ ) and otherwise the same numeric behaviour. Numeric stability is an important point (+1) - though I guess it depends a lot the data whether that matters. E.g. I typically far too few cases and noisy measurements - so the stability of my models is typically not limited by the numeric stability of underlying SVDs but by the patients/cell culture batches or instrumental noise.
Comment: @amoeba, thank you very much for coming back and giving an example! I tried both epsilon examples in SPSS and got results like your's except for the very last line: instead of `3 0 -3.3307e-16` eigen in spss returned me `3 0 0`. It looks as if the function has some in-built and fixed tolerance value beyond which it zero-offs. In this example, the function appeared as if to hack the knot of numerical instability by zeroing both tiny eigenvalues, the "0" and the "-16".
Comment: Thanks for the answer and for thorough consideration of pros and cons.
Comment: amoeba, may that be that you find time to show a concrete example where numerical stability suffers by `eig()` approach? (Readers will benefit: there is a point of trade-off between speed and stability. How one may decide in a concrete practical situation?)
Comment: @ttnphns I rewrote the whole answer, providing a concrete example. Take a look.
Comment: I personally tend to think that the "numerical instability/inaccurateness" danger is only of minor importance. Eigenvalues below, say, e-13 can be safely considered as just zero - for the majority if not all practical statistical analysis tasks.
Comment: @ttnphns: Yes, rounding to zero (that SPSS might be doing) makes sense. Still, getting 0 for the last two eigenvalues is less accurate than getting `1e-20` which is the correct result and can be obtained via svd (as far as I understood your comment in SPSS as well). In general, I fully agree that in practice all of that is of minor importance. Whenever the difference in speed becomes substantial, I prefer to use the faster method.
Comment: Here's the thing: you wouldn't really know whether you'd be safe with the symmetric eigendecomposition or SVD unless you actually perform the computation (a so-called "wicked problem"). So, why not stick with the safe instead of gambling with the potentially unruly? (Regarding speed versus stability: I'd personally prioritize the latter over the former, unless someone is going to die unless those matrices just have to be decomposed RIGHT NOW AND FAST!!!1! ;))
Comment: @J.M., it's a matter of taste. As for me, I strongly vote for speed, not accuracy. A practicing statistician mostly works with medium to large datasets; plus to it, he may need doing a matrix decomposition many times in loops. Time is crucial. To me, it is "silly" to do linear OLS regression via svd (very accurate, but slow), packages would typically use system of linear equations solver, or the sweep-operation approach.
Comment: @ttnphs, sure, for most "garden-variety" (whatever that means) LS problems, "sweep" or the normal equations will do fine. But if you're angling for diagnostics (unless you also can't be arsed to check your results), or you have a sufficiently complicated basis set that you can't easily tell if some of those are in fact not independent, you'll use SVD. And honestly, people could stand to be in less of a hurry... :P
Answer: Here are my 2ct on the topic

The chemometrics lecture where I first learned PCA used solution (2), but it was not numerically oriented, and my numerics lecture was only an introduction and didn't discuss SVD as far as I recall.
If I understand Holmes: Fast SVD for Large-Scale Matrices correctly, your idea has been used to get a computationally fast SVD of long matrices.
That would mean that a good SVD implementation may internally follow (2) if it encounters suitable matrices (I don't know whether there are still better possibilities). This would mean that for a high-level implementation it is better to use the SVD (1) and leave it to the BLAS to take care of which algorithm to use internally.
Quick practical check: OpenBLAS's svd doesn't seem to make this distinction, on a matrix of 5e4 x 100, svd (X, nu = 0) takes on median 3.5 s, while svd (crossprod (X), nu = 0) takes 54 ms (called from R with microbenchmark).
The squaring of the eigenvalues of course is fast, and up to that the results of both calls are equvalent.
timing  <- microbenchmark (svd (X, nu = 0), svd (crossprod (X), nu = 0), times = 10)
timing
# Unit: milliseconds
#                      expr        min         lq    median         uq        max neval
#            svd(X, nu = 0) 3383.77710 3422.68455 3507.2597 3542.91083 3724.24130    10
# svd(crossprod(X), nu = 0)   48.49297   50.16464   53.6881   56.28776   59.21218    10

update: Have a look at Wu, W.; Massart, D. & de Jong, S.: The kernel PCA algorithms for wide data. Part I: Theory and algorithms , Chemometrics and Intelligent Laboratory Systems , 36, 165 - 172 (1997). DOI: http://dx.doi.org/10.1016/S0169-7439(97)00010-5
This paper discusses numerical and computational properties of 4 different algorithms for PCA: SVD, eigen decomposition (EVD), NIPALS and POWER.
They are related as follows:
computes on      extract all PCs at once       sequential extraction
X                SVD                           NIPALS
X'X              EVD                           POWER

The context of the paper are wide $\mathbf X^{(30 \times 500)}$, and they work on $\mathbf{XX'}$ (kernel PCA) - this is just the opposite situation as the one you ask about. So to answer your question about long matrix behaviour, you need to exchange the meaning of "kernel" and "classical".

Not surprisingly, EVD and SVD change places depending on whether the classical or kernel algorithms are used. In the context of this question this means that one or the other may be better depending on the shape of the matrix.
But from their discussion of "classical" SVD and EVD it is clear that the decomposition of $\mathbf{X'X}$ is a very usual way to calculate the PCA. However, they do not specify which SVD algorithm is used other than that they use Matlab's svd () function.

    > sessionInfo ()
    R version 3.0.2 (2013-09-25)
    Platform: x86_64-pc-linux-gnu (64-bit)

    locale:
     [1] LC_CTYPE=de_DE.UTF-8       LC_NUMERIC=C               LC_TIME=de_DE.UTF-8        LC_COLLATE=de_DE.UTF-8     LC_MONETARY=de_DE.UTF-8
     [6] LC_MESSAGES=de_DE.UTF-8    LC_PAPER=de_DE.UTF-8       LC_NAME=C                  LC_ADDRESS=C               LC_TELEPHONE=C
    [11] LC_MEASUREMENT=de_DE.UTF-8 LC_IDENTIFICATION=C

    attached base packages:
    [1] stats     graphics  grDevices utils     datasets  methods   base

    other attached packages:
    [1] microbenchmark_1.3-0

loaded via a namespace (and not attached):
[1] tools_3.0.2

$ dpkg --list libopenblas*
[...]
ii  libopenblas-base              0.1alpha2.2-3                 Optimized BLAS (linear algebra) library based on GotoBLAS2
ii  libopenblas-dev               0.1alpha2.2-3                 Optimized BLAS (linear algebra) library based on GotoBLAS2

Comment: @ttnphns: yes. But as svd is provided by the BLAS that could be different with a different BLAS. I'd have expected that a good optimized BLAS does something like this. It doesn't seem to be the case with OpenBLAS, however. I'm too lazy to check other BLASs, but maybe a few people could check their other BLASs so we find out which ones are optimized for this case and which aren't. (I emailed the OpenBLAS developer and sent him a link to this question, so maybe he can add some info, e.g. reasons for not switching the algorithm to the `svd (X'X)` for long matrices.)
Comment: So, your testing (3.5 sec vs 54 msec) supports my line that "way 1" is considerably slower. Right?
Comment: Some points need clarification (to me). Can the "kernel methods" be summarized as "work on $X'$ instead of $X$ when $n<p$"? if so, it is quite trivial. I don't know POWER but i know NIPALS, which computes the eigenvectors of $X'X$ by iterating $u_{n+1} = X'Xu_n / ||X'Xu_n||$ (his converges to the 1st eigenvector $v_1$, then you have to update $X$ to compute the second, etc). There are two ways to perform NIPALS, (1) you can precompute $X'X$, or (2) you can perform the product as $X' \times (X u_n)$, which way is used here? I guess (1) is used, which can be unfair.
Comment: @Elvis: a) There's more to kernel methods than just computing on  $\mathbf X \mathbf X^T$, see e.g. http://stats.stackexchange.com/questions/2499/explain-what-a-kernel-is-in-plain-english. The equivalence is trivial for PCA (it doesn't matter whether you start by getting right or left singular vectors) but not for other methods. b) "which way to do NIPALS" relies on the same overall principle. Which algorithm is used for the SVD depends on your BLAS, and actually my guess is that NIPALS is not involved here. Note that my timing includes the computation of the cross product.
Comment: I was talking about your update, where Nipals is involved. I confirm Nipals is not involved in Lapack's SVD. About your benchmark experiment, something like `microbenchmark(X <- matrix(rnorm(5e6), ncol=100), Y <- t(X), svd(X), svd(Y), control=list(order="inorder"), times = 5)` can be interesting as well.
Comment: ...and its difference with `X <- matrix(rnorm(5e6), ncol=100); Y <- t(X); microbenchmark(svd(X), svd(Y), control=list(order="inorder"), times = 5)` which was my motivation for redefining `X` each time.


Registering AzureFunctions App with AAD B2C

Question: I'm very new to Azure development and I only know a little JavaScript, no C#. I'm trying develop an iPhone App using React Native and Expo.
I have 2 Azure tenancies:
APP 1 - private API: AzureFunctions App registration in Private Tenancy
APP 2 - public client: Azure Web App registration in Azure B2C Tenancy (this is the logical app that represents the iPhone app)
MSAL doesn't support AAD B2C Third Party Identity Providers (Google, Facebook, etc.) on Javascript Apps running on iPhone, however I seem to have successfully gotten the token back into the Expo iPhone app.
When I pass my Access Token that was generated by Azure AD B2C to my anonymous HTTP Trigger AzureFunctions App (that has Service Authentication turned on) I am denied access.
I have tried following this guide: https://learn.microsoft.com/en-us/azure/app-service/configure-authentication-provider-aad
but I cannot verify the domain of the AzureFunctions app, even though I have tried to use proxies to route the AAD B2C validator to the .json file.

Possible errors:

I haven't configured AZFunctions proxies correctly, when I go to the URL I am submitting to AAD B2C "Publisher Domain Verifier" I am not served the .JSON file.

CORS: I am trying to access the API from exp:// instead of an http:// server location

Any advice would be appreciated!
Comment: I couldn't figure out how to get it to serve files, but I did a response override which worked
Answer: I couldn't figure out how to get it to point to the JSON file or indeed serve that file, but the response override did the trick!

However I still am unable to specify the other tenancy (xxxxxxxx-azurewebsites.net)

In the end I abandoned this "automated authentication" and wrote my own token validator.


Find irreducible components of the following projective algebraic set

Question: Consider the following projective algebraic set
$$\mathcal{V}(x_0^2+x_1x_2,x_0^2-x_1x_2+2x_3^2)\subset\mathbb{P}^3_{\mathbb C}.$$
I need to find the irreducible components. My approach has been the following.
If we set $x_0=0$ then we obtain $x_1x_2=0$ and $x_3=0$, so we get two points at infinite $P_1=[0:1:0:0]$ and $P_2=[0:0:1:0]$.
Otherwise, setting $x_0=1$ and adding both equations we get that $1+x_3^2=0$; and $1+x_1x_2=0$. But the first one can be decomposed as $(1+ix_3)(1-ix_3)$. So we get
$$ \mathcal{V}(x_0^2+x_1x_2,x_0^2-x_1x_2+2x_3)=\mathcal{V}(1+x_1x_2,1+ix_3)\cup\mathcal{V}(1+x_1x_2,1-ix_3)\cup P_1\cup P_2.$$
Is this correct? I feel like something is off.
Thanks in advance.
Answer: Your calculations in the affine patch look correct to me. You can simplify it a bit more by doing all the computations in projective space rather than passing to an affine patch (which is what you're doing when you set $x_0 = 1$). Indeed, adding the two defining equations yields $x_0^2 + x_3^2 = 0$, which can be factored as $(x_0 + ix_3)(x_0 - ix_3) = 0$, so we get
$$\mathcal{V}(x_0^2+x_1x_2,x_0^2-x_1x_2+2x_3^2)=\mathcal{V}(x_0^2+x_1x_2,x_0+ix_3)\cup\mathcal{V}(x_0^2+x_1x_2,x_0-ix_3),$$
with no need to treat the two points where $x_0 = 0$ separately. You also need to verify that these two components are not contained in each other and are themselves irreducible, but that should be pretty straightforward.
Note that the points you denote $P_1$ and $P_2$ are not irreducible components; in fact, they're the intersection of the two irreducible components. So for your work to give the decomposition into irreducible components, you need to re-homogenize the polynomials in $x_0$ so they're closed subvarieties in projective space, and then recognize that $P_1$ and $P_2$ are contained in these and hence aren't irreducible components themselves.
Comment: Thank you so much for your answer! I have a question - the number of irreducible components of an algebraic set is not unique? Or the way I expressed the algebraic set is not valid as a decomposition in irreducible components?
Comment: The decomposition into irreducible components is unique. The points $P_1$ and $P_2$ aren't irreducible components, they're the points in the intersection of the two irreducible components. I've edited my answer to clarify.
Comment: That was very clarifying, thank you kindly!


Issue understanding the difference between reduced row echelon form on a coefficient matrix and on an augmented matrix

Question: I have a problem understanding something about matrices and the difference between a coefficient matrix and an augmented matrix. One theorem in a book I'm reading states:

Suppose thate $A\mathbf{x = b}$ is a sytem of $m$ linear equations in
  $n$ variables. The system is consistent iff the rank of the
  coefficient matrix $A$ is equal to the rank of the augmented matrix
  $\left[\begin{array}{c|c}A&b\end{array}\right]$.

With the following definition for rank of a matrix:

The rank of a matrix $M$ is the number of leading 1's in the
  reduced row echelon form that is row equivalent to $M$.

What is confusing me about this statement is the difference between the rank of the coefficient matrix $A$ and the augmented matrix $\left[\begin{array}{c|c}A&b\end{array}\right]$. The definition of rank requires knowing the reduced row echelon form of a matrix, but how can one find the reduced row echelon form of the coefficient matrix $A$? Wouldn't you always need the augmented matrix? And once you've transformed $A$ into reduced row echelon form, why would it ever be different from the reduced row echelon form of $\left[\begin{array}{c|c}A&b\end{array}\right]$?
Answer: In short, row reduced echelon form(RREF) of a matrix $A$ is such that
i) Every leading entry is 1
ii) Any nonzero rows are above zero rows
iii) any leading entry is strictly to the right of any leading entries above that row
iv) any other entry in a column containing a leading entry is 0 except for the leading entry.
So it does not have to be put in augmented matrix $[A|b]$ to get a RRE form. You are comparing RRE form of matrix $A$ and $[A|b]$.
To see why the statement is true, suppose that you put the matrix $[A|b]$ into RRE form, so you have a matrix E. If E contains a leading entry in its last column, in terms of system of equations, what does it say? And what is the condition for E to not have any leading entry in last column?
Note: If RRE form of $[A|b]$ does contain a leading entry, then it is different from that of $A$. Also, note that RRE form of $[A|b]$ is m by n+1 whereas that of $A$ is m by n.

Solve:
$x+y=1$
$x+y=2$
Then we have
$\
A =
\left( {\begin{array}{cc}
1 & 1 \\
1 & 1
\end{array} } \right)
$
$\
b =
\left( {\begin{array}{cc}
1  \\
2
\end{array} } \right)
$
and $Ax=b$
If we turn A into RREF, we get
$\
E =
\left( {\begin{array}{cc}
1 & 1 \\
0 & 0
\end{array} } \right)
$
So A has rank 1
and if we put $[A|b]$ into RRE form, we get
$\
E' =
\left( {\begin{array}{cc}
1 & 1 & 0 \\
0 & 0 & 1
\end{array} } \right)
$
So augmented matrix has rank 2. Observe what last row says in terms of equations.
Comment: @user 3002473: It is not too much work, but it means you're going against the family...
Comment: Example of what exactly? If you are asking an example in which rank of $A$ and $[A|b]$ is different, try solving equation x+y=1, x+y=2 using RRE form of matrix. It should not be that difficult to see what is happening
Comment: Hmmm, I think I'm starting to understand. Could you provide an example (assuming it wouldn't be too much work to write all the mathjax code for matrices)?
Comment: @user160738 Yeah, an example in which the rank is different between $A$ and $[A|b]$. I'm more of a visual/hands-on learner, so I think I might have a better understanding if I could see the theorems working in a real application.
Comment: Oh! That makes perfect sense now! Thank you so much!


Many ISP's is block port 25, how do I choose an alternative port?

Question: I am building an application that will be acting as a combined MUA/MTA on different networks. However, many of the networks are with ISP's that block port 25 for SMTP. Therefore I would also like to open up a secondary port so that some of the installs can communicate on that if port 25 is closed.
How do I choose a second port? I know some people use port 26 or port 2525. What is the correct way to choose a port that won't interfere with existing software?
Answer: If you're just sending email on behalf of an end-user, who originates the email within your application, use authenticated SMTP to port 587.
You can send mail to some servers via the SMTPS port 465, if the remote MTA is SSL/TLS-enabled and listening on that port, but don't count on it.
Otherwise, advise the user that they need a business-class ISP.
Comment: @Xeoncross Attempting to circumvent ISP policy is ***EXPLICITLY*** off-topic per the Server Fault [FAQ]. If you need MTA-to-MTA SMTP communication, you *need* port 25. End of story, end of RFC. Anything else is (a) not "professional" (it's a hack), and (b) a violation of your ISP's terms of use, which is not something we're going to help you with...
Comment: @Xeoncross You can't magically make every other MTA on the Internet speak on some other port than 25.
Comment: That's basically what I was typing up as an answer, so here's the add-on link: [Wikipedia on SMTP ports](http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#Ports)
Comment: SMTP port 587 is only for MUA *submission* to a MTA - it's not for MTA's to talk with each other. There is no central MTA or MX server in this, each system install needs to run by itself.
Comment: "More professional" is to not use a residential ISP for business-class service. :)
Comment: @MichaelHampton, I'm not, however I need *these* installs to default to an alternative port so that even if they can't talk with other MTA, they can still talk with each other.
Comment: @Xeoncross Hm, well, pick a high numbered port at random like everyone else, then?
Comment: @Xeoncross If you have control over the servers, pick a different port and configure it. There's no forms or entry fees.
Comment: Well, I was hoping for something more professional since this might end up on thousands of system. ;)
Comment: Yep, in an ideal world that would be great. But, expecting them to pay the increased price just to open 25 isn't a reasonable cost/benefit ratio.
Comment: It's perfectly reasonable if they want to run their own email service. Bypassing such a residential ISP restriction is almost certainly a good way to get kicked off. Now if there's an ISP blocking port 25 on a business class service, then the ISP is at fault there...
Comment: I am not sure why you think it is somehow more unprofessional to use a nonstandard port for SMTP than it is to use a residential ISP for any sort of production service like mail.
Answer: Your question is not 100% clear to me. Are you looking to have incoming port 25 connections, or are you connecting 25 outbound?   I will split the answer into two to cover both bases.
Outbound 25
If by "installs communicate on port 25", you mean that the software installation tries to contact other servers using port 25, that is a bad idea. Do not ship software that calls out on port 25.
The only outbound port your software can rely on for "phoning home" or contacting other servers  is port 80. And not only the port, but the protocol has to be HTTP. Some firewalls allow port 80, but they also inspect the protocol and block it if it is not HTTP.   You can also rely on 443, HTTPS.  Still, I would provide an alternative installation/registration workflow which works if there is no port 80 connection.
ISP's block outgoing 25 because it's used by spammers to directly contact mail servers from subscriber lines and deliver spam.
That being said, of course you can ship software that sends mail, and that uses port 25. However, that mail should go to an SMTP server which is configured by the user. (Quite often, a relay provided by their ISP, which may require authentication.)
Inbound 25
An application that is going to act as a mail server for a domain will generally not be deployable by users who cannot have incoming port 25 open.
Those people will need a workaround, ranging from changing their subscription, changing to another ISP, or using some machine in some other network as a port-forwarding proxy which forwards port 25 connections to the real server, using some port that is open.
The choice of alternate port is best left as a configuration to be determined by the users.
There is no universal port number that is guaranteed to be open, regardless of ISP.
But the bigger problem is that all the mail software out there will expect to connect to port 25.   The mail exchange (MX) DNS record does not include a port number; it resolves to a host.
The MX host is expected to listen on port 25.
Comment: Too bad SMTP does not recognize [SRV records](http://en.wikipedia.org/wiki/SRV_record). That would allow the use of any inbound port.
Answer: Today I released a feature on ForwardEmail.net to allow custom port forwarding for emails.  This solution may work for you.  We've documented it along with a guide/how-to at our page on Port 25 blocked by ISP workaround.
If you're unfamiliar with ForwardEmail.net:
ForwardEmail is a free, secure, and open-source email forwarding service, which allows the use of custom domains and disposable addresses. It also has a strict zero tolerance policy for storing logs of any kind (see the Zero Tolerance Privacy Policy for more information).
Answer: You can pick any port you want, as long as you're building a closed private e-mail system and the MTA's all know to use that port to communicate with each other, as long as you stay consistent. If you need this system to communicate to the outside world though, at least one of your MTA's will have to use destination port 25 to relay, or use a trusted relay to get out to the rest of the world.
As long as you pick something in the high port range, and it's something you can remember, this is fine.
Answer: For incoming 25, you can use one the many smtp rerouting services, provided because many ISPs block port 25. See this post: http://thenubbyadmin.com/2010/06/14/list-of-inbound-smtp-redirection-services-some-for-free/.
My plug for DNSMadeEasy: Comcast blocked port 25 on me unannounced, killing my mail server. After reviewing the rerouting options, I chose DNS Made Easy. (EasyDNS in Canada, DNSExit raised prices, DYN ridiculously expensive). DNS Made Easy had me back up in under 10 minutes, and that's with manual MX Record entries. They also responded very promptly to a silly question I had because I was rushed and didn't hit the little blue "i" info button. They have a essentially infinite 1 GB buffer and my mail server is a lot more robust to downtime now that I'm using this rerouting service.
For outgoing port 25, you'll need to specify a smtp gateway anyway, especially if the IP is listed in spamhaus's PBL list -- even if your mail makes it out, it will almost certainly be blacklisted by most remote mail servers.
Answer: I agree with Michael Hampton but since your comments talk about the MTA as well:
-Make your MUA program allow for configuration by the end client/user to put in whatever their correct outbound MTA is with their ISP.
Then the MUA will send email out their residential ISP since it will allow it coming from its own network.
EDIT: course you said your app is a combined MUA/MTA.  That won't work for MTA functionality unless you then had the "thousands" of deployed apps then send all the email to another centralized MTA (via whatever port) and then that MTA relays email outbound to wherever over port 25.


How to fix this TemplateSyntaxError at error in django

Question: I get this error
TemplateSyntaxError at /

add requires 2 arguments, 1 provided

Screenshot:
ERROR SCREENSHOT
I want to add pagination there but i get this error again and again
Here is my html file

    {% if buy_list.has_previous %}

    {% endif %}
    {% for num in buy_list.paginator.page_range %}
        {% if buy_list.number == num %}
            {{ num }}
    {% elif num > buy_list.number|add: '-3' and num < buy_list.number|add: '3' %}
        {{num}}
        {% endif %}
    {% endfor %}
    {% if buy_list.has_next %}

    {% endif %}

Thanks
Answer: Remove spaces after the add: built-in filter in your template.
buy_list.number|add:"-3"

buy_list.number|add:"3"


Sinatra Github OAuth Problems

Question: I am using the sinatra_auth_github gem to try and put GitHub OAuth into my Sinatra app. I was testing using the sinatra-auth-github-test example app for a while and it worked. I went back to try and get some of that code to work in my app and though most of it worked, the redirect from GitHub doesn't.
Can anyone recommend anything to help?
Here's the code:
class SimpleApp < Sinatra::Base
  # [...]

  set :github_options, {
    :scope     => 'user',
    :secret    => ENV['GITHUB_CLIENT_SECRET'],
    :client_id => ENV['GITHUB_CLIENT_ID'],
  }
  register Sinatra::Auth::Github

  get '/' do
    erb :index
  end

  get '/login' do
    authenticate!
    "Hello there, #{github_user.login}!"
  end
end

Answer: It's possible that the issue you're experiencing is related to shotgun. This gem reloads your entire application on every request, so it's possible there's an issue with maintaining the OAuth-login session state across the login-related requests.
Running your application with rackup instead (as in the example app), I don't experience any problems:
bundle exec rackup -p 3000

Comment: Hi, I tried doing that and I'm getting the same result. Is it because the way I'm calling `/login` is from an `href` in `index.erb`? Is it because the Redirect URL in the Github app is set to `/`? @wjordan
Comment: @zander in order to provide a [Minimal, Complete, and Verifiable example](http://stackoverflow.com/help/mcve) that I or others can reproduce your issue, please add more complete contents of the Sinatra source code you're running, and the exact commands you're using to run your server in development. I'm curious how the `GITHUB_` environment variables are provided and if that's consistent between your test app and this one.
Comment: As seen in the code above, I am using a client ID and client secret from GitHub and what I listed above is pretty much all I have, I just left out the index.erb because that's just an href linking to `/login`. In terms of the server, I'm running `bundle exec shotgun -p 3000` and the redirect link is just `http://0.0.0.0:3000`, if I try to set it to `http://0.0.0.0:3000/login`, an error shows up that the redirect URI is incorrect. It is going to Github normally, but the response link is not sending the user back to the Sinatra app @wjordan
Comment: The code is in a class in an `app.rb` file and the config.ru links to it but in an example I [saw](https://github.com/atmos/sinatra-auth-github-test/blob/master/config.ru) it was working in a config.ru @wjordan
Comment: have you tried calling `enable :sessions` within your Sinatra app?
Comment: yeah its been in there the entire time
Comment: Until you provide a complete example I can run locally and reproduce your issue (e.g., a public GitHub repo I can clone), I can't help any further, sorry.
Comment: @zander with the actual code I managed to narrow down your issue to the use of `shotgun`. it works when using `rackup`. hope that helps!
Comment: Great, glad to hear it worked! If this answer helped solve your question, the last step is to [mark it as Accepted](http://meta.stackexchange.com/a/5235/327137). Welcome to Stack Overflow!

Sinatra Github OAuth Problems


How can we show that for $\lambda <0$ we get the trivial solution $X(x)=0$?

Question: Find the solution of the problem $$u_t(x, t)-u_{xx}(x, t)=0, 0<x<1, t>0 \tag {*} \\ u(0, t)=0, t>0 \\ u_x(1,t)+u_t(1,t)=0, t>0$$
I have done the following:
We are looking for solutions of the form $$u(x, t)=X(x) \cdot T(t)$$
$$u(0, t)=X(0) \cdot T(t)=0 \Rightarrow X(0)=0 \\ X'(1) \cdot T(t)+X(1) \cdot T'(t)=0 \Rightarrow \frac{X'(1)}{X(1)}=-\frac{T'(t)}{T(t)}$$
$$(*) \Rightarrow X(x) \cdot T'(t)-X''(x) \cdot T(t)=0 \\ \Rightarrow \frac{X(x) \cdot T'(t)}{X(x) \cdot T(t)}-\frac{X''(x) \cdot T(t)}{X(x) \cdot T(t)}=0 \\ \Rightarrow \frac{T'(t)}{T(t)}=\frac{X''(x)}{X(x)}=-\lambda$$
So, we get the following two problems:
$$\left.\begin{matrix}
X''(x)+\lambda X(x)=0, 0<x<1\\
X(0)=0 \\
\frac{X'(1)}{X(1)}=\lambda \Rightarrow X'(1)-\lambda X(1)=0
\end{matrix}\right\}(1)
$$
$$\left.\begin{matrix}
T'(t)+\lambda T(t)=0, t>0
\end{matrix}\right\}(2)$$
For the problem $(1)$ we do the following:
The characteristic polynomial is $d^2+\lambda=0$.

$\lambda<0$:
General solution: $X(x)=c_1 e^{\sqrt{-\lambda }x}+c_2e^{-\sqrt{-\lambda}x}$

$$X(0)=0 \Rightarrow c_1+c_2=0 \Rightarrow c_1=-c_2$$
$$X(1)=c_1e^{\sqrt{-\lambda}}+c_2e^{-\sqrt{-\lambda}}=c_1(e^{\sqrt{-\lambda}}-e^{-\sqrt{-\lambda}})$$
$$X'(x)=\sqrt{-\lambda}c_1e^{\sqrt{-\lambda }x}-\sqrt{-\lambda}c_2 e^{-\sqrt{-\lambda}x} \\ X'(1)=\sqrt{-\lambda}c_1e^{\sqrt{-\lambda }}-\sqrt{-\lambda}c_2 e^{-\sqrt{-\lambda}}=\sqrt{-\lambda}c_1(e^{-\lambda}+e^{-\sqrt{-\lambda}}$$
$$X'(1)-\lambda X(1)=0 \Rightarrow \sqrt{-\lambda}c_1(e^{\sqrt{-\lambda}}+e^{-\sqrt{-\lambda}})-\lambda c_1(e^{\sqrt{-\lambda}}-e^{-\sqrt{-\lambda}})=0 \Rightarrow c_1 [ e^{\sqrt{-\lambda}}(\sqrt{-\lambda}-\lambda)+e^{-\sqrt{-\lambda}}(\sqrt{-\lambda}+\lambda)]=0$$
How could we continue to show that for $\lambda <0$ we get the trivial solution $X(x)=0$ ??
$$$$
EDIT:

$\lambda <0$ :
$X(x)=c_1 \sinh (\sqrt{-\lambda} x)+c_2 \cosh (\sqrt{-\lambda}x)$
Using the initial values we get that $X(x)=0$, trivial solution.
$\lambda=0$ :
$X(x)=c_1 x+c_2$
Using the initial values we get that $X(x)=0$, trivial solution.
$\lambda >0$ :
$X(x)=c_1 cos (\sqrt{\lambda}x)+c_2 \sin (\sqrt{\lambda}x)$
$X(0)=0 \Rightarrow c_1=0 \Rightarrow X(x)=c_2=\sin (\sqrt{\lambda}x)$
$X'(1)-\lambda X(1)=0 \Rightarrow \tan (\sqrt{\lambda})=\frac{1}{\sqrt{\lambda}}$

That means that the eigenvalue problem $(1)$ has only positive eigenvalues $0<\lambda_1 < \lambda_2 < \dots < \lambda_k < \dots $ that are the positive roots of the equation $\tan \sqrt{x}=\frac{1}{\sqrt{x}}$.
Is this correct??
Why can we say that the number of the eigenvalues is countable ??
How can we show that $$\lim_{k \rightarrow +\infty} \frac{\sqrt{\lambda_k}}{k \pi}=1$$ ??
Answer: It may be easier in this case to take the general solution as
$$
X(x)=c_1\sinh \mu x+c_2\cosh \mu x,\quad \mu:=\sqrt{-\lambda}.
$$
Then the first boundary condition implies that $c_2=0$ and the second is that either
$$
c_1=0
$$
or
$$
\tanh \mu =-\frac{1}{\mu}.
$$
But the last equation has no solutions, hence $c_1=0$.
Comment: How do we know that the equation $$\tanh \mu=-\frac{1}{\mu}$$ has no solutions ?? @Artem
Comment: I have edited my initial post... Could you take a look at it ?? @Artem
Comment: @MaryStar It looks fine to me. To answer your two last questions try to sketch the graphs of $\tan h$ and $1/h$.
Answer: Using an integrating factor, we can show that the solution to
$$
X''+\lambda X=0\tag{1}
$$
for $\lambda\gt0$, is
$$
X(x)=a\cos(\sqrt\lambda\,x)+b\sin(\sqrt\lambda\,x)\tag{2}
$$
for $\lambda=0$, is
$$
X(x)=a+bx\tag{3}
$$
and for $\lambda\lt0$, is
$$
X(x)=a\cosh(\sqrt{-\lambda}\,x)+b\sinh(\sqrt{-\lambda}\,x)\tag{4}
$$
Furthermore, we also have
$$
X(0)=0\tag{5}
$$
and
$$
X'(1)-\lambda X(1)=0\tag{6}
$$

If $\lambda\gt0$, $(2)$ and $(5)$ say that $a=0$. Then $(2)$ and $(6)$ say that
$$
b(\sqrt\lambda\cos(\sqrt\lambda)-\lambda\sin(\sqrt\lambda))=0\tag{7}
$$
if $b=0$, then $X(x)=0$, otherwise, we need
$$
\tan(\sqrt\lambda)=\frac1{\sqrt\lambda}\tag{8}
$$
which has an infinite set of solutions, the smallest being $\lambda=0.7401738843949670422$.

If $\lambda=0$, $(3)$ and $(5)$ say that $a=0$. Then $(3)$ and $(6)$ say that $b=0$. Therefore, $X(x)=0$.

If $\lambda\lt0$, $(4)$ and $(5)$ say that $a=0$. Then $(4)$ and $(6)$ say that
$$
b(\sqrt{-\lambda}\cosh(\sqrt{-\lambda})-\lambda\sinh(\sqrt{-\lambda}))=0\tag{9}
$$
if $b=0$, then $X(x)=0$, otherwise, we need
$$
\tanh(\sqrt{-\lambda})=-\frac1{\sqrt{-\lambda}}\tag{10}
$$
which has no solutions since $\tanh(x)\gt0$ when $x\gt0$ and $\tanh(x)\lt0$ for $x\lt0$.

Thus, as you have added since I started writing this up, there is a sequence of solutions with $\lambda\gt0$, where $\lambda$ satisfies $(8)$.
As $\lambda$ gets larger, $\frac1{\sqrt\lambda}\to0$. The places were $\tan(x)$ is positive and near $0$ are slightly greater than multiples of $\pi$. Thus, $\sqrt\lambda$ must be slightly greater than $k\pi$ for some non-negative $k\in\mathbb{Z}$. Certainly, we can label each such $\lambda$ as $\lambda_k$. In that case, we have
$$
\lim_{k\to\infty}\frac{\sqrt{\lambda_k}}{k\pi}=1\tag{11}
$$
Comment: To divide with $X(1)$ are we sure that it isn't zero?? $$X'(1) \cdot T(t)+X(1) \cdot T'(t)=0 \Rightarrow \frac{X'(1)}{X(1)}=-\frac{T'(t)}{T(t)}$$ ?? @robjohn
Comment: Indeed. I am not sure why you are bringing this up. Are you worried that we get that $a=0$ and $b=0$ imply that $X(x)=0$? Well, $X(x)=0$ *is* a solution, so we don't need to worry about the division by $0$. However, if you want to worry about it, you can assume that $X(x)\ne0$ for some $x$ and then when we derive $X(x)=0$, it is a contradiction; that is $b\ne0$.
Comment: Ok... When we have an other problem and we end up having $$
\tanh(\sqrt{-\lambda}\pi)=-\frac1{\sqrt{-\lambda}}\tag{10}
$$ would we say again that it has no solutions since $\tanh(x)\gt0$ when $x\gt0$ and $\tanh(x)\lt0$ for $x\lt0$ ?? @robjohn
Comment: Yes. That is just what I wrote. To be inclusive, I wanted to also knock out the negative $\sqrt{-\lambda}$, too. If we use $\sqrt{-\lambda}\gt0$, the left hand side is positive and the right hand side is negative. If we use $\sqrt{-\lambda}\lt0$, the left hand side is negative and the right hand side is positive.
Comment: So can we write it in that way although at tan we have $\sqrt{-\lambda }\cdot \pi $ and at the other side of the equation just $\sqrt{-\lambda}$ ?? @robjohn
Comment: I am really not sure what your point is. The two sides of the equation are opposite signs. What is the problem?
Comment: Now we have the equation $$\tanh (\sqrt{-\lambda} \pi )=-\frac{1}{\sqrt{-\lambda}}$$ Can we justify that it has  no solutions as before , $$\tanh (x)>0 \text{ when } x>0 \text{ and } \tanh (x)<0 \text{ when } x<0$$ although now we have at $\tanh $, $\sqrt{-\lambda } \pi$ and at the other side of the equality $\sqrt{-\lambda}$ without $\pi$ ?? @robjohn
Comment: What does the $\pi$ change? You still have opposite signs on each side of the equation.


What programming language will enable me to enter a very long number without converting it to floating point?

Question: What would be the best way to do the following.
Enter a very long number, lets say 500,000 digits long without it going into scientific notation; and then am able to do math with it, like +2 etc.?
Thank you in advance.
EDIT: It is a 500,000 digit, positive integer.
Comment: Python does it without any kind of special library at all.

Also, the bc program for Unix, which is a full programming language, can also do it without any kind of special library.
Comment: Pretty much any language would. You may have to write some code, though :-)
Comment: @snicker: yes it can, it just needs a *lot* of digits. Double-precision floating point numbers (which are just one implementation based on scientific notation) don't have enough digits, by a long way, but there's no inherent limit to scientific notation in general.
Comment: @unknown: Any turing-complete programming language that accepts textual input and has access to enough memory, anyway.
Comment: What do you have against scientific notation?
Comment: Surely he means floating point.
Comment: Any language will. If not out of the box, you can always write your own handler.
Comment: @Gurdas because scientific notation is not a number. It's a notation format that cannot express a number such as 10^500000 + 2 without **rounding error**.
Answer: Python and Java have native support, libraries exist for C++, C, .NET,  ...
Comment: I wouldn't describe Java's BigInteger class as "Native Support".
Comment: I dunno, BigInteger isn't as "native" or as pleasant to use as what Python provides, but at least it comes with the language.
Answer: I know that Erlang has support for unlimited size int arithmetics.
Comment: +1 Erlang is very popular for this. And it is free... unlike Mathematica.
Comment: Erlang does not have arbitrary precision floats.
Comment: Further clarification: having browsed the source code of Erlang, the integer arithmetic is written in C so it very much native IMO.
Answer: Python does this out of the box with no special library.  So does 'bc' (which is a full programming language masquerading as a calculator) for Unix systems.
Answer: Python is pretty good on its own, but better with gmpy (which bridges it to the GMP library others have mentioned, or alternately to the MPIR kinda-work-alike one [[work in progress;-)]]).  Consider:
$ python -mtimeit -s'x=int("1"*9999); y=int("2"*9999)' 'x*y'
100 loops, best of 3: 6.46 msec per loop

i.e., in pure Python, multiply two 10K-digits ints takes 6.5 milliseconds or so.  And...:
$ python -mtimeit -s'from gmpy import mpz; x=mpz("1"*9999); y=mpz("2"*9999)' 'x*y'
1000 loops, best of 3: 326 usec per loop

...with gmpy at hand, the operation will be about 20 times faster.  If you have hundreds rather than thousands of digits, it's even more extreme:
$ python -mtimeit -s'x=int("1"*199999); y=int("2"*199999)' 'x*y'
10 loops, best of 3: 675 msec per loop

vs
$ python -mtimeit -s'from gmpy import mpz; x=mpz("1"*199999); y=mpz("2"*199999)' 'x*y'
100 loops, best of 3: 17.8 msec per loop

so, with 200k digits instead of just 10k, gmpy's speed advantage is 38 times or so.
If you routinely need to handle integers of this magnitude, Python + gmpy is really a workable solution (of course I'm biased, since I did author and care for gmpy over the last few years exactly because I ♥ Python (hey, my license plate is P♥thon!-) and in one of my hobby (combinatorial arithmetic) I do have to deal with such numbers pretty often;-).
Answer: Mathematica allows you to do such math and you can write complete programs in it.
Otherwise, what you seek is a "library" to extend the built-in functionality of another programming language, such as Python or Java.
In the case of Python, the decimal module enables you to specify a precision in which math operations will be peformed.
Comment: I'll try Mathematica presently.
Comment: Are you working with whole numbers, or with numbers which also have a fractional part?  In either case, Mathematica will work, I am just curious as to what other solutions apply to you.
Comment: If you specify a precision, you can't do things like add 2 to a number with 500,000 zeros.. unless you use a bignum module or... Erlang.. which is free. Last I checked, Mathematica was expensive... and was WAY more powerful than just needing (nearly)infinite precision.
Comment: I thought there was a trial... but you must apply for it. I will probebly try Erlang or Python.
Answer: Haskell (when using GHC) also has builtin support for arbitrarily long integers.  Here's a snippet showing the length of a number converted to a string.
Prelude> length $ show $ 10
2
Prelude> length $ show $ 1 + 2^2000000
602060
Prelude> let x = 2^200000
Prelude> let y = 2^200000 + 5
Prelude> y - x
5

Or you could just type 2^200000 at the interactive console and wait a couple minutes for it to print out all 600k+ characters.  I figured this way was a little simpler to demonstrate.
Comment: Depends on the version of Haskell. Helium, for example, does not support arbitrary-precision arithmetic.
Answer: Common Lisp has built-in support for arbitrary large numbers as well...
Answer: Perl has a bignum module to do that sort of thing, and Python supports it natively.
Comment: Python doesn't have a bignum module: its integers are natively unlimited precision.
Answer: Perl, Python, Ruby, and Java can all do that. External libraries exist for everything else.
I rather like Ruby and Python because they automatically switch from Fixnum to Bignum. (Python: int to long.)
Comment: Erlang also does this, as do several other programming languages not on your list.
Answer: What you're looking for isn't necessarily a language, but an arbitrary-precision library.
GMP would be a fast implementation in C/C++, and scripting languages that handles big integers would probably use something like that.
Answer: In C or C++, you can use GMP (Gnu Multi-Precision library).
In Perl, you can use the bignum module.
Answer: MIT/GNU Scheme has support for arbitrarily large numbers.
Answer: Many functional languages natively support arbitrary-precision numbers. Some have already been mentioned here, but I'll repeat them for completeness:

Most versions of Haskell.
Miranda, a predecessor to Haskell.
Some implementations of Scheme. In particular, PLT Scheme.
Erlang.
Answer: I find Python quite good for this.


how to append list of DOM elements objects to another list of DOM element objects

Question: I have listed some DOM elements from a XML file and kept in list1.
list1 = [<DOM Element: DeviceInfo at 0x1915c88>, <DOM Element: ManagementServer at 0x1961b70>, <DOM Element: GatewayInfo at 0x19cd5d0>]

I have another list2, which also has DOM elements objects,
list2 = [<DOM Element: VendorConfigFile__INSTANCE__ at 0x1915f08>, <DOM Element: SupportedDataModel__INSTANCE__ at 0x191bda0>]

i want to append the list2 to list1.
I use the command.
newList = list1.append(list2)

expected output:
newList = [<DOM Element: DeviceInfo at 0x1915c88>, <DOM Element: ManagementServer at 0x1961b70>, <DOM Element: GatewayInfo at 0x19cd5d0>,[<DOM Element: VendorConfigFile__INSTANCE__ at 0x1915f08>, <DOM Element: SupportedDataModel__INSTANCE__ at 0x191bda0>]]

but I'm getting it as 'None' in the newList
what exactly is the problem and how to get this done.
Thank you
in my lists it contains <'DOM Element:SupportedDataModel__INSTANCE__ at 0x191bda0'> and many such DOM element objects, contents inside <> are not displayed inside the list.
Answer: Updated!
return type of append is None.
so list1.append(list2) gives out None but also changes list1.
so you should do.
newList = list(list1)
newList.append(list(list2))

Comment: hi Dinesh, i want it exactly like lists of lists: my new list should look like, newList = [ list1 , [list2] ]
Comment: If your `list1` and `list2` doesn't contain lists with in them, you can use this. else use `deepcopy()` instead of `list()`


Vim search and replace digits

Question: I'm using VIM to search and replace something like so
:0,$s/min-width:\s\d+/min-width: 61111/gc
But it says no pattern found, even though i see very clearly there are strings like min-width: 768px.  If I change my vim command to
:0,$s/min-width:\s768/min-width: 61111/gc
THen it works.  How come the \d isn't being recognized?
Comment: @henices It has to be `\vmin\-width\:\s\d+`. According to the help `-` and `:` are not in the list of characters that must not have special meaning: "Use of "\v" means that in the pattern after it all ASCII characters except '0'-'9', 'a'-'z', 'A'-'Z' and '_' have a special meaning." It does not mean they actually do have one, but they *may*.
Answer: You have to use \+ in your pattern instead of +.
See :help pattern, it says:
\+  Matches 1 or more of the preceding atom, as many as possible.

With your current pattern, you are searching for a single digit followed by a +.

Vim search and replace digits


IEnumerable in C# mvc

Question: I have two tables, Student (name, id, UniversityID) and Universities (Id, Name). One has the information about the student details and the other about the universities. In the first table I am storing the Student.UniversityID that matches with Universities.ID. The Universities are shown as a drop down in a form about the student details.
My question is when I am editing a specific student how to have the UniversityName choosen in the previous form while adding the student details, preselected.
What i am doing so far is that i am populating the dropdown like this
public ActionResult Edit(int? id)
{
    IEnumerable<SelectListItem> items = db.Universities
        .Select(c => new SelectListItem
        {
            Value = c.UniversityId.ToString(),
            Text = c.UniversityName.ToString(),
            Selected = true,
        });
    ViewBag.UniversityID = items;
}

I guess the best thing to do is a join between the two tables based on the
{select UniversityName from Universities Join Student where  Student.UniversityID=Universities.ID }
Any idea how to do this join these two tables in Linq and to select the UniversityName that fits this contidition, using  IEnumerable SelectListItem???
Answer: If you are using EntityFramework you can specify ForeignKey constraints, in which case you can just do:
db.Students.Select(student => new {
    UniversityName = student.University.UniversityName
})

To add ForeignKey constraints, in you EntityFramework data entity do:
[Table("Student")]
public class Student
{
    [Key] public int Id { get; set; }
    public string Name { get; set; }
    public int UniversityId { get; set; }

    [ForeignKey("UniversityId")] public University University { get; set; }
}

IEnumerable in C# mvc


is there an OnActivate method for angular?

Question: I currently have an angular page which implements ngOnInit().
Currently i am on /home and after clicking the home link it doesn't trigger ngOnInit(), is there any onActivate()?
ngOnInit() {
    this.getPage(1);
}

Answer: Not a very elegant solution, but i solved it. Below are the codes i have came up with in case anybody needs it too. feel free to comment on how i can improve the code. Thanks!
//app-routing.module.ts
{
    path: 'home',
    component: HomeComponent,
    data: {
        title: 'Home'
    },
    runGuardsAndResolvers: 'always',
}
...
@NgModule({
    imports: [RouterModule.forRoot(routes, {onSameUrlNavigation: 'reload'})],
    exports: [RouterModule]
})

//home.component.ts

export class HomeComponent extends OnActivate implements OnInit {
    constructor(public router: Router) {
        super(router);
    }

    initialisePage(): void {
        this.getPage(1);
    }

}

//OnActivate.ts
export class OnActivate {

    navigationSubscription;

    constructor(
        public router: Router
    ) {

        this.navigationSubscription = this.router.events.subscribe((e: any) => {
            if (e instanceof NavigationEnd) {
                this.initialisePage();
            }
        });
    }

    initialisePage() {

    }

    ngOnDestroy() {
        if (this.navigationSubscription) {
            this.navigationSubscription.unsubscribe();
        }
    }

}

Comment: I think it's better use some like if (this.router.url=='/Home') this.getPage(1) else this.router.navigateTo(['/Home'];
Comment: problem is, if the current route is `/home`, clicking a link with `routerLink='/home'` doesn't do anything. OnInit and the constructor is only triggered when you first visited the link.


Authorization user actions using spring aop theory

Question: Today I want to ask about authorizing user actions. For example I've got a List containing all authorized users. Shall I now create an advice which shall search for all actions requiring authorization. Use @Around and check if user is authorized. Then call users method. Otherwise if user isn't authorized do nothing. Am I right? Are there any other ways to do that?
Thanks in advance.
Answer: I think it will work. You can also use Spring Security. It is a part of Spring framework. It also use AOP.
http://docs.spring.io/spring-security/site/docs/3.0.x/reference/technical-overview.html
You can find a lot of tutorial on the Internet. Just google "spring security authorization tutorial"


C# List.ConvertAll in .NET 2.0

Question: I have a list of strings. All of the strings have whitespace that needs to be converted to underscores. I am fully capable of using a for or foreach loop to do this. I am still relatively new to C# and would like to become more familiar with it. With that said, my question is:
How can I get the following code to work in .NET 2.0? When I check fieldList at the end of the ConvertAll operation, nothing has changed. Is there an issue with passing the string by value instead of reference?
string fields =
  "First Name,Middle Name,Last Name,Birth Date,Gender,Address,City,State,Zip,Email";
List<string> fieldList = new List<string>(fields.Split(','));
fieldList.ConvertAll<string>(new Converter<string, string>(
    delegate(string str)
    {
        str = str.Trim();
        str = str.Replace(' ', '_');
        return str;
    }
));

Please, keep in mind, that I am using .NET 2.0 and cannot currently switch, so I do not have the luxury of using LINQ or Lambdas.
Answer: You need to assign the results of the ConvertAll method to the variable like this:
fieldList = fieldList.ConvertAll<string>(new Converter<string, string>(
    delegate(string str)
    {
        str = str.Trim();
        str = str.Replace(' ', '_');
        return str;
    }
));

The ConvertAll method returns a new List<T> so you need to assign the result of the method.  If you want to re-use the fieldList variable you can but it may be better to create a new variable to improve the clarity of your code:
List<String> convertedFieldList
    = fieldList.ConvertAll<string>(new Converter<string, string>(
        delegate(string str)
        {
            str = str.Trim();
            str = str.Replace(' ', '_');
            return str;
        }
));

As Marc Gravell points out in a comment below, you can simplify the syntax of this expression by doing this:
List<String> convertedFieldList
    = fieldList.ConvertAll<String>(delegate(String str) {
            return str.Trim().Replace(' ', '_');
        });

Comment: Just for completeness, you can simplify to `...ConvertAll(delegate (string str) { ... });`
Comment: I got this error:
The type arguments for method 'System.Collections.Generic.List.ConvertAll(System.Converter)' cannot be inferred from the usage. Try specifying the type arguments explicitly.
Comment: Sorry, I fixed the error.  I am used to using version 4 of the C# compiler :)
Comment: I re-added the new Converter.

fieldList = fieldList.ConvertAll(new Converter(
            delegate(string str)
            {
                return str.Trim().Replace(' ', '_');
            }
        ));
Answer: ConvertAll doesn't change the input list. It returns a new list containing the converted stuff. By the way, you can remove the new Converter<string,string> with C# 2.0+:
List<string> converted = fieldList.ConvertAll<string>
    (delegate(string s) { return s.Trim().Replace(' ', '_'); });

Besides, nothing prevents you from using a C# 3.0 compiler and LINQBridge and target .NET 2.0.
Comment: For some reason, C# 2.0 could not infer the type without the new Converter declaration.

C# List.ConvertAll in .NET 2.0


Nagiosgraph mouseover fails

Question: I have setup Nagios 4 together with Nagiosgraph and they are both working perfect.
However the javascript Mouseover effect does not work.
I have copied the following line in ../nagios/share/ssi/common-header.ssi as mentioned in Nagiosgraph Documentation and it is the only line:

However when I make a Mouseover effect over the Nagiosgraph icon I get the following error in apache error.log:
[Thu Nov 27 11:48:21 2014] [error]  ../nagios/share/ssi/common-header.ssi: line 1: syntax error near unexpected token `<'
[Thu Nov 27 11:48:21 2014] [error]  ../nagios/share/ssi/common-header.ssi: line 1: `'

Are there any ideas on how to resolve this?
Answer: Finally managed to resolve the issue.
The following steps should be followed:

The ../nagios/share/ssi/common-header.ssi file should be in shell script format:
 #!/bin/bash
 echo ""

Also the file ../nagios/share/ssi/common-header.ssi should be readable and executable, which means I had to run the following command:
chmod a+rx common-header.ssi

Comment: Unfortunately this doesn't seem to work for me so it doesn't look like a universal solution
Answer:
The first step to installing nagiosgraph, is to download the debian package and install it. The package can be downloaded via WGET, here:

a.       http://www.mirrorservice.org/sites/downloads.sourceforge.net/n/na/nagiosgraph/nagiosgraph/1.4.4/nagiosgraph-1.4.4-2.deb

Once you've run wget and have the debian installation file on your Alienvault system, you need to install it with the following command:

a.       dpkg –i nagiosgraph-1.4.4-2.deb

The final piece of the install is to modify the file, /etc/nagios3/conf.d/generic-service_nagios2.cfg. In this file, you need to a the following line just above the register line.

a.       action_url /nagiosgraph/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$' onMouseOver='showGraphPopup(this)' onMouseOut='hideGraphPopup()' rel='/nagiosgraph/cgi-bin/showgraph.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&period=week&rrdopts=-w+450+-j
  4.    After that, you need to add the nagiosgraph javascript file to the nagios header with the following command:
              a.    cp /usr/share/nagiosgraph/examples/nagiosgraph.ssi /usr/share/nagios3/htdocs/ssi/common-header.ssi

5.       Once you're done modifying the file, you simple restart nagios3 and apache2 with the following commands:
a.       service nagios3 restart
b.    service apache2 restart
Comment: As mentioned in the question NagiosGraph has been installed and is working perfectly. Could you please help with the MouseOver issue I have?

Nagiosgraph mouseover fails


how to express big-M formulation as an indicator constraint

Question: I want to solve the following LP using CPLEX. In the CPLEX documentation it is suggested to use indicator constraint instead of big-M formulation. I searched on google the term "indicator constraints" but except some research papers I could not find much information on this type of constraints.
I would appreciate it if someone could guide me on how to convert the big-M equation into an indicator constraint.
Problem Formulation
$$ \min \sum_{(i,j) \in E} \sum_{f \in F} c_{ij}^fx^f_{ij}$$
$$ \sum_{j \in V}x^f_{ij} - \sum_{j \in V}x_{ji}^f = \begin{cases}
d_f  &, & i = s_f \\
-d_f&, & i = t_f\\
0 &,& \text{otherwise}
\end{cases} $$
$$ \sum_{i,j \in E} x_{ij}^f \le u_{ij}$$
$$ \sum_{i,j \in E} w_{i,j}y_{ij}^f \le W^f$$
$$0 \le x^f_{i,j} \le My^f_{i,j}, \,\, y^f_{i,j} \in \{0, 1\} \quad \forall (i,j) \in E, f \in F$$
This is a modified multicommodity flow problem that for a flow $f$ the sum of all the weights $w_{ij}$ of the edges for which $x^f_{i,j} \gt 0$ should be less than $W^f$.
Solution 1
$$ \min \sum_{(i,j) \in E} \sum_{f \in F} c_{ij}^fx^f_{ij}$$
$$ \sum_{j \in V}x^f_{ij} - \sum_{j \in V}x_{ji}^f = \begin{cases}
d_f  &, & i = s_f \\
-d_f&, & i = t_f\\
0 &,& \text{otherwise}
\end{cases} $$
$$ \sum_{i,j \in E} x_{ij}^f \le u_{ij}$$
$$ \sum_{i,j \in E} w_{i,j}y_{ij}^f \le W^f$$
$$ y_{i,j}^f = 0 \Rightarrow x_{i,j}^f = 0 \quad \forall (i,j) \in E, f \in F$$
$$ x^f_{i,j} \ge 0, \,\, y^f_{i,j} \in \{0, 1\} \quad \forall (i,j) \in E, f \in F$$
Answer: The indicator constraint is
$$y_{i,j}^f = 0 \Rightarrow x_{i,j}^f = 0 \quad \forall (i,j) \in E, f \in F$$
Comment: Thank you LinAlg. Would you please guide me on  how to add this indicator constraint into my formulation. I have modified my formulation in my question, is this correct?
Comment: @Corey yes, but you should write either $\Rightarrow$ or "if .. then". Use \text in latex for text.
Comment: I removed the "if". LinAlg would you please explain this indicator constraint for me. Since we are saying that if $y^f_{ij}$ is zero then $x^f_{ij}$ should also be zero. I have a bit of doubt on how the solvers know that whether $y^f_{ij}$ is zero or one; as no where in the formulation we are changing the value of $y$
Comment: The solver will optimize $y$ just as in your original formulation. The formulations are equivalent.
Comment: Just one more question, there is no need to add $y$ to the objective function?
Comment: @Corey that's right.


Iterações dentro de TransactionScope

Question: Tenho um projeto Asp.Net MVC com Entity Framework 5.0 em .Net Framework 4.0.
Quais são os prejuízos trazidos por uma TransactionScope muito longa?
Tenho o seguinte trecho de código:
using(TransactionScope scope = new TransactionScope())
{
   foreach(var categoria in categorias)
   {
      // Sequencia de instruções
      db.Categoria.Add(categoria);
   }
   foreach(var produto in produtos)
   {
      // Sequencia de instruções
      db.Produto.Add(produto);
   }
   // Sequencia de instruções ou outra iterações.
   db.SaveChanges();
   scope.Complete();
}

Ele seria melhorado de alguma forma dividindo em várias Transactions?
Answer:
Quais são os prejuízos trazidos por uma TransactionScope muito longa?

Penalidades de performance, basicamente. Se o escopo da transação ainda pedir lock exclusivo, o sistema como um todo fica lento.

Ele seria melhorado de alguma forma dividindo em várias Transactions?

Depende. Se as entidades não possuem correlação, você pode paralelizar. Caso contrário, pelo fato de o contexto não ser thread-safe, não é possível paralelizar.
Isto não significa que ficará mais rápido. A paralelização é recomendada para volumes grandes de dados. Para transações pequenas, a diferença de performance não é muito sentida.
Outra coisa que você pode fazer é usar métodos assíncronos, que melhoram a concorrência entre requisições, mas você terá que ir para o .NET 4.5 e Entity Framework 6:
using(TransactionScope scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled)) // Só no .NET 4.5 em diante dá pra fazer isso
{
   foreach(var categoria in categorias)
   {
      // Sequencia de instruções
      db.Categoria.Add(categoria);
   }
   foreach(var produto in produtos)
   {
      // Sequencia de instruções
      db.Produto.Add(produto);
   }
   // Sequencia de instruções ou outra iterações.
   await db.SaveChangesAsync();
   scope.Complete();
}

Iterações dentro de TransactionScope


why isn't java.lang.Throwable an abstract class?

Question: Possible duplicate: why-is-java-lang-throwable-a-class
Hi! I doesn't understand why Throwable isn't abstract class. I see only one use case for these: in logging systems for figure out call hierarchy. But it can be some static method for this or other class. So, why?)
Thanks.
upd
from  java.util.logging.LogRecord
// Get the stack trace.
StackTraceElement stack[] = (new Throwable()).getStackTrace();

Why it can't be Throwable.getStackTrace();  or as in  java.lang.Thread
(new Exception()).getStackTrace();

In this way we can avoid throw new Throwable();
upd2
from javadoc

The Throwable class is the superclass
  of all errors and exceptions in the
  Java language.

So, as a superclass it should be abstract, imho. Using it for getting stacktrace isn't good case by this definition.
Comment: Where does it ever "throw new Throwable();" ?  Not sure you need to avoid something which never happens. ;)
Comment: @Peter Lawrey : Somebody can write it. API should avoid bad using (not only my IMHO).
Comment: @Stas, It is highly unlikely to compile accidently as it is checked exception. You have to handle it or add it to the throws clause. I think that is avoidance enough.  Not convinced there is a serious problem which needs another solution.
Comment: @Stas, My IDE also has a warning "Prohibited Exception thrown" This can be changed to a compiler error level.
Comment: possible duplicate of [why is java.lang.Throwable a class?](http://stackoverflow.com/questions/2890311/why-is-java-lang-throwable-a-class)
Comment: Still doesn't answer the question. Being a superclass isn't a sufficient reason to want it to be abstract.
Comment: @EJP, If we use class only as superclass, we can use only protected constructor. If we doesn't create its instances - why it shouldn't be abstract?
Comment: Still doesn't answer the question. Why *should* it be abstract? Who said that classes used only as superclasses should be abstract? This is not a hard and fast principle. The only Java language rule about abstract classes is that classes with abstract methods must be declared as abstract. You're trying to create a new rule, and you haven't given anything that approaches a justification. And who said we only use it as a superclass in the first place?
Comment: @EJP, to get answer for your first question I start new topic: http://stackoverflow.com/questions/4311271/. About second. I think we should use classes only in use cases that docs gives as. In other hand, API should avoid using classes in wrong ways.
Comment: You're still begging the question. You are defining something as a 'wrong way' without any actual justification, and then using that as the basis for your conclustion. It doesn't follow, and your premiss is invalid. There is nothing in the Javadoc that prevents using Throwable directly.
Comment: @EJP, ok. I give up) You are right. But if It was I why develop this API, I've made it in other way.
Answer:
I doesn't understand why Throwable
  isn't abstract class.

The answer is clearly stated here.

Why it can't be
  Throwable.getStackTrace(); or as in
  java.lang.Thread

Quite simply, the getStackTrace() calls the getOurStackTrace() method which is non-static. If getStackTrace() was static, so should getOurStackTrace(). This won't happen as printStackTrace() method uses the getOurStackTrace(). This is elaborated in the JavaDoc:

Provides programmatic access to the
  stack trace information printed by
  printStackTrace().

Source for java.lang.Throwable:
 public StackTraceElement[] getStackTrace() {
        return (StackTraceElement[]) getOurStackTrace().clone();
    }

Also, if you read the code of getOurStackTrace() method, you'll see it calls the following method:
private native int getStackTraceDepth();

As far as I know, native cannot be static (I may be wrong).
Comment: @Stas: They would have completely different semantics if it were static.  `getStackTrace()` for a subclass of `Throwable` is supposed to return the stacktrace at the point that `fillInStackTrace()` was called, which is at construction of the `Throwable` typically.  Calling it statically would have to return whatever the stacktrace is *at the time of the method call* which would be much less useful (e.g. it would print the stacktrace to the logging code, not to where the exception occurred).
Comment: Native methods can be static (see pretty much any method in `java.lang.StrictMath`).
Comment: @Peter Lawrey, it's on the `Throwable` class in the JDK (mine is 1.6.0-21, it was introduced since JDK 4 I believe?)
Comment: If native can be static, I think `Throwable.getStackTrace()` better idea than `new Throwable().getStackTrace()`
Comment: @The Elite Gentleman, java.lang.Thread also has this method. It is marked @since 1.4. Not sure there was a JDK 4. ;)
Comment: @Stas, Throwable.getStackTrace() isn't static because it gets/populates the stack trace for the throwable. (Where it was first created)  Calling it more than once, in different places won't give you a different result.
Comment: @Peter Lawrey, the `Thread.getAllStackTraces()` method effectively calls `private native static StackTraceElement[][] dumpThreads(Thread[] threads);` so it's still done natively.
Comment: @Peter Lawrey, @Mark Peters, `Throwable` can initialize itself by calling this static method, cant it?
Comment: @Stas, This method isn't used for good reasons. It initializes the stack trace lazily to improve performance. Most of the time, this array isn't needed, e.g. catch(Exception e) but taking the stack trace when it is used isn't very useful either.
Comment: @The Elite Gentleman, Not sure how your point relates to what I said.
Comment: @Stas: A static method has no reference to an "instance", so no instance can be initialized from a static method unless you pass the instance in.  If you're wondering if the data could be stored statically, the answer is yes, but that is absolutely horrible.  It would prevent Throwable from being used from multiple threads (without jumping through dozens of hoops), for starters.
Comment: @Peter Lawrey, sorry...I got lost somehow... :)
Comment: @The Elite Gentleman, still a good point. Another useful method worth knowing about.
Comment: @Mark Peters : I mean smt like `Throwable(){ stackTrace = stackTrace();}  public static StackTrace stackTrace()`. Why it's bad?
Comment: @Stas: Well, for starters it would make almost no sense to have such a static method in `Throwable`, since at this point it has nothing to do with throwing/catching, and everything to do with the state of the `Thread`.  And as has been explored, Thread already *has* a method for retrieving the stack trace.
Comment: @Mark Peters, I got lost too. Let's put `native getStackTrace()` into `Thread` and will use it in `abstract class Throwable`. why these solution is bad? PS : off course now API frozen.
Answer: I use it quite often for logging, so I am glad it isn't abstract.  There is a method to get the call stack, Thread.currentThread().getStackTrace() but this returns a StackTraceElement[] which isn't very useful for logging.
EDIT:
StackTraceElement[] stes = Thread.currentThread().getStackTrace();

Note: this method also works to get a stack trace of another thread which can be handy.
Comment: I don't think making Throwable abstract would really stop you...you could either do `new Throwable(){}.printStackTrace()` for throwaway logging, or for logging that you keep you should probably be using the cleaner `Thread` solution anyway (or at least a library method that wraps it).
Comment: Can you show how do you use it? In java.util.logging it used to get StackTraceElement[] "(new Throwable()).getStackTrace();"
Comment: @Mark, logging StackTraceElement[] is tedious to make readable.  You don't want it to be split by other log messages for example. esp another stack trace.
Comment: fair enough, though for non-throwaway logging I would expect one to be using a logging framework that does all this for you anyway (all logging frameworks I've seen have methods to log a Throwable directly).


STL List using c++ ....in passing array of pointer of list

Question: i am facing problem in passing STL list obj b reference, here is the code
using prototype
void input(list<string>& *x[])

declaring in main
list<string> *table[SIZE];
input(table);     //calling

any help would be apprecited.
Comment: Are you sure that you want `table` to be an array of pointers? It is hard to say without further information, but a `vector>` or a `vector*>` would be better in the vast majority of cases.
Answer: Your function is attempting to take a pointer to a pointer to a reference to a list<string>. Your problem is that it is impossible to create a pointer to a reference.
You should instead try making a reference to an array of pointers to list<string>:
void input(list<string> *(&x)[SIZE])


When to Use Azure Caching Local Cache

Question: I want to start using the Azure Distributed Caching and came across the concept of LocalCache. But the fact that it can go out of sync with the Distributed Cache, makes me wonder, why I would want to use it and how I could use it safely.
When enabled, items retrieved from the cache cluster are locally stored in memory on the client machine. This improves performance of subsequent get requests, but it can result in inconsistency of data between the locally cached version and the actual item in the cache cluster.
Calling DataCache.GetIfNewer is one option to ensure that I get the latest version, but that requires that I still do a call to the Distributed Cache, passing in the object that I want to check, in order to see if the two versions differ.
I could use Notifications to invalidate the LocalCache object, but that is done on a polling basis, which opens up the opportunity for an update to occur within the poll period leaving me with stale data.
So,why would I ever use LocalCache, and if there is a reason to do so, how do I use it safely?
Answer:
"There are only two hard things in Computer Science: cache invalidation and naming things" - Phil Karlton

You would use LocalCache when a) performance is critical b) you don't care that the retrieved object might be stale.
There are many cases where the object is never going to be out of date (e.g. list of public/bank holidays), or when you are not too worried about being 100% up-to-date (e.g. if item has > 1000 units in stock, use local cache, otherwise re-fetch from database).
Don't try and invalidate the local cache. If you need more up-to-date objects, get them from the cluster. If you cannot tolerate out-of-sync data, get it from the database. Caching is always a performance-inconsistency compromise — LocalCache more than the server cache, but the server cache is still a compromise.
Comment: Thanks Simon, the examples help to clarify things. I will keep them in mind when looking at my caching strategy.

When to Use Azure Caching Local Cache


turtle keeps crashing and won't run on python 3.1.2 on mac os x 10.6.8

Question: I am brand new to coding. The book I'm learning from said to type:
import turtle

And then to type:
t = turtle.pen()

That last command brought up an empty window titled:

Python Turtle Graphics.

This window is staying in the background and I get the spinning pinwheel.  The book said if this happened to try and use the Python console instead of the shell.  Tried bringing that up but couldn't.  Book said:

open Terminal and enter python.

This did nothing.  Does anyone know how I can get turtle working?
Answer:
t = turtle.pen()

This is a common error starting out with turtle -- either an error reading the code in the book or an error when the book was typeset (or simply a bad choice of font!)  What is really needed at this point in the code is:
t = turtle.Pen()

The pen() and Pen() methods are completely different.  Since Pen() is a synonym for Turtle(), I instead suggest:
t = turtle.Turtle()

to avoid the confusion.


Como ordenar um struct dentro de um struct

Question: Estou querendo fazer ordenação de um struct que está referenciado dentro de outra struct:
Detalhe: pode ser uma ordenação usando selection sort
typedef struct{
    int tipo;
    char dispositivo[50];
    int prioridade;
}Perif;

typedef struct no{
    Perif info;
    struct no* prox;
}Aux_Perif;

Aux_Perif* criarLista(){
    return NULL;
}

void imprimirLista(Aux_Perif** lista, int size){
    Aux_Perif* aux = *lista;
    while (aux != NULL){
    std::cout << "DISPOSITIVO: ";
    std::cout << aux->info.dispositivo << std::endl;
    std::cout << "PRIORIDADE: ";
    std::cout << aux->info.prioridade << std::endl;
    aux = aux->prox;
    }
}

void inserirFinal(Aux_Perif** perifericos, Perif barramento){
    Aux_Perif* novo = (Aux_Perif*)new Aux_Perif;
    novo->info = barramento;
    Aux_Perif* aux = *perifericos;
    Aux_Perif* anterior = NULL;
    if (*perifericos == NULL){
        *perifericos = novo;
    }else{
        while(aux != NULL){
            anterior = aux;
            aux = aux->prox;
        }
    anterior->prox = novo;
    }

}

int main(){
int solic;
int cont = 0;
int aux = 0, prior = 0;
Aux_Perif* perifericos;
perifericos = criarLista();
std::cout << "Entre com o número de solicitações: " << std::endl;
std::cin >> solic;
std::cout << "Periféricos:\n1. Impressora\n2. Mouse\n3. Teclado\n4. Scanner\n5. Roteador\n6. Disco Rígido" << std::endl;
std::cout << "De acordo com os números e Periféricos acima, insira a ordem de entrada das solicitações e sua prioridade" << std::endl;
while (cont < solic){
    Perif ordem;
    std::cin >> aux >> prior;
    switch (aux){
        case 1:
            //ordem.dispositivo = 'Impressora';
            strcpy(ordem.dispositivo, "Impressora");
            break;
        case 2:
            strcpy(ordem.dispositivo , "Mouse") ;
            break;
        case 3:
            strcpy(ordem.dispositivo , "Teclado");
            break;
        case 4:
            strcpy(ordem.dispositivo , "Scanner");
            break;
        case 5:
            strcpy(ordem.dispositivo , "Roteador");
            break;
        case 6:
            strcpy(ordem.dispositivo , "Disco Rígido");
            break;
        default:
            std::cout << "Error" << std::endl;
            break;
    }

    ordem.tipo = aux;
    ordem.prioridade = prior;
    cont++;
    inserirFinal(&perifericos, ordem);
    //delete[] ordem;

}
std::cout << "Daisy Chaining utilizando método de ordem a chegada ou entrada." << std::endl;
imprimirLista(&perifericos, solic);
std::cout << "Método por Prioridade: " << std::endl;
imprimirLista(&perifericos, solic);
return 0;
}

Nesse caso, eu quero fazer a ordenação de acordo com o Aux_perif -> info -> prioridade. Ajuda aê, pessoal!
Answer: Você pode fazer isso:
void selectionsort_AuxPerif(Aux_Perif **vetor, int tamanho) {
    int i, j;
    for (i = 0; i < tamanho - 1; i++) {
        int menor = i;
        for (j = i + 1; j < tamanho; j++) {
            if (vetor[menor]->info->prioridade > vetor[j]->info->prioridade) menor = j;
        }
        if (menor != i) {
            Aux_Perif *temp = vetor[menor];
            vetor[menor] = vetor[i];
            vetor[i] = temp;
        }
    }
}

void ordena_AuxPerif(Aux_Perif **perif) {
    // 1. Se a lista está vazia, então nem faz nada.
    if (perif == NULL) return;

    // 2. Descobre o tamanho da lista.
    int tamanho = 0;
    Aux_Perif *v;
    for (v = *perif; v != NULL; v = v->prox) {
        tamanho++;
    }

    // 3. Monta um vetor com os elementos da lista.
    Aux_Perif **vetor = (Aux_Perif **) malloc(sizeof(Aux_Perif *) * tamanho);
    int i = 0
    for (v = *perif; v != NULL; v = v->prox, i++) {
        vetor[i] = v;
    }

    // 4. Ordena o vetor.
    selectionsort_AuxPerif(vetor, tamanho);

    // 5. Corrige os ponteiros de acordo com a nova ordenação.
    for (i = 0; i < tamanho - 1; i++) {
        vetor[i]->prox = vetor[i + 1];
    }
    vetor[i]->prox = NULL;

    // 6. Corrige o ponteiro para o início da lista.
    *perif = vetor[0];

    // 7. Deleta o vetor auxiliar.
    free(vetor);
}

Ah, um outro detalhe que vi no seu código é que a função imprimirLista nunca utiliza o parâmetro size. Você pode retirar este parâmetro se preferir.
Além disso, caso você deseje substituir o método de ordenação por algum outro, é só trocar a chamada à função selectionsort_AuxPerif por alguma outra que tenha os parâmetros do mesmo tipo. Por exemplo:
void mergesort_AuxPerif(Aux_Perif **vetor, int tamanho) {
    if (tamanho <= 1) return;
    int meio = tamanho / 2;
    mergesort_AuxPerif(vetor, meio);
    mergesort_AuxPerif(&(vetor[meio]), tamanho - meio);

    Aux_Perif **temp = (Aux_Perif **) malloc(tamanho * sizeof(Aux_Perif *));
    int i, a = 0, b = meio;
    for (i = 0; i < tamanho; i++) {
        if (b == tamanho || (a < meio && vetor[a]->info->prioridade < vetor[b]->info->prioridade)) {
            temp[i] = vetor[a];
            a++;
        } else {
            temp[i] = vetor[b];
            b++;
        }
    }
    for (i = 0; i < tamanho; i++) {
        vetor[i] = temp[i];
    }
    free(temp);
}

Comment: Não me importo com o downvote, mas fico curioso com ele. Qual é a razão para votar contra? Se alguém vir dizer que é porque dei a resposta de graça, basta ver que isso aqui é apenas um pequeno detalhe de implementação dentro do contexto do que o OP está fazendo (converter a lista em array, usar o selectionsort que você acha facilmente na internet e arrumar de volta os ponteiros, o que não é nada demais). Se for por alguma outra razão, eu realmente gostaria de saber.


How to insert another webform record inside current webform and go back to previous page?

Question: I have two webform diagnosis webform and bill webform .
In diagnosis webform i have patient name , diagnosis detail and billId . In Bill webform I have BillId billAmount
which i want is that i want to save patient diagnosis info by click button_click event . after clicking i want to redirect billId page and save bill info . then go back to previous page and save patient diagnosis info with billId .
How to do that ?
Comment: But you saved diagnosis info already so why do you want to go back and save it again?
Comment: I want to save diagnosis with bill Id
Answer: Put your data inside Session State. You can acces it from any form.Tutorial


pipenv FileNotFoundError when trying to create virtual environment

Question: I am trying to setup a virtual environment using pipenv with Python 3.8.5. I installed Miniconda3 and used pip install --user pipenv and it appeared to install fine. Running pipenv --version works fine.
Eamon@DESKTOP-J37G7TQ MINGW64 ~
$ pip install --user pipenv
Requirement already satisfied: pipenv in c:\users\eamon\appdata\roaming\python\python38\site-packages (2020.11.15)
Requirement already satisfied: virtualenv in c:\users\eamon\appdata\roaming\python\python38\site-packages (from pipenv) (20.2.2)
Requirement already satisfied: pip>=18.0 in c:\users\eamon\miniconda3\lib\site-packages (from pipenv) (20.2.4)
Requirement already satisfied: certifi in c:\users\eamon\miniconda3\lib\site-packages (from pipenv) (2020.12.5)
Requirement already satisfied: setuptools>=36.2.1 in c:\users\eamon\miniconda3\lib\site-packages (from pipenv) (50.3.1.post20201107)
Requirement already satisfied: virtualenv-clone>=0.2.5 in c:\users\eamon\appdata\roaming\python\python38\site-packages (from pipenv) (0.5.4)
Requirement already satisfied: distlib<1,>=0.3.1 in c:\users\eamon\appdata\roaming\python\python38\site-packages (from virtualenv->pipenv) (0.3.1)
Requirement already satisfied: six<2,>=1.9.0 in c:\users\eamon\miniconda3\lib\site-packages (from virtualenv->pipenv) (1.15.0)
Requirement already satisfied: appdirs<2,>=1.4.3 in c:\users\eamon\appdata\roaming\python\python38\site-packages (from virtualenv->pipenv) (1.4.4)
Requirement already satisfied: filelock<4,>=3.0.0 in c:\users\eamon\appdata\roaming\python\python38\site-packages (from virtualenv->pipenv) (3.0.12)

However, when trying to create a virtual environment with pipenv --three, the result is:
$ pipenv --three
Creating a virtualenv for this project...
Pipfile: C:\Users\Eamon\python_practice\pipenv-project1\Pipfile
Using C:/Users/Eamon/miniconda3/python.exe (3.8.5) to create virtualenv...
[   =] Creating virtual environment...FileNotFoundError: [Errno 2] No such file or directory: 'c:\\users\\eamon\\miniconda3\\Lib\\venv\\scripts\\nt\\python.exe'

Failed creating virtual environment

[pipenv.exceptions.VirtualenvCreationException]:
Failed to create virtual environment.

I tried adding the path 'c:\\users\\eamon\\miniconda3\\Lib\\venv\\scripts\\nt\\python.exe' to my environment variables path list but that didn't seem to work. What else can I do?
Answer: Try to use pip package virtualenv. I've tried it on Ubuntu 20LTS and also Windows 10 and this works for me.
Install a python virtual environment builder:
python3 -m pip install virtualenv

Create a virtual environment named venv inside the project's directory: virtualenv venv
To activate this environment use this command inside project's directory: source venv/bin/activate
Install python modules dependencies listed in a requirements.txt:
python3 -m pip install -r requirements.txt

You should activate virtual environment when you working with python in this directory for package installation and for running commands in the project directory. When you need to deactivate the virtual environment do it using deactivate command.
To deactivate environment simply run: deactivate
Comment: Thanks for the suggestion! Unfortunately after I installed virtualenv and tried to create venv with `virtualenv venv`, I again got a FileNotFoundError `c:\\users\\eamon\\miniconda3\\Lib\\venv\\scripts\\nt\\python.exe`
Comment: @EamonGibbons well it looks like it has something to do with `miniconda3` you've installed. What do you see if you open command prompt and type `python --version` or `python3 --version`. Also when I was playing around with python on Windows 10 I was also using Power Shell.
Comment: So when I run `python --version` I get 3.8.5, although when I run python3 --version it shows ```bash: /c/Users/Eamon/AppData/Local/Microsoft/WindowsApps/python3: Permission denied```. Is this where the problem is coming from?
Comment: @EamonGibbons this is what I see when in command prompt on my PC:

`C:\Windows\system32>python --version`
`Python 3.9.1`

`C:\Windows\system32>python3 --version`
`Python 3.9.1`

`C:\Windows\system32>where python`
`C:\Users\dubak\AppData\Local\Microsoft\WindowsApps\python.exe`

`C:\Windows\system32>where python3`
`C:\Users\dubak\AppData\Local\Microsoft\WindowsApps\python3.exe`

So I suggest as first thing to make sure both commands work on your PC, then maybe remove `miniconda3` first, setup `venv` and then instal `miniconda3` again.


paired t-tests and interpretation of results

Question: My study is about comparing the scores a group of students (sample, N=14) in two different conditions. They play the same game in a tablet and using a computer. They were administered with a questionnaire to evaluate the dependent variable (experience), which is made up of five different components. I found out that the mean value of each component is slightly greater in the tablet game. However, the p-value was greater than the standard alpha level of 0.05 in each case. which leads us to fail to reject the null hypothesis. The null hypothesis, Ho: Difference of the Means is zero. Alternative Hypothesis H1: Difference of the Means is not equal to zero.
How do we interpret this result?
Can we say that they had the same experience in both Tablet and Computer environment, since the p-value was not significant. But how do we interpret the fact that the Mean Value was slightly greater for each component in the Tablet vs. Computer Game. For instance, component A: (µTablet:3.64 and µComp: 3.54); component B (µTablet:3.95 and µComp: 3.23); component C (µTablet:3.73 and µComp: 3.71); component D (µTablet:3.80 and µComp: 3.63); component E (µTablet:3.75 and µComp: 3.50).
Is it that N=14 (is not normally distributed), and is too small of a sample?
Kindly advise.
Comment: Regarding "is too small of a sample"... that's what pilot studies and power calculations are for - to help you find a suitable sample size. In some situations, 14 might be plenty big enough to pick up the size of difference that might be interesting to you; in others it might fall well short. One thing that's not clear from your description is whether you considered the potential for order effects (e.g. that people tended to prefer whatever they played first).
Comment: Yes, order effects were taken into consideration
Answer: You have found the problem with small sample studies!  You do not know whether there is truly no difference in the means, and the differences you observed were just due to random variation, or whether if you had obtained more data you would have gotten a significant p-value.
The interpretation to your study results is simply that the study did not reject the null hypothesis.  You have not proven that the conditions are the same, you just did not prove they were different.
In the future, consider your sample size and also the possibility of false positive results.  You said you did hypothesis tests on 5 different components.  You need far more data to support the amount of hypothesis testing you performed.  If you corrected for false positive rate using a standard Bonferroni correction for 5 tests, your p-value would have to be less than 0.01 for you to claim you can reject the null hypothesis.
Comment: Can you please clarify about which conditions were not proven to be same or different. Is it that the sample size is too small to arrive at a conclusion? How should I best report the findings, as it is? The dependent variable "experience" is composed of five different components that I individually ran the paired t-tests, and for each I obtained the p-value not to be significant. I am not sure how to proceed in order to correct for :false positive:  Do you want me to run a test for significance for the Global Average Mean value of all the 5 components. Please advise. Thanks so much
Comment: Are you suggesting that there are 5 different measurements that make up your dependent variable? If so, you may want to do multivariate paired T-test, called the Multivariate Paired Hotelling's T-Square.  See:  https://onlinecourses.science.psu.edu/stat505/node/116.  This should also help guard against multiple comparison problems.
Comment: @user39531 You can only report that the study failed to show any significant differences between the interventions (tablet or computer) on any of the 5 component scores.  I think that's all you can do now for this study.  In terms of whether a larger study would have found a significant difference, now that you have the means and variability for these 5 scores, you could perform a power calculation to see how many subjects would be needed to find a significant difference at alpha=0.01.  Keep in mind that there is a difference between "statistical difference" and "meaningful difference."
Comment: Hello User39531: Can I also add the following: the two platforms were not statistically different, but on the practical significance basis, we can say that users preferred the tablet as the µ was relatively larger in each case.
Comment: Hello StatsStudent: That is new to me, and great resource. Any tips how to conduct the multivariate paired t-test in SPSS. I believe that method minimizes errors..thanks so much
Comment: @user39531, I would reword to: "There was no statistically significant difference found between the two platforms.  The mean values for all 5 scores were higher for the tablet than the computer.  It may be that there is a small preference for the tablet that a larger study would have the power to detect."    However, I'd point out to you that your mean values seem quite close, so you should consider that in the proper context... what is your minimum score and maximum?  If you are dealing with a 10 point scale and finding the tablet mean to be 0.10 better, it's hard to argue that is meaningful.
Comment: Thanks for the kind clarification. The Likert Scale ranges from 1-5 (1:strongly disagree; 2:disagree; 3:neutral; 4: agree; 5:strongly agree). Am I supposed to disregard the "3:neutral" in my data compilation. I assume that the min score would be 1 and max score is 5. Please advise if I am calculating the score from each scale correctly.


Zen sub theme: page--node--name.tpl.php not working!

Question: I'm hoping someone can help me because I'm at my wits end!  I'm using the zen theme - then created a sub-theme and now I can't seem to get the custom templates to be recognized by the custom content type name (page--node--custom_content_name.tpl.php).
I've created a new template.php in my sub-theme and placed text that I've found in other blogs and forums to help make it recognize the content type so I can use it in the tpl.php name but nothing seems to work.  If I use the dev module it only says that I can name them page--node--1.tpl.php, which would be fine if we weren't going to be adding more content on an ongoing basis.
Right now I have so many tpl.php just so that I can show people what the pages will look like - it's crazy!! What am I doing wrong?!
Help!!
Comment: It looks like you are combining two content types (page & node). Are you trying to modify pages or nodes?
Answer: Drupal doesn't provide content type based template suggestions for page.tpl.php but for node.tpl.php. That means you can use a template like node--custom-content-name.tpl.php.
However, if you want to have a page.tpl.php for your content type, you need to tell Drupal about your template suggestion. Put this code in your template.php file:
function YOURTHEME_preprocess_page(&$variables) {
  if (isset($variables['node']->type)) {
    $content_type = $variables['node']->type;
    $variables['theme_hook_suggestions'][] = 'page__' . $content_type;
  }
}

You should name your template file like: page--custom-content-name.tpl.php.
Comment: Thanks I've tried that too.  It still won't work and the devel module isn't showing that it's even possible to name the templates with the content type.  so I must be missing some PHP somewhere... I just don't know what.
Comment: my devel module shows me my options are this: Template called:
    sites/all/themes/mazin_theme/templates/page.tpl.php
Candidate template files:
    page--node--31.tpl.php < page--node--%.tpl.php < page--node.tpl.php < page.tpl.php
Comment: Ok, I thought Zen theme would provide those template suggestions for page.tpl.php but it is not the case as I read. AdaptiveTheme has an option to activate those template suggestions. In your case you should tell Drupal about them manually. I have updated my answer, please check.
Comment: Yup - that's it!  I just had to create a new template.php and put it in my sub-theme and the only thing in it was a slightly different version: function mazin_theme_preprocess_page(&$vars) {
    // - page--example.tpl.php

if (isset($vars['node'])) {
$vars['theme_hook_suggestion'] = 'page__'.$vars['node']->type; //
}
}