Inverse quadratic optimization with linear equality constraints

Question: considering following quadratic optimization problem with linear equality constraints
\begin{equation}
\begin{aligned}
\max_w~& r^\top w - \frac{1}{2}w^\top Q w \\
\mathrm{s.t.}& \sum w = 1
\end{aligned}
\end{equation}
By given $r\in\Re^{N\times 1}$ and positive definite $Q$, it is easy to find the analytical solution:
\begin{equation}
\begin{bmatrix}
-Q & \mathbf{1}_{N\times 1} \\ \mathbf{1}_{1\times N} & 0
\end{bmatrix}\begin{bmatrix}
w\\ \lambda
\end{bmatrix} = \begin{bmatrix}
r\\ 1
\end{bmatrix}
\end{equation}
where $\lambda$ is the Lagrange multiplier. I was wondering, by given the optimal $w^*$, is it possible to derive the vector $r$?
I thought it would be easy, because
\begin{equation}
w = Q^{-1} (r + \mathbf{1}_{N\times 1} \lambda) \Longrightarrow \sum(w) = \mathbf{1}_{1\times N}Q^{-1} (r + \mathbf{1}_{N\times 1} \lambda) = 1
\end{equation}
hence
\begin{equation}
\begin{bmatrix}
I_N & \mathbf{1}_{N\times 1} \\ \mathbf{1}_{1\times N}Q^{-1} & \mathbf{1}_{1\times N}Q^{-1}\mathbf{1}_{N\times 1}
\end{bmatrix}\begin{bmatrix}
r\\ \lambda
\end{bmatrix} = \begin{bmatrix}
Qw\\ 1
\end{bmatrix}
\end{equation}
However, I found that matrix is not invertible due to rank deficiency. Can anyone help me to obtain the $r$ given $w^*$? Thanks!
Answer: The problem you pose is minimization of $||w-Q^{-1/2}r||_{Q^{1/2}}$ over the simplex, i.e. some weighted projection to the simplex. Given a projection $w^{*}$ there are generically infinitely many $r$ which yield the same projection (thus the issue with non-invertible). A picture in 2d says it all hopefully (here drawn for simple case with $W=I$, red line is the simplex, green line all possible $r$, two centers $r^{*}$ with corresponding level sets of the quadratic, and both of them yield the same projection $w^{*}$
Answer: Forming the Lagrangian
$$
L(w,\lambda) = r^{\top}w-\frac 12 w^{\top}Qw+\lambda(1^{\top}w-1)
$$
we have
$$
\nabla L = 0 = r^{\top}-w^{\top}Q+\lambda 1^{\top}=0
$$
multiplying by $w$ we have
$$
r^{\top}w-w^{\top}Qw+\lambda 1^{\top}w = 0\Rightarrow r^{\top}w-w^{\top}Qw+\lambda  = 0
$$
and finally
$$
r^{\top}-w^{\top}Q+(w^{\top}Qw-r^{\top}w) 1^{\top}=0
$$
or
$$
r - (w^{\top}r) 1 = Q^{\top}w-(w^{\top}Q^{\top}w)1\Rightarrow (I_n-W) r = Q^{\top}w-(w^{\top}Q^{\top}w)1
$$
then
$$
r = (I_n-W)^{-1}\left(Q^{\top}w-(w^{\top}Q^{\top}w)1\right)
$$
Here
$$
\cases{
1^{\top} = (1,\cdots,1)\\
I_n = \text{Identity matrix}\\
W = (w,\cdots,w)^{\top}
}
$$
Comment: $I_n - W$ will not be invertible, you are equivalent solving \begin{equation}
\begin{bmatrix}
I_N & \mathbf{1}_{N\times 1} \\ w^\top & 1
\end{bmatrix}\begin{bmatrix}
r\\ \lambda
\end{bmatrix} = \begin{bmatrix}
Qw\\ w^\top Q w
\end{bmatrix}
\end{equation}, which is rank deficient
Comment: Invertibility depends on the **size** of $W$. If $\|w\| << 1$ is invertible for sure.
Comment: Thanks, but I don't know how to verify your answer, because there is not inequality constraints, how can I ensure that abs(w) is much smaller than 1?
Comment: This is an extreme example. The identity matrix being dominant, $I_n-W$ will be invertible. Ill conditioning in the inversion may appear as the identity matrix is not dominant, but even in this case,$I_n- W$ may be numerically invertible.


large group of people ($>365$), what is probability that no one has a birthday today

Question: What is the probability of a large group of employees, say $2,728$, and no one having a birthday today? and, if there are $254$ actual hiring days in a year, what is the probability that also no one was hired on this date? This actually occurred at my friends company today and we are curious.
Comment: Can we make the (faulty, but not egregiously so) assumption that a person's birthday is equally likely to fall on any day of the year?
Answer: The probability that the first employee does not have a birthday today is
$$
P_1 = \frac{364}{365}
$$
in a normal year (and $\frac{365}{366}$ in a leap year).
The probability that no one has a birthday is the product of all these. If $N$ is the total number of employees, then it is:
$$
P = \prod_{n=1}^N P_n = \left(\frac{364}{365}\right)^N
$$
With $N=2\,728$ you get $P = 0.0005618758$.
So if the employee count stays constant, then what you describe happens on one day in about $2\,000$ on average.
EDIT: The question about the hirings is a bit more complicated: Imagine that there is only one hiring day per year, and it's a different day each year. This means that in 2013, maybe hundreds of people are hired on October 23, in 2014 maybe dozens on March 23 etc. In this case, the probability that no one has his anniversary today is maybe zero, if today is a day that has never been a hiring day.
With 254 hiring days, the distribution is much more constant, but there are other difficulties, for example a special day like March 26 (among others), which is one of the most common Good Fridays. This day is a free day in many countries, so the probability that it is a working day is lesser than that of other days, so fewer people have been hired on that day than on other days.
So, the answer on the hirings question depends on how the hiring days are distributed among the year.
For simplicity, let's assume that the hirings are distributed in a regular pattern over the year. For example, Monday-Friday are hiring days (even on holidays), Saturdays and Sundays are not. Then the probability that no one has his hiring day anniversary is the same as above:
$$
P = \left(\frac{364}{365}\right)^N
$$
Another simple case is if the hirings are always on the same dates. For example, on the first 20 days of a month, people are hired, then for the rest of the month they're not. In this case the probability is what you expected, if the total number of hiring days is $254$:
$$
P = \left(\frac{253}{254}\right)^N
$$
On non-hiring days, the probability is $P=1$ always, as no one was hired that day anyway.
I have to add that in my calculation of birthday probability, I assumed that birthdays are evenly distributed. This is not the case in reality. As far as I know, there are more people with birthdays in Summer, due to the fact the people live closer to each other in Winter (at least in northern countries).
Comment: Thank you! Also, the hiring question was for those who have already been hired, and would have an "anniversary" of their hiring on that day.
Comment: I suppose the hiring days are not always the same, but they depend on the distribution of Mon-Fri vs. Saturdays/Sundays?
Comment: well, wouldn't it be enough to know that there are 254 days of the year that "hiring days"? So, by the logic of the first answer, would it be (253/254)^2728 and then times that by previous answer?
Comment: which i get 0.0000212 for the hiring day anniversary probability, times it by the other one (or should I add) I get 1.19 x 10^-8, and that is a chance of 1 in 83,950,721?
Comment: Yes, if you have the two probabilities (no birthday, no anniversary), then you can multiply them to get the probability that both conditions are true.
Comment: Has your question been fully answered?
Comment: Could you clarify what ``"one day in about 2000"`` mean? If employees are employed for life and it hasn't happened after 365 days, then it never will. If the turnover is 100% in that everyday starts with a new set of employees, then we have to wait 2000 days (samples) before it happens? Thanks.


Machine learning for discovering formulas

Question: I have a vector of desired values that I want to fit based in some generated predictors. The tricky part is that I wish to have the explicit formula. For example, giving the below input I would like to get the following formula
Input:
Y=[1,2,3,6,7]
X=[1,2,3,4,5;0,0,0,1,1]

Output:
Transf={'multiply',1,'transform',linear;...
'multiply',2,'transform',linear}

Y=Transf(X) %which would be equal to: Y=1*X(1,:)+2*X(2,:)

This is a not so complicated linear regression. but generalizing it could give, log transforms, sin, cosin, roots, exponents.
What software  could I start with? Or how might I implement this?
Comment: Did you see [Eureqa](http://www.nutonian.com/products/eureqa/)?
Comment: @Aksakal I did not known it existed as I had not been able to formulate the search. It seems it is called symbolic regression. I will try it and come back to this point if it makes sense.
Comment: @Aksakal apparently some progress has been made towards a free alternative: http://scicomp.stackexchange.com/questions/14818/seeking-a-free-symbolic-regression-software
Comment: Could you clarify what it is that you mean by "explicit formula to access to what extent it makes sense"? Makes sense in what way? And what is it that you want to generalize to give logs, exponentials, etc?
Comment: @rocinante I edited the question. I hope is more clear now.
Comment: for example one idea would be to define every transformation allowed and run a GA to get the best combination. yet, I would guess this is really non-efficient
Comment: @Aksakal I tried eureqa which is a solution to my problem. yet I had some problems constraining the size of the solution. I will give it a go again in the next few days.
Answer: In addition to nice answer by @DankMasterDan (+1), I would like to share further information on the topic. It seems that an approach that you're looking for is symbolic regression. It seems to be closely associated with and usually is implemented via evolutionary algorithms, such as the most popular genetic programming (GP). However, other approaches are also proposed, especially in relation to analytical models of physical systems. For example, see a paper by Shmidt and Lipson (2009), published in Science. By the way, this small open source Java-based project presents an implementation of the above-mentioned approach.
In terms of software, available for symbolic regression, before I concentrate on my favorite open source solutions, I'd like to mention that Eureqa is definitely an interesting product, which has grown from an open source project. However it is quite expensive, as many commercial statistical or machine learning solutions, available on the market today.
I will start a brief review of open source solutions with a hybrid solution GPTIPS, which is an open source plug-in software for commercial MATLAB. It is referred to by authors as a "symbolic data mining platform for MATLAB".
Now, turning to a full open source software, we can find several IMHO very interesting solutions. A well-known language-agnostic (but still Python-based) system SageMath offers symbolic regression functionality via SymPy Python library, which can also be used independently as well. Another very interesting comprehensive open source software system is .NET-based HeuristicLab. While HeuristicLab is labeled "a framework for heuristic and evolutionary algorithms", it offers a much wider range of functionality beyond symbolic computations and evolutionary/GP solutions.
In addition to already-mentioned SymPy libarary, Python ecosystem offers DEAP open source project, where DEAP abbreviation refers to Distributed Evolutionary Algorithms in Python.
My brief analysis of open source software for symbolic regression and related solutions would be incomplete without mentioning what my favorite R ecosystem offers in that regard. An interesting R package for GP and symbolic regression is rgp (available on CRAN), which is referred to as "R genetic programming framework" (RGP). The RGP package is a part of a larger set of open source tools for symbolic computation in R, developed under the umbrella of a larger Rsymbolic project. There are also several optimization-focused GP packages (http://cran.r-project.org/web/views/Optimization.html), however it is highly unlikely that they offer symbolic regression functionality out-of-the-box, as RGP package does.
References
Schmidt, M., & Lipson, H. (2009). Distilling free-form natural laws from experimental data. Science, 324(5923), 81–85. doi:10.1126/science.1165893 Retrieved from http://creativemachines.cornell.edu/sites/default/files/Science09_Schmidt.pdf
Comment: this answer provides a good overview of the solutions to the problem above. I did tried Eureqa which is definitely a solution for my problem (yet did not really gave what I wished). As I am a MATLAB programmer I will try the GPTIPS to see whether I have more flexibility in defining constraints
Comment: @ASantosRibeiro: I'm glad you liked my answer and found it helpful. Good luck with GPTIPS!
Answer: It sounds like your goal is to find an arbitrary function g(:) such that y=g(x).
The answer to your question depends on what you mean by explicit. Specifically, you should note that there are an infinite number of ways to specify any function g(:), for example g(x)=x on -1

If by 'explicit' you mean find a a g*(:) which is functionally equivalent to g(x) - ie gives the same output of g(x) for all values of x then yes, there are many ML algorithms which can do this for arbitrary functions g(:). These algorithms could give you perfect predictive power from x to y, albeit at the possible expense of overfitting the data. They include SVM with RBF kernels and decision trees (ie any algorithm w/ infinite VK dimension).
However, if by explicit you mean the SIMPLEST g(:) which describes y=g(x), then things get much more complicated because simplicity/sparsity/complexity is a human notion which can only be quantified very clumsily.  For example, in theory, SVM regression w/ rbf kernel could perfectly fit y=sin(x), but it wouldn't output g=sin(:) but rather an uninterpretable series of coefficients, and you would have to do the work to piece them together into 'sin(x)'.
Now done with that theoretical mumbo-jumbo, I think a good way of getting started is to fit the the data with a taylor expansion as this is at least somewhat interpretable.
Good luck, hope this helps!
Comment: definitely a number of functions will be able to match y. yet, if one constrains the length of the solution less options will be available. what i meant is what is described as symbolic regression.
Answer: I strongly prefer Mathematica for such tasks.  Given data of the form $mydata = \{ \{ x_1, y_1 \}, \{ x_2, y_2 \}, ... \}$, one searches for the unknown parameters such as $a$, $b$, $c$, and $d$ in a non-linear function of your choice in this way:
NonlinearModelFit[mydata, {a + b x + c x^2 + d Sin[x]}, {a, b, c, d}, x]

The symbolic output is
FittedModel[2.8 + 1.075 x + .292 x^2 + 4.9 Sin[x]]

A plot of that nonlinear model and the data looks like this:

You can put in as many basis functions as you like, and some may be "fit" with $0$ coefficients and such.  Of course, too, you need more data points than free parameters for your fit to be meaningful.
Comment: The student version of *Mathematica* isn't too expensive.  My view is that if a software tool makes you 10% more efficient than another (and *Mathematica* makes me 50% more efficient than other tools), then it is worth it.  But of course, everyone has different values and desires.
Comment: One always approaches such a problem with a prior (explicit or implicit) about the basis functions one wishes to consider.  Are you really accepting as candidates Ackerman's function, or Hypergeometric function, the inverse of the Jacobi Elliptic Function, the derivative of the even Mathieu function, and so on?  Even genetic programming methods require the modeler to provide a set of basis functions.  In the approach I outlined, you can specify a larger set than will ultimately be found, and hence is hardly "trivial" at all.  In fact, it is the technique used by experts.
Comment: _Mathematica_ is definitely a nice piece of software. However, it's rather expensive. For those, who can't afford it, is cost-conscious or simply prefer open source solutions :-), _SageMath_, which, among other options, I've mentioned in my answer, seems to be a reasonable alternative.
Comment: I don't pay much attention to student versions of software, as their lower prices are just a fast road to a vendor lock-in. However, I understand your point and agree with you on this. Sometimes, it makes sense to prefer less open solutions, if they offer enough benefits versus alternatives.
Comment: I think this is solution misleads the reader. Any routines that solves a non-linear least-squares curve fitting problem will do. We did not *discover* anything; eg. why not use `log(x)` or `x^3`. We simply defined a small optimization problem to solve, `R` 's `nls` would do the same, `MATLAB`'s `lsqnonlin` would do the same, `scipy`'s `curve_fit` would do the same. No formula discovery here...
Comment: although i appreciate the idea, in your solution you are already defining the model, and are just looking for the coefficients. that is a rather "trivial" approach. the idea here is that we wish to understand which model it is and its coefficients.
Comment: @ASantosRibeiro One can put in as many basis functions as one likes, and if the data doesn't support significant contribution of a particular basis component, then the fit will return 0 (or $,\epsilon$) for that component.  In this way one prunes away from a large function set the irrelevant components.  For the "objectors" to this approach:  State a method *even in principle* that would find (or deduce) a model that wasn't implicitly in the model database ahead of time.  I do not believe you can.  Certainly not the genetic programming approach.  Once you realize this, the issue is closed.
Comment: @David G. Stork The problem here I would say is that you need to define every single component of the model. Lets say that you have a square function. your model will never be able to model this as you would need to define infinite number of sines such that the model would be able to fit every one. yet in a GA, if you allow it to use sines (note here that you did not define how many), it will keep adding then and reducing the error. Even in the example you gave, if you had one higher frequency sin there, you would never be able to fit it properly (without the user knowing it existed).
Comment: @ASantosRibeiro  You are right.  You must express--explicitly or implicitly--the bases for your fit.  In some cases that can be done simply (all polynomials of the form $\sum a_i x^i$).  But there is simply no other way.  What other way can you even *imagine*?
Comment: as said before you can define general basis functions that genetic algorithms can use (for example you define  a*sin(x)). by evolution processes different combinations will be tested, this mean in one iteration you can have 1*sin(10), in another 1*sin(10x)+1*sin(100x), and in another 1*sin(10x)+3*sin(100x)+5*sin(37x). here we defined only a*sin(b*x) and the method returned 3 combinations of sines that approximate the function. in the approach you showed we would need right at the beginning saying that the model was composed of a1*sin(b1*x)+a2*sin(b2*x)+a3*sin(b3*x).
Comment: in truth the best approach is probably the one that combines both these approaches. you first choose a model based in GA, then optimize it,then determine a new model based in evolution processes.
Comment: @ASantosRibeiro the GA approach you mention is not fundamentally different from the basis set approach.  You have "available" `Sin[a x]` for different *a*s and continually increase the number of *a*s.
Comment: @DavidG.Stork this thread is starting to be too large so I will just reply once more. you do also have basis functions as discussed. yet the chosen of the next model is entirely different. in your case if you had 2 models entirely different yet with a reasonable fit how would you join both? you would give all the basis of one and another and start from scratch? GA does has a set of better rules for that, so it does not start from the beginning in every turn. Yes you could join your models differently and improve, but now it starts to be a bit different from the presented solution.
Comment: @ASantosRibeiro Yes.  This discussion is extending too long so I too will reply just once more.  I see no substantive difference between traditional model fitting using a basis set yielding a low coefficient to a component and a GA that includes the same component "late" in evolution (and, presumably, with a small coefficient).  It isn't clear that fewer components means "more interpretable" model.  A traditional model consisting of ten harmonic components may be easily interpretable (e.g., acoustics) while a GA gives just 3 very complicated, unrelated components.  Which would you prefer?


rotating a circle around axis

Question: When I imagine a circle($x^2+y^2=1$) and rotate it around the x axis,
I get a shape of sphere.
So my question is why $(2πr)\timesπ$ and $(πr^2)\timesπ$ do not represent the area and volume of sphere.
I see that there are overlapped points especially in case of the volume,
but still resulting values are too far away from the values obtained using spherical coordinates system.
What am I missing?
Comment: For the surface area of the sphere, consider a point along the perimeter of the original circle. The distance that it travels around the axis of rotation depends on where the point originally is. It could be that it doesn't move at all during the rotation, or it could make a large circle. As you can intuitively see from this, you need to use integration to derive the surface area of the sphere.
Comment: By the way, if you want to make a sphere from  $x^2 + y^2 = 1$, maybe you want to rotate around the $z$-axis, no? Otherwise it doesn't change at all.
Answer: In this way we have some overlapping which cause an overestimation of the volume.
The correct method to use this idea is based on Pappus's Centroid Theorem for the surface and volume
Comment: Ah!!!!Wonderful as usual!!!!Greetings.
Comment: @PeterSzilas Credits to Pappus, a very clever boy! :)
Comment: https://en.m.wikipedia.org/wiki/Pappus_of_Alexandria .Greetings.
Comment: In case of the surface area of sphere, I think (2πr)×π is the overestimated value as you also mentioned about V. Correct value would be 4πr^2. Problem is,  4πr^2 is greater than overstimated value (2πr)×π when r=5. Why is it so?

rotating a circle around axis


Deriving four equally-sized bins for student papers in R

Question: I regularly teach a class with approximately 160 students in it.  I need to return graded papers back to the students as efficiently as possible.  I split up the papers into a set of four approximately even-sized bins.  I typically use the four corners of the room to do this; one bin per corner to reduce traffic problems and therefore get the papers back rapidly.
The problem is knowing how to split the bins as equivalently as possible.  This varies a bit because the distribution of the first letter of the last names is 1), not uniform throughout the alphabet, and 2), the empirical distribution changes, slightly, from semester-to-semester.  A set of bins for one semester might be A-D, E-H, I-K, and L-Z, but the start and end points could change (except, of course, for A in the first bin and Z in the last bin) each semester.  Note that the bin A-D, for example, would include students with the first letter of their last names for "A", "B", "C", and "D".  Moving the letters out-of-sequence can't be allowed.
Does someone have an idea on how to use R (I use R for everything else in class and the student data is in R) to generate the most efficient (i.e., approximately equally-sized) start and end points of each of the four bins each semester?
I thought perhaps that using "expand.grid" might help to generate all possible combinations of bins and recursively work backwards by eliminating incorrect bins.  I thought about a strictly algorithmic approach with a set of nested loops that not only builds the various bin combinations but also checks the bin sizes with cumulative sums too.  I also thought this might be a variant of a Integer Programming/Constraint Logic/knapsack program for an LP solver.
Any ideas?
Thanks
Comment: Here's a vector of 8-letter nonsense names we can use for test data: `set.seed(123); who = apply(matrix(sample(letters,160*8,TRUE),ncol=8),1,paste0,sep="",collapse="")`
Comment: Is it completely necessary to have a bin's boundaries defined by only the first letters? For example, you could divide up the class by four and have each bin defined by the students' names that occupy the boundaries. And if everyone's last name is Smith, could you just assign each student a bin that they'll go to every time?
Answer: Using this bunch of odd-named students:
set.seed(123); who = apply(matrix(sample(letters,160*8,TRUE),ncol=8),1,paste0,sep="",collapse="")

First get the first letters of their names:
fl = substr(who,1,1)

The cumulative sum of the table of first letters:
cs = cumsum(table(fl))

looks like this:
> cs
  a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t
  3   8  15  25  29  35  44  48  56  62  71  79  86  89  95 100 105 114 118 123
  u   v   w   x   y   z
130 135 138 148 155 160

Which tells us that the 40th student in alphabetical order starts with F, the 80th with L, the 120th with S. We can get those like this:
> names(tail(which(cs <40) ,1))
[1] "f"
> names(tail(which(cs <80) ,1))
[1] "l"
> names(tail(which(cs <120) ,1))
[1] "s"

So our report piles are labelled A-F, G-L, M-S, and T-Z
Converting letters to numbers and cutting the first letter vector into ranges defined by those intervals (again converted to letter numbers) gives us the number of student running to each corner:
> table(cut(as.numeric(factor(fl)),c(0,7,12,19,27)))

  (0,7]  (7,12] (12,19] (19,27]
     44      35      39      42

Comment: This looks nice.  Are the cut points fully inclusive, or are some letters inadvertently distributed across more than one bin?
Comment: The letters printed by the `names(tail....` lines, plus "z", are the right inclusive limits of the categories. All names A-F are group 1, all G-L are group 2, all M-S are group 3, all T-Z are group 4. The finds the 40th (80th, 120th) student in alphabetical, and decides that their first letter is the cutoff letter. That might include more than 40 students.
Comment: Note this fails if, for example there are 42 students starting with "A". Because then the first letter of the second group will also be A... As long as you don't have more than 40 students in a letter I don't think anything like this can happen.
Comment: Spacedman, thanks for your insight, code, and documentaton.  I put it into working production today.  Again, thanks.


Use input pipe and manage keyboard

Question: I try to run a PHP script which read a flow by incoming pipe, and I want to manage the keyboard too : cat /etc/passwd|./readSlow.php
The script will read each character from pipe and display it slowly on stdout until the user press the 'q' key (without pressing RETURN).
Actually, the script read the pipe and display the text slowly. But it doesn't stop when I press the 'q' (it display it and I need to press the RETURN key after the 'q' to activate the stop).
#!/usr/bin/php
<?php
// ReadSlow

// This tool take a file in input and read it, character by caracter to the
// output.
// It add a sleep time between each character.

// Wait 0.2s between each char
$defaultSpeed = 0.5;

if (($input = fopen ("php://stdin", "r")) === false)
  die ("Can not open stdin\n");
if (($keyboard = fopen("/dev/tty", "r")) === false)
  die ("Can not open /dev/tty\n");
stream_set_blocking($keyboard, false);

$speed = $defaultSpeed;

while (($char = fgetc ($input)) !== false)
{
  if (($key = fgetc ($keyboard)) === "q") // get input from keyboard
    break;
  usleep ($speed * 1000000);
  echo "$char";
}

fclose ($input);

At the end I would like to have more keys available.
Question : How should I mix the pipe in stdin and the keyboard (to not display the pressed key, to not waiting the return press) ?
I try to play with "stty", but each time, I have "stty: 'standard input': Inappropriate ioctl for device"
I am on Linux, with Bash if it change something.
Thanks
Answer: The problem is that stdin and tty are the same input channel. If you can get away without a pipe, try below code via "./readSlow.php /etc/passwd". This takes the file to print  as a command line argument. The command "stty -echo" suppresses the output of the pressed keystroke.
    #!/usr/bin/php
    <?php
    // ReadSlow

    // This tool take a file in input and read it, character by caracter to the
    // output.
    // It add a sleep time between each character.

    // Wait 0.2s between each char
    $defaultSpeed = 0.5;

    if (($input = fopen ($argv[1], "r")) === false)
      die ("Can not open $argv[1]\n");
    if (($keyboard = fopen("/dev/tty", "r")) === false)
      die ("Can not open /dev/tty\n");
    stream_set_blocking($keyboard, false);

    system("stty -echo");

    $speed = $defaultSpeed;

    while (($char = fgetc ($input)) !== false)
    {
      if (($key = fgetc ($keyboard)) === "q") // get input from keyboard
        break;
      usleep ($speed * 1000000);
      echo "$char";
    }

    fclose ($input);
    echo "\n";
    system("stty echo");

Comment: This is incorrect; _usually_ stdin is the same as tty, but when stdin has been redirected (in this case by a pipe) /dev/tty is still the tty. Also `stty -echo` is not enough to get character-at-a-time input.
Comment: The -echo/echo remove the key displayed but must be linked to raw/-raw to catch the key without pressing the RETURN key. Thanks a lot
Comment: The call to _stream_set_blocking($keyboard, false);_ for the tty catches the pressed key without having to press the RETURN key.
Answer: You were close. Terminal/console aka 'tty' devices on Unix by default work in line-at-a-time mode, to allow you an opportunity to correct small typing mistakes. You need to turn this off and stty is a good way to do so -- but stty operates on its stdin, and you have redirected stdin (with the pipe) so you need to redirect it back for stty. Try system("stty </dev/tty raw") at the beginning of your program and similarly but with -raw at the end.
Comment: And I add the -echo/echo to not display the typed key ! Awesome ! Thanks a lot

Use input pipe and manage keyboard


$\int\limits_0^4\int\limits_\sqrt x^2 \dfrac x{1+y^5}\ dy\ dx$

Question: How to find
$$
\int_0^4\int_\sqrt x^2 \dfrac x{1+y^5}\ dy\ dx=\ ...?
$$
The correct answer from my textbook is $\frac1{10}\log33$. I can answer it if the integrand is  $\dfrac x{1+y^2}$ so I can let $y=\tan t$ but not for $\dfrac x{1+y^5}$.
This is my homework with 10 questions but I can only answer 8 questions, two of them I'm stuck. Can anyone help me for this one? Thanks.
Answer: The region of integration, $0 \le x \le 4$ and $\sqrt x \le y \le 2$, can also be defined as $0 \le y \le 2$ and $0 \le x \le y^2$. If you change the order of integration, using this new definition as your limits, you will find that everything falls out nicely.
Comment: can you elaborate a bit more?
Comment: i get it now, thank you


How to generate an XML feed?

Question: I am creating an XML feed (domain.com/content_type/feed).
I looked into RSS Publishing and have turned it off, but do see where I can generate an XML feed: Home » Administration » Configuration » Web services
Is this the correct way?
Answer: You can do it using the Views Data Export module (Most popular)

This module is designed to provide a way to export large amounts of
  data from views. It provides a display plugin that can rendered
  progressively in a batch. Style plugins are included that support
  exporting in the following types:

CSV
Microsoft XLS (or really simple HTML understood by Excel)
Microsoft DOC
Basic TXT
XML

Example:

Alternatively, you can also do it using the Views Datasource module

Views Datasource is a set of plugins for Drupal Views for rendering
  content in a number of shareable, reusable formats based on XML, JSON
  and XHTML. These formats allow content in a Drupal site to be easily
  used as data sources for Semantic Web clients and web mash-ups. Views
  Datasource plugins output content from node lists created in the
  Drupal Views interface in a variety of formats - XML data documents
  using schemas like OPML and Atom, RDF data documents using a
  vocabulary like FOAF, JSON data documents in a format like Exhibit
  JSON, and XHTML data documents using a microformat like hCard.

How to generate an XML feed?


Wrapping column name of pandas dataframe

Question: I would like to wrap the column names of a data frame to some specific width. For instance, transform
   aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
0                       -1.526078                        0.258110
1                        0.263490                       -0.268635
2                       -1.264121                       -1.257454

to something like
   aaaaaaaaaaaaaaaaa  aaaaaaaaaaaaaaaaa
    aaaaaaaaaaaaa      aaaaaaaaaaaaa
0      -1.526078         0.258110
1       0.263490        -0.268635
2      -1.264121        -1.257454

There is a solution to this problem in this stackoverflow post, but it only works in jupyter notebooks, not in regular python.
Code to reproduce the dataframe:
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(3, 2), columns=['a' * 30] * 2)
print(df)

Comment: This is the question I mentioned in my post. The answers provided in this thread work only for jupyter notebooks.
Comment: did you check this: https://stackoverflow.com/questions/43092345/wrapping-column-names-in-python-pandas-dataframe-or-jupyter-notebooks
Answer: You could wrap the DataFrame in your own class which would override the __str__ and/or __repr__ methods to achieve this.


extract csv content from an xml tag

Question: i have a source xml as below
<rest-adapter-response>
  <metadata>
    <status>success</status>
  </metadata>
  <status-line>
    200
    <reason>OK</reason>
  </status-line>

    <Cache-Control>private, max-age=0</Cache-Control>
    <Transfer-Encoding>chunked</Transfer-Encoding>
    <Content-Type>application/octet-stream</Content-Type>
    <Expires>Thu, 25 Apr 2019 08:51:55 GMT</Expires>
    <Last-Modified>Fri, 10 May 2019 08:51:55 GMT</Last-Modified>
    <Server>Microsoft-IIS/10.0</Server>
    <X-SharePointHealthScore>1</X-SharePointHealthScore>
    <X-SP-SERVERSTATE>ReadOnly=0</X-SP-SERVERSTATE>
    <DATASERVICEVERSION>3.0</DATASERVICEVERSION>
    <X-Download-Options>noopen</X-Download-Options>
    <Content-Disposition>attachment</Content-Disposition>
    <SPClientServiceRequestDuration>224</SPClientServiceRequestDuration>
    <X-AspNet-Version>4.0.30319</X-AspNet-Version>
    <SPRequestGuid>de31db9e-70cb-8000-7fba-6c3e85d9c810</SPRequestGuid>
    <request-id>de31db9e-70cb-8000-7fba-6c3e85d9c810</request-id>
    <MS-CV>ntsx3stwAIB/umw+hdnIEA.0</MS-CV>
    <Strict-Transport-Security>max-age=31536000</Strict-Transport-Security>
    <X-FRAME-OPTIONS>SAMEORIGIN</X-FRAME-OPTIONS>
    <X-Powered-By>ASP.NET</X-Powered-By>
    <MicrosoftSharePointTeamServices>184.108.40.20624</MicrosoftSharePointTeamServices>
    <X-Content-Type-Options>nosniff</X-Content-Type-Options>
    <X-MS-InvokeApp>1; RequireReadOnly</X-MS-InvokeApp>
    <P3P>CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI"</P3P>
    <Date>Fri, 10 May 2019 08:51:55 GMT</Date>

  <message-body>
    <non-xml-data-response>COA,COA Acct Desc,Acct Prefix,Revaluation Acct,Mapping Changes - Additions( A )  Deletions ( D ) Changes ( C ),MJE,OIM Recon,Comments for difference:10000274,"Citibank, Operating, USD, 31165975",1000,10009999,A,,X,10000374,"Citibank, Clearing, USD, 31165975",1000,10009999,A,,X,10006604,"HSBC, Operating, SAR, SA0345000000003179660002",1000,10009999,A,,X,10006605,"Citibank, Operating, ZAR, 0202099009",1000,10009999,A,,X,123,,,456,,,,</non-xml-data-response>
  </message-body>
</rest-adapter-response>

above XML is the response of a share point web service which tried to read a csv file and gave a response like this. as you can see in the above response xml , csv data did come but inside one xmltag called <message-body> , and also lost the new line after every row format!!
now i need to recreate the csv !!. and the worst part is , in the tool where i receive this format i have capability to write xslt and xml ! no hosting language code or libraries could be used.
also only xslt 1.0.
there is a question like this
question on creating csv from xml , but this is bit different from my requirement . am just learning xslt and xpath , can any one help me in this ?
below is the requested output :
click here to view the csv format
Comment: Do you have a specific ruleset when to make linebreaks? It would be useful to know what you need the csv for. Maybe we can find another solution without recreating the csv. E.g. it may be enough to access every comma separated value by itself. This could be done by a for-each over tokenize(.//non-xml-data-response, ',')
Comment: yea i get your point, my requirement is to pass the entire csv file in this case . so we need  not concentrate on the csv values induvidually.
how ever the larger problem for me (by now you would have guessed) is to retrieve a csv file from the share point via web services.

[link](https://sharepoint.stackexchange.com/questions/262992/retrieve-csv-file-data-using-sharepoint-webservices-with-foramt)

this link  can explain  the larger problem. **but for now as i said i need the whole csv file**
Comment: Huuuh. I do not think that there is a ruleset which we can use to produce linebreaks which would work on every csv document. Is the csv document consistent in its structure and is that all the data it can contain? If yes we could build a ruleset like: Put a linebreak in front of the first big number. After that we can put a linebreak after 7 commas which are not in quotes. Would this work for you?
Comment: i think we need not worry about the "big number" , every seven commas , if we are able to insert a line break  then we are good. that will create all the rows including header wont it ?
Comment: How would you approach this task if doing it manually? Unless you have some way of determining how many separate values are in each row, this problem is insolvable.
Comment: Ugh so you just want a table out of it? Thats kind of simple to group every 7 elements. Should I write the code to transform the csv into an xml where the first 7 entries are the nodenames and every further entry gets into the corresponding tag like: 21312"some text" ......?
Answer: If one makes the assumption that the header row is terminated by a colon and that there are 7 values in each data row*, then it's possible to use the following stylesheet:
XSLT 1.0
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="text" encoding="UTF-8"/>

<xsl:template match="/rest-adapter-response">
    <xsl:variable name="csv" select="message-body/non-xml-data-response" />

    <xsl:value-of select="substring-before($csv, ':')" />
    <xsl:text>:
</xsl:text>

    <xsl:call-template name="restore-csv">
        <xsl:with-param name="text" select="substring-after($csv, ':')"/>
    </xsl:call-template>
</xsl:template>

<xsl:template name="restore-csv">
    <xsl:param name="text"/>
    <xsl:param name="i" select="1"/>
    <xsl:choose>
        <xsl:when test="contains($text, ',')">
            <xsl:variable name="value">
                <xsl:choose>
                    <xsl:when test="starts-with($text, '"')">
                        <xsl:text>"</xsl:text>
                        <xsl:value-of select="substring-before(substring-after($text, '"'), '"')"/>
                        <xsl:text>"</xsl:text>
                    </xsl:when>
                    <xsl:otherwise>
                        <xsl:value-of select="substring-before($text, ',')"/>
                    </xsl:otherwise>
                </xsl:choose>
            </xsl:variable>

            <xsl:value-of select="$value"/>
            <xsl:choose>
                <xsl:when test="$i mod 7 = 0">
                    <xsl:text>
</xsl:text>
                </xsl:when>
                <xsl:otherwise>
                    <xsl:text>,</xsl:text>
                </xsl:otherwise>
            </xsl:choose>

            <xsl:call-template name="restore-csv">
                <xsl:with-param name="text">
                    <xsl:choose>
                        <xsl:when test="starts-with($text, '"')">
                            <xsl:value-of select="substring-after(substring-after($text, '"'), '",')"/>
                        </xsl:when>
                        <xsl:otherwise>
                            <xsl:value-of select="substring-after($text, ',')"/>
                        </xsl:otherwise>
                    </xsl:choose>
                </xsl:with-param>
                <xsl:with-param name="i" select="$i + 1"/>
           </xsl:call-template>
        </xsl:when>
        <xsl:otherwise>
            <xsl:value-of select="$text"/>
        </xsl:otherwise>
    </xsl:choose>
</xsl:template>

</xsl:stylesheet>

Applied to your input example, the result will be:
Result
COA,COA Acct Desc,Acct Prefix,Revaluation Acct,Mapping Changes - Additions( A )  Deletions ( D ) Changes ( C ),MJE,OIM Recon,Comments for difference:
10000274,"Citibank, Operating, USD, 31165975",1000,10009999,A,,X
10000374,"Citibank, Clearing, USD, 31165975",1000,10009999,A,,X
10006604,"HSBC, Operating, SAR, SA0345000000003179660002",1000,10009999,A,,X
10006605,"Citibank, Operating, ZAR, 0202099009",1000,10009999,A,,X
123,,,456,,,

This may need more work to handle possible escaped double-quotes within quoted values.

--
(*) The strange thing here is that there are 8 values in the header row, but only 7 in the data rows.
Comment: Great effort and guessing skills!
Comment: i just got a chance to verify the result pls wait. i will verify and close this thread
Comment: the transformation is working fine. but my tool accepts any result which is inside a n xml tag. so i inculded a tag ``  just after this ``  . also in cluded a `` just after ` ` but in the result am not still getting this result tag . my result should be something like below

`
csv content .......
`
Comment: Try changing the output method to `xml`.

extract csv content from an xml tag


Show that the derivatives of a $C^1$ function vanish a.e. on the inverse image of a null set

Question: Let $A \subset \mathbb{R}$ be such that $\lambda(A) = 0$, where $\lambda$ is the Lebesgue measure on the real line.
Let $\Omega \subset \mathbb{R}^N$ be an open set and let $u \colon \Omega \to \mathbb{R}$ be in $C^1(\Omega)$. Then, for every $i = 1, \dots , N$, we have that $$\frac{\partial u}{\partial x_i}(x) = 0$$ for $\lambda^N$-almost every $x \in u^{-1}(A)$. (where $\lambda^N$ is of course the lebesgue measure on $\mathbb{R}^N$)
Does anyone know a proof of this fact?
The lecture notes continue saying that since every singleton has Lebesgue measure $0$ in $\mathbb{R}$, we have that $$\frac{\partial u}{\partial x_i}(x) = 0$$ for $\lambda^N$-a.e. $x \in \Omega_c$, where $\Omega_c := \{x \in \Omega : u(x) = c\}$.
Comment: Something's very wrong here. If we choose a set $A$ that's contained in the set of regular values of $u$, then it fails.
Comment: since on the preimage of that set the gradient has to be surjective and hence it can't be zero, right? I don't know what to say, I probably messed up something taking notes during the class. I'll ask my professor for clarification! thanks for pointing out that!
Comment: Dear professor Shifrin, my professor uploaded the lecture notes ad I double checked. The result is exactly the same I posted a week ago, I edited the original post adding $\lambda^N$ just to be more clear. I also added the part starting with "The lecture..." which is what I actually need, but it is nothing more that a direct consequence of the first statement, which is the one I am interested in. Any second opinion on this fact would be very appreciated! thank you!
Answer: The statement is true, below is a proof. You will see from the proof that it works under the assumption weaker than $C^1$, namely, it suffices to assume that $f$ is absolutely continuous on almost every line parallel to a coordinate axis.
I will first give a proof in the 1-dimensional case and then use it to proof the general claim. I will use the notation $|Z|$ to denote the (1-dimensional) Lebesgue measure of subsets $Z\subset {\mathbb R}$. I will also denote by $B_r(z)\subset {\mathbb R}$ the closed ball of radius $r$ centered at $z\in {\mathbb R}$. If $z=0$, set $B_r=[-r,r]$.
We will need
Theorem (Lebesgue density theorem). Let $Z\subset {\mathbb R}$ be a (measurable) subset of positive measure. Then almost every point $z\in Z$ is a density point,  i.e.,
$$
\lim_{r\to 0} \frac{|B_r(z)\setminus Z|}{|B_r(z)|}=0.
$$
Let $A\subset {\mathbb R}$ be a measure zero subset. For a $C^1$-smooth function $f: {\mathbb R}^n\to {\mathbb R}$ set
$$
E=f^{-1}(A), E'= E\cap \{x: df(x)\ne 0\}.
$$
Note that both $E$ and $E'$ are measurable sets.
Lemma. Let $f: {\mathbb R}\to {\mathbb R}$ is a $C^1$-function. Then $E'$ has zero measure.
Proof. Suppose not. Then there exists a density point $z\in E'$. By making affine change of variables in domain and range, we can assume  that $z=0$ and $f'(0)=1$. By continuity of $f'$, there exists $r_0>0$ such that for all $x\in B_{r_0}$,
$$
\frac{1}{2} < f'(x)< 2.
$$
In particular, for every $r\in (0, r_0)$ and every measurable subset $S\subset B_r$,
$$
|f(S)|\le 2|S|
$$
(here is the only place where the assumption that the domain of $f$ is 1-dimensional is needed). Set
$$
E_r'= B_r\cap E', E_r''= B_r \setminus E'.
$$
Since $f(E_r')\subset A$, and $|A|=0$, we get $|f(E_r')|=0$. On the other hand, since $0$ is a density point of $E'$, for every $\epsilon>0$ there exists $0<r_1<r_0$ so that for all $r\in (0, r_1)$
$$
|E_r''|< \epsilon |B_r|.
$$
Thus,
$$
|f(E_r'')|\le 2\epsilon |B_r|.
$$
By compining this with the fact that $|f(E_r')|=0$, we see that, for $r<r_1$,
$$
|f(B_r)| \le 2\epsilon |B_r|.
$$
On the other hand, by the mean value theorem (taking into account that $0.5\le f'(x)<2$ for $x\in B_{r}$,
$$
|f(x)-f(0)|\ge \frac{1}{2}|x|
$$
for all $x\in B_r$. In particular, $|f(B_r)|\ge |B_r|/2$ for all sufficiently small $r>0$. Thus, we obtain:
$$
\frac{1}{2}|B_r|\le |f(B_r)|\le 2\epsilon|B_r|
$$
for all $r\in (0,r_1)$. Taking $\epsilon=1/8$ we obtain a contradiction. qed
Now, we can prove the claim for general $n$.
Theorem.  Let $u: {\mathbb R}^n\to {\mathbb R}$ is a $C^1$-function. Then the set
$$
E'=u^{-1}(A)\cap \{p: du(p)\ne 0\},
$$
has zero measure.
Proof. It suffices to show that for each $i=1,...,n$, the set
$$
E_i'=E\cap \{p\in {\mathbb R}^n: \frac{\partial}{\partial x_i} u(p)\ne 0\}
$$
has zero measure. By renumbering the cooridnates, it suffices to check this for $i=n$. For every $y\in {\mathbb R}^{n-1}$ consider the
line $L_y=\{(y,x): x\in {\mathbb R}\}$ in ${\mathbb R}^n$. Restriction of $f$ to $L_y$ is $C^1$-smooth. Therefore, by Lemma, for each $y$, the set
$$
E_y'= E\cap \{\frac{\partial}{\partial x} f (y,x)\ne 0\}
$$
has zero measure 1-dimensional. Therefore, by Fubini's theorem, the set
$$
E_n'= \bigcup_{y\in {\mathbb R}^{n-1}} E_y'
$$
has zero $n$-dimensional measure. qed
Comment: Thank you for your answer! I hope you don't mind if before accepting it I take my time to read it carefully. :D
Comment: Sure, not a problem.
Comment: Ok, got it, thank you!
Comment: Hi @studiosus. Ho do we adapt your proof to the case where $f$ is only AC. I am trying to solve this problem http://math.stackexchange.com/questions/838804/properties-of-absolutely-continuous-functions Thank you
Comment: As I said, you need absolute continuity on lines (ACL), not just absolute continuity. Do you mean ACL?
Comment: Sorry @studiosus, I forgo to write, my case is in one dimension. I want only the first part of the proof. I have linked my question, did you saw it?
Comment: @Thomas: I see. I will take a look at it.
Comment: lol, I came across my question again, and I am so terribly sorry that I can't upvote this answer again xD
Answer: I'm sorry. This continues to be just plain wrong. Try $u(x) = x_1^2+x_2^2$. The only place both partial derivatives vanish is at the origin. So, for any $c\in\Bbb R-\{0\}$, the gradient of $u$ vanishes at no point of $u^{-1}(\{c\})$.
What Sard's Theorem tells us (at least, if $u\in C^N(\Omega)$) is that almost every $c\in\Bbb R$ is a regular value; that is, for almost all $c$, for every $x\in u^{-1}(\{c\})$ some partial derivative of $u$ is nonzero at $x$.
The other issue I have with your edited version is that for almost all $c\in\Bbb R$ it is the case that $\lambda^N(\Omega_c)=0$. That is, we expect $\Omega_c$ to be a smooth hypersurface, which necessarily has measure zero in $\Omega$.
Comment: I think that the statement given in the question is in fact true. The example given here is not a counterexample because the inverse image of a zero measure set also has zero measure, so anything holds on it in an almost everywhere sense (i.e., it is a vacuous truth).


Cannot invoke findMax() on the array type double[] error. a method that returns the max in an array

Question: I am creating a very simple method that returns the maximum value in an array on Eclipse, and I get this following error: Cannot invoke findMax() on the array type double[]
Here is the code:
 class themax {
private double findMax(double[] array) {
double max = array[0];
for (int index =1; index < array.length;index++) {
    if (array[index]> max)
    {
        max = array[index];
    }
}
return max;
}

 }
 public class maxandmin2 {

public static void main(String[] args) {
    // TODO Auto-generated method stub

    double[] array;
    array = new double[5];
    for(int i=0;i<array.length;i++) {
        array[i] = Math.random()*10;
    }
    array.findMax();
}

}

thanks a lot!
Comment: `themax m = new themax(); m.findMax(array);`
Comment: I replaced array.findMax(); with that, and I get this error : "The method findMax(double[]) from the type themax is not visible"
Comment: oh, I think I just have to change the method to private from public, thanks!
Answer: The method findMax is in the themax class and requires that the array is passed
You can call this method by instantiating a new themax and then calling it.
Change your main to
public static void main(String[] args) {

    double[] array;
    array = new double[5];
    for(int i=0;i<array.length;i++) {
        array[i] = Math.random()*10;
    }
    System.out.println ("The max is " + new themax().findMax(array));
}

note The findMax method should be public
Comment: Thank you! and I just did a quick fix of changing the method to public afterwards  because I got this error: "The method findMax(double[]) from the type themax is not visible"  I appreciate your help!


OpenVPN work in China

Question: Next week, I'm going to China to work, and I'm worried about the connections there. To prevent this, I installed and configured OpenVPN on an EC2 instance in AWS. I'm navigating it right now.
My EC2 is in Virginia and my traffic is masked in California.
Does this mean I can access my gmail, whatsapp, google and etc ...?
Answer: From my personal experience, yes it works and you can access whatever you want if you are routing 0.0.0.0/0 through your VPN.
My company uses OpenVPN extensively and from our experience, you can connect OpenVPN from China but you cannot connect OpenVPN to China.
To illustrate this more clearly -

OpenVPN server in UK, client connects from Shanghai, works perfectly fine.
OpenVPN server in Beijing, client connects from US, cannot connect.

And we have OpenVPN servers in UK, US, HK, SG to confirm this behaviour, which is really weird. I do not understand why the authority would block inbound connection but not outbound one, but that is just how it is.
However if you have the time I would suggest setting up shadowsocks VPN which seems to be the best VPN to bypass detection these days.

OpenVPN work in China


Pandas - Find rolling average of past 3 days

Question: I am trying to find the average of the past 3 days of data for a specific group of elements as shown below. Given below is how my Dataframe looks like:
day, category, sub-category, count
2021-01-01, electronic, phone, 10
2021-01-02, electronic, phone, 12
2021-01-03, electronic, phone, 3
2021-01-04, electronic, phone, 20
2021-01-01, electronic, computer, 15
2021-01-02, electronic, computer, 12
2021-01-03, electronic, computer, 20

Expected output
day, category, sub-category, count, rolling_average_past_3_days
2021-01-01, electronic, phone, 10,
2021-01-02, electronic, phone, 12,
2021-01-03, electronic, phone, 3, 8.3
2021-01-04, electronic, phone, 20, 11.6
2021-01-01, electronic, computer, 15,
2021-01-02, electronic, computer, 12,
2021-01-03, electronic, computer, 20, 15.7

Answer: Try this:
df['rolling_average_past_3_days'] = df.groupby('sub-category')['count'].rolling(3).mean().droplevel(0)

or
df.groupby('sub-category',sort=False).rolling('3D',on='day')['count'].mean().to_numpy()

Comment: thanks for getting back.. I tried the above bu it returns a new column with all NANs
Comment: interesting, it worked for me... when you run just `df.groupby('sub-category')['count'].rolling(3).mean()` does it return what you are expecting (outside your df)
Comment: thats interesting... the only other thing I can think of is to check that your `count` column is a numeric type.
Answer: To be generic that you can have more than one main category beside 'electronic', you need to group by category too.  You need also to use .transform() so that the result get broadcasted back to the original structure so that you can get all entries.  Otherwise, you may get error like: TypeError: incompatible index of inserted column with frame index
You can try this:
df['rolling_average_past_3_days'] = df.groupby(['category', 'sub-category'])['count'].transform(lambda x: x.rolling(3).mean())

print(df)

          day    category sub-category  count  rolling_average_past_3_days
0  2021-01-01  electronic        phone     10                          NaN
1  2021-01-02  electronic        phone     12                          NaN
2  2021-01-03  electronic        phone      3                     8.333333
3  2021-01-04  electronic        phone     20                    11.666667
4  2021-01-01  electronic     computer     15                          NaN
5  2021-01-02  electronic     computer     12                          NaN
6  2021-01-03  electronic     computer     20                    15.666667

The NaN values shown on the first 2 entries of each sub-category is normal since you are rolling averaving over the 3 days and the first 2 days have no value.  If you want to make them blank instead of showing NaN, you can use this:
df['rolling_average_past_3_days'] = df.groupby(['category', 'sub-category'])['count'].transform(lambda x: x.rolling(3).mean()).replace(np.nan, '')

print(df)

          day    category sub-category  count rolling_average_past_3_days
0  2021-01-01  electronic        phone     10
1  2021-01-02  electronic        phone     12
2  2021-01-03  electronic        phone      3                    8.333333
3  2021-01-04  electronic        phone     20                   11.666667
4  2021-01-01  electronic     computer     15
5  2021-01-02  electronic     computer     12
6  2021-01-03  electronic     computer     20                   15.666667


Mobile device access policies

Question: HR wants us to prevent part-time hourly staff from checking email while off the clock.
Is it possble to disable Activesync and OWA via exchange policy for users in a particular
group? I have looked in EAC, but can't seem to find an option for creating these types of rules for mobile devices.
Rather than going into EAC everytime a new user is created disabling those options, it would be easier if they were applied automatically and enabled on a as-needed basis.
Answer: Unfortunately I can't think of a good way to do this for Exchange ActiveSync. The problem you'll run into is that a device which is currently syncing with EAS won't stop just because you assign a new policy to the mailbox-- it will continue to be able to sync until the next time it authenticates. Imagine that Jane is a part-time staffer. She syncs her mailbox at 0800, which establishes an open sync channel. Depending on the particulars of her sync, she may not have to reauthenticate for 12 hours or more, in which case her device will keep synchronizing even after you disable her for EAS.
On the other hand, doing this with OWA is pretty straightforward. See http://technet.microsoft.com/en-us/library/dd351097(v=exchg.150).aspx. You just need to assign the user to the appropriate policy for the time of day, although if the user has an open session it will stay open too.
To fix either of these problems, you could use iisreset to dump all the existing connections on the CAS, but that may also interrupt other users, which might not be acceptable.

Mobile device access policies


Showing that the integral of $f(z) dz$ over the upper semicircle tends to $0$

Question: Define $f(z) = \frac{e^{iz}}{z} $ and given $R > 0$, let $γ_R$ is the upper semicircle of radius $R$ and center $0$.
Show that $\int_{γ_R} f(z)dz → 0$ as $R → ∞$.
(Hint: after a suitable parametrization $γ_R : [0, π] → \mathbb C$, divide the interval of integration in the interval [$\frac{1}{\sqrt{R}}$, $π$ − $\frac{1}{\sqrt{R}}$] and
the rest).
I didn't use the hint because I didn't understand it. Though, I found a parametrization of $γ_R$: $γ_R(\theta) = cos(\theta) + i\sqrt{1-cos^2(\theta)}$ for $\theta \in [0, π]$
Here's my attempt without using the hint:
$\int_{γ_R} f(z)dz = \int_{γ_R} \frac{e^{iz}}{z} dz$ and thus $|\int_{γ_R} f(z)dz| = |\int_{γ_R} \frac{e^{iz}}{z} dz|$
Let $z=x+iy$, then $|\int_{γ_R} f(z)dz| = |\int_{γ_R} \frac{e^{iz}}{z} dz| ≤ \int_{γ_R} |\frac{e^{i(x+iy) }}{x+iy}|dx + idy = \int_{γ_R} \frac{|e^{i(x+iy)}|}{|x+iy|}dx+idy$.
But $|e^{i(x+iy)}| = e^{-y} ≤ 1$ for $y \geq 0$ (i.e in the upper half plane, specifically in the upper semicircle).
And $|x+iy| = \sqrt{x^2+y^2} ≤ R$ for $R$ big enough.
Hence, $|\int_{γ_R} f(z)dz| ≤ \int_{γ_R} \frac{|e^{i(x+iy)}|}{|x+iy|}dx+idy ≤ \int_{γ_R} \frac{1}{\sqrt{R}} dx+idy = \int_{γ_R} \frac{1}{\sqrt{R}} dx + i\int_{γ_R} \frac{1}{\sqrt{R}} dy$
$-(\int_{γ_R} \frac{1}{\sqrt{R}} dx + i\int_{γ_R} \frac{1}{\sqrt{R}} dy) ≤ \int_{γ_R} f(z)dz ≤ \int_{γ_R} \frac{1}{\sqrt{R}} dx + i\int_{γ_R} \frac{1}{\sqrt{R}} dy$
Now taking lim as $R → ∞$, we deduce by the sandwich theorem that $\int_{γ_R} f(z)dz = 0$
Is my attempt correct?
Any other answers? And what about the hint?
Comment: Try to fill in the details for your sandwich theorem argument.  You might run into a problem.
Comment: No. The inequality $\sqrt{x^2+y^2}\le R$ implies $\frac1{\sqrt{x^2+y^2}}\ge\frac1R$, not the other way around.
Comment: @BrianMoehring what kind of problem?
Comment: @DavidC.Ullrich how can it be fixed then starting from my inequalities?
Comment: What makes you think it _can_ be fixed, starting with your inequalities? Most of the inequalities you wrote here are not even wrong, they make no sense, because we're talking about complex numbers.
Comment: The limits you have claimed are zero actually diverge to infinity.
Comment: @BrianMoehring How is that? If $R$ tends to infinity then the integrals tend to $0$
Comment: @DavidC.Ullrich Can you point out to which inequalities make _no sense_?
Comment: You need to actually evaluate/estimate them rather than simply making an unjustified claim.  They are on the order of $R \cdot \frac{1}{\sqrt{R}} = \sqrt{R}$ which tends to $\infty$.  The entire point of the hint is to avoid this problem.
Comment: Oh okay, I see now! How can I avoid this?
Comment: Use the hint.  Split the contour into parts, on one of which we use $e^{-y}\leq 1$ and on the other we use $e^{-y} \leq \phi(R)$ for some function $\phi(R)$ that goes to zero just fast enough to make both estimates tend to zero  (Btw: David Ullrich's point still stands.  I just saw it as more "fixable" than the one I was pointing out)
Comment: Any inequality of the form $z<w$ or $z\le w$ where $z$ and $w$ are complex numbers.  Hence for example, the final three, which sort of seem like the whole point
Answer: Your answer has a few mistakes in it.
$1)$ When you say $$\int_{\gamma_R}\frac{e^{iz}}{z}dz\leq\int_{\gamma_R}\left| \frac{e^{i(x+iy)}}{x+iy}\right|dx+idy$$
You seem to be trying to use the fact that, in general: $$\left|\int_C f(z)\mathrm{d}z\right|\leq \int_C |f(z)||\mathrm{d}z|$$
Notice the $|\mathrm{d}z|$ term above - it is an important part of the inequality. It cannot simply be replaced with $\mathrm{d}z=\mathrm{d}x+i\mathrm{d}y$.
In your question, this lack of care leads to: $$|\int_{γ_R} f(z)dz| ≤ \int_{γ_R} \frac{|e^{i(x+iy)}|}{|x+iy|}dx+idy ≤ \int_{γ_R} \frac{1}{\sqrt{R}} dx+idy = \int_{γ_R} \frac{1}{\sqrt{R}} dx + i\int_{γ_R} \frac{1}{\sqrt{R}} dy$$
which is an inequality between a real number and a complex one.
In more extreme cases, the misuse of the inequality can be truly misleading.
For instance, consider $\int_{S_1} z^*\mathrm{d}z$ ($S_1$ denotes the complex unit circle at the origin). This integral equals $$\int_{S_1} (x-iy)(\mathrm{d}x+i\mathrm{d}y)=\int_{S_1} (x\mathrm{d}x+y\mathrm{d}y)+i\int_{S_1} (x\mathrm{d}y-y\mathrm{d}x)$$ and by Green's theorem, this evaluates to $2i$, so $\left|\int_{S_1} z^*\mathrm{d}z\right|=|2i|=2$.
But $\int_{S_1}\left|z^*\right|\mathrm{d}z=\int_{S_1} 1\mathrm{d}z=0$. So, if we applied $\left|\int_C f(z)\mathrm{d}z\right|\leq \int_C |f(z)|\mathrm{d}z$ in this case (where $f(z)=z^*$), we'd get $2\leq 0$ (this is just to emphasize the point).
$2)$ That $\sqrt{x^2+y^2}\leq R$ when $R$ is big enough is true. But you use this to derive (in combination with $\left|e^{i(x+iy)}\right|$) that: $$\int_{γ_R}\frac{|e^{i(x+iy)}|}{|x+iy|}dx+idy ≤ \int_{γ_R} \frac{1}{\sqrt{R}} dx+idy$$
This is faulty, most immediately, because it is an inequality involving a complex number (as mentioned before), but also because $\sqrt{x^2+y^2}\leq R\implies \frac{1}{\sqrt{x^2+y^2}}\geq \frac{1}{R}$ not $\frac{1}{\sqrt{x^2+y^2}}\leq \frac{1}{R}$.
$3)$ Finally, you give the parametrization of $\gamma_R$ as: $$\gamma_R(\theta)=cos(\theta)+i\sqrt{1-cos^2(\theta)},\quad\theta\in [0,\pi]$$ but it should actually be $$\gamma_R(\theta)=Rcos(\theta)+iR\sqrt{1-cos^2(\theta)},\quad\theta\in [0,\pi]$$ where the extra factor of $R$ accounts for the radius of the semicircle (otherwise the parametrization would always just trace out a unit circle).
I bring this up, even though you did not use the parametrization in your answer, because I will in mine (following what the "Hint" says).
A more convenient way of writing the transformation is $$\gamma_R(\theta)=Re^{i\theta},\quad \theta\in[0,\pi]$$
It is easy to see that $\mathrm{d}z=Rie^{i\theta}\mathrm{d}\theta$
So the integral becomes: $$\int_0^{\pi}\frac{e^{i\left(Re^{i\theta}\right)}}{Re^{i\theta}}\times\left(Rie^{i\theta}\mathrm{d}\theta\right)=i\int_0^{\pi}e^{i\left(Re^{i\theta}\right)}\mathrm{d}\theta=i\int_0^{\pi}e^{Ri\mathrm{cos}(\theta)-R\mathrm{sin}(\theta)}\mathrm{d}\theta$$
So (using the estimation lemma): $$\left|\int_{\gamma_R}\frac{e^{iz}}{z}\mathrm{d}z\right|=\left|i\int_0^{\pi}e^{Ri\mathrm{cos}(\theta)-R\mathrm{sin}(\theta)}\mathrm{d}\theta\right|\leq \int_0^{\pi}e^{-R\mathrm{sin}(\theta)}\mathrm{d}\theta$$
Now, using the hint, we break up the above integral: $$=\int_0^{\frac{1}{\sqrt R}}e^{-R\mathrm{sin}(\theta)}\mathrm{d}\theta+\int_{\frac{1}{\sqrt R}}^{\pi-\frac{1}{\sqrt R}}e^{-R\mathrm{sin}(\theta)}\mathrm{d}\theta+\int_{\pi-\frac{1}{\sqrt R}}^{\pi}e^{-R\mathrm{sin}(\theta)}\mathrm{d}\theta$$
It is clear that $\left|e^{-R\mathrm{sin}(\theta)}\right|\leq 1$ for all $\theta\in [0,\pi]$, so the left and right integrals are bounded above by $\frac{1}{\sqrt R}$, giving us $$\leq \frac{2}{\sqrt R}+\int_{\frac{1}{\sqrt R}}^{\pi-\frac{1}{\sqrt R}}e^{-R\mathrm{sin}(\theta)}\mathrm{d}\theta$$
Lastly, $\mathrm{sin}(\theta)\geq \frac{\theta}{2}$ whenever $\theta\geq 0$ and so long as $\theta$ is small enough. So, as long as $R$ is big enough, $\mathrm{sin}(\theta)\geq \mathrm{sin}(\frac{1}{\sqrt R})\geq \frac{1}{2\sqrt R}$ for all $\theta\in [\frac{1}{\sqrt R},\pi-\frac{1}{\sqrt R}]$, giving us a final inequality: $$\leq \frac{2}{\sqrt R}+\int_{\frac{1}{\sqrt R}}^{\pi-\frac{1}{\sqrt R}}e^{-R\left(\frac{1}{2\sqrt R}\right)}\mathrm{d}\theta\leq\frac{2}{\sqrt R}+\pi e^{-\frac{\sqrt R}{2}}$$
Remember that all of this is an upper bound on the original integral's magnitude, $\left|\int_{\gamma_R}\frac{e^{iz}}{z}\mathrm{d}z\right|$.
Then, since both $\frac{2}{\sqrt R}$ and $\pi e^{-\frac{\sqrt R}{2}}$ tend to $0$ as $R\rightarrow \infty$, the original integral too must approach $0$ as $R\rightarrow \infty$.
Comment: Thank you so much for pointing out to my mistakes and clarifying them! I understand now
Comment: @JOJO Glad to hear it.


Does a threshold effect the training or testing fold in cross validation?

Question: I am trying to better understand how changing a threshold affects a cross validation model. So if you trained a random forest model, the default threshold is threshold=0.5. And I understand that if the predicted result gets a score >0.5 it is considered a positive case and vice versa. But if you have a 5-fold cross validation model, is the model looking at what happens in the first four folds, and then looks at the threshold in order to give you the results on the test fold OR does it apply the threshold on just the test fold? Or in other words, what does the threshold change? The results of the training folds or just of the testing fold?
I guess more technically speaking, looking at the example below, it appears that the results given for each fold is for the testing fold. So does that mean the threshold is evaluated on the testing fold and the training folds make don't care about the threshold?
attach(iris)
#create a binary outcome on Sepal.Length
iris <- iris %>% mutate(Sepal.Length=ifelse(Sepal.Length>5.0,"aff","neg"))

ctrl <- trainControl(method="cv",
                     number=5,
                     summaryFunction=twoClassSummary,
                     classProbs=T,
                     savePredictions = T)

model <- train(Sepal.Length ~ ., data = iris, trControl = ctrl, method=
                    "rf", preProc=c("center","scale"), metric="ROC",importance=TRUE, tuneGrid =
                    data.frame(mtry = 2))

#examine outcome at every fold
print(model$pred)

> print(model$pred)
    pred obs   aff   neg rowIndex mtry Resample
#1    aff neg 0.616 0.384        7    2    Fold1
#2    neg neg 0.116 0.884       10    2    Fold1
#3    aff aff 0.602 0.398       15    2    Fold1
#4    aff aff 0.894 0.106       19    2    Fold1
#5    aff neg 0.706 0.294       25    2    Fold1
#6    aff neg 0.716 0.284       27    2    Fold1
#7    neg neg 0.020 0.980       43    2    Fold1
#8    neg neg 0.034 0.966       48    2    Fold1
#9    aff aff 1.000 0.000       51    2    Fold1
#10   aff aff 1.000 0.000       60    2    Fold1

Answer:
But if you have a 5-fold cross validation model, is the model looking
  at what happens in the first four folds, and then looks at the
  threshold in order to give you the results on the test fold OR does it
  apply the threshold on just the test fold?

You train a model based on the first 4 folds, then this model predicts the probabilities for the test fold. With this probability, the test fold are assigned labels and this is checked against the actual labels, with a 0.5 cutoff, to calculate the accuracy, ROC etc.

Or in other words, what does the threshold change? The results of the
  training folds or just of the testing fold?

The threshold doesn't change. At least not in the example you showed. If you want to change the threshold to make a prediction, its another thing altogether and has nothing to do with cross-validation. This involves re-writing a predict() function that sets the cutoff at something and incorporating this into randomForest() and train() .

It appears that the results given for each fold is for the testing
  fold.

To add to the above, yes every data point in your data has a chance to be in the test fold and be predicted by the other 4 folds.
Comment: So to recapitulate, it seems like the threshold affects the final prediction of the test fold (either `aff` or `neg` in this case). So if you are modifying the threshold to get a certain value for sensitivity for your overall model which will predict on separate data, then does this mean  you would want each of your training folds to have that sensitivity or you would want that trained model to have that sensitivity (i.e. average of all testing folds)?
Comment: Ok I think I get what you mean. So you are thinking of using another threshold for sensitivity calculation and your question is whether you can train your model on that?
Comment: Yeah I want to make a model with `sensitivity=0.5` and I don't know what's more appropriate: train a model and then modify `model$pred` so it uses the threshold I predetermined which would then give me `sen=0.5` for the final model. And that means all folds are evaluated on the same threshold but it also means each "set" (train fold1-4, test fold 5, etc) has a different sensitivity ind sen that averages to `sen=0.5`. Or the alternative, train each set so each has `sen=0.5` but the overall average of test folds is unlike to be `sen=0.5`. I think the latter undoes cv since each fold is different
Comment: sensitivity = TP/(TP+FN), how do you force a model to stop at 0.5 ? I think this is a bit contorted now.. So first of all, when you train a model, you usually choose that which has the highest defined score, ROC, accuracy etc. Now you have a very weird case, so you can choose the one, that is closest to 0.5 sensitivity
Comment: I was basically following along [this] (https://stats.stackexchange.com/questions/221409/varying-classification-threshold-to-produce-roc-curves) to find the threshold which uses the ROC curve to give me an overall sensitivity of 50%. I guess I'm trying to conceptually understand how it does that, and you make it sound like it does it for each of the individual test folds to get that overall `sen=0.5`. But now I'm not sure if it is right to affect the threshold on the training folds or the testing folds if that makes sense?
Comment: You might optimize for sensitivity (maximize), but you cannot get a sensitivity of 0.5 This means you are labeling 50% of your true labels false, and this makes is as good as random! Hope you see the contradiction here
Comment: Thanks this was helpful. I think I just need to think about it a little bit more!


Xcode 4.4 Connecting a .h and .m to a View Controller

Question: I'm using Xcode 4.4 and I'm trying to connect the .h and .m files to the view controller in my storyboard but the options I have for adding files are:
Objective-C class
Objective-C category
Objective-C class extension
Objective-C protocol
Objective-C test case class
Every tutorial and answer I have found is for the 4.2 version which goes the option of Objective-C subclass which I don't have. I was wondering if anyone knew how to do it in 4.4.
Thanks
Answer: why you have to connect  .m and .h to the view controller ?
try the easy way: file - new file - objective c class and in the second windows check "with XIB interface".
with this, you'll get .m .h and .xib !
Answer: I had the same problem and just found the answer.
First, make sure to read Xcode help about storyboard (in help menu, type story...)
Second, Make sure XCode show you his full interface (in the tool bar, select all three view icons, or at least the bottom right and bottom left icon.
There is a video in the help which explain how to do so.


I need help to find surface area of the function $\operatorname e^{-3x}$ within interval $[0, 7]$

Question: Can you someone give me a hint to start finding the surface area of the function $f(x) = e^{-3x}$ within interval $[0, 7]$? like which method I should choose ?
I tried find area by the following integral:
$$\int_{x=0}^{x=7} 2\pi \operatorname e^{-3x}\sqrt{1+9\operatorname e^{-6x}}\, \operatorname dx$$
Update:
I substitute
$\tan(\theta) = \operatorname e^{-3x}$
$\sec(\theta) = \sqrt{1+9 \operatorname e^{-6x}}$  and
I got answer as below
$\pi[(\sqrt{1+9\operatorname e^{-6x}}\operatorname e^{-3x}+\ln(|\sqrt{1+9e^{-6x}}|+\operatorname e^{-3x}))]|_0^7$
but that gives negative number as area. I think I made wrong somewhere, but unable to figure out what it is, please if you anyone tell me the write way, it will be great help.
Comment: The question makes no sense without the phrase "revolve around the $x$-axis" somewhere.
Comment: What went wrong when you were trying to evaluate that integral?
Comment: That looks like the right integral to me, by the way--it is an integral that you can evaluate.
Comment: actually, I don't the correct way to integrate this, that's my issue? can u tell me the correct substitution to use for this integral, please?
Comment: @user114408: Tell me if I screwed up the formatting.
Comment: Surface of what? the cylinder?
Comment: @nbubis: the surface area for f(x) = $e^{-3x}$ revolve around x-axis between interval [0,7]
Comment: If you put the stuff you are taking the square root of in braces, it will all be under the square root sign.  So \sqrt{1+9e^{-6x}} gives $\sqrt{1+9e^{-6x}}$  Also putting backslashes before functions gives the proper font:  \tan gives $\tan$
Answer: If $\tan \theta = e^{-3x}$, then $\sec \theta =\sqrt{1+\tan^2 \theta}=\sqrt {1+e^{-6x}}$.  You probably want $\tan \theta = 3e^{-3x}$
Comment: I got you. thanks a lot. :)
Comment: @user114408 Clicking the green pipe you can accept an answer, which is a big reward to the answerer.


dust.js reference object attributes using $idx

Question: I'm new to dust.js. Is there a way to reference object attributes using $idx. For ex:
<code>{
  "table": [
    {
         "row":
         {
               "ID": "1",
               "Domain":"Domain 1"
         }
    },
    {
         "row":
         {
              "ID": "2",
              "Domain":"Domain 2"
         }
    }]
}
</code>
How do I reference the ID attribute using $idx ? I might rename the columns ("ID","Domain") later-on and don't want the dustjs code to break.
Thanks
Answer: This is not possible for a few reasons.

Most importantly, order does not matter in a JavaScript object, so there is no way of knowing what member of he object will come first. The ordering may be different in different browsers, and could change if you change the names of the keys.
It is not currently possible to iterate over the members of an object using Dust.
Comment: Yes, that's what I guessed would be an issue. Thanks for the reply.


The smallest 12 digit natural number for which the sum of its digits is 80.

Question: So everytime I heard the word smallest natural number of 5 digits for example I do this in my head:
<code>100000...
^
|
</code>
which means that there must be a one in there:
So I type in 1000 and then add more digits to have $$1000\,9999\,9999$$
Now into my calculator I calculated that this is is not 80 rather just 73, so I'll have to add a 7 out there to have $$1007\,9999\,9999$$
But is my answer correct? If not why my method fails and what is the correct method?
Comment: I would say that your reasoning is valid, perfect in fact.
Comment: [Reasoning valid.](/a/1851837)
Answer: This seems correct. Any other 12-digit number with a digit sum of 80 has to be larger than this one, because one of the digits in the first, second, third or fourth spots would have to increase.


CORS error requesting php file on NGINX

Question: I am making a POST request to NGINX but getting a CORS error:
Failed to load https://knode.work/save.php: Response to preflight
request doesn't pass access control check: No
'Access-Control-Allow-Origin' header is present on the
requested resource. Origin 'https://www.knode.io'
is therefore not allowed access.
On NGINX I have this in /etc/nginx/sites-available/default:
location / {
     # First attempt to serve request as file, then
     # as directory, then fall back to displaying a 404.
     try_files $uri $uri/ =404;

     add_header "Access-Control-Allow-Origin"  *;
  }

location ~ \.php$ {
     include snippets/fastcgi-php.conf;
     fastcgi_pass unix:/run/php/php7.0-fpm.sock;

     add_header "Access-Control-Allow-Origin"  *;

  }

MORE INFO:

I've updated the NGINX configuration file.  Please see the Gist at https://gist.github.com/jxxcarlson/c17f9d89e06f5804170a0e44236b9d9a
NGINX is not sending back the Access-Control-Allow-Origin header: http://noteimages.s3.amazonaws.com/uploads/Screenshot%202018-08-13%2008.38.52.png
Comment: Are you sure the call is matching on the enclosing server block? Be great to see the rest of your config, if you can share it.
Comment: It could very well be wrong -- I am new at this.  Here is gist with `/etc/nginx/sites-available/default`:

https://gist.github.com/jxxcarlson/c17f9d89e06f5804170a0e44236b9d9a

Thanks so much for looking at this!
Comment: I've updated the `nginx` configuration and also the Gist in the previous comment.  Still no change. I believe that the problem is specific to requests for *.php files.  There is some discussion of this online, but no good answers that I can find so far.
Comment: `sites-available/default` — is that for knode.work or www.knode.io?
Comment: That is for knode.work -- the NGINX server with PHP
Answer: It turns out that one needs to install nginx-extras:
apt-get install nginx-extras
then configure /etc/nginx/sites-enabled/default using more_set_headers instead of add_headers, as in the listing below.  With these changes, there are no CORS errors.
location ~ \.php$ {

    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/run/php/php7.0-fpm.sock;

    if ($request_method = 'OPTIONS') {
        more_set_headers 'Access-Control-Allow-Origin: $http_origin';
        more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS, PUT, DELETE, HEAD';
        more_set_headers 'Access-Control-Max-Age: 1728000';
        more_set_headers 'Access-Control-Allow-Credentials: true';
        more_set_headers 'Access-Control-Allow-Headers: Origin,Content-Type,Accept,Authorization';
        more_set_headers 'Content-Type: text/plain; charset=UTF-8';
        more_set_headers 'Content-Length: 0';
        return 204;
    }

    location ~ /\.ht {
    deny all;
    }

}

Comment: Why configure  ($request_method = 'OPTIONS') {  if he is sending POST?
Comment: cause it's preflight request?

CORS error requesting php file on NGINX


Determine parameter m so equation has complex results for x:$(m+1)x^2+2(m-1)x+4m+1=0$

Question: $$(m+1)x^2+2(m-1)x+4m+1=0$$
So for results to be complex $b^2-4ac<0$
I then get to a part where I don't know what to do next: $-8m^2-28m<0$. How do I solve m?
Comment: You are trying to find which values of $m$ satisfy $8m^2 + 28m > 0$.  Have you tried completing the square?
Comment: @fractal1729 What do you mean by completing the square ?
Answer: Consider $-8m^2-28m$ as a parabola; its zeroes are $0$ and $-7/2$.  As it is downward open it will take negative values if $m<-7/2$ or $m>0$.
BTW I've got $-12m^2-28m$ ...
Comment: Yup. Made a mistake. It is 12
Answer: You will need $3m^2+7m>0$, so $m(m+\frac{7}{3})>0$. For this to be true, you will need to have $m$ and $(m+\frac{7}{3})$ of the same sign. Find values of $m$ that satisfy this condition.
Edit: you had a mistake in your calculations (Thanks to Michael Hoppe for seeing it)
Comment: Thanks!!!!!!!!!


Restoring SQL databases with an AzureRM Custom Script Extension

Question: I am trying to download and restore database backups against a SQL server with an AzureRM custom script extension, the files download fine but restores fail with access denied ('for database 'master'), I assume this is because the script runs as the local system account and has no permissions.
For lots of the other activities I'm doing as part of Azure automation, I can pass a credential object to handle this stuff without storing secrets (i.e. the result of get-credential passed as an argument) but Set-AzureRmCustomScriptExtension only allows arguments of type "string"
How can I set the restore-sqldatabasebackup commandlet in that custom script extension to use the existing user that has permission to do this?
Answer: well, you can just construct credential object "on the fly":
<code>$cred = [pscredential]::new('administrator',(ConvertTo-SecureString -String '!Q2w3e4r' -AsPlainText -Force))
</code>
you can obviously use variables in here and pass those into the custom script extension. another option - grant local system permission to restore databases ;)


Perl XML::Simple Print to Excel File

Question: Question:
I want to print the xml data to the excel file, but fail to print to the excel file.
xml file:
data.xml
<?xml version="1.0" encoding="UTF-8"?>
<ns1:BoardTestXMLExport numberOfIndictedComponents="11" testerTestStartTime=".00000:00" testTime=".00000:00" repairStationId="vvts03" testStatus="Reviewed Passed" testerTestEndTime=".00000:00" xmlns:ns1="http://tempuri.org/BoardTestXMLExport.xsd" numberOfIndictedPins="0" numberOfComponentsTested="113" numberOfJointsTested="0" numberOfDefects="11" repairStatus="Reviewed Passed">
          <ns1:BoardXML imageId="0" serialNumber="R174842442 1P00232G01 R02 1C31228G01 R07 JH1711327" assemblyRevision="1P00232G01-LF-SS" boardType="1P00232G01-LF-SS" boardRevision="1511417735000"/>
          <ns1:StationXML testerName="huaemsaoi004" stage="V510"/>
          <ns1:RepairEventXML numberOfVariationOkDefects="0" numberOfFalseCalledPins="0" numberOfRepairedComponents="0" numberOfVariationOkPins="0" numberOfRepairedPins="0" numberOfRepairLaterPins="0" numberOfFalseCalledDefects="11" numberOfActiveDefects="0" numberOfVariationOkComponents="0" repairEndTime="2017-11-23T14:31:05.000+08:00" repairStartTime="2017-11-23T14:30:58.000+08:00" numberOfRepairLaterDefects="0" repairOperator="1066689" numberOfRepairLaterComponents="0" numberOfActiveComponents="0" numberOfActivePins="0" numberOfRepairedDefects="0" numberOfFalseCalledComponents="11"/>
          <ns1:TestXML name="r1">
                    <ns1:IndictmentXML algorithm="rep-3a99979127l-lf" indictmentType="Left off, Right off">
                              <ns1:RepairActionXML repairOperator="1066689" repairTime="2017-11-23T14:30:58.000+08:00" repairActionType="-" indictmentType="Left off, Right off" comment="-" repairStatus="False Call"/>
                              <ns1:ComponentXML packageId="EP-3A99979127L-LF" partId="EP-3A99979127L-LF" designator="r1"/>
                    </ns1:IndictmentXML>
          </ns1:TestXML>
</ns1:BoardTestXMLExport>

my code as below:
# use module
 use strict;
 use warnings;
 use XML::Simple;
 use Data::Dumper;
 use Excel::Writer::XLSX;

# create object
 my $xml = XML::Simple->new;

# create object
 my $xml = XML::Simple->new;

 #create excel object
 my $OutFile = "Output". "\." . "xlsx";
 my $workbook  = Excel::Writer::XLSX->new($OutFile);
 my $worksheet = $workbook->add_worksheet('SotredProcedures');

 # Create a format for the column headings
 my $header = $workbook->add_format();
 $header->set_bold();
 $header->set_color('red');
# Create a Text Wrap format
 my $format1 = $workbook->add_format();
 $format1->set_text_wrap();
 $worksheet->write(0, 0, "RepairStationID", $header);
 $worksheet->write(0, 1, "BoardType", $header);
 $worksheet->write(0, 2, "ComponentsTested", $header);
 $worksheet->write(0, 3, "NumberOfDefects", $header);
 $worksheet->set_column(0, 1, 12);
 $worksheet->set_column(1, 2, 20);
 $worksheet->set_column(1, 3, 20);
 $worksheet->set_column(1, 4, 20);
# read XML file
 my $data = $xml->XMLin("data.xml",KeyAttr=>"ns1");
print  $data ->{"ns1:BoardXML"}->{"boardType"}->{$worksheet};

Result: fail to print the boardtype to the excel file, output as below:

how can i do for it?
Comment: Don't use XML::Simple. The library itself tells you not to use it. See [Why is XML::Simple "Discouraged"?](https://stackoverflow.com/q/33267765/1030675) for more details.
Comment: Please **don't post code or error messages as images**. We've told you this before.
Comment: Have you _read your error message_? Do you know what it means? One of the most important things about programming is paying attention to details. Your error message tells you exactly what's wrong. Since you cut off the line number, I can't tell you exactly where the problem is, but it's fairly obvious. You used something that is a string like it was a hash reference. That won't work. You are accessing your data structure in a wrong way. Output it again with Data::Dumper and and check what you're doing.
Comment: @huang: Please don't create new accounts just to get round the new-user restrictions. [As I said to you yesterday](https://stackoverflow.com/questions/47549993/perl-xmlsimple-print-issue#comment82099077_47551350), if you contribute to the site then you will quickly accumulate the reputation that you need in order to be able to post more questions.
Comment: @huang: You show many signs of trying to write code by cutting and pasting examples from many sources without really understanding what you are doing. This is a recipe for disaster. I urge you to step back from this and take the time to understand the code that you are trying to use. Or, alternatively, admit that this is beyond your current skill level and pay a programmer to do it for you.
Comment: Also your code looks like you are really struggling with your task. It seems you copy/pasted a lot and don't understand what's going on. There are some things there twice, like the declaration of `$xml`. This code will not even compile. And the actual problem is that you used your `$worksheet` in a wrong way.
Comment: @zdim Yesterday's post was originally another question but then edited into a different question. I guess the author couldn't post another question within 5 days or so, was in a hurry, and then created this new account.
Comment: @simbabque  This is a new account, created today; but the file seems the same as in [yesterday's post](https://stackoverflow.com/q/47549993/4653379)
Comment: @choroba we've told them in the last question. Dind't help. ;)
Comment: most of thing i have finished it, now only need to print to the excel file now.
Answer: Your error message says that you are trying to access the string 1P00232G01-LF-SS as a hash reference. That comes from this line:

print  $data->{"ns1:BoardXML"}->{"boardType"}->{$worksheet};

Your $worksheet is an object that represents the sheet in your Excel file. You cannot use it as a hash key, and the value of $data->{"ns1:BoardXML"}->{"boardType"} is a string. There is no hash reference there to access data from.
You already know how to write (not print) data to your worksheet. You do it above:

$worksheet->write( 0, 1, "BoardType", $header );

Now you need to do the same thing to write your data to the sheet.
$worksheet->write(1, 1, $data ->{"ns1:BoardXML"}->{"boardType"}, $format1);

Remember that you need to change the row your're writing to. That's now row 1, so the first argument to write needs to change.

Perl XML::Simple Print to Excel File


Criterions for $U_1(\mathbb{Z}G)=G$ i.e. units to be trivial in $\mathbb{Z}G$

Question: Notation- $U_1(\mathbb{Z}G)$ here means units of $\mathbb{Z}G$ with augmentation $1$, i.e.  $u=\sum_{g\in G} a(g)g \in U(\mathbb{Z}G)$ such that $\sum_{g \in G} a(g)=1$
1) I have done theorem by Higman which states that If $G$ is a torsion group then $U_1(\mathbb{Z}G)=G$ iff $G$ is either an abelian group of exponent $1,2,3,4$ or $6$ or a Hamiltonian $2$- group.
2) Another criterion I have done is that if $G \le U_1(\mathbb{Z}G)$ and [$U_1(\mathbb{Z}G):G $] is finite then $U_1(\mathbb{Z}G)=G$.
What are other interesting/ nontrivial criterion for units in $\mathbb{Z}G$ to be trivial?
References are welcome too.
Answer: In the book An Introduction to Group Rings by Milies and Sehgal there is in chapter 8.2 the following exercise:

(Parmenter and Polcino Milies [116]) Let $G$ be a finite group. Prove that the following conditions are equivalent.
$\phantom{\text{(iiii)}}\llap{\text{(i)}}\;$ $\mathcal{U}(\mathbf{Z}G)$ is nilpotent.
  $\phantom{\text{(iiii)}}\llap{\text{(ii)}}\;$ $\mathcal{U}(\mathbf{Z}G)$ is an FC group.
  $\phantom{\text{(iiii)}}\llap{\text{(iii)}}\;$ $T\mathcal{U}(\mathbf{Z}G)$ (the set of elements of finite order in $\mathcal{U}(\mathbf{Z}G)$) is a subgroup.
  $\phantom{\text{(iiii)}}\llap{\text{(iv)}}\;$ $G$ is either abelian or a Hamiltonian $2$-group.
  $\phantom{\text{(iiii)}}\llap{\text{(v)}}\;$ $\mathcal{U}(\mathbf{Z}G)=\pm G$.
[116] M.M. Parmenter and C. Polcino Milies, Group rings whose units form a nilpotent or FC group, Proc. Amer. Math. Soc. 68, 2 (1978), 247-248

You might also like some of the other exercises in this chapter.


Regex - Extract all text except square brackets and their contents

Question: I have a changelog that has the version numbers in square brackets like this:
## [0.3.0] - 2016-12-13
- FIX This is an example fix
- FEATURE An example feature
I want to extract only the version numbers so I am left with:
[Unreleased]
[0.3.0]
[0.2.4]
[0.2.3]
Ideally I'd like to extract only the first value in square brackets that isn't 'Unreleased' so I get the latest version number 0.3.0 although I'm not sure this is possible in regex.
Can anyone help? Thanks :)
Comment: What have you tried so far, and what about your attempt does not meet your needs?
Comment: What tool/language are you using?
Comment: `(\[.*?\])` Try this.
Comment: I'm confused - your title says "Remove all..." but your question says "extract only...". Remove and extract are two different things. Which is it?
Comment: Thanks @ClasG I've updated my title :)
Answer: You can use a regex like this:
.*(\[.*?\]).*

With the replacement string: \1
Btw, if you want to remove all the other text, then you can use:
.*(\[.*?\]).*|*

Working demo
Update: if you want to transform the changelog in a list of versions, I could come up with this regex:
.*(\[.*?\])[\s\S]*?(^$|\z)

Working demo
Comment: `[^[\r\n]+(?:\R[^[\r\n]+)*\R?|(\[[^]]*\])` should be a performance improvement for your last regex. `[^[\r\n]+|(\[[^]]*\])` for the one before.
Comment: @SebastianProske, very nice to know. Thanks for the comment.
Answer: Edit (misread at first):
Replace
[\s\S]*?(\[[\d.]+])[\s\S]*

with
\1

Check out this example at regex101.


Por que se fala tanto em segurança em programação orientada?

Question: Programo em C e agora resolvi me desbravar no mundo do POO pela linguagem C#. Vejo que muitas pessoas falam de segurança e é justamente essa a minha dúvida, ao programar vejo que o próprio ambiente de desenvolvimento me limita o acesso as classes privadas assim como seus atributos e seus métodos. Quando eu defino esta classe como pública tudo o que pertence a ela fica acessível dentro de outras classes.
1° Definir como private uma classe protege o código do programador ou a aplicação do usuário?
2° Se eu acessar atributos e métodos de uma classe A em uma classe B eu estarei comprometendo a segurança da aplicação?
Se alguém também puder me dar umas dicas relacionadas a organização do código eu ficarei grato, pois estou acostumado com C então sempre modularizei meu código separado por 2 headers (funções e bibliotecas) e 1 .c onde chamo todas as funções. Visto que em C# possui os adventos das classes e a segurança:
3° Qual seria a melhor forma de organizar tudo em scripts diferentes de forma segura?
Comment: Existem muitos livros que e pessoas que falam várias coisas sem nenhum fundamento.
Comment: A resposta resolveu sua dúvida? Acha que pode aceitá-la? Veja o [tour] se não souber como faz. Isso ajudaria muito indicar que a solução foi útil para você. Também pode votar em qualquer pergunta ou resposta que achar útil no site todo.
Answer: Segurança é uma propriedade que nada tem a ver como programação orientada a objeto. Talvez o termo fosse outro.
Definir algo como privado não protege qualquer coisa. O compilador não deixa você usar um código normal que chame aquele membro. Nada impede que o membro seja acessado por formas não normais, como reflexão. Inclusive não protege uma instância acessar membros privados de outra instância da mesma classe.
Essa anotação no membro é só um indicador do que não deve fazer. Isso tem a ver com encapsulamento, com ocultação de informação desnecessária ao código. Mas não entenda ocultação com proteção. É só que está dentro em um invólucro, se quiser abri-lo está tudo lá pra ver.
Acessar membros de uma classe não compromete segurança alguma. Se isso ocorresse nem daria para fazer alguma coisa.
É possível usar classes privadas, mas quase nunca elas são úteis, uma classe privada só pode ser acessada dentro de uma outra classe onde esta foi declarada.
C# pode até ser usada assim mas ela não é uma linguagem de script.
Fica complicado responder as outras questões, sugiro fazer perguntas mais específicas e com mais detalhes.
Sugiro também procurar uma forma estruturada para aprender esses conceitos, talvez um bom livro.
Comment: bigdown, obrigado pela resposta. Qual livro você recomendaria com base na sua experiência para eu poder entender esses conceitos que até o momento estão bastante desconexos para mim?
Comment: Não gosto de recomendar livros, mas tem uma lista em http://pt.stackoverflow.com/tags/c%23/info.


How to read a file from within a the JAR. When both are present on the server

Question: This is the sample code which we were using:
DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
Document doc = dBuilder.parse(new FileInputStream("Data.xml"));
doc.getDocumentElement().normalize();

So when using the same code on local we can get the path using system.getProperties('user.dir').
Is there any way that we can get the path on JasperReports Server So that we can read the file. This particular code mentioned above is available in the JAR and xml is placed outside the JAR.
Answer: If you know where your file is on your system, why don't you use absolute path ?
new FileInputStream("/path/on/my/system/Data.xml")

Comment: For local machine absolute path served the problem.But for the server it was not working it was not picking the path of the file to perform the further operations.
Comment: Is your local env running the same OS than the server ?
Comment: yes.The file is on the JASPER server and need to call this file from a JAR.
Comment: I mean what's the os ?


Simplify a parametric equation with hyperbolic trigonometric functions

Question: I've the following parametric equations for a curve:
$$\begin{cases}x(t)=a\cdot \operatorname{sech} (t)
\\
y(t)=a\cdot(t-\tanh(t))\end{cases}$$
Now let $\theta(t)=-\arctan(\sinh(t))$ how does the parametric form change? I need to substitute $\theta$ in above equations...I'm not good at simplifying these tricky expressions involving even inverse hyperbolic functions. Any help would be greatly appreciated.
Comment: Since $\theta(t)=-\arctan(\sinh(t))$ you have $t=\operatorname{arcsinh}(\tan(-\theta(t)))$, which you can subsitute for $t$. I'd start from there and try to simplify the thus generated terms.
Answer: Hints:
You can use that $t=-\sinh^{-1}(\tan\theta)$, along with the identities that
$\text{sech} (-x)=\text{sech}(x), \;\;\text{sech}(\sinh^{-1}t)=\frac{1}{\sqrt{t^2+1}},\;\;\tan^{2}\theta+1=\sec^{2}\theta$,  and
$\sinh^{-1}x=\ln(x+\sqrt{x^2+1}), \;\;\tanh(-x)=-\tanh (x),\;\;\tanh(\sinh^{-1}t)=\frac{t}{\sqrt{t^2+1}}$

Notice that $\displaystyle\text{sech}(\sinh^{-1}(t))=\frac{1}{\cosh(\sinh^{-1}(t))}$ and $\displaystyle\tanh(\sinh^{-1}(t))=\frac{\sinh(\sinh^{-1}(t))}{\cosh(\sinh^{-1}(t))}$
where $\cosh(\sinh^{-1}(t))=\sqrt{t^2+1}$ and $\sinh(\sinh^{-1}(t))=t$.


Rule to send mail to the users about the percentage of their profile completed(single & multiple profiles)

Question: I am trying to write a rule to send an email to the user if his/her one of the profiles is not completed. But I am not getting how to get the empty field's list to sent to the user. What actually i wanted to do is. List all the fields which are empty from profile (Single & multiple) and sent the list to the user, requesting to fill all the details or just percentage of profile completion . Please help me to sort out this problem.
Thanks for your time,
Answer: How about using Content Complete or Profile Complete Percent to track profile completion. Content Complete integrates with Rules.
Comment: As I am working on drupal 7 I am using profile2 and pcp modules. How to integrate pcp module with rules?


Trying to install phpmyadmin - getting error 500

Question: (Using PHP 5.4.15, Apache 2.2, CentOS 6.4 minimal)
I'm having trouble installing phpMyadmin, I followed the instructions here, specifically:

I uploaded the phpMyAdmin-*.tar.gz & unzipped it to /var/www/html/phpmyadmin/
Copied config.inc.php & CHMODed it to 600 (I have suexec enabled, though I also tried with 660)

Yet I'm having error 500 whenever I access http://my-ip-address/phpmyadmin/index.php
I checked the apache2 logs, I see this error repeated countless times:
[Fri Jun 07 00:24:26 2013] [error] [client 18.104.22.168] PHP Fatal error:  Call to a member function get() on a non-object in /var/www/_/8thpixel.com/public_html/phpmyadmin/libraries/Header.class.php on line 135
I checked the source of Header.class.php (didn't modify it), line #135 looks like this:
if ($GLOBALS['PMA_Config']->get('user_preferences') == 'session'
            && ! isset($_SESSION['userprefs_autoload'])
I checked $GLOBALS, there's no user_preferences at all!
I've been pulling my hear for two days now, anyone can tell me what's wrong anyway?
Comment: Try the config with 664 to see if it works. Does your apache user own the config file? www-data or apache usually.
Comment: @NathanC: You're a genius! chmodding config.inc.php with 664 did it man, please post this as an answer so that I can accept it!
Answer: chmod 664 config.inc.php should resolve the problem. You may be able to get away with 660 if the config file is owned by the web server's user.