Adding logic to a page

Question: I am new to Drupal, and I am trying to add some behavior to an existing page within Drupal.
Using the admin UI, I created an model with 2 numbers (num1, num2). I then rendered it into a form.
Now I would like to add a third field, and compute using the client the sum of num1 and num2 into this new field. This needs to be done as you type the numbers.
I wanted to use Javascript to do this, but I really don't know where to start. Is there an example on how to simply add some logic into an exiting screen that is generated by the admin tools?
This is more information.
Then, I just add these to fields into a page like so:

Again, at this point I would like to just add the sum of both numbers into a 3ed field that is not there yet (of course my project is much more complex, but this is a simple example). So I was going to add field_number3 into my Content type, and I wanted to do some simple Javascript to calculate the field.
I could really use some step by step examples, to just to know where to place a module, and what code to type in  the module to only add this behavior to my content.
I am very new to Drupal, and I am using Drupal 8.
Comment: Welcome to Drupal Answers. It's not clear exactly what you're trying to do. Please add examples and screenshots. Also, it's not clear what you mean by "model" - do you mean field? Which part of the admin ui are you referring to?
Comment: Sorry, I added more details.
Comment: Welcome to Drupal Answers! _Adding logic to a page_ is quite a broad task to explain. We cannot explain from zero how to create a module, how a module can add JavaScript to a node being shown, and how JavaScript code can interact with fields attached to an entity.
Answer: Attaching custom JS to form
Adding css and javascript to a drupal 8 module
Here's an example of using hook_form_FORM_ID_alter. For this you'll need to make a custom module (conveniently named custommodule in the example below).
Module directory:
custommodule/
|- js/
   |-- custommodule.js
|-- custommodule.info.yml
|-- custommodule.libraries.yml
|-- custommodule.module

File contents:

custommodule.info.yml
type: module
name: Custommodule
description: 'Custom stuff'
package: Custom
core: '8.x'

custommodule.libraries.yml
custommodule:
  version: 1.x
  js:
    js/custommodule.js: {}
  dependencies:
    - core/jquery

custommodule.module
<?php

use Drupal\Core\Form\FormStateInterface;

/**
 * Implements hook_form_FORM_ID_alter().
 */
function custommodule_form_node_article_edit_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  $form['#attached']['library'][] = 'custommodule/custommodule';
}

custommodule.js
(function($) {

  Drupal.behaviors.myBehavior = {
    attach: function (context, settings) {

      var num1 = $("#edit-field-num-1-0-value").value;
      var num2 = $("#edit-field-num-2-0-value").value;
      var result = (Number(num1)+Number(num2));
      $("#edit-field-num-3-0-value").value = result;
    }
  };

})(jQuery);

The example uses the default article node type, with 3 integer fields (field_num_1, field_num_2, field_num_3), and works on the article edit form (form id -> node_article_edit_form). The js part is also very basic so all it does is set the third field value once the page loads. Drupal_add_js is deprecated, so we use the $form['#attached']['library'] array to attach a library (defined in the custommodule.libraries.yml file) that contains the JS file (a single libraries.yml file can define more libraries, which can include more js/css files).
Comment: Oh my god. This is awesome.  May God bless you!
Answer: This question is very broad and should be easy to find answers to, but if you're new to Drupal I understand it can be hard to know where to look. You'll need to implement hook_form_alter or hook_form_FORM_ID_alter in a custom module to be able to add your JS via drupal_add_js:
MODULE_form_alter(&$form, &$form_state, $form_id) {
  if ($form_id == 'NAME') {
    drupal_add_js(PATH);
  }
}

You will then need to look at Managing JavaScript in Drupal 7 in order to work out how to actually code the JavaScript you need.
Comment: I added more details to make this question more specific. I think what you suggest is probably the correct way. Still, if you can just give me a more step by step approach, so I can try to code it.
Comment: The question is for Drupal 8 not for Drupal 7.

Adding logic to a page


Is this JQuery deferred working properly?

Question: Here is the line of code in question:
$.when(dashboard.update.call(this)).then(dashboard.updateCompleted);

The problem is, dashboard.update calls another series of 5 or so functions.  Does dashboard.update finish after it has finished calling those functions or after those functions have actually finished?
The reason I ask is because it seems like dashboard.updateCompleted is being called before the functions called from dashboard.update have completed.  Is there a way around this?
Comment: You'll have to show us what `dashboard.update()` does (does it have async operations) and whether it returns a promise that is resolved only when the async operations are done.  `$.when().then()` can only do it's job for async operations inside of `dashboard.update()` if `dashboard.update()` returns a promise that is resolved only when the async operations are done.  And, if it does that, you don't need `$.when()` either as you can just use `dashboard.update.call(this).then(...)` directly.
Comment: it depends on whether those functions uses async execution.. ie whether they have ajax request etc within them
Comment: I'm not sure whether the functions inside of dashboard.update are async or not... they are a series of ajax requests though.
Comment: Welp, AJAX literally has async in its name... so yes those functions do use async execution.
Comment: @Hubrid Perhaps, see also approach here http://stackoverflow.com/a/23862173/2801559
Comment: Simple question.  Does `dashboard.update()` return a promise?  If not, then it will not work with `$.when()`.
Answer: To use when(), you must pass a defered object to the functions in when().
I just seen very similar situation with you. What I have done to make the code work is :
$.when(func1($.Deferred()),
       func2($.Deferred()),
       func3($.Deferred()),
       func4($.Deferred())
).done(function() {
    alert('Finished!');
}

And in each function, resolved() or rejected() must be called to notify $.when(). For example,
function func1(def) {
    A_async_call(function (status) {
        if (status == OK) {
           def.resolve(); /// Tell $.when, this function finished
        } else {
           def.reject();  /// tell $.when, a function failed
        }
    })
    return def; // return defered object to when, for later use
}

Comment: You hardly ever pass deferreds-to-be-resolved around. Having the `def` as a parameter *and* returning it looks like an antipattern to me.


DIfferentiable function and generalized mean value theorem

Question: Let f be continuous on [a,b] and differential on (a,b). Prove that if a >= 0 there are x1, x2, x3 $\in$ (a,b) such that
$$f'(x_1) = (b+a){f'(x_2)\over 2x_2} = (b^2 + ba + a^2){f'(x_3)\over3x_3^2}$$
I think this problem will use the generalized mean value theorem to solve.
However, I don't know how to apply it.
Can you suggest a solution?
Thank you very much!
Answer: By the mean value theorem,
$$\exists x_1 \in (a,b)\,: \quad \frac{f(b)-f(a)}{b-a} =f'(x_1) \,. \tag{1}
$$
Consider $g:[a^2,b^2] \to {\mathbb R}$ defined by $g(x)=f(\sqrt{x})$. Applying the mean value theorem to $g$, we infer that there is some $t \in (a^2,b^2)$ such that
$$\frac{g(b^2)-g(a^2)}{b^2-a^2} =g'(t) \,,$$
or equivalently, writing $ x_2:=\sqrt{t}$,
$$\frac{f(b)-f(a)}{(b-a)(b+a)}= \frac{f'(\sqrt{t})}{2\sqrt{t}}=\frac{f'(x_2)}{2x_2} \,. \tag{2}
$$
Similarly, apply the mean value theorem to $h:[a^3,b^3] \to {\mathbb R}$ defined by $h(x)=f(x^{1/3})$. We deduce that there is some $s \in (a^3,b^3)$ such that
$$\frac{h(b^3)-h(a^3)}{b^3-a^3} =h'(s) \,,$$
or equivalently, writing $ x_3:= {s^{1/3}}$,
$$\frac{f(b)-f(a)}{(b-a)(b^2+ba+a^2)}= \frac{f'(s^{1/3})}{3s^{2/3}} =\frac{f'(x_3)}{3x_3^2} \,. \tag{3}
$$


How to read one line at a time from a data file and to perform calculations in it before moving to the next line in C Programming?

Question: I'm a beginner at C programming and I would appreciate some help in order to understand the problem.
Alright so, I have a data file (input.dat) with data like this: (first line) 0 2 3 4 5; (second line) 1 2 3 5 4, (third line and so on...). I'm required to read the data one line at a time until the end of file and print it. This is what I have done so far:
int main(void)
{
float coeffs[5];
FILE *input;  /* File pointer to the input file */
fopen_s(&input, "input.dat", "r"); /* Location of the input file */
int count = 0;
/* Loops to read data set*/
while (fscanf_s(input, "%f %f %f %f %f ", &coeffs[0], &coeffs[1], &coeffs[2], &coeffs[3], &coeffs[4]) != EOF)
{
    printf("a=%.4f; b=%.4f; c=%.4f; d=%.4f; e=%.4f\n", coeffs[0], coeffs[1], coeffs[2], coeffs[3], coeffs[4]);
    count++;
}
return 0;
}

This is showing all of the lines in the data file at once. But this is not what I want. I need to read one line at a time and perform some calculations and conditions for that one line first before I move to the next line. So how can I do that?
Next problem is, for the first line, I need to implement a loop from -10 to +10 with increment of 2 (to get 11 results in total). For example the program will read the first line, display it on the screen, then for the first value -10, the program will calculate and again display something . Then it will do the same for -8, then for -6 and so on until +10. After the 11 results are displayed, the program will then and ONLY then, move to the second line and so on. Hence for each line in the data file, the program will have 11 results. How can I use the loop function with increment of 2 to achieve these 11 results?
I would appreciate if anyone can provide me a simple layout of the structure of the codes which I've to write. NOTE: The formats are a bit different than other compilers as I must use Microsoft Visual Studio to do it.
Comment: I imagine the downvotes are related in part to that, and in part to the fact that you show no effort to solve the second part of your question.
Comment: As for you being a beginner, this is not a tutorial site, nor really even a help site.  That we provide help is a happy side effect of our mission to create a database of on-topic questions and answers that will be useful to others.  The downvoters do not think your question meets our criteria, which you can and should read about here: https://stackoverflow.com/help/asking.
Comment: What are the down-votes for? I'm just trying to understand the base of the problem. :/
Comment: If your input is formatted with exactly five whitespace-separated numbers per line, then your code will indeed read one line at a time.  It indeed does perform some calculations (`printf()` and `count++`) for each such line.  What's not working about that?
Comment: The output of my code displays all the lines of the data file at ONCE when I run. Is it possible that it only displays the first line, then if I press ENTER, it displays the second line and so on?
Comment: I actually have no idea how to solve the second part of the question being a beginner. That's why I'm here.
Comment: The program's output for each line immediately follows that for the previous line because you don't output anything else in between.  You don't notice a delay because computers can do this sort of thing very swiftly.  If you want to do additional work per input line (prompt, read a response from the user, ...) then code for that would go into the loop.
Comment: Ahhh I get it now. So it indeed does read one line at a time.
Comment: @John...Oh I see. I will try to keep that in mind next time I'm asking a question. Thanks for your time to comment. Really appreciated!
Comment: It seems you really are missing some fundamentals of programming - repeating a numeric pattern and running a code block for each number in there is day 1 or day 2 of learning how to code.
That is likely the reason for the downvotes - we can't write a coding basic tutorial on every question here.

Try reading this to start fiiling up your knowledge gap: https://www.tutorialspoint.com/cprogramming/c_loops.htm
Comment: I will take a look at that. Thank you very much! @jsbueno
Answer: Add your calculations to your while loop. You are reading one line at a time anyway.
If you want to loop from -10 to 10 with increments of 2, use a for loop.
for(count = -10; count <= 10; count = count + 2)
    {
        // Calculations
    }

Comment: The for loop will go inside the while loop, am I right?


Evento .on('click', não funciona em input contendo data-md-icheck

Question: Olá, em minha função tem o evento .onClick que deveria ativar ao marcar o checkbox, porém quando uso o data-md-icheck ele é ignorado pela função.
<input type="checkbox" data-md-icheck class="ts_checkbox"/>

Evento:
$(document).on('click', 'input[type="checkbox"]', function () {
    $this.click = $(this);
    $this.get();
});

Comment: Bom pessoal, acabei encontrando a documentação e a solução. http://icheck.fronteed.com/#callbacks
Comment: Poste como resposta à sua pergunta e, após, marque-a como certa.
Answer: Documentação http://icheck.fronteed.com/#callbacks
Usei o comando:
$('input').on('ifChecked', function(event){
  alert(event.type + ' callback');
});
Answer: For me the event ifChanged works fine.
$(document).ready(function(){
       var callbacks_list = $('.demo-callbacks ul');
            $('.demo-list input').on('ifCreated ifClicked ifChanged ifChecked ifUnchecked ifDisabled ifEnabled ifDestroyed check', function(event){
                $('#'+this.id+':checkbox').attr('checked', function(idx, oldAttr) {
                   return !oldAttr;
                }); //this manipulates checked state of checkbox
            }).iCheck({
              checkboxClass: 'icheckbox_square-blue',
              radioClass: 'iradio_square-blue',
              increaseArea: '20%'
            });
          });

Comment: Este site é em Português, portanto, é recomendável que sua resposta também seja.


Accuracy of K Nearest Neighbor in near-ties

Question: Suppose you have a case where there's a near-tie in K Nearest Neighbor Algorithm - 3 of the K Nearest Neighbors are in class A and 2 of the K Nearest Neighbors are in class B. (Obviously, K = 5). Effectively, that means that a single document is determining which class your item should be in. In this case, would K = 5 actually be significantly better than K = 1? Would it be any better at all? Or am I missing something?
Suppose, for example, that there's actually a 50% chance that something will fall into class A and a 50% chance of it falling into class B, and that the documents are essentially "spread" randomly. This is a rather contrived example, but under this situation at least it seems that K = 5 is no better than k = 1 (in each case, there's a 50 - 50 chance of it being in one class vs. another). For cases like that, do we just say "that kind of a distribution is unlikely to happen in 'real life,' and K Nearest Neighbor is just a probabilistic algorithm anyway"?
Put another way, can K Nearest Neighbor appear to find patterns that don't exist in random (or close to random) data, or in data that's fairly "uniformly" distributed? (I suppose that, for a large number of classifications, it probably wouldn't show any pattern, but for a small number of classifications it might appear to show a nonexistent pattern).
Comment: Define "find patterns". On random data, the predicted class will look random, so what?
Comment: @Anony-Mousse That's my point, though - it may *look* like it's finding patterns when there aren't any. Just because the data is random overall doesn't mean that it won't *look* like there are patterns locally.
Comment: The thing is, I wouldn't call it a pattern. kNN doesn't have a notion of "pattern".
Answer: It's true that KNN, when run on bogus data, can appear to find patterns which don't exist. However, you could say the same for soft-margin SVM, or logistic regression, or nearly any classification algorithm.
Therefore it's necessary to perform cross-validation on classification models to ensure that the model is not overfitting to random noise.
About your example with $k = 5$. I think the better way to look a it, is instead of the classification of your datapoint being determined with just a single vote, it's now determined by 5 votes, which on average, may reflect reality better than just one vote (this depends heavily on the dataset though).
Suppose 999 students elected a class president, and the final vote came down to 500 vs 499. Sure, it all came down to the last person to vote, but that doesn't mean the vote couldn't have just been replaced by one random student voting.
Comment: Well for classifying random noise, whatever $k$ is, you're not going to get good results out of it, and you are correct that it is completely random whatever $k$ is. But the idea is that when there are actually patterns in the data, then a larger $k$ can reduce the effect of randomness in your classification (at the cost of a larger bias, see bias-variance tradeoff)
Comment: This is very difficult to say in general. Even if the data is not completely random, most data has some degree of randomness to it, so you can still pick up a "fake" pattern. In general, larger $k$ reduces the effect of this randomness. On the other hand, smaller $k$ can be more sensitive to patterns which will be smoothed out by larger $k$. Selection of $k$ is a nontrivial task, but you should try something like cross-validation to help you choose.
Comment: Would that imply that the probability of the last person voting for one of the candidates isn't necessarily 50 - 50 (i.e. that it's not necessarily like picking a candidate for Class President at full random)? I suppose the fact that? I suppose that you *could* argue that the fact that 500 students voted in favor of the guy who won implies that there's a slightly higher probability of someone voting for that candidate than the guy who lost (although it seems vaguely circular to me).
Comment: So, unless the data actually *is* truly random, the odds are you're still picking up on a real pattern (even if it *was* a near-tie in that case)?


USAMO problem solution

Question: I had asked for a hint here USAMO problem hint. i had tried induction once but i thought it won't work so i left it, but after seeing @lulu 's comment, i decided to give it a go again. Please see if my solution is correct.
[USAMO 2003] Prove that for every positive integer n there exists an n-digit number divisible by $5^n$ all of whose digits are odd.
MY SOLUTION: So first thing, i checked some small cases and figured we could generate the number with (n+1) digits satisfying the property by adding a number to it's front, ie. adding b 10$^n$ to the number with n digits.
We'll proceed with induction, let P(n) mean there exists an n-digit number divisible by $5^n$ all of whose digits are odd.
P(1) is true as 5|5.
Let P(k) be true, ie. let 5$^k$ | $a_ka_{k-1}...a_1$  with $a_i$ $\neq$ 2l for i $\in$ {1,2...k} .
I'll attempt to prove that by adding $ b \cdot 10^k $  with $ b \in {1,3,5,7,9} $ . we can have a number that is divisible by $5^{k+1}$.
So we want 5$^{k+1}$ $|$ $ b \cdot 10^k $ +  $a_ka_{k-1}...a_1$ . --> eq.1
Let $a_ka_{k-1}...a_1$ = $5^km  $
So from eq. 1, inputting $a_ka_{k-1}...a_1$ = $5^km  $ , we will get
5$^{k+1}$ $|$ $ b \cdot 10^k $ +  $5^k$m, then dividing throught by $5^k$ , we need
5 $|$ $2^k \cdot b + m$
as  $ b \in {1,3,5,7,9} $ , $\equiv$ 0,1,2,3,4 (mod 5)
So now m $\equiv$ 0,1,2,3,4 (mod 5) , let m $\equiv$ r (mod 5),
We need $2^k \cdot b + r  =0 (mod 5)$
now,$2^k \equiv$ 1,2,3,4 (mod 5)
so painstakingly going through each and every case of possible values of $2^k$ and m (mod 5) (there are 16 cases), we prove that we can find a $ b \in {1,3,5,7,9} $ such that 5 $|$ $2^k \cdot b + m$ .
This is the first time i have written so much in latex, so i am sorry if there is any mistake.
If you were a grader, out of 7, how many points would you give me?
Answer: I don't think math.se answers can address how they'd mark it, but I can advise on a cleaner way to write answers, because your ideas are right but they could do with algebraic clarity, and clarity regarding modulo arithmetic. (If you find yourself claiming that if we do something we eventually get a particular outcome, try to state this as an existence theorem that's either obvious, well-known or proven within your work.) To wit:

We claim some sequence $a_n$ of $n$-digit numbers in base $10$, all
digits odd, satisfies $5^n|a_n,\,10^n|a_{n+1}-a_n$. In particular
write $a_n=5^nb_n,\,a_{n+1}=a_n+10^nc_n$, so $b_1=1$ (because $a_1=5$)
and$$5^{n+1}b_{n+1}=a_{n+1}=c_n10^n+5^nb_n\iff5b_{n+1}=c_n2^n+b_n,$$so
it suffices to choose $c_n\in\{1,\,3,\,5,\,7,\,9\}$ with
$5|c_n2^n+b_n$. This choice is possible because these $5$ choices of
the $c_n$ each achieve a different residue class modulo $5$ (because
$5\nmid k2^n$ for $k\in\{2,\,4,\,6,\,8\}$), and exactly one achieves
$5|c_n2^n+b_n$.
Comment: @Aditya_math I wish I could remember which chapter of which book I wanted to recommend. You can start with guides like [these](https://www.google.com/search?q=proof+writing+guide+math&pws=0), but be warned some of their advice is about typesetting, instead of what will matter in your handwritten exploits. What may help the most is to read a lot of proofs to see how they articulate an idea clearly.
Comment: thanks, so should i work more on my proof writing skills, like is it necessary to write a really well polished solution such as yours. Actually i dont have much experience with proof writing, so is there a specific way we need to write our proofs in, and if there is, do you know where i can learn that, thank you
Comment: @Aditya The argument after "this choice is possible" is a special case of a general criterion for solvability of linear congreunces. You should invoke this criterion instead of repeating its proof inline as above - see my answer. This criterion (and related) results are fundamental so you should be sure to know them well.
Comment: @BillDubuque are you reffering to the last three lines?
Comment: @Aditya_math Yes.
Answer: There is a $\,x\in\Bbb Z\,$ with $\,5\mid 2^k x - m\!\iff\! \bmod 5\!:\ 2^k x \equiv m\,$ has a root $\,x.\,$ By general theory
$$\begin{align} \color{#c00}c\ x &\equiv \, d\!\!\pmod{\!\color{#0a0}n}\ \ \text{has a root}\ x\!\iff\! \gcd(\color{#c00}c,\,\color{#0a0}n)\mid d\\[.3em]
{\rm thus}\ \  \color{#c00}{2^k} x&\equiv m\!\!\!\pmod{\!\color{#0a0}5}\ \ \text{has a root}\ x,\ \, {\rm by}\ \ \gcd(\color{#c00}{2^k},\color{#0a0}5)\!=\!1\end{align}\qquad$$
and we can choose a root $\,x\in \{1, 3, 5, 7, 9\}\,$ since it is a complete residue system $\!\bmod 5;\,$ alternatively: $ $ if $\,0\le x < 5\,$ is even then  $\,x':= x\!+\!5\,$ is odd $< 10,\,$ and $\,x'$ remains a root by $\,x'\equiv x\pmod{\!5}$.
Comment: @Aditya  If you know modular inverses then we can write $\bmod 5\!:\ x\equiv m(2^{-k}) \equiv m(3^k)\,$ by $\,2^{-1}\equiv 3.\,$ But this problem only requires the *existence* of a solution, not its value. Of course it is always correct to repeat the proof of a theorem in a special case application, even though that is not optimal.
Comment: Thanks, i knew what this says but i never knew it had a name. So it is essentially like finding an inverse modulo $2^k$ . Also, i have one question, if sometimes, in a contest i don't remember the name of a theorem/ can't frame it, would it be ok to do the case bash? although unpleasent, does it work?

USAMO problem solution


Removing outliers renders a new distribution that has its own outliers

Question: I'm trying to remove all the outliers from a data set. However, after removing them, data points that weren't outliers before are now outliers due to the new distribution. What is the correct statistical procedure for this situation?
Comment: Have you considered that what you are regarding as 'outliers' may be useful observations for analyzing the data at hand. For example, if data are exponentially distributed, then 'outliers' by the 'boxplot' criterion are natural values of the distribution. As you eliminate some 'outliers' the reduced dataset may well show 'outliers' of its own, and so on. For severely skewed distributions such as Pareto (after several such iterations) you can wind up ignoring a substantial and legitimate subset of the data. // Usually it's a bad idea to delete 'outliers' unless you _**know**_ they are faulty.
Comment: To echo @BruceET a bit, defining outliers as those observations a certain distance from the mean/median/IQ range is not generally helpful, because there is nothing necessarily "wrong" with that data. For example, if you're trying to estimate the variance and you eliminate those points, you will be underestimating the variance. Outliers should be defined as those observations that you believe were *generated by a different distribution* than the rest of your data, and hence don't help you estimate something about the distribution of interest.
Comment: How do you decide what is an outlier?
Comment: Winsorization and truncation are just two examples of how to handle outliers. You seem to have conducted truncation. However, the respective quantiles you "cut off" might have been too small, as you suggest. You could just choose larger quantiles you cut off, if you want to hang on to truncation.
Comment: I can understand how removing outliers only to find that the new distribution has (its own) new outliers can look like a bit of a paradox
Comment: To use Hawkins' definition [Hawkins 1980]: "An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism." You should have either strong evidence or strong prior belief that some of your data is generated by a different mechanism/distribution before you throw those data points away.
Comment: Just keep deleting them until there is no more data.  Problem solved.
Comment: I keep deleting them until i get less than 0.5% outliers
Answer: Lognormal data is often encountered in practice. Here is a session in R
that begins with $n_1 = 1000$ observations from a lognormal distribution.
At the start, the sample mean is $\bar Y_1 = 1.70$ and the sample standard
deviation is $S_1 = 2.49.$ Then we go through several iterations, removing
boxplot outliers at each step. [Computations in R.]
set.seed(2020)
y1= rlnorm(1000)
summary(y1); length(y1); sd(y1)
    Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
 0.04217  0.49016  0.94409  1.69558  1.89374 40.56466
[1] 1000
[1] 2.493404

y2 = y1[y1 <= boxplot.stats(y1)$stats[5]]
summary(y2); length(y2); sd(y2)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.04217 0.46651 0.85522 1.11458 1.53644 3.97491
[1] 911
[1] 0.8619062

y3 = y2[y2 <= boxplot.stats(y2)$stats[5]]
summary(y3); length(y3); sd(y3)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.04217 0.44789 0.82218 1.01352 1.41563 3.13268
[1] 874
[1] 0.7214308

y4 = y3[y3 <= boxplot.stats(y3)$stats[5]]
summary(y4); length(y4); sd(y4)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.04217 0.44615 0.80870 0.98817 1.38893 2.86639
[1] 863
[1] 0.6898908

y5 = y4[y4 <= boxplot.stats(y4)$stats[5]]
summary(y5); length(y5); sd(y5)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.04217 0.44576 0.80567 0.97515 1.38133 2.79356
[1] 857
[1] 0.6744334

With successive 'outlier' removals, the sample size has fallen
to 911, 874, 863, and finally, $n_5 =$ 857. So I'm down to less than 86% of my
original data, with no end of removals yet in sight. The sample
mean has decreased from 1.70 for the full sample to 0.975, and the standard deviation
from 2.49 to 0.674.

One has to wonder what population the final sample in this sequence might represent. Certainly, not the same population that the original sample came from.

Here are boxplots for the original sample (at left) and the multiply-truncated sample y5 (right).
boxplot(y1,y2,y3,y4,y5, col="skyblue2")

To be clear, there is nothing unusual about the original sample above. Here are boxplots for 20 samples of size 1000 from the same population. Every one of the 20 samples has multiple boxplot outliers.
set.seed(1021)
m = 20;  n = 1000
x = rlnorm(m*n);  gp = rep(1:20, each=n)
boxplot(x ~ gp, col="skyblue2", main="Boxplots of 20 Lognormal Samples")

A simulation with 100,000 samples of size 1000 shows that the average number of
(first pass) boxplot outliers per sample is about 76.
set.seed(1234)
nr.out = replicate(10^5, length(boxplot.stats(rlnorm(1000))$out))
mean(nr.out)
[1] 77.53626

If an outcome is known to have resulted from equipment failure or data entry error,
then, of course, it should be excluded. Also, there may be times when you know
a value is simply impossible (e.g., a negative height or a human lifetime over 900 years)
and needs to be ignored. But I think it is a serious mistake to establish an
automated process for 'outlier' removal--particularly an iterative one. Sometimes real data show surprisingly good or disastrously bad news. It's a good idea to pay attention when that happens.
Comment: (+1): What made you choose lognormal distribution? Is there a clear flaw in outlier identification in boxplots that helped in making the choice of distribution?
Comment: when do i stop? when outlier is less than xx % of the data?


Finding f(x,y) given its partial derivatives

Question:
This is from Fitzpatrick's advanced calculus chapter 13 which is a analysis text and the start of analysis of several variables. Now the solution manual uses a multivariable version of the mean value theorem and this problem is actually from the chapter on the multivariable mean value theorem. Now based on what we did in multivariable calculus, an easy way to solve this problem is to just integrate with respect to x which gives us an an arbitrary constant and an arbitrary function of y, then differentiate that. However, because this is an analysis text, I am not sure if that is correct from a rigorous standpoint. For one thing, indefinite integrals has never been defined, but I do want to ask whether that is a valid solution method.
Answer: This is correct from a rigorous standpoint. We can define a function $g: \mathbb{R} \to \mathbb{R}$ by \begin{equation}
g(t) = f( t , 0 ). \end{equation}
Since $f$ has all partial derivatives at each point in $\mathbb{R}^2$ it follows that the function $g$ we defined is differentiable (in the single variable sense). This is because $g'(t) = \frac{d}{dt} f(t,0) = \partial_x f(t,0). $ Now, also from single variable calculus we know from the fundamental theorem of calculus that \begin{equation} f(t,0) - f(0,0) = g(t) - g(0) = \int_0^t g'(s) ds = \int_0^t  \partial_x f(s,0) ds. \end{equation}
Since we assume that $\partial_x f (x,0) \equiv 2$ it follows from this equation that $$ f(t,0) - f(0,0) = 2t.$$
Moreover, if we do the same thing for $y$ and we define a function $$ h_x(t) = f(x,t)$$ then we get the similar conclusion that $$ f(x,t)  - f(x,0) = 3t .$$
Finally we can notice that $$f (x,y) - f(0,0) = f(x,y) - f(x,0) + f(x,0) - f(0,0) = 3y + 2x$$ and so $$ f(x,y) = 3y + 2x + f(0,0) = 3y + 2x + 1. $$
This should be totally rigorous since we are just leveraging facts we know from single variable calculus.
Comment: Thank you for the clear explanation


How Can I Store Objects from LinearLayour when the Activity is destroyed?

Question: I have a activity that merely holds up a fragment, and this fragment has 2 objects that inherits from LinearLayout. I had a problem to manage the screen rotation, but i solved it up restoring and saving the state of these objects on their own onRestore and onSavedInstance methods. But this does't help when the activity that hold the fragment is destroyed. How can i proceed?
My activity
<code>public class LifeCounterActivity extends SingleFragmentActivity{
private String[] navMenuTitles;
private TypedArray navMenuIcons;
static String sFragmentTag;
private static String TAG = "LifeCounterActivity";
private static String KEY = "LifeCounterFragement";

LifeCounterFragment mFragment;

public static Intent newIntent(Context packageContext, String fragmentTag){
    Intent intent = new Intent(packageContext,LifeCounterActivity.class);
    intent.putExtra(KEY_ID_FRAGMENT, fragmentTag);
    sFragmentTag = fragmentTag;
    return intent;
}

@Override
public void onCreate(Bundle savedInstanceState){
    super.onCreate(savedInstanceState);
    Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
    setSupportActionBar(toolbar);

    if (savedInstanceState != null){
        mFragment = (LifeCounterFragment) savedInstanceState.getSerializable(KEY);
        getSupportFragmentManager().beginTransaction().replace(R.id.fragment_container,mFragment).commit();
    }else{
        android.support.v4.app.FragmentManager fm = getSupportFragmentManager();
        Fragment fragment = createFragment();
        fm.beginTransaction().add(R.id.fragment_container,fragment).commit();
    }

    /////////////////////////////////////////////////////////////////////////////////
    navMenuTitles = getResources().getStringArray(R.array.nav_drawer_items); // load titles from strings.xml
    navMenuIcons = getResources()
            .obtainTypedArray(R.array.nav_drawer_icons);//load icons from strings.xml
    set(navMenuTitles, navMenuIcons);
    navMenuIcons.recycle();
}

@Override
protected void onSaveInstanceState(Bundle outState) {
    outState.putSerializable(KEY,mFragment);
    super.onSaveInstanceState(outState);
}

@Override
protected Fragment createFragment() {
    if (mFragment == null){
        mFragment = new LifeCounterFragment();
    }
    return mFragment;
}
</code>
}
The fragment:
<code>    public class LifeCounterFragment extends Fragment implements LifeCounter.EventHistoryRecorder,Serializable{
    public static String TAG = "LifeCounterFragment";
    private static String KEY_PLAYER_1 = "Player1";
    private static String KEY_PLAYER_2 = "Player2";
    private static String KEY_HISTORY = "History";
    LifeCounter mFirstPlayer;
    LifeCounter mSecondPlayer;
    HistoryManager mHistoryManager;

    private static float ROTATION_COUNT = 180f;
    public LifeCounterFragment(){
        setArguments(new Bundle());
    }

    @Override
    public void onSaveInstanceState(Bundle outState) {
        super.onSaveInstanceState(outState);
        Bundle b = new Bundle();
        b.putSerializable(KEY_HISTORY, mHistoryManager);
        getArguments().putAll(b);
    }

 @Override
 public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        View v = inflater.inflate(R.layout.fragment_life_counter, container, false);

        savedInstanceState = getArguments();
        if (savedInstanceState != null && savedInstanceState.size() > 0){
            mHistoryManager = (HistoryManager) savedInstanceState.getSerializable(KEY_HISTORY);
            mFirstPlayer = (LifeCounter) savedInstanceState.getSerializable(KEY_PLAYER_1);
            mSecondPlayer = (LifeCounter) savedInstanceState.getSerializable(KEY_PLAYER_2);
        }

        if (mHistoryManager == null) mHistoryManager = new HistoryManager();
        if (mFirstPlayer == null) mFirstPlayer = (LifeCounter) v.findViewById(R.id.player_1_counter);
        if (mSecondPlayer == null) mSecondPlayer = (LifeCounter) v.findViewById(R.id.player_2_counter);
</code>
The methods i used to save and recover on the object
<code> @Override
public Parcelable onSaveInstanceState()
{
    Bundle bundle = new Bundle();
    bundle.putParcelable("superState", super.onSaveInstanceState());
    bundle.putInt(KEY_LIFE, this.mLife);
    bundle.putString(KEY_NAME, this.mTextViewPlayerName.getText().toString());
    bundle.putInt(KEY_LASTLIFE, this.lastLifeAmount);
    bundle.putSerializable(KEY_LISTENER, (Serializable) mListener);
    bundle.putBoolean(KEY_ASTRAL,this.isAstral);
    return bundle;
}

@Override
public void onRestoreInstanceState(Parcelable state)
{
    if (state instanceof Bundle) // implicit null check
    {
        Bundle bundle = (Bundle) state;
        this.mLife = bundle.getInt(KEY_LIFE); // ... load stuff
        this.lastLifeAmount = bundle.getInt(KEY_LASTLIFE);
        this.mListener = (EventHistoryRecorder) bundle.getSerializable(KEY_LISTENER);
        this.isAstral = bundle.getBoolean(KEY_ASTRAL);
        state = bundle.getParcelable("superState");

        recoverData(bundle.getString(KEY_NAME));
    }
    super.onRestoreInstanceState(state);
}
</code>
Answer: Did you consider using SQLight? You may save and load the objects to the sql of your phone and reload them on your activity reopening.
If you want to try this solution follow this link for sql tutorial: http://developer.android.com/training/basics/data-storage/databases.html
Comment: I do use already a database for another thing, if i create another one, would be to hold up only 6 variables D:
Comment: It is great that you already have a database, just add a new table to hold those values, and use same database :)


O bindValue() do PDOStatement não funciona com o nome da tabela?

Question: Pois bem, tenho seguinte código:
public function Ler($tabela="funcionarios"){
        $this->sql = $this->pdo->prepare("SELECT * FROM :tabela");
        $this->sql->bindValue(':tabela',$tabela);
        $this->sql->execute();
        return $this->sql->fetchAll();
    }

Aí ele retorna um array vazio.
Mas Já este código:
public function Ler(){
        $this->sql = $this->pdo->prepare("SELECT * FROM funcionarios");
        $this->sql->execute();
        return $this->sql->fetchAll();
    }

retorna o array com os registros, por que com o bindValue não funciona?
Desde Já, Obrigado :)
Answer: Não, a substituição de valores com bindValue não funciona com todas as partes do comando, como o comando em si (select, update, etc), nem com nomes de campos ou da tabela, funciona com os valores no values, where, etc.
Isso precisa ser feito antes, assim por exemplo:
$sql = "SELECT * FROM $tabela";
$this->sql = $this->pdo->prepare($sql);


Showing $P \land (P \lor Q) = P$ using only laws of propositions.

Question: How can I show $P \land (P \lor Q) = P$ using only laws of propositions?
I see how $P \land (P \lor Q) = P$ simplifies to just $P$ using a truth table, but I can't see how to get it using propositional calculus. The distributive law just changes it to $P \lor (P \land Q)$, leaving me essentially in the same position.
Comment: Welcome to Math StackExchange! You will find that people are more willing to answer your question if you tell us what you have tried and where you are stuck. This shows some genuine effort from your side, as people are generally not willing to solve your homework problems for you. Also, it may help you get more tailored answers.
Comment: Welcome to MSE! When you ask a question, please tell your question clearly and show your efforts when you are struggling in this problem. Thank you!
Comment: I see how P AND (P OR Q) simplifies to just P using a truth table, but how do I show that using just laws?
The distributive law just changes it to P OR (P AND Q), leaving me essentially in the same position.
I've spent an about two hours just applying different combinations of laws, but I just don't see how to reduce the left hand side to the right hand side.
Answer: It is indeed true that $P \land (P \lor Q) = P$, this is called the absorption law. Sometimes this law is taken as a rule itself (for interested logicians, see the end of this answer). You mention that you are allowed to use the distributive law, and we can actually prove the absorption law with some clever use of it. We will also need the constant $\bot$, which I use to denote false. Remember that we always have $A \lor \bot = A$ and $A \land \bot = \bot$ for every proposition $A$. So now we can just write:
$$
P =
P \lor \bot =
P \lor (Q \land \bot) =
(P \lor \bot) \land (P \lor Q) =
P \land (P \lor Q).
$$
The rules we use here are (from left to right):

$P = P \lor \bot$ (this holds for all propositions, so in particular for $P$),
$\bot = Q \land \bot$ (this holds for all propositions, so in particular for $Q$),
distributive law,
$P = P \lor \bot$ (again).

The promised footnote for the interested logicians: the absorption law is in a sense more fundamental, because it is one of the axioms when defining a (bounded) lattice in the algebraic way. When you add the distributive law, you get a distributive lattice. The above shows that the absorption law can be proved in a distributive lattice, so you no longer need that axiom.
Comment: That was amazing, thank you. I never would of thought to use the law of contradiction in reverse, beautiful.
Comment: Little did you know you just helped me complete my DOOMSDAY DEVICE!
Answer: Here is one way to show this using natural deduction:

The equivalence will be derived using the biconditional introduction inference rule. I will consider each side of the biconditional as separate subproofs and then reference both at the end.
I consider $(P\land (P\lor Q))\to P$ on lines 1-2. I assume the antecedent $(P\land (P\lor Q))$ on line 1 and derive the consequent $P$ using conjunction elimination ($\land$E) on line 2.
I consider the other direction $P\to (P\land (P\lor Q))$ on lines 3-5. I assume the antecedent $P$ on line 3 and derive the consequent $(P\land (P\lor Q))$ using disjunction introduction ($\lor$I) on line 4 and conjunction introduction ($\land$I) on line 5.
Line 6 uses biconditional introduction on the two subproofs mentioned above  referencing each subproof as a range of lines.

Kevin Klement's JavaScript/PHP Fitch-style natural deduction proof editor and checker http://proofs.openlogicproject.org/
Answer: Well, the following analogy doesn't apply directly, but if you had two sets, $A$ and $C$ then $A \cap (A \cup C) = A$ $\iff$ $C\subset A$
Comment: $A\cap (A\cup C)=A$ is unconditionally true just as $P\wedge (P\vee Q)\dashv \vdash P$ is.
Comment: Adding to K B Dave's comment, that means that the claim in this answer is actually false. We do not have $A \cap (A \cup C) \Rightarrow C \subseteq A$. Any two sets with $C \not \subseteq A$ should do as a counterexample, e.g. $A = \mathbb{N}$ and $C = \mathbb{R}$.
Answer: To show statements are equivalent, show each implies the other. Simplification gives $\to$ while addition plus adjunction gives $\leftarrow$.


Showing $E[X_{n+1}|X_1,...,X_n] = a_0+\Sigma_{k=1}^n a_kX_k$

Question: $X_1,...,X_n,X_{n+1}$ are jointly distributed with a Gaussian distribution. We let $X^* = E[X_{n+1}|X_1,...,X_n]$. Show that there exists constants $a_1,...,a_n,a_{n+1}$ such that $X^* = a_0+\Sigma_{k=1}^n a_kX_k$.
Additional info:
$X_{n+1} - X^*$ is indep. of $X_1,...,X_n$ and $X^*$ satisfies the system of equation:
$E[X_{n+1}-X^*] = 0$
$E[(X_{n+1}-X^*)X_k] = 0, k = 1,...,n$
There is a suggestion to work backwards and argue that there exist $a_k$ solving the last two equations; show $X_{n+1} − X^*$
is independent of G = $\sigma(X_1, . . . , X_n)$; and consider
$\int_G X_{n+1} dP = \int_G(X^* + X_{n+1} - X^*)dp$
Attempt:
Suppose n = 1. Then the conditional expectation of $E[X_1|X_2]$ is almost surely uniquely defined as that Borel function of $X_2$ for which $E[(X_1 - E[X_1|X_2])g(X_2)] = 0$ for all Borel functions g. In the jointly Gaussian case, it suffices to verify that there is an affine combination $a_0 + \Sigma_{k=1}^n a_kX_k $ such that $X - (a_0 + \Sigma_{k=1}^n a_kX_k)$ is uncorrelated with the random variable $X_2$ and has zero mean. This is because, since $(X - (a_0 + \Sigma_{k=1}^n a_kY_k),X_2)$ is a linear transformation of $(X_1,X_2)$, these variables are jointly Gaussian and so this uncorrelatedness would imply that
$X - (a_0 + \Sigma_{k=1}^n a_kX_K)\scriptstyle\coprod$ $X_2$, which implies that for all Borel functions g:
$E[(X - (a_0 + \Sigma_{k=1}^n a_kX_k))g(X_1,...,X_n)] = E[X-(a_0 + \Sigma_{k=1}^n a_kX_k)]E[g(X_1,...,X_n)] = 0$
where the second line used $E[X-(a_0+\Sigma_{k=1}^na_kX_k)]$ = 0. This then implies $E[X_1|X_2] = a_0+\Sigma_{k=1}^na_kX_k$ by definition of the conditional expectation. Writing down the equals corresponding to this uncorrelatedness and the equation $E[E[X_1|X_2]] = E[X_1]$ gives a collection of simultaneous linear equations that can be solved for the coefficients $a_0,a_1,...,a_n$.
Answer: Hints:

Let $\arg\min_{{b_0,b_{-0}}}E\left[(X_{n+1}-b_0-b_{-0}'X)^2\right]=(\beta_0,\beta_{-0}')'=\beta\in\mathbb{R}^{n+1}, \quad X=(X_1,\dots,X_n)'.$
$\beta_0=E[X_{n+1}]-\beta_{-0}'E[X], \quad \beta_{-0}=Var(X)^{-1}Cov(X,X_{n+1}).$
$Cov(X, X_{n+1}-\beta_0-\beta_{-0}'X)=0.$
Use normality of $(X_1,\dots,X_{n+1})'.$
Show that $E[X_{n+1}-\beta_0-\beta_{-0}'X|X]=0$ and conclude.


Do parallel streams treat upstream iterators in a thread safe way?

Question: Today I was using a stream that was performing a parallel() operation after a map, however; the underlying source is an iterator which is not thread safe which is similar to the BufferedReader.lines implementation.
I originally thought that trySplit would be called on the created thread, however; I observed that the accesses to the iterator have come from multiple threads.
By example, the following silly iterator implementation is just setup with enough elements to cause splitting and also keeps track of the unique threads that accessed the hasNext method.
class SillyIterator implements Iterator<String> {

    private final ArrayDeque<String> src =
        IntStream.range(1, 10000)
            .mapToObj(Integer::toString)
            .collect(toCollection(ArrayDeque::new));
    private Map<String, String> ts = new ConcurrentHashMap<>();
    public Set<String> threads() { return ts.keySet(); }
    private String nextRecord = null;

    @Override
    public boolean hasNext() {
        var n = Thread.currentThread().getName();
        ts.put(n, n);
        if (nextRecord != null) {
            return true;
        } else {
            nextRecord = src.poll();
            return nextRecord != null;
        }
    }
    @Override
    public String next() {
        if (nextRecord != null || hasNext()) {
            var rec = nextRecord;
            nextRecord = null;
            return rec;
        }
        throw new NoSuchElementException();
    }

}

Using this to create a stream as follows:
var iter = new SillyIterator();
StreamSupport
    .stream(Spliterators.spliteratorUnknownSize(
        iter, Spliterator.ORDERED | Spliterator.NONNULL
    ), false)
    .map(n -> "value = " + n)
    .parallel()
    .collect(toList());

System.out.println(iter.threads());

This on my system output the two fork join threads as well as the main thread, which kind of scared me.
[ForkJoinPool.commonPool-worker-1, ForkJoinPool.commonPool-worker-2, main]

Answer: Thread safety does not necessarily imply being accessed by only one thread. The important aspect is that there is no concurrent access, i.e. no access by more than one thread at the same time. If the access by different threads is temporally ordered and this ordering also ensures the necessary memory visibility, which is the responsibility of the caller, it still is a thread safe usage.
The Spliterator documentation says:

Despite their obvious utility in parallel algorithms, spliterators are not expected to be thread-safe; instead, implementations of parallel algorithms using spliterators should ensure that the spliterator is only used by one thread at a time. This is generally easy to attain via serial thread-confinement, which often is a natural consequence of typical parallel algorithms that work by recursive decomposition.

The spliterator doesn't need to be confined to the same thread throughout its lifetime, but there should be a clear handover at the caller's side ensuring that the old thread stops using it before the new thread starts using it.
But the important takeaway is, the spliterator doesn't need to be thread safe, hence, the iterator wrapped by a spliterator also doesn't need to be thread safe.
Note that a typical behavior is splitting and handing over before starting traversal, but since an ordinary Iterator doesn't support splitting, the wrapping spliterator has to iterate and buffer elements to implement splitting. Therefore, the Iterator experiences traversal by different threads (but one at a time) when the traversal has not been started from the Stream implementation's perspective.

That said, the lines() implementation of BufferedReader is a bad example which you should not follow. Since it's centered around a single readLine() call, it would be natural to implement Spliterator directly instead of implementing a more complicated Iterator and have it wrapped via spliteratorUnknownSize(…).
Since your example is likewise centered around a single poll() call, it's also straight-forward to implement Spliterator directly:
class SillySpliterator extends Spliterators.AbstractSpliterator<String> {
    private final ArrayDeque<String> src = IntStream.range(1, 10000)
        .mapToObj(Integer::toString).collect(toCollection(ArrayDeque::new));

    SillySpliterator() {
        super(Long.MAX_VALUE, ORDERED | NONNULL);
    }

    @Override
    public boolean tryAdvance(Consumer<? super String> action) {
        String nextRecord = src.poll();
        if(nextRecord == null) return false;
        action.accept(nextRecord);
        return true;
    }
}

Depending on your real life case, you may also pass the actual deque size to the constructor and provide the SIZED characteristic.
Then, you may use it like
var result = StreamSupport.stream(new SillySpliterator(), true)
    .map(n -> "value = " + n)
    .collect(toList());

Comment: I assumed that it was just an example. That's why I suggested adding the sized characteristic "depending on your real life case" instead of just adding it to my example. I also tried to emphasize the general pattern behind implementing spliterator directly. Whenever you have some sort of "*get next element or tell me there are no more*" operation like `readLine()`, `poll()`, `fetchNext()`, etc., a `Spliterator` is much simpler to implement. Even using two-method APIs is not harder with a `Spliterator` compared to an `Iterator`. It's just a matter of getting familiar with it.
Comment: This is an excellent answer, thank you. I don't often need to implement spliterator and actually look for examples in the JDK. Example on the deque is for demonstration, impl in question is related to hadoop. Thanks again for this answer.


r stacked bar total value

Question: I want to create a plot (EDA) about data that is collected from houses (devices in houses). But I'm stuck..
The data is looking like this:
df$device    df$date       df$time    df$value
boiler       2015-01-13    12:15      0.0009
boiler       2015-01-13    12:30      0.0007
boiler       2015-01-13    12:45      0.0005
boiler       2015-01-13    13:00      0.0010
TV           2015-01-13    12:15      0.0009
TV           2015-01-13    12:30      0.0007
TV           2015-01-13    12:45      0.0005
TV           2015-01-13    13:00      0.0010
boiler       2015-01-14    12:15      0.0009
boiler       2015-01-14    12:30      0.0007
boiler       2015-01-14    12:45      0.0005
boiler       2015-01-14    13:00      0.0010
TV           2015-01-14    12:15      0.0009
TV           2015-01-14    12:30      0.0007
TV           2015-01-14    12:45      0.0005
TV           2015-01-14    13:00      0.0010

The data is measured over 8 months (Jan till September). I want to make a plot, with:
    Y- axis = measured value
    X- axis = months
    Plots = all the devices
I tried, with:
df$monthnumber <- month(df$Date)
test <- table(df$Device, df$monthnumber)

barplot(counts1,col=rainbow(7), xlim = c(1,15),
        legend = c(rownames(counts1)), bty = "L")

What lead to the next image:
Result:
But, that is wrong. Because it contains how often an device is measured. I do not know how I can add the TOTAL VALUE per month for each application (a summation).
I tried:
test$value <- aggregate (df$Measurevalue, by = list(genergy$Device), sum)

But an error was given:

Error in $<-.data.frame(tmp, "value", value = list(Group.1 = c("Boiler",       :
    replacement has 7 rows, data has 285896

How can I tackle this problem?
- A stacked bar, with the TOTAL VALUE of each device, with (y= energy), (X= months)
Comment: Have you tried plotting using ggplot?
Comment: Yes, but the main problem was to get the summation of the value. It didn't work in my case, because (i guess) the dataframes differs from lengt. But i really have no idea how i could fix that.
Comment: So:
- with month nummers (x)
- with measured value (y)
- with total value (in the bar)

And i do not understand how i can add (to my table) an extra variable with the measured value per device)
Answer: So, you could use a combination of dplyr and ggplot
library("dplyr")
library("ggplot2")

df <- data.frame(device = c("boiler","boiler","boiler","boiler","TV","TV","TV","TV","boiler","boiler","boiler","boiler","TV","TV","TV","TV"),
                     date = c("2015-01-13","2015-01-13","2015-01-13","2015-01-13","2015-01-13","2015-01-13","2015-01-13","2015-01-13","2015-01-14","2015-01-14","2015-01-14","2015-01-14","2015-01-14","2015-01-14","2015-01-14","2015-01-14"),
                     time = c("12:15","12:30","12:45","13:00","12:15","12:30","12:45","13:00","12:15","12:30","12:45","13:00","12:15","12:30","12:45","13:00"),
                     value = c(0.009,0.007,0.005,0.001,0.009,0.007,0.005,0.001,0.009,0.007,0.005,0.001,0.009,0.007,0.005,0.001)
                     )
    df <- mutate(df,device = as.factor(device),
                 date = as.Date.character(date,format = "%Y-%m-%d")) %>%
      group_by(device) %>%
        mutate(sum(value))

The result is a summary by device on a daily basis

as you mentioned in your comment, you were looking to do it for a monthly basis - group the table based on the month
df<- group_by(df,m=as.factor(month(date)),device) %>%
summarise(s = sum(value))
ggplot(df,aes(x=m,y=s,fill=device)) + geom_bar(stat="identity")

As, I don't have enough monthly data, the graph looks as follows..

update: 27/06/2016
following the input of the data, you run the following command. And then try summarising the data based on the month.
    df <- mutate(df,device = as.factor(device),
                  date = as.POSIXct((strftime(date,format = "%Y-%m-%d")))) %>%
         group_by(device)

Comment: Thank you, but my data contains almost 800,000 rows. So after e.g. device = c("boiler"....) is it possible to do this automatically?
Comment: Really thanks! But when im run your second code:

df<- group_by(df,m=as.factor(month(date)),device) %>%
summarise(s = sum(value))
ggplot(df,aes(x=m,y=s,fill=device)) + geom_bar(stat="identity")

I got this error:
Error in eval(expr, envir, enclos) :
  do not know how to convert 'x' to class "POSIXlt"
Comment: class(df$Date)  --> "Date"
Comment: Thanks, i have the feeling that i'm almost there, but a new error is showed up:

Error in eval(expr, envir, enclos) : object 'm' not found

While, the first time I run the code block:

    plotje <- group_by(plotje, m= as.factor(month(date)),device) %>%
      summarise(s = sum(value))

It gives no error (till I want to run the ggplot). So i cannot understand why m is not fount. (because i assign m to as.factor(month(date))
Comment: ggplot(plotje,aes(x= m,y= s,fill=device)) + geom_bar(stat="identity")
Comment: Hi Arun, no M doesn't exist. I also installed both packages.. but i guess that the bug is in:

Error in eval(expr, envir, enclos) :
  do not know how to convert 'x' to class "POSIXlt"

The first time I run the code, no error is given, till the ggplot line. After that, while I run the code again, I get an error (about to convert x to class posixit)
Comment: Thank you so much @Arun. Your solution works, I made an extra column df$month <- month(df$date)

And then i applied your function. Thanks a lot

r stacked bar total value


Proving $\gcd(a,b) = ax+by \Rightarrow \gcd(x,y) = 1$

Question: I'm not entirely sure of how I should prove this statement:

$$
\gcd(a,b) = ax+by \Rightarrow \gcd(x,y) = 1
$$

So I've tried
$$
\begin{align}
&\gcd(x,y) = d \Rightarrow x =x'd, y=y'd\\
\Rightarrow &\gcd(a,b) = ax'd+by'd = d(ax'+by')\\
\Rightarrow & \gcd(a,b) = d(\gcd(a,b)) \\
\Rightarrow &1= d
\end{align}
$$
But I'm not sure if this is correct. Even so, is there perhaps a better way to solve this?
Comment: @pushpen.paul Sorry if I'm unclear, but I am trying to prove that implication.
Answer: Denote $\gcd(a,b)=c$, then $a=ca'$ and $b=cb'$,
$$\gcd(a,b)=ax+by\Leftrightarrow c=ca'x+cb'y\Leftrightarrow a'x+b'y=1$$
so $\gcd(x,y)=1$.
Answer: \begin{align*}
d \mid x \text{ and } d \mid y
&\implies d \gcd(a,b) \mid ax \text{ and } d \gcd(a,b) \mid by \\
&\implies d \gcd(a,b)  \mid ax + by = \gcd(a,b) \\
&\implies d = \pm 1.
\end{align*}
As others have mentioned, your own solution has an error: just because $ax + by = \gcd(a,b)$ for this specific value of $x,y$ certainly does not mean it will be equl to $\gcd(a,b)$ for any $x,y$.  For instance this does not hold in the cases $x,y = 0$ and $x, y = 1$.
Answer: Why would $ax'+by' = \gcd(a,b)$ hold? Your solution is $\color{#c00}{\text{wrong}}$.
Please note that the statement $\gcd(a,b) = ax+by$, holds for some integers $x,y$, and not all integers $x,y$ satisfy this relation. perhaps, you are mistaken here.
For example, $\gcd(2,3)=1$, that is we take $a=2, b=3$; so you would get $2\cdot2+3\cdot(-1)=1$. So, it holds with $x=2,y=-1$. Note that there are infinitely many choices of $x,y$; such as $(-4, 3), (5, -3), (8, -5)$ etc. But not all integer $x,y$ are applicable. You may like to test with, say, $x=0, y=0$, or $x=-1, y=2$ etc.
Comment: @fvel Please mention that $x\neq0,y\neq0$. This is true (I checked), but what I was telling (in response to your comment "I'm not sure if this is correct"): *Your solution is wrong*. Kindly see other answers, if that helps, for *correct* solution of your question. *I am not posting an answer to it, since it is already here.*
Comment: @fvel Your argument is valid if it is known that $gcd(a,b)=ax+by = \mathbf1$ (note that there is one **1**).
Comment: @fvel Summary: Your question is correct (for all $\gcd$'s, may not be $1$). Here are correct solutions to it. But, what you did was wrong (unless you know that the $\gcd$ is $1$). Hope, that makes clear the fact. `:-)`
Comment: @fvel You are always welcome.
Comment: OK @Goos, I apologize. Even if I am so, it was not intentional... `:-(`.
Comment: No worries I probably misread your tone.
Comment: yes but if x and y could be factored by a common divisor, $ax'+by'$ would be a solution. So if $t|-1$ and $t|2$, then we would have a smaller solution $-1/t$ and $2/t$. A contradiction? Although I'm not sure if that would prove anything...
Comment: @pushpen.paul for any $a,b$ that $\gcd(a,b) = ax+by \Rightarrow \gcd(x,y) = 1$
Comment: @pushpen.paul I want to prove the implication whenever $\gcd(a,b) = d$, for any $d$. Is that false?
Anyways, people are more likely to correct something wrong than they're willing to prove something right. I was pretty sure my proof was flawed and quickly got a lot of responses right away.
Comment: @pushpen.paul ah thanks now I understand what you meant... anyways thanks for your time and help!
Answer: The statement is probably that there exists such a pair of values $x,y$ that $\gcd(a,b)=ax+by$. Therefore, you cannot say that $ax'+by' = \gcd(a,b)$, your mistake comes from this replacement.


Filter dataframe by finding keyword inside a chunk of text

Question: I'm working with Twitter data in a dataframe. I want to filter the column that holds the text of each tweet according to a certain keyword found within the text.
I've tried str.contains but that doesn't work, as the column is a Series. I want to filter the "text" column for all the tweets containing the keyword 'remoaners'.
remoaners_only = time_plus_text[time_plus_text["text"].str.contains("remoaners", case=False, na=False)]

This produces either an empty dataframe or a lot of NaNs.
pandas version is 0.24.1.
Here's the input data: time_plus_text["text"].head(10)

0    [ #bbcqt Remoaners on about post Brexit racial...
1    [@sarahwollaston Shut up, you like all remoane...
2    [ what have the Brextremists ever done for us ...
3                     [ Remoaner in bizarre outburst ]
4    [ Anyone who disagrees with brexit is called n...
5    [ @SkyNewsBreak They forecasted if the vote wa...
6    [ but we ARE LEAVING THE #EU, even the #TORIES...
7    [ Can unelected Remoaner peers not see how abs...
8    [@sizjam68 @LeaveEUOfficial @johnredwood It wo...
9    [ Hey @BBC have you explained why when award w...
Name: text, dtype: object

Comment: Welcome on stackoverflow, it would be easier for people to help you if you share some input data to be able to reproduce your problem. Here it would be the first few lines of your column `time_plus_text["text"]`. See [How to create a Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve) for more details
Comment: Could you add sample data? This should work I think. Which pandas are you using?
Comment: updated, thanks for tips, hope its clearer now
Answer: Your code works.  So you are left to check your input data or the pandas bug fix version, 0.24.1 vs 0.24.2.
0.24.2
   index                                               text
0      0     [ #bbcqt Remoaners on about post Brexit rac...

import pandas as pd
import sys
if sys.version_info[0] < 3:
    from StringIO import StringIO
else:
    from io import StringIO

print(pd.__version__)

csvdata = StringIO("""0,   [ #bbcqt Remoaners on about post Brexit racial...
1,   [@sarahwollaston Shut up, you like all remoane...
2,   [ what have the Brextremists ever done for us ...
3,                    [ Remoaner in bizarre outburst ]
4,   [ Anyone who disagrees with brexit is called n...
5,   [ @SkyNewsBreak They forecasted if the vote wa...
6,   [ but we ARE LEAVING THE #EU, even the #TORIES...
7,   [ Can unelected Remoaner peers not see how abs...
8,   [@sizjam68 @LeaveEUOfficial @johnredwood It wo...
9,   [ Hey @BBC have you explained why when award w...""")

df = pd.read_csv(csvdata, names=["index", "text"], sep=",")

result = df[df["text"].str.contains("remoaners", case=False, na=False)]

# results
print(result)

Answer: The problem is that the string you want to find the substring remoaners in is contained in a list in each cell. You need to access this string by doing str[0] before doing str.contains such as:
# input
time_plus_text = pd.DataFrame({'text':[['#bbcqt Remoaners on about post Brexit racial...'],
                                       ['@sarahwollaston Shut up, you like all remoaners...'],
                                       ['what have the Brextremists ever done for us ...']]})
print (time_plus_text["text"].str[0].str.contains("remoaners", case=False, na=False))
0     True
1     True
2    False
Name: text, dtype: bool

so you should do:
remoaners_only = time_plus_text[time_plus_text["text"].str[0]\
                                             .str.contains("remoaners", case=False, na=False)]


(Euclid's Algorithm) What does b←b-a and a←a-b mean?

Question: In the diagram below, what do steps 4 and 6? Is it the same as ⇒ where if the right is true, the left is true as well?
Diagram:
https://upload.wikimedia.org/wikipedia/commons/thumb/d/db/Euclid_flowchart.svg/399px-Euclid_flowchart.svg.png
Comment: "Replace $b$ with $b-a$" and "Replace $a$ with $a-b$."
Comment: Welcome to Mathematics Stack Exchange.  It looks like "Let the new $B$ be the old $B$ minus $A$
Answer: If you're familiar with a coding language like Python, C, or Java, $a \gets b$ is equivalent to the statement $a = b$ in one of those languages.
If you're not familiar with those languages, allow me to give a brief explanation.
When one discusses algorithms, one uses variable names like $a$ and $b$ to represent the "contents of a box". Instead of $a$ representing a fixed value, $a$ is really the name of a box which holds a value.
When we want to use the value contained in box $a$, we refer to this value as $a$. For example, if one wrote the expression $2 \cdot a$, evaluating this expression would require you to look inside box $a$, get the value, and multiply that value by 2.
However, on some occasions, one might want to change the value inside box $a$. To do this, we use an "assignment statement" of the form $a \gets E$, where $E$ is some expression.
For example, in the pseudocode
<code>a <- 1;
b <- a + 1;
a <- b;
b <- a + 1;
</code>
the final value in box $a$ is 2, and the final value in box $b$ is 3.
Answer: What it amounts to is finding the difference between $A,B$ and then repeating the pattern with the "smaller of $A,B$" and the "difference" until, effectively, $A-B=0$ e.g. $GCD(20,15)$.
$$
20-15=5\\
15-5=10\\
10-5=5\\
5-5=0$$
So $GCD(20,15)=5$
A faster implementation requiring only $2$ iterations for this example uses "mod" as shown in the BASIC below.
<code>  100 input i1
  110 input i2
  120 x1 = i1
  130 x2 = i2
  140 r1 = x1 mod x2
  150 c1 = c1+1
  160 if r1 > 0
  170    x1 = x2
  180    x2 = r1
  190 goto 140
  200 endif
  210 print c1 "iterations",;
  220 print "GCD(" i1 i2 ") = " x2

</code>


Playing legato while staccato in Rachmaninoff's "Etude-Tableaux Op. 33"

Question: In Rachmaninoff's 'Étude-Tableaux Op.33 No.4 in D-minor' there are plenty of places where you are supposed to play staccato while playing legato in the left hand (for example, measures 32-34 below). How is this really executed the way he intended it (particularly also the pedaling)?
Thank you for your answers.
Edit: Forgot to say, I know that the top voice must be legato, but I don't understand how this is possible, while using the pedal and playing staccato. Especially since in measure 33 it goes over 2 octaves.
Comment: There are reasons why this particular Etude-Tableau is a posthumously published one.
Comment: In my book this is Op.33 No5 (not 4). Maybe someone can also suggest how to play the passages like m.37.
Even "Rachmaninov's legendary handspan" is insufficient to keep that 2-octave plus third with the left hand. In my score it even says "m.d." there. You have to KEEP that chord while playing the followingRH notes, which are again staccato in m.38.In 37 they are not staccato marked but I think you should not use pedal for the RH there to make it sound ok
Answer: As @Howlium says, the legato applies only to the "melody line" in the top voice of the left hand, not to the staccato chords. As for execution...
Execution
EDIT: It's significant that the chords in measure 32, from the beginning of beat 2 until the "grace-chord" on 4, all lack staccato markings.
Alexander Kobrin
In addition to the legato melody over staccato chords, there are particular problems in the left hand's tied D4 on m. 32 beats 2-3, and the tied C4 in m. 33, beats 1-2.
In Alexander Kobrin's performance on YouTube, you can watch his hands in this section. The relevant portion is from 1:40 - 1:49. The entire LH "singing" line is executed by Kobrin with an accented staccato articulation, using the sustain pedal to create the legato. Note that in the case of D4, he re-executes it on  beat 3, but he does not re-execute the C4 in the following measure.
Sviatoslav Richter
There is also a "synthesiafied" Richter performance. The color-bars in mm. 32-34 suggest two possibilities:

That his hands are big enough, for example, in m. 32 to play the beat 2 chord with his thumb on D4, then cross fingers 2 and 3 over the thumb to execute the chord on the half-beat. This technique would work for the remainder of the passage, excepting m. 33 beat 2.
On the D4, there are slight breaks in the color-bar, so perhaps he's actually re-articulating the note very quietly. (Pretty unbelievable feat, IMO, and less likely, but it's Richter.)

He does seem to use very light sustain pedal in that passage, though not so much in the piece as a whole.

EDIT
Me
Giving this passage a try (but short of performance speed), I would most likely execute it by changing the pedal just after I play each melody note. So, for example:
Beat 2: play all four notes with pedal, but near immediately release the pedal and all notes but D4, then reapply pedal to catch D4 (which I would continue to hold with my thumb).
Beat 2&: play chord with 5-1-2, releasing the pedal, but again holding onto the D4 and immediately reapplying the pedal.
Beat 3: play the 2-note chord, then release all three keys, but keep the pedal until playing the chord on ...
Beat 3&: same technique of catching the melody note with the pedal applies here and after.
Beat 4: keep the pedal through the "grace chord", but change it upon playing the F4.
And so forth...
Comment: @Matriz Just added a couple of edits: one observing that in m. 32 the chords under the ties/legato markings are not marked staccato, and a second attempting to describe how I'd execute the passage.
Comment: Thank you for the answer. I watched Kobrin's version and I am not sure, what he is actually doing with the pedaling. Does he slightly lift it while execute the staccato part but not fully?

Not sure if I understood what you wrote about the Richter correctly.
Comment: Did you mean by re-articulating the D4 just the top voice?
Comment: @Matriz Kobrin's pedaling is difficult to discern, but it sounds to me like he's either half-pedaling, likely clearing the pedal between "melody" notes. In Richter's performance, I'm suggesting he articulates the tied D4, for example, 3 times: the initial attack, the chord on the half-beat, and quietly again on beat 3. Truthfully this seems highly unlikely; I'm reacting to the fact that the Synthesia score indicates slight breaks in the note.
Answer: The legato mark applies to the phrase in the upper voices in the bass clef, not to the bass notes that have the staccato marks.
Comment: Thanks, but how exactly has one to do that, I mean you need to play the pedal for playing the upper voice legato, while playing the bottom voice staccato?
Comment: You wouldn't need a pedal if you have monstrously large hands. Rachmaninov's legendary handspan was an octave plus a fifth. Personally, I would need a pedal or a third hand.
Answer: Half pedalling is a way to get round this. It involves lifting the dampers partially off the strings, which means there is some sustain, but any notes played staccato still sound shortened.


Why am I getting this 403 Forbidden error?

Question: SOLUTION
Apache needs at least execute permissions on all the parent directories of the vhost dir.

I'm trying to set up a Rails app running under Apache on CentOS 4.3. I'm using Phusion Passenger (mod_rails) I made sure the app deployment dir is owned by group Apache and the permissions are set to allow the group to read it.
In my Apache config I have...
<VirtualHost *:80>

  ServerAdmin firstname.lastname@example.com
  DocumentRoot "/home/sneedmoure/deployed_rails_apps/myapp/current/public"
  ServerName myapp.com
  ServerAlias www.myapp.com
  ErrorLog "logs/myapp.com-error_log"
  CustomLog "logs/myapp.com-access_log" common
  CustomLog "logs/myapp.com-deflate_log" deflate

  <Directory "/home/sneedmoure/deployed_rails_apps/myapp/current/public">
    Order allow,deny
    Allow from all
#AllowOverride all       # Not sure whether I need this
    Options -MultiViews  # Passenger docs say to include this
  </Directory>

</VirtualHost>

When I try to access my app in the browser I get a 403 error.
In the error log I'm getting entries like...
[Wed Nov 25 03:01:19 2009] [error] [client 123.456.789.12] (13)Permission denied: access to / denied

I've tried different paths within the app and get the same result. It does work fine on my development machine.
Any notion what I'm doing wrong?
Answer: The only reason I can think of is that the web server user doesn't have access to that directory (/home/sneedmoure/deployed_rails_apps/myapp/current/public).
You could verify that by su-ing to the web server user account and trying to list the directory.
Comment: Can't su to user apache. Apparently not a type of user that can log in. However, I'm pretty sure the permissions are OK.
Comment: To su to it you'd need to edit /etc/passwd and temporarily change its shell to /bin/bash (or whatever your normal shell is).
Comment: I moved it to a different dir under /. I think the deal was that user apache needs read and execute on the entire line of dirs leading to the application.
Comment: It would need execute permissions on the entire directory tree, yes. Read permissions would only be needed on any web directories you want to list files in.
Answer: Is there an index file (e.g. index.html, index.htm, etc) in that directory?  If there isn't, it could be complaining b/c any request for that dir appears as an attempt to retrieve the directory listing.  Can you create a blank file in that directory (e.g. test.txt) and successfully get it to load in a browser?
Comment: Good thought. But yes, there is an index file.


Finding primary decomposition of module over a PID

Question: I've been studying the decompositions of a finitely generated module over a PID (the structure theorem), and appyling them successfully to obtain decompositions of finite abelian groups (finding all groups of order $n$ up to isomorphism, for example).
However, I found this exercise which I haven't been able to solve:
Find the primary decomposition of the $\mathbb{Z}$-module $M$, and describe its elementary divisors, where $M=\mathbb{Z^2}/K$, and $K$ is the ideal generated by $(6,6)$, $(3,6)$, $(3,12)$.
By the theorem, I know $M \simeq \mathbb{Z}^r$ $ \oplus$ $(\bigoplus_i \mathbb{Z}/(q_i))$, where the $q_i$ are prime powers. Also, $M = T(M)$, so I know $r=0$. Any hints or help finding the $q_i$ would be appreciated, I don't know where I could start.
Comment: Do you know how to use the Smith normal form of a matrix for finding the invariant factors of a f.g. module over a PID?
Comment: Not really, in fact this is the first time I've heard of it. I searched around a bit, doesn't seem too complicated. For which matrix should I find the Smith normal form? I do know how to find the elementary divisors, after finding the primary factors.
Comment: For the matrix having the rows (or columns) the three generators.
Comment: Thanks! I'll look into this more, seems like it would be especially useful when dealing with a bigger module.
Answer: In this case inspection yields that
$-(6,6)+3(3,6)=(3,12)$, so the last generator is superfluous.
Then $(6,6)-(3,6)=(3,0)$, and then $(3,6)-(3,0)=(0,6)$.
So, the same subgroup is generated by $(3,0)$ and $(0,6)$, and $K=3\Bbb Z\times 6\Bbb Z$
Then the quotient $\frac{\Bbb Z\times \Bbb Z}{3\Bbb Z\times 6\Bbb Z}\cong\frac{\Bbb Z}{3\Bbb Z}\times \frac{\Bbb Z}{6\Bbb Z}$ via a likely map. You said you know how to take it from here, right?
Of course, behind the secenes this is just an ad-hoc Smith Normal form.
Comment: Thanks, I think this is the way we were meant to do it, since we didn't cover the Smith normal form in class. Now that we have that, the elementary divisors must be 3, 3, and 2, right?
Comment: Hmm, isn't $\Bbb Z_3\times \Bbb Z_6\cong \Bbb Z_3\times \Bbb Z_3 \times \Bbb Z_2 $?
Comment: @Mauro Of course you're right. For some reason I thought I had to collapse them into two and get distinct prime powers. That isomorphism is correct.
Comment: Great then, thanks again!


CSS refuse to obey

Question: I'll make this short, I got this webpage :

    <link rel="Stylesheet" type="text/css" href="css/style.css" />
    <link rel="Stylesheet" type="text/css" href="css/nav.css" />

foo

bar

With this CSS
style.css
body
{
    padding:0;
    margin:0;
    background-color:#000000;
    background-repeat:no-repeat;
    background-image:url('../img/bg.jpg');
}

nav.css
#navList
{
    padding:0;
    margin:0;
    background-image:url('../img/menu.png');
    list-style-type:none;
    padding:12px 150px;
}

#navList li
{
    display:inline;
}

#navList li a
{
    color:#bfbfbf;
    padding:14px 25px;
    text-decoration:none;
}

#navList li a:hover
{
    color:#000000;
    background-color:#bfbfbf;
    text-decoration:none;
}

#navActive
{
    color:#000000;
    background-color:#bfbfbf;
}

It looks like the CSS from the navActive id is never being applied... Could someone tell me why and/or suggest me a way to correct this.
Thanks.
Comment: Looks o.k. to me. Do you happen to have a live example? A quick look with Firebug usually solves things like this.
Comment: Can you try using `#navList li a#navActive` instead of `#navActive` as the selector? I think using an ID *should* always have precedence but I'm not sure.
Answer: It's because the selector
#navList li a

is more heavily weighted than
#navActive

as it is more specific.
You can overcome it by adding
color:#000000 !important;

or using
#navList li a#navActive

as the selector
Comment: I thought that an ID selector has the highest specificity, is this not correct?
Comment: Alex is correct. The scores are added together, so an id with 2 tags is more specific than an id by itself.
Comment: Is the "!important" tag compatible with every browser?
Answer: Try this:
#navActive
{
    color:#000000 !important;
    background-color:#bfbfbf;
}

Here's a demo.  It's a specificity fight, in this case the original #navList li a is winning.  Another alternative is using #navList li a#navActive without the !important.

CSS refuse to obey


Relationship between ridge regression and PCA regression

Question: I remember having read somewhere on the web a connection between ridge regression (with $\ell_2$ regularization) and PCA regression: while using $\ell_2$-regularized regression with hyperparameter $\lambda$, if $\lambda \to 0$, then the regression is equivalent to removing the  PC variable with the smallest eigenvalue.

Why is this true?
Does this have anything to do with the optimization procedure? Naively, I would have expected it to be equivalent to OLS.
Does anybody have a reference for this?
Comment: Did you notice that p. 14 of that latest reference explicitly answers your question?
Comment: Your reference appears to demonstrate the opposite of what you are saying in your comments: for small $\lambda$, there is very little change in the results.  Nothing is removed at all.  In fact, several slides seem aimed at pointing out the difference between $L^2$ penalized regression (in which estimates are shrunk towards $0$) and "PCA regression" (in which the smallest components are entirely removed--which can be a very bad thing in some circumstances).
Comment: The connection (as I understand), is that if you use a very very small  regularization penalty, an L2-regularized regression would be removing the variable that has the smallest eigenvalue.

Therefore, doing SVD on the design matrix, and removing the variable with the smallest eigenvalue is equivalent to a regression with a "soft" regularization penalty...

This is the closest explanation I've found to this: http://sites.stat.psu.edu/~jiali/course/stat597e/notes2/lreg.pdf
Comment: Mmm.. found another reference:
http://statweb.stanford.edu/~owen/courses/305/Rudyregularization.pdf

In the slide, "$y^{ridge}$ and principal components", it says that ridge regression projects y onto these components with
large dj

* sigh *
Comment: Could you explain more explicitly how PCA and regression are connected in your statement? Regression distinguishes dependent from independent variables, whereas nothing of the sort occurs in PCA. So what variables are you applying PCA to?  It cannot be just the independent variables, for that would have little to do with the regression. But if it's applied to all the variables, then the eigenvectors are linear combinations of them all. What could it possibly mean to remove *any* such component from the dataset, since it involves the dependent variable?
Comment: But you can see that in practice this happens.  If you take a fully-singular design matrix (say you have both "dummies" for male and female) and you use an L2-regularization implementation, and set the hyper-parameter to 0, it still comes up with an answer, where none should exist in "vanilla" regression
Comment: That appears to be a different question altogether, having to do with collinearity, identifiability, and conventions for solving the normal equations.
Answer: Let $\mathbf X$ be the centered $n \times p$ predictor matrix and consider its singular value decomposition $\mathbf X = \mathbf{USV}^\top$ with $\mathbf S$ being a diagonal matrix with diagonal elements  $s_i$.
The fitted values of ordinary least squares (OLS) regression are given by $$\hat {\mathbf y}_\mathrm{OLS} = \mathbf X \beta_\mathrm{OLS} = \mathbf X (\mathbf X^\top \mathbf X)^{-1} \mathbf X^\top \mathbf y = \mathbf U \mathbf U^\top \mathbf y.$$ The fitted values of the ridge regression are given by $$\hat {\mathbf y}_\mathrm{ridge} = \mathbf X \beta_\mathrm{ridge} = \mathbf X (\mathbf X^\top \mathbf X + \lambda \mathbf I)^{-1} \mathbf X^\top \mathbf y = \mathbf U\: \mathrm{diag}\left\{\frac{s_i^2}{s_i^2+\lambda}\right\}\mathbf U^\top \mathbf y.$$ The fitted values of the PCA regression (PCR) with $k$ components are given by $$\hat {\mathbf y}_\mathrm{PCR} = \mathbf X_\mathrm{PCA} \beta_\mathrm{PCR} = \mathbf U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\}\mathbf U^\top \mathbf y,$$ where there are $k$ ones followed by zeroes.
From here we can see that:

If $\lambda=0$ then $\hat {\mathbf y}_\mathrm{ridge} = \hat {\mathbf y}_\mathrm{OLS}$.

If $\lambda>0$ then the larger the singular value $s_i$, the less it will be penalized in ridge regression. Small singular values ($s_i^2 \approx \lambda$ and smaller) are penalized the most.

In contrast, in PCA regression, large singular values are kept intact, and the small ones (after certain number $k$) are completely removed. This would correspond to $\lambda=0$ for the first $k$ ones and $\lambda=\infty$ for the rest.

This means that ridge regression can be seen as a "smooth version" of PCR.
(This intuition is useful but does not always hold; e.g. if all $s_i$ are approximately equal, then ridge regression will only be able to penalize all principal components of $\mathbf X$ approximately equally and can strongly differ from PCR).

Ridge regression tends to perform better in practice (e.g. to have higher cross-validated performance).

Answering now your question specifically: if $\lambda \to 0$, then $\hat {\mathbf y}_\mathrm{ridge} \to \hat {\mathbf y}_\mathrm{OLS}$. I don't see how it can correspond to removing the smallest $s_i$. I think this is wrong.

One good reference is The Elements of Statistical Learning, Section 3.4.1 "Ridge regression".

See also this thread: Interpretation of ridge regularization in regression and in particular the answer by @BrianBorchers.
Comment: Would it ever make sense to soft-threshold the singular values,
max( $s_i -$ thresh, 0 ) ?
(Lasso regression soft-thresholds $\beta_{Least-squares}$, not the spectrum.)
Comment: One correction to your otherwise great answer: the fitted values in the regression on the first $k$ PC's are actually
$$
\mathbf{U} {\text{diag}}(1_1,1_2,...,1_k,0,...,0)\mathbf{U}^T\mathbf{y}
$$
This is an exercise at the end of the chapter that you mention.
Comment: This is beautiful.
Comment: I don't think using singular value as the criterion for component removal is a good idea in the context of regression. Instead, you should apply regression with all principal components first then remove the components that don't contribute much to the fitting. See also: [Examples of PCA where PCs with low variance are "useful"](https://stats.stackexchange.com/questions/101485/examples-of-pca-where-pcs-with-low-variance-are-useful)
Answer: Elements of Statistical Learning has a great discussion on this connection.
The way I interpreted this connection and logic is as follows:

PCA is a Linear Combination of the Feature Variables, attempting to maximize the variance of the data explained by the new space.
Data that suffers from multicollinearity (or more predictors than rows of data) leads to a Covariance Matrix that does not have full Rank.
With this Covariance Matrix, we cannot invert to determine the Least Squares solution; this causes the numerical approximation of the Least Squares Coefficients to blow up to infinity.
Ridge Regression introduces the penalty Lambda on the Covariance Matrix to allow for matrix inversion and convergence of the LS Coefficients.

The PCA connection is that Ridge Regression is calculating the Linear Combinations of the Features to determine where the multicollinearity is occurring. The Linear Combinations of Features (Principle Component Analysis) with the smallest variance (and hence smaller singular values and smaller eigenvalues in PCA) are the ones penalized the hardest.
Think of it this way; for the Linear Combinations of Features with smallest variance, we have found the Features that are most alike, hence causing the multicollinearity. Since Ridge does not reduce the Feature set, whichever direction this Linear Combination is describing, the original Feature corresponding to that direction is penalized the most.
Answer: Consider the linear equation
$$
  \mathbf X \beta =  \mathbf y\,,
$$
and the SVD of $\mathbf X$,
$$
\mathbf X =  \mathbf U \,\mathbf S \,\mathbf V^T,
$$
where $\mathbf S = \text{diag}(s_i)$ is the diagonal matrix of singular values.
Ordinary least squares determines the parameter vector $\beta$ as
$$
  \beta_{OLS} =  \mathbf V \,\mathbf S^{-1} \,\mathbf U^T \, \mathbf y
$$
However, this approach fails as soon there is one singular value which is zero (as then the inverse does not exists). Moreover, even if no $s_i$ is excatly zero, numerically small singular values can render the matrix ill-conditioned and lead to a solution which is highly susceptible to errors.
Ridge regression and PCA present two methods to avoid these problems. Ridge regression replaces $\mathbf S^{-1}$ in the above equation for $\beta$ by
\begin{align}
\mathbf S^{-1}_{\text{ridge}} &= \text{diag}\bigg(\frac{s_i}{s^2_i+\alpha}\bigg),\\
\beta_{\text{ridge}} &= \  \mathbf V \,\mathbf S_{\text{ridge}}^{-1} \,\mathbf U^T \, \mathbf y
\end{align}
PCA replaces $\mathbf S^{-1}$ by
\begin{align}
\mathbf S^{-1}_{\text{PCA}} &= \text{diag}\bigg(\frac{1}{s_i} \, \theta(s_i-\gamma)\bigg)\,,\\
\beta_{\text{PCA}} &= \  \mathbf V \,\mathbf S_{\text{PCA}}^{-1} \,\mathbf U^T \, \mathbf y
\end{align}
wehre $\theta$ is the step function, and $\gamma$ is the threshold parameter.
Both methods thus weaken the impact of subspaces corresponding to small singular values. PCA does that in a hard way, while the ridge is a smoother approach.
More abstractly, feel free to come up with your own regularization scheme
$$
\mathbf S^{-1}_{\text{myReg}} = \text{diag}\big(R(s_i)\big)\,,
$$
where $R(x)$ is a function that should approach zero for $x\rightarrow 0$ and $R(x)\rightarrow x^{-1}$ for $x$ large. But remember, there's no free lunch.
Comment: Where is $y$ in your estimators ? Isn't it $\beta_{ols} = \mathbf{VS^{-1}V}^\intercal y$?
Comment: Also, could you explain how you found $\mathbf{S}^{-1}_{PCA}$?
Comment: @Cryckx: sure, in all betas the right-hand side is missing. I'll update the question. The PCA stuff is rarely seen in this notation -- usually the recipe goes as "drop all entries of the pseudoinverse that correspond to singular values below a chosen threshold". The heaveside step function is just a mathematical notation for that.


Vulnerability index for every node of graph

Question: Given a particular undirected graph with n nodes, is there an index that would characterize the vulnerability of each of the nodes? By vulnerability I mean the susceptibility of the graph to fragmentation.
Let's say that I remove a particular node i which results in the graph being fragmented into five parts; and when I remove node j it results in the graph being fragmented into two parts (and there would be nodes which would not fragment the graph)...is there an index which gives me this information for all the nodes?
The goal of the above is to identify vulnerable nodes in a graph (made from healthy subjects) and look at node properties of these nodes in another graph (made from patients suffering from various medical conditions).
For the sake of completeness, my nodes are representatives of different brain regions and the edges between these nodes represent correlation coefficient between some form of time series from each of these brain regions. However, the problem can be generalized to other contexts (the "key player" problem in graph theory...identifying key people in a social network, for example).
Comment: There are metrics for identifying *cut-sets*. [Moody and White's](https://networkx.github.io/documentation/latest/reference/generated/networkx.algorithms.connectivity.kcomponents.k_components.html#networkx.algorithms.connectivity.kcomponents.k_components)  algorithm was intended for graph level metrics, but nodes in leaves farther down are less vulnerable. If you start from there, how many separate connected components in the next stage down would be the fragmentation, and the those at further down levels would be less vulnerable.
Comment: You may just find the cut-set's though for the entire graph, and then count up the separate components when they are removed compared to the whole graph, that is pretty simple.
Answer: I think you should have a look at the block-cut tree of our graph. The number of biconnected components a node is contained in is exactly the number of fragments the graph decomposes into when this node is removed. Or, put differently, your index is the degree of that node in the block-cut tree.


Como ler yaml com php

Question: Olá, estou abrindo um servidor de Minecraft (jogo java) e me deparei com a seguinte necessidade:
Para cada servidor de minecraft é necessário a construção de um site informativo. Ao construir o site, criei uma pagina simples apenas com a lista da equipa do servidor. A equipa é constituída por "Ajudantes", "Moderadores" e "Administradores". Para atribuir um destes cargos a um jogador, é necessário introduzir um comando dentro do jogo, que logo este será validado no ficheiro config.yml do servidor.
Este é o ficheiro config.yml que define o estatuto dos jogadores:
users:
      JamesMarch:
        group:
        - Admin
      TheCountess_:
        group:
        - Admin
      mumiant_:
        group:
        - Moderador
      Pequena20:
        group:
        - Ajudante
      nana34piu:
        group:
        - Ajudante

O que eu estava pensando era criar um "leitor" em php, que lesse o código yaml e os cargos dos jogadores no meu site do seguinte jeito:

        <title>Alderaan Minecraft Server</title>

Staff

Administradores

JamesMarch

TheCountess_

Moderador

mumiant_

Ajudantes

Pequena20

nana34piu

Gostaria ainda que em cada paragrafo da staff aparecesse o nome de cada membro, por ordem alfabética, exemplo:

Administradores

JamesMarch
TheCountess_

e não

Administradores

TheCountess_
JamesMarch

Gostaria também que, por exemplo, se houver apenas um Administrador, apareça "Administrador" e não "Administradores", exemplo:
Casa haja um administrador:

Administrador

JamesMarch

Casa hajam dois ou mais administradores:

Administradores

JamesMarch
TheCountess_

Tambem queria que fosse possivel aparecer apenas os tres cargos no web site, uma vez que o site possui mais que tres cargos, eu gostava que apenas os cargos de "Administrador", "Moderador" e "Ajudante" aparecessem na pagina.
Por ultimo, queria que aparecessem os cargos pela hierarquia, exemplo:
Certo:

Administradores

JamesMarch
TheCountess_

Moderador

mumiant_

Ajudantes

nana34piu
Pequena20

Errado:

Ajudantes

nana34piu
Pequena20

Administradores

JamesMarch
TheCountess_

Moderador

mumiant_
Answer: Instalando leitores de Yml
Se o teu servidor permitir instalar algo você pode instalar/compilar o http://pyyaml.org/wiki/LibYAML, baixe http://pyyaml.org/download/libyaml/yaml-0.1.7.tar.gz e depois compile:
$ ./configure
$ make
# make install

E depois instale via PECL, pode baixar aqui http://pecl.php.net/package/yaml
Symfony
Você pode instalar o pacote do Symfony usando composer, execute no seu projeto isto:
composer require symfony/yaml

Ou adicione no seu arquivo composer.json o symfony/yaml e depois execute:
composer update

Para usar faça isto:
use Symfony\Component\Yaml\Yaml;
use Symfony\Component\Yaml\Exception\ParseException;

try {
    $value = Yaml::parse(file_get_contents('arquivo.yml'));
} catch (ParseException $e) {
    printf("Impossivel decodificar o YAML: %s", $e->getMessage());
}

Lendo o yml da pergunta
O teu .yml parece estar em um formato inválido, suponho que o correto seria isto:
users:
    JamesMarch:
        group:
        - Admin
    TheCountess_:
        group:
        - Admin
    mumiant_:
        group:
        - Moderador
    Pequena20:
        group:
        - Ajudante
    nana34piu:
        group:
        - Ajudante

O resultado deve ser mais ou menos isto:

Note que tive que reorganizar o array em um estrutura para poder montar o HTML mais facilmente, fazendo o uso da variável $porcargo

<?php

use Symfony\Component\Yaml\Yaml;
use Symfony\Component\Yaml\Exception\ParseException;

$conteudo = file_get_contents('config.yml');

include 'vendor/autoload.php';

$porcargo = array();

try {
    $ymlParsed = Yaml::parse($conteudo);
    foreach ($ymlParsed['users'] as $nome => $cargos) {
        $grupo = $cargos['group'];
        $j = count($grupo);

        for ($i = 0; $i < $j; $i++) {
            if (!isset($porcargo[ $grupo[$i] ])) {
                $porcargo[ $grupo[$i] ] = array();
            }

            $porcargo[ $grupo[$i] ][] = $nome;
        }
    }
} catch (ParseException $e) {
    die(sprintf("Unable to parse the YAML string: %s", $e->getMessage()));
}
?>

        <title>Alderaan Minecraft Server</title>

Staff

            <?php foreach ($porcargo as $cargo => $usuarios): ?>

<?php echo $cargo; ?>

                <?php for ($i = 0; $i < count($usuarios); $i++): ?>

<?php echo $usuarios[$i]; ?>

                <?php endfor; ?>

            <?php endforeach; ?>

Comment: [yaml-parse-file](http://php.net/manual/pt_BR/function.yaml-parse-file.php) é a forma estruturada de Yaml::parse correto?
Comment: @Everson Creio eu que `Yaml::parse` é um pacote de symfony independente do `LibYAML` e do http://pecl.php.net/package/yaml, de uma olhada no source-code: https://github.com/symfony/yaml/blob/master/Parser.php#L88

Como ler yaml com php


Fixed effect tests

Question: I have done a fixed effect regression after doing the Housman test. I am curious what tests should I use to see whether the results of regression are accurate.
Comment: Check the other model assumptions, see e.g. Wooldridge's introductory text book.
Answer: The type of tests you might want depend somewhat on the type of regression you've done (e.g. linear, generalized linear (logistic, poisson, etc.)).
In linear regression, people often use the R2 as a measure of how much variance the data explains.  This is read as a proportion and can be thought like a percentage, e.g. if R2 = 0.7 ~ 'model explains 70% of the variance in the data'.
There are measures that apply to other types of regression, e.g. confusion matrices, or the C-statistics (Area Under the Receiver Operator Characteristic/AUC) for logistic regression.  The C-statistic can be intercepted in a similar way to the R2.
Steer clear of measures such as AIC if you want to know how 'accurate' you model is.  These are very useful for comparing similar models, but don't tell you how accurate they are.
I'd suggest plotting your observed and fitted values.  If it predicts well they should form a (fairly) straight line with little spread.  It's also worth plotting the fitted values and residuals (but read up about what residuals are).
I'd recommend the brilliant UCLA stats pages on the following link.  They have annotated examples of lots of regressions/analyses in a various stats programs including Stata, R and SAS.
https://stats.idre.ucla.edu/other/annotatedoutput/
Answer: Thanks for your answer, info was very useful. I have done a fixed effect linear panel regression in stata. What I want to do is to run additional tests to check whether the regression is accurate. This is my first time with panel regression. For example when I did an OLS regression I run several tests (VIF,hettest and so on. I would like to know what similar tests are appropriate for fixed effect regression.

Fixed effect tests


Is $2^\sqrt{2}$ is rational or irrational?

Question: There are many popular problems on irrational power rational. A very interesting read is this blog. And there are many specific examples where rational power irrational is rational (e.g. $2^{\log_2(3)}$). But what about $2^{\sqrt{2}}$? Is it rational or irrational?
Comment: @N74 Yes, indeed, but also very popular with hundreds of [duplicates](https://math.stackexchange.com/questions/446647/irrationality-of-sqrt2-sqrt2?noredirect=1&lq=1).
Answer: According to the Gelfond-Schneider theorem, it is transcendental (and, in particular, irrational).


use a custom textgroup from t()

Question: I've defined my own textgroup using:
function foo_locale($op = 'groups') {
  switch ($op) {
    case 'groups':
      return array('foo' => t('foo strings'));
  }
}

The problem is that when I use t('bar') in a template, the entry generated in the database (e.g when viewing the template in /fr) is associated with the default textgroup. This doesn't allow me to use the foo textgroup I've previously defined, thus all my custom strings are mixed with the other "Built-in interface" ones.
How can I solve this?
Answer: t('String') always belongs to the "Built-in interface". If you have t() texts, then you don't need to create textgroups.
They are meant for strings that are not translated with that function, usually content/configuration. The I18n modules define lots of these groups for things like variables, menus, taxonomy terms and so on.
Answer: Since the t() only for UI and you can not select which exactly the grouptext that you prefer. You can use this:
$context = $field_name . ':' . $entity_type . ':label';
$label = i18n_string_translate('field:' . $context, $field_info['label']);

Example:
$context = 'field_custom:article:label';
$label = i18n_string_translate('field:' . $context, 'Custom title');

You have to prefix the context with "field:"
Answer: As others have mentioned use i18n module, the specific function is
i18n_string_translate($name, $string, $options)

This is part of i18n_string module. Here pass $textgroup . $context as $name, the string to be translated as $string. You can also pass $options['langcode'] if translation in a specific language is desired.
Answer: You can add strings to a custom text-group by first extracting them to a POT file. Then when you import them at /admin/config/regional/translate/import you can assign the strings to your text-group.
Or for user-defined strings you could use the i18n module.
NOTE: Text-groups will be removed in Drupal 8. See http://drupal.org/node/1188430
Comment: This will work, but I do not prefer this way since it will duplicate the phrases and it is not the best solution. Thanks for your help.

use a custom textgroup from t()