Nginx, second WordPress site not loading, could the issue be with sites config file?

Question: I am trying to install a second WP site (dancortes.press) on an Ubuntu 16 server running Nginx. The first site (microurb.club) has been successfully installed. I am not having a problem with that one.
I have successfully installed WordPress according to my server results for the second website (dancortes.press), but I still only get the Nginx home page. I thought it might be the fact that the Group Owner was the User, so I changed group owner on web files to www-data:
microurb@vps148370:/var/www/dancortes.press/public_html$ ls -l
total 192
-rw-r--r--  1 microurb www-data     0 Oct  5 15:36 index.html
-rw-rw-r--  1 microurb www-data   418 Oct  8 17:07 index.php
-rw-rw-r--  1 microurb www-data 19935 Oct  8 17:07 license.txt
-rw-rw-r--  1 microurb www-data  7413 Oct  8 17:07 readme.html
-rw-rw-r--  1 microurb www-data  5447 Oct  8 17:07 wp-activate.php
drwxrwxr-x  9 microurb www-data  4096 Oct  8 17:07 wp-admin
-rw-rw-r--  1 microurb www-data   364 Oct  8 17:07 wp-blog-header.php
-rw-rw-r--  1 microurb www-data  1627 Oct  8 17:07 wp-comments-post.php
-rw-rw-r--  1 microurb www-data  2599 Oct  8 17:17 wp-config.php
-rw-rw-r--  1 microurb www-data  2853 Oct  8 17:07 wp-config-sample.php
drwxrwxr-x  5 microurb www-data  4096 Oct  8 17:19 wp-content
-rw-rw-r--  1 microurb www-data  3286 Oct  8 17:07 wp-cron.php
drwxrwxr-x 18 microurb www-data 12288 Oct  8 17:07 wp-includes
-rw-rw-r--  1 microurb www-data  2422 Oct  8 17:07 wp-links-opml.php
-rw-rw-r--  1 microurb www-data  3301 Oct  8 17:07 wp-load.php
-rw-rw-r--  1 microurb www-data 34327 Oct  8 17:07 wp-login.php
-rw-rw-r--  1 microurb www-data  8048 Oct  8 17:07 wp-mail.php
-rw-rw-r--  1 microurb www-data 16200 Oct  8 17:07 wp-settings.php
-rw-rw-r--  1 microurb www-data 29924 Oct  8 17:07 wp-signup.php
-rw-rw-r--  1 microurb www-data  4513 Oct  8 17:07 wp-trackback.php
-rw-rw-r--  1 microurb www-data  3065 Oct  8 17:07 xmlrpc.php

I thought the problem may have been that index.html file and so I removed it and restarted nginx and still, I get only Nginx home page. I restarted Nginx a couple times.
I checked access and error logs and all I get is an error about code on the first site.
I checked the nginx access logs:
184.108.40.206 - - [08/Oct/2017:19:10:55 -0400] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"
220.127.116.11 - - [08/Oct/2017:19:10:56 -0400] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36"

The error logs have an error regarding code on my first site:
PHP message: PHP Notice:  Undefined index: success in /var/www/microurb.com/public_html/index.php on line 295" while reading upstream, client: 126.96.36.199, server: microurb.club, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "microurb.club"
2017/10/08 19:10:01 [alert] 22798#22798: *25 open socket #11 left in connection 3
2017/10/08 19:10:01 [alert] 22798#22798: aborting
2017/10/08 19:10:52 [alert] 23108#23108: *3 open socket #3 left in connection 3
2017/10/08 19:10:52 [alert] 23108#23108: aborting

I really do not know what to look for next. Why can I not see the WP site?
This is my /etc/nginx/sites-available/dancortes.press server file:
server {
        listen 80;
        listen [::]:80;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # include snippets/snakeoil.conf;

        root /var/www/dancortes.press/public_html;

        # Add index.php to the list if you are using PHP
        index index.php;

        server_name dancortes.press;

        include global/restrictions.conf;
        include global/wordpress.conf;

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
        }

}

Below I am adding a screenshot of my DNS configuration:
Comment: I thnk the issue is that yo have your virtualhost file on sites-available, so you need to create a symlink to sites-enabled
Comment: @DiegoVelez, the symlink was created, but my second site was not configured for a 301 redirect like the first one, instead it was configured for CNAME www dancortes.press. In such a case, you do have to make the server_name www.dancortes.press, instead of just dancortes.press. Once I added the www to server_name, WP core site displayed just fine. No more Nginx welcome page.
Answer: The error messages says the PHP in index.php is malformed on line 295. The Wordpress index.php is 17 lines long. It could be that the included files are counted as lines in that file.
Check what file the PHP interpreter is running. Reinstall Wordpress. Check your Nginxs settings that call PHP are correct. Some themes aren't compatible with PHP7.
Comment: Tim, the php error on line 295 is referring to the first website, which successfully loads in the browser. Are you suggesting that a coding error on the first site is affecting the ability of nginx loading the second site?
Comment: Please edit your question to make it clear there are two websites, perhaps calling them website1 and website2, saying which is working. Make sure any logs or configuration files are labeled with which website they're for.
Comment: I attempted to make it clearer by editing above, but the log files, I just did a less /var/log/nginx/error.log. I do not know how to ask it to just give me the error for the second site.
Comment: Tim, have no idea if you are still assisting but I just checked with info.php and PHP 7 is working, and its working through FPM/FastCGI from what I saw in Server API.
Comment: I don't think you've given enough information to solve the problem, at least not enough for me to help. Your question is also somewhat vague. You probably need to do more diagnostics yourself to work this out, or to hire a consultant.
Comment: Hi Tim, I have provided more information above. This should help you help me.
Comment: I think you need to hire a consultant for this. You don't seem to understand the problem well enough to give us the information we'd need to help solve the problem.
Comment: Unfortunately, I cannot answer my own question because you blocked it as off-topic, which is not off topic. I have resolved this issue, but I cannot even properly close this. There is a reason why I posted the DNS console. The answer is right there.
Comment: Suggest you edit your question to include the answer at the end. Interested to hear what it was.


Does conditional indepence imply this?

Question:
Definition
Let $X,Y,Z$ be random variables for a common probability space $(\Omega,\mathscr{F},P)$
Then, $X,Y$ are said to be conditionally independent given $Z$ if $P(X\in A, Y\in B|Z)=P(X\in A|Z)P(Y\in B|Z)$ for all borel sets $A,B$.

If $X,Y$ are conditionally independent given $Z$, $P(X\in A|Y,Z)= P(X\in A|Z)$?
This is true if we assume $(X,Y,Z)$ is jointly absolutely continuous. (Because $$p(x|z)=p(x|z)p(y|z)/p(y|z)=p(x,y|z)/p(y|z)=p(x,y|z)p(z)/p(y,z)=p(x,y,z)/p(y,z)=p(x|y,z)$$)
I am curious if this holds for general $X,Y,Z$'s.
Answer: This is certainly true.
Let's say that the $\sigma$-algebras $\mathcal{F}$ and $\mathcal{G}$ are independent given $\mathcal{H}$ if
$$
P_{\mathcal{H}}(F \cap G) = P_{\mathcal{H}}(F) P_{\mathcal{H}}(G)
$$
for every $F \in \mathcal{F}$ and $G \in \mathcal{G}$.
Then we will show that
$$
P_{\mathcal{G} \vee \mathcal{H}}(F) = P_{\mathcal{H}}(F)
$$
for every $F \in \mathcal{F}$.
First $P_{\mathcal{H}}(F)$ is certainly $(\mathcal{G} \vee \mathcal{H})$-measurable.
Second, we have, for every $F \in \mathcal{F}$, $G \in \mathcal{G}$ and
$H \in \mathcal{H}$,
\begin{align}
P(F \cap G \cap H) &=  E( P_{\mathcal{H}}(F) P_{\mathcal{H}}(G) I_{H})\\\\
&= E( P_{\mathcal{H}}(F) P_{\mathcal{H}}(G \cap H)) \\\\
&= E( P_{\mathcal{H}}(F) I_{G \cap H})
\end{align}
where the first equality comes from the conditional independence of $\mathcal{F}$ and $\mathcal{G}$.
Since the equality $P(F \cap G \cap H) = E( P_{\mathcal{H}}(F) I_{G \cap H})$ is true for every set of the form $G \cap H$, it will be true for every set in $\mathcal{G} \vee \mathcal{H}$, by standard arguments.
Comment: What's $P_\mathscr{H}(A)$? Do you mean $P(A|\mathscr{H})$?
Comment: Yes sure ......

Does conditional indepence imply this?


If $X$ is a continuous random variable with support $A$, does this imply that the cdf of $X$ is strictly increasing on $A$?

Question: If $X$ is a continuous random variable with support $A$, does this imply that the cdf of $X$ is strictly increasing on $A$?
My guess is yes. But just in case, let me know if you can think of any counterexamples.
Answer: A counter example is made by taking the cdf as the Cantor distribution $\mathfrak c(\cdot)$ on $[0,1]$, whose support is the Cantor set $\mathfrak C$:

it is a continuous (if not absolutely continuous) distribution
its support $\mathfrak C$ is closed and of Lebesgue measure zero
the points 1⁄3 and 2⁄3 are adjacent in $\mathfrak C$, i.e., $\mathfrak C\cap(1/3,2/3)=\emptyset$ while $1/3,2/3\in\mathfrak C$
hence $\mathfrak c(1/3)=\mathfrak c(2/3)$ and $\mathfrak c(\cdot)$ is not strictly increasing on $\mathfrak C$
Comment: @Cross Wherever the density might be zero, the cdf is not strictly increasing. It is possible for densities to equal zero on the support of the random variable--they just cannot be zero throughout any open interval.
Comment: Brilliant counter-example. What happens if the density exists as discussed in the other answer?
Comment: Thanks, Xi'an! This is very helpful.
Comment: @Xi'an@whuber Then requiring the density to be everywhere positive would suffice, right?
Answer: Assuming the density exists, and ignoring the boundary points, a function strictly increases if its derivative is strictly positive. CDF's derivative is PDF and it is strictly positive in the support. So, CDF must strictly increase in the support.
Comment: A continuous random variable being a random variable whose cumulative distribution function (cdf) is continuous everywhere, this cdf may be differentiable nowhere, meaning there is no density (pdf).
Comment: @gunes Thank you very much!


Classification in a applied financial setting

Question: Many academic papers focusing on statistical learning in the applied setting of finance train a model such that their parameter set, B,  defines the relationship
y(t) = B*x(t) + e(t)
y could be a 0-1 coded response for an increase (or not) of the asset price at time t.  Ignoring the details of the model my question is as follows:
given that you are predicting the response at the SAME time that you record your features, how would you make use of this model?  Intuitively I think ok, my input space says we will
see an increase in y but since they are at the same time, I can actually go and see what y has in fact done.  Do I take the view of my model to be the truth and create a trading signal based on this information (for example)?
Any practical insight would be greatly appreciated.
Answer: If you want to predict the future, and then use that information to profit, you need to build a model like this:
Next(y)~x or y(t+1)=B*x(t)+e(t)
i.e. specify your problem as "what will happen tomorrow" not "what did happen today."
Look at the quantmod package for R for more examples.
Comment: Yes, but all of your explanatory power goes out the window in this setting.  Besides, if it is this obvious why do so many academic papers ignore it?
Comment: @SpeedBoots Maybe the academic papers you are reading are focused on explaining the past rather than predicting the future.  Furthermore, it could be that there are certain variables that predict a stock's price going up.  The January effect would be an example of one such model.


Exception - instance error

Question: When you start catching exception and the exceptions autoloader controller error pops up:
Argument 1 passed to Controller_Exception:: __construct () must be an instance of Autoloader_Exception, string given, Called in 'source'

try {} catch (Autoloader_Exception $ e){ throw new Controller_Exception ('Some Error');  }}

Controller_Exception class { public function __construct ($ Autoloader_Exception e) { } }

Question two, how can I build an exception class? Does anybody know a good example?
Comment: What was the *first* question?
Comment: What language are you using? Can you provide some context?
Answer: The first issue is that you are using
new Controller_Exception ('Some Error')

According to the error message 'Some Error' must be an Autoloader_Exception. Depending on what you want to do you could use:
new Controller_Exception ($e)

For the second question you can read about creating your own exceptions here.

Exception - instance error


Por que não pode declarar uma variável dentro de um case?

Question: Por que isso não compila?
#include <stdio.h>

int main(void) {
    int valor = 0;
    scanf("%d", &valor);
    switch (valor) {
    case 0:
        int variavel = 1;
        printf("%d", variavel);
        break;
    default:
        int variavel = 2;
        printf("%d", variavel);
    }
}

Answer: Um engano muito comum é as pessoas acharem que o case é um bloco de comandos e gera um novo escopo. Na verdade o case é apenas um label. Então é apenas um nome para um endereço do código usado para provocar um desvio. Na verdade um switch é apenas um goto baseado em um valor.
Isto já funciona:
#include <stdio.h>

int main(void) {
    int valor = 0;
    scanf("%d", &valor);
    switch (valor) {
    case 0: {
        int variavel = 1;
        printf("%d", variavel);
        break;
    } default: {
        int variavel = 2;
        printf("%d", variavel);
    }
    }
}

Veja funcionando no ideone. E no repl.it. Também coloquei no GitHub para referência futura.
As chaves criam um bloco e um escopo, aí pode criar as variáveis.


How to retrieve cart_item_data with WooCommerce?

Question: During the add_to_cart function, there is a filter to add "cart item data". The filter is woocommerce_add_cart_item_data. I expected to store my custom plugin data in this, so that the data is stored relative to the item and multiple products can be added with different data.
This all seemed to work, but I am not able to retrieve the data. I can't figure it out. The data is there, I can see it in a serialized string, but I can't pull it out.
echo '
';
var_dump( WC() );

foreach( WC()->cart->get_cart() as $cart_item ) {
  var_dump( $cart_item );
  var_dump( WC()->cart->get_item_data( $cart_item ) );
}
echo '
';

The first dump of WC() has a property: session->_data->cart->(serialized data). The _data property is protected, though, but I can see my custom field inside the serialized data.
The $cart_item is an array with product_id and some other data, but it does not include my custom data :(
Finally, using the get_item_data() method I thought I had it all figured out. I passed in the cart item object, and... an empty string. Same if I pass the key, rather than the cart item itself.
How am I supposed to access the cart item data?

Here is the "Add cart item data" function, which works (or at least seems to work):
function save_class_menu_selection( $cart_item_data, $product_id, $variation_id ) {
  if ( !product_is_class( $product_id ) ) return $cart_item_data;

  // Save the date, or give a fatal warning. Date is required.
  if ( !empty($_REQUEST['class-date']) ) {
    $cart_item_data['class-date'] = stripslashes($_REQUEST['class-date']);
    return $cart_item_data;
  }else{
    wp_die('
Invalid Class Date Selected

You tried to add a class to your cart, but the date selected was invalid. Please try again.
');
    exit;
  }
}
add_filter( 'woocommerce_add_cart_item_data', 'save_class_menu_selection', 10, 3 );

Answer: I was in the same situation today and stumbled over this question after some research.
After some reverse engineering I found the problem and want to provide a solution for other which may also stumble over this question.
The problem is that the data gets sanitized when the cart items get restored from the session. So the extra cart item data IS stored into the session but on the next request it does not get restored.
There is a filter "woocommerce_get_cart_item_from_session". As first parameter you get the sanitized cart item (without extra data) and as second all data which got stored into the session (including extra data).
The solution is to hook in there and also restore your custom cart item data.
Example Code:
add_filter( 'woocommerce_add_cart_item_data', function ( $cartItemData, $productId, $variationId ) {
    $cartItemData['myCustomData'] = 'someCustomValue';

    return $cartItemData;
}, 10, 3 );

add_filter( 'woocommerce_get_cart_item_from_session', function ( $cartItemData, $cartItemSessionData, $cartItemKey ) {
    if ( isset( $cartItemSessionData['myCustomData'] ) ) {
        $cartItemData['myCustomData'] = $cartItemSessionData['myCustomData'];
    }

    return $cartItemData;
}, 10, 3 );

To also show the data at the cart/checkout page you can use the following code:
add_filter( 'woocommerce_get_item_data', function ( $data, $cartItem ) {
    if ( isset( $cartItem['myCustomData'] ) ) {
        $data[] = array(
            'name' => 'My custom data',
            'value' => $cartItem['myCustomData']
        );
    }

    return $data;
}, 10, 2 );

The final thing now is to save the data when the order is made:
add_action( 'woocommerce_add_order_item_meta', function ( $itemId, $values, $key ) {
    if ( isset( $values['myCustomData'] ) ) {
        wc_add_order_item_meta( $itemId, 'myCustomData', $values['myCustomData'] );
    }
}, 10, 3 );

You dont have to do anything else the show the data inside the backend, all order item meta data gets display automatically.
Comment: Thanks for this Stefan. It sounds like this is very similar to a "query var" in WordPress. That is, any data passed via POST/GET does not register as a query_var unless you specifically tell WordPress to do so. Do you know if this session data gets saved when the cart data is converted into an order? (eg, when a payment is made)
Comment: The data does not get saved automatically but I have added some code examples to my answer on how to do this.
Comment: I'm going to make this the accepted answer as my solution was a workaround. This looks more properly implemented. I haven't actually tested it myself, though.
Comment: This answer was EXTREMELY helpful! Thank you for taking the time to post. Might you be able to clarify how to also show this custom data within customer payment emails?
Comment: This was really helpful. Now I am trying to figure out how to make changes in the item price based on the custom value. Anyone interested to help?
Comment: could you please help https://stackoverflow.com/questions/47646781/woocommerce-hook-and-filters-for-changing-product-price-when-adding-to-cart
Comment: Thought I would point out that you do not need the "woocommerce_add_cart_item_data" hook if you are adding the product to the cart with PHP. The function `WC()->cart->add_to_cart()` has a  5th parameter that accepts cart item data. The other hooks are still necessary.
Answer: I could not get the default cart item data to work, unfortunately. I feel it may not be properly implemented, or may even be deprecated, as there is a lack of support and documentation.
Instead, I used a cart session variable to accomplish the same thing. It's simply an array where each key is the cart_item_key. The value of each array is yet another array, containing a key-value pair of custom fields. So it's essentially the same thing as the built-in cart item data, except stored as cart session data instead.
Here is a Gist containing some utility functions to make it easy:
https://gist.github.com/RadGH/e3444fc661554a0f8c6f

Or if you want to build it yourself, the magic is in WC()->session. Here are the two key functions for this to work:
WC()->session->get('_my_cart_item_data');
WC()->session->set('_my_cart_item_data', $cart_item_data_array);

These are the action hooks you will need:
<<<EXAMPLES
action: woocommerce_add_to_cart
desc: When an item is added to the cart.
args: $cart_item_key, $product_id, $quantity, $variation_id, $variation, $cart_item_data

action: woocommerce_add_order_item_meta
desc: When a cart item is converted to an order item, save metadata to the order item
      using the function "wc_add_order_item_meta( $item_id, $key, $value )"
args: item_id, $values, $cart_item_key


Insignificant F-test in linear regression - when to stop?

Question: I repeatedly read that one can stop looking further when the F-test is insignificant for linear regression; for example in the comments here. However, according to this very insightful answer only relying on the F-test could lead to missing an actual significant variable when some other explanatory variables mask a significant overall result. My question: In case of an insignificant F-test, when shall I stop and when might it be legitimate/worthwhile to continue and look deeper?
Comment: This probably needs more details to matter at all for any actual data analysis. I think you're viewing statistical testing as a means-to-an-end, and as a geometric process: how many p-values do you need 'til one is significant? No, all hypotheses need to be prespecified prior to data collection and analysis. Post-hoc tests are just that: incidental findings. A rather nice but paywalled paper called "MUNCHAUSEN'S STATISTICAL GRID,WHICH MAKES ALL TRIALS SIGNIFICANT" shows interestingly that with enough variables, you can "grid" an analysis enough to get significant results regardless of truth.
Comment: I understand and agree.Thanks! (found this also helpful: https://projects.fivethirtyeight.com/p-hacking/) But when my model is theoretically well versed and I have an insignificant F-test. In case I stop, I might miss an actually valid and significant IV, isnt it? Is it a rule of thumb to stop when the F-test is insignificant?
Comment: Just wanted to say apologies for the answer that keeps appearing and disappearing!  I keep thinking I'm totally content with my answer and then I discover something else.  I'm still working through it.  Quite fascinating.
Answer: Short Answer
After spending several days thinking about this and running simulations, I can't agree with the usual recommendations.  But if someone can see a flaw in my logic (which there easily might be) please do comment.
My conclusion is this.  If your goal is to look at a series of predictors of interest one-by-one, then I think the ANOVA is not really relevant to that question and a better "general approach" (if we must have one) would be to Bonferroni correct your individual $t$-tests.  The key points are:

If you want to keep your false positives under control, ANOVA followed by unadjusted $t$-tests does not do this.  Especially if some of your predictors are real and some not (it only takes one "real" predictor successfully detected to unlock the ANOVA and open up the without-ANOVA false positive chances of the remaining tests).

Adjusting your $t$-tests (e.g. Bonferroni) does control your false positives and is in some cases, actually more powerful than the ANOVA.  (When aggregated across all adjusted tests).

Adding an over-arching ANOVA to your adjusted $t$-tests is therefore not necessary (false-positives already under control).  And it actually will (by its very design) mask some of the true positives you would otherwise have discovered.

The real strength of ANOVA is in detecting combinations of predictors and there are many situations where it is brilliant at that (dummies of categorical variables, combinations that are too weak to detect individually, correlated predictors that can be "lumped together").  It is precisely its excellent power in these situations that causes the trade-off that makes it unable to capture all the cases that Bonferroni-adjusted tests would make.

If you want a better intuition for why Bonferroni can beat ANOVA think of it this way: under the null both must reject 5% of cases (or close to that for the aggregate of your Bonferroni-tests).  So the set of datasets where Bonferroni can reject cannot be a subset of those for ANOVA.  Which is to say, if ANOVA does better at one thing, it must do worse than Bonferroni at something else.

Wouldn't it be better anyway if we estimated our coefficients and their errors? ;p  A significant predictor might be small with small error; then you're pretty confident it isn't that interesting.  A non-significant predictor might be very large but with a large error; OK, you're not confident in the result you got, but it looks promising!

Long Answer
As a stats educator I have always told my (unfortunate) students "well, what is the point of looking at the individual $t$ statistics if the $F$ tells us that the overall picture looks just like noise?".  So, this question (and @whuber's example) actually kept me awake last night (not even joking), trying to figure out if I've been talking rubbish.  And I think I have.  The final answer I've arrived at isn't what I've been taught, nor what I've been teaching for the last decade, so if anyone with more experience/education than I can point out a flaw in my thinking I'd certainly be grateful.
Edit: Most of this was done looking at uncorrelated predictors.  After taking the post down a couple of days to look at correlated predictors, the picture is a bit more murky, but I still come to the same conclusion.  Small section on correlated predictors added right at the bottom.
My Answer
The answer I have reached is the opposite from what I have always thought.  I now think the $F$-statistic does not devalue the results of any given significant $t$-test and should not be a barrier to checking the $t$-tests.  But, that it can add value to non-significant $t$-tests.
That is to say: don't use the $F$ as a filter for your individual $t$-stats.  But do use it to test interesting compound hypotheses.
The key here is that the $F$ has a different goal than the individual $t$-tests and should, I think, be used in line with this.  If your goal is to test the predictors individually, then you might do better to just Bonferroni-adjust your $t$-tests-of-interest and look only at those.  If your goal is specifically to see whether a bunch of predictors together can predict your dv, then the $F$-statistic has the upper hand and can detect things that your individual $t$-tests (even before corrections) could not.
The fact that ANOVA is so well crafted to pick up combinations of predictors actually directs it away from detecting individual significant predictors and means it can quite naturally appear to contradict your $t$-tests, just because it is asking a different question.
This conclusion came from looking at simulations based on @whuber's example, and a geometric visualisation to help me think this through.  In case you're interested, more details follow...  Usual caveats apply, that we are usually better informed by estimating effects and their CIs than looking at significance; also these results are what I've found with some preliminary simulations -- but someone might leap in and say I'm all wrong!
The Simulation
I wanted to understand @whuber's example more generally -- rather than one isolated example, so I ran a whole bunch of simulations that ran similar examples 10000 times.  I also realised that this specific case (perfectly uncorrelated predictors) can be transformed into a super intuitive geometric interpretation.  You can see a simulation along with the geometric interpretation below.
Each row in the table in the figure corresponds to adding an extra predictor to the model -- you can see the beta of the added predictor, and the % significant according to the ANOVA vs the % of cases where at least one of the Bonferroni-adjusted $t$-tests came out significant.  So, row 4, for example has 4 predictors, with betas of 0, 0, 0 and 0.  Clearly this is the null case, so we're relieved to see lots of 5% values in there.
The plot shows all the simulations that correspond with the second row of the table: where we have two predictors.  The axes give the $t$ statistics for the slopes on those two variables; each dot is a simulation.  Those dots outside the circle will be significant under the ANOVA, those dots outside the square will have at least one significant Bonferroni-corrected $t$.  (The circle was calculated based on the fact that the $F$ in this uncorrelated case is the mean of the squared $t$ statistics).  Brightly coloured dots are those simulations that are called significant by only one of our two methods (ANOVA, Bonferroni $t$).

Now, here is the plot of 10000 simulations based on @whuber's example, with number of predictors set to 10:

It really surprised me to find that, in this case, the Bonferroni approach is consistently more powerful than the ANOVA?!  But, looking at the graph, we can see why: all those red dots that land in that little "blind spot" of the ANOVA, which is exactly where we'd like it to be.
This held even with a lower effect size and so, lower power (correction to earlier, due to change in approach to the Bonferroni):

Where the ANOVA comes into its own is when looking at combinations of predictors.  Say, if we add a second meaningful predictor to the model:

...albeit, the Bonferroni lags but is not too too far behind!  But still, we can see that, whereas the ANOVA has a blind spot on the "only one meaningful predictor" case, the Bonferronied-$t$ has its corresponding blindspot in the "combination of predictors" case (the cyan dots).  And the more meaningful predictors we add to the model, the further and further the Bonferronied-$t$ starts to fall behind:

So What?
The key takeaway for me from all of this, was that the $F$-statistic is actually somewhat apples and oranges with the $t$-tests.  It isn't necessarily the perfect umbrella test, if you want to check predictors one-by-one for significance.  It is (I think?) the most powerful option for detecting departures from noise overall: by sitting its circle on the contours of the distribution of dots in the plots, it means the non-rejection region is the smallest possible.  But we gain that power in detecting combinations that are very hard for us to interpret.
If you are in a situation where you have ten predictors that all have $t$-statistics of 1.4, it will be able to detect that as "unusual" (a.k.a. significant).  But what are you going to do with that?  How are you going to then decipher which of those is really behind this?  Your $t$-tests aren't going to be much use.  If, instead, you use the Bonferroni-adjusted $t$ statistics, you do lose power overall, potentially quite a lot.  But you actually gain some power in the situations that you can interpret and that are in line with your objectives.  And, crucially, here is the key point: doing so is completely consistent; starting with an ANOVA is mixing together two different questions as if they were one.
So, I think (as it stands at present), I can no longer justify this approach of using the $F$ as a gateway to the $t$ tests.
On the other hand, you might be specifically interested in the combination of a set of predictors.  For example, if there is too much noise to detect them individually, or if they don't make sense individually (such as dummies of a categorical variable).  In that case, ANOVA really is the stuff!
Code
Here is the code, in case you want to play with it.  Sorry it's so long, but I like that you can tweak almost every aspect of the simulation:
<code>library(ggplot2)
library(gridExtra)

# choose your betas to try different possiblities.  You can change the number of betas and it will change the number of xs -- which will also change the dataset size according whuber's approach
betas            <- c( 0,  0, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c( 1,  0, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c(.5,  0, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c(.7, .7, 0, 0, 0, 0, 0, 0, 0, 0)
betas            <- c( 1,  1, 1, 1, 1, 1, 1, 1, 1, 1)*.5

data.multiplier <- 1 # set this at 1 to reproduce @whuber's example.  A higher number adds more data to the dataset (e.g. 2 will double the rows up)

# crude attempt to always start out with 80%ish power for the one-predictor cases
num.predictors <- length(betas)
num.data       <- 2 ^ num.predictors * data.multiplier
noise.sigma    <- sqrt(1/8) * sqrt(num.data)
num.sims       <- 500 # Number of simulations(reduced to 500 here as better to play with)

set.seed(1)

run.anova.on.subset = function(i, x, y){
  test <- summary(lm(y ~ x[, 1:i]))
  F <- test$fstatistic

  num.coeffs <- nrow(test$coefficients)
  p.F        <- 1-pf(F["value"], F["numdf"], F["dendf"])
  p.ts       <- as.numeric(test$coefficients[2:num.coeffs, "Pr(>|t|)"])
  ts         <- as.numeric(test$coefficients[2:num.coeffs, "t value" ])

  names(p.F) <- NULL

  c(
    t1         = ts[1],
    t2         = if(i > 1) ts[2] else NA,
    Bonferroni = any(p.ts < .05 / i),
    ANOVA      = p.F < .05
  )
}

generate.dataset.and.run.all.Fs = function(){
  # this is @whuber's setup with orthogonal predictors of which only one contributes to y
  x <- as.matrix(do.call(expand.grid, replicate(num.predictors, c(-1, 1), simplify = FALSE)))
  x <- do.call(rbind, replicate(data.multiplier, x, simplify = FALSE))
  y <- x %*% betas + rnorm(n=num.data, mean=0, sd=noise.sigma)
  sapply(1:num.predictors, run.anova.on.subset, x, y)
}
runs <- replicate(num.sims, generate.dataset.and.run.all.Fs())

# convert simulation results into a table of powers and dataframe of ts from each simulation
power.table <- apply(runs, 1:2, mean)[3:nrow(runs),]
power.df <- as.data.frame(t(power.table))

ts.df = as.data.frame(t(runs[, 2, ]))

predictors.df = data.frame(
  beta  = betas,
  Bonf. = sprintf("%.0f%%", power.df$Bonferroni * 100),
  F     = sprintf("%.0f%%", power.df$ANOVA      * 100)
)

# critical values.  f is F transformed to scale of t
Fcrit = qf(.95,2,num.data-3)
fcrit = sqrt(2*Fcrit)
Boncrit = qt((1-.05/(2*2)), num.data-3)

# coords for the circle and square on the plot
Boncrit.df = data.frame(
  t1 = c(Boncrit, Boncrit, -Boncrit, -Boncrit, Boncrit),
  t2 = c(Boncrit, -Boncrit, -Boncrit, Boncrit, Boncrit)
)
thetas = seq(0,2*pi, length.out=300)
circle = data.frame(
  t1 = cos(thetas) * fcrit,
  t2 = sin(thetas) * fcrit
)

# expand xlimits to fit in power table
xlims <- c(min(c(ts.df$t1, circle$t1)), max(ts.df$t1))
RHside <- xlims[2]
xlims[2] <- xlims[2] + diff(xlims)*.5

plot = ggplot(aes(t1,t2), data=ts.df) +
  geom_point(aes(colour=interaction(Bonferroni,ANOVA))) +
  geom_path(data=circle, size=1) +
  geom_path(data=Boncrit.df, linetype="dashed", size=1) +
  scale_colour_manual(values=c("grey", "red", "cyan", "black")) +
  scale_x_continuous(limits=xlims) +
  annotation_custom(tableGrob(predictors.df, theme=ttheme_minimal()), xmin=RHside) +
  theme_classic() +
  theme(legend.position = "none")

print(plot)

</code>
Correlated Predictors
Once we start to look at correlated predictors, things get a little more complicated.  The Bonferroni approach suffers badly when we have two correlated predictors, as the individual $t$-tests must incorporate uncertainty about "which" of them is the "cause" (please read that word loosely!).  The ANOVA, on the other hand can just say, hey, these two look interesting; I award you significant!
But, here's the thing.  Every way I look at it, it loops back to the same problem.  If anything, the points made in the earlier section are more acute here: the ANOVA is far more powerful, by focussing on detecting scenarios that we cannot interpret (that is, if our goal is to look predictor by predictor).  In the figure below, the "cyan" column shows that (in this scenario) we have about 20% of experiments come back significant on the ANOVA, but not on the Bonferroni-$t$ (cyan dots in the figure), so we are stuck trying to interpret them.
And, again, because the rejection region must always cover 95% (ish for Bonferroni) of cases in the null, this increased power on the cyan cases must come at a cost.  The cost is that a proportion of cases (up to 6% for this scenario) are red dots -- significant under Bonferroni, but not under the ANOVA.  Those are cases that would have been interpretable, but we stopped because of the ANOVA.  If we just focus on those cases where we would successfully detect the first predictor with Bonferroni (Bonferroni $t$ True Positives -- "BtTP"), the last column of the table ("BtTPs.blocked") shows that the ANOVA would actually cause us to miss 20% of those in the 10-predictor case!
If you think, hey but you're using the Bonferroni-adjusted $t$ values, that's why you're getting these rubbish results...  Firstly, the "FtFP" (F-and-then-$t$ False Positives) shows that using the ANOVA followed by unadjusted $t$s produces at least one false positive in up to 22% of our samples.  Secondly, the ANOVA isn't meant to allow us to use non-adjusted $t$s here.  Indeed, we hope that the ANOVA will be significant to "allow" us to look at the our true positives on predictor 1.  But that then does nothing to control the errors on predictors 2-10.

(For full clarity: in the above simulation, each odd numbered predictor has a correlation of 0.6 with the predictor after it.  But the rest of the correlations are zero.)
Again, the final point is: this is not anti-ANOVA!  Indeed, this suggests that if you have two strongly correlated predictors that sit together and make a nice "thing" -- why not blob them together into one $F$ statistic and test both at the same time?  That, as well as the cases from before (dummies and combinations that are hard to detect individually).  There are many cases where the ANOVA is far the better tool.  It is just a case of picking the right tool for the job.  But, when it comes to testing a series of individual predictors of a regression, I don't believe ANOVA is the right umbrella for that job.


understanding setTimeout()

Question: I'm trying to make jcarousel work both sides
If I use the following it returns the correct alert
this.timer = window.setTimeout( (a === 0.5)? alert("c.next() a = " + a):
                                (a === 0.6)? alert("c.prev() a = " + a):
                              function(){},500 ) ;

but, if I replace the alert with the actual call it doesn't scroll. just moves to the next image
this.timer = window.setTimeout( (a === 0.5)? c.next.bind(c):
                                (a === 0.6)? c.prev.bind(c): function(){},500 ) ;

without bind() it doesn't work either
this.timer = window.setTimeout( (a === 0.5)? c.next():
                                (a === 0.6)? c.prev(): function(){},500 ) ;

Does anyone has an idea why is this happening? if I don't use any condition inside Timeout and use one of the two calls, for example c.next() then it would scroll but only from Right to Left
Any guideline will be greatly appreciated
Sandra
Comment: What do you mean, "it doesn't scroll, just moves to the next image"?  Isn't that what your `c.next()` function does? Your second sample of code looks like the right one; the first one doesn't really do anything and the third one is wrong.
Comment: ok. you are right. c.next() moves to the next. But, if I use: window.setTimeout( function() {c.next()}, 500 ) it scrolls to the right until the user moves the mouse out. Based on that, I've been trying to use the same function (because I can't find another way) to scroll to the left if the user hovers over the left button
Answer: Make it easier for yourself and define your functions ahead of time and in a way that saves you the trouble of scope and closure
var next = function () {return c.next();},
    prev = function () {return c.prev();},
    noop = function () {};

this.timer = window.setTimeout(
    (a === 0.5 ? next : a === 0.6 ? prev : noop),
    500
);

Now

If you want to debug a function, you can debug it separately to setTimeout.
It's clear which function is being passed into setTimeout.

Edit Also note how I'm not invoking the functions as they're passed as the argument; when one uses setTimeout it is because they want something invoked later, not immediately.
Comment: This works correctly, but without scrolling. Here is where we started: http://stackoverflow.com/questions/18139359/jcarousel-autoscroll-in-two-different-directions

understanding setTimeout()


Adding custom block problems

Question: I've created my own module in drupal 8. I also created block in my module.
I tryied to add my custom block (which is type of custom form) to view. But when I am trying to add custom block in view, its not appearing in the list. Block is only appearing in block layout. What is wrong or is there other way of adding custom block from module to your view in drupal 8?
<code><?php
/**
 * @file
 * Contains \Drupal\reviewhelper\Plugin\Block\ReviewHelperBlock.
 */
namespace Drupal\reviewhelper\Plugin\Block;

use Drupal\reviewhelper\Form;
use Drupal\Core\Block\BlockBase;
use Drupal\Core\Form\FormStateInterface;

/**
 * Provides a 'ReviewHelper' Block
 *
 * @Block(
 *   id = "ReviewHelper",
 *   admin_label = @Translation("Review helper block"),
 *   category = @Translation("Custom Blocks"),
 * )
 */
class ReviewHelperBlock extends BlockBase {
  /**
   * {@inheritdoc}
   */
  public function build() {
    $form = \Drupal::formBuilder()->getForm('Drupal\reviewhelper\Form\ReviewHelperPush');
    return $form;
  }
}
</code>
Comment: It's not clear in this context what you mean by "view", do you mean a view from the Views module or something else?
Comment: Yes Cottser I meant views module. The problem itself is to how add custom block to view in views module.
Answer: There are two different "block things". Actually, three.

First, there are block plugins. That's code, that's what you have. It's the thing you select on the block layout page. It's not something that you can list with views (Well, theoretically, it would be possible, but you would have to write a views query plugin that queries plugins.)
Then, there block config entities. That's a plugin + configuration, a block placement.
Last, there are custom blocks or content blocks. Those are content entities, similar to nodes with different types, fields and so on. Those are the user-created data. Each of them is exposed on the block layout page as a plugin derivate. There's one plugin class for them and then there are derivates for each content entity. Those are the things you can list with views. Only those.

Everything is connected, you can create custom/content blocks, which are exposes as a block plugin, which is then placed and becomes a block config entity. But they're all different things.
Comment: Berdir can you explain/elaborate about views query plugin that queries plugins in drupal 8?
Comment: Not really, no. It doesn't exist, it's just an idea. And it very likely wouldn't allow you to do what you have in mind. Maybe you're looking for something like https://www.drupal.org/project/page_manager, It's not clear to me at all why you'd want to add a block plugin to a view and how.

Adding custom block problems


MySQL new db and troubleshooting some connections to old DB still running

Question: I just setup a new DB server replacing one that has been live for 5+ years. I normally set the db connections in the host file but now that the new server is up I still see some connections to the old server how can I track down where those connections are coming from?
Answer: If you log into the old MySQL server and run the following:
mysql> show processlist;

You should see a list of all active connections, with the source host in the 3rd column. It should be of the form:
serverhost:12345

Where serverhost is the hostname you're looking for and 12345 is the source port. If you want to track down the process that's connecting to MySQL, on serverhost run the following:
serverhost# lsof -i :12345

You should get something like this:
COMMAND   PID USER   FD   TYPE     DEVICE SIZE NODE NAME
program 5954 root   27u  IPv4 -780315302       TCP serverhost:12345->mysqlserver:mysql (ESTABLISHED)

So in this example, command program at PID 5954 is the offender.
Hope this helps.
Answer: You can view same informations explained by vmfarms also using Mysql Administrator (old gui) or Mysql WorkBench, into the section "Server Connections", after you login into the server as root.


StackingClassifier: Different Feature Sets

Question: I have a few different feature sets (so, with same number of rows and the labels are the same) that I use for different ML models. In my case these sets are DataFrames.
I want to use them to train a StackingClassifier. But the fitting step only allows to specify a single feature set. Goal is to fit clf1 with df1 and clf2 with df2, etc.
stack_clf = StackingClassifier(estimators=[('df1-clf', clf1), ('df2-clf', clf2)],final_estimator=LogisticRegression())
stack_clf.fit(...)

How should I proceed with this kind of situation? I have been trying to find a solution using pipelines but am not sure if that is the way to go. There was a similar question several years ago with voting classifiers, but at the time the solution was apparently not natively supported by SKlearn, and the custom functions suggested were specific to voting classfiers.
Thanks in advance for any help.
Comment: This is more a programming question; I've answered a similar one at https://stackoverflow.com/a/70599700/10495893
Answer: StackingClassifier does not support training each base estimator on different feature sets. I do not see how you could use a Pipeline to cope with this either. With its current implementation, you need to call the fit method with one feature set for all classifiers because StackingClassifier will otherwise not expose attributes ending a trailing _ which are needed to identify the classifier as fitted.
If you are not bound to use scikit-learn, an easy way out would be to use the h2o library and train a H2OStackedEnsembleEstimator. As written in the documentation, it explicitly supports training each base estimator on different feature sets:

The models must be trained on the same training_frame. The rows must be identical, but you can use different sets of predictor columns, x, across models if you choose.

An example for Python can be found here.
Comment: Thanks - really appreciate your answer. I will look into the H2O library.
Answer: There's actually a way to pull this off using Pipeline base estimators with simple custom transformers.

Create a custom transformer for each classifier. This will be a simple transformer simply slicing the data. It can be achieved by using a lambda function inside sklearn's FunctionTransformer.
Create a two-step Pipeline for each base estimator with its corresponding transformer
Create a StackingEstimator with its base estimators being the Pipelines from (2)

    from sklearn.datasets import make_classification
    from sklearn.ensemble import StackingClassifier
    from sklearn.preprocessing import FunctionTransformer
    from sklearn.linear_model import LogisticRegression
    from sklearn.svm import LinearSVC
    from sklearn.pipeline import Pipeline

    # Dummy data, `X` is the super set containing all features:
    X, y = make_classification(n_features=10)

    # Base estimator 1 uses the first 3 features with a logistic regression:
    clf_1 = LogisticRegression()
    clf_1_transformer = FunctionTransformer(lambda X: X[:, :3])
    tclf_1 = Pipeline(
        [('transformer_1', clf_1_transformer), ('clf_1', clf_1)]
    )

    # Base estimator 2 uses the last 7 features with a SVM:
    clf_2 = LinearSVC()
    clf_2_transformer = FunctionTransformer(lambda X: X[:, 3:])
    tclf_2 = Pipeline(
        [('transformer_2', clf_2_transformer), ('clf_2', clf_2)]
    )

    # The meta-learner uses the transformed-classifiers as base estimators:
    sclf = StackingClassifier([('tclf_1', tclf_1), ('tclf_2', tclf_2)])
    sclf.fit(X, y)

    # Sanity check: check how many coefficients each estimator uses:
    print(sclf.estimators_[0].steps[-1][1].coef_.size)  # 3
    print(sclf.estimators_[1].steps[-1][1].coef_.size)  # 7

Note that a more proper way to create the column-selection transformer might be to inherent from BaseEstimator and TransformerMixin instead of hacking the FunctionalTransformer.
Comment: Hi. Usefull comment, but I think it won't work with sclf.predict/predict_proba(). At least in my case where I have for one model OHE and other none it doesn't work.


iOS giving cookie.mom error when building MFP project

Question: We have a worklight project version 184.108.40.206 and I have migrated it to MFP 7.
During my build (Ant or eclipse) I'm getting this error for iOS:
ANT:
<code> builtin-copyPlist --convert binary1 --outdir /Users/Sam/Desktop/ProjectName/temp/source/ProjectName/apps/ProjectName/iphone/native/CordovaLib/build/ProjectName.app -- WorklightSDK/resources/Cookies.mom
 [exec]

/Users/Sam/Desktop/ProjectName/temp/source/ProjectName/apps/ProjectName/iphone/native/WorklightSDK/resources/Cookies.mom:0:
error: reading data: The file "Cookies.mom" couldn't be opened because there is no such file.
</code>
If I built the project through eclipse and ran it to xCode i get:

Inside WoklightSDK, I don't see the cookies.mom

How to fix this error?
Comment: Are you storing your project in some source control system? could it be that you are not uploading everything and so when you downloaded the files into Studio 7 they were incomplete? Try deleting the native folder first before upgrading.
Comment: Yes my project is stored to our main svn, which we have been always using and there is no native folders in our svn, we use "NativeResources" folder instead since our hybrid project contains some native codes. Is there some values I need to insert or configure in the xCode project inside the"NativeResources"folder to migrate the iOS?
Comment: Are you also storing there the xcodproj file? If you're storing this file then it may cause problems when migrating. Try two things: 1) remove the nativeResources folder and then try to migrate and then re-add the files and re-build in xcode 2) Try to migrate with *native* folder after it was generated in 6.2.
Comment: Yes of course there is the xcodeproj stored, it contains our native code. I tried "1" and it didn't work, got the same error. I tried "2" and got this error on eclipse "Migrating the ProjectName project from version 6.2.0.01.20141106-2034 to version 7.0.0.00.20150402-2001 has failed because java.io.IOException: Unable to delete file: /Users/Sam/Documents/workspace/ProjectName/apps/ProjectName/ipad/native/CordovaLib/Classes/UIDevice+Extensions.m"
Comment: The only other thing I can think of is to delete the iPhone folder > import > add iPhone folder > add native code again.
Comment: Idan, in WL 6.2 when I build the project, I see in iPhone>native>WoklightSDK>resources > cookies.mom. However, when I do the build for same project on MFP 7, I navigate to the iPhone>native>WokrlightSDK, but I don't see the "resources" folder inside. It is not being generated which contains the missing files that blocking my project from being built. any idea why?
Comment: At this time I do not know, I don't have time to look into it. Feel free to open a PMR.
Answer: The mentioned files in the screen shots no longer exist in a MFP project (so it is OK that they are not there in a fresh project) The migration process was supposed to remove them and so the build should not have errored on them.
But you're getting an error, this means to me that the project was not migrated to MFP 7.

Edit: per the test project in the PMR you've opened, this is happening because in your project you use using the nativeResources folder. Use of this folder is no longer considered a best practice you should stop using it.
The error here is that After the migration process, that part of it removes the old cookies.mom files, the builder takes the contents of the nativeResources folder and copies it to the native folder. In this process it replaces the updated .xcodeproj (now with removed references to the cookies.mom files that no longer exist), with your old .xcodeproj file that does contain these references - to files that do not exist... thus when building in Xcode, it fails because it has references to files that do not exist.
Delete the .xcodeproj file from your nativeResources folder and re-building. Next, in Xcode, re-references your custom class files. After the build succeeds in Xcode and your app is working, copy back the updated .xcodeproj file to the nativeResource folder, if you choose to continue using it.
By continuing to use this folder, you void your support - as I mentioned, this folder is no longer considered a best practices and may disappear in the future.
Comment: even MFP6.3 removed those files?
Comment: Eclipse console when I check out the code from SVN "Migrating the ProjectName project from version 6.2.0.01.20150214-1613 to version 6.3.0.00.20150214-1702 has succeeded." (right click on iPhone > run as > build iPhone environment. again run as > xCode project > clean the project and build it got the same error)
Comment: Might be a migration bug then... pmr.
Comment: Idan, sorry, just last question, is there a fix pack for MFP 6.3, I would like to download the latest fix available for this version before raising PMR. I used this URL to get the MFP 6.3 (http://stackoverflow.com/questions/17493510/ibm-worklight-where-to-find-previous-releases). If there is a fix packs and you can share with me the way of getting it, it might solve the issue hopefully and avoid the PMR.
Comment: If you are an IBM customer or business partner you can download the latest available iFix from IBM fIX Central.
Comment: yes we are, would you please share the landing page for the IBM fix Central for MFP 6.3 and we can login with our credentials to download it? Thanks in advance.
Comment: Please google for "ibm fix central". There is no landing page, you'll need to search in the website for "mobilefirst platform 6.3".
Comment: Idan, would you please take a look at this PMR 54816,047,649? and provide your input. Thank you
Comment: I have sent a response back 3 min ago. If MFP is not able to migrate my older .xcodeproj by modifying it to remove the WorklightSDK, is that mean I need to rewrite my native code again?
Comment: Where in my response in the PMR did you at all see me asking you to re-write your native code?! I did not ever write this. The copying of the nativeResources folder happens AFTER THE MIGRATION, which is why you see the OLD .xcodeproj file. Remove the .xcodeproj file from the the nativeResources folder - ONLY this file. Then import the project to 6.3 to have it migrated. Then run it in Xcode again and re-reference your files. This should work. If it works, you can now put back the UPDATED .xcodeproj file back into nativeResources folder.
Comment: This is not a bug. It happens because of the way you are working per my description. As I wrote several times, using nativeResources is no longer considered best practices. DO NOT USE IT.
Comment: You can also directly edited in a text editor the .xcodeproj file to remove references to cookies.mom files - I do not guarantee this will be enough. There is no bug here. None. Do not use the nativeResources folder.
Comment: I will do what you have asked now and test it. But for your second comment, where should I keep my native code then if you are recommending to not use the nativeResource folder?
Comment: Inside the native folder obviously
Comment: Thank you Idan, I was able successfully to build the iOS project on MFP 6.3 and 7 after removing the OLD xcodeproj and replaced it with the new one. I also did the file reference to complete the build. Will close the PMR.
Comment: I have a slightly less urgent read of the situation.  While the migration process didn't remove the Cookies.mom from pbxproj, it did add all of the new Framework references to the pbxproj file, so it does seem that the migration actually took place. It just seems like if this nativeResource/Resource subdirectory existed in the app, it doesn't remove this Cookies.mom resource from the pbxproj.  I had 9 apps that got migrated, and only 3 of them had this issue so it was an interesting testcase.


Tratando JSON associativo em PHP

Question: Tenho uma aplicação Java:
<code>    Gson gson = new Gson();
    String json = gson.toJson(pedido);

    DataServiceGenerator dataServiceGenerator = new DataServiceGenerator();
    Webservice service = dataServiceGenerator.createService(Webservice.class);
    Call<String> call = service.postJson(json);
</code>
Retrofit:
<code>@POST("/servidor.php")
Call<String> postJson(@Body String pedido);
</code>
Como posso recuperar estes dados para salvar em uma tabela no php:
<code>[{
"ped_data_hora": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"ped_hor_ent_producao": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"ped_hor_retorno": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"ped_hor_sai_producao": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"ped_hor_saida": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"ped_programado": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"ped_repique": {
    "year": 2018,
    "month": 6,
    "dayOfMonth": 4,
    "hourOfDay": 23,
    "minute": 47,
    "second": 51
},
"campo1": 1,
"campo2": 2,
"campo3": 3,
"campo4": 4
}]
</code>
Já tentei assim:
<code> $resultado=json_decode(json_encode( $_POST ), true );
... $resultado->ped_data_hora;
... $resultado[0]->ped_data_hora;
... $resultado['campo1'];
</code>
Assim:
<code>$resultado=json_decode($_POST);
</code>
Já verifiquei e minha JSON está valida, só não consigo obter os valores;
Comment: Qual é o resultado que obteve em suas duas tentativas?
Comment: null ou falha na decodificação
Comment: ERRO: SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'campo1' cannot be null
Comment: Ok, mas esse é um erro no SQL, não tem relação com a leitura do JSON.
Comment: Mas confesso que estou meio perdido com a lógica de tratamento, já que tenho parte associativa na json, e parte não
Comment: sim, mas tanto com um echo ou diretamente ao sql, apresenta resultado nulo
Comment: quando dou um var_dump(), retorna isto: array(0) {}
Comment: Aliás, seu JSON não é válido, não. `"campo4": 4,` esta vírgula no final não pode existir.
Comment: Correto, eu removi alguns campos e deixei ela, vou editar!
Comment: Pensando melhor, onde é que entra o JSON aí? Se você recebe os dados via `$_POST`, ele já será um *array*. Não faz sentido convertê-lo para JSON e depois voltar a *array*; bem como não faz sentido você decodificar o JSON em um *array*. O que **exatamente** você quer fazer?
Comment: está parte que estou perdido então, estou enviando de um cliente java uma json com dados de uma classe convertidos para json, e preciso salvar estes dados numa tabela mysql no servidor
Comment: E como você está enviando esses dados? Essa informação deveria estar na pergunta, assim como essa descrição do problema que comentou.
Comment: os dados vão exatamente como coloquei na json, a questão era como manipular ela do lado servidor, mas vou complementar então
Comment: Vamos [continuar esta discussão no chat](https://chat.stackexchange.com/rooms/79772/discussion-between-anderson-carlos-woss-and-rodrigo).
Answer: Foi resolvido com a ajuda do colega Anderson Carlos Woss, segue:
<code>$resultado = json_decode(json_decode($json), false );
</code>
Dai trabalho com os objetos desta maneira:
<code>$resultado[0]->campo1
</code>
E assim:
<code>$resultado[0]->campo1->sub_campo
</code>
E mais uma dica importante referente a datas, que é o meu caso, para salvar em um campo Date() no mysql por exemplo, segue a dica:
<code>$data = $resultado[0]->ped_hor_ent_producao;
$formatada = "{$data->year}/{$data->month}/{$data->day}";
</code>

Tratando JSON associativo em PHP


Alocamento dinâmico/estocástico de objectos em QML

Question: Estou de momento a tentar fazer o seguinte em QML, em simultâneo:

fazer loading dinâmico de objects previamente criados um ficheiro à parte;
ao fazer aquilo que é anteriormente aqui descrito, seleccionar os objectos estocasticamente

Para isso estou, muito basicamente, a usar o código abaixo:
import QtQuick 2.0

Item {
    id: randomMIDIkeyboardSelector;
    property var random: 0;

    function randomSelection(min, max) {
      min = Math.ceil(min);
      max = Math.floor(max);
      var out = Math.floor(Math.random() * (max - min + 1)) + min;
      console.log(parseFloat(out));
      return parseFloat(out);
    }

    function createMidiKeyboard(itemToBeInstantiated) {
        var component = Qt.createComponent("MidiKeyboard.qml");
        var midiKeyboard = component.createObject(itemToBeInstantiated, {});
    }

    function randomPicking() {
        random = parseInt(randomSelection(1, 8));
        if(random == 1) {createMidiKeyboard(MidiKeyboard);}
        if(random == 2) {createMidiKeyboard(MidiKeyboard2);}
        if(random == 3) {createMidiKeyboard(MidiKeyboard3);}
        if(random == 4) {createMidiKeyboard(MidiKeyboard4);}
        if(random == 5) {createMidiKeyboard(MidiKeyboard5);}
        if(random == 6) {createMidiKeyboard(MidiKeyboard6);}
        if(random == 7) {createMidiKeyboard(MidiKeyboard7);}
        if(random == 8) {createMidiKeyboard(Midikeyboard8);}
        return random;
    }

    Component.onCompleted: randomPicking();

}

Basicamente, estou a criar um número aleatório e a usar uma função de javascript para criar objectos dinamicamente, que instancio a partir de um if statement
Contudo, quando carrego o ficheiro no documento principal, o objecto de user interface desejado não é criado na janela
Para perceberem melhor o teor conteúdo do projecto sugiro, caso tenham interesse, que consultem o link abaixo:
https://github.com/tiagmoraismorgado/TMM_QML_UI_UX_FRAMEWORK_WIP
Comment: Luiz Vieira, obrigado pelo edit.
Comment: não, luiz, ainda não consegui dar resposta ao problema. obrigado pela dica :)
Comment: Por nada Tiago. :) Bem vindo ao SOPT. Retirei as saudações porque este site não é um fórum. Infelizmente não uso QML, então não sei mesmo responder à sua pergunta. Mas outros saberão! Boa sorte!
Comment: já agora, fica a sugestão, para quem quiser dar uma vista de olhos no framework me funcionamento, segue o link com o video abaixo https://vimeo.com/188973909
Comment: Tiago, isso significa que você conseguiu resolver a sua dificuldade? Se sim, não deixe de responder você mesmo à pergunta, pois assim pode ajudar outras pessoas que tenham o mesmo problema no futuro.
Comment: Ah, ok. Enquanto isso, dá uma lida [neste tópico](http://pt.stackoverflow.com/help/no-one-answers) da ajuda a respeito do que fazer se demorar pra ter respostas. Boa sorte.
Answer: 1º tens de dar uma tamanho ao componente
2º A operação createComponent é assíncrona por defeito. Por isso tens de esperar que o componente seja carregado.
3º É preciso definir a path relativa ou absoluta dos componentes que queres carregar porque estão em pastas/caminhos diferentes.
Aqui fica o meu fix:
import QtQuick 2.0

Item {
    id: randomMIDIkeyboardSelector;
    anchors.fill: parent  // fit item size to parent size
    property var random: 0;

    function randomSelection(min, max) {
        min = Math.ceil(min);
        max = Math.floor(max);
        var out = Math.floor(Math.random() * (max - min + 1)) + min;
        console.log(parseFloat(out));
        return parseFloat(out);
    }

    function createMidiKeyboard(itemToBeInstantiated) {
        var component = Qt.createComponent(itemToBeInstantiated)
        if (component.status === Component.Ready) {
            console.log("Component created with success!! ")
            var midiKeyboard = component.createObject(randomMIDIkeyboardSelector, {});
        }
        else if (component.status === Component.Error) {
            console.log("Error loading component: ", component.errorString())
        }
    }

    function randomPicking() {
        random = parseInt(randomSelection(1, 8));
        if(random == 1) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard.qml");}
        if(random == 2) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard2.qml");}
        if(random == 3) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard3.qml");}
        if(random == 4) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard4.qml");}
        if(random == 5) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard5.qml");}
        if(random == 6) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard6.qml");}
        if(random == 7) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/MidiKeyboard7.qml");}
        if(random == 8) {createMidiKeyboard("qrc:/_files_qml/_MIDIKeyboards/Midikeyboard8.qml");}
        return random;
    }

    Component.onCompleted: {
        randomPicking();
    }

}

Adicionalmente também precisei de adicionar no ficheiro principal (instantiationTest.qml) os seguintes imports:
import QtQuick 2.0
import QtQuick.Window 2.0
import "qrc:/_files_qml/_Buttons/"
import "qrc:/_files_qml/_MIDIKeyboards/"
import "qrc:/_files_qml/_Stochastic Selectors/"

Window {
  ....
}

Espero ter ajudado.
Comment: ajudou sim, luigi. em relação ao primeiro bloco de código, tudo bom. em relação ao segundo, parti o projecto em directorias, já depois de ter criado este post, o que acabou por fazer com que, de alguma forma, o projecto ficasse com erros. alias estou com problemas em compilar, mesmo depois de ter adicionado os ficheiros, quer da forma com que o fizeste, quer da forma com que o fiz. ja experimentei fazer um clean build e um rebuild, andar a modificar definições no ficheiro de configuração do projecto mas não adiantou de muito
Comment: muito grato pela ajuda. com amizade. tiago
Comment: bastou-me criar um projecto novo e copiar para lá os ficheiros, e o problema ficou automaticamente resolvido
Answer: no meu caso coloquei o código assim:
import QtQuick 2.0
Item {
    id: randomMIDIkeyboardSelector;
    anchors.fill: parent  // fit item size to parent size
    property var random: 0;
function randomSelection(min, max) {
    min = Math.ceil(min);
    max = Math.floor(max);
    var out = Math.floor(Math.random() * (max - min + 1)) + min;
    console.log(parseFloat(out));
    return parseFloat(out);
}

function createMidiKeyboard(itemToBeInstantiated) {
    var component = Qt.createComponent(itemToBeInstantiated)
    if (component.status === Component.Ready) {
        console.log("Component created with success!! ")
        var midiKeyboard = component.createObject(randomMIDIkeyboardSelector, {});
    }
    else if (component.status === Component.Error) {
        console.log("Error loading component: ", component.errorString())
    }
}

function randomPicking() {
    random = parseInt(randomSelection(1, 8));
    if(random == 1) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard.qml");}
    if(random == 2) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard2.qml");}
    if(random == 3) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard3.qml");}
    if(random == 4) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard4.qml");}
    if(random == 5) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard5.qml");}
    if(random == 6) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard6.qml");}
    if(random == 7) {createMidiKeyboard("./../_MIDIKeyboards/MidiKeyboard7.qml");}
    if(random == 8) {createMidiKeyboard("./../_MIDIKeyboards/Midikeyboard8.qml");}
    return random;
}

Component.onCompleted: {
    randomPicking();
}

}
import QtQuick 2.0
import QtQuick.Window 2.0
import "_Buttons/"
import "_MIDIKeyboards/"
import "_StochasticSelectors/"
import "_Sliders/"
import "_imgButtons/"
Window {
id: root
width: 640
height: 320
minimumWidth: 640
maximumWidth: 640
minimumHeight: 320
maximumHeight: 320
visible: true
title: qsTr("instantiationTest")
color: "black"

    // Button1 {}            // the button looks good and is working fine, feedback is welcome
    // Button1_1 {}          // the button looks good and is working fine, feedback is welcome
    // Button2 {}            // the button looks good and is working fine, feedback is welcome
    // Button2_2 {}            // the button looks good and is working fine, feedback is welcome
    // Button3 {}            // the button is working fine, but looking terribly. feedback is welcome
    // Button3_3 {}
    // Button4 {}            // the button is working fine, but looking terribly. feedback is welcome
    // Button4_4 {}
    // Slider1 {}            // slider looks good but is buggy and faulty. feedback is welcome
    // Toggle1 {}            // toggle looks good nad is working fine. feedback is welcome
    // Toggle2 {}            // toggle looks good nad is working fine. feedback is welcome
    // Radial {}             // radial looks good and is working fine. feedback is welcome
    // Switch {}             // switch works fine. images need to be treated in photoshop, to keep black background and same size. feedback is welcome
    // UpDownArrows {}       // working and looking fine. needs some twweaking within dimensions cropping
    // PlayStop {}           // looks and works perfectly.feedback is welcome, however
    // Click1 {}             // looks and works perfectly. needs mouseX and mouseY coordinates
    // Click2 {}             // looks and works perfectly. needs mouseX and mouseY coordinates
    // MidiKeyboard {}
    // MidiKeyboard2 {}
    // MidiKeyboard4 {}
    // MidiKeyboard5 {}
    // MidiKeyboard6 {}
    // MidiKeyboard7 {}
    // Midikeyboard8 {}
    // MidiKeyboardRandomSelect {}
    // ButtonStochasticSelector {}
    // ToggleStochasticSelector {}
    // MasterStochasticallocation {}

}


Normal QQ-plot of logarithm of data does not match log-normal QQ-plot of data itself

Question: I'm working in Python statsmodels, and trying to find the distribution of the data in a numpy array. From the histogram, it looked like the data might be log-normally distributed. So, I took the logarithm of the data and plotted normal pp- and qq-plots:
fig, ax = plt.subplots(1, 2, figsize=(10, 4))
probplot = sm.ProbPlot(np.log(values), fit=True)
probplot.ppplot(line='45', ax=ax[0])
probplot.qqplot(line='45', ax=ax[1])
ax[0].set_title('P-P Plot')
ax[1].set_title('Q-Q Plot')
plt.show()

Based on the plots above, I infer that the logarithm of my data is approximately normally distributed. Therefore, my data should be (?) log-normally distributed. But, when I fit the data itself to a log-normal distribution and plot the pp- and qq-plots, it looks way off:
fig, ax = plt.subplots(1, 2, figsize=(14, 4))
probplot = sm.ProbPlot(values, dist=lognorm, fit=True)
probplot.ppplot(line='45', ax=ax[0])
probplot.qqplot(line='45', ax=ax[1])
ax[0].set_title('P-P Plot')
ax[1].set_title('Q-Q Plot')
plt.show()

If the log of the data is approximately normally distributed, shouldn't the data itself be log-normally distributed? Am I doing something wrong in the code or inferring something incorrectly?
EDIT 1: my values aren't particularly large:
count    15174.000000
mean        57.769947
std         69.944390
min          1.486459
25%         23.676258
50%         41.553398
75%         69.389144
max       1913.764130

So why would my quantiles look like that in the Q-Q plot?
EDIT 2: if I do the prob plot directly in scipy, things look different but still don't seem right:
shape, loc, scale = lognorm.fit(values)
print(shape, loc, scale)

3.74176496393 1.48645695058 1.95280308726
fig, ax = plt.subplots()
stats.probplot(values, fit=True, dist=lognorm, sparams=(shape, loc, scale), plot=ax)
plt.show()

If the log of the data is normally distributed, why does the prob plot of my lognorm fit look so far off?
Comment: Do you know how it fits the lognormal distribution? Does it estimate parameters with maximum likelihood? Or what is it doing? Eg. the maximum likelihood estimator of $\mu$ for lognormal distribution is $\frac{1}{n} \sum_i \log x_i$. That's almost certainly *not* what it's doing though. FYI a general issue with log-normal distribution is that if you compute raw moments (eg. $\frac{1}{n} \sum_i x_i$ etc...) to estimate parameters $\mu$ and $\sigma$ you tend to perform horribly because all your estimation is driven by a few observations in far right tail.
Comment: Clearly from second figure, something f'ed up is going on when fitting the log-normal distribution.
Comment: I edited the question to provide a direct scipy version. I fit the data to a lognormal distribution, get the parameters, and make a probability plot accordingly. 1) why do the statsmodels and scipy plots look so different? 2) why does the scipy plot still look like it's not a very good fit?
Answer: Look how big your values are. Look at your Theoretical Quantiles on the second (bad) set of plots. Notice anything alarming?
statsmodel.ProbPlot is experiencing some, ahem, extreme numerical errors when you feed it such large values.
We can compare the behavior of scipy.stats.probplot with statsmodel.ProbPlot as a check.
Run this code first with s=0.954. Then run it with s=5.23.
import matplotlib
import numpy as np
import statsmodels.api as sm
from scipy import stats
from scipy.stats import lognorm
import matplotlib.pyplot as plt

# generate some lognormal values
s = 0.954 # shape parameter
#s = 5.23
mean, var, skew, kurt = lognorm.stats(s, moments='mvsk')
print(mean,var,skew,kurt)
x = stats.lognorm.rvs(s, size=1000)
print("max(x) = ",np.max(x),"  min(x) = ",np.min(x))

# make Scipy probplot
fig1 = plt.figure()
ax1 = fig1.add_subplot(111)
res1 = stats.probplot(x, fit=True, dist=stats.lognorm,sparams=(s,), plot=ax1)
#res1 = stats.probplot(x, fit=True, sparams=(s,), plot=ax1)
ax1.set_title("Scipy Probplot for lognormal dist")
plt.show()

# make statsmodel probplot
fig, ax = plt.subplots(1, 2, figsize=(14, 4))
#shape, loc, scale = lognorm.fit(x)
print(shape,loc,scale)
#probplot = sm.ProbPlot(x, dist=lognorm, fit=True, loc=loc, scale=scale, distargs=(shape,))
print("max(x) = ",np.max(x),"  min(x) = ",np.min(x))
#probplot = sm.ProbPlot(np.log(x), fit=True)
probplot = sm.ProbPlot(x, dist=lognorm, fit=True)  # bad line
#probplot = sm.ProbPlot(x, dist=lognorm, loc=loc, scale=scale,  fit=True)
probplot.ppplot(line='45', ax=ax[0])
probplot.qqplot(line='45', ax=ax[1])
ax[0].set_title('P-P Plot')
ax[1].set_title('Q-Q Plot')
plt.show()

Scipy also has an unfamiliar form for the lognormal equation, see fitting log normal distribution in R vs Scipy
Addendum:
Look at the size of your Theoretical Quantiles, with values like 120,000 and 500,000. I postulate that within sm.ProbPlot, somewhere it is doing exp(large number) and getting a lot of inf's and large numbers like 500,000.
Now run this code (below) Initially, s is set to 0.954. Note how the max of the data is around 25 or so. The quantiles and histogram all have maximum values close to the maximum of the data.
Then set s to 3.74176496393, which is the fit from your data. Now things start to go awry. I just ran it and got max(x)=485,096. The plots are scaled out to 500,000 to 700,000. This is aggravated by your original code line:
probplot = sm.ProbPlot(x, dist=lognorm, fit=True)  # original code line

... which doesn't give location, scale, and shape to the routine.
Yet your data don't have a max of 200,000 - 500,000, as a shape=3.74 will give you. The max of your data is 1913 -- when you fit it, you got a shape parameter of s=3.74. Your 75% is 69; could you have some outliers that are excessively influencing the fit?
I postulate you are getting a poor fit of the lognormal distribution to your data.
import matplotlib
import numpy as np
import statsmodels.api as sm
from scipy import stats
from scipy.stats import lognorm
import matplotlib.pyplot as plt

# generate some lognormal values
s = 0.954 # shape parameter
#s = 3.74176496393
mean, var, skew, kurt = lognorm.stats(s, moments='mvsk')
print("Generated random lognormal variable x")
print("shape = ",s)
print("Distribution parameters")
print("mean = ",mean," var = ",var," skew = ",skew," kurtosis = ",kurt)
x = stats.lognorm.rvs(s, size=1000)
print("max(x) = ",np.max(x),"  min(x) = ",np.min(x))

# make Scipy probplot
fig1 = plt.figure()
ax1 = fig1.add_subplot(111)
res1 = stats.probplot(x, fit=True, dist=stats.lognorm,sparams=(s,), plot=ax1)
#res1 = stats.probplot(x, fit=True, sparams=(s,), plot=ax1)
ax1.set_title("Scipy Probplot for lognormal dist")
plt.show()

# make statsmodel probplot
fig, ax = plt.subplots(1, 2, figsize=(14, 4))
shape, loc, scale = lognorm.fit(x)

#paramters from actual data
#shape=3.74176496393
#loc=1.48645695058
#scale=1.95280308726

print("shape (sigma) =",shape," loc (mu) = ",loc," scale (alpha) = ",scale)
print("max(x) = ",np.max(x),"  min(x) = ",np.min(x))
probplot = sm.ProbPlot(x, dist=lognorm, fit=True, loc=loc, scale=scale, distargs=(shape,))
#probplot = sm.ProbPlot(np.log(x), fit=True)
#probplot = sm.ProbPlot(x, dist=lognorm, fit=True)  # original code line
#probplot = sm.ProbPlot(x, dist=lognorm, loc=loc, scale=scale,  fit=True)
probplot.ppplot(line='45', ax=ax[0])
probplot.qqplot(line='45', ax=ax[1])
ax[0].set_title('Statsmodels P-P Plot')
ax[1].set_title('Statsmodels Q-Q Plot')
plt.show()

# histograms and fit pdf
fig3 = plt.figure()
ax3 = fig3.add_subplot(111)

#xp = np.linspace(lognorm.ppf(0.01, s),
#                lognorm.ppf(0.99, s), 100)

print("s =",s)
shape, loc, scale = lognorm.fit(x)
#actual values from dataset:
#shape=3.74176496393
#loc=1.48645695058
#scale=1.95280308726
print("Shape = %3f  loc = %3f   scale = %3f" %(shape,scale,loc))

xp = np.linspace(lognorm.ppf(0.01, shape),
                lognorm.ppf(0.99, shape), 100)

ax3.plot(xp, lognorm.pdf(xp,shape),'r-', lw=5, alpha=0.6, label='lognorm pdf',figure=fig3)

ax3.hist(x, normed=True, log=False, histtype='stepfilled', alpha=0.2,figure=fig3)
ax3.legend(loc='best', frameon=False)
plt.show()

Comment: My values aren't particularly large. I edited the question to include the descriptive stats. Why would my quantiles look like that in the Q-Q plot?


Select all rows in Python pandas

Question: I have a function that aims at printing the sum along a column of a pandas DataFrame after filtering on some rows to be defined ; and the percentage this quantity makes up in the same sum without any filter:
def my_function(df, filter_to_apply, col):
    my_sum = np.sum(df[filter_to_apply][col])
    print(my_sum)
    print(my_sum/np.sum(df[col]))

Now I am wondering if there is any way to have a filter_to_apply that actually doesn't do any filter (i.e. keeps all rows), to keep using my function (that is actually a bit more complex and convenient) even when I don't want any filter.
So, some filter_f1 that would do: df[filter_f1] = df and could be used with other filters: filter_f1 & filter_f2.
One possible answer is: df.index.isin(df.index) but I am wondering if there is anything easier to understand (e.g. I tried to use just True but it didn't work).
Comment: But ```pandas.apply(func)``` doesn't do any filtering. Isn't it enough for your use case? I guess you would need to swap ```print(...)``` with ```return ...``` to some extent there.
Comment: Yes, there is probably a workaround, I was just asking out of curiosity.
Comment: Your suggested `df[df.index.isin(df.index)]` is actually the only code I found so far that works in my example, thanks. Using `df[slice(-1)` seems to select only all but the last row…? An alternative is `df[df.index != None]`, but Pycharm will display a warning.
Comment: If you know that indexes are positive, you can also use `df[df.index >= 0]`.
Answer: A Python slice object, i.e. slice(-1), acts as an object that selects all indexes in a indexable object. So df[slice(-1)] would select all rows in the DataFrame. You can store that in a variable an an initial value which you can further refine in your logic:
filter_to_apply = slice(-1)  # initialize to select all rows

... # logic that may set `filter_to_apply` to something more restrictive

my_function(df, filter_to_apply, col)

Comment: Thanks !! Also realized in the meantime that `pd.Series(data = [True]*len(df), index = df.index)` could work but much less elegant and can't be used as default argument of a function as it depends on `df`. So you definitely nailed it !
Comment: not sure if I'm missing something, but for me, df[slice(-1)] selects all BUT the last row? :O
Comment: @MatthiasLuh that's actually right - I had not paid enough attention.
Answer: This is a way to select all rows:
df[range(0, len(df))]

this is also
df[:]

But I haven't figured out a way to pass : as an argument.
Comment: You can use `slice(-1)` to act like `:` such as: `df[slice(-1)]` and even better: `index = slice(-1); df[index]`
Answer: Theres a function called loc on pandas that filters rows. You could do something like this:
df2 = df.loc[<Filter here>]

#Filter can be something like df['price']>500 or df['name'] == 'Brian'
#basically something that for each row returns a boolean

total = df2['ColumnToSum'].sum()

Select all rows in Python pandas


Passar uma variável de uma função para outra PHP Laravel?

Question: Olá alguém poderia me ajudar? Estou com problemas de passar parâmetros de função para outra. Não sei se estou fazendo certo, sou nova com PHP.
public function verificarCondutor(Request $request)
{
    //dados do formulário
    $cartaoRFID = $request->cartaoRFID;
    $cancelaTipo = $request->cancela;

    $this->indexJson($cartaoRFID);

}

 public function indexJson($cartaoRFID)
{
    $cartaoRFID = $cartaoRFID;
    return $cartaoRFID;
}

Comment: O que você fez está certo (se ambas as funções pertencem à mesma classe), mas como `indexJson` não tem utilidade alguma no exemplo fica difícil entender o que está querendo fazer.
Comment: dê um `var_dump` em `$this->indexJson($cartaoRFID)` e olhe se obteve o resultado que queria
Comment: Está tudo certo, só que você não faz absolutamente nada com o valor, se der um dd() ou var_dump() no retorno ou dentro da função, vai conseguir testar os valores.
Comment: O seu código tem erros conceituais, o seu código tem um método sem retorno e no outro você faz a mesma variável receber ela mesmo que não é necessário e para utilizar um método no outro é com a palavra reservada `$this` e o **nome do método**. O que você deseja fazer? [pode melhorar a sua pergunta editando](https://pt.stackoverflow.com/posts/375345/edit)
Answer: O que faltou foi um 'return' no primeiro método.
return $this->indexJson($cartaoRFID);

Só que a segunda função não faz nenhuma alteração na variável.
Se queres retornar um json existe uma response... Tente no primeiro método.
return response()->json($request->all());

Vai retornar todo o request.
Se queres retornar somente a variável no formato json faça.
return response()->json(['cartao' => $cartaoRFID]);

Answer: Olá, Beatriz! Você tem nesse código dois métodos, que devem estar em uma classe. Partindo desse pre-suposto você pode passar parâmetros para um outro método utilizando o $this, que faz referência ao objeto instanciado da classe, obtém o seu retorno e salva em uma variável para usá-la posteriormente.
Dessa forma, você pode fazer o seguinte:
<?php

//...
public function getIndexJson($cartaoRFID)
{
    // Faz toda a lógica que precisa dentro dessa função.
    $indexJson = [];

    // Então retorna o resultado
    return $indexJson;
}

public function verificarCondutor(Request $request)
{
    // Salva os dados enviados pelo formulário em variáveis
    $cartaoRFID = $request->cartaoRFID;
    $cancelaTipo = $request->cancela;

    // Passar o valor de $cartaoRFID para o método getIndexJson() e salva o seu retorno na variável $indexJson.
    $indexJson = $this->getIndexJson($cartaoRFID);

    //... continua com a lógica de verificar condutor
}

Espero que tenha ajudado. :)
Answer: o PHP suporta basicamente dois tipos de passagens de parâmetros para funções: passagem por valor e passagem por referência.
A passagem por valor não altera o valor da variável fora da função após sua utilização interna. Exemplo:
<?php
function add_some_extra($string)
{
    $string .= ' e alguma coisa mais.';
}
$str = 'Isto é uma string,';
add_some_extra($str);
echo $str;    // imprime 'Isto é uma string,'
?>

Entretanto, se o parâmetro da função for passado por referência através de caractere & colocado imediatamente antes do parâmetro, o valor da variável será alterado tanto dentro quanto fora da função. Exemplo:
<?php
function add_some_extra(&$string)
{
    $string .= ' e alguma coisa mais.';
}
$str = 'Isto é uma string,';
add_some_extra($str);
echo $str;    // imprime 'Isto é uma string, e alguma coisa mais.'
?>

Além disso, se for utilizar orientação a objetos, você poderia utilizar os atributos de uma classe para modificar o valor da variável desejada dentro de um método, e poder acessá-la em outro método. Com base no seu código ficaria assim:
...
class Condutor{
 public $cartaoRFID;
 public function verificarCondutor(Request $request)
 {
    //dados do formulário
    $this->cartaoRFID = $request->cartaoRFID;
    $cancelaTipo = $request->cancela;

    $this->indexJson();

 }

 public function indexJson()
 {
    return $this->cartaoRFID;
 }
}


One-statement Insert+delete in PostgreSQL

Question: Suppose I have a PostgreSQL table t that looks like
 id | name | y
----+------+---
 0  | 'a'  | 0
 1  | 'b'  | 0
 2  | 'c'  | 0
 3  | 'd'  | 1
 4  | 'e'  | 2
 5  | 'f'  | 2

With id being the primary key and with a UNIQUE constraint on (name, y).
Suppose I want to update this table in such a way that the part of the data set with y = 0 becomes (without knowing what is already there)
 id | name | y
----+------+---
 0  | 'a'  | 0
 1  | 'x'  | 0
 2  | 'y'  | 0

I could use
DELETE FROM t WHERE y = 0 AND name NOT IN ('a', 'x', 'y');
INSERT INTO t (name, y) VALUES ('a', 0), ('x', 0), ('y', 0)
    ON CONFLICT (name) DO NOTHING;

I feel like there must be a one-statement way to do this (like what upsert does for the task "update the existing entries and insert missing ones", but then for "insert the missing entries and delete the entries that should not be there"). Is there? I heard rumours that oracle has something called MERGE... I'm not sure what it does exactly.
Comment: What's wrong with running two statements?
Comment: The work that that single statement would have to do, would exactly be the same as the work those two statements do.
Comment: The text of your question says "update the existing entries and insert missing ones" which is not what your two statements are doing.  At a minimum, your conflict target should be (name, y).
Comment: @TempestasLudi If you have two simple operations, use two simple statements in a transaction. Trying to smash them together into one is *more* complexity.
Comment: Well... If there exists one clear-cut statement that does the same, two statements is overkill and extra complexity.
Comment: As in [upsert](http://www.postgresqltutorial.com/postgresql-upsert/)?
Comment: The work that the single statement has to do may be the same (although I doubt that), but the amount of work required from the programmer is less. Since the complexity of a program (the amount of work required to understand it) is increased by the SQL queries it employs, having one statement instead of two would reduce the complexity.
Comment: I meant that "update the existing entries and insert missing ones" is what upsert is for, to illustrate a case of a "statement that does two things at the same time".
Answer: This can be done with a single statement. But I doubt whether that classifies as "simpler".
Additionally: your expected output doesn't make sense.
Your insert statement does not provide a value for the primary key column (id), so apparently, the id column is a generated (identity/serial) column.
But in that case, news rows can't have the same IDs as the ones before because when the new rows were inserted, new IDs were generated.
Given the above change to your expected output, the following does what you want:
with data (name, y) as (
  values ('a', 0), ('x', 0), ('y', 0)
), changed as (
  insert into t (name, y)
  select *
  from data
  on conflict (name,y) do nothing
)
delete from t
where (name, y) not in (select name, y from data);

That is one statement, but certainly not "simpler". The only advantage I can see is that you do not have to specify the list of values twice.
Online example: https://rextester.com/KKB30299
Comment: Thank you. However, if that is the best solution, I won't bother and just do it with two separate statements. :P
Comment: @a_horse_with_no_name, do you know if there is a performance benefit from a single statement? I think it is one snapshot per statement, so I would assume fewer snapshots is better perf wise.
Answer: Unless there's a tremendous number of rows to be updated, do it as three update statements.
update t set name = 'a' where id = 0;
update t set name = 'x' where id = 1;
update t set name = 'y' where id = 2;

This is simple. It's easily done in a loop with a SQL builder. There's no race conditions as there are with deleting and inserting. And it preserves the ids and other columns of those rows.
To demonstrate with some psuedo-Ruby code.
new_names = ['a', 'x', 'y']

# In a transaction
db.transaction {
  # Query the matching IDs in the same order as their new names
  ids_to_update = db.select("
    select id from t where y = 0 order by id
  ")

  # Iterate through the IDs and new names together
  ids_to_update.zip(new_names).each { |id,name|
    # Update the row with its new name
    db.execute("
      update t set name = ? where id = ?
    ", name, id)
  }
}

Fooling around some, here's how I did it in "one" statement, or at least one thing sent to the server, while preserving the IDs and no race conditions.
do $$
    declare change text[];
    declare changes text[][];
begin
    select array_agg(array[id::text,name])
      into changes
      from unnest(
         (select array_agg(id order by id) from t where y = 0),
         array['a','y','z']
      ) with ordinality as a(id, name);

    foreach change slice 1 in array changes
    loop
        update t set name = change[2] where id = change[1]::int;
    end loop;
end$$;

The goal is to produce an array of arrays matching the id to its new name. That can be iterated over to do the updates.
      unnest(
         (select array_agg(id order by id) from t where y = 0),
         array['a','y','z']
      ) with ordinality as a(id, name);

That bit produces rows with the IDs and their new names side by side.
    select array_agg(array[id::text,name])
      into changes
      from unnest(...) with ordinality as a(id, name);

Then those rows of IDs and names are turned into an array of arrays like: {{1,a},{2,y},{3,z}}. (There's probably a more direct way to do that)
    foreach change slice 1 in array changes
    loop
        update t set name = change[2] where id = change[1]::int;
    end loop;

Finally we loop over the array and use it to perform each update.
You can turn this into a proper function and pass in the y value to match and the array of names to change them to. You should verify that the length of the ids and names match.
This might be faster, depends on how many rows you're updating, but it isn't simpler, and it took some time to puzzle out.
Comment: This only works if you know exactly what the contents of the table (with `y=0`) are. I asked for a solution that requires no prior knowledge of the table.
Comment: @TempestasLudi You can select the IDs and loop over them in a transaction using a programming language. That's what the second part of the answer does in "one" statement. It's not really to use, more to demonstrate how complex smashing everything into one statement gets.
Comment: @TempestasLudi I think I'm confused by your use of "update". Do you want to update existing rows? Or blow away all rows where `y=0` and insert a possibly different number of new rows? If it's the latter, there's no `upsert` equivalent. An `upsert` works on a single set of provided values, it's just what to do when there's an insert conflict. You want to insert from a set, and delete everything which matches a query (`y=0`) but *does not* match the set. Those are two different sets, so two statements.


How to link ipsec clients with different connections in StrongSwan?

Question: I use strongswan ipsec as VPN gateway for mobile devices (Android). In StrongSwan config I've setup 2 connections (two different subnets 10.10.10.0/24, 10.10.20.0/24 with different routing policies) for 2 different groups of users.
And I don't understand (and can't find in manuals and forums) how to link user with connection. Where and how to setup a strict user>connection relation ?
Thank you!
My  ipsec  config:
cat /etc/ipsec.conf

config setup
    charondebug="ike 1, knl 1, cfg 0"
    uniqueids=no
conn any2ex
    auto=add
    compress=yes
    type=tunnel
    keyexchange=ikev2
    fragmentation=yes
    forceencaps=yes
    dpdaction=clear
    dpddelay=300s
    rekey=no
    left=%any
    leftid=*.*.233.132 #I've masked server IP for this post. Certificate was issued for the ip address.
    left=*.*.233.132
    leftcert=server-cert.pem
    leftsendcert=always
    leftsubnet=0.0.0.0/0
    right=%any
    rightid=%any
    rightauth=eap-mschapv2
    rightsourceip=10.10.10.0/24
    rightdns=188.8.131.52,220.127.116.11
    rightsendcert=never
    eap_identity=%identity

conn ex2loc
    auto=add
    compress=yes
    type=tunnel
    keyexchange=ikev2
    fragmentation=yes
    forceencaps=yes
    dpdaction=clear
    dpddelay=300s
    rekey=no
    left=*.*.233.132
    leftid=*.*.233.132
    leftcert=server-cert.pem
    leftsendcert=always
    leftsubnet=0.0.0.0/0
    right=%any
    rightid=%any
    rightauth=eap-mschapv2
    rightsourceip=10.10.20.0/24
    rightdns=184.108.40.206,126.96.36.199
    rightsendcert=never
    eap_identity=%identity

I configure Android clients with this
https://docs.strongswan.org/strongswan-docs/5.9/os/androidVpnClientProfiles.html#_example
Comment: Cross posted [here](https://github.com/strongswan/strongswan/discussions/974).
Answer: The connection choice is based on rightid and depends on what is used as IKE identity by the client. If your users present RFC822_ADDR as identity, you can use some sort of a wildcard with * on rightid instead of %any to differentiate them.
Comment: Thank you for your comment!
 I use EAP login-password authentication ( /etc/ipsec.secrets with `test : EAP "test"`)

so the user's  rightid is user's login, as far as I understand.
Is it possible somehow to group users? (instead cretaing connection for each user)
Comment: There's no direct groups support on these attributes, but you can fake one with user naming convention, i.e. user-group and then use the wildcard as suggested above. Or go full RADIUS and use plugins that would set connection per RADIUS attributes.
Comment: Thank you, could you please provide an exmaple. I'm playing wildcards, but still not working and I have no idea why :(
Answer: In /etc/ipsec.conf
config setup
    charondebug="ike 1, knl 1, cfg 0"
    uniqueids=no
conn net1
    ...
    firstname.lastname@example.com
    ...
conn net2
    ...
    email@example.com
    ...

And then in /etc/ipsec.secrets
email@example.com : EAP "user_password"
email@example.com : EAP "user_password"

With this configs  email@example.com will be connected to net1, and email@example.com to net2 accordingly.