Random forest low score on testing data (scikit-learn)

Question: I am trying to train my model using Scikit-learn's Random forest (Regression) and have tried to use GridSearch with Cross-validation (CV=5) to tune hyperparameters. I fixed n_estimators =2000 for all cases. Below are the few searches that I performed.

max_features :[1,3,5], max_depth :[1,5,10,15], min_samples_split:[2,6,8,10], bootstrap:[True, False]

The best were max_features=5, max_depth = 15, min_samples_split:10, bootstrap=True
Best score = 0.8724
Then I searched close to the parameters that were best;

max_features :[3,5,6], max_depth :[10,20,30,40], min_samples_split:[8,16,20,24], bootstrap:[True, False]

The best were max_features=5, max_depth = 30, min_samples_split:20, bootstrap=True
Best score = 0.8722
Again, I searched close to the parameters that were best;

max_features :[2,4,6], max_depth :[25,35,40,50], min_samples_split:[22,28,34,40], bootstrap:[True, False]

The best were max_features=4, max_depth = 25, min_samples_split:22, bootstrap=True
Best score = 0.8725
Then I used GridSearch among the best parameters found in the above runs and found the best on as
max_features=4, max_depth = 15, min_samples_split:10,

Best score = 0.8729
Then I used these parameters to predict for an unknown dataset but got a very low score (around 0.72).
My questions are;

I doing the hyperparameter tuning correctly or I am missing something?
Why is my testing score very low as compared to my training and validation score and how can I improve it so that I get good predictions out of my model?

Sorry, if these are basic questions as I am new to scikit-learn and ML.
P.S: The training (+Cross validation data) has 26138 samples with 6 features/inputs (columns) and one output. The testing data has 1416 samples.
Comment: You might be a little too tuned into your training set. This won't fix your CV error but try to do a 2/3rd training set and 1/3rd test set and see if that improves your test error.
Comment: @Kristofersen: You mean I should not do Cross-validation while tuning hyperparameters, and use 2/3rd data for simple training and 1/3rd for testing?
Comment: I'm certainly no expert in this area, but whenever I've run random forests it's always been a 2/3rd 1/3rd split and run CV on the training set still. Also,  isn't there some preprocessing step or postprocessing step that will clean up the random forrests? Why not just use the default settings?
Comment: What you mean by clean up procedure? I think the settings will be different for different problems so that's why I am tuning hyperparameters.
Comment: The algorithm should be able to automatically determine the number of parameters / depth of the tree. In post processing it will build the full tree and then start trimming nodes at the bottom that don't contribute much to the prediction power. Try to run everything with default settings with 2/3 in training and see what the accuracy looks like
Comment: How the algorithm will automatically determine the parameters? e.g. if min_samples_leaf is left as default then it will be equal to 1, which will different for different cases, no?
Comment: It uses (probably) the gini index to determine which feature has the highest prediction power and that goes to the top. Then it uses the next most useful feature and so on to build the tree
Answer: I guess you have tunneled in on tuning too many non-useful hyper parameters, because an easy to use grid-search functionality allowed you to do so.
Notice all your explained variances only differ on the fourth digit. You have found, what appears to be a negligible better model settings. But even that you cannot be sure off because:

the RF model is non-deterministic and performance will vary slightly
a CV only estimates future model performance with a limited precision
nfold CV is not perfect reproducible and should be repeated to increase precision
Grid tuning should be performed with nested CV, but that is not your problem here I think.

Only "grid-tune" max_features. It has only 6 possoble values. You can run each 5 times and plot it. Check if some setting is repetitively better, probably you find anything from 2-4 perform fine. Max_depth is by default unlimited and that is optimal as long data is not very noisy. You set it to 25, which in practice is unlimited because already $2^{15}$=32000 and you "only" have 26000 samples. Changing these other hyper parameter will only give you shorter training times(useful) and/or more robust models. Thumb-rule: as explained variance is way above 50%, you do not need to make your model more robust by limiting depth of trees (max_depth, min_samples_split) to  e.g. 3. Max_depth 15 is quite deep, and probably plenty deep enough, just as 2000 are trees enough. So raising and lowering number of trees and depth within the quite fine range does not change anything, and it will be really hard and non-rewarding to find the true best setting.
So you have performed a grid search and learned that RF will have the same performance in the parameter space you have tested.
If you obtain a testset from a different source you should expect a drop in performance. Your CV only estimate the model performance, if the future test set was drawn from the exactly same population.
With 1400 tests, the sample error alone could swing the measured performance +/- 0.03, I guess.
If your swapped e.g. to boosting algorithms grid-tuning of multiple parameters would be a more rewarding tool.
To improve your model maybe you can refine your features. Look to variable importance, to see what features work well. Could you maybe derive new features with an even higher variable importance? Since your explained variance is quite high(low noise), you may benefit from swapping to xgboost. You may also spend time wondering if this chase of a better model performance of some target by some metric (explained variance) is useful specifically for your purpose. Maybe you don't need the model being that accurate when predicting large values, so you log transpose your target e.g. Maybe you only want to rank your predictions so explained variance could be replace with Spearman rank coefficient.
happy modelling:)
Comment: Thanks for your detailed answer. I have tried to look at the feature importance and found that one of the variables is not that important. I removed it and trained it again but did not give me better results. I have also tried to use 9 years climatic conditions worth of data (previously it was just 3 years) and used a high values of max_depth =100 and it gave me perfectly fit model (both on training and testing sets). But using that much data worries me that the testing data may already exist in those 9 years data (even if it is outside the training data). What you suggest?
Comment: In regards of very optimistic RF when trees a fully grown,  it may be worth to check this question + 2 answers: http://stats.stackexchange.com/questions/66543/random-forest-is-overfitting
Comment: dropping a single poor variable will not change much. Dropping 80 poor out of 100 variables may help. Tell me how it goes :)
Comment: yeah using very distant in past time series etc. would be a problem if you expect the underlying system is an a transition these years. It's not easy to make a clear cut decision of how old data to use. *Back testing* may be the least poor option to know
Comment: Well, there are only 6 variables (features) so I could drop one, which was less importance but could give a try by dropping another one.


How can I resolve a collation error in my DB migration?

Question: I am trying to migrate my db from localhost onto cleardb. Upon importing my db onto cleardb I am getting the error Unknown collation: 'utf8mb4_0900_ai_ci'. I have no idea what a collation is, nor can I find anything online about what collation's cleardb accepts. I'm guessing it has something to do with the character set that is being submitted.
What are my option on how to fix this?
thanks
Comment: set charset="utf8" and collation="utf8_general_ci" in config/database.php of MySQL array
Answer: for collation error, you need to follow the following steps
1, go to config/database.php file
2, set charset="utf8" and collation="utf8_general_ci"
final MySQL array show like below
<code>'mysql' => [
            'driver' => 'mysql',
            'url' => env('DATABASE_URL'),
            'host' => env('DB_HOST', '127.0.0.1'),
            'port' => env('DB_PORT', '3306'),
            'database' => env('DB_DATABASE', 'forge'),
            'username' => env('DB_USERNAME', 'forge'),
            'password' => env('DB_PASSWORD', ''),
            'unix_socket' => env('DB_SOCKET', ''),
            'charset' => 'utf8',
            'collation' => 'utf8_general_ci',
            'prefix' => '',
            'prefix_indexes' => true,
            'strict' => true,
            'engine' => null,
            'options' => extension_loaded('pdo_mysql') ? array_filter([
                PDO::MYSQL_ATTR_SSL_CA => env('MYSQL_ATTR_SSL_CA'),
            ]) : [],
        ],
</code>


prove that if $L(f,P)=U(f,P)$ then $f$ is constant on $[a,b]$

Question: Suppose that $f$ is a bounded function on $[a,b]$ and there exists a partition $P $of $[a,b] $such that $L(f,P)=U(f,P)$. Prove that $f$ is constant on $[a,b]$
I know that $L(f,P)=U(f,P)$ meaning $f$ is Darboux integrable, I also know that I need to prove that $f'(x)=0$ to show that $f$ is constant on $[a,b]$. But I don't know how to link those two.
Comment: Please look at william Wade's introduction to anaysis book. It is explained there so good.
Comment: do you have any free version link of this book?
Comment: There may be exist on the website " libgen.info "
Comment: I'll check that out, thanks
Answer: Hint: Suppose $f$ is a real-valued function defined on some closed interval $I = [\alpha, \beta]$, and $\inf\{f(x) : x \in I\} = \sup \{f(x) : x \in I\}$. What does this tell you about the set of values of $f$ on $I$?
Comment: oh, the least upper bound and the greatest lower bound are the same so the values of $f$ on $I$ are the same , thank you.


Como é que o Laravel "guarda" as rotas?

Question: No Laravel, tanto no 4 como no 5, podemos utilizar uma classe chamada Route para poder criar determinadas rotas.
A minha curiosidade é: como é que o Laravel (ou outros frameworks) fazem para "saber" que determinada rota corresponde à uma url?
Por acaso tudo isso é salvo em algum array e depois, quando a requisição é feita, a rota é chamada?
Onde fica a lista dessas rotas?
Isso é importante saber, principalmente para quem quer aprender a fazer uma framework.
Comment: Negativo? O que há de errado com a pergunta?
Comment: Até onde eu sei é feito um "mapeamento" entre a rota, e controller, função ou que será executado, quando está rota for acessada. Acredito que seja feito algum tipo de comparação (if aninhado) até encontrar a definição que bate com a url requerida.
Comment: Pelo nível de código fonte do Laravel duvido realmente que seja "ifs aninhados". Inclusive o Laravel 5 tem como fazer cache de rotas, não faz sentido ser "um monte de ifs"
Comment: Tá, foi um palpite apenas .... Mas não importa bem como ele faz, por que no fundo, em qualquer linguagem de programação, a tarefa de tomar um caminho, ou outro, é sempre feito por meio de condicionais, seja ela no próprio script, ou em baixo nível, talvez em uma operação requisitada pelo interpretador ao SO, executada diretamente no processador ... hehe ...
Answer: Eu não entendi bem a pergunta, mas vou supor que usem $_SERVER['PATH_INFO'] ou $_SERVER['REQUEST_URI'], no .htaccess do Laravel/Symfony não deve usar PATH_INFO porque ele esta assim:
RewriteRule ^ index.php [L]

O path_info só é gerado no php se fizer algo semelhante a isto:
RewriteRule (*.?) index.php/$1 [L]

No caso PATH_INFO não funciona muito bem em alguns servidores diferentes do Apache, as vezes o resultado é diferente, mas o REQUEST_URI funciona de maneira idêntica no Apache, Ngnix, Lighttpd e IISExpress (não pude testar no IIS padrão mas acredito ser a mesma coisa).
A estrutura PHP deve ser algo como (essa é a estrutura que usei em um framework pessoal, no entanto esta simplificada, sem verificação de POST, GET, PUT, etc):
class Route
{
    private static $currentPath;
    private static $routes = array();

    public function path()
    {
        if (self::$currentPath) {
            return $currentPath;
        }

        //Pega o nome do script atual
        $sname  = $_SERVER['SCRIPT_NAME'];

        //Remove a query string da url
        $reqUri = empty($_SERVER['REQUEST_URI']) ? null : preg_replace('#\?(.*)$#', '', $_SERVER['REQUEST_URI']);

        //Subtrai o nome do script (se necessário)
        $pathInfo = substr(urldecode($reqUri), strlen(substr($sname, 0, -9)));
        $pathInfo = '/' . ($pathInfo === false ? '' : $pathInfo);

        return self::$currentPath = $pathInfo;
    }

    public function add($path, $controller)
    {
        self::$routes[$path] = $controller;
    }

    public function exec()
    {
         if (empty(self::$routes[$path])) {
              return false;
         }

         return self::$routes[$path];
    }
}

Existem muitas maneiras de "guardar" tais rotas, no caso da minha classe eu guardo em um array na mesma classe:
private static $routes = array();

Mas pode-se até guardar em lugares separados, vai depender do objetivo de uso final.
O uso seria algo como:
require 'route.php';

Route::add('/', 'ClasseController@indexaction');
Route::add('/blog', 'ClasseController@blogaction');
Route::add('/admin', 'AdminClasse@action');

echo 'Path: ', Route::path(), '
', PHP_EOL; //Retorna o PATH equivalente ao PATH_INFO
echo 'Controller: ', Route::exec(), '
', PHP_EOL; //Retorna o controler

Não posso afirmar que funciona exatamente desta maneira no Laravel/Symfony, mas esta explicação é pra entender como usar o $_SERVER['REQUEST_URI'] ou PATH_INFO.
Laravel e o cache de Rotas
No Laravel existe uma estrutura de cache para as rotas (o qual você me citou) que após o comando (não funciona com funções anonimas):
php artisan route:cache

É gerado um arquivo em projeto/bootstrap/cache/routes.php, ele contem uma variavel php "serializada" (e codificada em base64) e usa o método setRoutes pra definir todas rotas que estão em cache pro "Collection", um exemplo de cache:
app('router')->setRoutes(
    unserialize(base64_decode('Código base64'))
);

Isto pode ser razoavelmente vantajoso para os servidores de produção.
Comment: Sim, o `Laravel` usa o `REQUEST_URI`. Na verdade, não é bem o `Laravel`, é o Symfony, que é usado pelo `Laravel`. O exemplo retrata sim o que acontece no `Laravel`, mas claro de maneira mais simplificada, visto que o `Laravel` usa 3 classes só pra fazer o roteamento :d
Answer: Bem, vamos explicar por partes:
O Laravel, quando utilizamos a chamada da classe  Route, na verdade você está chamando um facade da classe Illuminate\Routing\Router.
Um Router serve para facilitar a criação de rotas (Illuminate\Routing\Route), que por sua vez será criado dentro de uma coleção de rotas pré-definidas (Illuminate\Routing\RouteCollection).
Ao você invocar o Route::get('/', 'HomeController@getIndex') por exemplo, você está na verdade fazendo o seguinte:
 app('route')->get('/', 'HomeController@getIndex');

Que é a mesma coisa que :
app('route')->addRoute(['GET', 'HEAD'], '/', 'HomeController@getIndex');

Então temos que entender que:
1 - Route é um facade (fachada) para classe Router.
2 - Router é uma classe responsável pela criação de rotas dentro de uma coleção.
3 - RouteCollection é a lista de rotas. Cada vez que é chamado Route::get ou Route::post, essa rota é inserida dentro dessa classe.
4 - Route (não o facade, mas Illuminate\Routing\Route) é a rota propriamente dita. É a classe que representa uma entidade que é uma rota.
Internamente, o Laravel tem um método em RouteCollection que pega a primeira rota que bate com a expressão regular da URI da rota.
Quando nenhuma é encontrada, retorna uma exceção (que é o erro 404).
Veja o método RouteCollection::match, que faz esse serviço.
/**
 * Find the first route matching a given request.
 *
 * @param  \Illuminate\Http\Request  $request
 * @return \Illuminate\Routing\Route
 *
 * @throws \Symfony\Component\HttpKernel\Exception\NotFoundHttpException
 */
public function match(Request $request)
{
    $routes = $this->get($request->getMethod());

    // First, we will see if we can find a matching route for this current request
    // method. If we can, great, we can just return it so that it can be called
    // by the consumer. Otherwise we will check for routes with another verb.
    $route = $this->check($routes, $request);

    if ( ! is_null($route))
    {
        return $route->bind($request);
    }

    // If no route was found, we will check if a matching is route is specified on
    // another HTTP verb. If it is we will need to throw a MethodNotAllowed and
    // inform the user agent of which HTTP verb it should use for this route.
    $others = $this->checkForAlternateVerbs($request);

    if (count($others) > 0)
    {
        return $this->getOtherMethodsRoute($request, $others);
    }

    throw new NotFoundHttpException;
}

Então, podemos concluir, conforme foi perguntado, que as rotas são salvas numa coleção, que é responsável por retornar uma rota registrada de acordo com o conteúdo da requisição (o método http e a uri).
Comment: mas aí é um pouco diferente do que perguntou... esse é a parte onde as rotas vão sendo armazenadas quando são encontradas..O que entendi da pergunta é como que o laravel traduzia as rotas..
Comment: Estou sentindo um tom de represálias nesses negativos. Ainda dá pra perceber que alguns usuários na comunidade precisam amadurecer
Comment: "Encontradas" não, quando são "declaradas". São "encontradas" (ou não) de acordo com a uri passada.
Comment: não entendi.. rsr encontrada ou declarada dá no mesmo pois a lógica do  padrão psr4 é "encontrar (o arquivo)" e carregar se estiver "declarada (a classe)"
Comment: Esquece então, deixa pra lá. Vamos ver se outros usuários também estão com dúvida sobre o que eu perguntei.
Answer: O Laravel usa o autoloader para identificar rotas. Você pode optar também por rotas pré definidas.
Para os casos onde não há pre definição de rotas e o Laravel identifica-as automaticamente, muito provavelmente está aplicando o padrão PSR4, o qual pode ser definido no composer
 "autoload": {
         "psr-4": {
             "App\\": "app/",
              "Mylib\\": "src/"
        }
    },

exemplo ilustrativo
https://laracasts.com/lessons/psr-4-autoloading
Sempre que uma rota é invocada e encontrada, é armazenada em cache.
Não estou dizendo que o Laravel utiliza somente o PSR4. Isso é opcional nas configurações.
Esse cache das rotas é algo que pode ser bom ou ruim. Pois sempre carregará um arquivo contendo todas as rotas a cada requisição ou a cada vez que incoca, por exemplo. Está carregando ali uns kb que podem ser mais pesados do que verificar a rota em tempo de execução. Num projeto pequeno de 10, 20 rotas, pode ser vantagem, mas acima de 100, acredito que não.
Para otimizar, existe o route group onde podem ser divididos os arquivos de rotas, deixando-os em tamanhos menores. Por isso, quem usa muitas rotas deve usar esse recurso.
Comment: Para entender melhor porque você foi negativado, consulte o manual: https://laravel.com/docs/5.4/routing#basic-routing :)
Comment: O autoload é parte essencial do routing. É isso alguns aqui não entendem e por isso negativaram.
Comment: Esta pergunta não tem relação alguma com autoloading, não por acaso nenhuma das outras respostas abordam isso. Pegue o código de routing do Laravel, jogue tudo num arquivo só, adicione rotas fictícias com closures e você ainda poderá responder a pergunta. Não vou nem entrar no mérito do caching porque sua argumentação também está completamente equivocada.

Como é que o Laravel "guarda" as rotas?


Does the bonferonni correction change the outcome of the p-value or is it just changin what you accept as significant?

Question: For some reason I can't seem to integrate the bonferonni corrected p adjustment into my multiple comparisons t test in r so I need to do each of my 3 tests individually.
What I wondered was do I need to change something about the t tests or do I just look at the p values they produce and see if they're greater or less than my bonferonni corrected p value (0.016) after having done a normal t test.
Comment: Answering the title question: it is the latter. The correction changes the chosen significance level (0.016 instead of 0.050), not the test statistic with its associated p-value.
Answer: You can look at it from either angle.
I would tend to agree with @RichardHardy that it is easier to understand in terms of the latter: reject a null $H_i$ if the corresponding $p$-value $p_i$ is less than $\alpha/n$, where $\alpha$ is the desired level and $n$ the number of tests performed: reject if $p_i\leq \alpha/n$.
But you can also calculate "adjusted" $p$-values - so do something which might fall under the header of "changing" the test statistic - that allow to you continue to use the default cutoff $\alpha$. In the Bonferroni case, all you need to do is to multiply the $p_i$ by $n$ and reject if the adjusted $p$-values are less than $\alpha$: reject if $n\cdot p_i\leq \alpha$.
This is also what p.adjust in R does:
> p.adjust(c(.01,0.04), method = "bonferroni")
[1] 0.02 0.08


Servlet 3.0 file upload (@MultipartConfig) with embedded jetty

Question: When using jetty embedded, ie registering a servlet like this:
ServletContextHandler context = new ServletContextHandler(server, "/", false, false);

s = new UploadServlet(jee);
context.addServlet(new ServletHolder(s), "/upload");

Tthe MultipartConfig annotation does not seem to be respected. How do register URL's that support file upload under Servlet 3.0? Reverse engineering the Jetty code reveals it must be something like this, but what is the "multi"???
ServletHolder sh = new ServletHolder(new UploadServlet(jee));
Object multi;
sh.getRegistration().setMultipartConfig(new MultipartConfigElement(multi));
context.addServlet(sh, "/upload");

Answer: The MultipartConfigElement actually has multiple constructors. One takes an instance of a MultipartConfig annotation. The others take only the location or all the available options as parameters. So the easiest possiblity using all default will be
sh.getRegistration().setMultipartConfig(new MultipartConfigElement(""));

Answer: Jacob,
Processing annotations with Jetty is currently only supported on WebAppContexts, and then also only if annotation processing is enabled. The jetty-9 doco talks about enabling annotations for a web app with stand-alone jetty here: http://www.eclipse.org/jetty/documentation/current/annotations.html
So, convert your ServletContextHandler to a WebAppContext, and call setConfigurationClasses() on it with the following list:
    "org.eclipse.jetty.webapp.WebInfConfiguration",
    "org.eclipse.jetty.webapp.WebXmlConfiguration",
    "org.eclipse.jetty.webapp.MetaInfConfiguration",
    "org.eclipse.jetty.webapp.FragmentConfiguration",
    "org.eclipse.jetty.webapp.JettyWebXmlConfiguration",
    "org.eclipse.jetty.annotations.AnnotationConfiguration"

and make sure the annotation processing-related jars on your classpath - from the jetty distro these are:
javax.annotation-1.1.0.v201108011116.jar  org.objectweb.asm-3.1.0.v200803061910.jar
Jan
Comment: Thanks. I checked out the link in your post, but I couldn't see anything about "enabling" annotations for embedded jetty, only for standalone jetty.
Comment: @Jacob here's an [example of using Embedded Jetty with Servlet 3.0 Annotations](https://github.com/jetty-project/embedded-servlet-3.0/blob/master/src/test/java/com/company/foo/EmbedMe.java).


To show that $\lim_{n\rightarrow\infty}$ $\frac{n^{2}}{2^{n}} = 0$

Question: We need to show that that $\lim_{n\rightarrow\infty}$ $\frac{n^{2}}{2^{n}} = 0$.
This makes sense intuitively since exponentials always rush to infinity faster than do polynomials, but I don't know how to prove it.
Cool thing is, this is actually not from a math class, but from my CS class, where we have to compare runtimes. Regardless, can someone show me how to prove this without using differentiation? Thanks!
Answer: Hint: Can you show that $0 \le \frac{n^2}{2^n} \le \frac{1}{n}$ for $n \ge 10$?
Comment: @user00000000001899 Try induction or check the great answers on the duplicate question.


Proving $abcd+3\geq a+b+c+d$

Question:
If $a,b,c,d$ are non negatives and $a^2+b^2+c^2+d^2=3$ prove that  $$abcd+3\ge a+b+c+d$$

The inequality is not as simple as it looks.The interesting part is that the equality occurs when $a=0,b=c=d=1$ upto permutation .(I don't know if there are further equality cases.)
I tried rewriting the inequality as $$a^2+a(bcd-1)+b^2+c^2+d^2-b-c-d\ge 0$$ As its a quadratic in $a$ it suffices to show  the discriminant$$\Delta_a={(bcd-1)}^2-4(b^2+c^2+d^2-b-c-d)\le 0$$ which unfortunately is wrong (when $a=\sqrt{3},b=c=d=0$)
P.S ;I am not aware of using Lagranges multipliers
Its from here
Update :An answer has been posted in the link above (AOPS) similar to Dr Mathva's answer
Comment: -- *FYI* --
The statement holds more generally for three, or more than four real variables without positivity assumption:
$$\text{If $\,3\leqslant n\,$ and}\;\sum^n_{d=1}a_d^2=n-1,
\text{ then }\:\sum^n_{d=1}a_d\:\leqslant\:\prod^n_{d=1}a_d+n-1,$$
cf page 57 in the [GAZETA MATEMATICA](https://ssmr.ro/gazeta/gma/2020/gma3-4-2020-continut.pdf#page=57) .
Notice that $\,a_d\,$ encodes Albus Dumbledore ...
Comment: I haven't figured out how to use that but in [post](https://artofproblemsolving.com/community/c6h404087p2252927) #6 arqady (I guess a.k.a. @MichaelRozenberg at MSE) seems to give a hint...
Comment: @VIVID nice find! thanks for your interest
Comment: As the problem is symmetric with respect to all variables, I tried to take $a=b=c=d=\frac{\sqrt 3}{2}$ and got $$a b c d-(a+b+c+d)\ge \frac{9}{16}-2 \sqrt{3}\approx -2.9$$ which is the exact minimum. I don't write this as an answer because without analysis I can't prove that the minimum happens when the variables are equal.
Comment: @Raffaele interesting ,that means  this  inequality is infact  weaker......
Comment: @AlbusDumbledore Yes. Pierrecarre has proved it, using real analysis methods (Lagrange multipliers). (Please edit to correctly write Lagrange's name, greatest Italian mathematician :)
Answer: Due to symmetry, assume wlog that $a\geqslant b\geqslant c\geqslant d\geqslant 0$. Since I will use this fact later, I am proving it before the cases:

Lemma: We have that $2\geqslant a+d$.

In fact, this follows from Cauchy-Schwarz:  $$2= 2\sqrt{\frac{a^2+b^2+c^2+d^2}{3}}\geqslant 2\sqrt{\frac{a^2+3d^2}{3}}=\sqrt{1+\frac13}\cdot \sqrt{a^2+3d^2}\geqslant a+d$$
We will now consider some cases depending on the position of the number $1$:

Case 1: $1\geqslant a\geqslant b\geqslant c\geqslant d\geqslant 0$. Observe that one may rewrite the inequality as $$3+abcd-(a+b+c+d)=(1-a)(1-b)+(1-c)(1-d)+(1-ab)(1-cd) $$ Which is clearly positive.

Case 2: $a\geqslant 1\geqslant b\geqslant c\geqslant d\geqslant 0$. If $ab\geqslant 1$, we might proceed as in the next case. So we will work with $ab<1$. Thus \begin{align*}3+abcd-(a+b+c+d)&=(1-a)(1-b)+(1-c)(1-d)+(1-ab)(1-cd)\\&\geqslant \underbrace{(1-a)}_{<0}(1-b)+(1-c)(1-d)\\&\geqslant (1-a)(1-c)+(1-c)(1-d)\\&=(1-c)(2-(a+d))\geqslant 0  \end{align*} Where the last inequality follows from the lemma.

Case 3: $a\geqslant b\geqslant 1\geqslant c\geqslant d\geqslant 0$. This implies that \begin{align*}6+2abcd-2(a+b+c+d)&=a^2+b^2+c^2+d^2+3+2abcd-2(a+b+c+d)\\
&=(a-1)^2+(b-1)^2+(c+d-1)^2+2cd(ab-1)\geqslant 0\end{align*} Or, equivalently $3+abcd\geqslant a+b+c+d$.

Case 4: $a\geqslant b\geqslant c\geqslant 1\geqslant d\geqslant 0$. As @dezdichado noticed, this case is straightforward, since it forces directly $a=b=c=1, d=0$ due to the constraint $a^2+b^2+c^2+d^2=3$.

Case 5: $a\geqslant b\geqslant c\geqslant d\geqslant 1$. Clearly impossible, since this would yield $a^2+b^2+c^2+d^2\geqslant 4>3$.

Done!
Comment: your case $3$ would have forced $a = b= c = 1$ and $d = 0$ anyway due to the constraint. +1
Comment: I don't think so? E.g. $a=\sqrt{3}, b=c=d=0.$ Then L.H.S. $=0$ but R.H.S. =$\sqrt{3}+1$.
Comment: @AlbusDumbledore I also miss Michael Rozenberg; he's able to solve (almost) every inequality, and some of them with crazy computations (such as BW or uvw ones)!! It's sad that he does not answer questions anymore on this site (does anybody now why?). Anyway, he still comes and looks around, since he's been active at least yesterday. Hope he comes back
Comment: @Dr.Mathva :) He is still answering questions in AOPS, I am a fan of him too!
Comment: Ingenious(+1) I doubt a better answer will be there
Comment: Thanks @AlbusDumbledore! Although I am disappointed that I could not use Michael Rozenberg's hint on aops. I thought that [Turkevichi's inequality](https://artofproblemsolving.com/community/c6h1709871p11020490) would lead somewhere, but was unable to prove that $ab+bc+cd+da+ca+bd\geqslant a+b+c+d+1$ (I am not even sure if this is true ...)
Comment: @Dr.Mathva that was my same thought of using the turkevicus inequality :) Micheal Rozenberg has surprisingly been not answering questions for the past 3 months else i am sure he would have solved the problem using  his hint:).Nevertheless this answer is worth a  bounty!
Comment: @SeeHai Thanks for pointing that out; I feel so dumb to have missed it ...
Comment: @Dr. Mathva please don't say so! Your solution was very nice indeed.
Comment: One way I was thinking of doing this problem (probably will get the best bound) was to extend the $uvw$ to a $uvwx$ for four variables, because this is useful for inequalities where the number of variables is important. But by the time I could figure out five to six base equalities to work with, your answer was up! +1
Comment: How do you say that $b^2 + c^2 + d^2 \geq 3d^2$?
Comment: I assumed wlog that $a\geqslant b\geqslant c\geqslant d$. Therefore, since the variables are positive $$b^2\geqslant d^2, c^2\geqslant d^2, d^2\geqslant d^2\implies b^2+c^2+d^2\geqslant 3d^2$$
Comment: @Dr.Mathva An answer has been posted in AOPS forum see https://artofproblemsolving.com/community/c6t243f6h404087_inequality
Answer: Since $a^2+b^2+c^2+d^2 = 3$ defines a compact set in $\mathbb{R}^4$, $f(a,b,c,d)=abcd+3-a-b-c-d$ will have a global minimum and maximum over that set, that will occur (can be easily justified) in critical points of the Lagrangian
$$
L(a,b,c,d,\lambda) = abcd+3-a-b-c-d -\lambda(a^2+b^2+c^2+d^2-3)
$$
So, just compute the critical points, compute the value of $f$ over each of these points and get the global minimum. If the global minimum is $\ge 0$, you are done.
When computing the critical points, you will get some complex solutions, some solutions with negative components but, in the end, the relevant solutions are the ones you already mentioned and also $a=b=c=d=\frac{\sqrt{3}}{2}$. The minimum value of $f$ (even admitting negative values for $a,b,c,d$) is zero (and the maximum is $\frac{57}{16}+2\sqrt{3}$).
Edit: Missed one critical point, but the result stands.
Comment: My point is, yes, one doesn't need to provide all the calculations, but your answer is simply *try Lagrangian Multiplier*. It doesn't provide any further information **specific to this question** other than the generic method. E.g. Is it a straight forward process as solving a quadratic polynomial or what should OP expect from using this method to this question?
Comment: IHMO, the bulk o f Lagrange multiplier approach is **always** working on the derivatives so as to find the global extrema, which your answer didn't shed any light on. Your answer might just be a comment as VIVID posted.
Comment: @QuangHoang the solution is fine. Smooth function on a compact set attains its maximum and minimum in that set and they can only be achieved at a critical point. Then, you use Lagrange to find all the critical points and then find the maximum and minimum and by just looking at their values. Your concern would be valid if there happen to be an infinite number of critical points, so inspecting them individually would be hard, but it's not the case here assuming his computation is right.
Comment: sorry but what is a compact set and langrangian thing i am not at all aware
Comment: @AlbusDumbledore A compact set in $\mathbb{R}^4$ is a bounded and closed set. THis is relevant because a continuous function ($f$) always attains a global maximum and minimum values over a compact set.
Comment: @AlbusDumbledore Most likely, that is the [Lagrange Multiplier](https://en.wikipedia.org/wiki/Lagrange_multiplier) method
Comment: The Lagrangian is the the function $L$ I defined in the post and is used as a tool to compute candidates to min/max with restrictions.
Comment: The critical points of $L$ are the  vectors $(a,b,c,d,\lambda)$ that make all the partial derivatives be zero simultaneously.
Comment: @QuangHoang Providing an answer does not mean making all the calculations. I think I outlined the milestones.
Comment: @PierreCarre This is supposed to be a contest question and such methods i am not aware even if  one is able to solve those equaions,I had also mentioned i am not ware of langranges multipliers .Any way maybe i van fathom your idea after i learn more mathematics
Comment: @AlbusDumbledore I did not interpret your comment on not being aware of the method as a way of dismissing it as a solution process. I'm sure there are purely algebraic methods to solve the question and, I would gladly provide one if I had found it!
Comment: @PierreCarre OK ,thanks for your effort grateful for that!
Comment: @dezdichado No. The restriction places us in the surface of the ball. The process I proposed allows the calculation of the global max/min on the surface of the ball.
Comment: Yeaj my bad lol i confuser myself with the second derivative test
Comment: global extrema attains at **either** critical points or on boundary. Your solution did not consider boundary point, and also **did not** show that the critical point you (sort of guess) is a maximum or a minimum. The computed value is obviously larger than the case when $a=b=c=1, d=0$, which gives the minimum stated in the problem.
Comment: @QuangHoang I think you are confused about how Lagrange multipliers work. What do you think is the boundary in this case? We are computing the max/min over a set that is the surface of a sphere. Considerations about the rank of the Jacobian matrix of the restrictions allow you to say that in this case the max/min will occur on critical points. And the global max/min will be one of those.
Comment: @QuangHoang And of coarse the value on this other point will be greater than the global minumum! But it is necessary to look at this additional point to make sure that the global mimumun is attained at the other critical points.
Comment: $a=b=c\approx0.721545, d\approx1.19922$ and $a=b\approx0.556325, c=d\approx1.09110$ are other critical points
Answer: This is a remark on Dr. Mathva's answer, which may seem very mysterious at first but becomes more transparent when you consider a simpler problem:

If $a, b \geq 0$ with $a^2 + b^2 = 1$, prove that
$$ ab+1 \geq a+b \,.$$

(It is the problem from the question if you impose $c = d = 1$.)
Here it is clear that you can factorize it as $(a-1)(b-1) \geq 0$, so you are naturally led to distinguish cases according to whether $a, b \geq 1$ or $\leq 1$. If both are $\geq 1$ or both are $\leq 1$, we are done. So suppose wlog $a \geq 1 \geq b$. Then plug in the constraint to get
$$ab + a^2+b^2 \geq a+b \,.$$
Now play a bit to recover every term in the RHS from the LHS. Because $a \geq 1$, we can just remove factors $a$ from the LHS. That is, $ab \geq b$ and $a^2 \geq a$, and we are done.
By considering this simpler problem, you can more easily get the idea of assuming wlog $a \geq b \geq c \geq d$, and of distinguishing cases according to the position of $1$.
Answer: See, in the link below, our extension to n real variables (GMA Problem 494, page 57, with the our solution):
https://ssmr.ro/gazeta/gma/2020/gma3-4-2020-continut.pdf

Screenshots from the link provided:
Comment: Sorry but link is not working for me
Comment: It's impossible. The link is valid and it's Open Access for everyone, not for Romania only.
Comment: If the issue still persists, leave me your email adress and I'll send to you the pdf.
Comment: @user120123 Just to support: the link is indeed working now.
Comment: Great. I'm glad to hear this.
Comment: @VIVID its not working for me :( says the site cat be reached
Comment: Then leave your email adress and the problem is solved.
Comment: @AlbusDumbledore I added the screenshots from the file if user120123 does not mind.
Comment: Great initiative. Bravo!
Comment: @VIVID wonderful thanks! and to  user120123 thanks

Proving $abcd+3\geq a+b+c+d$


Determination of effect size for a repeated measures ANOVA power analysis

Question: This is a follow-up to the repeated measures sample size question.
I am planning a repeated measures experiment. We record energy usage for 12 months, then give (a randomly assigned) half of the customers continuous information about their energy usage (perform the treatment), and record their energy usage for another 12 months. A similar study performed in the past showed a 5% reduction in energy usage.
I want to estimate the required sample size using $\alpha=0.05, \beta=0.1$. G*Power 3 has a tool for repeated measures power analysis. However, it requires two inputs that I am not entirely familiar with:

$\lambda$ - the noncentrality parameter (How do I estimate this?)
$f$ - the effect size (I believe that this is the square root of Cohen's $f^2$)

According to Wikipedia's effect size page:

Cohen's $f^2= {R^2_{AB} - R^2_A \over 1 - R^2_{AB}}$ where $R^2_A$ is the variance accounted for by a set of one or more independent variables $A$, and $R^2_{AB}$ is the combined variance accounted for by $A$ and another set of one or more independent variables $B$.

However, my expected 5% change in energy consumption does not tell me how much variability will be explained. Is there any way to make this conversion?
If you know of a way to do this power analysis in R, I would love to hear it. I am planning to simulate some data and try using lmer from the lme4 package.
Answer: Assuming you are going to average the first 12 months to form a baseline measure and the second 12 months to form as a follow-up measure, your problem reduces to a repeated measures t-test.
G*Power
You might want to check out the following menu in G*Power 3:
Tests - Means - Two Dependent Groups (matched pairs).
 Use A priori, $\alpha=.05$, Power = 0.90.
Use the Determine button to determine effect size. This requires that you can estimate  time 1 and 2 means, sds, and correlation between time points.
If you know nothing about the domain, based on my experience in psychology, I'd start with something like
M1 = 0, SD1 = 1, SD2 = 1
correlation = .60

This means that M2 is basically a between subjects cohen's d.
You could then examine a few different values of M2 such as 0.2, 0.3, ... 0.5, ... 0.8, etc. Cohen's rules of thumb suggest 0.2 is small, 0.5 is medium, and 0.8 is large.
R
UCLA has a tutorial on doing a power analysis on a repeated measures t-test using R.
Side point
As a side point, you might want to consider having a control group.


Passando informação para uma outra Activity

Question: Bom dia estou tentando passar 2 dados de uma activity para outra mas não estou conseguindo segue abaixo o código relevante na primeira Activity :
 @Override
    public void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (resultCode == Activity.RESULT_OK) {
            Uri contactData = data.getData();
            Cursor c =  getContentResolver().query(contactData, null, null, null, null);
            if (c.moveToFirst()) {

                String phoneNumber="";
                String name = c.getString(c.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME));
                String contactId = c.getString(c.getColumnIndex(ContactsContract.Contacts._ID));
                String hasPhone = c.getString(c.getColumnIndex(ContactsContract.Contacts.HAS_PHONE_NUMBER));

                if ( hasPhone.equalsIgnoreCase("1"))
                    hasPhone = "true";
                else
                    hasPhone = "false" ;

                if (Boolean.parseBoolean(hasPhone))
                {
                    Cursor phones = getContentResolver().query(ContactsContract.CommonDataKinds.Phone.CONTENT_URI, null,ContactsContract.CommonDataKinds.Phone.CONTACT_ID +" = "+ contactId,null, null);
                    while (phones.moveToNext())
                    {
                        phoneNumber = phones.getString(phones.getColumnIndex(ContactsContract.CommonDataKinds.Phone.NUMBER));
                    }
                    phones.close();
                    Bundle bundle = new Bundle();
                    bundle.putString(name, String.valueOf(name));
                    bundle.putString(phoneNumber, String.valueOf(phoneNumber));

                }
                //mainActivity.onBackPressed();
                // Toast.makeText(mainactivity, "go go go", Toast.LENGTH_SHORT).show();

            //    tvname.setText("Name: "+name);
              //  tvphone.setText("Phone: "+phoneNumber);
              //  Log.d("curs", name + " num" + phoneNumber);
            }
            c.close();
        }
        finish();

    }

Segue abaixo o código relevante da Segunda Activity :
 intent = getIntent();
 Bundle getBundle = null;
 getBundle =  intent.getExtras();
 Bundle bundle = intent.getExtras();
 String name = bundle.getString("name");
 String phone = bundle.getString("phoneNumber");
 messageContact = name+'\n'+ phone;

Comment: Diego, alguma resposta resolveu seu problema? Teria algo a mais que possamos explicar?
Comment: @Diego, na verdade, o que voce precisa, pelo que pude ver, e iniciar a activity corretamente, ja que voce coleta os dados no metodo `onActivityResult` e isso so funciona se voce chamar sua segunda activity com `startActivityForResult(Intent intent, int requestCode)` e retornar dela para a primeira activity chamando o metodo `setResult (int resultCode, Intent data)`
aqui tem um bom exemplo: https://www.javatpoint.com/android-startactivityforresult-example
Answer: Você chama o bundle corretamente, mas falta passar o bundle pelo "Intent".
Segue exemplo (Com bundle):
        Bundle bundle = new Bundle();
        bundle.putString("id_do_item", "seu_dado");
        Intent intent = new Intent(context, MinhaOutraActivity.class);
        intent.putExtra(bundle);
        startActivity(intent);

Para acessar na outra view,use o seguinte codigo:
id_user = getIntent().getExtras().getString("id_do_item");

Comment: Matheus! mas é o seguinte! eu dou finish em uma activity! se voce reparar eu abro uma activity pra pegar o numero de contato(Agenda do usuário) que o usuário selecionar, logo depois eu finalizo e retorno pra primeira Activity
Comment: É na vdd quando eu postei isso vc n tinha feito a edição ainda....
Eu vou Analisar seu código de novo e edito a minha resposta
Comment: certo Matheus, estou no aguardo
Answer: Erro
O seu erro é estar criando um novo bundle e não passá-lo para activy, ou poderia usar o bundle do onCreate.

Como fazer
Há 2 formas:
Sem o Bundle (porque seu intent já tem um próprio no onCreate)
public void teste2(View v) {

    Intent i = new Intent(this,Teste.class);
    i.putExtra("site","site2.com");
    startActivity(i);
}

Com um novo Bundle:
public void teste(View v) {

    Intent i = new Intent(this,Teste.class);

    Bundle bd = new Bundle();
    bd.putString("site","google.com");

    i.putExtras(bd);
    startActivity(i);
}

Com novo bundle, usando o putExtras.
Com o bundle do onCreate, seria o putExtra.
Comment: Raoni! mas é o seguinte! eu dou finish em uma activity! se voce reparar eu abro uma activity pra pegar o numero de contato(Agenda do usuário) que o usuário selecionar, logo depois eu finalizo e retorno pra primeira Activity
Comment: Então você tem tratar na outra activity. Cada activity passa os dados para a próxima.
Comment: Não entendi muito bem
Comment: Por que tu nao chama `startActivity(intent);` apos dar `finish()` na primeira activity?
Comment: Ou salve em algum `sharedpreferences` seus dados entao, e recupere na próxima `activity`
Comment: Toda vez que sai de uma `activity` para outra, precisa passar os dados. Então se você está na `activity 1`, passou os dados para a `activity 2`, e quer voltá-los para a `activity 1`, então também precisa fazer o mesmo processo na `activity 2`. Ou como o LMaker disse, usar `sharedpreferences`, ou outro tipo de `armazenamento` dependendo do que você precisa.
Comment: @LMaker consegui! porém ele está enviando 2 vezes a mensagem para o usuário
Comment: @LMaker se eu chamar pra abrir a activity ele vai dar nullpointer nao?


Run a hadoop cluster on docker containers

Question: I want to run a multi-node hadoop cluster, with each node inside a docker container on a different host. This image - https://github.com/sequenceiq/hadoop-docker works well to start hadoop in a pseudo distributed mode, what is the easiest way to modify this to have each node in a different container on a separate ec2 host?
Comment: did you found the solution, if yes then please guide me
Answer: I did this with two containers running master and slave nodes on two different ubuntu hosts. I did the networking between containers using weave. I have added the images of the containers on docker hub account div4. I installed hadoop in the same way, as its installed on different hosts. I have added the two images with coomands to run haddop on them here:
https://registry.hub.docker.com/u/div4/hadoop_master/
https://registry.hub.docker.com/u/div4/hadoop_slave/.
Answer: The people from sequenceiq have created a new project called cloud-break that is designed to work with different cloud providers and create hadoop clusters on them easily. You just have to enter your credentials and then it works the same for all providers, as far as I can see.
So for ec2, this will now probably be the easiest solution(especially because of a nice GUI):
https://github.com/sequenceiq/cloudbreak-deployer


bash flock: Why 200?

Question: Regarding that thread:
bash flock: exit if can't acquire lock
I'll appreciate if someone can explain to me what does the '200' stand for.
I've read about flock and it seems that 200 if to specify a File Descriptor, but what is so good about this number?
Comment: It's just an arbitrary file descriptor number.
Answer: Theres nothing special about the number 200. It just happens to be the example used in the man page of the flock command; and it happens to be a large number, so it's unlikely to conflict with the the file descriptor of any other file you open during your script.
In your comment, you ask about:
(
  flock -e 200
  echo "In critical section"
  sleep 5
) 200>/tmp/blah.lockfile
echo "After critical section"

The parentheses () create a subshell; a new process, separate from the parent process. The 200>/tmp/blah.lockfile causes that process to open up /tmp/blah.lockfile for writing, on file descriptor 200. The commands inside the parentheses are executed within that shell.
flock -e 200 obtains an exclusive lock on the file pointed to by file descriptor 200. An exclusive lock means that anyone else who tries to obtain a lock on that file, either exclusive or shared, will block (wait) until this lock has been relinquished, or fail if they hit a timeout or asked not to block. So during the remainder of the body of the subshell (the echo and sleep commands), the lock will be held by that subshell, and no one else can obtain that lock. Once the subshell finishes, the file will be closed and lock relinquished.
Comment: @Rapher It's not technically necessary; you could take the lock before executing the code, and release it after. However, it makes it convenient to manage the lock; you are guaranteed that the lock will be dropped when the subshell exits, so you can't forget to drop the lock (or fail to drop it for some other reason), and it's easy to see the range for which the lock applies by just matching parentheses.
Comment: so if you are using this technique in multiple scripts, you use different numbers? or different lock file names?
Comment: Thanks for the answer.
Anyway, is there an easy way to make sure there's no conflict? "Unlikely" may not always be enough.
Comment: @didi_X8 Use a real programming language and not Bash. Bash is for quick and dirty scripts; if you want guarantees about behavior, it's best not to use a language that has so many weird and surprising behaviors (like executing the content of environment variables). That said, in Bash you don't open new file descriptors unless you explicitly do a redirection using that file descriptor; `exec 3>/some/file`, or a subshell with a similar numbered redirect. So you can just keep track of the numbers manually, keeping in mind that 0, 1 and 2 are stdin, stdout, and stderr.
Comment: Thanks. I'm new to bash scripting, could you please explain to me the rest of the code: `(
    flock -e 200
    echo "In critical section"
    sleep 5
) 200>/tmp/blah.lockfile
echo "After critical section"`
Comment: Thanks! There is one thing I don't understand, though. Why is the subshell necessary for? Why not just take the lock before executing the code and let it go after the code execution has finished ?
Comment: Great answer. Thanks!
I was also wondering about these exact same issues. (the parentheses and the 200).
Comment: It seems that if I run two instances of the same script with this code in parallel that one of them never obtains the lock and just sits there. Any thoughts?
Comment: @SanderVerhagen Usually that's what you try to achieve with a lock - not being able to run more than one instance of the same thing at the same time.
Comment: Obviously (?!) what I meant was: never obtains the lock _after other process(es) have gone away_. But I moved on to a different solution, thank you.
Comment: What happens if you use the same number for two different files?
`200>/tmp/${client_one}.lockfile` and `200>/tmp/${client_two}.lockfile`  Will they conflict? the idea is that we call the same bash script, once per client.

bash flock: Why 200?


Roland TD11KV Problem recording using a flash drive

Question: So basically I've been trying to record myself playing drum using a USB flash drive and when I plug the flash drive on the td11 module it assums that the flash drive is actually there, but then when I plug the flash drive on my computer so I can hear what I record it sounds completely different from what I played...
Does anyone knows how to solve this?
I've red that this problem can be because i'm not using a flash drive from Roland but I dont know
Comment: Can you clarify the problem - is the recording incorrect? The drive is being recognised? You are recording audio or digitally? Could you upload a clip of the sound so we can hear the problem?
Answer: The Roland TD11 is an electronic drum set.   The USB stick has two purposes.  You can play an audio file from it, so that you can play drums along to an already recorded song.   Or you can record your drum playing as a MIDI file.
Now think what a MIDI file is.  It doesn't contain any sound.  It just has data of which pad you hit, how hard and in what rhythm.   Play it back through the drum set, you'll get the sounds you expect.   Play it on your computer, you'll get whatever sounds the computer has.
Answer: The flash drive might be too large. Many older devices can't use flash drives larger than 8GB. Also, make sure there is enough space on the drive and that it is formatted in FAT32.


Filter from master list using array of objects in where clause linq C# query

Question: I have a master list which contains objects of type A , refer the below screen shot

Now I want to filter out the master list based on the values in array of object matching Ids in master list.
Comment: What have you tried and what are you getting stuck on?
Comment: [Please do not upload images of code/errors when asking a question.](https://meta.stackoverflow.com/q/285551/8967612)
Comment: It should be the other way around... `masterlist.Where(x => values.Contains(x.Id)).Select(x => x.Id).ToList()`. Select statement will get you just the IDs u want
Comment: return masterSpecList.Where(x => x.ID.Contains(values.ToString())).ToList();
Comment: this what I tried using contains in linq from master to filter out but its values.ToString() returns objects not the string of values , may be that is causing error . I am not sure about that
Comment: yes, its working  , can you please post it as answer or do I need to update my question with more details ?
Answer: <code>masterSpecList.Where(x => values.Contains(x.ID)).Select(x => x.ID).ToList().
</code>
Select statement will get you just the IDs u want. Contains should be the other way around like in the example above


Rendering form manually

Question: When making my ticket-system I want managers to be able to reply to tickets when viewing the ticket. On the page to view any ticket I load my form. Because I want it to be a bit more fancier than just the default 'element under element' style I render each item by item manually.
<?php print drupal_render($form['open']); ?>

Now the problem is, my form doens't work. Clicking on my rendered submit doesn't perform any action.
My guess is that rendering each item manually doesn't include the 'form container'. How should I include is? Or is my style of working completely wrong?
Answer: The best way to implement custom rendering of a form would be to use a custom theme hook for the form:
/**
 * Builder function for the ticket form.
 */
function MODULE_ticket_form($form, $form_state) {
  // Build the $form array here...

  // Let the theme API knows that this form use it own theme hook.
  $form['#theme'] = 'ticket_form';
  return $form;
}

/**
 * Implements hook_theme().
 */
function MODULE_theme() {
  return array(
    // Custom theme hook for the ticket form.
    'ticket_form' => array(
      'render element' => 'form',
    ),
  );
}

/**
 * Default implementation for the ticket_form theme hook.
 */
function theme_ticket_form($variables) {
  $form =& $variables['form'];

  //Custom rendering of forms elements goes here...
  $output = ...;

  // Render every other elements (incl. hidden system ones).
  $output .= drupal_render_children($form);
  return $output;
}

Instead of using only a theme function, you can also use a template file. The theme function can also return a render array instead of a string.
Answer: Try following will work:
<?php $form = drupal_get_form('yourform_id'); ?>

<?php print '<form id="'.$form['#id'].'" accept-charset="UTF-8" method="'.$form['#method'].'" action="'.$form['#action'].'">'; ?>
<?php print drupal_render($form['form_id']);?>
<?php print drupal_render($form['form_build_id']);?>
<?php print drupal_render($form['form_token']);?>
<?php print '</form>'; ?>

Rendering form manually


Stop double click from happening after single click

Question: I developing an application in WPF where a single click on an item will change the view to something else. The problem is that if the user double clicks on the item, the first event will be the single click that opens the other view and the next event will be the double click on the new view. This double click on the new view will cause other things to happen.
So, the question is;
Can I stop the double click from happening after I get the single click event to my first view? I have tried to set Handled = true on the MouseButtonEventArgs in the single click method but that does not work. The other view will still get the double click and random things will still happen. Any smart solution of my problem out there?
/Daniel
Answer: RaymondChen is once again a valuable source.
I suggest you read the linked half second delay article for deeper details but that one shows how the delay manifests itself to the user.
Comment: the problem with this solution is that during this half second, the user could click something else and then two things will happen. The thing that happens after the first click on the first item and the thing that happens after the second click on the second item. This could be solved by setting some kind of state in a model for the app so that it knows that a first item already has been klicked. Not so nice solution though.
Comment: If your users are clicking that fast on multiple different elements of the UI I fear custom code and state stored in the appropriate places is the only solution...


Maximize browser window for specific test case

Question: code:
Open browser to homepage
    Run Keyword If  '${browser}' != 'firefox'
    ...     Set Selenium Implicit wait    30
    Run Keyword If  '${platform}' == 'mobile'
    ...     Open with emulator
    ...     ELSE IF    '${platform}' == 'desktop'
    ...     Run keywords
    ...     Open Browser    url=${homepage}     browser=${browser}    #remote_url=${HUB_URL}
    ...     AND    Maximize Browser Window

How to maximize browser window for specific test case only?
Comment: It looks like you're doing it. How is what you're asking different from what you're already doing?
Comment: I want to do this for one test case only. And other test cases should not use this.

Actually, I've added one more test setup:
"Open browser to homepage with non maximized window"

But it returns:

Setup failed:
Keyword 'Selenium2Library.Open Browser' expected 1 to 6 arguments, got 0.
Comment: Have you tried simply closing the maximized window when you're done? The next time you open a browser window it won't be maximized.
Answer: Add an argument to your keyword Open browser to homepage. If you give it a default value, you will not have to touch existing calls to the keyword.

Open browser to homepage
    [Arguments]    ${maximize}=false
    ...
    Run Keyword If    "${maximize}".lower()=="true"    Maximize Browser Window


Why is the equation $z=(x+y)^2+y^2$ a paraboloid?

Question: Why is the equation $z=(x+y)^2+y^2$ a paraboloid?, as the right side is expanded, it is $z=x^2+2xy+2y^2$.
The equation of a paraboloid in some sources is simply  $z=\frac{x^2}{a^2}+\frac{y^2}{b^2}$ and there is no the product of $x$ and $y$ in the equation.
Answer: A paraboloid is something which can be described by such an equation after possibly making a change of coordinate. On the right hand side, if you replace $x = x' - y$ you get the standard form, $z= (x')^2 + y^2$.
Comment: what do you mean by making change of coordinate?
Comment: Well in the answer we changed from (x,y,z) coordinates to (x',y,z) coordinates. The shape is the same but viewed from a different angle. In general a change of coordinate usually mean making an invertible linear substitution like above.
Answer: The equation $z={x^2\over a^2}+{y^2\over b^2}$ isn't the equation of a paraboloid, any more than ${x^2\over a^2}+{y^2\over b^2}=1$ is the equation of an ellipse in the $x$-$y$ plane. Those sources you quote likely mentioned somewhere that this is the equation of a paraboloid in "standard position," that it's the "standard form" of the equation, or some other wording along those lines. It describes an elliptic paraboloid that has its vertex at the origin, the coordinate axes as its principal axes, and opens "upward" in the direction of the positive $z$-axis. Often, we also assume that $a\ge b$. This is a very special subset of all possible elliptic paraboloids.
The paraboloid that you have has been rotated approximately 32° from this standard position, which you can see if you plot the surface. If you were to apply the inverse rotation to it, you'd end up with an equation in the standard form. We don't need to do this, however, as the original form of the equation of this paraboloid makes it easy to find its spectrum, which can be used to distinguish among the various types of quadric surfaces: it's $(1,1,0,-1)$, which is the spectrum that all elliptic paraboloids have. (Compare this to, say, the spectrum of an ellipsoid, which from the standard equation is $(1,1,1,-1)$.)
Answer: You may get rid of the term in $xy$ by performing the transformation $$x=u+\frac{1+\sqrt 5}{2}v,\,y=\frac{1+\sqrt 5}{
2}u-v,$$ which gives you an equation in standard form.
Comment: The $uv$ term still there. You need $x=u+\phi v,\ y=\phi u-v$ with $\phi=\frac{1+\sqrt{5}}2$ to keep an orthogonal basis and eliminate $uv$.
Comment: @zwim Oh, thanks. That's true. The squares here have different coefficients.


Which of these 2 formulas is the right one for estimating sample size for proportions?

Question: I've seen two formulas for estimating a sample size proportion, the first one:
s = 1.96^2 * p * (1-p) / MOE^2

Where:

1.96 is the Z score for getting a 95% confidence interval.
p is the estimated proportion (0.5 if unknown)
MOE is the margin of error (0.05)

This formula is independent from the total population. There's also another one that reads:
S = N * s / (N + s - 1)
Where:

N is the population.
s is the sample size estimated with the previous formula.

My question is, which of these would you use and when? (the second one seems more complete since it takes into account the population size).
Answer: I think it depends on the population you want to make inferences about. The latter one just corrects for a finite or small population size.
I think often people assume infinite populations since they want to make inferences about a population that goes beyond the group they have sampled from.
So if you want to make some test on the exam pass rate of this year in faculty X and you know that only N exams were taken and you are only interested in this year in your faculty a correction is appropriate (e.g. because your sample size can obviously not be larger than your population size). If you want to make a general statement about pass rates of e.g. boys in general an infinite population size would have to be assumed.
As you can see S approaches s as N increases
p = 0.5
MOE = 0.05

s = 1.96^2 * p * (1-p) / MOE^2

df = data.frame()
for(N in seq(10,10^6,10)){
    S = N * s / (N + s - 1)
    d1 = data.frame(N, S)
    df = rbind(d1,df)
}

plot(df$N, df$S-s)

Answer: I would use the first one, assuming that the population is very large (in theory, infinite). However, I believe you misunderstand one of the concepts. MOE (margin of error) is not 1-confidence level (so in your example, not necessarily 0.05). It is the error that you allow for you proportion estimation, so it could be 0.01, 0.02, or whatever you want. it is not related to the confidence level (at least not directly). You choose each of them separately. You want your proportion estimate to be accurate within some percentage (say +- 0.02), and you want to claim this with a confidence of 95%.
Comment: Thanks Zahava, I understand the difference. My bad anyway, I probably expressed myself wrong.


git rm --cached file vs git reset file

Question: I'm trying to learn Git. I'm confused between
git rm --cached file

and
git reset file

both of the commands seem to take the file from staged to un-staged area. How do the commands differ?
Comment: possible duplicate of [What's the difference between \`git rm --cached asd\` and \`git reset head -- asd\`](http://stackoverflow.com/questions/5798930/whats-the-difference-between-git-rm-cached-asd-and-git-reset-head-asd)
Answer: git rm --cached <file> will completely remove the file's contents from the index. This means that on commit the file will be removed from the HEAD commit. (If the file was only added to the index and not yet tracked this is a "no-op".)
git reset -- <file> resets the contents of the file in the index to be the same as the head commit. This means that on commit no changes will be committed to the file. This operation is not valid if there is no tracked version of the file in the HEAD commit.
Comment: Maybe as an addition: `git rm --cached` will add the *delete* action of the file to the index, just like `git add` will add an *add* action.
Comment: is there a way to view files that are in the index? (even if they match the HEAD)
Comment: Is index==staging area? If a file is in the index, isn't it tracked by extension?
Comment: For git reset -- , there should be a tracked version of that file in HEAD commit. If it's new file added to index, then there will be no tracked version in HEAD commit
Comment: So `git reset` doesn't reset the index to what it was right after the last commit, namely _empty_. It keeps the list of staged files, but resets their _contents_ to that of the last commit. Therefore, `git reset` isn't the opposite of (doesn't undo) `git add`.
Comment: @bubaka I think you're looking for `git ls-files -s`. It lists all the files in the index, whether staged or not. What it shows will be what will be put in to your next commit.

git rm --cached file vs git reset file


Pandas count based on condition in current row from records before current row

Question: I have a dataframe show as follows:
import pandas as pd
df=pd.DataFrame({'col1':['a','a','a','b','b','c']})
df.sort_values('col1', inplace=True)
df['Ref']=0

Thus the dataframe looks like:
a  0
a  0
a  0
b  0
b  0
c  0

For the ref column, I want to show the number of reference of current row. For illustration purpose, following is what I want to achieve:
a  0
a  1
a  2
b  0
b  1
c  0

I can use df.iterrows() and loop row by row. Un fortunately in my case, it will take 15 minutes to run. I am wondering if there is a reasonable way to do so.
Comment: Can we assume `df['col1']` is ordered? Or is `[a,a,b,a,b]` possible, but requiring the same results?
Comment: Yes you can assume it is sorted!
Answer: Group the data by col1 and use cumcount
import pandas as pd

df = pd.DataFrame({'col1':['a','a','a','b','b','c']})

df['Ref'] = df.groupby('col1').cumcount()

df.sort_values('col1', inplace=True)

Output:
>>> df

  col1  Ref
0    a    0
1    a    1
2    a    2
3    b    0
4    b    1
5    c    0


How to load conditionally a script depending on whether element exists in DOM?

Question: I have a Drupal.behaviors.leftmenusticky script that converts a left menu to a sticky menu (100% height,position fixed and the like) but I want to drupal_add_js it only if the menu actually exists in the page shown.
(function ($) {
  if ($(".full-height").length > 0 ) {
  Drupal.behaviors.leftMenuSticky = {
      attach: function (context, settings) {
        var contentHeight = $("#main").innerHeight();
        var viewportHeight = $(window).height();
        var offsetTop = $(".full-height").offset().top;
        if (contentHeight < viewportHeight) {
          jQuery(".full-height").height(viewportHeight-offsetTop);
        }
        else {
          jQuery(".full-height").height(contentHeight-offsetTop);
        }
        leftColumnResizer();
        $(window).bind("resize", leftColumnResizer);
        function leftColumnResizer () {
          var toto = $(".region-sidebar-first").width();
          $("#block-paddle-menu-display-management-level-4").css("width", toto);
        }
      }
    }
  }
  else {console.log('notfound');}
})(jQuery);

The Javascript way would be to check whether or not the class I'm targetting exists or not:
  if ($(".full-height").length > 0 )

but 1) I don't know where to place this check exactly and 2) I'd prefer to have a template.php way of doing that by making use either of settings or of context...
Is this possible?
Answer: drupal_add_js() happens server-side, it can't possibly have exposure to what's in the DOM (unless you pass that info back to the server with AJAX, but that would be a very weird thing to do). The correct solution is the javascript one as you've mentioned.
You want that code to run whenever the DOM is ready, so in the Drupal paradigm immediately inside the attach function of your behaviour:
(function ($) {
  Drupal.behaviors.leftMenuSticky = {
      attach: function (context, settings) {
        if ($(".full-height").length > 0 ) {
          var contentHeight = $("#main").innerHeight();
          // ...


Java: Parse multiple String Date Time format into LocalDateTime/Date/Calender

Question: I am trying to parse string date Time into LocalDateTime or Date in Java.
I may get the inputs Like 12/21/2020 12:12:12 PM, 1/12/2020 2:6:8 PM, 10/2/2020 10:50:8 AM
I tried to use different types of patterns under DateTimeFormatter or DateTimeFormatBuilder Which Can Work. Brute force method Where I am Hard coding different combination of pattern. Here's the code copied from the link:
import java.time.format.DateTimeFormatter;
import java.time.format.DateTimeFormatterBuilder;
import java.util.Date;
import java.text.SimpleDateFormat;
import java.time.LocalDateTime;

class Main {
    public static void main(String[] args) {
        // many combinations
        final String DATE_FORMATS = "[MM/dd/yyyy HH:mm:ss a][M/dd/yyyy H:mm:ss a]";
        DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern(DATE_FORMATS);

        System.out.println(dateFormatter.parse("12/21/2020 11:12:12 AM"));
        System.out.println(dateFormatter.parse("1/12/2020 2:54:55 AM"));
    }
}

Is there any other solution as I don't have control over a input? Single pattern which,I can use in Formatter Instead of multiple combination ? Or Is it better to modify the input after receiving before parsing?
Comment: How can we tell you if there is a solution *different from the one you haven't shown here*? Please edit your question and add your version, thank you!
Comment: What do you mean by "brute force method"? The standard way to parse a `String` into one of the `java.time` classes is `DateTimeFormatter`. Please show a [mre] demonstrating the problem you're having.
Comment: As for your question, what sort of solution are you looking for? If you have multiple potential formats then you're going to need multiple patterns or a single pattern that can handle all the different patterns (if even possible). The better option is to control the input and require it to be a certain format.
Comment: Please show us what you tried, what you'd expected and what happened instead!
Comment: Talk is cheap, show the code!
Comment: I copied the code from your link into your question as questions should be self-contained.
Answer: tl;dr
LocalDateTime.parse(dateTimeString, DateTimeFormatter.ofPattern("M/d/yyyy h:m:s a");
is sufficient for all of your datetime examples.

The following example is able to parse all of your sample datetime Strings by using LocalDateTime.parse instead of DateTimeFormatter.parse, which are different:
public static void main(String[] args) {
    String first = "12/21/2020 12:12:12 PM";
    String second = "1/12/2020 2:6:8 PM";
    String third = "10/2/2020 10:50:8 AM";

    DateTimeFormatter dtf = DateTimeFormatter
            .ofPattern("[MM/dd/yyyy HH:mm:ss a][M/dd/yyyy h:m:s a][MM/d/yyyy HH:mm:s a]");

    System.out.println(LocalDateTime.parse(first, dtf));
    System.out.println(LocalDateTime.parse(second, dtf));
    System.out.println(LocalDateTime.parse(third, dtf));
}

Have a look at the second / middle pattern used, it uses h (lower case) for hours instead of an H (upper case) because the AM/PM of day will be derived from a capital H, too and that would lead to a conflict in the second datetime sample "1/12/2020 2:6:8 PM" where 02:06:08 is considered AM, but is followed by PM in the pattern.
The output of my solution is this:
2020-12-21T12:12:12
2020-01-12T14:06:08
2020-10-02T10:50:08

which correctly parses the time to PM (14 = 02 PM).
Note:
Don't use this DateTimeFormatter for output  LocalDateTime.format(DateTimeFormatter) because it would return all three formattings defined...
Very ugly: 10/02/2020 10:50:08 AM10/02/2020 10:50:8 AM10/2/2020 10:50:8 AM
Comment: Pattern `DateTimeFormatter dtf = DateTimeFormatter.ofPattern("[M/d/yyyy h:m:s a]");`  Works for all the different combination. If we Do `LocalDateTime.parse`
Comment: @Thilakraj OK, thanks, good to know. I hadn't checked just a single of the patterns, used all of them. Will edit that into the answer!
Answer: I dealt with a similar problem, where I needed to parse a String to date without knowing the format in advance. I wrote an article about the idea and the implementation. Here is a quote from the article:
So the solution I came up with is to have a set of formats stored in property file, and when a String needs to be parsed the formats are read from a file and attempts to parse the String are made sequentially with each format until it is parsed successfully or until we run out of formats. The advantages of this solution are that if you discover a  valid String that was not parsed successfully, all you will need to do is to add new format to your properties file and no re-compilation and re-deployment is needed. Also this way you can set your priorities: say if US date format is preferable to  European one just place US formats first and only after the European ones. Also in java 8 the format strings allow for optional format sections denoted by '[]'. So several formats actually may be combined into a single one with optional sections. For example instead of
MM/dd/yyyy
MM-dd-yyyy
MM.dd.yyyy

you can just write
MM['/']['-']['.']dd['/']['-']['.']yyyy

Here is the link to the full article
Comment: @deHaar - Thank you so much. BTW I have some other articles on the same site. Also, if you could leave a short comment for the article I would appreciate it very much
Comment: I didn't find good words for a comment so far, but I liked using the platform *like*-function ;-)
Comment: While informative it doesn't appear to me to be very relevant to the question asked.


Jeffreys prior on invariant likelihoods

Question: For a likelihood $p(y | \theta)$ and pdf $f(y)$:
Suppose that a likelihood is location invariant i.e.
$p(y | \theta) = f(y - \theta)$
Show that the Jeffreys prior is of the form $p(θ) ∝ 1$.
I understand that we have to use the Fisher information to solve this, but am confused about the idea of location invariance.
Fisher = $\sqrt{-E[d^2/(d\theta)^2 \log (f(y - \theta))]}$
which has to be proportional to $1$, but how do I use $f(y - \theta)$ in this equation for Fishers information?
Comment: This is an exercise and should have the `self-study` tag. You should also provide more details on why you cannot prove that the information is constant.
Comment: I don't really understand how this location invariance comes into this idea. I can back track and realize that the log of the second derivative of f(y - theta) must be a constant, but I can't see why that's the case.
Answer: When considering the Fisher information
$$\mathfrak{I}=-\int_\mathcal X\frac{\partial^2 \log(f(x-\theta))}{\partial\theta^2}  f(x-\theta)\,\text{d}x$$
we have that
$$\frac{\partial^2 \log(f(x-\theta))}{\partial\theta^2}=-\frac{\partial}{\partial\theta}\frac{f'(x-\theta)}{f(x-\theta)}=\frac{f'(x-\theta)}{f(x-\theta)}+\frac{f''(x-\theta)^2}{f(x-\theta)^2}$$
and making the change of variable $y=x-\theta$ (with Jacobian one) in the integral(s)
\begin{align*}
\int_\mathcal X\frac{\partial^2 \log(f(x-\theta))}{\partial\theta^2}  f(x-\theta)\,\text{d}x &= \int \frac{f'(x-\theta)}{f(x-\theta)} \,\text{d}x + \int\frac{f''(x-\theta)^2}{f(x-\theta)^2} \,\text{d}x\\
&= \int \frac{f'(y)}{f(y)} \,\text{d}y + \int\frac{f''(y)^2}{f(y)^2} \,\text{d}y
\end{align*}
which is constant in $\theta$.

Jeffreys prior on invariant likelihoods


Every group the homology of some space?

Question: Given a specific group, up to isomorphism, is there a way to determine a topological space, up to homeomorphism, with said group as the nth homology?
In other words, is there an established algorithm to work backwards from a specific group (like $\mathbb{Z}_2$) and end up with some topological space?
Answer: The only requirement is that the group is abelian. One such construction is the Eilenberg-MacLane space which is a space such that all homotopy groups except one are trivial. For existence see chapter 4 of Hatcher. These spaces answer your question because the Hurewicz Theorem tells us that the nth homology will be the nth homotopy group since the space is n-1 connected.
Another construction is the Moore Space which is the the homology analogue of the Eilenberg-MacLane space, also constructed in Hatcher.

Every group the homology of some space?


An Alternate Solution to a Differential Equation

Question: The main question being asked: is it possible an $f(x)$ which satisfies the below equation where $u=u(x)$;if so, how?
$$\boxed{ \frac{df(u^2)}{d(u^2)} = \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \quad ,u = u(x)\qquad(*)}$$
or similarly after using the transformation in the Remarks section:
$$\boxed{\frac{df(u^2)}{du} = 2u\left[\left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2}\right] \quad ,u = u(x) \qquad (**)}$$

Part 1:
This stems from the initial question assigned to me which was:
Find an $f(x)$ which satisfies:
$$ f'(\sin^2x) = \cos^2x + \tan^2x, \quad 0<x<1$$
(I have left this part in 'prime' notation because this is how the original question was presented.)
The proper solution involves the following method:
$$ f'(\sin^2x) = (1-\sin^2x) + \frac{\sin^2 x}{1-\sin^2 x}$$
Let $u = \sin^2x$, then:
$$ f'(u) = (1 - u) + \frac{u}{1-u}$$
$$ f(u) = \int\,\left[(1 - u) + \frac{u}{1-u} \right] du$$
$$ f(u) = \int\,\left[(1 - u) + \frac{u-1+1}{1-u} \right] du$$
$$  = \int\,\left[-u + \frac{1}{1-u} \right] du$$
$$ \therefore f(x) = -\frac{1}{2}u^2-\ln|1-u|=-\frac{1}{2}x^2-\ln|1-x|$$

Part 2:
My question is if this can  be solved with the following alternate method:
$$ \frac{d(f(\sin^2x))}{d(\sin^2x)} = \cos^2x + \tan^2x, \quad 0<x<1$$
then,
$$\frac{d(f(\sin^2x))}{d(\sin^2x)} = \cos^2x + \frac{\sin^2x}{\cos^2x}$$
Let $u = \sin x$, $du = \cos x\,dx$ then:
$$ \frac{d(f(u^2))}{d(u^2)} = \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \qquad$$
Can this equation be solved to find an $f(x)$ which satisfies this relation regardless of what $u(x)$ actually is?
See Main Edit 1&2 for clarification on why it is in this form
The following steps are INCORRECT, as this is not how the chain rule works. But there is something in my head grinding such that I feel that it can be solved in a way invoking the chain rule but I can't get my thoughts wrapped around it properly.
(I have left this part in 'prime' notation because this displays my erroneous thought process of misunderstanding differentials which probably led me to the wrong answer.)
$$ 2u\, f'(u) = \left(u'\right)^2 +\frac{u^2}{\left(u'\right)^2}$$
Clearly $ f'(u^2) \ne 2u\, f'(u)$, however continuing with this incorrect thought..
$$ f'(u) = \frac{1}{2} \left[\frac{\left(u'\right)^2}{u}+\frac{u}{u'}\right]$$
$$ f(u) = \int\frac{1}{2} \left[\frac{\left(u'\right)^2}{u}+\frac{u}{u'}\right] $$
Is there anyway I can actually correctly use the chain rule starting from $(*)$ to reach this point and then integrate, or solve it through some method for differential equations? I've never experienced a question with the square of a derivative so I was wondering if someone could give me insight on this.

Main Edit 1:
As @JJacquelin noted, the notation with 'prime' causes quite problem with what the question being asked is:
Using the fact that $u = \sin x\,\, \text{is a function of}\, x$
The line:
$$ f'(u^2) = \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \qquad$$
can be written as such:
$$ \frac{df(u^2)}{d(u^2)} = \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \quad ,u = u(x) \qquad(*)$$
And as so, I have replaced all appropriate parts of this post with Leibniz notation to suggest clarity.
Now can I find an $f(x)$ which satisfies this equation?
I have left my previous work as is, so that I may reference it one day in case I come across a problem of pinpointing what my prime notation means.
Please do let me know if there are any further clarifications that need to be made before this can be solved properly.

Edit 2: Proof of why [incorrectly] believed it is $\frac{d}{dx}$:
Thank you to @JJacquelin once again for helping me clarify the differentials in the question. I have edited the general question with reflection to their analysis, and hope to recieve the answer I was looking for now that the notion of differentials is cleared up. I was incorrect in my reasoning before and it is actually $\frac{df(X)}{dX}$ for any dummy variable $X$, contrary to what I had thought as you can see below.:
If we start from the solution of Part 1:
$$f(x) = -\frac{1}{2}x^2-\ln|1-x|$$
$$\frac{df(x)}{dx} = -x+\frac{1}{1-x}$$
$$\frac{df(x)}{dx} = -x+\frac{1-x+x}{1-x}$$
$$\frac{df(x)}{dx} = -x+\frac{1-x}{1-x}+\frac{x}{1-x}$$
$$\frac{df(x)}{dx} = -x + 1 + \frac{x}{1-x}$$
$$\frac{df(x)}{dx} = 1 - x +\frac{x}{1-x}$$
Now $x \mapsto \sin^2x$:
$$\frac{df(\sin^2x)}{dx} = 1 -\sin^2x+\frac{\sin^2x}{1-\sin^2x}$$
$$\frac{df(\sin^2x)}{dx} = \cos^2x + \tan^2x$$
But this turns out to be wrong, when $x\mapsto \sin^2 x$ it becomes:
$$\frac{df(\sin^2x)}{d(\sin^2x)} = \cos^2x + \tan^2x$$
$$f'(\sin^2x)=\cos^2x + \tan^2x$$
Which is what we started with.
Now I am beginning to doubt my self, because I am not sure if the $dx$ changes to $d(\sin^2x)$ when $x\mapsto\sin^2x$. I would appreciate it if someone could help me realize which one it is, because then I can't begin to even grasp the main question without formulating the correct statement.
Thank you for @JJacquelin clearing this up.

Remarks:
On another note, I looked back to some of my differential work from university and came across a homework problem we proved which shows that:
$$\frac{d^2x}{dy^2}= -\frac{\frac{d^2y}{dx^2}}{\left(\frac{dy}{dx}\right)^3}$$
Which leads me to wonder, can the terms $\left(\frac{du}{dx}\right)^2$ be transformed into some $n$-th derivative? More generally, are there possible transformations to convert powers of derivatives to $n$-th derivatives in order to simplify this problem into a differential equation which can be solved using usual methods?
Also something @JJacquelin helped me realize about use of the chain rule, which may be of some help to progress:
$$\frac{df(u^2)}{d(u^2)}= \frac{df(u^2)}{2udu}$$
which may simplify the original question to:
$$\frac{df(u^2)}{du} = 2u\left[\left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2}\right] \quad ,u = u(x) \qquad (**)$$
These two identities might be of some help...possibly.
Comment: Sorry to be clumsy, but it appears still ambiguous for me. For example does $f'(\sin^2(x)$ means $\frac{d}{d(\sin^2(x))}f(\sin^2(x))$ or $\frac{d}{dx}f(\sin^2(x))$ ? In the wording, they are still a lot of primes. Excuse me to repeat, in this kind of problem, I think that the symbol prime should be suppressed everywhere.
Comment: @JJacquelin I have changed the primes into Leibniz notation in every part of the question except for Part 1, because that was how the original question was presented to me on a work sheet, and in the second half of part 2 because I want to make clear my erroneous work that might have stemmed from using prime notation.

Everywhere else however, the prime notation has been removed and replaced with d/dx, not d/d(sin^2x) because if you take the solution $f(x)$ from part 1 and then take $df(x)/dx$ of that function and then substitute $\sin^2x\, \text{for}\, x$ you will get the original question.
Answer: The trouble comes from the ambiguity of the symbol prime, which doesn't specifies with respect to which variable the differentiation is done.
Does $\quad f'(u)=
\begin{cases}
\frac{d}{du} f(u)\qquad(1)\\
\text{or} \\
\frac{d}{dx}f(u) \qquad(2)
\end{cases}\quad $ ?
Does $\quad f'(\sin^2(x))=
\begin{cases}
\frac{d}{d(\sin^2(x))} f(\sin^2(x))\qquad(1)\\
\text{or} \\
\frac{d}{dx}f(\sin^2(x)) \qquad(2)
\end{cases}\quad $ ?
It seems that, in writing
$\quad f'(\sin^2x) = (1-\sin^2x) + \frac{\sin^2 x}{1-\sin^2 x}\quad$
you understand "prime" in sens of definition $(2)$ ,
while in writing
$\quad f'(u) = (1 - u) + \frac{u}{1-u}\quad$
you understand "prime" in sens of definition $(1)$.
Both are different and leads to different further calculus. So, it is difficult to give a definitive answer to your question.
When two different variables, sometimes $u$ sometimes $x$, are involves in differentiations, it should be better to avoid the symbol prime. I suggest to edit your question with no symbol prime but with symbols $d$, in order to make it not ambiguous.
Comment: Indeed, a valid point that I should've noticed earlier. I am currently going to bed right now, but I will edit it as soon as I am up and awake, and hope to further receive a definitive answer. Thank you for the apt response :)
Comment: I have added a main edit in reference to your post, I hope this clears up the confusion with the prime notation. Thank you for letting me know about this clarity issue.
Answer: Since the original question was presented on the form :
$$f'\left(\sin^2(x)\right)=\cos^2(x)+\tan^2(x)$$
probably the symbol prime means
$$f'(X)=\frac{df(X)}{dX}\text{ , for any dummy variable }X.$$
So, with $X=\sin^2(x)\quad\to\quad f'\left(\sin^2(x)\right)=\frac{df\left(\sin^2(x)\right)}{d\left(\sin^2(x)\right)}=\cos^2(x)+\tan^2(x)$
With this interpretation, your first part is correct because, with symbol $X=u=\sin^2(x)$ the equation to be solved is
$$f'(u)=\frac{df(u)}{du}=(1-u)+\frac{u}{1-u}$$
$$f(u)=\int \left((1-u)+\frac{u}{1-u}\right)du$$
leading to
$$f(u)=-\frac{1}{2}u^2-\ln|1-u|+C \quad\to\quad f(x)=-\frac{1}{2}x^2-\ln|1-x|+C$$
SECOND PART, with change of variable $v=\sin(x)$
I don't use the same symbol $u$ in order to avoid confusion.
The equation to be solved is
$$f'(v^2)=\frac{df(v^2)}{d(v^2)}=(1-v^2)+\frac{v^2}{1-v^2}$$
$\frac{df(v^2)}{d(v^2)}=\frac{df(v^2)}{2vdv}=(1-v^2)+\frac{v^2}{1-v^2}$
$\frac{df(v^2)}{dv}=2v\left((1-v^2)+\frac{v^2}{1-v^2}\right)$
$$df(v^2)=\left((2v-2v^3)+\frac{2v^3}{1-v^2}\right)dv$$
$$f(v^2)=\int\left(2v-2v^3+\frac{2v^3}{1-v^2}\right)dv$$
$$f(v^2)=-\frac{1}{2}v^4-\ln|1-v^2|+c$$
As expected, the result is the same as above :
$\quad f(x)=-\frac{1}{2}x^2-\ln|1-x|+c$
NOTE :
It is possible to solve the original equation without any change of variable :
$$f'\left(\sin^2(x)\right)=\frac{df\left(\sin^2(x)\right)}{d\left(\sin^2(x)\right)}=\cos^2(x)+\tan^2(x)$$
$$f'\left(\sin^2(x)\right)=\frac{df\left(\sin^2(x)\right)}{2\sin(x)\cos(x)dx}=\cos^2(x)+\tan^2(x)$$
$$df=2\sin(x)\cos(x)\left(\cos^2(x)+\tan^2(x)\right)dx$$
$$f=\int df=\int 2\sin(x)\cos(x)\left(\cos^2(x)+\tan^2(x) \right)dx$$
$\int 2\sin(x)\cos(x)\left(\cos^2(x)+\tan^2(x) \right)dx=-\frac{1}{2}\sin^4(x)-\ln(\cos^2(x))+c$
$$f\left(\sin^2(x)\right)=-\frac{1}{2}\sin^4(x)-\ln|\cos^2(x)|+c$$
$$f\left(x\right)=-\frac{1}{2}x^2-\ln|1-x^2|+c$$
As expected, this is the same result as above.
Comment: Thank you for the lengthy and detailed answer on showing me what the prime symbol actually means in this question, and even showing me how to do it without any transformations. However this is only helps me clarify part of the big question, which is solving the differential equation without knowing that the first term on the R.H.S can be replaced with $1-v^2$ but rather $\left(\frac{dv}{dx}\right)^2$. I will edit my post to reflect your clarity, and I really appreciate your efforts in helping with this so far.
Comment: So far, I cannot understand because you wrote : << This stems from the initial question assigned to me which was:

Find an $f(x)$ which satisfies:
$$ f'(\sin^2x) = \cos^2x + \tan^2x, \quad 0> Putting $v=\sin(x)$ into it gives $=1-v^2+\frac{v^2}{1-v^2}$ .
Comment: Yes putting $v=\sin(x)$ results to that. However, when I saw the question initially I didn't think of using $\sin^2x+\cos^2x=1$ identity. I saw that the derivative of $\sin(x)$ was $\cos(x)$ and $\tan(x)$ can also be represented by $\sin(x)$ and $\cos(x)$ so I replaced everything with sin and its derivative.

$$f'(\sin^2x) = \cos^2x + \tan^2x, \quad 0<x<1$$
$$f'(u^2)= \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2}$$

And that's what I'm looking to see if solvable. To find an $f(x)$ which satisfies this equation regardless of what u(x) is.
Comment: I only added part 1 about the initial question to show where this all started from, but now I'm not looking to solve it when $u(x) = \sin(x)$. Part 2 is what I really am looking for, while part 1 is the answer that the professor posted online and when I first looked at that question I didn't think of using that transformation to $1-v^2$ but rather that the derivative of $\sin(x)$ is $\cos(x)$ and going on from there. I hope this makes more sense, but if you want to talk in a chat room to really understand I am okay with that as well.
Comment: I added the detailed steps at the end of my previous answer.  Is it OK ? In fact, I am a bit lost about what you need exactly. If you have more questions, please, edit them in the comments, not in the above main question which is now very long and too confusing for me.
Comment: Okay forget everything that's in the main post now. Imagine there was a new post and I gave you the following equation:
$$ \frac{df(u^2)}{d(u^2)} = \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \quad ,u = u(x)\qquad(*)$$

Is it possible to find an $f(x)$ that satisfies this equation, regardless of what $u(x)$ is.

Forget everything from the main post about $\sin and \cos$, just look at the equation in this comment as if you never read the main post.
Comment: According to this wording of the problem, see the new answer.
Answer: In fact, the preceding posts are more comments than answers. The aim was to clarify the problem. After discussion, this was done. So, these preliminary posts could be deleted.
In fact, the question is :
Is it possible an $f$ which satisfies the below equation where $u=u(x)$;if so, how?
$$\boxed{ \frac{df(u^2)}{d(u^2)} = \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \quad ,u = u(x)}$$
From what was discussed before :
$$f(u^2)=\int \left( \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \right)2udu$$
One can take any differentiable function $u(x)$ and put it into the above integral. Doesn't mater if there is, or not, a closed form for the integral. This proves that the function $f$ exists (fonction defined by an integral).
Let $F(x)$ a given function. With $u(x)=F(x)$ the derivative $\frac{du}{dx}=F'(x)$ is known.
In the next expression
$$\left( \left(\frac{du}{dx}\right)^2 + \frac{u^2}{\left(\frac{du}{dx}\right)^2} \right)2udu=\left( \left(F'(x)\right))^2 + \frac{  \left(F(x)\right)^2}{ \left(F'(x)\right)^2 } \right)2F(x)F'(x)dx$$
all the functions are known.
Thus the function $I(x)$ defined by the next integral is known (even if no closed form can be expressed) :
$$I(x)=\int \left( \left(F'(x)\right))^2 + \frac{  \left(F(x)\right)^2}{ \left(F'(x)\right)^2 } \right)2F(x)F'(x)dx$$
$$f(u^2)=I(x)$$
The inverse function of $u=F(x)$ is $x=F^{-1}(u)$
$$f(u^2)=I\left(F^{-1}(u)\right)$$
With $X=u^2$
$$\boxed{ f(X)=I\left(F^{-1}(X^{1/2})\right)}$$
This shows how the function $f(X)$ can be obtained.
EXAMPLES :
This was already done in case of $F(x)=\sin(x) \quad\to\quad f(X)=-\frac{1}{2}X^2-\ln|1-X|$.
We will do it with another function, for example the very simple $F(x)=x$
$F'(x)=1 \quad\to\quad I(x)=\int \left( 1 + x^2 \right)2xdx = x^2+\frac{2}{3}x^3$
$x=F^{-1}{u}=u=X^{1/2} \quad\to\quad F(X)=I\left(F^{-1}(X^{1/2})\right)=(X^{1/2})^2+\frac{2}{3}(X^{1/2})^3$
$F(X)=X+\frac{2}{3}X^{3/2}$
OTHER EXAMPLE :
More complicated, with $F(x)=\ln(x)$
$F'(x)=\frac{1}{x}$
$I(x)=\int \left( \frac{1}{x^2} + \frac{  \left(\ln(x)\right)^2}{ \frac{1}{x^2} } \right)2\ln(x)\frac{1}{x}dx $
$I(x)= x^2\ln^3(x)-\frac{3}{2}x^2\ln^2(x)+\frac{3}{2}x^2\ln(x)-\frac{\ln(x)}{x^2}-\frac{3}{4}x^2-\frac{1}{x^2}$
$x=F^{-1}\left(u\right)=e^u=e^{X^{1/2}}$
$$f(X)=e^X X^{3/2}-\frac{3}{2}e^X X+\frac{3}{2}e^X X^{1/2}-\frac{X^{1/2}}{e^X}-\frac{3}{4}e^X-\frac{1}{e^X}$$
As already pointed out, a closed form for $f(X)$ cannot be derived in all cases. This requires that closed forms be known for the integral and for the inverse function.
Comment: Wow, that was a very interesting approach. This is the answer I was looking for, and it helped me understand a lot. Sorry for the confusion beforehand in clarifying the question, I appreciate all the work you put into helping me.


In Hybris what the difference between ant deploy and ant all

Question: Can anyone please tell the difference in hybris of ant deploy and ant all.
According to me ant deploy will deploy code without any re-compliation.
It is said it takes only changes in configurations.
I dont see this making any sense as property and .xml files dont need any ant all.
So does ant deploy build items.xml and beans files?
Answer: You call ant deploy when you use ant server. This command is used to update the configuration done for tomcat/tcserver in /hybris/config to /hybris/bin/platform. It restarts the server if it's already started.
ant all executes ant build and ant server. ant build is in charge of compilation of java code, hybris beans, and items.
Answer: No, ant deploy doesn't build the items.xml or beans.xml.
The ant task description says
"Deploys platform to bundled server. Will copy files of config/tomcat or config/tcServer to config folder of configured bundled server."

It allows you to use some hooks in your extension's buildcallbacks.xml if your extensions have specific deploy requirements.
But in my almost five years of hybris experience though I have never had the need to run the ant deploy task.
(In fact, until I read this question I wasn't even aware of it as a task).
The day-to-day task (and usually also in actual deployments for go-live etc) "ant all" is what is being used.
Hope that helps!
Answer: My  Experience  Has  Been  Slightly Different From Sebastian.
Ant   Deploy  Used  for  Faster  Deployment   Without   Re  -  Compiling  Platform.
Comment: So you keep deploying without making any changes ?


Why do I get a redirect loop (302) when using path-auto patterns on taxonomy terms?

Question: I am using pathauto to set up my URL's at the moment and they are working fine for nodes.
However when I set up a rule for taxonomy terms using [term:name] as the pattern I end up with redirect loops.
Can anyone explain why this is and how to remedy.
Thanks for any help.
Comment: Any altrations in `.htaccess` ? pls post pathauto screenshot ?
Comment: It seems to have been due to caching and the rebuilding of aliases table. Once I left it for a while and cleared caches it was working as expected.
Answer: It seems to have been due to caching and the rebuilding of the alias table.
Once I left it for a while and cleared caches it was working as expected.


Best way to integrate "customization" to the front page

Question: #First question
We are making a Drupal 7 theme for one of our customers.
The company wants to be able to edit the frontpage easily, without any coding skills.
Atm we got this:
Image 1

Image 2

But this is totally not user friendly.
We made a block, and put block content in it, for this one it was just html.
But what we want is more like a wordpress "customizer" or something?
Maybe this is something that is totally basic in Drupal 7 but we just missed it?
#Second Question
The people (Employes of our client) that will manage the website when launched. Will they have to work in de basic Drupal 7 dashboard, or is there a way to make your own dashboard in the style of your website?
Comment: You are asking two questions, please remove one of them from this question and create a new question with the second one.
Comment: Welcome to Drupal Answers! _The best way_ is rather subjective, and we cannot give an answer about that. Actually, it's not much clear what you are asking, since it's not clear what exactly you don't understand in achieving the task you described. Then, @sanzante is right when he says you are asking more questions in a single post. It could be that for you are two related questions, but for as are two independent questions.
Answer: As a possible starting point, you might use the Panels and Panelizer modules.
When you define a custom CTools page (using the Page Manager module included with the CTools module), you can define its layout and content.
Concerning your second question, it is also possible to edit the page without moving to the administrative interface, by using the IPE (In Place Editor) provided by Panels.
You should be able to find many tutorials about Panels, in drupal.org too: https://www.drupal.org/node/1857644
Answer: I think there is no out of the box solution like you want to have on drupal7.
the question is what they want to customize.
Only the text and Images?
Just Install an Html editor and use Panels.
If you have some fancy Sliders or things like this I would use "paragraphs" but its alot work cause you have to write the templates for every element.
Or you just keep it userunfriendy and hope they pay you for changing the content:)
Answer: I agree with kerim, it would help to have a scope of what the client needs to be able to customize.
Drupal does offer a lot of themes and modules that have "customizable UIs" built into them, but I personally feel they all still assume a great deal of skill and can put you in a position to offer endless support.
My suggestion is scope out what they want to customize, and build those items into custom content types.  Then just assign the regions to different blocks throughout the page, i.e. "header photos," "sidebar links," "main content" - then they can fill one form, and it will dynamically flesh out the site for them.
I used this approach for a major real estate company to build property templates - they completed one node form, and it snaps everything right into place.  It may not work exactly how you want, but it may set you in the right direction.
Personally I would advise you against exposing blocks to your client directly - they're difficult to parse off with permissions and assume a lot of base knowledge.

For your second question, there are plenty of ways to provide multiple themes throughout your site - you can even set your client's UI pages to a custom URL path, and then use something like https://www.drupal.org/project/page_theme to set a theme to only those pages, i.e. site.com/client/front-page, site.com/client/editor, etc
Best of luck!