Improve extremely slow 'drupal_process_form'

Question: I am having problem investigating why our profile edit page is damn slow.
Call Stack
#   Time    Memory  Function    Location
1   0.0003  276344  {main}( )   ...\index.php:0
2   1.6889  39666816    menu_execute_active_handler( )  ...\index.php:21
3   1.6899  39668968    call_user_func_array:{C:\wamp\www\hyperDrupal\includes\menu.inc:527} ( )    ...\menu.inc:527
4   1.6899  39669448    entity_ui_get_form( )   ...\menu.inc:527
5   1.6901  39696720    drupal_build_form( )    ...\entity.module:1282
6   1.7842  40984120    drupal_process_form( )  ...\form.inc:385
(7) 17.3247 58502824    form_execute_handlers( )    ...\form.inc:903
8   17.3247 58503344    profile2_form_submit_handler( ) ...\form.inc:1519
9   17.3265 58492400    profile2_save( )    ...\profile2.module:621
10  17.3265 58492528    Profile->save( )    ...\profile2.module:311
11  17.3265 58491736    Entity->save( ) ...\profile2.module:918
12  17.3265 58492096    EntityAPIController->save( )    ...\entity.inc:333
13  17.3737 58517352    EntityAPIController->invoke( )  ...\entity.controller.inc:462
(14)    23.5502 90243232    module_invoke_all( )    ...\entity.controller.inc:355
15  23.5511 90247648    call_user_func_array:{C:\wamp\www\hyperDrupal\includes\module.inc:951} ( )  ...\module.inc:951
16  23.5511 90248096    bdEvent_entity_update( )    ...\module.inc:951
17  23.5511 90248832    bdE\bdEvent->dispatch( )    ...\bdEvent.module:44
18  23.5567 90332840    bdE\Services\profileService->run( ) ...\bdEvent.php:38
19  23.5568 90333120    bdE\Services\profileService->entity_get_values( )   ...\profileService.php:24

I created a stack trace in a new module and times are huge. Check step 7 and 14 both are after long operations.
What could be taking so much between
6   1.7842  40984120    drupal_process_form( )  ...\form.inc:385
7   17.3247 58502824    form_execute_handlers( )    ...\form.inc:903

and between
13  17.3737 58517352    EntityAPIController->invoke( )  ...\entity.controller.inc:462
14  23.5502 90243232    module_invoke_all( )    ...\entity.controller.inc:355

Comment: This is difficult to answer without knowing what that form is doing to take so long.
Comment: `drupal_process_form` executes the validators and submit handlers, including custom ones. Please post the form code, and any relevant submit handlers/validators etc. Without that, your guess is as good as mine. Also XHProf is probably more useful than a stack dump.
Comment: it's a simpler form I inherited using around 7 fields, all selects eg: country, city.
Comment: Useful starting point for XHProf: [Profiling Drupal 7 performance with XHProf and Devel](http://www.pixelite.co.nz/article/profiling-drupal-7-performance-xhprof-and-devel/)
Answer: Based on the backtrace, you won't know why your site is slow.
You should run a proper PHP profiler and analyse the results which can identify the slowest components.
It doesn't have to be Drupal it-self, but could be lack of caching or misconfiguration.
Some suggestions:

disable modules which you don't need (e.g. Devel, core Database Logging module, etc.),
upgrade your PHP to the latest or higher branch,
recompile your PHP for 64-bit or higher architecture depending on your CPU,
use the faster storage device for your db files or whole LAMP environment (e.g. SSD or memory-based filesystem),
use PHP debugger or profiler to find out any performance bottleneck (e.g. XDebug Profiler, DTrace or NuSphere PhpED PHP Profiler),
run some time-consuming drush command under gprof profiling tool, so you can find some performance bottleneck as well.

If disabling slow modules or profiling won't help, see: How do you improve Drupal performance?


Lemma for Jacobi's formula proof

Question: In the wikipedia article on Jacobi's formula
$$\frac{d}{dt}\det A(t) = \textrm{tr}\left(\textrm{adj}(A(t))\frac{dA(t)}{dt}\right)$$ they offer a proof via the chain rule that i am trying to understand.
The lemma states that $\det'(I) = \textrm{tr}$, where $I$ is the identity matrix and $\det'$ is the differential of $\det$. This is where I'm confused. What is the differential of $\det$? I figured it meant the derivative of the determinant of a matrix with respect to the matrix's indices, but then how does it make sense to apply it to the specific matrix $I$?
Comment: $det$ may be regarded as a smooth map of manifolds, usually as $det: GL_n(\mathbb{R}) \rightarrow \mathbb{R}$.  Hence at a matrix $A$ it has a full derivative $d(det)_A$, a linear map on the associated tangent spaces.
Comment: @Randall I don't know anything about manifolds or tangent spaces
Answer: The differential of a function $\mathrm{d}f$ is its best linear approximation at some point $x_0$. In this case $\det$ is a function $\det:M_{n\times n} \rightarrow\mathbb{R}$ where $M_{n\times n}$ is the space of $n\times n$ square matrices. Therefore, a matrix is the equivalent of a point for real functions. The best linear approximation to $\det$ near the identity is given by:
$$ \det(\mathbf{I}+\mathbf{M})=\det(\mathbf{I})+\mathrm{d}(\det(\mathbf{I}))\mathbf{M}+R(\mathbf{I},\mathbf{M}),\qquad \lim_{\|\mathbf{M}\|\rightarrow 0} \frac{R(\mathbf{I},\mathbf{M})}{\|\mathbf{M}\|}= 0 \tag{1}\label{eq1}.$$
The analogous expression for $f: \mathbb{R} \rightarrow\mathbb{R}$ is
$$ f(x_0+\epsilon)=f(x_0)+\mathrm{d}(f(x_0))\epsilon+R(x_0,\epsilon),\qquad \lim_{|\epsilon|\rightarrow 0} \frac{R(x_0,\epsilon)}{|\epsilon|}= 0 \tag{2}\label{eq2}.$$
Regarding the original question, $\det'(\mathrm{I})=\mathrm{tr}$ is equivalent to the following:
$$ \frac{\mathrm{d}}{\mathrm{d}t} \left. \left [ \det(\mathbf{I}+t\mathbf{B})\right] \right|_{t=0} = \mathrm{tr}(\mathbf{B}) \tag{3} \label{eq3}$$
You can use induction to prove $\eqref{eq3}$. The first step is to prove it for $n=2$:
$$\frac{\mathrm{d}}{\mathrm{d}t}  \left . \begin{bmatrix}
1+tB_{11} & tB_{12}  \\
tB_{21} & 1+tB_{22}
\end{bmatrix} \right|_{t=0} = \frac{\mathrm{d}}{\mathrm{d}t} \left. \left[
(1+tB_{11})(1+tB_{22}) - t^2B_{12}B_{21}    \right] \right|_{t=0}=B_{11}+B_{22}.  \tag{4} \label{eq4}$$
Assuming that $\eqref{eq3}$ is valid for $n=k-1$, we will prove it for $n=k$.
$$ \frac{\mathrm{d}}{\mathrm{d}t} \left. \left [ \det(\mathbf{I}+t\mathbf{B})\right] \right|_{t=0} = \frac{\mathrm{d}}{\mathrm{d}t} \left. \left [
\sum_{i=1}^k (\delta_{i,k}+tB_{i,k})M_{i,k}(t)
\right] \right|_{t=0}  \tag{5} \label{eq5}$$
here $M_{i,k}(t)$ represent the $(i,k)$ minor of $\mathbf{I}+t\mathbf{B}$.
$$ \frac{\mathrm{d}}{\mathrm{d}t} \left. \left [
\sum_{i=1}^k (\delta_{i,k}+tB_{i,k})M_{i,k}(t)
\right] \right|_{t=0}=\sum_{i=1}^k\left. \left [
 B_{i,k}M_{i,k}(t)+(\delta_{i,k}+tB_{i,k})\frac{\mathrm{d}}{\mathrm{d}t} M_{i,k}(t)
\right] \right|_{t=0}  \tag{6} \label{eq6}$$
evaluating the first term in the RHS of $\eqref{eq6}$ at $t=0$ gives zero except when $i=k$ since each minor has lower order proportional to $t$. On the other hand, $M_{k,k}(0)=\det(\mathbf{I})=1$. Therefore,
$$ \sum_{i=1}^k\left. \left [
 B_{i,k}M_{i,k}(t)+(\delta_{i,k}+tB_{i,k})\frac{\mathrm{d}}{\mathrm{d}t} M_{i,k}(t)
\right] \right|_{t=0}=B_{k,k}+\frac{\mathrm{d}}{\mathrm{d}t} M_{k,k}(0)= \sum_{j=1}^k B_{j,j}\tag{7} \label{eq7}$$
In the last step we use the hypothesis $n=k-1$ since $M_{k,k}$ is  $\det(\mathbf{I^{(k-1)\times(k-1)}}+t\mathbf{B^{(k-1)\times(k-1)}})$.
Answer: I probably can't help with the original question, but I can offer a proof that could be perhaps more understandable.
First, let's expand the determinant. We know that $\det A = \sum_\sigma (-1)^{\sigma} A^{\sigma_1}_1 A^{\sigma_2}_2 \ldots A^{\sigma_n}_n$, with the sum going over all permutations $\sigma$ of natural numbers from 1 to $n$. The derivative is then obtained using the Leibniz rule:
$$ \frac{\mathrm d}{\mathrm d t} \det A = \sum_\sigma (-1)^{\sigma} \sum_k A^{\sigma_1}_1 A^{\sigma_2}_2 \ldots \frac{\mathrm d A^{\sigma_k}_k}{\mathrm d t} \ldots A^{\sigma_n}_n. $$
When you use the determinant definition again, you get a rule that is best expressed with this pseudo-formula
$$ \frac{\mathrm d}{\mathrm d t} \det A = \\ \sum_k \det(\text{$A$ with all elements in the $k$-th column replaced by their derivatives}). $$
Now when you have a look at the RHS: $\operatorname{adj} A \frac{\mathrm d A}{\mathrm d t}$. Let's now consider any matrix $R = \operatorname{adj} A \cdot B$. Using the Laplace's expansion rule, we will see that the element in the $i$-th row and $j$-th column of this matrix is exactly determinant of $A$ with $i$-th column replaced by the $j$-th column of $B$.
And the trace just takes the diagonal elements and adds them up, so you get a sum of determinant of $A$ with first column replaced by the first column of $\mathrm d A / \mathrm d t$, i.e. its derivatives, plus $\det A$ with 2nd column replaced with derivatives and so on. Exactly what we found on the left.
This line of reasoning is really clumsy to explain, but hopefully should be easy to follow.

Lemma for Jacobi's formula proof


Can a variable be statistically insignificant but its components be significant?

Question: There is a paper (here) which shows that the beta coefficient of a standardised variable (see equation 3 in the paper), which represents the correlation between the standardised variables, can be split into two additive components (see equation 4).
My question is, can the beta coefficient from the regression be statistically insignificant, but one or both of the components be significant?
Answer: Yes, in my experience it can but it is rare. I have had a variable that was comprised of three components. I have seen one and two of the components come out significant but the overall not have a significant beta coefficient.


How to define arbitrary partial order relation and prove its properties?

Question: I have a simple data type with all nullary constructors and wish to define a partial order for it, including a Relation.Binary.IsPartialOrder _≡_.
My use case: the type is the type of sorts in an abstract syntax tree (statement, expression, literal, item), and i want a constructor of the AST which effectively upcasts a term (item ≤ statement, expression ≤ statement, literal ≤ expression).
data Sort : Set where stmt expr item lit : Sort

So far i have this:
data _≤_ : Rel Sort lzero where
    refl : {a : Sort} → a ≤ a
    trans : {a b c : Sort} → a ≤ b → b ≤ c → a ≤ c
    expr≤stmt : expr ≤ stmt
    item≤stmt : item ≤ stmt
    lit≤expr : lit ≤ expr

I can define isPreorder but have no idea how to define antisym:
open import Agda.Primitive
open import Data.Empty using (⊥)
open import Data.Unit using (⊤)
open import Relation.Binary
open import Relation.Binary.PropositionalEquality using (_≡_)
import Relation.Binary.PropositionalEquality as PropEq

module Core.Sort where

data Sort : Set where
    stmt expr item lit : Sort

data _≤_ : Rel Sort lzero where
    refl : {a : Sort} → a ≤ a
    trans : {a b c : Sort} → a ≤ b → b ≤ c → a ≤ c
    lit≤expr  : lit  ≤ expr
    expr≤stmt : expr ≤ stmt
    item≤stmt : item ≤ stmt

≤-antisymmetric : Antisymmetric _≡_ _≤_
≤-antisymmetric =
    λ { refl _ → PropEq.refl;
        _ refl → PropEq.refl;
        (trans refl x≤y) y≤x → ≤-antisymmetric x≤y y≤x;
        (trans x≤y refl) y≤x → ≤-antisymmetric x≤y y≤x;
        x≤y (trans refl y≤x) → ≤-antisymmetric x≤y y≤x;
        x≤y (trans y≤x refl) → ≤-antisymmetric x≤y y≤x;
        x≤z (trans z≤y (trans y≤w w≤x)) → _ }

I'm not sure what to do in the last clause (and all further clauses like it), and in any case this is cumbersome.
Am i missing a more convenient method to define an arbitrary partial order?
Comment: The last line of case 1 is an unreachable clause. Without that line, notice that `lit ≤ stmt` is not inhabited, which may explain why you couldn't define `IsPartialOrder` (it's not transitive). For case 2, I'm guessing you're not seeing how you can use recursion to prove `antisym`. Or maybe you're not making use of case-splitting on implicit arguments. It might help narrow the scope of your question if you presented a complete, small program, with all required imports, and with holes inserted at the places you feel you are stuck providing a proof.
Comment: Given that your type has only four values, you should be able to implement `antisym`, or anything else, really, in the worst case by just exhaustively enumerating all 16 cases.
Comment: m0davis: Ah yes, idea 1 was incomplete. I added the work i've done so far to the question.
Comment: Cactus: Yes, but i may add more values in future. I hope to define the partial order as the reflexive transitive closure of a directed acyclic graph, essentially.
Answer: Notice that, for any given x and y, whenever x ≤ y is provable, there are infinitely many such proofs. E.g., stmt ≤ stmt is proved by refl and by trans refl refl and so forth. This may (but probably doesn't) explain why it's troublesome (and maybe impossible) to prove ≤-antisymmetric.
In any case, the following definition of "less than or equal", _≼_, has the property that whenever x ≼ y is provable, there is exactly one proof of it. Bonus: I can prove antisym for it.
-- x ≺ y = x is contiguous to and less than y
data _≺_ : Rel Sort lzero where
    lit≺expr  : lit  ≺ expr
    expr≺stmt : expr ≺ stmt
    item≺stmt : item ≺ stmt

-- x ≼ y = x is less than or equal to y
data _≼_ : Rel Sort lzero where
    refl : {a : Sort} → a ≼ a
    trans : {a b c : Sort} → a ≺ b → b ≼ c → a ≼ c

≼-antisymmetric : Antisymmetric _≡_ _≼_
≼-antisymmetric refl _ = PropEq.refl
≼-antisymmetric _ refl = PropEq.refl
≼-antisymmetric (trans lit≺expr _)                   (trans lit≺expr _)     = PropEq.refl
≼-antisymmetric (trans lit≺expr refl)                (trans expr≺stmt (trans () _))
≼-antisymmetric (trans lit≺expr (trans expr≺stmt _)) (trans expr≺stmt (trans () _))
≼-antisymmetric (trans lit≺expr (trans expr≺stmt _)) (trans item≺stmt (trans () _))
≼-antisymmetric (trans expr≺stmt _)                  (trans expr≺stmt _) = PropEq.refl
≼-antisymmetric (trans expr≺stmt (trans () _))       (trans lit≺expr _)
≼-antisymmetric (trans expr≺stmt (trans () _))       (trans item≺stmt _)
≼-antisymmetric (trans item≺stmt (trans () _))       (trans lit≺expr _)
≼-antisymmetric (trans item≺stmt (trans () _))       (trans _ _)

Comment: Thanks! Since reading your answer i found [`Star`](https://agda.github.io/agda-stdlib/Data.Star.html) in the stdlib which i believe we can easily define `_≼_` in terms of: `_≼_ = Star _≺_`, tho antisymmetry must be proven nonetheless.


Git stash uncached: how to put away all unstaged changes?

Question: Suppose two set of changes are made in a project versioned by git. One set is staged and the other is not.
I would like to recheck staged changes by running my project at this state (before commit). What is a simple way to put away all unstaged changes and leave only staged? So I need unstaged changes to disappear from my project, but to be stored somewhere for further work.
This is sounds very much like git stash command. But git stash would put both unstaged and staged changes away from my project. And I can't find something like git stash uncached.
Comment: I have a dream: `git stash --unstaged` (a.k.a. `git stash -U`).
Comment: As of today with my git 2.21, there's still not a good answer to this. All answers below are either incorrect (`-k` option) or cumbersome to use.
Comment: @Maggyero feature request?
Comment: @Maggyero Maybe they will. Given that the opposite `--staged` option or shorter `git stash -S` was added since 2022. Till then, the accepted answer works well.
Answer: Update 2:
I'm not sure why people are complaining about this answer, it seems to be working perfectly with me, for the untracted files you can add the -u flag
The full command becomes git stash --keep-index -u
And here's a snippet from the git-stash help

If the --keep-index option is used, all changes already added to the
  index are left intact.
If the --include-untracked option is used, all untracked files are
  also stashed and then cleaned up with git clean, leaving the working
  directory in a very clean state. If the --all option is used instead
  then the ignored files are stashed and cleaned in addition to the
  untracked files.

And this is a gif of how it looks:

Update:

Even though this is the selected answer, a lot have pointed out that the [answer below](https://stackoverflow.com/a/34681302/292408) is the correct one, I recommend checking it out.
I tested my answer again today (31/1/2020) against git version 2.24.0, and I still believe that it's correct, I added a small note above about the untracked files.
If you think it's not working please also mention your git version.
Old answer:
If the --keep-index option is used, all changes already added to the index are left intact:
git stash --keep-index

From the documentation of git-stash:

Testing partial commits
You can use git stash save --keep-index when you want to make two or
  more commits out of the changes in the work tree, and you want to test
  each change before committing:
# ... hack hack hack ...
$ git add --patch foo            # add just first part to the index
$ git stash save --keep-index    # save all other changes to the stash
$ edit/build/test first part
$ git commit -m 'First part'     # commit fully tested change
$ git stash pop                  # prepare to work on all other changes
# ... repeat above five steps until one commit remains ...
$ edit/build/test remaining parts
$ git commit foo -m 'Remaining parts'

But, if you just want to visually check the staged changes only, you can try difftool:
git difftool --cached

Comment: Note that this answer will also stash the changes you have staged.
Comment: This answer works. The question was about staged vs. unstaged and does not mention anything about untracked files.
Comment: see also `git stash [-p|--patch]` which feels like interactive stashing.  From `man git stash` "With --patch, you can interactively select hunks from the diff between HEAD and the working tree to be stashed."
Comment: This answer *works fine* for me: the only things in the resulting stash are the unstaged changes. ♂
Comment: This answer isn't really useful as it will result in confusion. This answer is better https://stackoverflow.com/a/34681302/292408.
Comment: For me it puts both the staged and unstaged files in the stash.
Comment: Here's a demonstration of the problem with this answer:
https://gist.github.com/bobpaul/bded251b8637061e6a7094e6141ef38f
Comment: This answer confuses `untracked` with `unstaged`. Untracked files are files that are new and not in Git yet. Unstaged files are files that have been changed, but those changes haven't been selected for commit yet. So a new file can be both untracked and unstaged. But if you make changes to an existing file that is already in Git, that file will not be untracked, but it will be unstaged. What the OP wants is a way to stash only those changes that are not selected for commit (unstaged), which is completely separate from untracked.
Comment: I usually `add -p`, `checkout -p` and `reset -p`, never tried `stash -p`, thanks for the tip :D
Comment: @ElijahLynn I've linked to the other answer since i found a lot of people saying it's the better answer, thanks for your comment
Comment: "I'm not sure why people are complaining about this answer, it seems to be working perfectly with me, for the untracted files you can add the -u flag" ... The issue isn't the untracked files. The issue is that the changes on the index _are also included in the stash_. This is a problem if you want to check out a different commit and pop your stash there, but don't want the changes that were on the index.
Comment: This fails if a single file has both staged and unstaged changes. Your example has files that are either wholly staged or wholly not staged
Comment: I just contributed this 'all in one' answer that includes how to do 'only staged', 'only unstaged',  'unstaged and untracked', 'staged/unstaged but also keep staged in index', 'staged/unstaged/untracked but also keep staged in index'. Hopefully it helps! https://stackoverflow.com/a/68241237/1137085
Comment: This answer worked for me: `git version 2.16.6`
Comment: Hey @bobpaul, Thanks for your comment, I will check it out this weekend and get back to you
Comment: Not a good answer. If you pop the stash immediately, it can sometimes create merge conflicts. You'd expect it to apply on the staged changes, but it does something else because the staged changes are also stashed (just _not_ removed from working directory). It's impossible to resolve the conflicts while still keeping the unstaged changes unstaged, because you have to stage your resolution.
Comment: This does not work. It stashes staged changes as well (but also keeps them intact in the working directory after the stash). What we want is to only stash unstaged changes, and `--keep-index` is not doing that, which is very confusing.
Comment: This is a good answer.  It's even the top suggestion in "Creative Stashing" from [the git docs](https://git-scm.com/book/en/v2/Git-Tools-Stashing-and-Cleaning)
Answer: The accepted answer also stashes staged changes as a few have pointed out. Here's a way to do it without getting your staged changes in the stash.
The idea is to do a temporary commit of your staged changes, then stash the unstaged changes, then un-commit the temp commit:
# temp commit of your staged changes:
$ git commit --message "WIP"

# -u option so you also stash untracked files
$ git stash -u

# now un-commit your WIP commit:
$ git reset --soft HEAD^

At this point, you'll have a stash of your unstaged changes and will only have your staged changes present in your working copy.
Comment: This is really the correct answer IMO.  The `--keep-index` option in the current accepted answer still stashes what's in the index, it just *also* keeps it in the index.  So then it's duplicated, and hilarity ensues.
Comment: @KenWilliams hilarity tragedy
Comment: @PengheGeng you can run `git commit` with `--no-verify` to disable the commit hooks for this particular commit
Comment: This is the best answer so far, as the --keep-index option in the accepted answer is misleading. This should be the accepted answer.
Comment: The `git add .` step might want to be improved by `git add --all` as that should grab the files in a directory above the current working directory too.
Comment: -1 This approach doesn't work in all scenarios. It commits the staged changes but the commit might fail if you have a pre-commit hook that checks the current working code. If your staged changes are good but your unstaged changes are not, this approach will fail to commit staged changes.
Comment: I just contributed this 'all in one' answer that includes how to do 'only staged', 'only unstaged',  'unstaged and untracked', 'staged/unstaged but also keep staged in index', 'staged/unstaged/untracked but also keep staged in index'. Hopefully it helps! https://stackoverflow.com/a/68241237/1137085
Comment: Why didn't @kim123 come back and say, "Hey, I tried this and hilarity/tragedy ensued."? I checked at the official documentation and it seems like the index WON'T be restored redundantly. The command to force that to happen would be `git pop --index` which is described as "Tries to reinstate not only the working tree's changes, but also the index's ones." [https://git-scm.com/docs/git-stash#Documentation/git-stash.txt---index] Also the DAG of how `git stash` saves the tree shows that there is a direct delta from the HEAD to the working state. So it seems like the accepted answer works.
Comment: In powershell console `HEAD^` doesn't work for me, `HEAD~` does.
Comment: @cardiffspaceman What if you make changes to the index after the stash (in the same source locations), commit the changes, then try to pop the stash? Will there not be a conflict? --keep-index is confusing at the very best.
Answer: I found the marked answer did not work for me since I needed something which truly stashed only my unstaged changes.  The marked answer, git stash --keep-index, stashes both the staged and unstaged changes.  The --keep-index part merely leaves the index intact on the working copy as well.  That works for OP, but only because he asked a slightly different question than he actually wanted the answer for.
The only true way I've found to stash unstaged changes is to not use the stash at all:
git diff >unstaged.diff
git apply -R unstaged.diff

You can also use git restore . or git checkout -- . instead of git apply -R unstaged.diff.
Work work work...
git apply unstaged.diff
rm unstaged.diff

Comment: @Fritz: No temp commits are not possible in some scenarios. It might fail if you have a pre-commit hook that checks the current working code. If your staged changes are good but your unstaged changes are not, this approach will fail to commit staged changes.
Comment: This should be the accepted answer! It is the only one in multiple stackoverflow threads that  does what it claims and not relying on making temp commits!
Comment: @user643011: Temp commits are not a bad thing in git. They cost nothing and do not harm anyone.
Comment: This doesn't include untracked files.  You need to use "git ls-files" to find & include those in the diff patch
Comment: This definitely should be the accepted answer, better than temp commits when you are scripting a pre-commit hook.
Comment: **The only answer that really answers the question with the correct and sufficient solution. Thank you.**
Comment: Here on `git version 2.6.1.windows.1`, `git stash -k` worked as described.
Comment: I just contributed this 'all in one' answer that includes how to do 'only staged', 'only unstaged',  'unstaged and untracked', 'staged/unstaged but also keep staged in index', 'staged/unstaged/untracked but also keep staged in index'. Hopefully it helps! https://stackoverflow.com/a/68241237/1137085
Comment: Amazing, especially `git apply -R unstaged.diff`.
Comment: @PengheGeng Just skip the commit hook.
Answer: As of Git 2.35 (Q1 2022), you can now stash staged changes via git stash --staged.
Thus, to stash unstaged (and untracked) changes:
git stash --staged          # stashes staged, leaving unstaged
git stash -u                # stashes unstaged & untracked
git stash pop "stash@{1}"   # pops staged back (but they're now unstaged)
git add -A                  # restages the previously staged changes

HOWEVER, this would FAIL if the staged & unstaged changes affects the same lines, as git stash --staged would "partially" fail in this situation by stashing the staged changes, but leave behind both staged & unstaged.
In this case, the below method (using git commit) might be better.

If you wanna add an git alias for it, add this to your .gitconfig:
[alias]
    stash-unstaged = "!cd "${GIT_PREFIX:-.}"; f() { \
        git stash --quiet --staged;                 \
        git stash -u \"$@\";                        \
        git stash pop --quiet \"stash@{1}\";        \
        git add -A;                                 \
    }; f"

Usage:
git stash-unstaged -m "Stash Message"

For Git versions <2.35,
You can stash unstaged (and untracked) by exploiting git commit to separate the staged and unstaged changes:
git commit -m "TEMP"        # commits staged, leaving unstaged
git stash -u                # stashes unstaged & untracked
git reset --soft HEAD^      # undo commit, bringing staged back

Unlike the previous method (using git stash --staged), this will still work even when staged & unstaged changes affects the same lines.
 Tho, you MIGHT need to resolve merge conflicts when u pop the stash.

FYI, to stash staged changes, simply stash the unstaged first:
# Stash unstaged as shown previously
git commit -m "TEMP"
git stash -u
git reset --soft HEAD^

git stash -u                # stashes remaining staged
git stash pop "stash@{1}"   # pops unstaged back

Alias for these 2:
[alias]
    stash-unstaged = "!cd "${GIT_PREFIX:-.}"; f() { \
        git commit --quiet -m \"TEMP\";             \
        git stash -u \"$@\";                        \
        git reset --quiet --soft HEAD^              \
    }; f"

    stash-staged = "!cd "${GIT_PREFIX:-.}"; f() {   \
        git stash-unstaged --quiet;                 \
        git stash -u \"$@\";                        \
        git stash pop --quiet \"stash@{1}\";        \
    }; f"

Answer: Since the various answers here so far seem to have their own intricacies/limitations, I wanted to provide a few more alternatives that cover all of the specific edge cases that I personally have needed.
tl;dr
List staged (without deleted) files:
git diff --staged --diff-filter=d --name-only

List unstaged (without deleted) files:
git diff --diff-filter=d --name-only

List unstaged/untracked files:
git ls-files --modified --others --exclude-standard

Stashing only staged files (originally from this StackOverflow answer, but tweaked slightly):
git stash push --include-untracked -- $(git diff --staged --diff-filter=d --name-only)

Stashing only unstaged (not untracked) files:
git stash push --keep-index -- $(git diff --diff-filter=d --name-only)

Stashing unstaged and untracked files:
git stash push --keep-index --include-untracked -- $(git ls-files --modified --others --exclude-standard)

Stashing staged/unstaged files, while also keeping the staged files in your index as well:
git stash push --keep-index

Stashing staged/unstaged/untracked files, while also keeping the staged files in your index as well:
git stash push --include-untracked --keep-index

Full Explanation
git stash push allows us to provide a <pathspec>, and only stash the files that match it:
git stash push -- <pathspec>

push [-p|--patch] [-k|--[no-]keep-index] [-u|--include-untracked] [-a|--all] [-q|--quiet] [-m|--message <message>] [--pathspec-from-file=<file> [--pathspec-file-nul]] [--] [<pathspec>…]
Save your local modifications to a new stash entry and roll them back to HEAD (in the working tree and in the index). The  part is optional and gives the description along with the stashed state.

<pathspec>…
This option is only valid for push command.
The new stash entry records the modified states only for the files that match the pathspec. The index entries and working tree files are then rolled back to the state in HEAD only for these files, too, leaving files that do not match the pathspec intact.

-u, --include-untracked, --no-include-untracked
When used with the push and save commands, all untracked files are also stashed and then cleaned up with git clean.

git diff allows us to list the currently unstaged files with --name-only:
git diff --name-only

git diff [<options>] [--] [<path>…]
This form is to view the changes you made relative to the index (staging area for the next commit).

--name-only
Show only names of changed files.

--diff-filter=\[(A|C|D|M|R|T|U|X|B)…\[*\]\]
Select only files that are Added (A), Copied (C), Deleted (D), Modified (M), Renamed (R), have their type (i.e. regular file, symlink, submodule, …) changed (T), are Unmerged (U), are Unknown (X), or have had their pairing Broken (B). Any combination of the filter characters (including none) can be used.
Also, these upper-case letters can be downcased to exclude. E.g. --diff-filter=ad excludes added and deleted paths.

git ls-files allows us to list both the --modified files AND the untracked (--others) files:
git ls-files --modified --others --exclude-standard

git-ls-files - Show information about files in the index and the working tree

-m, --modified
Show modified files in the output

-o, --others
Show other (i.e. untracked) files in the output

--exclude-standard
Add the standard Git exclusions: .git/info/exclude, .gitignore in each directory, and the user's global exclusion file.
Comment: Perfect Answer. thanks @Glen
Comment: Some of the higher-voted answers are simpler but none are this comprehensive.  You deserve more upvotes!
Comment: Your solution does not seem to allow stashing unstaged *changes* (per file), only unstaged *files*, does it?
Answer: Git: Stash unstaged changes
This will stash all modifications that you did not git add:
git stash -k

Note that newly created (and non-added) files will remain in your working directory unless you also use the -u switch.
git stash -k -u

Also, your working directory must be clean (i.e. all changes need to be added) when you git stash pop later on.
http://makandracards.com/makandra/853-git-stash-unstaged-changes
Comment: This is equivalent to `git stash --keep-index`. Staged files are included in the stash.
Comment: This is answer is misleading, `git stash --keep-index` stashes all the changes from the HEAD state in the working tree, i.e. both staged and unstaged changes. The difference with bare `git stash` is that it also puts the working tree in *index* state (instead of in *HEAD* state), which creates a conflict when you `git stash pop` since the latter expect the working tree to be in the HEAD state.
Answer: Here is a way to make this very simple:

Add the alias definitions below to your .zshrc or .bashrc or .bash_profile
Now, anytime you are in this situation, just type gss and you will have 2 stashes - 1 with all the changes, another with only the staged changes

So, you can now apply the staged changes, see if those work, commit them if you want. Then, later you can also bring in the unstaged changes by applying the 'all WIP' stash and try those out.
alias gsts='git stash save'
alias gsv="git stash save --keep-index"

# How to Git Stash preserving staged/unstaged situation.
# 1. gsv will stash all and reset (remove) the unstaged, leaving only staged in staged state.
# 2. gsts will make a stash with your "good" (staged) files
alias gss='gsv all WIP && gsts staged WIP'


OSError: [Errno 98] Address already in use - Heroku

Question:
I'm trying to deploy a project https://github.com/kc1/flask-huey-example to heroku to allow usage of a task queue and flask. The project layout is as above. You can also take a look at the github repo which I have updated. The project is running without issue locally on windows (gunicorn is not used locally) .
My Procfile:
web: gunicorn run_app:app
worker: python /app/.heroku/python/bin/huey_consumer run_huey.huey

I had previously gotten an error Failed to find application object 'app' in 'run_app' . I tried to fix that by changing run_app.py to:
from app import create_app, config

if __name__ == '__main__':

    import logging
    logging.basicConfig()
    logging.getLogger().setLevel(logging.DEBUG)

    app = create_app()
    app.run(host='localhost', port=6060, debug=True)

else:

    app = create_app()
    app.run(debug=False)

Unfortunately , now when I push to heroku, I see:
2018-08-16T16:37:59.253114+00:00 app[web.1]: [2018-08-16 16:37:59 +0000] [8] [ERROR] Exception in worker process
2018-08-16T16:37:59.253118+00:00 app[web.1]: Traceback (most recent call last):
2018-08-16T16:37:59.253121+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
2018-08-16T16:37:59.253123+00:00 app[web.1]:     worker.init_process()
2018-08-16T16:37:59.253124+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
2018-08-16T16:37:59.253126+00:00 app[web.1]:     self.load_wsgi()
2018-08-16T16:37:59.253128+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
2018-08-16T16:37:59.253130+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2018-08-16T16:37:59.253132+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
2018-08-16T16:37:59.253133+00:00 app[web.1]:     self.callable = self.load()
2018-08-16T16:37:59.253135+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
2018-08-16T16:37:59.253137+00:00 app[web.1]:     return self.load_wsgiapp()
2018-08-16T16:37:59.253139+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
2018-08-16T16:37:59.253140+00:00 app[web.1]:     return util.import_app(self.app_uri)
2018-08-16T16:37:59.253142+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
2018-08-16T16:37:59.253144+00:00 app[web.1]:     __import__(module)
2018-08-16T16:37:59.253146+00:00 app[web.1]:   File "/app/run_app.py", line 21, in <module>
2018-08-16T16:37:59.253148+00:00 app[web.1]:     app.run(debug=False)
2018-08-16T16:37:59.253149+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/flask/app.py", line 841, in run
2018-08-16T16:37:59.253151+00:00 app[web.1]:     run_simple(host, port, self, **options)
2018-08-16T16:37:59.253153+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/werkzeug/serving.py", line 814, in run_simple
2018-08-16T16:37:59.253155+00:00 app[web.1]:     inner()
2018-08-16T16:37:59.253157+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/werkzeug/serving.py", line 774, in inner
2018-08-16T16:37:59.253159+00:00 app[web.1]:     fd=fd)
2018-08-16T16:37:59.253161+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/werkzeug/serving.py", line 666, in make_server
2018-08-16T16:37:59.253162+00:00 app[web.1]:     passthrough_errors, ssl_context, fd=fd)
2018-08-16T16:37:59.253164+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/site-packages/werkzeug/serving.py", line 577, in __init__
2018-08-16T16:37:59.253165+00:00 app[web.1]:     self.address_family), handler)
2018-08-16T16:37:59.253167+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/socketserver.py", line 453, in __init__
2018-08-16T16:37:59.253169+00:00 app[web.1]:     self.server_bind()
2018-08-16T16:37:59.253171+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/http/server.py", line 136, in server_bind
2018-08-16T16:37:59.253173+00:00 app[web.1]:     socketserver.TCPServer.server_bind(self)
2018-08-16T16:37:59.253181+00:00 app[web.1]:   File "/app/.heroku/python/lib/python3.6/socketserver.py", line 467, in server_bind
2018-08-16T16:37:59.253183+00:00 app[web.1]:     self.socket.bind(self.server_address)

How can I fix this?
Comment: You cannot have two applications listening on the same port at the same time. Perhaps you started the application twice, or some other application has taken the port.
Answer: The port you are trying to bind you application to is already being used by another app. Close the other app and re-deploy
Answer: I got the same error when trying to run a Flask application on Heroku and solved it by removing the
app.run

statement entirely from my app file (mine is named app.py), for example your run_app.py.
My Procfile:
web: gunicorn -b 0.0.0.0:$PORT app:app

When I want to run the app locally, I run in terminal (MacOS zsh)
FLASK_ENV=development flask run


How to create a $_SESSION variable and then carry it over to another page

Question: I want to create a session variable so I can use it on another page. (Like make the session variable on page1.php and then call it again on page2.php).
edit : What I meant by Form validation was making sure that the variables were set (like there was something in the input box) and then make them session variables so I could use them on other pages.
Here is my code for page1.php:
<?php

session_start();

?>
<!DOCTYPE html>

<?php

 if(!isset($_POST["submit"])) {

 //matri70boss suggested this code. Thanks matri70boss!
 if(!empty($_POST) && empty($_POST['username']){
  echo "error , username was not submited";
 } else {
  $_SESSION["username"] = $_POST["username"];
 }

 }

 ?>

 <title>NoterSharp</title>
 <link rel="icon" href="Images/icon.png">
 <link rel="stylesheet" type="text/css" href="style.css">

Welcome to NoterSharp!

 <form action="page2.php" method="post">

 <input type="username" id="username" name="username" placeholder="Username">
 <input type="submit" id="submit" name="submit" value="Submit">

 </form>

Here is my page2.php code:
<?php
session_start();

?>

<!DOCTYPE html>

<title>NoterSharp</title>
<link rel="icon" href="Images/icon.png">
<link rel="stylesheet" type="text/css" href="style.css">

Your username is: <?php echo $_SESSION["username"]; ?>

For some reason the username isn't being displayed on page2.php
Comment: fix this line $_SESSION["username"] == $_POST["username"]; to $_SESSION["username"] = $_POST["username"];
Comment: Your input fields don't have `name`s. And your form has `page2` as its action, so `$_POST` won't ever be filled in `page1`.
Comment: Can you clarify what you mean by "form validation" Are you wanting to check for blank fields?
Comment: What I meant by Form validation was making sure that the variables were set (like there was something in the input box) and then make them session variables so I could use them on other pages.
Comment: Thanks Nicolas Aoki, I'll fix that.
Comment: TiiJ7, I thought name and id were the same. I'll add name's to the inputs.
Comment: TiiJ7. About the form action, How can I make so that the $_POST is filled in page1 and is then carried over to page2?
Answer: change
     $_SESSION["username"] == $_POST["username"];
to
$_SESSION["username"] = $_POST["username"];

and maybe add something like
    if(!empty($_POST) && empty($_POST['username']){
      echo "error , username was not submited";
    }

to check if it the username was sent , which would echo "error , ..." in case the form was submited but username is empty (either empty string or not set).
Maybe set the  $_SESSION["username"] in the else such as
    if(!empty($_POST) && empty($_POST['username']){
      echo "error , username was not submited";
    } else {
      $_SESSION["username"] = $_POST["username"];
    }

Answer: for more information you can use the below links as reference. :)
https://code.tutsplus.com/tutorials/how-to-use-sessions-and-session-variables-in-php--cms-31839
https://www.w3schools.com/pHp/php_sessions.asp
http://php.net/manual/en/reserved.variables.session.php


When should I *not* use R's nlm function for MLE?

Question: I've run across a couple guides suggesting that I use R's nlm for maximum likelihood estimation.  But none of them (including R's documentation) gives much theoretical guidance for when to use or not use the function.
As far as I can tell, nlm is just doing gradient descent along the lines of Newton's method.  Are there principles for when it's reasonable to use this approach? What alternatives are available? Also, are there limits on the size of the arrays, etc. one can pass to nlm?
Answer: There are a number of general-purpose optimization routines in base R that I'm aware of: optim, nlminb, nlm and constrOptim (which handles linear inequality constraints, and calls optim under the hood). Here are some things that you might want to consider in choosing which one to use.

optim can use a number of different algorithms including conjugate gradient, Newton, quasi-Newton, Nelder-Mead and simulated annealing. The last two don't need gradient information and so can be useful if gradients aren't available or not feasible to calculate (but are likely to be slower and require more parameter fine-tuning, respectively). It also has an option to return the computed Hessian at the solution, which you would need if you want standard errors along with the solution itself.
nlminb uses a quasi-Newton algorithm that fills the same niche as the "L-BFGS-B" method in optim. In my experience it seems a bit more robust than optim in that it's more likely to return a solution in marginal cases where optim will fail to converge, although that's likely problem-dependent. It has the nice feature, if you provide an explicit gradient function, of doing a numerical check of its values at the solution. If these values don't match those obtained from numerical differencing, nlminb will give a warning; this helps to ensure you haven't made a mistake in specifying the gradient (easy to do with complicated likelihoods).
nlm only uses a Newton algorithm. This can be faster than other algorithms in the sense of needing fewer iterations to reach convergence, but has its own drawbacks. It's more sensitive to the shape of the likelihood, so if it's strongly non-quadratic, it may be slower or you may get convergence to a false solution. The Newton algorithm also uses the Hessian, and computing that can be slow enough in practice that it more than cancels out any theoretical speedup.
Answer: When to use and not to use any particular method of maximization depends to a great extent on the type of data you have. nlm will work just fine if the likelihood surface isn't particularly "rough" and is everywhere differentiable. nlminb provides a way to constrain parameter values to particular bounding boxes. optim, which is probably the most-used optimizer, provides a few different optimization routines; for example, BFGS, L-BFGS-B, and simulated annealing (via the SANN option), the latter of which might be handy if you have a difficult optimizing problem. There are also a number of optimizers available on CRAN. rgenoud, for instance, provides a genetic algorithm for optimization. DEoptim uses a different genetic optimization routine. Genetic algorithms can be slow to converge, but are usually guaranteed to converge (in time) even when there are discontinuities in the likelihood. I don't know about DEoptim, but rgenoud is set up to use snow for parallel processing, which helps somewhat.
So, a probably somewhat unsatisfactory answer is that you should use nlm or any other optimizer if it works for the data you have. If you have a well-behaved likelihood, any of the routines provided by optim or nlm will give you the same result. Some may be faster than others, which may or may not matter, depending on the size of the dataset, etc. As for the number of parameters these routines can handle, I don't know, though it's probably quite a few. Of course, the more parameters you have, the more likely you are to run into problems with convergence.


making graphs with xCode

Question: Hi I need to graph some arrays to analyse signals, I have a cocoa project going.  Can anyone tell me where I can find tools to simplify this task?
Basically i want to display my arrays like on a graphing calculator.
Comment: ..I'm looking for the mac equivalent of ZedGraph which is windows c# .net. I cannot use excel and matlab as i am debugging step by step. (In windows I would use messageBox as a sort of breakpoint and The program would redraw all the graphs on each messageBox call )...thx
Answer: Even though it's a young library, I'd suggest looking at the open source Core Plot framework.  It works on Mac and iPhone and can currently plot line and bar charts.  As I said, it's young and a lot of the core functionality is still being written, but it might work right now for your case.
Comment: Got it working.  This thing is awesome.
Comment: thx that seems to be what im looking for
Answer: Take a look at this post to the Apple mailing lists. http://lists.apple.com/archives/scitech/2007/Jun/msg00022.html
It mentions 3 potential graphing packages that you can try.  I have not had personal experience with them, so buyer beware, but it will at least give you a start on where to look.
Answer: XCode allows you to create data formatters to present object data in the debugger, but it doesn't provide any way to graphically represent the data using charts or graphs. (At least, not that I know of!) I think that's beyond the scope of XCode. You might consider dumping the data from your array to a file using a bit of code or the console, and then loading it into Excel or MATLAB. Then you can create a graph of your signal with no problem.
Just out of curiosity, are you familiar with another IDE that provides graphs of array data? I know tools like MATLAB and Mathematica do it pretty nicely, but I've never seen it in an IDE.
Comment: *Just of curiosity, are you familiar with another IDE that provides graphs of array data?* Jupyter Notebook for Python.

making graphs with xCode


Find groups that have the same values in another two columns

Question: I'd like to find the rows that group A and group B have the same values in the feature column and the value column.
My data:
  group feature value
1     A      f1     1
2     B      f1     1
3     A      f2     1
4     B      f2     0
5     A      f3     0
6     B      f3     1

df = data.frame(group = rep(c("A", "B"), 3),
                feature = c("f1", "f1", "f2", "f2", "f3", "f3"),
                value = c(1,1,1,0,0,1))

Desired outputs:

Shared:

  group feature value
1     A      f1     1
2     B      f1     1

Group-specific:

  group feature value
1     A      f2     1
2     B      f2     0
3     A      f3     0
4     B      f3     1

I've got the results by doing it manually:

split the dataframe by the group column --> get two dataframe, one only has group A and another only has group B.
create a new column feature_value using paste(df$feature, df$value, sep = "_").
use intersect and setdiff to find the shared and group-specific feature_value.
filter the orginal df using the extracted feature_value.

This is laborious and I hope there is a better way.
Answer: Solution 1: Grouped Filter
Group by feature, then filter:
library(dplyr)

df %>%
  group_by(feature) %>%
  filter(n_distinct(value) == 1) %>%
  ungroup()

# A tibble: 2 × 3
  group feature value
  <chr> <chr>   <dbl>
1 A     f1          1
2 B     f1          1

Same code, but with != instead of ==:
# A tibble: 4 × 3
  feature group value
  <chr>   <chr> <dbl>
1 f2      A         1
2 f2      B         0
3 f3      A         0
4 f3      B         1

Solution 2: Pivoting
A bit more cumbersome, but may be more flexible e.g. if you want to make comparisons among >2 features: pivot wider, then filter, then pivot back to long:
library(dplyr)
library(tidyr)

df_wide <- df %>%
  pivot_wider(names_from = group)

shared <- df_wide %>%
  filter(A == B) %>%
  pivot_longer(A:B, names_to = "group")

group_specific <- df_wide %>%
  filter(A != B) %>%
  pivot_longer(A:B, names_to = "group")

Comment: Nice solutions! I think the best one is 2, because it has the flexibility to clearly accommodate more `feature`s (although I think it should also work with 1, I haven't thought about it enough).
Comment: I like solution 1 better because my real data have a lot more extra columns which I want to keep untouched. Pivoting the data frame seems messed up those columns.
Answer: Using data.table:
For the first desired output:
library(data.table)
setDT(df)

df[, .SD[.I[.N > 1]], by = .(feature, value)]

   feature value group
1:      f1     1     A
2:      f1     1     B

And, for the second desired output, we could do:
df[, .(group = df$group[.I[.N == 1]]), by = .(feature, value)]

   feature value group
1:      f2     1     A
2:      f2     0     B
3:      f3     0     A
4:      f3     1     B

For more info about .SD, .I, and .N see here: ?special-symbols.


How is it possible to have a significant correlation between two variables but a low covariance?

Question: How is it possible to have a significant correlation between two variables but a low covariance?  What does this mean?
Comment: The covariance dependes on the scale of x and y... multiple both by a 10*10^10 and you get a large covariance... the correlation remains unchanged... Covaince is poor choice for getting an idea about the relationship between x and y...
Comment: "significant" doesn't mean "meaningful", it means "statistically different from 0". so if the true covariance is 0.0001 (and both variables have SD 1), getting significance is simply a matter of getting a sample size big enough to detect that confidently.
Answer: Welcome to site, @Debbie! First of all, if this is a homework question, please revise it and add the "self-study" tag.
The answer is, covariance equals the product of correlation and the standard deviations, i.e.,
$$\mathrm{cov}(X,Y)=\mathrm{cor}(X,Y)\mathrm{sd}(X)\mathrm{sd}(Y).$$
As shown in the small simulation below, when the standard deviations are small (0.1 and .1*$\sqrt{2}$ in the example), even the correlation is high and significant (cor = 0.68, p-value = 4.4e-15), the covariance can be low (0.008 here).
<code>> set.seed(1)
>
> x = rnorm(100,sd=.1)
> y = x + rnorm(100,sd=.1)
>
> cor.test(x,y)

    Pearson's product-moment correlation

data:  x and y
t = 9.2729, df = 98, p-value = 4.441e-15
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.5627820 0.7758677
sample estimates:
      cor
0.6836311

> cov(x,y)
[1] 0.008059066
</code>
Answer: In short, anything can be significant if $n$ is large enough.
When estimating and significance testing the correlation coefficient $r$, we most often use the estimator $s$:
\begin{equation}
s=\frac{s_{xy}}{s_xs_y}
\end{equation}
Where $s_{xy}$ denotes the sample covariance and $s_x$ , $s_y$ the sample standard deviations. The standard error of this estimator is given by:
\begin{equation}
se_s=\sqrt{\frac{1-s^2}{n-2}}
\end{equation}
Significance testing is done by the estimator $t=\frac{s}{se_s}$ which is approximately t-distributed with $n-2$ degrees of freedom.
As $n$ becomes large, two things happen. Our estimator $t$ comes closer and closer to a  $N(0,1)$ distribution, and the standard error gets smaller. In a $N(0,1)$ distribution, we would reject the hypothesis $H_0: r = 0$ at the 5% level, claiming it significant, if $|t|>1.96$.
But then, for any given variance and covariance, and thus any given $s$, a large enough $n$ will make this happen.


Can I only traverse a DatabaseConnection object once?

Question: I have a database query result, and I want to go through the records twice. However, the second time I go through it, nothing happens. It seems you can only iterate through this object once?
My code looks like this:
$query = db_select( ... );
$result = $query->execute();
if ( $result->rowCount() > 0 ) {
  foreach ( $result as $record ) {
    ... processing happens here
  }
}

// now loop through again, to mark records as processed, upon success.
foreach ( $result as $record ) {
  ... additional processing here
}

And the additional processing doesn't happen.
As a work around, I added $records = $result->fetchAllKeyed(); before the initial processing, and then the result object had no records!
I found nothing in the documentation about resetting the object or anything.
Is this how the object works? Can I reset the iterator again, or must I fetch the keys initially to have an array that I can iterate over many times?
Answer: This is expected behaviour; Drupal wraps around PDO and this is how PDO query functions behave also. Have a look at Is it possible to rewind a PDO result for a more in-depth discussion.
The standard way to get around this would be to use either fetchAllKeyed() as you've mentioned, or fetchAll() instead. The latter will produce an array of object results that can be iterated over as many times as you want.
$query = db_select( ... );
$result = $query->execute();
if ( $result->rowCount() > 0 ) {
  $data = $result->fetchAll();

  foreach ($data as $item) {
    // do something with data
  }

  foreach ($data as $item) {
    // do something with data again
  }
}


How to calculate week number corona sdk?

Question: In my application I have to get current week number of this year. And display current the week date. After I change the week number date also update.
local current_Year = tonumber(os.date("%Y",os.time(t)))
datastring = current_Year.."-01-01T00:00:00"
start_datevalue = makeTimeStamp(datastring)
diffday = os.time()-start_datevalue
week_txt.text=math.floor((diffday/86400)/7)+1

This code returns the current week. But how can I get the first date on this week and when in change week number how retrieve first date of that week number.
Answer: Have you looked into using the modulo command which you could use to return the day of the week with.
It would be something like weekday_txt.text=(diffday/86400)%7.
You could also use  os.date and os.time. Here is some documentation  on those functions.


What type of statistical analysis should I use?

Question: I have collected data for 30 patients for a certain test. This test was administered at 6 different but consistent intervals, and each test for each patient has lots of variables. If I want to do an analysis that compares both the differences among the intervals AND between the patients at the intervals, what sort of statistical methods are recommended?
Answer: Sounds like a multilevel modeling approach would be useful for you. Check out this tutorial by Woltman et al. or this introduction by Snijders.
Multilevel models in a longitudinal framework allow you to describe differences within individuals (level 1) and between individuals (level 2).


Find the current and previous designation of a Employee in q/kdb+

Question: Find the current and previous designation employees by comparing current date with previous updated
flip dc:`eid`name`titleID`title`lastupdate!(("1234" ;"1234"; "1234" ;"9835" ;"9835"; "9835"; "0982");("Jhon";"Jhon";"Jhon";"Chris";"Chris";"Chris";"jack");("09561";"08123";"08123" ;"07812"; "05234"; "05234"; "08123");("SalesManager";"Salesexecutive";"Salesexecutive";"SeniorSalesExecutive";"Salesexecutive";"Salesexecutive";"salesmanager");2023.06.26 2023.06.25 2023.06.20 2023.06.26 2023.06.25 2022.06.20 2022.01.19)
EmployeeTAB

eid
name
titleID
title
lastupdate

"1234"
"Jhon"
"09561"
"Sales Manager"
2023.06.26

"1234 "
"Jhon"
"08123"
"Sales executive"
2023.06.25

"1234"
"Jhon"
"08123"
"Sales executive"
2022.06.20

"9835"
"Chris"
"07812"
"Senior Sales executive"
2023.06.26

"9835"
"Chris"
"05234"
"Sales executive"
2023.06.25

"9835"
"Chris"
"05234"
"Sales executive"
2022.06.20

"0982"
"Jack"
"08123"
"Sales Manager"
2022.01.19

Query select from EmployeeTAB where lastupdate<td,lastupdate=(max;lastupdate) fby eID; results the latest updated result
output

eID
name
titleID
title
lastupdate

"1234"
"Jhon"
"09561"
"Sales Manager"
2023.06.26

"1234"
"Jhon"
"08123"
"Sales executive"
2023.06.25

"9835"
"Chris"
"07812"
"Senior Sales executive"
2023.06.26

"9835"
"Chris"
"05234"
"Sales executive"
2023.06.25
Comment: When asking a question involving such sample tables please include a `q` statement to create them to help those answering. The question also needs more detail. Are you unhappy with the query included and it's output? or is it that the query is what you have attempted but did not give the correct output?
Answer: Remove employees with only 1 role using fby.
Sort remaining by id and latest date and group the table on eid.
Then just take the first 2 entries and ungroup
`eid xasc ungroup 2#''`eid xgroup`eid`lastupdate xdesc select from employeeTAB where 1<(count;i)fby eid

Answer: One way to achieve desired result is to

Get reference table of eids and two most recent lastupdates
Join reference table with other data by eid and lastupdate

Like below
(ungroup select -2#asc lastupdate by eid from EmployeeTAB)
  ij `eid`lastupdate xkey EmployeeTAB

Deleting employees, who only had one role makes query more complicated:
delete from distinct (
  (ungroup select -2#asc lastupdate by eid from EmployeeTAB)
    ij `eid`lastupdate xkey EmployeeTAB
) where 1=(count;eid) fby eid

Answer: Another approach (assumes data is sorted by lastupdate as per the sample):
q)select from EmployeeTAB where({(1<count x)&x&not 2<sums x};lastupdate<=.z.D)fby eid
eid    name    titleID title                  lastupdate
--------------------------------------------------------
"1234" "Jhon"  "09561" "SalesManager"         2023.06.26
"1234" "Jhon"  "08123" "Salesexecutive"       2023.06.25
"9835" "Chris" "07812" "SeniorSalesExecutive" 2023.06.26
"9835" "Chris" "05234" "Salesexecutive"       2023.06.25

q)select from EmployeeTAB where({(1<count x)&x&not 2<sums x};lastupdate<=2023.06.25)fby eid
eid    name    titleID title            lastupdate
--------------------------------------------------
"1234" "Jhon"  "08123" "Salesexecutive" 2023.06.25
"1234" "Jhon"  "08123" "Salesexecutive" 2023.06.20
"9835" "Chris" "05234" "Salesexecutive" 2023.06.25
"9835" "Chris" "05234" "Salesexecutive" 2022.06.20
q)

Answer: res:(select  from employeeTab where lastupdate<=tradingDate,lastupdate=(max;lastupdate) fby eid) lj 1! select eid,prevTitleID:titleID,prevTitle:title,preLastupdate:lastupdate from employeeTAB where lastupdate<=tradingDate,lastupdate=(max;lastupdate) fby eid
 select from res where titleID<>prevTitle, not prevtitle like ""


Self Learning Neural network

Question: Is it possible for a neural network to learn based on its own predictions? So what I really want to do is to have my neural network model to learn from its own predictions but I'm not sure what the implications would be. On my own intuition, I think that it might lead the model into a state where it doesn't work well with new input since what it knows too much only about itself. I would want any advice for the matter.
Also I thought of having the model learn from its own predictions that pass a certain uncertainty. But I also have a hard time determining uncertainty for a neural network model and Im using keras.
Comment: Is this unsupervised learning ? It seems that you are training the model to be more confident about predictions that it might not have been confident about. This will very likely lead to bad solutions.
Comment: No I have labelled data. So the model's predictions is based on my labelled data. after the model predicts new and unlabelled data, I would like to use them (the predictions that pass the certainty threshhold) as part of my training data for the model to improve the model.
Answer:
Is it possible for a neural network to learn based on its own predictions?

This is called a recurrent neural network (RNN).

On my own intuition, I think that it might lead the model into a state where it doesn't work well with new input since what it knows too much only about itself.

Not very clear what you are stating here, but it sounds like you might want to check out Hopfield Networks, a specific type of neural network which have a tendency to converge towards stable local minimum states, and considered to be closer to the way human memory works than standard feedforward neural networks.

Also I thought of having the model learn from its own predictions that pass a certain uncertainty.

Check Boltzmann Machines, a probabilistic version of Hopfield Networks.

But I also have a hard time determining uncertainty for a neural network model and Im using keras.

Again, not very clear what you are asking. But you can create RNN in Keras, and Boltzmann Machines with TensorFlow.
Comment: I don't see how this is recurrent, there are no input sequences. I also don't see how this is Boltzmann machines as there is no low-rank optimization...
Comment: I edited my question to add a workflow on what I meant by training with its own predictions.
Comment: @VincentPakson Yup, it's seems like you're looking for a variation on Hopfiled Nets or Boltzman Machines.
Comment: With that. I'd love to have more idea on how to use Hopfield or Bolzmann Machines in my use case
Comment: @Curious seq-to-seq or seq-to-vec are only one application of RNN. Any NNet where the outputs feed back into the network are RNN. The dynamics of the network described in the OP are similar to the way Hopfiled and Boltzmann converge to stable attractors/local energy minima, even if the way the OP formulates it is completes different.
Comment: He is basically training a feed-forward network while changing the target values each epoch. You can call it RNN if the backpropagation considers more than one forward pass on the same datapoint in one big step, which doesn't in this case. What he is suggesting is the pseudo-labeling approach (in section 3 in https://www.analyticsvidhya.com/blog/2017/09/pseudo-labelling-semi-supervised-learning-technique/ ) which is an old method for semi-supervised learning.
Answer: Idk if this is what you are looking for, but this looks a little bit like semisupervised learning, which I'd check out. You start with a few labeled observations, but then you can feed the most certain (unlabeled) predictions back in as if they were actuals in order to supplement the small amount of data you do have.  https://en.wikipedia.org/wiki/Semi-supervised_learning

Self Learning Neural network


Dropbox user authorization without invoking a web browser for login

Question: I'm relatively new to the Dropbox APIs, so I apologize if I'm asking something simple.
I'm developing an application that will synchronize its database via the Dropbox Datastore APIs (or Sync APIs) but I can't seem to find a way to authorize a user for the Dropbox service connection. The only authorization that I was able to find is the /oauth2/authorize API, that according to the documentation will display a web page that will let user log in.
I am not really interested in displaying any web pages in my app (since it is not web based, nor that it has a web browser.) I just want to have my own UI incorporate the user name and password fields with the note Please log in to your Dropbox account where a user can type in their credentials. After which I could pass them to the Dropbox API call.
Is there such a way to authorize a user account?
Answer: By design, the Dropbox API doesn't allow this. (We don't want third-party apps to ever handle user credentials.)
Note, though, that the user only has to do this authorization once. From then on, your app can continue to use the access token you got in the first place. The normal flow for a command-line app is to send the user to the browser to authorize and then copy/paste an access code into the console. The app then gets the access token and saves it for future use.
Comment: Thanks. But I'm sorry, can you explain again how this copy and paste mechanism should work? If I understand correctly, you say that my app should redirect a user to a web browser. (This in itself will most certainly guarantee that the user will have a hard time switching back to my app, but let's assume that he/she eventually will...) But then I should expect a user also to copy and paste something into my app? Man, do you know how many people will be able to do this? 10 out of a 100 of your general users. There must be some other way to do this...
Comment: You can try it out for yourself. The Python, Ruby, PHP, and Java SDKs all come with a command-line sample.


Bloqueio de caracteres com exceção campo e-mail

Question: Estou usando este codigo para bloqueio dos caracteres especiais, mas esta bloqueando também no campo do e-mail.
como resolvo? ou como permito @ . e _
$(function(){
 var regex = new RegExp('[^ 0-9a-zA-Zàèìòùáéíóúâêîôûãõ\b-]', 'g');
 // repare a flag "g" de global, para substituir todas as ocorrências
 $('input').bind('input', function(){
 $(this).val($(this).val().replace(regex, ''));
});
})

Comment: Você só precisa adicionar dentro dos colchetes:
Answer: Você só precisa adicioná-los dentro dos colchetes:
var regex = new RegExp('[^ 0-9a-zA-Zàèìòùáéíóúâêîôûãõ\b-@_]', 'g');

Answer: Não sei se entendi corretamente sua pergunta, mas em seu código você aplica a expressão a todos os inputs.
Uma alternativa é definir os inputs que deseja validar através de uma classe.
Obs.: o comando .bind() está obsoleto, utilize o .on()

As of jQuery 3.0, .bind() has been deprecated. It was superseded by
  the .on() method for attaching event handlers

Exemplo:

$(function(){
   var regex = new RegExp('[^ 0-9a-zA-Zàèìòùáéíóúâêîôûãõ\b-]', 'g');
   $(".validar").on('input', function(){
      $(this).val($(this).val().replace(regex, ''));
   });
});

 <input type="text" name="nome" id="nome" class="validar"    placeholder="Digite seu nome" />

 <input type="text" name="idade" id="idade" class="validar" placeholder="Digite sua idade" />

 <input type="text" name="email" id="email" placeholder="Digite seu e-mail"/>


What's the difference between session storage and container in Zend?

Question: I'm trying to learn how to use sessions with ZF2/Doctrine for a login form. The documentation lists both Container and Session Storage for accessing the SESSION variable. But which do I use? And how do I use the Session Storage?
Answer: As per i knowledge:
Zend\Session\Container as SessionContainer; Container extends of ArrayObject means to you can easily iterate through properties and read/write them, e.g.
use Zend\Session\Container as SessionContainer;
$this->session = new SessionContainer('post');
$this->session->post_id = 1;
var_dump($this->session->post_id);

Session Storage:
use Zend\Session\Storage\ArrayStorage;
use Zend\Session\SessionManager;

$populateStorage = array('foo' => 'bar');
$storage         = new ArrayStorage($populateStorage);
$manager         = new SessionManager();
$manager->setStorage($storage);
var_dump($manager->getStorage());


PAM problems with "success=N" action

Question: I want to run a script when an authentication failure occurs. But, with the following conf file, authenitcation always fails even when the proper credentials are supplied.
auth       [success=1 new_authtok_reqd=ok ignore=ignore default=bad] pam_unix.so
auth       optional     pam_exec.so  /usr/bin/log_failure
password   required     pam_unix.so  md5

Authentication works by changing "success=1" to "success=ok", though obviously the failure script will always run.
I tried looking through the source code of _pam_dispatch_aux and it doesn't seem to return the first rule's return when there is no rule to jump to. If I add another rule so there is something to jump to, everything works.
auth       [success=1 new_authtok_reqd=ok ignore=ignore default=bad]  pam_unix.so
auth       optional     pam_exec.so  /usr/bin/log_failure
auth       optional     pam_exec.so  /usr/bin/noop
password   required     pam_unix.so  md5

Am I doing something wrong in my conf file, trying something that shouldn't work, or is there a bug?
Answer: This is working as intended.
success=n does not contribute to the return of the module stack. The result is that a successful authentication against pam_unix.so returns failure (because nothing returns a success of ok or done), and a failed authentication returns failure (as expected, because pam_exec.so returning success in this context would be Very Bad).
Try the following instead:
auth       sufficient   pam_unix.so
auth       optional     pam_exec.so  /usr/bin/log_failure
password   required     pam_unix.so  md5

sufficient means "stop here if we were successful, otherwise keep going". This should have the desired effect.
If /usr/bin/log_failure is a script though, the security nut in me would be more inclined to try this...
auth       sufficient   pam_unix.so
auth       [default=ignore]     pam_exec.so  /usr/bin/log_failure
password   required     pam_unix.so  md5

This prevents a success returned by log_failure from allowing a login.
Comment: I've updated my answer to be slightly more security conscious.

PAM problems with "success=N" action


SSH key does not work from a remote shell

Question: I've come across an interesting problem and I literally don't know where to start looking to resolve it!
The VCS machine in our office is running gitweb and my public key is on this machine fine. I can access the repositories from my machine without any problems, using my key.
However, if I am on any kind of remote shell, key authentication does not work! It falls back to password authentication, but the git user is configured to not have a password, so this is no good!
I have confirmed the problem by SSHing into another machine in the office and from there back into my machine. Also, by physically using another machine to SSH into my machine and even by going through a VPN to access my machine from a different network. In all cases, authentication with the git server fails although they are identical commands to those I run locally.
My first reaction was that gitosis (which automatically handles the keys) was misconfigured. It does include a number of additional security options on the keys:
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty

However, the problem is confined to me - other machines seem to be able to authenticate without a problem whether the shell is remote or local, which suggests that it is my SSH that is misconfigured. I did change the default port away from 22 by editing /etc/ssh/sshd_config but everything else should be at the default values. It's possible someone else has edited the config, so any things to look out for would be appreciated!
My machine is running Ubuntu 9.04 (obviously, upgrading might be a solution). Really, I'm just after a starting point - something to google for to resolve the problem. If this is something simple that someone has come across before then that's fantastic but as I say, I'm really at a loss for now!
Thanks in advance!
Comment: 'ssh -vvv username@hostname' may be of use...
Comment: What do you mean by "if I am on any kind of remote shell"
Comment: When you SSH to your machine, do you use the same user ID and username as you would have when you're using the machine directly (same `uid` and same `$HOME`)?
Comment: I mean any shell that was opened from another physical machine. It's the same user and username, yes. I'm logging into my physical machine with my username (and password). From that state, I am failing to authenticate with the git server (via public key) even though a shell opened from the same physical machine (without need for login) authenticates as expected.
Comment: I'll try -vvv at work, tomorrow!
Answer: I'm unclear on your description of what keys and configuration files reside on what systems.  However, if you are accustomed to being able to use your private key from a box to which you have not copied it, then you are doing so through use of SSH agent forwarding.  Try removing 'no-agent-forwarding' from the option list that you quoted above and see if that resolves your problem.
Comment: Thanks for the answer, but I don't think this is the issue. I can use key authentication from a local shell (opened in the gui, but not when I've logged in externally). My key seems to be set up ok, given that it works fine from a locally opened shell.
Answer: I had multiple keys in my ~/.ssh folder. Clearing out all of them but the main id_dsa key and re-adding that key to gitosis resolved the problem.
It's still a mystery why I was getting such strange behaviour! It must have been processing the keys differently via SSH / on a local shell. If anyone can shed any light on it, I'd be interested (I have the -v and -vvv output for local shells and remote SSH shells if you need it).
I wanted to add this in case someone else comes across the problem! (although it's very hard to search for, because there are so many questions on how to set up ssh keys at all)
Thanks very much to everyone who helped out.


conda commands do not work under Mac OS X 10.5.3

Question: I am trying to solve a problem as seen at Missing or syntactical error for styles in matplotlib, and the suggestion was that I had a corrupted install of python library.
I tried a complete uninstall and reinstall of anaconda, and now none of the conda commands work.I have no idea how to diagnose this problem, and I fear I'm going to go to the lengths  I did in the above thread, destroying things.
Answer: This is apparently a known bug with Catalina on OS X. I had to add the conda binaries to my PATH variable and all was well.


DictProperty has no attribute "requests_recycle_view". Python

Question: I need to refresh recycle view when I recive data from stream. When I run below code I see

AttributeError: 'kivy.properties.DictProperty' object has no attribute
  'requests_recycle_view'

It comes from methodrecycle_view_list I think there is some issue with instances but can't solve it. When I comment self.ids.requests_recycle_view.data = self.requests_tab and run refresh_recycle_view from button it works - I mean it refresh the list on the screen but I need it to run automatically when data in requests_tab changes from stream.
from kivy.config import Config
Config.set('graphics', 'multisamples', '0')
from kivy.app import App
from kivy.lang.builder import Builder
from kivy.uix.screenmanager import ScreenManager, Screen
import pyrebase
from kivy.properties import  ListProperty

config = {
    "apiKey": "String",
    "authDomain": "String",
    "databaseURL": "String",
    "projectId": "String",
    "storageBucket": "String",
    "messagingSenderId": "String"
  };

firebase = pyrebase.initialize_app(config)
db = firebase.database()

Builder.load_file('kv/main.kv')

#-------------------------------------------------------------ScreenManager

class ScreenManagement(ScreenManager):
    pass

#--------------------------------------------------------------RequestScreen

class RequestScreen(Screen):

    requests_tab =  ListProperty()

    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def refresh_recycle_view(self):
        print('refresh',  self.requests_tab)
        self.ids.requests_recycle_view.data = self.requests_tab

    def recycle_view_list(self, listFromStream):
        print("recycle_view_list", listFromStream)

        self.requests_tab = listFromStream
        self.ids.requests_recycle_view.data = self.requests_tab

    @classmethod
    def setRequestTab(cls, tab):
        print("setRequestTab", tab)
        cls.recycle_view_list(RequestScreen ,tab)

# ------------------------------------------------------------stream_handler
def stream_handler(message):
    print(message["event"])  # put
    print(message["path"])  # /-K7yGTTEp7O549EzTYtI
    print(message["data"])  # {'title': 'Pyrebase', "body": "etc..."}

    pyreMessage = [{'text': message["data"]}]

    RequestScreen.setRequestTab(pyreMessage)

db.child("Requests").stream(stream_handler)

#---------------------------------------------------------------StreamApp
class StreamApp(App):
    def build(self):

        sm = ScreenManagement()
        sm.add_widget(RequestScreen(name="Bajot II"))
        return sm

if __name__ == '__main__':
    StreamApp().run()

main.kv
#main.kv
#:import FadeTransition kivy.uix.screenmanager.FadeTransition

<ScreenManagement>:
    transition: FadeTransition()
    RequestScreen:

<CustLabel@Label>
    halign: 'right'
    valign: 'middle'
    width: 80

<RequestScreen>
    name: "RequestScreen"

    requests_tab:   requests_recycle_view.data

    Button:
        size_hint_x: None
        width: 42
        text: "Done"
           on_press:   root.refresh_recycle_view()

    BoxLayout:
        #size_hint_y: None

        RecycleView:
            id: requests_recycle_view
            viewclass: 'CustLabel'
            data: root.requests_tab
            RecycleBoxLayout:
                spacing: 15
                default_size: 100, dp(25)
                default_size_hint: 1, None
                size_hint_y: None
                height: self.minimum_height
                orientation: 'vertical'

Answer: I have not been able to test this (I don't have your database), but I think it will work. I think your statement about instances is right. In the modified code below, I changed setRequestTab to an instance method:
from kivy.config import Config
Config.set('graphics', 'multisamples', '0')
from kivy.app import App
from kivy.lang.builder import Builder
from kivy.uix.screenmanager import ScreenManager, Screen
import pyrebase
from kivy.properties import  ListProperty

config = {
    "apiKey": "String",
    "authDomain": "String",
    "databaseURL": "String",
    "projectId": "String",
    "storageBucket": "String",
    "messagingSenderId": "String"
  };

firebase = pyrebase.initialize_app(config)
db = firebase.database()

Builder.load_file('kv/main.kv')

#-------------------------------------------------------------ScreenManager

class ScreenManagement(ScreenManager):
    pass

#--------------------------------------------------------------RequestScreen

class RequestScreen(Screen):

    requests_tab =  ListProperty()

    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def refresh_recycle_view(self):
        print('refresh',  self.requests_tab)
        print(self.ids.requests_recycle_view)
        self.ids.requests_recycle_view.data = self.requests_tab

    def recycle_view_list(self, listFromStream):
        print("recycle_view_list", listFromStream)

        self.requests_tab = listFromStream
        self.ids.requests_recycle_view.data = self.requests_tab

    def setRequestTab(self, tab):
        print("setRequestTab", tab)
        self.recycle_view_list(tab)

# ------------------------------------------------------------stream_handler
def stream_handler(message):
    print(message["event"])  # put
    print(message["path"])  # /-K7yGTTEp7O549EzTYtI
    print(message["data"])  # {'title': 'Pyrebase', "body": "etc..."}

    pyreMessage = [{'text': message["data"]}]

    App.get_running_app().requestScreen.setRequestTab(pyreMessage)

db.child("Requests").stream(stream_handler)

#---------------------------------------------------------------StreamApp
class StreamApp(App):
    def build(self):

        sm = ScreenManagement()
        self.requestScreen = RequestScreen(name="Bajot II")
        sm.add_widget(self.requestScreen)
        return sm

if __name__ == '__main__':
    StreamApp().run()

I have saved a reference to the RequestScreen in the StreamApp, and then used that reference in the stream_handler(). That allows me to call setRequestTab() as an instance method. Again, this code is not tested, so there may be some errors.
Comment: Error: `App.get_running_app().requestScreen.setRequestTab(pyreMessage)
 AttributeError: 'StreamApp' object has no attribute 'requestScreen'`. If you can help I will create database for tests.
Comment: Did you make the change in the `build()` method? (`self.requestScreen = RequestScreen(name="Bajot II")` and `sm.add_widget(self.requestScreen)`)
Comment: Yes I have. That's a bit odd for me now.
Comment: Another possibility is that your `stream_handler` is getting called before the `StreamApp` is setup. If that is the case, then perhaps moving the `db.child("Requests").stream(stream_handler)` into a method of the `StreamApp` and calling that method via a `Clock.schedule_once()` in the `build()` method might fix it. You could also make `stream_handler` another method of the `StreamApp` and eliminate the need for the `App.get_running_app()` call.
Comment: Ok I will try. In the meantime I have set `Clock.schedule_interval` for `refresh_recycle_view`. I can see that clock works but the view does not refresh. I need to get a bit more knowledge about live cycle and how it accually works rather than just searching for ready solutions. Now it is a magic. Thanks for your help. I will come back later.


Probability Notation Question, Distribution of a function of Random Variables

Question: What is the meaning of $P(Y\le y)$ in simplest terms? I am having trouble understanding random variables and when they are used. What is the difference of random variable and functions? I was having trouble understanding the following problem:

My understanding is:
In the solution it looks like they replace $Y$ with $X^3$, and then take a cube root. That value is the upper limit of integration, but why id $0$ the lower limit? The integrand is $f(x)$. So the integral of the pdf is the cdf. Where does the $0<y<1$ come from? Is that part just necessary to show that a probability must be between $0$ and $1$?
I feel lost with the big idea, can someone give me some pointers, and what I should be thinking when I look at the above mentioned notation and problem?
THANK YOU!
Comment: If you're studying probability and are learning about transformations *and* you don't know what a random variable is by now, you have a significant review to do.
Comment: Please be more constructive with your comment, it may help to keep this in mind:Your Answer
Thanks for contributing an answer to Mathematics Stack Exchange!

Please be sure to answer the question. Provide details and share your research!
Comment: There is a reason why I have a comment here rather than an answer. You are asking about a topic which is covered at the end of an undergraduate-level probability course and yet you do not understand what a random variable is, which is taught at the very beginning of such a course. I see further of evidence of this similarly, considering that you do not know how to read $P(Y \leq y)$.

It is like asking a question on solving a calculus problem without knowing algebra. Hence, I suggested that you review your concepts. I'm not going to proceed further with this discussion.
Comment: Imagine that salt crystals are cubical, and let $X$ be the side length of a randomly chosen crystal. Since $Y=X^3$, the random variable $Y$ represents the volume of a randomly chosen crystal. Then for any fixed $y$, $\Pr(Y\le y)$ is the probability that a randomly chosen crystal has volume $\le y$.
Answer:
My understanding is:
In the solution it looks like they replace $Y$ with $X^3$, and then take a cube root. That value is the upper limit of integration, but why id $0$ the lower limit?

$0$ is the lower limit of the support for $f_X(x)$.  For all $x<0$, $f_X(x)=0$.

The integrand is $f(x)$. So the integral of the pdf is the cdf. Where does the $0<y<1$ come from? Is that part just necessary to show that a probability must be between 0 and 1?

The support for $\,f_X(x)\,$ is $\;0<x<1\;$, so $\;0^3<x^3<1^3\;$ and since $\,y=x^3\,$ thus $\;0<y<1$ is the support for $F_Y(y)$.
Comment: is the support the domain? and how do you read $f_X(x)$ aloud?
Comment: @Guest The **support** of a random variable is the range of values where the probability density is not zero. Also $f_X(x)$ would be read aloud as "$f$ of capital $X$ at $x$".
Comment: Notation-wise, is using f(y) correct? My textbook switches to g.
Comment: @Guest, No, you should not use the same symbol for the density functions of two random variables with different distributions as they will not be the same function.  Either use a different symbol, such as $f, g$, or use subscripts to distinguish, such as $f_X, f_Y$.
Comment: So the solution of the given example is notationally incorrect?
Comment: @Guest Oh, yes.  It's definitely poor form.


MLE of variance is biased in a Gaussian distribution

Question: Referring to: How to understand that MLE of variance is biased in a Gaussian distribution
at some point during calculation the formula of the sum of the expected value becomes a single expected value:

The explanation given is:
With the last step following since due to $E[X^2_n]$
being equal across n due to coming from the same distribution.
I can't understand:

What does the expectation of a value mean, in this case what is even $E[X_n]$
Why should the expected value of an observation value be equal?
What's the proof for $Var[1/N *...] = (1/N)^2 Var[...]$

Thanks for the help and sorry for the bad format I don't know how to write formulas, this is my first post xD
Answer: Note that $x_n$'s are random variables, not specific numbers. Assuming that $x_n$ is gaussian distributed with density $p$, it's expectation is per definition
$$ E[x_n] = \int_\mathbb{R} x \cdot p(x) dx. $$
If we apply a function to $x_n$, it's expectation is
$$E[f(x_n)] = \int_\mathbb{R} f(x) \cdot p(x) dx, $$
in your case $f(x) = x^2$.
The calculation you reference, assumes that the $x_n$ all come from the same distribution, i.e. they have the same density $p$ and thus the same expectation.
For your last question see e.g. here for a more general statement and proof.
Comment: @Cristian It's not different for each observation because they're sampled from the same dist.
Comment: According to book though the Xn are the values of one observation, as stated:
"we obtain the maximum likelihood solution which is the sample mean, i.e., the mean of the observed values {xn}"

and

"We first note that the maximum likelihood solutions µML and σ2ML are functions of the data set values x1,...,xN. Consider the expectations of these quantities with respect to the data set value"
Comment: If this is the case the expected value shouldn't be different for each observation?
Answer: $X_n$ is a Gaussian RV for each $n$; since iid, they all have the same mean, variance, moments etc. i.e. simply we have $$E[X_1^2]=E[X_2^2]=\dots=E[X_n^2]$$
Therefore, $\frac{1}{N}\sum_{n=1}^N E[X_n^2]=\frac{1}{N}NE[X_1^2]=E[X_2^2]=E[X_n^2]$ for any $n$. This is why we can remove the summation.
The proof for scaling property of variance can be: $$\operatorname{var(cX)}=E[(cX)^2]-E[cX]^2=c^2(E[X^2]-E[X]^2)=c^2\operatorname{var}(X)$$
Comment: Wonderful I think I get it, so it's the same also for the variance for example, technically we could evaluate the variance of a single value (coming from a sample) but since we don't know the pdf we need multiple observation to attempt to determine it, right?
Comment: According to book though the Xn are the values of one observation, as stated:
"we obtain the maximum likelihood solution which is the sample mean, i.e., the mean of the observed values {xn}"

and

"We first note that the maximum likelihood solutions µML and σ2ML are functions of the data set values x1,...,xN. Consider the expectations of these quantities with respect to the data set value". If this is the case the expected value shouldn't be different for each observation?
Comment: @Cristian This is the chicken & egg problem most novice practitioners are trapped in. $X_n$'s are observations sampled from a distribution. So, they're actually random variables o/w we wouldn't have sampled them and they wouldn't belong to a distribution. They're not fixed constants. Knowing (or pretending to know) and using them in our calculations doesn't mean that they're not RVs. In ML estimators, you pretend if you know $X_n$, use them in your estimator formulas, but when it comes to evaluating the bias of your estimator, we just take the expectation. How hypocrite of us :)
Comment: Yeah I'm quite new to random processes and this is really confusing for me, so you say that even tough the values come from an observation we consider them as variables? How would the expectation formula change? would we have a constant in the integration or what?
Comment: In any expected value calculation, you don't deal with $x_n,X_n$ or etc. When you know the PDF, e.g. $x_n\sim f(x)$, the expected value can be written with any variable: $$E[x_n]=\int \tau f(\tau)d\tau$$
Comment: Ok, so basically you don't care at all about the value, you say this is a value given by a certain pdf, so it's expected value is given by the formula which is the same for all values since they have the same PDF, that feels so wrong though! Is that the case in general? As far as I understood the expected value referred to a distribution, so I was wrong on that since it's about a variable that comes from that distribution? Also when we multiply Xn do we multiply the values or the distributions?
Comment: While calculating statistics from your data sample, you use the values numerically, but the expected value is like asking if regardless of the current value, what would be the average value  I'd get if I draw a sample. Consider a population, and we sample 10 people and measure their heights: name them as $X_1,..,X_{10}$ belonging to John, Jack etc. But, $X_1$ is not always John, it is the first person you choose from the population. So, it's going to change if you sample tomorrow, which means $X_1$ is a RV, but John is not. So, in expected value, you care about $X_1$, not John.
Comment: yes, variance is also an expected value: $$var(X)=E[(X-E[X])^2]$$ a note: variance of a known value is $0$ since it doesn't vary, e.g. $var(3)=0$. But, variance of a RV is calculated through the distribution. Same idea applies.


Regarding boosting, bagging and bootstrapping

Question: How to understand the relationships, comparative advantages, and comparative disadvantages of boosting, bootstrapping and bagging in terms of their respective applications in data mining.
Comment: Can you be more specific? It is a topic for a book.
Answer: Both boosting and bagging are ensemble techniques -- instead of learning a single classifier, several are trained and their predictions combined. While bagging uses an ensemble of independently trained classifiers, boosting is an iterative process that attempts to mitigate prediction errors of earlier models by predicting them with later models.
Bootstrapping doesn't really fit into this context (unless you mean something different by that term) and is a way of estimating the accuracy of a learned classifier on unseen data, similar to standard train/test data set splits or cross validation.
Comment: Bagging involves bootstrapping.


Given a sheaf $\mathcal{N}$ of $\mathcal{O}_Y$-modules defined by a finitely generated $\mathbb{C}[x]$-module, what is $f^*\mathcal{N}$?

Question: Say $f : \mathbb{A}^2 \rightarrow \mathbb{A}$ is projection on the first coordinate. Given a sheaf $\mathcal{N}$ of $\mathcal{O}_Y$-modules defined by a finitely generated $\mathbb{C}[x]$-module, what is $f^*\mathcal{N}$? Does it come from a $\mathbb{C}[x, y]$-module?
Comment: Let $A \to B$ be a ring morphism, and $f: X=Spec B \to Spec A=Y$ be the associated map of affine schemes. For an $A$-module $N,$ the pullback $f^{\star}(\mathcal{N}) \cong (N \otimes_A B)^{\sim}$ where $\mathcal{N}=(N)^{\sim}$ is the associated $\mathcal{O}_Y$-module. So in your case, $f^{\star}(\mathcal{N}) \cong (N \otimes_{\mathbb{C}[x]}\mathbb{C}[x,y])^{\sim}\cong \mathcal{N}\otimes_{\mathcal{O}_Y}\mathcal{O}_X.$ Finitely generated assumption is of course redundant.
Comment: Neglect the last statement!
Answer: I assume by $Y$ you mean $\mathbb A^1 = Spec(\mathbb C [x])$. Then the answer to your last question is yes, which is the content of Proposition 5.8, Chapter II in Hartshorne's book.
(just as Ehsan Kermani says in his comment)
Since you are dealing with affine schemes, you can identify $\mathcal N$ with its set of global sections $N=\Gamma(\mathbb A^1, \mathcal N)$. Then, again as Ehsan says, to compute the pullback, you just tensor with $k[x,y]$, where now $k[x,y]$ acts on $N$ as follows: $x$ acts as before, but $y \cdot n = 0$ for all $n$.


$C_b(X)$ is non-separable for $X$ non-compact

Question: If $X$ is a non-compact space then prove that $C_b(X)$ is not separable, where $C_b(X)$ is space of all bounded continuous functions on $X$.
I was trying like this, but got stuck at middle: Take a sequence $S$ consisting of distinct points without having a limit point. Now for all subset $A$ of that sequence we can get a continuous function $f_A$ such that $f_A\rvert_{A}=1$ and vanishes at $S-A$. So $A\neq B$ implies $\lvert f_A-f_B \rvert=1$ and thus we'll get an uncountable collection of functions $\{f_I\}$ such that $\lvert f_i-f_j \rvert=1 \;\forall i,j \in I$. Now I'm stuck.
Comment: Why don t you accept my anwer?
Answer: Now you have finished. Indeed you have found an uncountable number of disjoint balls. For example the balls of center $f$ and radius $1/2$. A dense subset should intersect all these disjoint balls, so there is at least one point in every ball. So the dense set must be uncountable.


Is this conditional sum function valid for linear programming?

Question: Linear programming requires a linear function to maximize in order to operate properly.
I am trying to figure out whether I could use linear programming to solve a problem. However, I would need a function that looks like the following.
Consider three non-negative integer variables $x1$, $x2$ and $x3$. The maximization function $f$ is defined as following:
<code>f(x1,x2,x3) = (x1+x2+x3) if (x1+x2+x3) < 10
            = 0 otherwise
</code>
Is there any theoretical objection to using such a maximization function in linear programming?
Comment: Is there any reason not to make the (x1+x2+x3) < 10 a constraint and keep the maximization function simply as f(x1,x2,x3) = (x1+x2+x3)? I otherwise see no problem, although more constraints on x1, x2, and x3 would be needed to make linear programming effective.
Comment: Ohhh, nice idea. Many thanks.
Answer: NK1406 provides a nice workaround:
<code> Make (x1+x2+x3) < 10 a constraint
 and keep the maximization function simply as f(x1,x2,x3) = (x1+x2+x3)
</code>


Coalgebras and Coideals: Why does $\ker(\pi \otimes \mathrm{id}_C ) = I \otimes C$ hold?

Question: In a proof on comodules and coideals I found the following passage:

Let $C$ be a coalgebra, and $I \subset C$ a vector subspace.
Let $\pi \colon C \rightarrow C/I $ be the canonical projection. Consider the linear map $f := (\pi \otimes \mathrm{id}_C) \circ \Delta \colon C \rightarrow C/I \otimes C$.
By the universal property of quotient vector spaces, there is a unique map $F \colon C/I \rightarrow C/I \otimes C$ with $F \circ \pi = f$ if and only if $I \subset \ker(f)$.
Now $I \subset \ker(f)$ is equivalent to $\Delta(I) \subset \ker(\pi \otimes \mathrm{id}_C) = I \otimes C$, i.e., $I$ is a right coideal.

Why does the last equality $\ker(\pi \otimes \mathrm{id}_C ) = I\otimes C$ hold?
Comment: Well, $\ker(\pi) = I$ and $\ker(\operatorname{id}_C) = {0}$, so the kernel of $\pi \otimes id_C$ is $I \otimes C + C \otimes \{0\} = I \otimes C$.
Comment: Thanks a lot, are you using https://math.stackexchange.com/a/541554/752637?
Comment: I kind of assumed it was true. I should be more careful. That the sum in my previous comment is a subset of the kernel of the tensor product map is clear. But the other inclusion is less clear in general. I think this is what they discuss in the answer you linked to.
Answer: For every two $$-linear maps
$$
  f_1 \colon V_1 \to W_1 \,,
  \quad
  f_2 \colon V_2 \to W_2
$$
between vector spaces, we have the equality
$$
  \ker(f_1 ⊗ f_2)
  =
  \ker(f_1) ⊗ V_2 + V_1 ⊗ \ker(f_2) \,.
$$
(If $f_1$ and $f_2$ are both surjective, i.e., quotient homomorphisms, then this is also true if we replace $$ by an arbitrary commutative ring.)
Proofs of this can be find in the question What is the kernel of the tensor product of two maps?.
It follows in the given situation that
$$
  \ker(π ⊗ \mathrm{id}_C)
  =
  \ker(π) ⊗ C + C ⊗ \ker(\mathrm{id}_C)
  =
  I ⊗ C + C ⊗ 0
  =
  I ⊗ C \,.
$$
Alternatively, we could consider the short exact sequence
$$
  0 \longrightarrow I \longrightarrow C \xrightarrow{\enspace π \enspace} C/I \longrightarrow 0 \,.
$$
Tensoring over $$ is exact because it is a field, whence the sequence
$$
  0
  \longrightarrow I ⊗ C
  \longrightarrow C ⊗ C
  \xrightarrow{\enspace π ⊗ \mathrm{id}_C \enspace} (C/I) ⊗ C
  \to 0
$$
is again exact.
This tells us that $I ⊗ C$ is the kernel of $π ⊗ \mathrm{id}_C$.