Table scan on internal tables

Question: Assume I have 2 tables - TABLE-1 & TABLE-2 and each of the table has 1 million rows with 10 columns and index on col1..
Now I build a internal table on this 2 tables ( 1 + 1 = 2 million) rows,
select * from
(select col1, col2,....col10 from table-1
union all
select col1, col2,....col10 from table-2) x

Questions,
how will the internal table will be treated in Oracle since its a internal table..
1. Will the internal table will be treated as a table with index on col1?
2. Will this be captured in the Explain plan?

Comment: "internal tables" is a bit misleading here. You are simply executing a query. No tables are created, nor are they internal.
Answer: Yes and yes.
Oracle will effectively treat this inline view as a table.  It can use predicate pushing to apply a filter on the inline view to the base tables, and potentially use an index.  The explain plan will show this.
Tables, indexes, sample data, and statistics
create table table1(col1 number, col2 number, col3 number, col4 number);
create table table2(col1 number, col2 number, col3 number, col4 number);

create index table1_idx on table1(col1);
create index table2_idx on table2(col1);

insert into table1 select level, level, level, level
from dual connect by level <= 100000;
insert into table2 select level, level, level, level
from dual connect by level <= 100000;

commit;

begin
    dbms_stats.gather_table_stats(user, 'TABLE1');
    dbms_stats.gather_table_stats(user, 'TABLE2');
end;
/

Explain plan showing predicate pushing and index access
explain plan for
select * from
(
    select col1, col2, col3, col4 from table1
    union all
    select col1, col2, col3, col4 from table2
)
where col1 = 1;

select * from table(dbms_xplan.display);

Plan hash value: 400235428

----------------------------------------------------------------------------------------------------
| Id  | Operation                             | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |            |     2 |    40 |     2   (0)| 00:00:01 |
|   1 |  VIEW                                 |            |     2 |    40 |     2   (0)| 00:00:01 |
|   2 |   UNION-ALL                           |            |       |       |            |          |
|   3 |    TABLE ACCESS BY INDEX ROWID BATCHED| TABLE1     |     1 |    20 |     2   (0)| 00:00:01 |
|*  4 |     INDEX RANGE SCAN                  | TABLE1_IDX |     1 |       |     1   (0)| 00:00:01 |
|   5 |    TABLE ACCESS BY INDEX ROWID BATCHED| TABLE2     |     1 |    20 |     2   (0)| 00:00:01 |
|*  6 |     INDEX RANGE SCAN                  | TABLE2_IDX |     1 |       |     1   (0)| 00:00:01 |
----------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   4 - access("COL1"=1)
   6 - access("COL1"=1)

Notice how the predicates happen before the VIEW, and both indexes are used.  By default everything should work as well as can be expected.
Notes
This type of query structure is called an inline view.  Although a physical table is not built, the phrase "internal tables" is a good way of thinking about how the query logically works.  Ideally, an inline view would work exactly like a pre-built table with the same data.  In reality there are some cases where things don't quit work that way.  But in general you are definitely on the right path - build a large query by assembling small inline views, and assume that Oracle will optimize it correctly.
Answer: for your particular query no any index will be used, but I suppose you do some filtering, ie where x.col1 = ###, I'm not sure that oracle will be able to use table-1/table-2 indexes to filter, so I suggest you to put where statements inside "union query"

Table scan on internal tables


Is the offset_column parameter in H20's random forest algorithm the same as the offset option in SAS Proc Logistic?

Question: I am trying to use H2O in R to run a random forest.  http://docs.h2o.ai/h2oclassic/Ruser/rtutorial.html
In the documentation, I saw that there is an option for an offset parameter but I cannot find much information about how it is leveraged.
In logistic regression, I have used an offset in two ways:
1.  To adjust for oversampling a binary event (http://support.sas.com/kb/22/601.html)
2.  To do a two stage model where the first stage logit is calculated and then I used the logit score as an offset in the second stage so in effect the residual is modeled in stage two.
I would like to replicate both of these through random forest, if possible, but did not think it was possible until I saw the offset parameter in the H2O implementation of Random Forest.  Does anyone know if the H2O offset parameter functions the same as the offset option in SAS proc logistic?
Answer: thanks for pointing this out! The offset parameter is actually not supported for H2O's distributed random forest. The parameters will be remove in a future release. A jira ticket for the issue can be found here:https://0xdata.atlassian.net/browse/PUBDEV-5191


Is Con(PA) true in standard model?

Question: It is well known that there is a fomula which means the consistency of PA in PA,that is Con(PA).
Then, Is Con(PA) true in standard model? or not?
Moreover,how to prove it?
I'm sorry. I edit this question as follows.
I can't understand well the relation of between [...is true in a relevant meta-theory] and [...is true in the standard model].
One requires a meta-theory to treat PA as object-theory.
Here, I adopt ZFC as meta-theory.
In ZFC, It is provable that PA has a model,and then Con(PA) is true.
On the other hand, it is said that Con(PA) is true in the standard model iff
Con(PA) is evaluated true at usual interpretation.
Above two notions slightly different (at least on definitions).
I can't understand well the relation of two notions - still after reading
Reese's answer.
Comment: What do you understand by "the standard model"? Since your metatheory is ZFC, it seems you mean the structure $(\omega,+,\times,S,<,0) $ defined within the universe of sets (or some variant of this, depending on what specific signature you use to state the axioms of PA). Here, $+,\times $ are ordinal addition and ordinal multiplication, 0 is $\emptyset $, $S (\alpha)=\alpha\cup\{\alpha \} $, and $a <b $ iff $a\in b $. If this is indeed the structure you mean, you can easily prove that the second-order formulation of PA holds for it arguing about ordinals in ZFC.
Comment: Possible duplicate of [Why bother with Mathematics, if Gödel's Incompleteness Theorem is true?](https://math.stackexchange.com/questions/54982/why-bother-with-mathematics-if-g%c3%b6dels-incompleteness-theorem-is-true)
Comment: I'm sorry about the lack of context. I've edited the question.  But, your linked question isn't close to mine (at least in my thought)
Comment: Once you have this, the rest is the usual sketch: The statement that PA is consistent holds, since you exhibited a model. Its formalization is a statement about numbers, and the argument in the given answer can be quickly formalized in ZFC.
Comment: Thank you for your comment! Hmm, so [Con(PA) is true in ZFC] and [Con(PA) is true in the standard model(also in ZFC)] are equivalent?
Comment: A posteriori.${} $
Comment: I've understand it at last! I'm sorry about very very basic misunderstanding and bothering you. My misunderstanding was as follows.
Comment: One of most basic factors of Godel's incompleteness theorem is, many properties of a logical system(here PA) are formalized in that system. Particularly, provability is formalizable, and in Sigma_1. So, Con(PA) is in Pi_1, actually "Sigma_n is true" is formalizable for all n. However, for all formulas P, "P is true" is not definable (i.e. Tarski's theorem) .
Comment: So, one can take a proposition Con(PA)(in PA) such that Con(PA) is evaluated true at the standard model(model theoretically defined notion) iff "PA is consistent"(not in PA! but in meta-theory, naively meta-theoretically defined notion) is true. Then, provided PA is consistent, "Con(PA) is true in the standard model" holds obviously!
Comment: I've not understand well such a very very basic notion. I'm so sorry!
Answer: Yes, provided $PA$ is consistent. $Con(PA)$ is just the statement "there is no proof of $1 = 0$ from $PA$". If it were not true, then there would be such a proof, and that proof would be coded by some standard natural number $n$. But then we could decode it back into the proof - here the standardness is important, because otherwise the "proof" might have nonstandard length and hence not be a real proof - and we then have a proof that $PA$ implies $1 = 0$. Since $PA$ proves $1 \neq 0$, this would make $PA$ inconsistent.
Comment: Thank you for your answering!
Comment: But I can't understand enough.  I want to derive [Con(PA) is true in the standard model], so in order to use proof by contradiction, I have to assume [Con(PA) is not true in the standard model] but not [Con(PA) is not true in a meta-theory] precisely? But in your answer, it seems that the latter is assumed...
Comment: @maimai If you don't assume that $Con(PA)$ holds in the metatheory, then there *is* no "standard model", because $PA$ is inconsistent and therefore has no models at all. My argument works like this: we suppose for contradiction that $Con(PA)$ fails in the standard model. We then deduce that $PA$ is actually inconsistent (so $Con(PA)$ fails in the metatheory), which we consider to be a contradiction because we're operating under the assumption that $PA$ is consistent (in the metatheory).
Comment: I'm so sorry about very basic misunderstanding!
I've understand it at last.
About it, please see comments of my question.
So now, I can understand "Provided PA is consistent, Con(PA) is evaluated true at the standard model" and your proof by contradiction.
Comment: But, different problems occur.
I can't understand your statements [the standardness is important, because otherwise the "proof" might have nonstandard length and hence not be a real proof ].
Comment: As long as I understand, any given proof,it would be coded by a natural number n,
but the natural number n code "the proof itself" but not "the length of the proof".
I'm sorry about bothering you still.
Comment: @maimai You're correct that $n$ codes the proof rather than the length, but the coding techniques we usually assume use a fixed library of (standard) numbers to represent the symbols. Since nonstandard numbers are too big to be constructed in a standard number of steps from standard pieces, any nonstandard number that codes a proof codes a proof that has nonstandard length (either because it includes a nonstandard number of steps or because one of the sentences it uses has a nonstandard number of symbols).
Comment: First, in general,it is possible that a proposition P is true in a model A,
but not in other model B.
So, it is possible that Con(PA) is true in the standard model,but not in a
nonstandard model.
But,this argument isn't interesting and doesn't require your explanation.
Comment: Indeed, your explanation contains more connotation.
and this just make your explanation interesting, intuitive and
attractive.
Probably, it can expressed as follows.
(In the following, I assumed a nonstandard model is fixed.)
Comment: Con(PA) is originally defined to have the property

(1) Con(PA) is true in the standard model iff PA is consistent(in usual sense).

But, consequently Con(PA) doesn't only have the property (1), but also have the property

(2) Con(PA) is true in the non-standard model iff PA is nonstandard-consistent
(i.e. standard natural numbers replaced nonstandard in usual sense ).
Comment: Is this thought right?
Did you mean such thing?
Comment: @maimai Not really. $Con(PA)$ isn't defined to have anything to do with standardness or nonstandardness - the statement "$Con(PA)$" literally is "there does not exist a number $n$ which codes a proof of $0 = 1$ from $PA$". Importantly, we *can't* say anything about standardness inside of a model - there is no way to formally express the notion of "standard" in the language of $PA$. Also, at no point in my answer did I move into a nonstandard model, so no, I haven't said anything about (2) (though it's true). My answer was proving that (1) holds, which is what you asked in your question.
Comment: But, Con(PA) is just "a string of symbols".  Although Con(PA) means nothing itself, we can say "Con(PA)" literally is "there does not exist a number n which codes a proof of 0=1 from PA" because we refer to some model.
Comment: Of course, there is no way to formally express the notion of "standard" in the language of PA, but PA is **object-theory** but not **meta-theory** here.  So, can we refer to "standard"?  Otherwise, how does Con(PA) -which is just a "string of symbols"- code "PA is actually consistent"?
Comment: @maimai $Con(PA)$ is a string of symbols in the metatheory, not in $PA$. In order to interpret it in $PA$ and ask whether it is true in a model, we must first replace it with its definition in $PA$, which is "there does not exist a number $n$ which codes a proof of $0=1$ from $PA$". Importantly, this is a sentence in the language of $PA$ - that's why it makes sense to ask whether it's provable from $PA$. $Con(PA)$ most certainly does *not* code "$PA$ is actually consistent", which is a meta-theoretical statement and cannot be expressed in the language of $PA$.
Comment: > __Con(PA)__ most certainly does _not_ code "__PA__ is consistent"  Probably not, rather __Con(PA)__ certainly _does_ code "__PA__ is consistent"  Of course, that is a meta-theoretical statement, but, that can be expressed in the language of __PA__.  That's just the meaning of "coding"!
Comment: Otherwise, ___Con(PA)___ is true in the standard model __doesn't__ follow from " ___PA___ is consistent".    that's why your answer is right, I think.
Comment: @maimai $Con(PA)$ codes "$PA$ is consistent", not "$PA$ is 'actually' consistent". $Con(PA)$ is interpreted *within whichever model we're in at the moment*, not meta-theoretically. It expresses the idea that a number with a certain property does not exist; if that number *did* exist, and happened to be standard, then it would represent a "real" proof of a contradiction.
Comment: Hmm, I'm not convinced.  But, I'm sure that your answer to my question is right. Moreover, your explanation is interesting. So, we'd better stop this too long argument.
Comment: I'm sorry about bothering you and, I appreciate your kindness very much!

Is Con(PA) true in standard model?


Infiniteness of the set of primes such $f$ have root $\bmod p$

Question: Let $f \in \mathbb{Z}[x]$ be non constant. How to prove that there exist infinitely many primes $p$ such $f$ have root in $\mathbb{Z}/p\mathbb Z$?
Answer: By contradiction using the same argument as Euclid' proof of infinitude of primes.
First you can assume that $f$ has no root in $\Bbb Z$ otherwise it's obvious that $f$ has a root in every $\Bbb Z_p$.
Take $g(x)=\dfrac{f(xf(0))}{f(0)}$ (this is a well defined polynomial over $\Bbb Z$), and we have also $g(0)=1$.
Now assume that $g$ has a root modulo only finitely many primes $p_1,\dots,p_k$. This means that for every integer $n$ the only primes which are allowed to divide $g(n)$ are only $p_1,\dots,p_k$.
Consider $m=\prod_{i=1}^kp_i$. Let $t\neq 0 $ be an integer. We have $g(tm)\equiv g(0)\equiv 1\bmod m$, so no prime from $p_1,\dots,p_k$ divides $g(tm)$, hence $g(tm)=1$ for all values of $t\neq 0$ and finally $g$ is a constant because it's a polynomial and here we have a contradiction as the polynomial $g$ is assumed to be non-constant.


Testing a binomial model

Question: I have a computational experiment that takes a size parameter $s$ and returns true or false.
It has been suggested that the result of the experiment is independent of $s$, but I don't think so.  I expect a general decrease in the numbers, though only logarithmically dependent on $s$.  (This makes it hard to detect and is the essential reason for my posting -- a larger effect would be easier to detect.)
For s = 5..30 I have generated 1000 samples for which I give the number of occurrences, respectively:
108, 108, 99, 99, 98, 87, 82, 85, 84, 94, 97, 87, 86, 102, 82, 84, 83, 87, 75, 69, 66, 79, 91, 81, 74, 74
Another wrinkle: the time needed to generate the samples depends on the size parameter.  Making 1000 tests at size 10 took less than a second; at size 30 it takes nearly a minute.  So I could generate many more examples at small sizes if that would be helpful.  ($s$ relates to $10^s$, so I can't pick too many more than $10^s$ for a given $s$.  But I could run a billion tests at s = 10 without trouble [in a week or two].)
The flip side, of course, is that it's difficult to generate large examples, even though those are the ones that would be most useful.  With a bit more effort (half a dozen processor-hours) I was able to do 1000 tests at higher levels:
s = 40: 75
s = 45: 81
s = 50: 74
s = 55: 73
s = 60: 74
s = 65: 68
So my question: How should I design an experiment to decide if these are identically distributed?  Once I have my data, how can I decide (for some $\alpha$) whether to reject or not reject the hypothesis that the distributions are identical, i.e., independent of $s$?
For example, I could continue to generate data for 1000 tests at each value of $s$ from 31 to 70.  But given the increasing time needed as the parameter increases I'm not sure that this is an appropriate way to sample the data.  But once I introduce non-uniformity I'm not sure what testing technique would be appropriate.  (I'm a statistical neophyte.)
Comment: You already have reasonably strong evidence of the logarithmic trend, even using only the data for s = 10,20,30,40,50 (which avoids worries about high leverage with s=40 or 50) and very strong evidence that there is *some* kind of trend.  In addition to that, what more are you looking for?
Comment: @whuber: Ideally, evidence whether the probability is decreasing inversely proportional to the logarithm.
Comment: The evidence for a log trend is not there in the 5..30 dataset (which shows a downward trend but suggests it's linear, not logarithmic) but with the addition of s=40,50 it is there.  The problems are that (1) the evidence is fairly sensitive to the two results at 40 and 50, implying you could use more data for higher s (but this sense of "sensitive" is *not* what experimental design formulas address!) and (2) it could help to confirm the logarithmic trend at intermediate values (in the range 30..50) and at more extreme values than 50, if they are of interest.
Comment: @whuber: OK, I put in some crunching time and generated more data at higher levels.  If more would be needed I could generate data for higher numbers (say, 1000 tests at s = 70, 75, 80) or increase the data for existing points from 1000 to 10,000 points with approximately equal effort.
Comment: @whuber: As I mentioned in a comment to one of the answers, there cannot be a linear trend unless it is constant, as it would cause the probability of the trait to go over 100% or under 0%.  A hyperbolic trend is _a priori_ possible, but linear is not.  The inverse log trend comes from theory rather than looking at the data; I might do some work to see if I can get more specific predictions from that model and compare to the data.
Comment: I make no claim about any trends; I only said that the data for s in 5..30 provide evidence for a linear trend *within that interval*.  It's foolish to extrapolate far outside the range of data you have, as you know: that's your point.  That's why I suggested you might be interested in values more extreme than 50, because you definitely would like to bracket the range of s you are interested in modeling.  (Incidentally, many experimental design formulas wind up telling you to take all your measurements at the two extremes and none in middle.)
Comment: I did add measurements up to 65 above; I hope to have measurements up to 80 tomorrow.  But what I actually want is information about how it performs around, say, 5000 -- a range that is entirely infeasible to test.  Of course statistics alone can't do this, I need domain-specific knowledge.  I have this, but I'd like to make sure my conclusions are reasonable based on the data I can test.
Comment: Can you perform even a *single* test for $s=5000$?
Comment: No.  At s = 150 I could perform maybe one test per day; at s = 300 I could not test anything.
Comment: Then there's really not much you can say beyond $s=300$ except completely on faith.
Comment: There's quite a bit I can say.  For example, I can say that the probability is greater than 0 for any positive s, even though I can't compute that far.  At present I can't prove the thing I want so I'm looking for statistical support, but it's just that -- support.
Answer: Sometimes, the data is sufficiently strong that you can get by with testing a much weaker assertion than what you are actually interested in. In this case, a very weak consequence of the null hypothesis (the probability is independent of $s$) would be that the cases $5 \leq s \leq 17$ and $18 \leq s \leq 30$ have the same probability of success. You can test this with a two-sample t test; acceptance of the null hypothesis wouldn't tell you very much (only that this weak consequence is acceptable), but rejection of the consequence implies rejection of the original, stronger, null hypothesis. I ran this test in two different ways - one by testing the $2 \times 13$ numbers of successes, and one by testing the $2 \times 13000$ zeroes and ones for the individual outcomes. In both cases, the $p$-value was pretty small (0.0016 and 0.00024, respectively), so for $\alpha > 0.00024$ I believe we can reject the hypothesis.
Of course there are many situations where this type of laziness doesn't work; then we need more advanced analysis to give us a definitive result. But sometimes a lazy approach is good enough.
(P.S. - I later realized that a cheap strengthening of the test would be to make it one-sided; that should improve the values even further with hardly any extra effort.)
Comment: Good -- glad to know I don't need a more powerful approach.  Do you think it's feasible to test the data for a logarithmic fit?  Something like probability ~ a/log s.
Comment: A standard quick approach is to divide the data into thirds along the s axis (with the intention of comparing the first and last thirds with a test of central tendency, usually a nonparametric one).  If you do this for the 5..30 data and produce three side-by-side boxplots, the trend will be clear and convincing.  The middle one, though, will *not* suggest a logarithmic tendency.
Answer: A good choice to analyze these data would be a generalized linear model for a binomial response and a logit link.  Conceptually it's simple: it says that each test has a chance $f(s)$ of succeeding, so that 1000 runs at a given value of $s$ should behave like 1000 flips of a coin having an $f(s)$ chance of success.  The chances are not expressed as probabilities, but as the logarithms of the odds: that's the logit.  Your supposition is that these log odds are a linear function of $\log(s)$.  The function $f$ is the combination of this logarithm and the logit.
However, you don't really need this level of sophistication because (a) you have run the same number of tests each time, (b) the number is large enough that the response counts are not small, and (c) the expected changes in response rate and its variance for this range of $s$ (from 5 through 70) are small.  Thus you will get essentially the same results if you do ordinary least squares regression of the logit of response/1000 against $\log(s)$.  This is simple enough even to do with a spreadsheet.

This plot shows the re-expressed data and their least squares fit.  It's a good fit; the residuals tend to be evenly scattered around it; and no points are extremely far to the right or the left.  There is a slight tendency for residuals to be positive for both small and large values of $\log(s)$, hinting at a quadratic term in $\log(s)$.  If that's a possibility, run more experiments for $s \le 10$ and $s \ge 50$.
The OLS table is
<code>           y |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         lns |   -.173364   .0238979    -7.25   0.000      -.22217   -.1245579
       _cons |  -1.874851   .0727821   -25.76   0.000    -2.023492    -1.72621
------------------------------------------------------------------------------
</code>
and, for comparison, the (more accurate) GLM table is
<code>    response |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
         lns |  -.1758287   .0305201    -5.76   0.000    -.2356469   -.1160104
       _cons |  -1.864569    .091082   -20.47   0.000    -2.043086   -1.686051
------------------------------------------------------------------------------
</code>
The closeness of the coefficients show that both techniques are fitting the same curve.  The logarithmic term is highly significant for both.  OLS is a little overconfident (it gets more extreme p values), but that doesn't matter here.
As a final check, we can back-transform the fit.  I have placed two bands at one standard deviation (as predicted by the fit at each point: the bands actually get narrower as $s$ gets larger).  If anything, the scatter is too tight: we would expect one-third or so of these 32 points, or 10 of them, to lie beyond the bands, but only 5 do.  (I wouldn't read much into that, but I would be worried if the opposite were true: if the majority of points fell outside these bands, I would worry about the accuracy of the binomial model.)

If you have the capability of running a GLM, then you can use it to analyze a dataset in which the numbers of tests per value of $s$ vary.  Because you already have about six times as much data as you need (to show the log trend is significant), I would recommend substantially reducing the sizes of future experiments, perhaps down to 100 or even less, rather than 1000 at a time.  You certainly can optimize your experimentation to get the most additional information for the extra work, but if you're almost done, it's less work overall just to continue with these rough heuristics.
Answer: This is a neat design of experiments exercise. You have a regression model, something like ${\rm Prob}[{\rm test}=1|s] = a + b \log s + \epsilon$, and you want to estimate the parameter $b$ subject to computational costs. This is akin to Neymann-Chuprow allocation in survey statistics. As a first approximation, you can utilize Neymann formula with $h$ being each individual value of $s$ and $C(s)$ being the time it takes to produce a single test. Basically, you'd want to take the number of tests that is inversely proportional to $\sqrt{C(s)}$. As a more accurate approximation, you can develop a model for the cost as a function of $s$, and plug it into the variance expression for the coefficient $b$ (the corresponding entry in the standard $\sigma^2 (X'X)^{-1}$ formula assuming the variances are constant... which is not exactly right, but for a small effect, is not a terrible approximation).
Update: per whuber's suggestion in the comments section: it might be worth checking the assumptions of your model. As long as we are in an agreement that this is an experimental design situation, the (rather small) subfield of the expertimental design literature that deals with deviations from the function form is called "robust design" literature. If you knew you had a linear trend, the best design would be to collect the data in the two extreme points ($s=5$ and $s=30$). However, to test for deviations from linearity, you would need to use a design that spreads the points across the whole domain (in the simple case of constant costs and constant variance, uniformly between the two extremes). For you, this basically means an extended analysis of the (existing or to be collected) data, in which you would run say a regression $\mbox{Prob}[\mbox{test}=1|s] = b_0 + b_1 s + b_2 s^2 + b_3 s^3 + \epsilon$ to see if you have higher order wiggles in your line (or add your favorite $\log s$ term there if you insist on the log dependence).
Comment: (The link associated with Neymann-Chuprow allocation makes no mention of it.)  This is a nice start.  It would be helpful to add a few remarks about the need to collect data to test the model specification: often, optimal designs are inadequate for this (crucial) purpose.
Comment: This is exactly what I was looking for.  Is there a good introduction to Neymann-Chuprow somewhere?  As I said, I have very little statistical background so simple is good.
Comment: @Charles, you can take a look at http://www.amazon.com/Sampling-Analysis-Advanced-Sharon-Lohr/dp/0495105279 or http://www.amazon.com/Sampling-Techniques-3rd-William-Cochran/dp/047116240X.
Comment: @whuber: I can link to a more specific subsection of the Wikipedia article. It is (inappropriately) called "Neyman's allocation" there. I will also add a couple of word about robust designs.
Comment: In response to the update: I absolutely guarantee that the trend is not linear (unless it is constant), as that would cause the probability that the tested quality is present to be negative for large s.  Needless to say, this does not happen.  As my primary interest is extrapolation to large s (which I cannot reasonably sample because of the time required) I cannot ignore this effect.
Comment: Do you have a substantive theory that states that the dependence must be logarithmic? E.g., if you were dealing with sort algorithms, there are good reasons to state that the model must have the form time$=a + b n + c n \log n + $random noise (and you'd expect that contributions of $a$ and $b$ would be small compared to $c$). Or is your logarithmic assumption just a convenient one?
Comment: @StasK: It's based on the Hardy-Ramanujan theorem and some handwaving. :)  It was a theoretical prediction made before I calculated the data.

Testing a binomial model


Adicionar espaços ao final do campo - python

Question: tenho um arquivo *.txt no qual existe uma coluna com o código do produto e outra coluna com o valor da quantidade conforme abaixo:
Cód.          qtd
7513020087041;5.0

879705341017;24.0

11713777;8.0

17565097;2.0

181420;20.0

181421;20.0

Porém o sistema para o qual exporto este arquivo não aceita pois o campo código precisa estar formatado com 13 caracteres, e se caso não possuir os 13 caracteres ele precisa ser preenchido com espaços em branco como no exemplo abaixo:
7513200870410;5.0

879075341017_;24.0

11713777_____;8.0

17565097_____;2.0

181420_______;20.0

181421_______;20.0

Obs: Lêia-se '_' = ' ' espaços.
Comment: Para essa operação talvez seja melhor usar [AWK](https://pt.wikipedia.org/wiki/AWK) do que Python. Ficara algo assim em AWK `awk -F';' '{ if ($1 != "") printf "%-13s;%s\n", $1, $2; else print }' NOME_ARQUIVO
`
Answer: Podes usar ljust:
print('7513200870410'.ljust(13)) # '7513200870410'
print('181420'.ljust(13)) # '181420       '

Ou format (python3.x):
print('{:<13}'.format(7513200870410)) # '7513200870410'
print('{:<13}'.format(181420)) # '181420       '

Exemplo com os teus dados para gravares posteriormente num ficheiro:
dados = '''
    7513020087041;5.0
    879705341017;24.0
    11713777;8.0
    17565097;2.0
    181420;20.0
    181421;20.0
'''

dados_new = ''
for d in dados.split():
    cod, qtd = d.split(';')
    dados_new += '{:<13};{}\n'.format(cod, qtd)

# guardar dados_new em um novo ficheiro:
""" OUTPUT
7513020087041;5.0
879705341017 ;24.0
11713777     ;8.0
17565097     ;2.0
181420       ;20.0
181421       ;20.0
"""

DEMONSTRAÇÃO
Se quiseres gravar em um ficheiro outra vez com tudo formatado podes, a seguir ao código do exemplo acima:
...
print('\n'.join('{};{}'.format(x,y) for x,y in dados_new), file=open('new_file.txt', 'w'))

# output:
"""
7513020087041;5.0
879705341017 ;24.0
11713777     ;8.0
17565097     ;2.0
181420       ;20.0
181421       ;20.0
"""

DEMONSTRAÇÃO
Comment: :-)  eu ia comentar do uso do format (e no Python 3.6, as fstrings também aceitam os parâmetros usados pelo format)
Comment: Graças a sua ajuda ;)
Comment: Posso fazer isso com um 'for' ou ' while'?  pois o arquivo possui muitas linhas, muita mesmo...
Comment: Claro @CelsoLopes, vou colocar um exemplo com o teu cod
Comment: @jsbueno, eu sabia que format também dava mas tive de testar porque já não me lembrava bem da sintaxe. Obgado pelo reparo
Comment: @Miguel, eu testei e funcionou, mas quando faço no arquivo não está funcionando, veja meu código e o erro:
Comment: with open('Produtos.txt', 'a') as arq:
    arq_new = []
   for d in arq.split():
        cod, qtd = d.split(';')
        arq_new.append(('{:<13}'.format(cod), qtd))
    print(arq_new)
Comment: @CelsoLopes: https://repl.it/repls/BlandRichPiranha
Comment: @Miguel, exatamente isso, muito obrigado cara.  Agora, quando jogo para um novo arquivo ele está criando um salto de linha '\n' me deixando com um linha em branco entre um dado e outro, como removo isso?
Comment: @Miguel, consegui, valeu!! só remover o '\n' da última linha onde voce me forneceu :  print('\n'.join('{};{}'.format(x,y) for..., então ficou assim :print(''.join('{};{}'.format(x,y) for...   Muito obrigado mesmo!!!
Comment: @CelsoLopes ainda bem. Boa sorte
Answer: Você pode inicialmente criar uma lista vazia para armazenar os produtos. Ao percorrer as linhas do arquivo, você cria um dicionário para armazenar os dados de cada produto e adiciona à lista. No final, você percorre a lista de produtos e imprime os dados na tela. O try - except serve para o programa não parar ao percorrer as linhas vazias. Tente deste jeito:
arquivo = open("<nome_do_arquivo>.txt")
produtos = []

for linha in arquivo.readlines():
    if 'Cód' not in linha:
        try:
            produto_list = linha.split(";")
            codigo = produto_list[0].ljust(13)
            valor = produto_list[1]
            produto = {"codigo": codigo, "valor": valor}
            produtos.append(produto)
        except:
            pass

for produto in produtos:
    codigo = produto["codigo"]
    valor = produto["valor"]
    print(f"{codigo};{valor}")

Python é minha linguagem favorita para este tipo de operação. Com apenas um arquivo de poucas linhas e um comando de terminal, a gente faz mostruosidades.
Comment: Tem muitos problemas com essa abordagem. A maneira como abriu o arquivo e não o fechou causa vazamento de memória. A leitura integral dos dados e seu espelhamento, a depender do tamanho do arquivo, pode consumir toda a memória de trabalho o que associado ao uso do tratamento inespecífico de exceções pode derrubar o Sistema Operacional ao tratar com `pass` com um `MemoryError`.  A reiteração e a forma de uso das estruturas de dados amentam muito a complexidade de tempo do algoritmo o que poderia ser resolvido em tempo linear está operando em tempo quadrático.
Comment: Sem ônus ou demérito, esse código foi gerado com BARD ou ChatGPT?
Comment: Não. Eu escrevi este programa na minha máquina e executei. Agora, quanto a fechar o arquivo depois das operações, de fato foi vacilo meu. Deveria ter fechado o programa com o comando "arquivo.close()". No meu computador não deu problema provavelmente porque o arquivo era pequeno. Por que você achou que eu recorri à Inteligência Artificial?
Comment: O que me levou a pensar que fosse IA foi a quantidade de erros no código para uma pessoa que disse que python era a linguagem preferida para a tarefa, sendo que python não é a melhor linguagem para a tarefa. Na verdade python é muito verborrágico para processamento de arquivos texto, existem linguagens muito mais eficientes para a tarefa por exemplo perl e awk que fazem essa atividade em uma linha.
Comment: Não sei programar nem em perl nem em awk. Sei poucas linguagens. Sou iniciante. Só sei mesmo Java, Python, JavaScript e PHP (Laravel) e um pouco de C e C++ e R.
Comment: Eu já precisei manipular arquivos grandes várias vezes que consegui em Python. Provavelmente, eu utilizei o comando para fechar o arquivo - aqui foi esquecimento meu. Quando quero fazer uma coisa simples e rápida que depende de programação, você acha que eu deva aprender a linguagem ideal do zero, sendo que a linguagem que eu domino já faz?
Comment: Eu estou tentando dizer que não acho que você domine a linguagem e que cometeu muitos erros no seu código de exemplo e que não achei lega você ter dito que *"...Python é minha linguagem favorita para este tipo de operação. Com apenas um arquivo de poucas linhas e um comando de terminal, a gente faz mostruosidades..."*. Veja se com um exemplo em Python você entende os problemas do seu código: https://replit.com/@Augusto-Vasques/TrivialRudeAddresses#main.py
Comment: Veja também esse comentário que anexei a pergunta https://pt.stackoverflow.com/questions/268558/adicionar-espa%c3%a7os-ao-final-do-campo-python#comment1020629_268558


Sufficient Statisitics and Discrete Distributions

Question: I am trying to master minimal/complete sufficient statistics, however I am having trouble when the distributions are discrete and involve indicator functions. Here is my 3 part question:
Let $X$ be a random variable from the following distribution:
$$f(x;\theta) = \left\{\begin{array}{ccc} \theta & , & x = -1 \\ 1 - 2\theta & , & x = 0 \\ \theta & , & x = 1\end{array}\right.$$
where $0\leq \theta \leq\frac{1}{2}$.

Find a minimal sufficient statistic for parameter $\theta$.
Answer: Define the indicator functions:
$$I_1(x) = \left\{\begin{array}{ccc} 1 & , & x = -1 \\ 0 & , & {\rm otherwise}\end{array}\right.$$
$$I_2(x) = \left\{\begin{array}{ccc} 1 & , & x = 0 \\ 0 & , & {\rm otherwise}\end{array}\right.$$
$$I_3(x) = \left\{\begin{array}{ccc} 1 & , & x = 1 \\ 0 & , & {\rm otherwise}\end{array}\right.$$
Then,
\begin{eqnarray*}
f(x;\theta) & = & (\theta)^{I_1(x)}(1 - 2\theta)^{I_2(x)}(\theta)^{I_3(x)} \\
& = & \theta^{I_1(x) + I_3(x)}(1 - 2\theta)^{I_2(x)}.
\end{eqnarray*}
Therefore, by the Factorization Theorem, a sufficient statistic for $\theta$ is:
$$T(\underline{X}) = (I_1(x) + I_3(x), I_2(x))$$
To show that this is a minimum:
$$\frac{f(x;\theta)}{f(y;\theta)} = \frac{\theta^{I_1(x) + I_3(x)}(1 - 2\theta)^{I_2(x)}}{\theta^{I_1(y) + I_3(y)}(1 - 2\theta)^{I_2(y)}}$$
is independent of $\theta$ if and only if $I_1(x) + I_3(x) = I_1(y) + I_3(y)$ and $I_2(x) = I_2(y)$. Therefore, $T(\underline{X})$ is a minimal sufficient statistic.

Is $X$ a complete statistic?
Answer: By definition I know I need to show $E[u(z)] = 0$ implies $u(z) = 0$ for all $\theta$. However, I am unsure how to show $X$ is complete.

From Part 1, is the statistic complete? Why?
Answer: Once again I understand the definition of complete, but am unsure on how to reach the conclusion based on:
\begin{eqnarray*}
E[u(z)] & = & 0 \\
\sum\limits_{z = -1}^1 u(z)\theta^{I_1(z) + I_3(z)}(1 - 2\theta)^{I_2(z)} & = & 0
\end{eqnarray*}
Comment: I do not think your proposal is a minimum (if the sample size is fixed and known)
Comment: Please add the [tag:self-study] tag & read its
[wiki](https://stats.stackexchange.com/tags/self-study/info).
Answer: The minimal sufficient partition is
$$\{ \{-1,1\}, \{0\} \} $$
(since $\frac{f_\theta(1)}{f_\theta(-1)}$ does not depend on $\theta$).
So the minimal sufficient statistics(MSS) is
\begin{eqnarray}
S(x)=\left\{
\begin{array}{cc}
a & x=0 \\
b & x=-1,1
\end{array}
\right.
\end{eqnarray}
Is it a complete statistics?
$\forall \theta\in(0,\frac{1}{2})$
\begin{eqnarray}
0=E(g(S(X))) &=& g(b)\theta +g(a) (1-2\theta )+g(b)\theta \\
&=& g(a)+\left(g(b)-2g(a)+g(b)\right) \theta
\\
&=& g(a)+2\left(g(b)-g(a)\right) \theta
\end{eqnarray}
So you can easily check it.
Comment: @Xi'an, Thanks, It is corrected.


Enviado POST de text "" via AJAX

Question: Boa tarde, tenho este código:
            <?php foreach ($resultado as $row) { ?>

                    <?php echo $row['seis']; ?>
                    <?php echo $row['ban']; ?>
                    <?php echo $row['bon']; ?>
                    <?php echo $row['quantidade']; ?>
                    <?php echo "R$ " . $row['preco']; ?>
                    <?php echo $row['Vendedor']; ?>

                        <input name="comprar" id="comprar" value="Comprar" class="btn btn-success" type="submit"></input>

            <?php } ?>

Ele faz uma tabela para cada coluna no BD. Eu preciso enviar estas informações (realizando um POST) do que está dentre os . Tentei com este código:

Mas ao clicar no botão comprar, todos os  estão sendo enviado, sendo que é pra enviar de acordo com a coluna que eu cliquei o botão comprar
Exemplo: valorTd: 498401 498406 547874 547874 547874 547874 547874 547874 547874 547874
Comment: Não ficou muito claro, mas... 1) seu `foreach` vai gerar vários td's com o mesmo valor no atributo `name`; 2) no jquery você faz referência à classe `$('.seis').text()` e deveria ser ao atributo `name`
Comment: No seu *input* você poderia criar um atributo referencial apontando para a `tr` e então pelo jquery você 'navega' pelos `td's` que precisa recuperar.
Comment: Tentei desta forma: `  var valorTd = $('#seis').text();
         var valorTd2 = $('#ban').text();
         // Envia a requisição ajax
         $.ajax({
             url: "ajax/recebe_pedido.php", // Arquivo php que vai receber os dados
             data: { // Recebe os dados das td´s e passa em um json
             valorTd: valorTd,
             valorTd2: valorTd2, ` Mas o POST vem vazio (eu alterei de class para name)
Answer: Assumindo que no seu foreach tenha uma coluna id, tente assim:
<?php foreach ($resultado as $row) { ?>

        <td class="seis_<?=$row['id']?>"><?php echo $row['seis']; ?>
        <td class="ban_<?=$row['id']?>"><?php echo $row['ban']; ?>
        <td class="bon_<?=$row['id']?>"><?php echo $row['bon']; ?>
        <td class="qntd_<?=$row['id']?>"><?php echo $row['quantidade']; ?>
        <td class="pre_<?=$row['id']?>"><?php echo "R$ " . $row['preco']; ?>
        <td class="ven_<?=$row['id']?>"><?php echo $row['Vendedor']; ?>

            <input name="comprar" id="comprar" value="Comprar" class="comprar btn btn-success" type="submit" coluna="<?=$row['id']?>"></input>

<?php } ?>

Comment: Funcionou, mas só o primeiro botão **Comprar** está enviando requisição, o resto não funciona
Comment: Sabe dizer por que isso acontece ?
Comment: Tente agora, referenciei uma classe ao invés do id do botão
Comment: Deu certo ? Caso sim, marque a resposta como resolvida

Enviado POST de text "" via AJAX


Front-Back Transformation - copy

Question: Given a string S, consisting only of english alphabets, replace all the alphabets with the alphabets occuring at the same position when counted in reverse order of alphabets. For example, 'a' would be replaced by 'z', 'b' by 'y', 'c' by 'x' and so on. Any capital letters would be replaced by capital letters only.
Comment: Welcome to SO! please show us what you have tried.
Answer: <code>k=[]

s= 'my name is talha'

small = [chr(x) for x in range(97,123)]
#['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

small_reversed = small[::-1]
#['z', 'y', 'x', 'w', 'v', 'u', 't', 's', 'r', 'q', 'p', 'o', 'n', 'm', 'l', 'k', 'j', 'i', 'h', 'g', 'f', 'e', 'd', 'c', 'b', 'a']

d=dict(zip(small,small_reversed))
#{'a': 'z', 'b': 'y', 'c': 'x', 'd': 'w', 'e': 'v', 'f': 'u', 'g': 't', 'h': 's', 'i': 'r', 'j': 'q', 'k': 'p', 'l': 'o', 'm': 'n', 'n': 'm', 'o': 'l', 'p': 'k', 'q': 'j', 'r': 'i', 's': 'h', 't': 'g', 'u': 'f', 'v': 'e', 'w': 'd', 'x': 'c', 'y': 'b', 'z': 'a'}

for x in s:
    if x in d:
        k.append(d[x])
    else:
        k.append(x)

print(''.join(k))
#nb mznv rh gzosz
</code>

Front-Back Transformation - copy


Split a column list into columns

Question: Suppose I have a DT as -
id         values          valid_types
 1            2|3             100|200
 2              4                 200
 3            2|1             500|100

The valid_types tells me what are the valid types I need. There are 4 total types(100, 200, 500, 2000). An entry specifies their valid types and their corresponding values with | separated character values.
I want to transform this to a DT which has the types as columns and their corresponding values.
Expected:
id   100   200  500
 1     2     3   NA
 2    NA     4   NA
 3     1    NA    2

I thought I could take both the columns and split them on | which would give me two lists. I would then combine them by setting the keys as names of the types list and then convert the final list to a DT.
But the idea I came up with is very convoluted and not really working.
Is there a better/easier way to do this ?
Answer: Using tidyr library you can use separate_rows with pivot_wider :
library(tidyr)

df %>%
  separate_rows(values, valid_types, sep = '\\|', convert = TRUE) %>%
  pivot_wider(names_from = valid_types, values_from = values)

#     id `100` `200` `500`
#  <int> <int> <int> <int>
#1     1     2     3    NA
#2     2    NA     4    NA
#3     3     1    NA     2

A data.table way would be :
library(data.table)
library(splitstackshape)

setDT(df)
dcast(cSplit(df, c('values', 'valid_types'), sep = '|', direction = 'long'),
                 id~valid_types, value.var = 'values')

Comment: Is there a way to do this without `tidyr` ?
Comment: Thanks. But I was trying to achieve this with minimal packages. Hence going with the other answer although this solves it as well.
Answer: Here is another data.table approach:
dcast(
  DT[, lapply(.SD, function(x) strsplit(x, "\\|")[[1L]]), by = id],
  id ~ valid_types, value.var = "values"
)

Comment: @leoOrion `id + date ~ valid_types`
Comment: Ah. I was doing this and you solved it before I could. Thanks. This is what I want
Comment: What if my key id is a composite of 2 cols -> `id,date`. I can change the by cluase but how do i do the same for `id~valid_types` ?

Split a column list into columns


gSOAP: No socket in soap_ssl_accept()

Question: I am trying to setup an example gSOAP server with SSL. I created a self signed certificate as this:
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

Then joined the certificates to one file:
cat cert.pem key.pem >both.pem

And then run this code:
#include <stdsoap2.h>
#include <stdexcept>
#include <sstream>
#include <iostream>

#define PORT 6667

int main(int argc, char **argv) {
    struct soap engine(SOAP_C_UTFSTRING);
    engine.bind_flags=SO_REUSEADDR;

    soap_ssl_noinit();

    int rc = soap_ssl_server_context(&engine,
            SOAP_SSL_DEFAULT,
            "both.pem", // keyfile
            nullptr, // password
            nullptr, // cacert
            nullptr, // capath
            nullptr, // dhfile
            nullptr, // randfile
            nullptr  // serverid
    );

    if (rc) {
            std::stringstream error;
            soap_stream_fault(&engine, error);
            throw std::runtime_error("soap_ssl_server context failed " + error.str());
    }

    rc = soap_bind(&engine, NULL, PORT, 100);
    if (rc < 0) {
            throw std::runtime_error("soap_bind failed " + std::to_string(rc));
    }

    int master_socket = rc;
    std::cout << "master socket: " << master_socket << std::endl;

    engine.accept_timeout = 50;
    engine.send_timeout = 50;
    engine.recv_timeout = 50;

    rc = soap_ssl_accept(&engine);
    if (rc) {
            soap_stream_fault(&engine, std::cout);
            throw std::runtime_error("soap_ssl_accept failed " + std::to_string(rc));
    }
    std::cout << "soap_ssl_accept " << rc << std::endl;

    return 0;
}

But I get this output:
master socket: 3
Error 30 fault detected[no subcode]
"SSL/TLS error"
Detail: No socket in soap_ssl_accept()
terminate called after throwing an instance of 'std::runtime_error'
  what():  soap_ssl_accept failed 30

What am I doing wrong?
Answer: I figured it out. One must call soap_accept before calling soap_ssl_accept. The correct code is:
    rc = soap_accept(&engine);
    if (rc) {
            soap_stream_fault(&engine, std::cout);
            throw std::runtime_error("soap_accept failed " + std::to_string(rc));
    }

    rc = soap_ssl_accept(&engine);
    if (rc) {
            soap_stream_fault(&engine, std::cout);
            throw std::runtime_error("soap_ssl_accept failed " + std::to_string(rc));
    }

and then the error is fixed.

gSOAP: No socket in soap_ssl_accept()


Do i need my Root note, 3rd or 5ft note to be at down beat to be in Chord?

Question: Do i need my Root note, 3rd or 5ft note to be at down beat to be  in Chord ?
Comment: Only if you want all your music to sound the same (and not very interesting either). Putting all the chord changes exactly on the down beats is a common beginner's mistake. There is no "rule" that says you can't put them anywhere you like.
Answer: If you play the whole chord (Root, 3rd and 5th) on the down beat, it will clearly give the chord to the listener. So, if this is what you aim for, do it. But, if you play all your chords like this, the song might get boring away after a while, and that is why many musicians prefer to not give away all the notes at the down beat. You can play some chord notes on the down beat and some on the up beat or some on the down beat and some on the next beat (really depends on your rhythm).
Try out a few ways and see what you like best!
Answer: No.  An arpeggio counts as a chord. And chords, broken or otherwise, don't have to come on a strong beat.


Rules for computer aided chord progression

Question: I'm working on a program to help musicians write chord progressions. You pick the chords one at a time in sequence, and the computer to try to "guess" what chords should come next, similar to the way iphone guesses what word i'm trying to type.
For example, given C-Am-F, the next is likely G.
Here are some rules I've come up with so far. ("home" is the first chord in the song, "last" is the last chord so far, "next" is the chord the computer will guess)

the tonic note of next is in the scale of the last (eg. G is in the F major scale)
the dominant note of next is in the scale of the last (D is in the F major scale)
the tonic note of next is in the scale of home (G is in C major)
the next and the last have some notes in common

A score is determined based on the number of rules that are satisfied. (Some might be weighted more than others.) The top 10 chords based on score are then presented to the user to pick from.
Has this already been done?  Can you help me to think of more rules?  I know that my rules so far are not very good.
Comment: Personally I would prefer to set an expectation and then challenge it to create surprise for the listener. I would prefer to look at it from a totally different perspective, and since I'm trying to write to a feeling, I can't figure out how a computer could be programmed to help me with that. I definitely would want something that suggests the next chord I might use based on my writing history - unless perhaps I use it to make sure I do something new and different every time. But then the algorithm can't learn from me.
Comment: Yeah, you might have more success with a Markov chain, where a chord is followed by a *selection* of other chords, each with a certain probability of being chosen.
Comment: @JohnHenckel this may interest you: it's a statistical study of the occurrence of chords in rock music http://www.midside.com/publications/declercq_temperley_2011.pdf, especially jump to p. 61 chord transitions
Comment: The quality of your results is going to depend heavily on the quality of programming, and what "rule set" you allow.  See, for example, Hindemith's book on theory and harmony.
Comment: I might add that this sort of thing has been done many times - with expert programmers setting up neural nets with adjustable weights for far more properties than just chord progressions.
Comment: To add to @Dekkadeci's comment, you could generate different chains for different genres, or just mash them all together.
Comment: THANK YOU everyone for the awesome feedback. I am writing in "Tone.js" so it runs in a browser. I'll post progress here. @ToddWilcox I agree, emotion content often requires the unexpected. I will certainly allow exceptions.
Answer: Percy Goetschius, an American music theoretician of the 19th and early 20th centuries, gave the following table in one of his books. Should get you started. As he says, the ordinary progressions are those of one class to the left, so these should get the largest probability weight. Make sure to give the "digressions" (motions to the right) some love too, however, otherwise you would never leave the tonic class anymore, and also the larger digressions (two classes to the right) for the sake of surprise. What these relative probability weights should be numerically depends on the particular style you want to recreate. Of course all of this doesn't account for modulations, which have their own "rules".

Answer: The Project looks like a good idea, However take a look at piano companion(android) mobile app ( I think it has something similar to this feature). Also check garage band(ipad) app.
The example rules listed in the questions are not clear to me, I think you have mixed note and chord in your example rules (the tonic note of next is in the scale of the last (eg. G is in the F major scale) this is confusing, you can reword the rules)
Meanwhile also take a look at chord progressions using "circle of fifth" in music theory, that will help you come up with possible next chords for a given scale, given first chord and given last chord.
Also consider adding major,minor natural,minor harmonic, minor melodic,pentatonic scales support and possible chord progressions including major/minor/augmented/diminished/sus2/sus4/7th/dominants/9th/11th/13th chords and midi instruments selection.
Answer: Just browsing the paper (indicated by Michael Curtis) and it is a gold mine!  Particularly this table.


Consider the complex $A_{*}: A _ {i \in \{0,1,2\}} = \mathbb{R}^2$, where all other $A_i = 0$. Find all relevant homologies.

Question: I should include the maps here:
$f_{0\to 1} (x,y) = (x,0)$, and $f_{1 \to 2} (x,y) = (0,y)$.
The first thing we are asked is to show this is indeed a complex, which is really trivial for me. (simple composition of maps).
The follow up is to find all relevant homologies.
$A_{*}: 0 \to A_0 \to A_1 \to A_2 \to 0$. I know that the definition of a homology is
$H_i (A_{*}) = \dfrac{ker f_i}{im f_{i-1}}$
Using this, I think there are only 3 relevant homologies (meaning non-trivial). Is this correct? I also need to say what they are.
For example, I think that $H_1 (A_{*}) = \dfrac{ker f_{0 \to 1}}{0} = A_1$. Is this correct?
Answer: Note that $H_0(A_*)=\ker f_{0\to 1}= 0\times \mathbb R\cong\mathbb R$.
The complex is exact at $A_1$, i.e. $\ker f_{1\to 2}=\operatorname{im} f_{0\to 1}$, hence $H_1(A_*) = 0$.
Finally $H_2(A_*)=A_2/\operatorname{im}f_{1\to 2}= \mathbb R^2/(0\times \mathbb R)\cong \mathbb R$.
Comment: Very nice. Thank you very much.


WebDriver PhantomJS Unable to find element, but works fine with Firefox

Question: I have been banging my head into the wall for a long time now so I thought I would ask the "experts" why the below code would not work (entering password) with PhantomJS but works just fine with Firefox. The most disturbing of all is that one field entry (username) is successful but the second would not work at all. The page loads just fine and I have included test code to verify other components are loaded just fine.
See code below:
import java.io.File;
import org.openqa.selenium.*;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.phantomjs.PhantomJSDriver;

public class login {

public static void main(String[] args) {
    WebDriver driver;
    Boolean verbose = false;  //Change to true to test it with firefox
    String phantomPath = "../phantomjs-1.9.8-linux-i686/bin/phantomjs";
    String url = "https://www.britishairways.com/travel/redeem/execclub/_gf/en_us";

    if (verbose) {
         driver = new FirefoxDriver();
         }
    else{
        File file = new File(phantomPath);
        String userAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; cs; rv:22.214.171.124) Gecko/2009032609 Firefox/3.0.8";
        System.setProperty("phantomjs.binary.path", file.getAbsolutePath());
        System.setProperty("phantomjs.page.settings.userAgent", userAgent);

        driver = new PhantomJSDriver();
        }
    driver.get(url);
    try{
        driver.findElement(By.id("membershipNumber")).sendKeys("1234");
        System.out.println("ID input successful");
        if (driver.findElement(By.id("ecuserlogbutton")).isDisplayed()) {
            System.out.println("Login Button is present");
        }
        //This is where it fails with PhantomJS but work with Firefox
        driver.findElement(By.cssSelector("#pintr > #password")).sendKeys("1234");
        System.out.println("password input successful");
        }
    catch (Exception e){
        System.out.print(e.getMessage());
        }
    driver.close();
}
}

Comment: It might be timing issues. Try to use Thread.Sleep(2000) before each findElement and observe the behavior. If it works, then you know it's timing issue. Also there is a method called WaitForPagetoLoad. You can call that before you input into elements.
Comment: Well, solved my own problem. Seems like the css selector will not work with PhantomJS I used by.xpath with .//*[@id='password'] and now it works.
Comment: Thanks neo, I actually tried that one too by very slowly debugging the code with eclipse. Still not sure why the css selector is not working.
Comment: But still it's a good practice to check if the document is loaded or checking if an element exists before trying to access it. WebDriver performance might change on different computers. I had situations where tests run fine in one pc, but in another pc fails because of timing issues.
Answer: PhantomJS 1.x has a problem with element IDs. The site is broken, because it uses password for two elements on the page which should never happen. Simply replacing the id in the selector with the element type (input) solves it.
driver.findElement(By.cssSelector("#pintr > input")).sendKeys("1234");

Comment: You rock sir! I don't have the reputation to give you a thumbs up, but thanks SO much! I did not notice that the duplicate name for password.
Comment: Yes thanks! I used the screen capture to look at the page but all I saw was the empty field. I guess I would have to do find on the element ID to look for duplicates the next time. Thanks again for your help!!!
Comment: Yes, I also saw the empty field, but I also noticed that somewhere else some text appeared. You will receive an error when the field doesn't exist or is not visible. So that means, when the field should be filled, but is not, you probably selected the wrong field.
Comment: hi, if i have By.id("userid") how i can write your code?
Comment: @AntonioBalduzzi I don't know what you mean. If you can't describe the problem in detail in a comment, it should be a [question](http://stackoverflow.com/questions/ask).
Comment: @ArtjomB. the problem is the same the driver can't find the element if i use: driver.findElement(By.id("userid"));
Comment: @AntonioBalduzzi It isn't the same case as in this question. Make sure that the element actually exists in the page. Maybe you need to wait for it to appear.
Answer: Try the methods from this link
From my experience with WebDriver, it's usually timing issues. Call the method in above link at the beginning of your code so you can make sure everything loads before you try to find them. Or you can simply use Thread.Sleep with long enough time before finding elements.


Is it possible to have a higher train error than a test error in machine learning?

Question: Usually it is called over-fitting when the test error is higher than the training error. Does that imply that it is called under-fitting when the training error is higher than the test error? Also what is the difference between test error and validation error?
Answer: These simplified formulae from Stanley Сhan's Introduction to Probability for Data Science provide some good intuition on the train/test error:
MSEtrain = σ2 (1 - d/N)
MSEtest = σ2 (1 + d/N)
where σ2 is noise variance (data measurement errors and such), N is the sample size, d is a measure of model complexity (d<=N).
Overfitting means that a model fits too closely to the training samples so that it fails to generalize (d is comparable to N): train error is low, test error is high.
When the model is too simple and uderfits data (d<<N), train and test error are about equally high.
When your test error is suspiciously low, that likely means you've got some sort a of data leakage from a test set (e.g. when you preprocess a whole dataframe before splitting).
As of validation vs test error, a separate validation split normally serves as an intermediate test for e.g. best model/hyperparameter selection. Since you purposely select what performs best on this set, it can be treated as a sort of leakage as well. Thus the final test data should remain unseen until the very end of the model development process.


Compute explicitly $\sum_{x=1}^\infty\sum_{y=1}^\infty \frac{x^2+y^2}{x+y}(\frac{1}{2})^{x+y}$

Question: It is possible to compute explicitly the following series?
$$\sum_{x=1}^\infty\sum_{y=1}^\infty \frac{x^2+y^2}{x+y}\Big(\frac{1}{2}\Big)^{x+y}$$
@Edit I tried to sum and subtract $2xy$ in the numerator.
In this way I get the following
\begin{align}
\sum_{x=1}^\infty\sum_{y=1}^\infty (x+y)\Big(\frac{1}{2}\Big)^{x+y}-2\sum_x x\Big(\frac{1}{2}\Big)^x\sum_y\frac{y}{x+y}\Big(\frac{1}{2}\Big)^y.
\end{align}
The first series should be easy to be computed. It remain the second one, in particular the following
\begin{align}
\sum_y\frac{y}{x+y}\Big(\frac{1}{2}\Big)^y=\sum_y\Big(\frac{1}{2}\Big)^y-x\sum_y\frac{1}{x+y}\Big(\frac{1}{2}\Big)^y.
\end{align}
Therefore the problem is reduced to computing:
\begin{align}
\sum_{y=1}^\infty \frac{1}{x+y}\Big(\frac{1}{2}\Big)^y.
\end{align}
But I don't know well how to do it.
Comment: What did you try ?
Comment: $\sum_{y=1}^\infty \frac{1}{x+y}$ never converges.  Fortunately, your third equation is wrong: you left out a $(1/2)^y$.
Comment: Thank you. I typed wrong
Answer: Little easier way is to do a substitution (since $n+m$ arrives many times). What I mean is, we can rewrite (where $x \in (0,1)$ let's say to not bother about changing the order of summation (but any $|x| < 1$ will be good due to absolute convergence of double series) $$ \sum_{m =1}^\infty \sum_{n=1}^\infty \frac{n^2 + m^2}{n+m} x^{n+m} = \sum_{m=1}^\infty \sum_{n=1}^\infty (n+m)x^{n+m} - 2\sum_{n=1}^\infty \sum_{m=1}^\infty \frac{nm}{n+m}x^{n+m} $$
Now, let $k=n+m$. If $n,m$ can be any numbers between $1$ and $+\infty$, then $k$ is any number between $2$ and $+\infty$. If $k \ge 2$ is fixed, $n$ can be any number between $1$ and $k-1$, hence $$ \sum_{m=1}^\infty \sum_{n=1}^\infty (n+m)x^{n+m} = \sum_{k=2}^\infty \sum_{n=1}^{k-1} kx^k = \sum_{k=2}^\infty k(k-1)x^k $$
Similarly $$ \sum_{n=1}^\infty \sum_{m=1}^\infty \frac{nm}{n+m}x^{n+m} = \sum_{k=2}^\infty \sum_{n=1}^{k-1} \frac{n(k-n)}{k}x^k = \sum_{k=2}^\infty \frac{x^k}{k} \sum_{n=1}^{k-1} n(k-n) = \frac{1}{6}\sum_{k=2}^\infty (k-1)(k+1)x^k $$
Both sums can be easily calculated via differentiating under sum sign, I will leave it to you
Answer: $$ \sum_{y=1}^\infty \frac{1}{x+y} \left(\frac{1}{2}\right)^y = \Phi(1/2,1,x) - 1/x$$
where $\Phi$ is the Lerch Phi function.
Comment: Thank you. I never heard about the Lerch Phi function. This means that I should be able to compute $\sum_x x2^{-x}\phi(1/2, 1, x)$. I hope this is possible. I will study about the Lerch Phi function. Thanks.


Gmail does not accept mail from a new VPS server with PTR setup

Question: I have PTR records setup but I am not able to deliver mail to Gmail accounts:
host gmail-smtp-in.l.google.com[2607:f8b0:4004:c08::1b] said:
550-5.7.25 [2a01:4ff:f0:a7b6::1] The IP address sending this message does not
550-5.7.25 have a PTR record setup, or the corresponding forward DNS entry does
550-5.7.25 not point to the sending IP. As a policy, Gmail does not accept
550-5.7.25 messages from IPs with missing PTR records. Please visit
550-5.7.25  https://support.google.com/mail/answer/81126#authentication for more
550-5.7.25 information.
u1-20020ac87501000000b00315c38791casi6985465qtq.683 - gsmtp (in reply to end of DATA command)

Can someone please walk me through on how to setup my PTR records correctly. I am hosted on Hetzner Cloud and used Rainloop to create the email accounts.
Answer: Your PTR record for IPv6 is setup correctly (and strictly speaking it seems to be no longer required by Gmail).
Saying that one hint is in the message Gmail sends to you:
or the corresponding forward DNS entry does 550-5.7.25 not point to the sending IP

Which is the case here:
tkepczyn-linux1:~> dig mail.saveneur.com aaaa +short
tkepczyn-linux1:~>

There is no mapping from the mail.saveneur.com back to the IPv6 address.
Saying the above please also consider setting up SPF record. If it is missing this may be the real cause of this error. Unfortunately I cannot help with this as you haven't provided the sending domain name.
Comment: AAAA record requies full IPv6 address, not a network address. Just put 2a01:4ff:f0:a7b6::1 there. I don't know if it would suffice, ask Google. But according to the message it should.
Comment: I edited post to show newly added AAAA record. Would that suffice?


Defining arrays in Javascript

Question: While defining variables at the start it fails to define the second 2d array. Why? How should it be done?
var NPC = new Array(10);
for (var i = 0; i < 10; i++)
{
    NPC[i] = new Array(7);
};

var previous_bead_NPC = new Array(10);
for (var i = 0; i < 10; i++)
{
    previous_bead_NPC[i] = new Array(7);
};

Comment: I see both in Firefox/Firebug: http://jsfiddle.net/userdude/AKZyD/
Comment: What do you mean, it "fails to define the 2d array".  How can you tell?
Comment: I don't see any problem with the 2nd array. Did you get an error message while doing this or when you want to use these arrays?
Comment: In Firebug I can see only the NPC array and second one is absent.
Comment: Still, in DOM tab in Firebug I only see NPC array. WTF I'm stupid or what?
Comment: Yes your code definitely defines both "NPC" and "previous_bead_NPC". What exactly is it that makes you think it isn't doing so?
Comment: See: http://i.imgur.com/FplTs.png It can be a little confusing, I suppose, in jsFiddle, since you have to find the `iframe`. It's there, though. Personally, I don't know why you're traipsing through the DOM tab to find your declared global arrays. Is there a root cause for you doing this?
Answer: There's no real point to using the Array constructor:
var NPC = [];

is fine. JavaScript doesn't preallocate space anyway (at least, modern runtimes don't). If you want an array to have a particular length, you can do this:
NPC[9] = undefined; // now it'll be 10 elements long

Thus if you want to make an array of 10 rows, each of which being an array of length 7:
var NPC = [];
for (var i = 0; i < 10; ++i) {
  NPC[i] = [];
  NPC[i][6] = undefined;
}

(I don't know why you'd want that, but I guess it could serve some purpose.)
Comment: Old habits can be hard to break, I suppose.
Comment: What's the point of setting an element somewhere to undefined? Setting the length property (best at initialisation with the Array constructor) seems less confusing.
Comment: You know, you're right; I haven't had enough coffee this morning. :-) Well also I personally dislike the Array constructor because the semantics are so odd and inconsistent. This whole question is just kind-of pointless I think.
Answer:
This is what i see when i paste your code in a sample web site. No modifications made. So i guess your problem can be solved by fixing/updating your firebug ;)
Comment: I don't know why it did not list properly previous_bead_NPC - I changed it to Previous_bead_NPC and I can see it now just as you do. Stupidest thing ever.
Comment: Also in the jsFiddle demo: http://i.imgur.com/FplTs.png
Comment: @user1510009 no problem dude. I'll flag this question for a delete, because in the end there is no value in this question or it's answers.

Defining arrays in Javascript


Extract a specific string inbetween strings by regex

Question: I have a url like this
  http://localhost:4970/eagle
&Account=001
&FruitSlad=Apple, Rambutab
 &Fruits=Canada

and I want to extract the text between  FruitSlad and the next & and I tried to write a regex for it
I tried ((GroupByMultiples)(.$&?)) but it didnt work .  Um seeking for a help to extract the text between &FruitSlad and the next &
Comment: post the expected output..
Comment: you mean this `(?<=&FruitSlad=)[^&\n]*` or `(?<=&FruitSlad=).*`
Comment: expected output is Apple,rambutan
Comment: @kirov which language, to know wich interpreter and see if you need capture groups or not, etc.
Comment: its working for Javascript and Python @AvinashRaj thank you
Comment: it won't work for js but it do work for python..
Comment: Let me check again . i wont to use with js
Comment: @AvinashRaj it works for JS smoothly , thank you
Answer: /FruitSlad=([^&]+)/

See here.
Answer: In JavaScript you could do the following.
var url = "\
  http://localhost:4970/eagle \
&Account=001 \
&FruitSlad=Apple, Rambutab \
 &Fruits=Canada";

var result = url.split('&')[2].split('=')[1];
console.log(result.trim()); //=> "Apple, Rambutab"

Or if you prefer using regex ...
var result = url.match(/&FruitSlad=([^&]+)/);
if (result)
    console.log(result[1].trim()); //=> "Apple, Rambutab"

Answer: The convenient way to get values of query variables in Python 2.x is to use urlparse module:
import urlparse

url = 'http://localhost:4970/eagle?Account=001&FruitSlad=Apple, Rambutab&Fruits=Canada'
vars = urlparse.parse_qs(urlparse.urlparse(url).query)
print vars['FruitSlad'][0]

In Python 3.x use urllib.parse module:
import urllib.parse

url = 'http://localhost:4970/eagle?Account=001&FruitSlad=Apple, Rambutab&Fruits=Canada'
vars = urllib.parse.parse_qs(urllib.parse.urlparse(url).query)
print(vars['FruitSlad'][0])

This module decodes also various encoded characters (%nn), so is more suitable than regex in this case.


Elastic search not get the all of documents in python code

Question: I have elastic search single node cluster and it has some index.The index contain 6000 or more documents.Document mapping has lot of fields like(30 or more).I can query all of duplicate records in (Kibana dev tool) but i am querying with python result like 10 records or 20 records.What is the reason?
my index and document count
<code>yellow open   test_index kjioInpQRAqT3o1LZHI92g   1   2       7652         1267      20.7mb          20.7mb
</code>
my mapping
<code>{
 "settings": {
  "number_of_shards": 1,
  "number_of_replicas": 2
 },

 "mappings": {
  "test_index" : {
    "properties": {
        "name": {
            "type": "keyword"
        },
        "address": {
            "type": "keyword"
        },
        # MORE 35 FIELDS ....
        }
      }
   }
}
</code>
Answer: I found the reason.Document has lot of fields.It has 35 fields.I can query in (kibana dev) tool.Query returned 1000 documents for one time.Python elastic search library can't get this count of documents reason is the single document is too large (Document has 35 fields).Then i change the query and i put the useful fields for query result.Now it's work file.
Change your query like this (normal query)
<code>`{
   "_source": ["_id","_index", "quote_date", "asset"],
   "query": {
       "match_all": {
       }
   }
}`
</code>
Change your query like this (aggregation query)
<code>` "aggs": {
  "duplicateDocuments": {
     "top_hits": {
        "_source": ["_id","_index", "name", "address"],
                    "sort": [{"create_at": { "order": "desc"} } ]
                }
            }
   }`
</code>


Why is time series forecasting different for each software?

Question: I have 2 different software programs: SPSS, and Statgraphics.  I am using them for time series forecasting but

Each one gives different arima parameters when using the auto ARIMA model, and
The forecast is also different when using the same arima parameters in both software .

Can you please clarify why there are differences between them?
Comment: Could you please give an example model&data, and show the different results?
Comment: You should be able to edit your own post, see the *edit* button below the post! Then please edit the following information: Models used (maybe commands), link to the data, output & predictions. If you cannot include some, make a link here to some pastbin and I can include it.
Comment: See https://robjhyndman.com/hyndsight/estimation/
Comment: You need to include all essential information within the question itself.
Comment: I appreciate your edits.  However, we prefer that the information be wholly contained in your post itself, because links to other sites eventually rot.  That would make the entire thread incomprehensible.
Comment: Take 6 screen shots from your analysis.doc file and create a new question and include them in your new self-contained question.
Comment: When you have decided that my answer and comments have been useful to you , please upvote it and accept it
Comment: thank you for passing by my question, the data is from investing.com for one of the indices called (DFMGI), the data gathered monthly from Jan.2004 to Oct 2019.  the result from Statgraphics was ARIMA (1,1,1)(0,2,2)12, where ARIIMA parameters where all significantly less than 0.05 and the ljung box test was more than 0.05.  On the other hand for SPSS, the same data has been used giving ARIMA (0,1,2)(1,0,1)12, all parameters are significant but ljung box was less than 0.05 (0.012)
Comment: I'm sorry I couldn't find a way to paste the forecasted results for 12 months.  I will be grateful if you can help in order to see that the forecast is different between the two mentioned software
Comment: When using the model found by statgraphics which is ARIMA (1,1,1)(0,2,2)12 in SPSS the result for the MA seasonal was not significant (0.889 for lag 2) although the same was significant in Statgraphics !
Comment: thank you very much for the help, but believe me that from yesterday I was thinking about this miss, how both SPSS and Statgraphics give two different answers in term of ARIMA parameters and results when using the Auto ARIMA in both software as well as different forecasts based on two ARIMA parameters.  but also I tried to use the same ARIMA parameters for both software but again different forecast results and different ljung box results!  really I'm very confused.  the data is the same with no transformation used but unfortunately, the results are different.
Comment: how can I know which software gives a better answer to either R or SPSS or Statgraphics while they are giving different results?  I know that all expectations are wrong but the forecast is giving at least a proper direction or clue.  I really need help
Comment: I tried to get your data ,,, but had problems .. please post it here and I will try and give you some guidance.
Comment: @IrishStat please find the excel sheet for your further analysis and help https://1drv.ms/x/s!Ajj3wvWS6XTwgRknwzbUvZuqCxyZ
Comment: @kjetilbhalvorsen please find attached a link to the data https://1drv.ms/x/s!Ajj3wvWS6XTwgRknwzbUvZuqCxyZ
Comment: @IrishStat please see the attached analysis as per my question above https://1drv.ms/w/s!Ajj3wvWS6XTwgRt1lOBGfxIJo2j-
Comment: @whuber I gave all the explanations needed, I was clear enough in my question, in addition I added the data and the results obtained from SPSS and Statgraphics softwares in my above comments.  Please unhold my question since I have an intention to learn why each software give different forecast although I used the same ARIMA parameters in both softwares (1,1,1)(2,0,2)12 ?
Comment: @whuber thanks for the guidance, I modified the question to have all needed details
Comment: @whuber Thank you for your support, but I can not put all the information in the same question in addition to the data analysis because it takes more space and will not be understood too.  So I hope you accept this way so that we provide an easier opportunity to answer the question.
Comment: @whuber please be informed that i removed the links and i added a photo, is it ok now ? if yes please remove the unhold, thanks
Comment: @IrishStat thank you for your support, I added the jpg for the analysis and data for your considerations.  thanks,
Answer: To properly answer your question regarding automatic model differences requires a little bit of history https://autobox.com/pdfs/econometrics.pdf to explain some different approaches to ARIMA model identification.
Model identification has and always will be an iterative process much like peeling an onion where clues are found and followed and possibly discarded culminating in a possibly useful model . Box and Jenkins in 1968 suggested this approach and https://autobox.com/pdfs/ARIMA%20FLOW%20CHART.pdf reflects a modification/update of this where additional/possible model complexity is found out in an exploratory mode . Think of the EDA approach of J. Tukey but in a time series context.
My first attempt in my disseration topic in 1968 was to do a one pass brute-force approach of trying a family of possible models to determine the best model via minimizing the error variance. In this regard Auto.arima was based upon an early piece of my software called AUTOBJ (which is the forerunner of AUTOBOX which I have helped to develop) while using a relative of the error variance the AIC to suggest the "best model".
Neil Polhemus , a friend of mine , the founder of Statgraphics was also aware of AUTOBJ and decided to use what was then a promising model identification tool call the Extended Autocorrelation (EACF or ECF With regard to ARMA time series, what exactly is eacf (extended auto-correlation function)? ) as the basis for his approach. The EACF failed to detect either latent deterministic structure of the need for error variance remedies and is no longer a tool of choice.
What you have are two different approaches ( two pieces of software) to model identification , neither of which deal with the effect of possible latent deterministic structure (like level/step shifts or local time trends) and complications dealing with error variance structure (like power transforms or GLS).
Visually and analytically your selected time series has error variance issues (higher variability at higher levels) and a level shift. A simple ar(1) model in a log transform yields one possible pulse. Neither of your selected tools of choice deal with this opportunity space. The residuals from the model I suggest are free of structure.
A series can be non-stationary in the mean. There are (at least) two different ways to remedy this .. 1) to difference ... 2) to Demean ie. adjust for one or more level/step shifts or local time trends .
In closing your series ( although monthly) exhibits no provable seasonality. Incorporating unwarranted seasonal differencing INJECTS structure which needs a counter-balancing seasonal component to provide a remedy.
I developed a useful model ( 1 level shift needed to deal with non-stationarity (period 58) and a simple ar(1) component ) here  with Gaussian Noise (here) which provided the following Actual & forecast graph here  with wide limits due to the need for a log transform (N.B. how the error variance is smaller at lower levels of the original series ) . These forecasts look nothing like any of your three candidate models.
Also note that there is no seasonality in the original data , the model and the errors from the model . Tools that assume seasonal structure ..thus injecting seasonality into the residuals and then incorporating seasonal structure to counter-balance that had been erroneously injected should be critically examined.
The appropriate model is approximately a random-walk (.956 is nearly 1.0 ) with a positive drift (1.48) in logarithms.
Comment: I seriously doubt seasonality of 95 as there is no logic to this systematic behavior. You had I believe 190/191 values in the history thus at best you have two replications of  a seasonality of "95" . Again your series is non-stationary in the expected value BUT that does not always imply that the series should be regularly differenced either 1 or 2. In summary a plot of the residuals from either model should suggest sufficiency along with a clean acf . Be careful about the "buttons" you press.
Comment: When one uses AUTOBOX and the user does not know the seasonality , there is hyper-search where for each potential seasonality index , the "best model" is found by allowing both stochastic/adaptive seasonality and deterministic seasonality to be found thus empirically hopefully finding the best seasonality index. The user has to specify the range for this search. With 190 values it would be "silly" to consider a seasonality of 95 as there would only be two reps .I am not an expert in spectral analysis  to help you  correct your opinion that spectral software suggests 95 as the seasonality
Comment: When you are satisfied with my answer please upvote it (if you can ) and accept it to close the question . I suggest that you ask another question , posting all of your input/output suggesting to you that  a seasonality of 95 is appropriate to possibly  get help from frequency-domain experts.. You can refer that question to this one to provide background.
Comment: thank you very much for your valuable comments, on the other hand I have just showed the differences among the commercial software which are giving a very tedious auto Arima, wherein my case as an engineer who is not expert in performing ARIMA model without the auto button, I'm really frustrated from the results which I believe that the responsibility for both software owners/producers have to illustrate why this is happening?
Comment: regarding the cyclic of the data and from some analysis I came to know that the data is repeating itself every 95 months and thus if I used this seasonality I will end up with a more complex Arima model like (1,2,2)(2,0,0)95.  I hope from those who are producing such commercial software to review the auto Arima methodology and to include not only the false auto Arima but to give the right tools which help the researchers who are not specialized in scripting.
Comment: Thank you very much for your multiple and very useful answers.  I also know that 95 seasons are very ridiculous, but spectral analysis is very useful to know the periods in which the time series repeats itself.  For example, through the data attached to the question and through spectral analysis it is useful to know the seasonality during the 95 months which is the complete cycle for this data.  Thanks a lot for your support, I will try to use AUTOBOX to see if it is useful


A question about linear independence

Question:
Let $S$ be a linearly independent set. Let $S'$ be a proper subset of $S$.
$Span$$(S') \neq Span(S)$
Let $v$ be an element of $S$ which is not contained in $S'$; such an element must exist because $S'$ is a proper subset of S. Since $v \in S$, we have $v \in$ $span$$(S)$. Now suppose that $v$ were also in $span$$(S')$. This would mean that there existed vectors $v_1, \ldots, v_n \in S'$ (which in particular were distinct from $v$) such that
$v = a_1v_1 + a_2v_2 + . . . + a_nv_n$,
or in other words
$(−1)v + a_1v_1 + a_2v_2 + . . . + a_nv_n = 0$.
But this is a non-trivial linear combination of vectors in S which sum to zero (it's nontrivial because of the $−1$ coefficient of $v$). This contradicts the assumption that $S$ is linearly independent. Thus $v$ cannot possibly be in $span$$(S')$. But this means that $span$$(S')$ and $span$$(S)$ are different, and we are done.

So, we have $v = a_1v_1 + a_2v_2 + . . . + a_nv_n$. We can subtract $v$ from both sides of the equation and then argue that elements of $span$$(S')$ are linearly dependent.
Suppose a vector $v = a_1v_1 + a_2v_2 + . . . + a_nv_n$ is linearly independent. Can't we just subtract $v$ from both sides of the equation and say that the linear combination is linearly dependent every time?
Thanks.
Comment: "we are done" ? With **what** ? And one single vector cannot be linearly independent, though if it is the zero vector it is lin. dependent.
Comment: It's a part of bigger proof that I didn't want to post in full. Would take up too much space.
Comment: But then don't write that as people reading your post may think you forgot to add some info...
Comment: I edited my OP. Now it should be more clear.
Answer: Consider $a_{0}v - \sum_{i=1}^{n} a_{i}v_{i} = 0$. The vector $v$ is linearly independent with $v_{1}, ..., v_{n}$ if and only if $a_{0}, ..., a_{n}$ are all $0$ is the only linear combination satisfying the one I gave above.
So what we're really saying is this. If we have a set of linearly independent vectors, then no vector $v_{1}$ in the set $S$ can be formed as a linear combination of the vectors $v_{2}, ..., v_{n}$. If it could, then we would have two linear equations $-v_{1} + \sum_{i=2}^{n} a_{i}v_{i} = 0$ and $\sum_{i=1}^{n} 0v_{i} = 0$. Clearly, having two such equations violates the definition linear independence. Hence, the contradiction.

A question about linear independence


If $0\to M'\to M\to M''\to 0$ is exact, why does $\operatorname{Ass}(M)\subseteq \operatorname{Ass}(M')\cup \operatorname{Ass}(M'')$.

Question: I'm stuck on a proof I'm reading.

Let $0\to M'\stackrel{\mu}\to M\stackrel{\sigma}\to M''\to 0$ be a sequence of $A$-modules. Then $\operatorname{Ass}(M)\subseteq \operatorname{Ass}(M')\cup \operatorname{Ass}(M'')$.

Let $P\in\operatorname{Ass}(M)$, so $P=\operatorname{ann}(m)$ for some $m\neq 0\in M$.
If $Am\cap im(\mu)=0$, the proof shows $P=\operatorname{ann}(\sigma(m))$, so $P\in \operatorname{Ass}(M'')$.
When $Am\cap im(\mu)\neq 0$ it says there exists $m_0\in Am\cap im(\mu)$ nonzero. Write $m_0=am=\mu(m_0')$ for $m_0'\in M'$. Then $P=\operatorname{ann}(m_0)=\operatorname{ann}(\mu(m_0'))=\operatorname{ann}(m_0')$ since $\mu$ is injective. So $P\in\operatorname{Ass}(M')$.

The only equality above I don't follow is $P=\operatorname{ann}(m_0)$. The containment $P\subseteq\operatorname{ann}(am)=\operatorname{ann}(m_0)$ is clear, since $P=\operatorname{ann}(m)$. But why does the converse hold?
Comment: Hmmm...what is $\;Ass\,(M)\,$ ?
Comment: @Timbuc It's the set of associated primes.
Answer: Let's consider the generic example of short exact sequence, that is, $M'$ is a submodule of $M$ and $M''=M/M'$.
So, $P=\operatorname{ann}(m)$, $m\in M$, $m\ne 0$.
There are two cases: $Am\cap M'=0$ or not.
In the first case $P=\operatorname{ann}(\hat m)$, where $\hat m$ is the residue class of $m$ modulo $M'$.
In the second case, let $m_0\in Am\cap M'$, $m_0\ne 0$. Set $m_0=am$. Now you want to show that $P\supseteq\operatorname{ann}(m_0)$ (the other containment being obvious): if $bm_0=0$ then $bam=0$, so $ba\in P$ and since $P$ is prime and $a\notin P$ (why?) we get $b\in P$.


Magento Reindexing doesn't work

Question: I recently created around 700 attributes through script, all attributes look fine at backend.
But when I reindex, I get following error:

exception 'PDOException' with message 'SQLSTATE[42S22]: Column not
  found: 1054 Unknown column 'e.additional_information_s' in 'field
  list'' in  /lib/Zend/Db/Statement/Pdo.php:228

Note: this attribtue exist in Database (eav_attribtue) table.
I would highly appreciate suggestion.
Comment: which reindex exactly ? If product flat, take a look here : http://stackoverflow.com/questions/12893772/magento-product-attributes-recommended-maximum
Comment: Reindex got failed on catalog_fulltextsearch
Comment: For, I didn't set backend model for the attribute so I was getting such reindex fail error. When I corrected this, reindex works pretty well.
Answer: The following will reindex each index.
<code>for ($i = 1; $i <= 9; $i++) {
    $process = Mage::getModel('index/process')->load($i);
    $process->reindexAll();
}
</code>
You can also use the Magento collection model to load each index rather than hard coding the id in the for loop.
<code>/* @var $indexCollection Mage_Index_Model_Resource_Process_Collection */
$indexCollection = Mage::getModel('index/process')->getCollection();
foreach ($indexCollection as $index) {
    /* @var $index Mage_Index_Model_Process */
    $index->reindexAll();
}
</code>
But if you want to reindex just the price the id is 2
<code>$process = Mage::getModel('index/process')->load(2);
$process->reindexAll();
</code>
You could also call the function getProcessByCode as follows:
<code>$process = Mage::getModel('index/indexer')->getProcessByCode('catalog_product_price');
$process->reindexAll();
</code>

Magento Reindexing doesn't work


Mix two colors in LAB color space

Question: I have two colors in LAB color space, for example:
blue: 32.303, 79.197, -107.864 (hex code: #0000ff)
yellow: 97.138, -21.556, 94.482 (hex code: #ffff00)
I want to mix these two colors, and get the result color (hopefully some green in this case).
Any ideas how to accomplish this?
More about LAB color space:

http://www.colourphil.co.uk/lab_lch_colour_space.html
http://www.broadhurst-family.co.uk/lefteye/MainPages/Lab.htm
http://en.wikipedia.org/wiki/Lab_color_space
Comment: The [tag:coloring] flag is for graph colourings. The tag descriptions are displayed in the tag selection interface, so you can use them to decide which tags are appropriate.
Comment: Ok, thank you for pointing this out.
Comment: Similar question of mine on SO: http://stackoverflow.com/questions/1351442/is-there-an-algorithm-for-color-mixing-that-works-like-mixing-real-colors
Answer: I assume you are writing a computer implementation.
LAB is device independent. Your color codes are not, as they will be processed both by software and hardware before you see the colors. As a first approximation, convert your RGB to sRGB, which in this case will act as a reference device-independent color space.
When the sRGB values are obtained, convert to CIEXYZ and then to LAB, as the Wikipedia article on LAB suggests. Thus, write a function to compute $rgb\rightarrow sRGB \rightarrow XYZ \rightarrow L*a*b*$. Once the latter is obtained, you can interpolate linearly, e.g., take the average values of each value, and then perform the inverse transform to get your new RGB codes.
Comment: Thank you for your answer. This is what I just finished implementing. It works good for mixing red with yellow (resulting in orange), but mixing blue with yellow results in a kind of purple. It should be a kind of green, if I'm correct..
Comment: "[B]lue and yellow do not make green but a pinkish tint" -J.C. Maxwell
Answer: Simen's answer won't work for yellow and blue mixing.
There are two types of mixing of colors: mixing light and mixing pigments. When you are mixing light, then adding something will make it brighter. When you are mixing pigments, you are taking light out... making it darker.
Hence the confusion. On a computer monitor, yellow (255R, 255G, 0B) plus blue (0, 0, 255) is equal to white. If you do some sort of average, you get gray. With pigments, you get green, of course.
Simulating what the pigments do is very difficult. You need to convert to reflectance and form a product, red with red, green with green, and blue with blue. Even that won't work as you expect, but it will highlight the fact that a mixture of pure yellow pigment and pure blue pigment will give you black, and not green.
To make it work out, you need to start with the correct pigments. Cyan (0, 255, 255) mixed with yellow (255, 0, 0) gives you green when you multiply.
Answer: In L*a*b* the difference between two colors is the euclidian distance.
If you want to mix "equal" amounts, then that is the midpoint of the line between the two colors.
$ L^*a^*b^*_{mix} = \frac{L^*_1 + L^*_2}2, \frac{a^*_1 + a^*_2}2,\frac{b^*_1 + b^*_2}2 $
However, you might also want to look at CIELCh, which converts LAB to polar coordinates, allowing you to keep chroma constant and then ust adjust hue as the angle between the two colors. You'll find that many color blends in LAB will end up near grey if the colors are nearly opposite. This is because the midpoint line will go through the center of the LAB space, which is a chroma of zero.
Here's an example of four types of mixing between the colors.

The mid color for just mixing between the two sRGB values is #808080, for LAB it's #C98AA9 and for LCh it's #FF006A

Mix two colors in LAB color space