How to explain black and white keys on a piano keyboard

Question: I was going over quite a lot of answers that are already posted and I've figured out that I could ask this question anew. My perspective is that I have melodies coming to my head and  I record them with my voice. I will be learning music theory, piano and Ableton Live.
There is basically C, D, E, F, G, A and H on a piano keyboard and there is these black keys, which are Sharps / Flats. I am wondering how any melody that can come to my head (I have like 12,000 recorded so far, for like 4 months) translates into this. Any of them can be always played on a piano keyboard? Are sharps / flats some secondary type of notes, or is this something like 1-1-1-1-1-1-1-1-1-1-1-1 (12 sounds of equal importance) just layed out like this on a keyboard. And with this, there is no more sounds, they dont exist? So what exists in the end is these 12 and different octaves, which are on the keyboard (low to high). So there could be more octaves but no more than 12 of these sounds in each set? Or there can be, can there be like 15 or 17 or 30-something making up an octave? Do these 12 sounds translate somehow in any actual value (like Hertz)?
Just trying to understand - connect the music that comes or may come to my head with this piano keyboard. This is just "engraved in stone", that's how this is and it cant be different or what this is? I've always had an impressions that white keys are the main ones and the black keys are somehow secondary ones. I was watching some vids on Youtube and reading answers from here, but I don't get it yet.
Comment: Something along the line of "physics of sound", maybe some mathematics - what this really is scientifically (or at least based on the current knowledge that we have). This is what I am interested in.
Answer: Your question is very broad. It sounds like you are asking 'how does music work?' That can't be answered in a Q & A forum, but here are a few things I suggest you look into...
The basic Western music system uses 12 tones.
All 12 tones are called the chromatic scale. That will be all the keys on the keyboard.
But most music focuses on 7 of those tones called the diatonic scale.
On the keyboard the basic diatonic scale is all the white keys. Depending on which tone you start the scale you will get either a major scale or a minor scale (or one of several modes.)
In practice the diatonic scale mixes in the other 5 notes from the chromatic scale. Those will be the black keys on the keyboard. Melody and harmony then is a mixture of diatonic and chromatic sounds.
Regarding the pattern of black and white keys and this diatonic/chromatic system you must realize these elements can be transposed to 12 different starting notes. So, all the white keys on the piano are the diatonic scale. If you start that on C, you get the C major scale. But you can transpose that major scale to a different starting note like Db. When you do that the sequence of black and white keys will change, but the pattern will still be diatonic. The relative relationships stay the same. In the beginning this can be hard to understand, but it's a critically important aspect of music theory.

EDIT re. the scientific background.
This is in no way complete, but read these articles about why some pitch intervals seem more significant that others...

The harmonic series, this is more the acoustical science.
the Chord of Nature, this is more of a historic perspective but related to acoustics.
pitch class and octave equivalency, this is combination of acoustics, culture, and psychology.

Also, here are a few articles about musical naming and labeling...

Helmholtz and Scientific pitch notation, here is a page with both systems labeling the piano keys
intervals and their qualities
degrees of the scale

That's a lot of reading and a lot to thing about! It's probably best to stop at this point and give yourself time to digest all of those ideas. It's important to note there are philosophical debates on many of these topics. Don't approach them as absolute truths. Remember music is art. Beauty does not need to be reconciled with the harmonic series and such!
Comment: Thanks a lot for the info. Trying to start from "the bottom of this" mentally, something that "in fact" exists. Like a human can hear a range of sound (lets say 1 to 1000 Hertz or some other unit / number). Pitch is what divides this to human understanding and this can be divided into Pitch 1 to 1000, or something like that. It then translates onto a piano keyboard and this is this for this reason, and this is ths for that reson... Something like this. Still not sure why different pitches have the same names like c1, c2, c3 (like something in common, but would this be like physics of sound?)
Comment: Meaning, what is the scientific explanation of c1, c2, and c3 - something in common between them (tones on different scales, I think is the name).
Comment: I basically want to start like from the physics of sound then do the theory of music, then do the piano (not advanced, just what is needed for Ableton Live; like chords but not really some acrobatic things with the keyboard as this is not a needed thing, at least not now). Then kind of understand the synthesizers, all parameters of sound in a pc, anything like this. Understand. Also what are the rules of getting this together. Something like figuring this out for 10 years maybe, before starting to make my own songs. Maybe also playing a lot of already made songs (what I've liked).
Comment: This understanding can translate into the quality of music that I would produce. Easy to say, but this also takes time to complete... Again, sorry for bothering. I am kind of "hungry for info" but on a very beginner level of this. I am close to 40 years old and I want to learn like 3-5 new things (website programming, music, some sport, nutrition). Like starting all anew :) But yes, these melodies come to my head, I can make like 100 per hour or maybe more. Need to make a use of that, so I need to learn.
Comment: You would need to understand overtones and the harmonic series. You could start by reading up on the way a string oscillates.
Answer: These are all question under the subject of Music Theory. If you're writing music in the Western Tradition, then 12 notes per octave will usually be sufficient. Your keyboard controller has a pitch bender that can get subtle pitches in between the notes of the keys.
If you're just starting out transcribing your own melodies, then an important early step should be determining if it has a Major or Minor kind of sound. Use your "do-re-mi" to find how the melody lays on the scale and if "do" is the tonal center then it's major, or if "la" is the tonal center then it's minor. The tonal center of a melody can usually be found if you sing the melody and play a bass note on the keyboard that sounds good with the last note of the melody. The bass note should feel like "home" if you have the right one.
If use the key of C for Major melodies and use the key of A minor for Minor melodies then you can (mostly, usually) stay on the white keys.
Bon chance!
Comment: So "pitch" is the only parameter that differentiates the piano keys? And with the above, I could ass well have 174 piano keys (88 times two) (meaning, the same range of sound as the 88 keys, but more accurate, more spread out)? I am a total novice so I need to learn these things here before bothering here. This is literally like my first day ever of looking into this.
Comment: Also kind of wondering what is the difference between a good song (hit) and a bad song or an average one. Meaning, to what extent this can be explained theoretically / scientifically (lets say with the computer "equipment" that we've had, like people were able to move forward with this knowledge, I am sure, meaning analyze / compare).
Comment: Yes, absolutely. The piano keys are just a selection of pitches that evolved in the 17th-19th centuries to make music in the Western Tradition. You could select different pitches and other traditions *do*.
Comment: So what different peaches (assuming that they would be all unique) have in common as we have the various degrees of the same sounds (octaves) - like c, d, e, f, g, a h? Do they share something mathematically or physics-wise? Thanks a lot for the info, by the way. So I am assuming that a piano keyboard could also be 120 keys, and the sounds would also have repetitive names? Sounds like the division is based on what a human can differentiate... But again, different pitches but coming in the sets of 8. I think that this can be measured in some way (like more engraved in stone type of thing).
Comment: Like the range of what a human can hear is 1 to 1000 and we have these 100 sounds on a piano keyboard (lets say) and this is based on the fact that this is what a person can differentiate - like 1 and 10 sounds different to a human, but 1 and 3 sounds the same, hence the division. Kind of trying to get to the bottom of this, what this really is - like physics, mathematics. And also, how this gets translated onto the piano keyboard as this is the instrument that I want to learn (like the main one lets say, makes more sense, can use two hands and so on)
Comment: The 12 notes per octave allow you to have a Major Scale starting on any key. Other tunings do not share this property. But the importance of the Major scale goes back into earlier Greek Music and evolved under the Just Tuning system.
Comment: So this "Major Scale" is kind of something "engraved in stone", "physically" exists? This is just what it is, by nature, type of thing? Or is this somehow arbitrary too? Again, not really understanding the scales yet (but I have like 10 the best books on learning piano and music theory).
Comment: And what could be the explanation of different pitches having the same names (or related), like C1, C, c, c1, c3, c3. Do they have somethig in common, lets say physics-wise?
Comment: @user54956 all of these questions have been answered over and over again on this site. We can point you to them. Search for terms like octave, harmonic series, registers, ect. It's hard to figure where your gaps in knowledge are when it seems like you are reasking a lot of questions we already have on the site.
Answer: Oner a long period of time, Western music has been adjusted as far as note pitch is concerned, so each octave (between any note and the one in the same place to the left or right) is divided into twelve. It's called 12tet, and is a little bit of a compromise for some notes, which can sound a touch out of tune. But it now means, on piano at least, that the same tune can be played in different keys and sound pretty good. Older tunings meant tunes sounded best in one particular key - on a well-tuned instrument - but sounded pretty awful in other keys.
The layout of pianos is more for convenience. A piano could have all 88 notes spread out left to right, but only people with exceptionally long arms would be able to play it. So, it's compressed into two layers - black and white keys. It would be ludicrous to have them alternating: you wouldn't be able to pinpoint which notes are which.
It just happens that the way they are placed gives key C an easy one to find. Starting on that C, the diatonic notes are in a particular pattern: TTSTTTS. That's T=tone, S=semitone spaces between successive notes.The black keys are just as important, they're for other musical keys.Let's start at E. Using TTSTTTS, those notes are now E F# G# A B C# D#, back to the next E.
There is no direct relationship between notes, keys on a piano and frequencies. If you are mathematically/scientifically minded, you can delve into the ramifications, but I don't think it will help with understanding what's going on, except to know that every octave is split into 12 equal parts. Taking A=440Hz, and its octave being twice that at 880Hz, divide the difference by 12, there are your note frequencies! Maybe that helps - I doubt it, somehow!
Back to your (many) tunes. By playing C E G before you try to play a tune, it will centre the ears on the key of C. Search amongst the white keys, find the appropriate start note, and away you go, using probably only white keys. If you play, for example, E G# B before you start, the ears are going to be in the key of E. Now, you may well have to use up to and including four black keys.
For simplicity's sake, I've only discussed diatonic notes - which a lot of tunes use exclusively - they're the seven specific to any key. In their easiest to understand form, they're the white keys - constituting key C on piano.
You ask about any other ways into which notes can be split or ordered. Yes, but to Western ears they often sound 'out of tune'. Oriental/Indian music uses different scales, which play notes that I call 'in the cracks' - which is where they could be found on a piano. Not quite as high as one key, but lower than the next... Although this is academically interesting, it may not help your quest to play your tunes on a piano.
Comment: Thanks a lot for the info. Like I said, I should not be bothering here before learning what the existing system is. Still thinking about it though. So, if I could ask a) piano keyboard is arbitrary - correct? It does not have to be this, it could be something else. To what extent this is arbitrary to you, 0% to 100%? B) there is a range of sound a human can hear; this can be divided into various degrees of pitch that a human can differentiate (lets assume there is no differences between humans); a question could be - what is the best unit to measure this (like from Physics, I think)?
Comment: C) And then there is this repetition of pitches; they are of different pitch level (that we would want to measure as in the point B above), but they have something in common, so as an example we have these C1, C, c, c1, c2, c3 - totally different pitches but having something in common, so this is something that in fact exists (like in nature, non-arbitrary) - would this be correct? D) going further with this thinking, "from the bottom up" (I mean this is how I imagine this, it can be different) - what other layers of this could be. Let say I am interested in  differentating the actual real
Comment: aspects of this from the arbitrary aspects of this. Piano keyboard that we have today, starting from the basement, then floor 0, floor 1 - how this is mentally build? Not really this, more along the lines of a range of sound a human can hear, then differentiate - what this really is and not some historical progress of how this has evolved and why. What is REAL and not arbitrary with the sounds. The real aspects of organization of sound that can be created, but based on what actually exists (it seems like this can be bordering on something like "philosophy of music").
Comment: @user54956 - for simplicity's sake, C1, C2, C3 etc are all different pitches of the same note name. Whatever the frequency of C1, C2 is double, and C3 double that. So, yes, there is a mathematical formula, and each octave (between C1 and C2, or C2 and C3, is divisible into 12 equal parts, giving the 12 semitones of the chromatic scale. There are plenty of questions and answers here already to enlighten you about this side of music.


Snowboard boot socks

Question: I'll start practicing Snowboard and want to know what kind of socks are best. I have read on the Internet that can be a good idea to thin socks, but I'm not sure it's the right thing.
Comment: Welcome to Sports Stack Exchange! Please don't ask shopping questions on this site. These question don't have any definite answers and are of little value to others.
Answer: Look at the following link - http://www.snowboarding-master.com/snowboarding-socks.html
it's from a well known snowboarding site.
the main answer is:

For snowboarding you want to wear WOOL, SMARTWOOL or a Synthetic
  material specifically made for skiing or snowboarding...Some Brand
  Name examples of synthetic fabrics are HOLOFIBER, THERMASTAT or
  MICROFLEECE
For snowboarding, one pair of midweight socks is best... Do not wear 2
  pairs of socks unless the base pair are very thin lightweight socks...

also take a look at this post - http://answers.yahoo.com/question/index?qid=20100116060636AAkOrH2

Snowboard boot socks


Allowed memory size of (whatever) bytes exhausted - Windows PHP

Question: I have been trying to find advice on troubleshooting this issue for while but have been unable to find anything which relates to my situation.
I am running PHP 5.6.0 on a brand new Windows 2012 R2 VPS. My PHP App was originally on another 2012 VPS and ran absolutely fine with 128 memory limit set. I have simply copied over my MySQL database and code to this new VPS, and I am now getting the "Allowed memory size of (whatever) bytes exhausted" error. It doesn't matter what I set the limit to, and if I set it to -1 it just eats everything and still errors.
Literally everything I can find online regarding this error either says to up the memory limit, or use some debugging tools that are only applicable to LAMP stacks. No one seems to have had this problem using PHP on windows.
How do I go about debugging what is causing this, clearly my app hasn't changed from copying it from the old VPS, so it must be the PHP setup on this new box, but I can't for the life of me figure it out.
I have tried older PHP versions from 5.3.28 to 5.4.24. I have also checked out my code around the lines it errors at, but there is never anything relevant there, and sometimes the line is just a comment.
If I open Task Manager and then load my site, I can see "CGI / FastCGI (32bit)" memory usage shoot up really quickly until it reaches whatever the limit is then my page errors.
Any advice very welcome.
Thanks
Comment: Probably not gonna be of much help but are you sure the memory limit changes have been applied properly? Have you checked the values reported by phpinfo(); before and after you make the changes?
Comment: Yeah they have definitely been applied, it eats whatever it's set to.
Answer: There are a few windows tools that can help a lot here. One is a download, the only link I could find in short order is from CNET so I think you can trust it, it is a Microsoft program called PROCMAN (or Process Manager) ProcMan
The other tool is the Resource Monitor built into Task Manager in Server 2012, it is much improved over Server 2008 and provides a detailed look at what is happening on the server, including memory usage.


Datepicker do Bootstrap

Question: Desenvolvi uma aplicação usando o bootstrap (getbootstrap.com).
Crei um campo data  e vinculei o datepicker do próprio bootstrap.
Como fazer para alterar o tamanho do calendário? Preciso exibi-lo menor.
Answer: Abre o Console do Navegador, inspeciona o elemento, pegue as classe e as utilize em um CSS separado editando-as diretivas e inserindo a flag !important.
Answer: Use no seu CSS o seguinte:
<code>.datepicker table tr td{
   width:auto !important;
   height: auto !important;
   font-size: 11px !important;
}
</code>
Para alterar o tamanho do datepicker basta alterar o tamanho da font.

Datepicker do Bootstrap


Como fazer paginação no SQLServer 2008 R2?

Question: Testei com duas formas, mas não obtive exito.
select * from table OFFSET 10 ROWS

select top 10 * from table OFFSET 10 ROWS FETCH NEXT 5 ROWS ONLY

Curiosamente eu nunca precisei utilizar offset nessa versão do SQLServer, e agora estou tendo dificuldades com algo relativamente simples.

Erros exibidos no console:
Mensagem 102, Nível 15, Estado 1, Linha 3
   Incorrect syntax near
  'OFFSET'.
  Mensagem 153, Nível 15, Estado 2, Linha 3
  Invalid
  usage of the option NEXT in the FETCH statement.

Posso estar equivocado, mas confesso que não encontrei respostas no site que realmente respondessem a minha dúvida (o que é curioso, já que é algo relativamente simples).
Comment: Quero estar enganado, mas acredito que o LIMIT não funciona no SQLServer. Este é um comando do MYSQL, não é? Eu pelo menos utilizo TOP 10 no SQLServer, e LIMIT 10 no MYSQL.
Comment: As construções que utilizou somente funcionam na versão 2012 (em diante) do SQL Server.
Comment: Vide "Comparing performance for different SQL Server paging methods" https://www.mssqltips.com/sqlservertip/2696/comparing-performance-for-different-sql-server-paging-methods/
Answer: Há duas maneiras de fazer isso. Vamos assumir que a consulta que você irá fazer é esta:
SELECT * FROM Orders WHERE OrderDate >= '1980-01-01' ORDER BY OrderDate

Neste caso, voce deve determinar o total de resultados usando:
SELECT COUNT(*) FROM Orders WHERE OrderDate >= '1980-01-01'

...pode ser inapropriado considerando a eficiência do resultado.
Agora, para conseguir realizar uma consulta mais adequada, o seguinte modelo poderia ser o mais eficiente:
SELECT  *
FROM    ( SELECT    ROW_NUMBER() OVER ( ORDER BY OrderDate ) AS RowNum, *
          FROM      Orders
          WHERE     OrderDate >= '1980-01-01'
        ) AS RowConstrainedResult
WHERE   RowNum >= 1
    AND RowNum < 20
ORDER BY RowNum

Aqui irá retornar 1-19 linhas da query original.
Agora, só adaptar ao seu código.
Answer: Thiago, Para paginar é necessária uma ordenação (ORDER BY) dos registros para o seu caso:
SELECT * FROM table
ORDER BY 1 --Nome campo
OFFSET 10 ROWS FETCH NEXT 5 ROWS ONLY

Comment: Então na documentação da microsoft estava escrito a partir do SQL 2008 mas ao ler o documento mais a fundo, a função de Offset está disponível a partir da versão 2012 do SQL server. https://docs.microsoft.com/en-us/sql/t-sql/queries/select-order-by-clause-transact-sql#Offset
Comment: Cheguei a testar utilizando order by também, mas recebo os seguintes erros:

Mensagem 102, Nível 15, Estado 1, Linha 3
Incorrect syntax near 'OFFSET'.
Mensagem 153, Nível 15, Estado 2, Linha 3
Invalid usage of the option NEXT in the FETCH statement.
Comment: Thiago, não esqueceu do ORDER BY? pois os Offset é um parametro adicional do ORDER BY. Segue doc da microsoft com a documentação da funcção [link] (https://docs.microsoft.com/en-us/sql/t-sql/queries/select-order-by-clause-transact-sql)
Comment: Tentei sim!
A query que tentei rodar foi essa: SELECT TOP 10 * FROM users
ORDER BY id
OFFSET 10 ROWS
FETCH NEXT 5 ROWS ONLY
Comment: Continua nao funcionando!
SELECT * FROM PessoaFisica
ORDER BY pf_id
OFFSET 10 ROWS
FETCH NEXT 5 ROWS ONLY

Antes mesmo de rodar a query, já é apresentado "erro de sintaxe", as palavras sublinhadas em vermelho...
Comment: O banco está com nível de compatibilidade?
Comment: Pra ser sincero eu nunca tinha acessado a propriedade, e pra variar eu não sou o SA do banco... Mas isso implica em que?


AB testing vs testing the null hypothesis

Question: I'm trying to understand the difference between

testing the null hypothesis (i.e. testing that the probability of a "goal" is the same across 2 different populations, similar to prop.test in R)
an A/B test using a bayesian formula such as described here: http://www.evanmiller.org/bayesian-ab-testing.html

Is there a difference? Is one preferable?
The problem I'm facing looks something like this:
control group has 100,000 impressions and 100 reactions
test group has 50,000 impressions and 55 reactions
Answer:
Is there a difference?

Yes. A null hypothesis test produces a test statistic and a p-value, the probability of a test statistic as extreme as the that of the data, under the assumption that the null hypothesis is true. In your example, prop.test tests the assumption that the $p_A$ and $p_B$ are equal. This is distinct from the probability described in your link, $Pr(p_B \gt p_A)$:
On your data, prop.test produces a p-value of 0.6291; we interpret this to mean that if $p_A = p_B$, we would expect to see data this extreme in roughly 63% of experiments. But this isn't directly interpretable as the probability that the alternative outperforms the control. Using the linked post's formula, one arrives at $Pr(p_B \gt p_A) \approx 0.726$, which is directly interpretable as such. (Python code after the break.)
To gain a little intuition about this, observe the two posterior densities for $p_A, p_B$.

The mode of $p_B$ is clearly to the right of the mode of $p_A$. In other words, our point estimate for $p_B$ is higher. Expected, since $\frac{55}{50000} \gt \frac{100}{100000}$.
The posterior for $p_B$ is more dispersed. Intuitively satisfying: since we've observed A twice as many times, we're more confident in a narrower posterior.
There's still plenty of overlap—it's conceivable that the two treatments just don't meaningfully differ.

For one last intuitive aid, we can plot the distribution of the difference of the posteriors, and observe that roughly three-quarters of its area lies to the right of $0$:

To reiterate, the p-value only tells us that the data fail to reach the extremity at which we'd be convinced a difference exists.

Is one preferable?

That question is an instance of the broader Bayesian v. Frequentist choice, and often veers into matters of opinion. In general, I believe the answer depends on many factors, including application, audience, and analyst preference. Here are a few ways to view the difference between the two, which will hopefully help show when one might be preferable.
One nice introduction to Bayesian A/B testing puts it like so:

Which of these two statements is more appealing:
(1) "We rejected the null hypothesis that A=B with a p-value of 0.043."
(2) "There is an 85% chance that A has a 5% lift over B."
Bayesian modeling can answer questions like (2) directly.

For another take, theoretical statistician Larry Wasserman nicely describes the two schools of thought:

But first, I should say that Bayesian and Frequentist inference are defined by their goals not their methods.
The Goal of Frequentist Inference: Construct procedure with frequency guarantees. (For example, confidence intervals.)
The Goal of Bayesian Inference: Quantify and manipulate your degrees of beliefs. In other words, Bayesian inference is the Analysis of Beliefs.

>>> from scipy.special import betaln as lbeta
def probability_B_beats_A(a_A, b_A, a_B, b_B):
...     total = 0.0
...     for i in range(a_B):
...         total += exp(lbeta(a_A+i, b_B+b_A) - log(b_B+i) - lbeta(1+i, b_B) - lbeta(a_A, b_A))
...     return total
>>> probability_B_beats_A(101, 100001 - 100, 56, 50001 - 55)
0.72594700264280843


Permutations with chess pieces

Question: Let's say we have chess table $n^2$ and we want to put 8 rooks on the table, so that none of them are under eachother's fire.
I've come up with this:
$$n^2 + n^2(n-1)+n^2(n-1)(n-2) ...$$
I think that I'm not taking into account that the pieces are indifferentiable. That means I should substract number of permutations, but how many? and is the first equation even correct?
Answer: You can choose $8$ rows and $8$ columns in $\binom n8^2$ ways, and then there is a standard rook placement (permutation) problem left, giving a factor $8!$. So you get
$$
  \binom n8^28!=\frac{n^2(n-1)^2\ldots(n-7)^2}{8!}\quad\text{solutions.}
$$
This is also what you get from your method if you divide out the $8!$ ways to arrive at the same placement (given that the rooks are indistinguishable).
Answer: You have $8^2$ choices for the first rook, then $7^2$  for the second, $6^2$ for the third and so on. This gives $(8!)^2$ configurations, but you can permute the rooks in $8!$ ways, leaving $8!$.
Another way to see it is that you must have one rook per column. You have $8$ choices for the row of the one in the first column, then $7$ for the position in the second column, again leading to $8!$
Comment: I understand that I have $n^2$ different options to place the first rook, for the next rook I must make the table 1 row and 1 column smaller which makes it $(n-1)^2$ and so on... How exactly do you get $8^2$ choices for the first rook? Doesn't the number of permutations have to take table size into account?
Comment: I misread to think you specified an $8\times 8$ board. You are correct it is $n^2$ for the first, $(n-1)^2$ for the second
Comment: would you say that the answer is $$n^2(n-1)^2(n-2)^2(n-3)^2...$$ ?
Comment: No, you need to divide by $8!$ but otherwise OK
Comment: Are you sure that I have to divide by $8!$. I tried it out with smaller table (3x3) and 3 rooks, then the answer should be simply $3^22^21^2$
Comment: For $3\times 3$ there are only $3!=6$ arrangements of three rooks. As they are indistinguishable, you need to divide by $3!$ permutations of the rooks. The configurations are the two diagonals plus each corner and the two squares a knight's move away for six.

Permutations with chess pieces


Fees for commercial Apps published into Google Apps Marketplace

Question: while reading through the online documentation I do not understand if:

there are some fees for commercial Apps published into GAM (or CWS) or the one time 5$ would suffice;
it is mandatory to have a Google Wallet account or not.

Thank you.
Answer: Before you publish your first item, you must pay a one-time US$5.00 developer registration fee. They charge this fee in order to verify developer accounts and better protect users against fraudulent activity.
The 5$ fee is just to verify the developer account.
Check this page for more details.
I believe all the payments to Google goes through Google wallet only. However, you have to provide your card information in Google wallet while you are paying.
Hope that helps!


Find a likelihood to calculate a posterior probability

Question: I am having trouble understanding a basic Bayesian inference exercise:

Suppose we are interested in inferring the proportion $\theta$ of
individuals in a given population suffering from a certain disease. A
group of epidemiologists initially determined that this prevalence can
be:

Low $(C_1: \ \theta = 0.10)$
Medium $(C_2:\ \theta = 0.50)$
High $(C_3:\ \theta = 0.75)$

and that $\mathbb{P}(C_1) = 0.4$, $\mathbb{P}(C_2) = 0.20$ and
$\mathbb{P}(C_3) = 0.40$.
Subsequently, the epidemiologists had access to a random sample of 100
individuals from that population, in which they observed that 27 of
them had the disease. Given this information, what is the probability
that the prevalence is now low?

My attempt:
To solve the problem I tried to simply use Bayes' theorem, considering the event $D:$ the person is sick. So for me it was simply to look at it as follows:
$$\mathbb{P}[C_1 \vert D] = \dfrac{\mathbb{P}[D\vert C_1]\cdot \mathbb{P}[C_1]}{\mathbb{P}[D]} = \dfrac{\theta \cdot 0.4}{0.27} = \dfrac{0.1 \times 0.4}{0.27} \approx 0.15 $$
Therefore concluded that the probability that the prevalence of the disease is low was 0.15.
Thinking that I had got it right, I decided to take out the probabilities of medium and high prevalence respectively, but I ran into a problem when calculating the high prevalence:
$$\mathbb{P}[C_3 \vert D] = \dfrac{\mathbb{P}[D\vert C_3]\cdot \mathbb{P}[C_3]}{\mathbb{P}[D]} = \dfrac{\theta \cdot 0.4}{0.27} = \dfrac{0.75 \times 0.4}{0.27} \approx 1.11 \ !! $$
My first thought was that the wrong data I was using was to use that the probability of the disease is 0.27 and I proceeded to calculate it using the law of total probability:
$$\mathbb{P}[D] = \sum_{i=1}^{3} \mathbb{P}[D\vert C_i]\cdot \mathbb{P}[C_i] = 0.44$$
So
$$\mathbb{P}[C_1 \vert D] = \dfrac{0.1 \times 0.4}{0.44} \approx 0.09 $$
However, here I understand that I am also doing something wrong, as I am not using the information provided to me from the observed data.
Looking at some examples, I understand that to find the probability I want, it is not as simple as what I am proposing, since I need to define a prior distribution and a likelihood.
Guided by the same steps as some examples, I established that I could propose that $\theta \sim Beta(0.1,0.9)$ as a prior distribution, since $\mathbb{E}[\theta] = 0.1$ (for the low prevalence case). And, using that 27 out of 100 people in the sample are sick, I thought of proposing a binomial likelihood, using $n=100$ and $k=27$.
Making use of that, it is intuitive for me to conclude, however, I realize that by doing this procedure, at no point do I use the information that $\mathbb{P}[C_1] = 0.4$.
The idea I have is that the error would be to assume a binomial likelihood, taking into account that we have 3 different scenarios: high, medium and low prevalence. So intuitively I think in that case I should use a multinomial distribution for the likelihood, but I think that complicates things too much to be able to find a posterior distribution.
I am really confused with this exercise, to the point of doubting whether I am interpreting correctly that what the exercise asks for is to find $\mathbb{P}[C_1 \vert D]$. I would appreciate any help to be able to state the exercise correctly.
Answer: We can just apply Bayes' Theorem.
$$\mathbb{P}(\text{low}|27) = \frac{\mathbb{P}(27|\text{low}) \cdot \mathbb{P}(\text{low})}{\mathbb{P}(27)} $$$$=\frac{\binom{100}{27} (0.1)^{27} (0.9)^{73}\cdot (0.4)}{\binom{100}{27} (0.1)^{27} (0.9)^{73}\cdot (0.4) + \binom{100}{27} (0.5)^{27} (0.5)^{73}\cdot (0.2) + \binom{100}{27} (0.75)^{27} (0.25)^{73}\cdot (0.4)} $$$$= \frac{(0.1)^{27}(0.9)^{73}\cdot (0.4)}{(0.1)^{27}(0.5)^{73}\cdot (0.4) + (0.5)^{27}(0.9)^{73}\cdot (0.2) + (0.75)^{27}(0.25)^{73}\cdot (0.4)} $$$$\approx 53.7 \text{%}.$$
Answer: It's still a binomial likelihood because your data have only two possible outcomes (diseased or not diseased). Rather, it's your prior distribution on $\theta$ that is to be multinomial/categorical. The Beta distribution is commonly used with the binomial likelihood but is unrelated to this problem.
To be precise, the prior distribution on the binomial success probability parameter $\theta$ is the categorical distribution on [0.1, 0.5, 0.75] with probabilities P(C1), P(C2), P(C3).
You're also calculating the binomial probabilities wrong. It's not just the proportion of diseased in the sample, you need to use the formula:
$$ {n\choose k} \theta^k(1-\theta)^{n-k}$$
where $n=100$ and $k=27$ in your case.


Why is Bayesian Statistics becoming a more and more popular research topic?

Question: Browsing through the research area of the top 100 US News statistics program, almost all of them are heavy in Bayesian statistics. However, if I go to lower tier school, most of them are still doing classical/frequentist statistics research. For example, my current school (ranked between 150 to 200 on QS world ranking for stats so not considered a top tier school) has only one professor focusing on Bayesian stats and there is almost a resentment towards Bayesian stats. Some grad students I talked to even says that Bayesian Statisticians are doing Bayesian stats for the sake of it which I of course disagree strongly.
However, I wonder why this is the case. I'm having several educated guesses:
(a) there is not enough room left for advancements in the methodology of classical/frequenting stats and the only viable research in classical/frequentist stats research is on applications which will be the main focus of lower tier school as top tier school should be more inclined towards theoretical and methodological research.
(b) It is heavily field dependent. Certain branch of stats is simply more suitable for Bayesian stats such as many scientific application of stats method while other branch is more suitable for classical stats such as financial area. (correct me if I'm wrong) Given this, it seems to me that top tier schools have a lot of stats faculties doing applications in scientific field while lower tier schools stats department are mainly focusing applications in financial area since that helps them generate income and funding.
(c) There are huge problems with frequentist method that can't be resolved for example the prone to overfitting of MLE, etc. And Bayesian seems to provide a brilliant solutions.
(d) Computational power is here hence Bayesian computation is no longer a bottleneck as it was 30 years ago.
(e) This one may be the most opinionated guess I have. There is a resistance from classical/frequentist Statistician that just don't like a new wave of methodology that can potentially overtake the role of classical stats. But like Larry Wasserman said, it depends on what we are trying to do and everyone should keep an open mind, especially as a researcher.
Comment: @Peter Leopold Half seriously: 99%? and who is "us"? Many naive users of statistics indeed have serious misconceptions about CIs, but if you're addressing the CV community, I would hope that 99% really doesn't fit. Statistical people can be just as bad as anyone else at making up numbers in the absence of hard data!
Comment: While opinions of CV denizens will vary, and such opinionation is considered off-topic, it is worth pointing out that exactly this question is answered in introductory chapters of modern texts on Bayesian analysis. In particular, Chapter 1 of  Gelman, et al *Bayesian Data Analysis* 3rd Ed. It boils down to a) "common sense" and b) the highly problematic frequentist confidence interval that 99% of us can't help but misinterpret. The *way* we misinterpret it is intrinsically Bayesian, so we might as well do Bayesian analysis from the get-go.
Comment: @NickCox I was addressing the OP, and the "us" is meant to be empathetic and inclusive. Hopefully, "99%" is recognized as an order of magnitude estimate $\sim 10^{-2}$. It is also the larger fraction of the *have/have-not* trope in popular culture, where have-nots in this context indicate the event: "I took stats 101, but I have not fully internalized the frequentist vs. Bayesian interpretation of confidence intervals vs. credible intervals." And now that you called me out, I'll  assert (:D) that it is my official prior for that event! Naturally, I am willing to be convinced otherwise! :D
Answer: Personally, I would venture a few guesses:
(1) Bayesian statistics saw a huge uptick in popularity in the last couple decades. Part of this was due to advancements in MCMC and improvements in computational resources. Bayesian statistics went from being theoretically really nice but only applicable to toy problems to an approach that could be more universally applied. This means that several years ago, saying you worked on Bayesian statistics probably did make you a very competitive hire.
Now, I would say that Bayesian statistics is still a plus, but so is working on interesting problems without using Bayesian methods. A lack of background in Bayesian statistics would certainly be a minus to most hiring committees, but getting a PhD in statistics without sufficient training in Bayesian methods would be pretty surprising.
(2) Bayesian statisticians will mention "Bayesian" on their CV. Frequentists will usually not put "Frequentist" on their CV, but much more typically the the area they work in (i.e., survival analysis, predictive modeling, forecasting, etc.). As an example, a lot of my work is writing optimization algorithms, which I'd guess implies you would say means I do Frequentist work. I've also written a fair chunk of Bayesian algorithms, but it's certainly in the minority of my work. Bayesian statistics is on my CV, Frequentist statistics is not.
(3) To an extent, what you've said in your question also holds truth as well. Efficient general Bayesian computation has more open problems in it than the Frequentist realm. For example, Hamiltonian Monte Carlo has recently become a very exciting algorithm for generically sampling from Bayesian models. There's not a lot of room for improvement of generic optimization these days; Newton Raphson, L-BFGS and EM algorithms cover a lot of bases. If you want to improve on these methods, you generally have to specialize a lot to the problem. As such, you're more like to say "I work on high-dimensional optimization of geo-spatial models" rather than "I work on high-dimensional Maximum Likelihood Estimation". The machine learning world is a bit of an exception to that, as there's a lot of excitement in finding out new stochastic optimization methods (i.e., SGD, Adam, etc.), but that's a slightly different beast for a few reasons.
Similarly, there's work to be done on coming up with good priors for models. Frequentist methods do have an equivalent to this (coming up with good penalties, i.e., LASSO, glmnet) but there's probably more fertile ground for priors over penalties.
(4) Finally, and this is definitely more of a personal opinion, a lot of people associate Frequentist with p-values. Given the general misuse of p-values observed in other fields, lots of statisticians would love to distance themselves as far as possible from current misuses of p-values.
Comment: So the answer to why it's become more popular includes (1) it's become more popular. Puzzled by that, but I imagine it's just a matter of needing some rewording.
Comment: @NickCox: my point is that it **is** more popular, but also it's popularity may be somewhat overstated. That is, the OP saw that Bayesian Statistics was on the CV of almost every professor at a sample of top tier universities. But that doesn't mean that every one of those professors only does Bayesian statistics. Another point on (1) was that I think there *was* a time when doing Bayesian statistics as your research area was very important for getting a top-tier position. I'm not sure it's as strict of a requirement anymore, but many of the professors you see now were hired during that time.
Comment: Oh, I see your point. I was focusing on discussion of the "Is it because there is not enough room for advancement in classical/frequentist statistics?" and not so much on the "why is this happening?"


Colocar numeração sobre as curvas do gráfico (matplotlib)

Question: Tenho um código em Python para a simulação de um modelo matemático que estou estudando. Para gerar os gráficos, estou utilizando apenas a função plot() do matplotlib, dentro de um for (são diversas curvas numa mesma figura). Não convém colocar o código inteiro aqui, mas a estrutura é basicamente essa:
import numpy as np
import matplotlib.pyplot as plt

T = np.linspace(0, 100, 1000)
L=[10,20,30]
fig1=plt.figure()

for l in L:
    plt.plot(T,T+2*l, label='size= '+str(l))

plt.legend()
plt.show()

A questão é: preciso colocar sobre cada curva, a numeração referente aos dados testados, de forma similar à imagem abaixo:

Tem algum argumento na função plot() que eu possa adicionar, ou algum outro método que eu possa usar para fazer isso?
E também na legenda, ao invés de mostrar a linha com cor, mostrar o número.
Obs: Eu sei que eu posso fazer cada curva de uma cor,espessura ou desenho diferente, porém (infelizmente) para esse trabalho preciso que o gráfico tenha este formato específico, com a legenda sendo indicada pela numeração, e não por cores, por exemplo.

EDIT (problema resolvido)
Baseado no que o @darcamo fez, consegui colocar as numerações na curva.
Baseado nessa resposta consegui ocultar os handles e deixar as legendas da forma que eu queria.
O código final ficou assim:
import numpy as np
import matplotlib.pyplot as plt

T = np.linspace(0, 100, 2000)
L=[10,20,30]
fig1=plt.figure()
for n, l in enumerate(L):
    y=T+2*l
    label=str(n+1)+' => Size = '+str(l)
    plt.plot(T,y,label=label)
    y_shift=0.05*y[-1]
    plt.text(T[-200], y[-200]+y_shift, str(n+1))

leg = plt.legend(handlelength=0, handletextpad=0, fancybox=True)
for item in leg.legendHandles:
    item.set_visible(False)

plt.show()

Gerando o seguinte gráfico:
Answer: Você pode usar plt.text para colocar um texto em um local específico.
import numpy as np
import matplotlib.pyplot as plt

T = np.linspace(0, 100, 1000)
L=[10,20,30]

fig1=plt.figure()

for l in L:
    y = T+2*l
    plt.plot(T, y, label='size= '+str(l))
    y_shift = 5  # Um deslocamento pra cima para o texto ficar acima da curva
    plt.text(T[500], y[500] + y_shift, str(l))

plt.legend()
plt.show()

Quanto a legenda, se todas as curvas vão ser iguais você realmente precisa da legenda? Afinal já vai ter um texto próximo a cada curva indicando qual é a numeração da curva.
Comment: Obrigado pela resposta, vou tentar assim que possível. Mas então, as curvas não são iguais, são parecidas com as da imagem que eu coloquei. Esse código só foi pra exemplificar a estrutura que eu tava usando, não faria sentido eu colocar as equações reais que eu tô usando, pois ficaria enorme e não agregaria nada ao problema. Eu preciso da legenda sim, pois terei por exemplo, as curvas 1, 2 e 3, cada uma é reflexo de uma informação diferente, e como eu disse, eu poderia fazer isso de outras formas mais simples, mas para o trabalho que estou fazendo, preciso desse formato específico da imagem.
Comment: Me referi iguais quanto ao estilo. Quando o estilo diferencia as curvas, como com cores diferentes, você precisa da legenda para dizer o que o estilo significa (a curva vermelha corresponde a esse caso, a amarela aquele outro, etc.). Se todas tem o mesmo estilo e você estiver adicionando um número em cada curva esse número já é, de certa forma, uma legenda.
Comment: Ah sim, entendi o que você quis dizer. Mas a legenda seria como a da imagem, por exemplo 1 -> Size=100um... . Ao invés do estilo da linha, seria a numeração na legenda.
Comment: obrigado pela ajuda!


find all tr in a table element with xpath?

Question: def parse_header(table):
    ths = table.xpath('//tr/th')
    if not ths:
        ths = table.xpath('//tr[1]/td') # here is the problem, this will find tr[1]/td in all html file insted of this table

    # bala bala something elese

doc = html.fromstring(html_string)
table = doc.xpath("//div[@id='divGridData']/div[2]/table")[0]
parse_header(table)

I want to find all tr[1]/td in my table, but table.xpath("//tr[1]/td") still find all in html file. How can I find in just this element instead of all html file?

EDIT:
    content = '''

<root>

            content from table 1

                content from embeded table

</root>'''

root = etree.fromstring(content)
table_one = root.xpath('table[@id="table-one"]')
all_td_elements = table_one.xpath('//td') # so this give me too much!!!

now I do not want the embeded table content, how can I do this?
Answer: To find the elements that are sub-elements of your context node, prepend the period . operator to your XPath. So, I think the XPath you are looking for is:
.//tr[1]/td

This will select td elements that are sub-elements of the current table, not in the entire HTML file.
As an example:
from lxml import etree

content = '''

<root>

            content from table 1

            content from table 2

</root>'''

root = etree.fromstring(content)
table_one = root.xpath('table[@id="table-one"]')

# this will select all td elements in the entire XML document (so two elements)
all_td_elements = table_one.xpath('//td')

# this will just select the single sub-element because of the period
just_sub_td_elements = table_one.xpath('.//td')

Comment: I have another question, I have update my question, how can I ignore the embeded table?
Comment: I do not want to get `content from embeded table` with `table_one.xpath('//td')`
Comment: OK - well in that case your XPath needs to be more specific - something like: `table_one.xpath('table[@id="table-one"]/tr/td')`
Comment: maybe the example is too simple,  but in my situation I think `table.xpath('./tr/td|./tbody/tr/td')` is better
Comment: I dont think `all_td_elements = table_one.xpath('//td') ` will select all TD ..the context  table_one sub dom ...


When should I use Validation rather than Cross Validation

Question: I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation set as this gives a more unbiased model selection step and reduces randomness in model results due to the selection process of the validation data.
Aside from increased computational expense, are there any other drawbacks to cross validation compared to normal validation? Is it safe to say that, if computational complexity weren't an issue, one would always be better off cross validating rather than just using normal validation?
Comment: Highly relevant: https://stats.stackexchange.com/q/104713, though this question does have a validation aspect that makes it not a duplicate.
Comment: Thank you very much
Answer: Cross-validation as an alternative in case of a lack of training data is quite an understatement. Unless your sample size is very large, validation performance can vary widely among different random splits.
Cross-validation suffers less from this, since it considers the results from multitude folds. Even better would be to average over multiple runs of cross-validation, each with a different random split into $k$ folds.
What you consider ordinary validation is really just cross-validation with a single fold. Examples of where you might intentionally want to use $k=1$ over multiple folds include:

Your cannot afford $k>1$ computationally;
You have, say, millions of records and can confidently split your data randomly;
You are performing external validation and want to demonstrate that your model still performs well on data from a source never before seen by the model.

In case of the latter, your model would likely generalize better if you include data from multiple sources for training (e.g. data from different institutes, studies, or data bases). However, if you use all sources for training, you still don't have a real estimate of the performance on a new source.
Answer: Another drawback of CV (in addition to those in Frans Rodenburg's excellent answer) is when there is a dependency between samples, such as in a time series. In that case you can split into train/valid such that no training data has a dependency on any validation data. (E.g. in the case of time series, your validation data has the later timestamps.)
Comment: Good point. Rob Hyndman has a nice [blogpost](https://robjhyndman.com/hyndsight/tscv/) about time series cross-validation


SVM, Overfitting, curse of dimensionality

Question: My dataset is small (120 samples), however the number of features are large varies from (1000-200,000). Although I'm doing feature selection to pick a subset of features, it might still overfit.
My first question is, how does SVM handle overfitting, if at all.
Secondly, as I study more about overfitting in case of classification, I came to the conclusion that even datasets with small number of features can overfit. If we do not have features correlated to the class label, overfitting takes place anyways. So I'm now wondering what's the point of automatic classification if we cannot find the right features for a class label. In case of document classification, this would mean manually crafting a thesaurus of words that relate to the labels, which is very time consuming. I guess what I'm trying to say is, without hand-picking the right features it is very difficult to build a generalized model ?
Also, if the experimental results don't show that the results have low/no overfitting it becomes meaningless. Is there a way to measure it ?
Answer: In practice, the reason that SVMs tend to be resistant to over-fitting, even in cases where the number of attributes is greater than the number of observations, is that it uses regularization.  They key to avoiding over-fitting lies in careful tuning of the regularization parameter, $C$, and in the case of non-linear SVMs, careful choice of kernel and tuning of the kernel parameters.
The SVM is an approximate implementation of a bound on the generalization error, that depends on the margin (essentially the distance from the decision boundary to the nearest pattern from each class), but is independent of the dimensionality of the feature space (which is why using the kernel trick to map the data into a very high dimensional space isn't such a bad idea as it might seem).  So in principle SVMs should be highly resistant to over-fitting, but in practice this depends on the careful choice of $C$ and the kernel parameters.  Sadly, over-fitting can also occur quite easily when tuning the hyper-parameters as well, which is my main research area, see
G. C. Cawley and N. L. C. Talbot, Preventing over-fitting in model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007. (www)
and
G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (www)
Both of those papers use kernel ridge regression, rather than the SVM, but the same problem arises just as easily with SVMs (also similar bounds apply to KRR, so there isn't that much to choose between them in practice).  So in a way, SVMs don't really solve the problem of over-fitting, they just shift the problem from model fitting to model selection.
It is often a temptation to make life a bit easier for the SVM by performing some sort of feature selection first.  This generally makes matters worse, as unlike the SVM, feature selection algorithms tend to exhibit more over-fitting as the number of attributes increases.  Unless you want to know which are the informative attributes, it is usually better to skip the feature selection step and just use regularization to avoid over-fitting the data.
In short, there is no inherent problem with using an SVM (or other regularised model such as ridge regression, LARS, Lasso, elastic net etc.) on a problem with 120 observations and thousands of attributes, provided the regularisation parameters are tuned properly.
Comment: Not that I know of, the point was that the SVM has some theoretical results that suggest that they are resistant to over-fitting due to the number of attributes, but feature selection methods don't even have that.  FWIW, the monograph by Miller on "subset selection in regression" advises to use regularisation methods if predictive performance is all that is required.  There have been some open challenges on feature selection, which suggest that regularisation tends to outperform feature selection (see http://www.nipsfsc.ecs.soton.ac.uk/ and http://www.causality.inf.ethz.ch/challenge.php).
Comment: Are there any papers showing feature selection methods like Information Gain based ranking are more likely to overfit ?
Comment: As stated in the first paragraph, *SVMs tend to be resistant to over-fitting,  is that it uses regularization (parameter C)*, so by default, `svm` has this regularization parameters *C*? And I'm not sure what variables controlled by this *C*, you mean those **slack variables**?
Comment: `...feature selection algorithms tend to exhibit more over-fitting as the number of attributes increases...`. This statement does not seem correct. Due to the curse of dimensionality, if the number of attributes increases, the algorithm will tend to overfit and feature selection, for obvious reasons, will try to decrease the number of attributes, which means less overfitting.
Comment: @avocado C is not regularization parameter actually but it can be thought of a tradeoff between preferring large margin and preferring small number of misclassified data points. A small C prefers large margins but allow potentially large number of misclassified points (ignore errors/underfit the data). A large C prefers small number of misclassified examples at the expense of smaller margin (high tendency to overfit). Therefore, it is basically deciding between classifying the training data well (minimizing the empirical risk) and classifying the future examples well (generalization).
Comment: @ARAT no, feature selection algorithms are prone to over-fit the feature selection criterion, see my answer here https://stats.stackexchange.com/questions/27750/feature-selection-and-cross-validation/27751#27751 .  Essentially feature selection algorithms are optimising a feature selection criterion evaluated on a finite sample of data, and so can be minimised by exploiting the variance in that criterion due to the particular sample of data on which it is evaluated.  The more features, the more degrees of freedom are available to exploit that variance and hence over-fit.
Comment: That trade-off *is* exactly what regularisation achieves in machine learning algortihms.  The sum of slack penalties is effectively a data misfit measure and the margin maximisation term |w|^2 is equivalant to a weight-decay regularisation term that is minimising the norm of the weights.  SVM is only unusual in having the regularisation parameter associated with the data-misifit term, but the problem is identical if you divide both sides by C.
Answer: I will start with the second and last questions.
The problem of generalization is obviously important, because if the results of machine learning cannot be generalized, then they are completely useless.
The methods of ensuring generalization come from statistics. We usually assume, that data is generated from some probability distribution that originates in reality. For example if you're a male born in year 2000, then there is a probability distribution of what is your weight / height / eye colour when you reach 10, which results from available gene pool at year 2000, possible environmental factors etc. If we have lots of data, we can say something about those underlying distributions, for example that with high probability they are gaussian or multinomial. If we have accurate picture of distributions, then given height , weight and eye colour of a 10 year old kid in 2010, we can get a good approximation of the probability of the kid being male. And if the probability is close to 0 or 1 we can get a good shot at what the kids sex really is. The most important part of this whole thing is: we assume there is some probability distribution that generates both training data, test data, and the real world data we would like to use our algorithm on.
More formally, we usually try to say that if the training error is $k$ then with high probability ($\delta$) the error on some data generated from the same distribution will be less than $k + \epsilon$. There are known relations between size of the training set, epsilon and the probability of test error exceeding $k+ \epsilon$. The approach I introduced here is known as Probably Approximately Correct Learning, and is an important part of computational learning theory which deals with the problem of generalization of learning algorithms. There are also number of other factors that can lower epsilon and increase delta in those bounds, ie. complexity of hypothesis space.
Now back to SVM. If you don't use kernels, or use kernels that map into finite dimensional spaces, the so called Vapnik-Chervonenkis dimension which is a measure of hypothesis space complexity, is finite, and with that and enough training examples you can get that with high probability the error on the test set won't be much bigger than the error on training set. If you use kernels that map into infinite-dimensional feature spaces, then the Vapnik-Chervonenkis dimension is infinite as well, and what's worse the training samples alone cannot guarantee good generalization, no matter the number of them. Fortunately, the size of the margin of an SVM turn out to be a good parameter for ensuring generalization. With big margin and training set, you can guarantee that the test error won't be much bigger than training error as well.
Answer: There are at least two major sources of overfitting you might wish to consider.

Overfitting from an algorithm which has inferred too much from the available training samples. This is best guarded against empirically by using a measure of the generalisation ability of the model. Cross validation is one such popular method.
Overfitting because the underlying distribution is undersampled. Usually there is little that can be done about this unless you can gather more data or add domain knowledge about the problem to your model.

With 120 samples and a large number of features you are very likely to fall foul of 2 and may also be prone to 1.
You can do something about 1 by careful observation of the effect of model complexity on the test and training errors.
Comment: Without additional data it is very difficult to know if you are overfitting against an undersampled population. You can divide the data into three sets using a a validation set to tune the model and then a separate test set to test it. Another approach is to modify the model complexity and see how the training and test set accuracies vary. A lift in the test set error will indicate some degree of model overfitting from  the data.
Comment: But, how do we detect if we are overfitting ? If the models do pretty good in cross-validation, are they doing it because it is actually good or because of overfitting due to 1 or 2 ? Also with thousands of features, analyzing each feature by hand would be cumbersome. If it is not possible to identify overfitting, how can we trust any cross-validation result ?
Comment: Applying repeated 50/50 holdout trials will give you some feel for if the training and test sets are balanced. They probably won't be. Some models, such as KNN, will achieve perfect accuracy on the training set so are inherently overfitted in some sense. But they may still achieve acceptable performance on the test set. So as with much in machine learning, in the end it all depends on the precise specifics of the data.
Comment: @image_doctor Can you elaborate on "modify the model complexity"?


Table view combined fields, separate sort

Question: I'm looking for some assistance in setting up a table view to be able to sort merged columns separately.  At a high level I have the following view table:

First Name
Last Name
Age/Weight^

Ken
Davidson
Age: 42Weight: 200

I have all three columns sortable, except that the third column only sorts by Age, even though it has two possible sort options.  Clicking the Age/Weight will only sort by Age, there is no way that I can see to make it sortable by either Age or Weight.
I found a couple links here that don't seem overly optimistic, but I'm hoping that things have changed:

How to sort View on rewritten field? My fields are re-written to just add the internal label Age{{ age__value }} I don't need to sort by the re-write.

Edit
The more I look into this, the more I'm starting to think that the only way to do this is by providing a custom template views-view-table--custom-view-name.html.twig and would need to overwrite the one section:

        {% for key, column in header %}
          {% if column.default_classes %}
            {%
              set column_classes = [
                'views-field',
                'views-field-' ~ fields[key],
              ]
            %}
          {% endif %}

              {%- if column.wrapper_element -%}
                <{{ column.wrapper_element }}>
                  {% if column.title == 'sort by Age/Weight' %}

                  {% else %}
                    {%- if column.url -%}
                    {{ column.content }}{{ column.sort_indicator }}
                  {%- else -%}
                    {{ column.content }}{{ column.sort_indicator }}
                  {%- endif -%}
                </{{ column.wrapper_element }}>
                  {% endif %}
              {%- else -%}
                {%- if column.url -%}
                  {{ column.content }}{{ column.sort_indicator }}
                {%- else -%}
                  {{- column.content }}{{ column.sort_indicator }}
                {%- endif -%}
              {%- endif -%}

        {% endfor %}

Before I follow this path, does it seem like an optimal solution? The issue here is that I would lose the ability to display the sort indicator, since the column is still directly related to Age/Weight (which is just Age) instead of the individual columns.
Is there a global variable available in the views-view-table renderer that I would be able to check for the actual sort column?  Is the URL available for parsing?
Answer: I was able to get this working - in the most Drupal best practice way I could think of.  If this is way off, I'd love some guidance on how to do this properly, but for those looking for the same:
I added a theme preprocessor to update the table $variables with the sort/order at the highest level:
function sc_preprocess_views_view_table(&$variables) {
  $view = $variables['view'];
  $handler = $view->style_plugin;
  $variables['sorting'] = array(
    'active' => $handler->active,
    'order' => $handler->order
  );
}

The twig file was updated with:
// views-view-table--custom-view--page.html.twig
{% if column.title == 'sort by Age/Weight' %}
  age
  {% if sorting['active'] == 'field_hfs_age' %}
    {{ sc_column_sort_indicator(sorting['order']) }}
  {% endif %} /

  weight
  {% if sorting['active'] == 'field_hfs_weight' %}
    {{ sc_column_sort_indicator(sorting['order']) }}
  {% endif %}
{% else %}
  // Original code
{% endif %}

And two custom Twig Functions were created
  /**
   * Provides common functionality for building the custom column sort URL required.
   * @param $name the name of the current field
   * @param $sorting the associated array of the current sort
   */
  public function scTableSortUrl($name, $sorting) {
    $nextSort = ($sorting['active'] == $name && $sorting['order'] == 'asc' ? 'desc' : 'asc');
    return "?order={$name}&sort={$nextSort}";
  }

  /**
   * Display the table sort indicator of the current custom table.   I'm sure this exists somehwere
   * else, but I can't find it.
   * @param $initial the initial sort order [asc | desc]
   */
  public function scTableSortIndicator($initial) {
    return [
      '#theme' => 'tablesort_indicator',
      '#style' => $initial,
    ];
  }

https://www.hashbangcode.com/article/drupal-9-creating-custom-twig-functions-and-filters  for reference.


Ordenando resultado

Question: Boa noite, preciso de ajuda para fazer uma ordenação de palavras desse código.

texto = '''amar e viver, Amar é sonhar, sonhar é viver, viver é curtir, curtir é amar'''

def converte_texto(texto):
    pontuacao = ['.',',',':',';','!','?','"','(', ')']
    novo_texto = ''.join(c for c in texto if c not in pontuacao).upper()
    lista = novo_texto.split()
    return lista

def palavras(texto):
    palavras = converte_texto(texto)
    contagem = dict()

    for palavra in palavras :
        contagem[palavra] = contagem.get(palavra, 0) + 1

    return contagem

print(palavras(texto))

Esse é o resultado:
{'AMAR': 3, 'E': 1, 'VIVER': 3, 'É': 4, 'SONHAR': 2, 'CURTIR': 2}

Estou querendo que ele saia de forma ordenada decrescente, mais estou com dificuldade.
Answer: Podes utilizar o método sorted, especificando a chave a utilizar na comparação. Neste caso:
def palavras(texto):
    palavras = converte_texto(texto)
    contagem = dict()

    for palavra in palavras :
        contagem[palavra] = contagem.get(palavra, 0) + 1

    return return dict(sorted(contagem.items(), key=lambda x: x[1], reverse=True))
Output:
{'É': 4, 'AMAR': 3, 'VIVER': 3, 'SONHAR': 2, 'CURTIR': 2, 'E': 1}

Answer: Para ordenar um dicionário em Python é bem simples já que a linguagem conta com um método próprio para isso que funciona tanto com os dicionários, como listas e tuplas.
A sintaxe é sorted(object, key, reverse), onde, no seu caso, Object é o próprio dicionário, key é a forma de organização e reverse é um booleano para dizer se você quer inverter a lista ou não. Ele é opcional e deixando em branco não vai inverter a resposta. O key também é opcional e se não for informado fará uma ordenação alfabética da suas chaves.
Como o seu caso necessita de ordenação por valores vamos precisar incrementar com mais uma função: items().
Assim, é só acrescentar a seguinte linha no seu código e depois retornar essa variável mais abaixo. A função lambda aqui funciona como o ordenador e é, basicamente, uma função sem nome. Não vou explicar ela aquo, pq não é o escopo da pergunta, mas se quiser saber mais a respeito, aqui tem uma boa explicação.
ordenado = sorted(orders.items(), key=lambda x: x[1], reverse=True)
return ordenado

Note que como você quer uma ordenação do maior para o menor, eu dei o valor true para a variável reverse. No final seu código fica:
texto = '''amar e viver, Amar é sonhar, sonhar é viver, viver é curtir, curtir é amar'''

def converte_texto(texto):
  pontuacao = ['.',',',':',';','!','?','"','(', ')']
  novo_texto = ''.join(c for c in texto if c not in pontuacao).upper()
  lista = novo_texto.split()
  return lista

def palavras(texto):
  palavras = converte_texto(texto)
  contagem = dict()

  for palavra in palavras :
      contagem[palavra] = contagem.get(palavra, 0) + 1

  ordenado = sorted(contagem.items(), key=lambda x: x[1], reverse=True)

  return ordenado

print(palavras(texto))

Ordenando resultado


Print to PDF via AirPrint

Question: I'm building an app using Cordova and I used this Plug-in to print the content of an HTML page. I would try to print this to a PDF, not using AirPrint printer.
How can I do it (also by modifying .m and .h files in the Plugin Folder)?
Here's the plugin I've used: https://github.com/phonegap/phonegap-plugins/tree/master/iPhone/PrintPlugin
Thank you!
<code> var html = document.getElementById("printHTML").innerHTML;
window.plugins.printPlugin.print(
                                 html,
                                 function(result) {
                                 alert("Printing successful");
                                 },
                                 function(result) {
                                 if (!result.available){
                                 alert("Printing is not available");
                                 }
                                 else{
                                 //Localised error description
                                 alert(result.error);
                                 }
                                 });
</code>
Comment: There is not PDF printer in iOS and the plugin you provided only allows printing to AirPrinter. In native code you could access `UIKit` `UIGraphicsBeginPDFContextToData `: http://developer.apple.com/library/ios/#documentation/2ddrawing/conceptual/drawingprintingios/GeneratingPDF/GeneratingPDF.html#//apple_ref/doc/uid/TP40010156-CH10-SW1
Answer: Let me know if you find a better answer as I am looking for the same solution. In the meantime, what I have done is open up a browser in a new window (skip the plugin all together) and run the html-document through pdfcrowd.com. Then I use airprint directly from the browser.

Print to PDF via AirPrint


Is the difference of sets of a semiring in the semiring itself

Question: For a semiring $\mathcal{H} $ of sets one of the requirements is that for $A,B \in \mathcal{H}$ there exist disjoint sets $C_1, C_2,...,C_n$ such that $A\setminus B = \cup_{i=1}^n C_i.$ It is unclear if it is meant for $A\setminus B$ to be in the semiring or not. Given that the property of being closed under finite unions is not a requirement in the definition of a semiring, one may expect that the difference must not be in general in the semiring. In the case of right half open intervals in $\mathbb{R}$, i.e. $[a,b)$, the difference will be in the semiring, but I do not have an example to show an opposite case, in which the difference of sets in the semiring is not in the semiring. Can you provide a reference or a proof. Thanks.
Answer: Consider $S=\{(a,b]:a\in\mathbb{R}, b\in\mathbb{R}\}$.
$(0,3]\in S$, $(1,2]\in S$ but $(0,3]\backslash(1,2]=(0,1]\cup(2,3]\notin S$.


Send data from Asp.net core to Asp.net Mvc with angualrjs

Question: I have two projects Asp.net Core and Asp.net Mvc.
I wand to send data from the first project(Asp.net Core) to the second project(Asp.net Mvc) with angualrjs
When i run two projects ,my problem is that when I want to post Model from Asp.net Core to ActionResult in Asp.net Mvc , this ActionResult doesn't have value.
Angularjs in first project(Asp.net Core):
$rootScope.PrintCArd2 = function () {
    $scope.TblFishCar = {};
    $scope.TblFishCar.IdFish = $rootScope.TblFishCarIdFish;

    $http.post("http://localhost:61779/Report/GetFishReport", $scope.TblFishCar).then(function (result) {
        alert("Ok2");
    },
        function (e) {
            alert("warning"+e);
        });
};

ActionResult in second projcet(Asp.net Mvc):
public class ReportController : Controller
{

    [System.Web.Http.HttpPost]
    public ActionResult GetFishReport([FromBody]TblFishCar model)
    {
        //do something
        return PartialView("_ShowReport");
    }
}

angularjs calls public ActionResult GetFishReport([FromBody]TblFishCar model) but model doesn't has value.
TblFishCar:
 public partial class TblFishCar
{
    public Guid IdFish { get; set; }
    public int ShFish { get; set; }
    public string DateFish { get; set; }
}

How can i solve this problem?
Comment: Can you show your `TblFishCar model` class? Also, see this answer - https://stackoverflow.com/questions/30957248/how-to-send-post-in-angularjs-with-multiple-params/30957308#30957308
Comment: yes edited my post
Comment: Change `Guid` to `string` in the model class and remove `[FromBody]` from the method argument. You could cast the string to guid later in method body(if need be!).
Comment: i use this link , but GetFishReport doesn't has value.
Comment: Please update your post as to what change did you make? Also, `console.log($scope.TblFishCar)` to check if `IdFish` property has value before post?
Comment: this $scope.TblFishCar has value
Comment: Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/151350/discussion-between-mohammad-daliri-and-sajal).
Answer: Look at the developer tools of the browser to see the real errors.
Also you should enable cross origin requests in your MVC app:
  <system.webServer>
    <modules runAllManagedModulesForAllRequests="true"></modules>
    <httpProtocol>
      <customHeaders>
        <add name="Access-Control-Allow-Origin" value="*" />
        <add name="Access-Control-Allow-Headers" value="*" />
        <add name="Access-Control-Allow-Credentials" value="true" />
      </customHeaders>
    </httpProtocol>
    <handlers>
      <remove name="ExtensionlessUrlHandler-Integrated-4.0" />

      <remove name="TRACEVerbHandler" />
      <add name="ExtensionlessUrlHandler-Integrated-4.0" path="*." verb="*" type="System.Web.Handlers.TransferRequestHandler" preCondition="integratedMode,runtimeVersionv4.0" />
    </handlers>
  </system.webServer>


What do these entries in my SSH logs mean?

Question: I recently noticed that I have entries from an unknown IP address in my SSH logs. I performed a grep to extract all entires that didn't contain my own IP address. I was presented with this:
Jul 24 22:06:54 server1 sshd[8261]: Accepted publickey for root from xxx.xxx.xx.xxx port 39721 ssh2
Jul 25 04:06:50 server1 sshd[8233]: Accepted publickey for root from xxx.xxx.xx.xxx port 40800 ssh2
Jul 25 04:08:30 server1 sshd[8233]: Received disconnect from xxx.xxx.xx.xxx: 11: disconnected by user

I have a few questions:

Are the first two lines successfull or attempted log ins?
Is the third line a disconnect resulting from a successfull or attempted log in?
Is the four digit number in square brackets after sshd the PID?

I'm running CentOS 5 on a dedicated server. I am using OpenSSH.
Answer: First two lines are successful logins.
Third line indicates that the connection made on the second line was disconnect (notice the PID in the square brackets, in this case 8233, are the same, indicating that the SSHD process that was spawned and accepted a public key has now disconnected... the same process generated the two lines in the log).
The PID is the method you can use to track a particular session. When a connection is made to SSHD a new process is spawned with a unique PID (unique in any one instance - that PID may be reused some time later, maybe several hours or days later as all PIDs are eventually recycled). That process stays with the connection as long as the connection lives. So if you grep for a particular PID you can get a history of what happened on that connection.
Comment: You're welcome. By the way "accepted publickey" means the user connected using a private/public key pair rather than typing the root user password to connect. In other words they must have had a valid entry in the `/root/.ssh/authorized_keys` file.
Answer:
Yes, they look successful.  sshd was presented with a public key which exists in ~root/.ssh/authorized_keys.
I've never seen this message from anything other than a successfully-logged-in connection.
Yes, it'd the PID.


What's the big deal with normalization constants in Bayesian inference?

Question: I read this sentence in a book:

"... therefore this method is particularly useful for Bayesian inference since it doesn't require a normalization  constant"

The method is a computational algorithm that takes a density (possibly un-normalized) and does some stuff with it.
I've seen this sentiment elsewhere too, that in Bayesian stats, we need methods that don't require normalization constants ... but I don't understand what the big deal is??
If we have a density $f$ on $R$ but we don't know the normalization constant ... give me two seconds to type this in R:

integrate(f, -Inf, Inf)

and out pops the (inverse of the) normalization constant.
So what is the big deal? Why do we care whether or not a method needs normalization constants when it takes 2 seconds to calculate them in R?
Comment: The problem is if there are many parameters/random variables in the density. High dimensional numerical integration is a very tough computational problem. Making it faster and more reliable in various settings remains a very active area of research.
Comment: Did you really just dismiss numerical integration of arbitrary functions in high dimensional spaces as trivial
Comment: The integration you do in R is still an approximation and carries some error. If you have a method that can avoid you this, why not?
Comment: Every computational method is an approximation dude... saying "hey u can't use that approximated normalization constant in that approximative algorithm" sounds a bit strange. I would agree with your statement if we wanted to use some exact, analytical formula or something, but that's not the case. Everything we do is an approximation. Besides, R's integrate is pretty darn accurate.
Answer: The normalization constant involves an integration, and integration can be intractable in high dimensional problems.
In one dimension, you can evaluate $f$ at a set of points $X = \{x_1, x_2, \dots, x_n\}$ and use something like trapezoid rule to approximate the integral.
This set $X$ grows exponentially with increase in dimensions. eg. for two variables $x,y$, this becomes $\{(x_1,y_1), (x_1,y_2),\dots, (x_1,y_n), (x_2,y_1),\dots,(x_n,y_n)\}$.
See curse of dimensionality and numerical integration on wikipedia
Answer: To illustrate the problem, I decided to simulate the simplest problem that I encounter in statistical estimation.  The sample is $\{-6,-1,1\}$.  It is drawn from $$\frac{\sigma}{\pi[\sigma^2+(x-\mu)^2]}.$$  What is the posterior, and more importantly for me, what is the predictive distribution?
I used:
<code>rm(list=ls())
options(digits = 12)

library(stats)
library(cubature)

f<-function(x){
  result<-1/(pi**3)*(x[1]**3)/(x[1]**2+(-1-x[2])**2)/(x[1]**2+(-6-x[2])**2)/(x[1]**2+(1-x[2])**2)
  return(result)
}

result<-cubintegrate(f,c(-Inf,-Inf),c(Inf,Inf),fDim = 2,method = "hcubature")
print(result)
</code>
There was no meaningful result because the error was too large relative to the answer.  That is just three observations.  It doesn't take much at all to break down multiple integrations over an arbitrary posterior, even of low dimension.
It is a big deal because it is difficult.  It appears to look simple.  It isn't.  It is even worse when the posterior is located in a very narrow region, so small that you start having significant digit issues.  Add just a little dimensionality or a lot of data and your built-in function is going to fall apart.
Finally, and this might be a small issue depending on your goals and your specific problem, but tools like integrate or cubintegrate can be biased estimators of the constant of integration.


Detection of jquery.tinymce.js

Question: We have a "browser test page" which is used to diagnose user problems. There are fields the user can fill out, plus quite a few detections as well. The data from that page can then be emailed into our tech support. The page is at http://www.rephunter.net/test-browser.php. The current page is no longer fully functional, for the following reason:
I have just converted the website from using dojo to jquery. Part of the test includes checking for tinymce support. We used to run the "regular" tinymce library, but have switched over to the jquery version. In the previous version, which used an older version of TinyMCE, support was tested by the following:
if(typeof tinyMCE == 'undefined')

However, with the new instance of tinymce, it always comes up as undefined, even if there is a tinymce editor on the page. I have done some testing and it appears that tinyMCE is defined only after the form is submitted. For example, here is a snippet that shows how this works:

form onsubmit='Validate()'>
<textarea class="tinymce"></textarea>
<input type="submit" />
</form>

No matter what I have done in the detection program, the only way to get a valid tinyMCE object seems to be to submit the page. That is, the 'body' alert does not work; the "validate" alert does.
One workaround is to put on the page instructions for the user to look at the page and see if there is an editor rather than just a textarea. However, with our users this would be less than ideal. We really want the email, and thus I need to have a way to detect tinyMCE.
I may be able to work this out by having an onsubmit function that detects tinymce, but I would also like something that can be displayed directly to the user on first display of the page.
Is there a simple solution? Something similar to the elegance of if(typeof tinyMCE == 'undefined')
Answer: How about this:
typeof jQuery.fn.tinymce != 'undefined'

I just looked at your plugin's source and saw this in the beginning:
(function(b){var e,d,a=[],c=window;b.fn.tinymce=function(j)
                                   ^^^^^^^^^^^^

Comment: I guess in the "addon replies" pressing enter submits the post! I wanted to type more, so have to do it as a "one-liner" I guess. I also wanted to say that the function now returns true for me. But I am wondering how to test the "not enabled" case, meaning that the browser can not load it. In this case I don't think I can fake it in code, but actually have to find a browser that supports Javascript but not jQuery. Any ideas?
Comment: Good luck with that. If you want to check for browser compatibility, try Modernizr. It's a good tool to use.

Detection of jquery.tinymce.js


URL of image not saved into database

Question: I am trying to add an image to profile in my application. The code returns the correct URL and also the image is stored into the specified folder. But the URL is not saved into the databaseenter image description here
Answer: I see two things :
1 - You move your picture with public_path() and you set the $url with URL::to() which is not really logic. Try this way :
$url = public_path() . '/profile_photo/'.$file->getClientOriginalName();

2 - At the end of your if statement, your set an exit() method, which exit from your method. Take it off !
EDIT ----
To save it to your db, do this way :
$user = User::find($id);
$user->profile_photo = $url;
$user->save();

Comment: Thank you ! But still the problem is not solved. The url is not saves to database
Comment: i've added the save part to my answer
Comment: Thank you for this. It works. But why don't you use update()
Comment: save acts like an update if you provide the model ID and acts like an insert if not. That's why i always use `save()`
Comment: How I delete a file from folder when I upload a new file in laravel 5.4

URL of image not saved into database


How DO I store/retrieve data and display them in templates?

Question: Coming from THE MVC pattern of Django, I am unable to grasp the concept in Drupal 8. I know Twig is used for templating, but what about storing/retrieving data in Drupal 8 and displaying them in templates? Suppose I have a content type (Activity) with fields like Image, Body, Files, etc. Someone can create multiple activities with a WYSIWYG editor. How do I retrieve all these nodes and display them in a page applying my own styles (not using views). Django Views are suitable for this. Can I do this without developing a module?
Answer: One way to achieve this is using preprocess function. In your theme's '.theme' file, write a preprocess function. Example:
In bartik.theme
function bartik_preprocess_html__node__28(&$vars) {

$nids = \Drupal::entityQuery('node')
        ->condition('type','article','=')
        ->condition('status',1,'=')
        -> execute();
$nodes = \Drupal::entityTypeManager()->getStorage('node')->loadMultiple($nids);
foreach ($nodes as $key => $value) {
$title = $value->title->value;

}
$vars['title'] = $title;

}

(node__28 is any node page where you want to see all results, I made this basic page)
Now in html--node--28.html.twig template, retrieve title of content type = article
{{ title }}

Comment: Any other way rather than preprocess method?
Comment: You can use views module - simple and easy way
Comment: Views are rather simple in a sense that it only supports fewer styling types like grid, list, etc. In django, I could directly create query into database, combine the data into json, and render the data to templates. How do I do so in Drupal 8?
Comment: You can use templates as per your style. You will get theme suggestions for building templates for views. Else if you want to enhance more styling options in views, then views' hooks are available to enhance the functionality. You can alter these styling types and enhance too by using Drupal's hook functions.
Answer: I would highly recommend using views to build simple lists of things where "list" means a list, table, slideshow, or map. It is more maintainable for site builders. This is how you would accomplish this task without building a module. In other words, use views.
However, here's the custom module route if you want to do it all with code rather than out-of-the-box tools.
Drupal uses a presentation-abstraction-control (PAC) framework, which is similar to model-view-controller (MVC). In Drupal 8 this effectively means you can create a route and route controller to do business logic and then return a render array for the presentation/theming layer to consume.

Implement a route and controller in your custom module. You can inject the entity.query service into your controller a few different ways. Then load the entities that you wish via the Entity Query API (similar to an ORM query builder). The controller could also work similar to an entity list builder.
Assemble a render array as the return value of the controller. You may want to use render elements or theme functions such as container or table or item_list, but you can implement your own templates with custom variables to use therein.


Is $\mathcal C^1([a,b],\mathbb R)$ endowed with the $C^1$-norm a Hilbert space?

Question: Let $\mathcal C^1([a,b],\mathbb R)$ be endowed with the $C^1$-norm $\|\cdot\|$ where $\|f\| = \| f \|_\infty + \| f' \|_\infty$ for $f \in \mathcal C^1([a,b],\mathbb R)$.
In class, we have proved that $\langle \mathcal C^1([a,b],\mathbb R), \|\cdot\| \rangle$ is a Banach space. I could not figure out if $\langle \mathcal C^1([a,b],\mathbb R), \|\cdot\| \rangle$ is a Hilbert space or not. In other words, I'm not sure if $\|\cdot\|$ is induced by some inner product.
Could you please shed me some light on this issue? Thank you for your help!
Comment: Check the parallelogram identity.
Comment: Also, you can check that the minimum principle fails in $C^1$, while it holds in every Hilbert space. That is, there exist non-empty closed convex sets in $C^1$ which have no vector of minimal norm (in contrast to Hilbert spaces), and/or have more than one such.
Comment: Hint: there surely are.
Comment: Check for example $f(x) = x$ and $g(x)=1$ on $[0,1]$.
Comment: Hi @amsmath, I've tried to use the parallelogram identity, but I don't know if there exist functions $f,g$ that violate the parallelogram identity.
Answer: On $[a,b]=[0,1]$ set $f(x) = x$ and $g(x) = 1$. Then
$$
\|f-g\|^2+\|f+g\|^2 = (\|x-1\|_\infty + 1)^2 + (\|x+1\|_\infty + 1)^2 = 4+9=13.
$$
But
$$
2\|f\|^2+2\|g\|^2 = 2(1+1)^2 + 2 = 10.
$$
Comment: Thank you so much for your help!, I just did the calculation and cross-check it with your answer :))


How can I get rid of the "Info:" prefix on my sl4j/logback NetBeans console?

Question: This "should" be trivial. My NetBeans 8.2 Payara(GlassFish) Output log prepends EVERY line with "Info: ".. no matter the logging level. How do you get rid of this?
My logback.xml:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <statusListener class="ch.qos.logback.core.status.NopStatusListener" />
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <Pattern>%d{ss.SSS} %-5level %-28logger{5} %msg%n</Pattern>
            <outputPatternAsHeader>true</outputPatternAsHeader>
        </encoder>
    </appender>
    <root level="debug">
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

Sample output:
Info:   NeuroCraft was successfully deployed in 767 milliseconds.
Info:   06.251 INFO  c.o.n.w.WebsocketRouter      Connection opened: 3972aac1-e72a-45f2-8fa6-2b8d91141d75
Info:   08.379 INFO  c.o.n.m.NetworkThreadRunner  Run started.
Info:   08.381 INFO  c.o.n.c.NetworkController    Calibrated.
Info:   10.059 INFO  c.o.n.m.NetworkThreadRunner  Walk started.
Info:   10.524 INFO  c.o.n.c.NetworkController    Calibrated.

Why does every line start with "Info:   "????
Oh, and my pom.xml fragment:
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.25</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.2.2</version>
</dependency>

Thanks in advance...
Answer: Don't log to CONSOLE, log to a file instead.
When your code is writing a log entry, Logback will build the log message starting with 06.251 INFO ..., then write that message to "console". The app server (GlassFish) is intercepting that output and logging it to the application log (file and/or real console) as an INFO-level log message.
You're basically sending out from one logging framework (Logback) into another logging framework (the one used by GlassFish).
Comment: Well. That makes perfect sense. Thanks! i guess it's worth leaving it as is just for the convenience of testing in NetBeans.


Theming translation links

Question: I need to change translation links attributes, which have been defined in translation_node_view function in these lines:
<code>$node->content['links']['translation'] = array(
  '#theme' => 'links__node__translation',
  '#links' => $links,
  '#attributes' => array('class' => array('links', 'inline')),
);
</code>
I'd like to override these attributes in my custom module.
Any help would be appreciated.
Answer: I found a solution by myself. May be useful for others I think:
<code>/**
 * Implements hook_node_view_alter()
 */
function MYMODULE_node_view_alter(&$build) {
    if (isset($build['links']['translation'])){
        $build['links'] = array(
            'translation' => array(
                '#theme' => 'links__node__translation',
                '#links' => $build['links']['translation']['#links'],
                '#attributes' => array('id' => 'translation_links')
            )
        );
    }
}
</code>
* Now my links have their own ID, so I can customize them as I need with css
* If you need to custom only these links, I suggest to use MYTHEME_node_view_alter()

Theming translation links


Aumentar tamanho da imagem

Question: Bom, eu tenho este código que após prencher 1 formulário fazendo upload com uma imagem, esse código coloca a imagem que foi feita o upload numa pasta.
Código:
<?php
error_reporting(0);
include("config.php");
$idcarro = $_POST["idcarro"];
    if(isset($_POST['upload'])){

        //INFO IMAGEM
        $file       = $_FILES['img'];
        $numFile    = count(array_filter($file['name']));

        //PASTA
        $folder     = 'imgcarros';

        //REQUISITOS
        $permite    = array('image/jpeg', 'image/png');
        $maxSize    = 1024 * 1024 * 5;

        //MENSAGENS
        $msg        = array();
        $errorMsg   = array(
            1 => 'O arquivo no upload é maior do que o limite definido em upload_max_filesize no php.ini.',
            2 => 'O arquivo ultrapassa o limite de tamanho em MAX_FILE_SIZE que foi especificado no formulário HTML',
            3 => 'o upload do arquivo foi feito parcialmente',
            4 => 'Não foi feito o upload do arquivo'
        );

        if($numFile <= 0){
            echo 'Selecione uma Imagem!';
        }else{
            for($i = 0; $i < $numFile; $i++){
                $name   = $file['name'][$i];
                $type   = $file['type'][$i];
                $size   = $file['size'][$i];
                $error  = $file['error'][$i];
                $tmp    = $file['tmp_name'][$i];

                $extensao = @end(explode('.', $name));
                $novoNome = rand().".$extensao";

                if($error != 0){
                    $msg[] = "$name : ".$errorMsg[$error];
                }else if(!in_array($type, $permite)){
                    $msg[] = "$name : Erro imagem não suportada!";
                }else if($size > $maxSize){
                    $msg[] = "$name : Erro imagem ultrapassa o limite de 5MB";
                }else{

                    if(move_uploaded_file($tmp, $folder.'/'.$novoNome)){
                        $sql = mysqli_query($link, "INSERT INTO imgcarros (idcarro, img) VALUES ('$idcarro', '$folder/$novoNome')");
                        $verifica = mysqli_query($link, "SELECT * FROM carros where id='$idcarro'");
                        $array = mysqli_fetch_array($verifica);

?>

<?php

                    }else{
                        $msg[] = "$name : Desculpe! Ocorreu um erro...";

                }

                foreach($msg as $pop){
                    echo $pop.'
';
            }
        }
    }
        }
    }

Porém eu quero fazer upload de uma foto com as dimensões: 2380x1422 pixeis, mas não consigo, porem se eu fizer upload de uma foto com os tamanhos 264*261 pixeis já consigo.
Como posso fazer para corrigir isso, para colocar tamanhos máximos de 4000x4000 pixeis?
Obrigado
Comment: No seu código não tem nada que limite a resolução da imagem, logo o problema de não conseguir enviar imagens com alta resolução não é em outro lugar. Talvez no tamanho do arquivo, verificou se tem 5Mb ou menos? Verifique o limite de upload do PHP no arquivo php.ini e no seu servidor web. Não sei quanto ao Apache, mas o Nginx tem um limite de upload também e você pode aumentar / diminuir no arquivo de conf.
Comment: `$maxSize    = 1024 * 1024 * 5;` Experimentou aumentar isso? Aparentemente é o que tem no seu código limitando tamanho do arquivo. Agora, pra que permitir upload de 4000x4000? Tem certeza que é isso mesmo que você precisa?
Comment: Problema Resolvido!
Answer: O código que apresentou não valida as dimensões (width, height).
Provavelmente o problema pode estar na validação do peso ou to mime type.
Limite de peso para upload, no PHP
Nas configurações do PHP, defina o valor na diretiva upload_max_filesize.
Caso não saiba quanto está definido, apenas faça isso:
echo ini_get('upload_max_filesize ');

Retornará o limite em bytes.
Input hidden MAX_FILE_SIZE
Verifique também se no formulário HTML contém MAX_FILE_SIZE. Pois possui prioridade sobre o upload_max_filesize do PHP, caso seja menor. Note que browsers mais modernos ignoram esse parâmetro. Por precaução, para browsers mais antigos, é bom especificar. Por fim, a escolha depende do público alvo.
Limite de tamanho no código
No código que postou há também um limite específico definido na variável $maxSize. O limite é de 5mb.
Aumente o limite caso seja necessário.
Mime type e variações de tipos JPG.
Outro ponto que pode estar impedindo de fazer um upload é os tipos de arquivos permitidos, definidos na variável $permite
$permite = array('image/jpeg', 'image/png');

Os tipos JPG possuem variações. É recomendável adicionar também tais variações:
$permite = array(
    'image/jpeg',
    'image/jpg',
    'image/pjpg',
    'image/pjpeg',
    'image/png'
);

Comment: votei positivo, mas tem um detalhe aqui: "Note que browsers mais modernos ignoram esse parâmetro." na verdade esse parâmetro independe do browser. quem cuida do MAX_FILE_SIZE também é o PHP. A função desse parâmetro é cortar o envio mais rapidamente, sem ter que aceitar o upload até "estourar" o espaço. Para evitar do lado do cliente, você precisa de JS e da api de arquivos.
Comment: Obrigado, o problema não estava no tamanho da imagem mas sim no tamanho do arquivo.

Aumentar tamanho da imagem


Multilevel multinominal logistic regression

Question: For the analysis of repeated measurements data by developing a multilevel multinominal logistic model, in what format should the data set be prepared? What about any other technical pearls?
Answer: In practice it will depend on which software you use to fit the model, but generally data will be in long format, with one row for every subject/unit