Can absolute pitch be learned or acquired by anyone?

Question: As a companion to questions about absolute (or perfect) pitch, discussions about if the trait is achievable through training are sure to follow. After these discussions I have read up a bit on the subject, and changed my views somewhat. I thought it would be interesting to have a dedicated discussion, instead of it being spread out in comments.
So, the question is: Can any adult (barring pathological conditions) achieve perfect pitch through training?
(I have searched the existing questions, but couldn't find one with this specific meaning. That pitch perception can be improved is quite clear, the question here is if absolute pitch can be fully obtained in adulthood.)
Comment: If you categorise fluent AP by the definition that it is only possible if acquired from a young age, then there's a danger any contradiction to this will be ignored, by assuming the subject must have had it all along. Until somebody figures out how to learn and hopefully teach this skill (which could be never), we're quite reasonably stuck with believing it is something like: 1. Only possible to acquire when young, and any child could learn it. or 2. Only possible if born with it, regardless if you realise it as a child... or some other theory about the subject having to be special.
Comment: "Can any adult achieve perfect pitch through training?" - Who can say no? Are you wondering if it's not possible for any adult at all? Isn't AP just RP but using memory as the reference?
Comment: There are AP tests online.  If one played enough, then I'm sure their results will improve.  I have learned many tunes by ear, but I definitely do not have AP or even RP, since I imagine there would be much less trial and error involved if I had.
Comment: @LeeKowalkowski: I'm also sure results will improve if the ability is exercised. The question is if full AP can be achieved.
Comment: www.audiocheck.net/blindtests_abspitch.php - I got 10/10 after listening to the c4 test file, refresh the page for a new test.  Without listening to the test files, without revisiting answers, it was 5/10, I'm way off having AP.  A faster paced game would be better for learning.
Comment: @LeeKowalkowski: If it is yes, then documented cases would exist and the question has an answer. As it is now, things are not clear, and I would like to know more. Hence this question.
Comment: I found a woman on this thread that claims to have acquired it later in life: http://www.classicalguitardelcamp.com/viewtopic.php?f=1&t=47885&start=0 - There could of course be a viewpoint that anybody who acquired this skill in later life must have always had the ability to learn it. How can there be a documented case of this phenomena if we don't know how to acquire it, we don't have a proven way of teaching the skill, so how can this be scientifically proven?
Comment: @LeeKowalkowski: It just "happened" for her, so likely it was there from the start. There are some criteria for it, apparently, and since there are courses there should be at least one person who did not have it before, took the course and over time was able to pass the criteria. A lot of people actually claiming they had obtained it through training would also be an indication of it. I do not demand 100% irrefutable scientific proofs, but any indications of people actually achieving it, rather than baseless claims that it can be achieved.
Comment: The Wikipedia article does say non-AP people can learn pseudo-AP, and basically fake-it. Although not the same thing, not as efficient, and probably not as impressive, the result is the same, no? Are you interested because you are interested in trying to acquire the skill?
Comment: @LeeKowalkowski: No, I am interested since I believe one can _not_ achieve this trait through training in adult age. "Faking it" as you describe I believe is possible, but the effortless, automatic recognition of pitches which I attribute to AP is something achieved at a young age, but then the opportunity is passed. It is this belief I want challenged or confirmed.
Answer: I achieved a degree of absolute pitch (AP) as an adult, but of course this was not done in a controlled manner, so the scientific reliability of my case is implicitly questionable.
Anyway, here's what happened:
As a student who specialized in music in high school, and then pursued a Bachelor of Music degree in college, my path follows a pretty standard one in terms of trained classical musicians (who are not identified as "prodigies" VERY early in life).
Being around people who did posses AP meant that I was able to self-define as not possessing it, since I was incapable of doing the same kinds of mental-aural recall and identification that they were. My general "aural skills" were still very good, as any of my teachers will tell you; my aural skills professor asked me "Do you have perfect pitch?" in one of the first few weeks of class. It is common for a well-trained sense of relative pitch (RP) to be mistaken for AP when you meet a younger student whose skills seem to be beyond the level of RP training he/she is expected to have undergone. At high levels of musical training and skill, a musician that possesses AP is not capable of much more than a similarly-skillful musician that only possesses RP. Music theory is based almost entirely upon RP phenomena, so all musicians must learn it.
Anyway, about 1-2 years into my undergraduate degree, I started getting really interested in aural skills in general as well as the AP phenomenon, and started doing some hardcore middle-of-the-night "internet research" on the topic. There appeared to be some anecdotal evidence that AP could be learned, but at the same time most of the "training methods" being advertised looked like snake oil salesmen.
In the end, it turned out that there was a significant mental block I had to overcome simply to the fact that B and C (or any other two notes) sound different to one another. After that, there's a lot of almost meditative work to do to listen for those differences, while connecting back to repertoire that you know and can accurately recall; and finally comes the unending process of designing and studying aural skills exercises that specifically target your learning needs with increasing difficulty. That's the basic 3-step process in my experience.
Right now, my AP skill is at the point where I can produce or recognize any given pitch out of context with a very high degree of accuracy, but with non-instantaneous response times. When I am incorrect, I'm never off by more than a semitone. In these ways, I draw a lot of parallels between training AP and training RP -- in the sense that a student makes the same kinds of mistakes at lower levels of facility. Metaphorically speaking, it's as if the aural definition I can perceive of pitch is very sharp for some notes and sometimes blurry for others, but given enough time I can "figure it out" in my head for any note. Timbral differences throw in another variable that doesn't apply as much to RP training, at least for me. I have found all of this to improve with continued study.
For these reasons, I feel that "perfect pitch" (PP) is a useful term to use as distinct from AP--though colloquially they are seen as synonyms, I like to use AP as we do in this post, but use PP to refer to the "hyper-skillful" sense of AP that we see in people who picked it up intuitively at a very young age (honed by copious amounts of formal musical training).
In conclusion, I believe that the similarities I have experienced between AP and RP in my own musical (self-)education indicate that it is possible for anyone to achieve AP or even PP through training. There is of course an understanding that people are predisposed to musical skill based on their exposure to music during their upbringing, but this impacts RP awareness and training just the same as it does AP--and no one has ever said, "Oh, you'll never be able to play like Coltrane no matter how hard you work at it, because you're not genetically predisposed." There appear to be some societal preconceived notions that you can achieve any level of skill through dedicated training and hard work, but for some reason PP is an exception that "you must be born with". I see no reason for this to be anything but that; a preconceived notion. This is the closest one I can find to it at the moment, but I am aware of a study that shows that people without musical training, when singing their favorite pop songs a cappella, more often than not, are able to do so in the correct key. This indicates that most people have some intuitive pitch memory even if they don't realize it, which is one of the primary building blocks of AP training.
I haven't been engaged in training my own AP for a couple of years since I've been out of school, but I have some long-term goals of developing software and training methods for this, coupled of course with further honing of my own skills in a more controlled context. Consider this post a prelude to more scientific findings I hope to come up with in the future!
Comment: This is great stuff. I think that to differentiate between absolute pitch and the prodigal version of it is useful. I see it as hard to achieve a level where different tones are described to be different like different colors through training, but what your are doing is slightly different. There is a societal notion that one can achieve any level of skill through hard work, but I think it is wrong. There must be a reason why the Coltranes, Davises, Mozarts and so forth are so very rare.
Comment: I'd suggest the reason for prodigious rarity is that not a lot of people are willing to put in the work! And it's worth noting that the people you mention as being truly anomalous both attained a high level of skill _and_ caused an artistic paradigm shift. In his most influential years, many trumpeters could outplay Miles, but that's not what he was known for. :-) I know many jazz and classical musicians who became what they are because they _put in the work_ in high school and college, regardless of PP possession. (My comment on that was in reference to seeing AP acquisition as an exception.)
Comment: I am not as naive to think that anyone at a high level has not put in a _lot_ of work. But the drive to put in the work has to come from somewhere too... I do not believe that anyone can play in a symphonic orchestra at professional level due to psychological and physiological (hence in part genetic) reasons. Anyone can improve their musical abilities and get great pleasure from it though. I think it is important to try to estimate the realistic goals before doing anything. The "anything is possible" mindset can lead to a lot of disappointments and failed attempts.
Comment: Good answer, good content, and I agree with a lot of your thoughts.  Hearing is very underdeveloped in the US, and it is not surprising that there are no standardized techniques for AP as there are for RP.
Comment: Hi, I'm reading it 3 years later. Have you made any new discoveries or a training plan to acquire perfect pitch by adults?
Comment: @BartoszWójtowicz Sorry, nothing new to report. Most of my work at present is not in the field of music, so that's not a negative result.
Comment: I think that when I was about 30, I gained absolute pitch to some extent. It just came. Quite often, I'm able to play a song in my head that I heard before less than a semitone off. I'm not always able to do it. I guess my brain was sluggishly improving with time with no conscious effort and then when I was about 30, it had improved so much that when I started watching the second season of "The Worst Witch" at that time after just a few episodes, I had the ability to reliably play the song in the introduction at the correct pitch. Some people seem to think that those who didn't aquire it
Comment: during childhood don't gain it later. Maybe it is true that it's a lot more common to aquire it in childhood than adulthood. Then the ones that aquired it in childhood did it naturally. However, if somebody does not have the type of brain where they were going to aquire it in childhood, it may actually be less help to aquire it trying to make them hurry up and gain it in childhood than to give them as much time as they want to aquire it. I don't think you're going to be able to get researchers to agree to invest in research just on how to train absolute pitch because the research would be too
Comment: inefficient and investing in research on that would slow down other research so much. Maybe researchers will develop a very complex technique for relating so many problems with so many other problems and forming a proper organized plan that prevents problems in the first place and taking over all research and conducting it very efficiently and that might include giving people a way to train their absolute pitch.
Answer: Before the discussions here, I thought that absolute pitch (AP from now on) was a given from birth to some lucky (or unlucky in some cases) individuals. From discussions and reading up on the subject, my understanding now is that it is related to early exposure to pitches and meaning attached to them. This could be childhood musical training, or languages like Chinese, where pitch determines the meaning of some words.
Just like children up to a certain age are very receptive to langauage, it seems as at a certain stage in development the brain is more aware of pitch. This ability is then lost, as other functions take over. (This is simplified, but so is my understanding.) Just like a child can learn several languages at an early age relatively effortless, but will struggle more without achieving e.g. the same pronunciation later in life, AP reached through training at an early age will outdo training as an adult.
The Wikipedia article states that no cases of AP achieved by adults have been documented.
As I understood the subject, the answer to the question is no, AP can not be achieved by training as an adult. It can be improved, but not reach the levels of those that "always" had the ability.
There is some discussion about this in the fascinating Musicophilia book by Oliver Sacks.
Comment: There is recent research that indicates that even those that self-identify as having the ability to perceive absolute pitch actually don't. http://news.uchicago.edu/article/2013/06/11/perfect-pitch-may-not-be-absolute-after-all
Comment: @RolandBouman: Very interesting. Would be interesting to do the same tests with people having synethesia, e.g. seeing different colors for different keys.
Answer: At the present level of science (can be summarised here, for instance), there is no obvious answer to this question.
Human ear internally contains a huge number of stereocilia of variable length. They resonate just like strings (or maybe more like rods as seem attached from only one side) in response to any sound, and the brain receives information which of the thousands of existing strings resonate, and how much. Stereocilia are arranged into staircase pattern that is also very important; if this pattern does not develop for some reason, the ear is deaf.
The picture below shows mice stereocilia, human stereocilia are similar:

These strings are "tuned" to various absolute, not relative frequencies inside the hear-able sound range, and the tuning range is narrow, just like for the normal strings. Hence the system seems providing all necessary information to tell the absolute frequency of the sound.
This anatomy is more or less the same for all humans, so it is really mystery why some are capable of absolute pitch and others not. The inner ear not just passively sends signals like a mic, but seems also receiving back some commands from the brain.
As a conclusion, you probably can always try to train and rely on your luck. With such a degree of scientific understanding, it may be fully possible that some people have the ability to train this skill while others do not. However do not forget that absolute pitch is not required to enjoy both listening and playing music and may be more sense to spend that time on the usual work with your instrument.
Comment: I'm not (consciously) working on my relative pitch, let alone absolute pitch, so I'm not interested in it due to that. I just see it as as such foreign concept for me personally, that I have a hard time seeing how it could be achieved by training. (Probably says something about my general musicality...) Science has no clear answer yet, but the lack of any documented case compared with the effort spent, says something.
Answer: I've once listened to an interview of someone with AP. The one thing that stuck to me was that he was hearing notes the way we see colors. I.e. he can identify notes by hearing them, but he has a hard time to distinguish intervals. Can you tell me how big the difference between blue and green is? Well, that's how he perceived notes. And the comparison is pretty much on spot, since light as well as sounds are just waves with a certain frequency.
So I hardly doubt anyone who's born without it can learn to percieve notes like someone born with AP. But with a lot of training, "normal" people can certainly become pretty good at identifying notes.
Answer: I can only answer from a completely personal point of view. For the last couple of years, I sing a C note as I walk past a piano, then check it. Just done it now, almost spot on.So, somehow, I think it can be trained. However, that's one note. If I hear a note, I have to imagine my C against it, to identify the note.Not sure if this qualifies.
Comment: I appreciate any input, but I don't think it qualifies. Absolute pitch as I understand it is the ability to identify all pitch classes. Someone with AP should be able to identify any tone from the piano immediately.
Comment: That's why it was worded that way.But it may encourage others.
Comment: And that's very good, since this ability can be improved with exercise. I'm interested if the goal of absolute pitch can be achieved. It seems many believe this, but I haven't heard of a case where it happened.
Answer: I have AP and I perceive notes in different regions of the brain.
In addition, different notes have their extra resonance in my ears.
( They enter into my ear in a different angles, that is the best way to describe it. It is a 3D experience for me. )
If you are able to notice these differences in tones, then you may be able to learn AP.
If you can not notice these differences, then maybe you won't learn it.
I will not notice it unless I want to. The tones are all equal to me.
The sensation is like wearing a shirt; you will not notice the shirt unless you turn your attention to it or when it feels uncomfortable.
It's just something that is there.
Answer: I read somewhere (scientific American I think) that the scientists differentiate between people who have perfect pitch, who are born with it, or "always have it" and people who have learned it, which is always by means of a "reference pitch". It's like they can remember what c sounds like and then when they hear a they start at c and go down step by step. People with "true" perfect pitch apparently just "know".
Answer: I was a bass drummer in a bagpipe band for 30 years and have heard the bagpipe note A (actually played Bb) a million times and can whistle it any time I want. The only problem is that not every band tune their bagpipes to the same Bb. So I am within the range of a semitone (up or down) all the time. I also learned  to play the violin as an adult and can recognize about 10 cents when a note is wrong. The point here is that my brain has been trained to recognize one note. I believe it would be possible to achieve some degree of AP but it would take a tremendous amoumt of work to burn it into the brain of an adult.
Answer: I'm autistic and I got a book prize for showing certain mathematical abilities in some of my courses during my graduation of undergraduate school. Once when I was maybe about 21, I was planning to find a way to train myself to have absolute pitch and never figured out a way but I no longer care because around when I was about 30, the ability to frequently play a song I head before in my head at a pitch that's less than a semitone off just came. I'm not always able to do it and I can't all the time readily correctly sense what note I'm hearing. I think my brain was very sluggishly heading towards gaining that ability because the brain makes that sluggish progress with no conscious effort and even if you do not one speck have absolute pitch, that doesn't mean your brain isn't closer to gaining it than it was in the past. There's probably a hidden factor in your brain which indeed makes you closer to gaining that ability than you were in the past.
Answer: A guy named David Burge (Not, I think the deceased American concert pianist of a similar name) has been peddling his 'Perfect Pitch' training course for many years.
It's interesting to compare the advertising in my 1980s editions of 'Keyboard' magazine with the current web adverts.   Like 'Mavis Beacon teaches typing' and the 'Linguaphone' language courses I suspect the packaging changes but the content stays much the same!
No-one has found it necessary exposed Burge as a charlatan.  His courses may not give everyone perfect pitch, but they will certainly help develop relative pitch, arguably a more useful skill.
If I blow my nose robustly, it sounds a pretty consistent B♭.  This has actually been useful at times.


Erro em galeria de imagens, botão voltar aleatorio

Question: Olá, tenho um js que faz uma lista de imagens avançarem e retrocederem com base em uma variável, ela avança perfeitamente, porém para voltar ela volta aleatoriamente. Alguém pode me ajudar a resolver? o código está abaixo:
var imgs = [
  'produtos/Slide92.png',
  'produtos/Slide93.png',
  'produtos/Slide94.png',

];
var num_imgs = 1;
var next_three = num_imgs;
var prev_three = 0;
var next_imgs, prev_imgs;
for(var i = 0; i < next_three; i++) {
    $('#galery').append('');
}
$('#next').on('click', function() {
  next_imgs = imgs.slice(next_three, next_three + num_imgs);
  if(next_imgs.length === 0) {
    next_three = 0;
    next_imgs = imgs.slice(next_three, next_three + num_imgs);
  }
  $('.imgDinamica').each(function() {
    if(typeof next_imgs[$(this).index()] === 'undefined') {
        $(this).hide();
        return true; // continue
    }
    $(this).show();
    $(this).prop('src', next_imgs[$(this).index()]);
  });
  next_three += num_imgs;
});

$('#prev').on('click', function() {
    prev_imgs = imgs.slice(prev_three - num_imgs, prev_three);
  console.log(prev_three);
  if(prev_imgs.length <= 0) {
        prev_three = imgs.length;
    prev_imgs = imgs.slice(prev_three - num_imgs, prev_three);
  }
  $('.imgDinamica').each(function() {
    if(typeof prev_imgs[$(this).index()] === 'undefined') {
        $(this).hide();
        return true; // continue
    }
    $(this).show();
    $(this).prop('src', prev_imgs[$(this).index()]);
  });
  prev_three -= num_imgs;
});

O código original é esse: https://jsfiddle.net/bwoep3wL/2/
Answer:
#next.click não atualiza prev_three nem #prev.click atualiza next_three;
prev_three não esta a ser corretamente inicializado:
var prev_three=imgs.length-imgs.length%num_imgs;
algoritmo no geral.

Sugiro a utilização de uma única variável de posição, modificada de acordo com o movimento:
var imgs = [...];

var num_imgs = 3;
var cur_img = 0;

function upd_imgs(index, count) {
  //Garantir que existem count elementos
  while($('.img_change').length<count) $('#imgs_wrapper').append("");
  //Corrigir count se superior a elementos disponíveis
  if (index+count>imgs.length) count=imgs.length-index;
  $('.img_change').each(function() {
    if ($(this).index()>=count) {
      //Ocultar elementos extra contagem
      $(this).hide();
    } else {
      //Atualizar src e mostrar imagens
      this.src=imgs[index+$(this).index()];
      $(this).show();
    }
  });
}

//Primeira atualização
upd_imgs(cur_img, num_imgs);

$('#next_imgs').on('click', function() {
  cur_img=cur_img+num_imgs;
  if (cur_img>=imgs.length) cur_img=0;
  upd_imgs(cur_img, num_imgs);
});

$('#prev_imgs').on('click', function() {
  cur_img=cur_img-num_imgs;
  if (cur_img<0) cur_img=imgs.length-imgs.length%num_imgs;
  upd_imgs(cur_img, num_imgs);
});

Comment: Cara, fantástico! Muito obrigado. Resolvido!
Answer: Creio que haja uma solução mais simples do que isso. caso você crie os elementos que simbolizem cada item do slideshow, (o que fica mais leve depois de processado), provavelmente você terá uma estrutura como essa:
https://jsfiddle.net/tq6gen43/4/


Placement of Unique Objects into Distinct Containers

Question: Say we have 5 distinct items and 2 distinct boxes, none of which can be empty.
There are $2^5$ ways to arrange them, but accounting for scenarios where all balls are in one box, we subtract 2, as there are two ways for this scenario to occur.
However, say we instead kept our 5 distinct items and added a new distinct box, so 3 of them, and none may be empty.  Following this same method, how would we calculate the number of valid arrangements, starting with the $3^5$ total options and removing the invalid ones?
Answer: This problem can be solved more generally, it is one of the problems that appears in the twelvefold way problem.
In general if you have $k$ boxes and $n$ objects then there are ${n\brace k} k!$ ways to place the objects. Note that $n\brace k$ denotes the stirling number of the second kind.
Comment: Yes, I have looked at the Stirling number formula, but I am just wondering if there is perhaps a more intuitive way at looking at a smaller example like this one.  I'd like to understand how and why a small example works so that a more complex example will later makes more sense.  Thank you!
Answer: Case 1 - Only 1 bin is empty.
Let we have all balls in bin 1 and bin 2. And bin 3 is empty. Then we have $2^5$ ways.
But its not necessary that empty bin is always bin 3. It may be bin 1 or bin 3.
So we have total cases $3 × 2^5$.
Case 2 - Two of them empty.
Let all balls in bin 1. And bin 2 and bin 3 empty. So we have $1^5 = 1$ way.
Again its possible that all balls in bin 2 or bin 3.
So we have total cases $3 × 1 = 3$ cases.

Total cases where no bin can be empty = $3^5 - 3 × 2^5 - 3$


Show that there exists a unique topology $\tau$ on $X$ such that $\mathcal{A} \subset \tau$.

Question:
Let $X$ be a set and $\mathcal{A} \subset \mathscr{P}(X)$ be a family of subsets of $X$. Show that there exists a unique topology $\tau$ on $X$ such that $\mathcal{A} \subset \tau$.

This is the same as showing that the topology generated by the subbase $\mathcal{A}$ is the coarsest topology we can put on $X$?
I don't know how to show that $$\mathcal{B}=\{B_1\cap B_2 \cap \dots \cap B_n \mid B_i \in \mathcal{A}\}$$ forms a basis for $X$. I think I need to assume that $X$ can be gotten from some of the intersections in order for this to work? If so, then $\forall x \in X : \exists B \in \mathcal{B}$ such that $x \in B$. Also if $U, V \in \mathcal{B}$, then $U = \bigcap_{k \in \Bbb N}B_k$ and $V=\bigcap_{j \in \Bbb N}B_j$ so $U \cap V$ is either empty or the smaller one of $U$ or $V$, but in both cases in $\mathcal{B}$?
Now elements of $\tau$ are arbitary unions and finite intersection of $\mathcal{B}$ how can I show that this is the coarsest topology I can put on $X$?
Comment: As the question is stated right now, this looks false. For example, what is $\mathcal{A}$ is already a topology? Then there might be bigger topologies which contain it. So a topology which contains $\mathcal{A}$ might not be unique.
Comment: I found the book online. You have left out one key word from the exercise. It should say "unique smallest topology."
Comment: @Lakritsa Are you sure you have copied it correctly? The current statement you have at the top is certainly false.
Comment: @Mark the problem is from introduction to topology by Gamelin & Greene. Problem 5 of chapter on bases.
Comment: Yes, just take $\mathcal{A}$ as a subbase or $\mathcal{A} \cup \{X\}$ if you're following Munkres' conventions. Unicity is a small argument.
Answer: This topology can be defined as $\tau = \bigcap \{ \tau'\mid \tau' \text{ a topology on } X: \mathcal{A} \subseteq \tau' \}$
The collection over which we take the intersection is non-empty as the discrete topology $\tau'=\mathscr{P}(X)$ is always in it, and an intersection of topologies is again a topology. Minimality is obvious. We just use that the set of topologies on $X$ is a complete lattice.
A simple argument will show $\tau$ is unique as well.


Piano methods (classical) that address musical meter specifically

Question: Czerny's op.500, book 1, lessons #12 and #13 are specifically about meter. I think the main point of the lessons is playing various rhythms which subdivide the beat while being able to count the meter properly.
Can anyone recommend other piano methods which use a similar approach?
Answer: This doesn't exactly answer your question, but I think there is an important point to make. At this point in my studies, I was getting all of my examples from the standard literature. It's my strong opinion that exercise books like this be kept to a minimum, and not be used at all once someone is sufficiently advanced (such as they would be by the time they get to the specified Czerny examples).
Naturally, there are many etudes which cover rhythm. For example, Chopin's Etude No. 1 in F Minor from Trois Nouvelles Etudes is a good example of 3 over 4, and is a good piece to couple with the Fantasy Impromptu which has a lot of 4 over 3. I don't know that these 2 examples are what you are looking for in terms of meter, but the larger point is that method books are generally not worth the time when there are so many good examples in real music.
Answer: If what you are looking for is accuracy in subdividing beats into 3,4,5,etc.  then it might be worth looking at the Scriabin Etudes.  They are not at all easy but there are lots of complex rythmic ideas there.


Determine an unknown number of real world locations from GPS-based reports

Question: I'm working on some software which should determine real world locations (f.e. speed cams) from several GPS-based reports. An user will be driving when reporting a location, thus the reports a very inaccurate. To solve that problem I have to cluster reports about the same location and calculate an average.
My question is about how to cluster those reports. I read about Expectation-maximation algorithms and k-means clustering, but as I understood I would need to determine the number of real locations in advance.
Are there any other algorithms, which don't need the exact number of real locations, but instead use some edge conditions (f.e. minimal distance) ?
A report contains longitude, latitude and accuracy (in meters). There is no name or anything else which could be used to identify duplicates.
Another obstacle could be that it will be common, that there is only one report for a real world location. That makes it difficult to distinguish outliers from good data.
Comment: Hi Pat. I'm talking about traffic jams or speed cams to make it more clear.
1. The "k" in k-means clustering stands for the number of clusters. In my case I would have one cluster for each location, whereas I don't know how many different locations there are.
2. Yes, they also include a timestamp. But I don't understand why that should be important, because I only care about the position.
Comment: I'm not sure what you mean when you say "...as I understood I would need to determine the number of real locations in advance..." Assuming I've understood you correctly, there's nothing in the algorithms which inherently requires this. Are you perhaps planning to increase the number of cluster components based on the number of reports?
Comment: 2nd question :) . If your reports are coming from someone who is driving, then there's possibly going to be significant changes in position between them. Do the reports come with a timestamp telling you when they were taken?
Comment: 1. Ahh, I see. So you have an unknown number of locations, and each location generates one or more reports. However, all you see are a stream of reports. You want to infer how many locations there are, and their position, based on the observed reports. Have I got it? 2. I worry about timestamps because you say the user will be *driving* when they give a report. As such, unless the reports come in very quick succession, or the speed is very low (possible, if it's a traffic jam) then a single location will look like a ragged line of reports following the road. Timestamps may be able to help here
Comment: 1. Yes, you've got it. 2. It's a manual task, so a location should only be reported once per user during one trip. But you're right, I have to handle duplicates, when a user accidently clicks twice, and reports will be inaccurate when driving fast. That's why I mentioned the edge condition of a minimal distance between locations. Let's ignore traffic jams, which might spread some miles, and assume that a location is very small.
Answer: I have found a software that maybe can help you. It looks like somebody had the same problem that you and they gave him a solution in this forum, so you will need to use ArcGIS, but if you are looking for an algorithm they suggest this paper. I think the paper is detailed enough to be a good start fro your algorithm.
Comment: Because *all* the information is this answer resides in its links, and links do eventually rot, please at least summarize what the forum posts and the paper recommend.


file .ics to delete appointment not recognized by outlook

Question: I send a icalendar when a user create an appointment on my application and the file ics is regularly recognized by outlook.
If the user delete the appointment on my application i send a icalendar with canceled notation, but the .ics file is not recognized (not supported calendar message.ics)
Icalendar which create the appointment on outlook (regular recognized):
BEGIN:VCALENDAR
PRODID:-//Schedule a Meeting
VERSION:2.0
METHOD:CANCEL
BEGIN:VEVENT
DTSTART:20190718T033000Z
DTSTAMP:20190718T113347Z
DTEND:20190718T040000Z
LOCATION:
UID:Tesis10910
DESCRIPTION:
X-ALT-DESC;FMTTYPE=text/html:
SUMMARY:
STATUS:CANCELLED
ORGANIZER:MAILTO:firstname.lastname@example.com
ATTENDEE;CN="firstname.lastname@example.com";ROLE=REQ-PARTICIPANT;RSVP=TRUE:mailto:email@example.com
BEGIN:VALARM
TRIGGER:-PT15M
ACTION:DISPLAY
DESCRIPTION:Reminder
END:VALARM
END:VEVENT
END:VCALENDAR

Icalendar which delete the appointment:
BEGIN:VCALENDAR
PRODID:-//Schedule a Meeting
VERSION:2.0
METHOD:CANCEL
BEGIN:VEVENT
DTSTART:20190718T033000Z
DTSTAMP:20190718T113347Z
DTEND:20190718T040000Z
LOCATION:
UID:Tesis10911
DESCRIPTION:
X-ALT-DESC;FMTTYPE=text/html:
SUMMARY:
STATUS:CANCELLED
ORGANIZER:MAILTO:email@example.com
ATTENDEE;CN="firstname.lastname@example.com";ROLE=REQ-PARTICIPANT;RSVP=TRUE:mailto:email@example.com
BEGIN:VALARM
TRIGGER:-PT15M
ACTION:DISPLAY
DESCRIPTION:Reminder
END:VALARM
END:VEVENT
END:VCALENDAR

Answer: Few things:

don't know if it is intended but your first iMIP message already has a METHOD:CANCEL instead of REQUEST.
your second iMIP message has a different UID. If this is really what you are sending, it is normal for the client not to recognise it as the cancel of the first message.
Your second message lacks a SEQUENCE property with an increased value. See https://www.rfc-editor.org/rfc/rfc5546#section-3.2.5
there is no point in adding the VALARM component. Clients will always ignore it.


jQuery: calling a function within an event handler's nameless function

Question: I want to know why this works:
$('#clientSearchFirstName').keyup(runLiveSearch);

But this doesn't:
$('#clientSearchFirstName').keyup(function() {
    runLiveSearch();
    });

The error returned for the latter has to do with the jQuery library (I have the minified version), apparently effected in some way:

TypeError: f.nodeName is undefined (jquery.js)
...,h=h.slice(c.length),c.type=g,c.matches=d);if(!c)break}return
  b?h.length:h?bc.er...

Is it impossible to call another function with an event handler's "default" function? For reference (I don't see why it would have to do with the contents of the function), here is the function:
function runLiveSearch() {
            var search_string = $(this).val();
            var which = this.id;
            which = which.replace("clientSearch","");

            if (search_string == '') {
                $("#statusBox").fadeOut();
            }else{
                $.ajax({
                    url: "echoSearchResults.php?searchQuery=" + search_string + "&which=" + which, // ...etc
                    type: 'GET',
                    dataType: 'json',
                    success: function(data) {
                        var filler = '';
                        if (data && data.length > 0) {
                            if (data.length < 30) {
                            filler = 'Found '+data.length+' results for "'+search_string+'":
'; }
                            else { filler = 'Showing first 30 results for "'+search_string+'":
'; }
                            filler = filler + '

ID	First name	Last Name	Cell phone	Home phone	Work phone	Primary Phone
';
                            for (x=0;x <= (data.length - 1);x++)
                            {
                                filler = filler +'
'+data[x][0]+'	' + data[x][1] + '	' + data[x][2] + '	'+convertphone(data[x][3],"touser")+'	'+convertphone(data[x][4],"touser")+'	'+convertphone(data[x][5],"touser")+'	'+convertphone(data[x][6],"touser")+'
';
                            }
                            filler = filler + '
';
                        } else { filler = 'No search results found for "'+search_string+'".'; }
                        document.getElementById('statusBox').innerHTML = filler;
                    }
                });
                $("#statusBox").fadeIn();
            }
        }

Answer: In short: change your code to
$('#clientSearchFirstName').keyup(function() {
    runLiveSearch.call(this);
});

In a bit more detail: the context (this) differs when function is called "as-is", while jQuery uses call method to provide functions with right objects. You can experiment with this a bit by logging values of this.
Comment: Wow. Lifesaver man. Very interesting. Really... I may have never found this, and so quickly to boot!


How to create an array with references to other variables?

Question: I'm trying to create an array (or collection) that can contain other variables that can be accessed or changed, and have the changes be reflected outside the array.
boolean autoTakeStepMode = false;
boolean[] settings = {autoTakeStepMode};
String[] settingsMessages = {"Enable auto-take-step mode (y/n)?"};
String response = "";
for (int i = 0; i < settingsMessages.length; i++) {
    response = ui.getYesNoResponse(settingsMessages[i]);
    if (response.equals("y"))
        settings[i] = true;
    else if (response.equals("n"))
        settings[i] = false;
}

When I try to access the autoTakeStepMode it is always equal to its default value, false.
Comment: You can't do this, basically. It's no different from setting `boolean x = autoTakeStepMode;` - if you change the value of `x`, that won't change the value of `autoTakeStepMode`.
Comment: Unlike C++, Java does not allow references to other variables.
Comment: you could use a map to store the booleans "by name"
Comment: This sounds line an [XY Problem](http://xyproblem.info).  Back away from how you _think_ this should be solved, and explain what you're actually trying to accomplish.
Comment: thats because you are using a primitive boolean type, if you change it to the wrapped class Boolean it is going to be a reference. So, if you change the value it would be reflected to every place where you are using this object. Any way, the map solutions it's better.
Comment: @MiguelResendiz Please explain how you would change the value of a `Boolean` (the wrapper type). Hint: You can't!
Answer: The type should not be primitive type. If you want to change the variable autoTakeStepMode, you should wrap it with class.
public class AutoTakeStepMode {
     boolean isOpen;
}

Comment: Or use the Boolean class... it is a native wrap for boolean.
Comment: No, it works, just replace boolean to AutoTakeStepMode. Here's a blog about difference between primitive type and class type: http://www.programmerinterview.com/index.php/java-questions/difference-between-a-primitive-type-and-a-class-type/
Comment: @MiguelResendiz Boolean is 'wrapped' but instances of it are not mutable, so it is no better for this Q.
Answer:
When I try to access the autoTakeStepMode it is always equal to its
  default value, false

This is the actual behavior . If you want to update value of autoTakeStepMode then update it as well along with settings[i] .
for (int i = 0; i < settingsMessages.length; i++) {
    response = ui.getYesNoResponse(settingsMessages[i]);
    if (response.equals("y"))
        settings[i] = true;
    else if (response.equals("n"))
        settings[i] = false;

       autoTakeStepMode = settings[i];

}

Comment: I think this defeats the purpose of having an array containing boolean values for each setting when I have to explicitly assign `autoTakeStepMode` this way.


If I already have files in an S3 bucket in the Glacier storage class, what happens if I apply a bucket policy to move to Glacier after 1 day?

Question: We have a pre-existing S3 bucket that we are looking to add a lifecycle policy to in order to move large files in S3 standard to Glacier. The bucket already contains items with the Glacier storage class (changed manually via console). How does the lifecycle policy handle files that are already in Glacier? Will they be skipped or will they be moved again?
Question comes mostly because of a previous experience adding a transition to Deep Archive policy to files that were in Glacier that resulted in an early deletion from Glacier charge that was not ideal (since the files were in the Glacier storage tier for less than 90 days).
Also have a small test going in a test bucket tonight to see how it behaves but was curious if anyone had experience in a production environment?
Answer: If you create a policy to move S3 standard tier objects to Glacier tier, objects already in glacier tier are not changed.
The reason you were charged when you moved from glacier to deep archive tier is the minimum time commitment policy. S3 standard has no minimum time policy.
Comment: That makes sense. Thanks for the confirmation.


Get entire BODY content using PHP DOM DOCUMENT

Question: I want to get entire body tag content using DOM Document.
I used following code:
$dom = new domDocument;

/*** load the html into the object ***/
$dom->loadHTML($html);

/*** the table by its tag name ***/
$tables = $dom->getElementsByTagName('body')->item(0)->nodeValue;

This gives me TExt. I want entire body content.
Comment: do you expecting to get html tags like ` `??
Comment: You....you don't mean that you're expecting to get things like images and videos as binary objects with `domDocument`, do you?
Comment: I'm expecting entire raw data inside BODY Tag
Comment: So....let's say we have `` somewhere inside the `body` of our page....you're expecting the image tag as an HTML element as part of the `domDocument` crawl, and **not**, say, the actual jpeg, right?
Answer: You can pass the body DOMElement to either DOMDocument::saveHTML() or DOMDocument::saveHTMLFile(), e.g.
<?php
$doc = new DOMDocument;
$doc->loadhtmlfile('http://stackoverflow.com');

$body = $doc->getElementsByTagName('body');
if ( $body && 0<$body->length ) {
    $body = $body->item(0);
    echo $doc->savehtml($body);
}

prints
Warning: DOMDocument::loadHTMLFile(): Unexpected end tag : p in http://stackoverflow.com, line: 2843 [...]

[...]

Comment: you still get the "outer"  tags though. You need to remove them with regex, or see here for another solution : https://stackoverflow.com/a/11254896/4360302
Answer: $dom = new domDocument;
$dom->loadHTML($html);

// ... change, replace ...
// ... mock, traverse ..

$body = $dom->documentElement->lastChild;
$dom->saveHTML($body);

Answer: It is safer to use PHP tidy extension which can fix invalid XHTML structures and also extract body only:
$tidy = new tidy();
$htmlBody = $tidy->repairString($html, array(
    'output-xhtml' => true,
    'show-body-only' => true,
), 'utf8');

Then load extracted body into DOMDocument:
$xml = new DOMDocument();
$xml->loadHTML($htmlBody);


interesting 2-3 bar ideas for a beginning improviser

Question: Sadly I won't succeed at the "Music Man" approach to learning to improvise. By that I mean "hear it, fumble around with no coaching, and then play inspired music by the end of the movie".
What I'd greatly appreciate are some strategies for creating interesting 2 to 3 bar arrangements within a given scale. This would be especially helpful for those lead sheets that end a long phrase with a "held" note taking two bars. That's where, for blues especially, I've heard some really clever transitions.
For background, I've been told to start with a "horizontal" approach ("sequence scale notes in order") interrupted by small interspersed loopbacks. I expect someone with experience could make that cool, probably using some syncopation. I've also tried, with a bit more success, to blues-ify the bars by picking notes from the right blues scale.
FWIW, my "ear" is also fairly untrained, so I find it hard to pick up ideas off a recorded performance.
Suggestions?
Comment: This doesn't directly answer your question for improvisation strategies, but I highly recommend ear training, especially if you intend to do much composition or improvisation. It's pretty straightforward (and fun) to learn on your own, and the payoff is well worth the investment. There's lots of answers about how to do that on this very site: http://music.stackexchange.com/search?q=ear+training
Answer: There are two basic approaches to take when improvising. One is a chord-first approach, and the other is a scale-first approach. Both are useful, and can feed back and forth on each other.
In the chord first approach, you begin with an existing chord progression (which you might possibly modify or elaborate) and determine your melody notes from that. You aren't limited to just chord-tones, though, as you can begin to insert so-called non-harmonic tones, such passing tones and appoggiaturas. This is what you want to use if you're improvising over an existing tune, since you don't want to stray too far from the melody.
In the melody first approach, you begin with notes from your chosen scale, and then harmonize them. To some extent, these notes can even be seemingly random, although for best effect, they should tend to have a lyrical quality. You're looking for something that makes sense as a singable melody. One good way to do that is to use mostly scale steps, with occasional leaps. This approach can be used when you're creating a brand new melody from scratch, and have no need to fit to pre-existing chords.
Lots more advice regarding creating melodies can be found here:

How to make melodies effectively?
Learning to create melodies


create trigger in PosgreSQL

Question: So I found this example:
create function eager.account_insert() returns trigger
  security definer
  language plpgsql
as $$
  begin
    insert into eager.account_balances(name) values(new.name);
    return new;
  end;
$$;

create trigger account_insert after insert on accounts
    for each row execute procedure eager.account_insert();

The thing I can't understand: function eager.account_insert() does not take any arguments, however, it operates with variable new. It returns it, but should't it return trigger?
Also, this: insert into eager.account_balances(name), this is not some certain record chosen, what it this?
Answer: The new (and old when it's an update statement) is the RECORD you're inserting or updating. You can get columns from it and do whatever you want with them. Many times BEFORE INSERT triggers check for valid values etc.
The function must return a RECORD with the same columns as the table, or NULL if the insert should not happen (usually for INSTEAD triggers).
The insert statement is just a regular insert with one column specified of the table and the value is taken from the newly inserted RECORD's column name.
The documentation explains the triggers very well.

create trigger in PosgreSQL


Foreignkey usando User em outra models

Question: Estou desenvolvendo um site de cotação pra fins de estudo da linguagem. Customizei a models User e nela inclui outro atributos, como endereço e cidade por exemplo. O problema, agora, é gravar o ID do usuário logado no campo relacionado, ou seja, a minha tabela de cotação tem um campo client que recebe o id do usuário logado, para que na hora de criar a cotação este id seja gravado. Se eu fizer isto pelo Admin dá certo! Se eu adicionar o campo 'client' via forms.py pela classe Meta, dá certo! por que eu posso escolher o usuário logado ou outro da lista. Mas o certo é gravar direto o id sem precisar mostrar ou escolher usuário.
O que fiz de errado nos códigos abaixo?
models.py

from django.db import models
from qnow_user.models import User
from django.conf import settings
from datetime import datetime,date
from django.contrib.auth import get_user_model

# Create your models here.
class Quotation(models.Model):
    #User owner quotation
    client = models.ForeignKey(settings.AUTH_USER_MODEL,
        on_delete=models.CASCADE, related_name='quotation',
        help_text='Cliente dono desta cotação')

    #Date create of quotation
    date_create = models.DateField('Data Criação',default=date.today,
        help_text='Data em que foi criada esta cotação.')

    date_update = models.DateField('Data Atualização',default=date.today,
        help_text='Data em que foi modificada esta cotação.')

    #House type to quotation
    HOUSETYPECHOICES = (
    ('Apartamento', 'Apartamento'),
    ('Casa', 'Casa'),
    ('Comercial', 'Comercial'),
    ('Escritório', 'Escritório'),
    ('Outros', 'Outros'),
    )
    house_type = models.CharField('Tipo do Imóvel',max_length=20,
                choices=HOUSETYPECHOICES,blank=False,default='Casa',
                help_text='Especifique o seu tipo de imóvel')

    STATUSCHOICES = (
        (0,'Pendente'),   #Client requested quotation
        (1,'Em Análise'), #Company analizing quotation
        (2,'Liberado'),   #Quotation released for provider
        (3,'Orçado'),     #Quotation provider

    )
    status = models.IntegerField('Situação do Pedido',choices=STATUSCHOICES, blank=True,default=0,
        help_text='Difinia a situação atual desta cotação.')

    #House set to quotation
    house_set = models.CharField('Tipo de Residencial', max_length=100, default='Casa particular',
                help_text='Especifique o nome do seu condomínio ou residencial.')

    class Meta:
        verbose_name = 'Cotação'
        verbose_name_plural = 'Cotações'

    def __str__(self):
        return self.house_set

forms.py

from django import forms
from .models import Quotation
from django.contrib.auth import get_user_model
from qnow_user.models import User

User = get_user_model()

class QuotationForm(forms.ModelForm):
    class Meta:
        model = Quotation
        fields = ('client','date_create', 'date_update', 'house_type',
            'house_set',
        )

views.py

from django.shortcuts import render, redirect, get_object_or_404
from django.http import HttpResponse, HttpResponseNotFound
from django.contrib.auth.decorators import login_required
from django.views.decorators.csrf import csrf_exempt
from django.contrib.auth import authenticate, login
from django.http import HttpResponseRedirect
import datetime
from .models import Quotation
from .forms import QuotationForm

@login_required
def quotation_client(request):
    template_name = '../templates/client_quotation.html'
    form = QuotationForm(request.POST)
    if request.method == 'POST':
        #if request.user.is_authenticated:
        #    quotation_client = Quotation.objects.get_or_create(client=request.user)
        if form.is_valid():
            form.save(commit=True)
            return HttpResponse("
Cotação Feita
")
        else:
            form = QuotationForm()
    else:
        form = QuotationForm()
    print('tetse')
    context = {
                'active_page_client_provider' : 'active',
                'form':form
                }
    return render(request,template_name,context)
#return HttpResponse("
Área do cliente no cliente2
")

admin.py

from django.contrib import admin
from .models import Quotation

# Register your models here.
admin.site.register(Quotation)

Answer: Você tira o client do form, com isso vc não vai exibir nada do cliente
class QuotationForm(forms.ModelForm):
    class Meta:
        model = Quotation
        fields = ('date_create', 'date_update', 'house_type',
            'house_set',
        )

Na hora de salvar, em vez de salvar direto o "form" você pega a instancia de Quotation, adiciona o user logado e salva:
@login_required
def quotation_client(request):
    template_name = '../templates/client_quotation.html'
    form = QuotationForm(request.POST)
    if request.method == 'POST':
        if form.is_valid():
            quotation = form.save(commit=False)
            quotation.client = request.user
            quotation.save()
            return HttpResponse("
Cotação Feita
")
        else:
            form = QuotationForm()
    else:
        form = QuotationForm()
    print('tetse')
    context = {
                'active_page_client_provider' : 'active',
                'form':form
                }
    return render(request,template_name,context)

Answer: Obrigado Ricardo, eu acabei a minutos atrás descobrindo via outros meios a mesma
coisa que tu me passou. E pelo que vi acertei. Veja como ficou agora.

from django.shortcuts import render, redirect
from django.contrib.auth.decorators import login_required
from django.contrib.auth import authenticate, login
from .models import Quotation
from .forms import QuotationForm

@login_required
def quotation_client(request):
    template_name = '../templates/client_quotation.html'
    form = QuotationForm(request.POST)
    if request.method == 'POST':
        if request.user.is_authenticated:
            Quotation = form.save(commit=False)
            Quotation.client = request.user
            if form.is_valid():
                form.save(commit=True)
                return redirect('qnow_site:site')
            else:
                form = QuotationForm()
        else:
            context = {
                'origin':'client',
                'active_page_client_provider':'active'
            }
            return redirect('qnow_user:login_client')
    else:
        form = QuotationForm()
    context = {
                'active_page_client_provider' : 'active',
                'form':form
                }
    return render(request,template_name,context)

Foreignkey usando User em outra models


Logo becomes pixelated when it spins

Question: http://jasperalani.com/
If you hover over the logo at the bottom left of the page it becomes super pixelated when you hover over it.
I am using

transform: rotate(360deg);

How can I fix this so it maintains its quality?
Comment: The browser does that so save ressources afaik
Comment: did you try `outline: none; ` in your logo css?
Answer: That's because you are  making you image to transform:rotate(360deg);, instead of that target you parent element .socialmedia and try it works.
.socialmedia {
    position: fixed;
    bottom: 0;
    right: 1;
    width:21px;
    height:21px;
    -webkit-transition: -webkit-transform 0.6s ease-in-out;
    transition: transform 0.6s ease-in-out;
}

img{
    width: 100%;
    height: 100%;
 }
.socialmedia:hover{
    transform:rotate(360deg);
}

.socialmedia {
 position: fixed;
 bottom: 0;
 right: 1;
 width:21px;
 height:21px;
    -webkit-transition: -webkit-transform 0.6s ease-in-out;
   transition: transform 0.6s ease-in-out;
}

img{
 width: 100%;
 height: 100%;
 }
.socialmedia:hover{
 transform:rotate(360deg);
}

Comment: Oh this is really better than my solution, nice. +1
Comment: Yes @modiX and thanks, because when we rotate our image without any kind of hover effect on it and then you check, it's border still pixelate. Whereas not when placing them in div.
Answer: This is a known problem on Google Chrome.
Usually, you can overcome the problem by adding -webkit-backface-visibility: hidden to your element you rotate. In your example it will remove the anti aliasing entirely, though.
This is, because you are using transition instead of transform, so to fix the problem you rather add outline: 1px solid transparent to the CSS of your image. This will solve the problem.

img {
    width: 21px;
    height: 21px;
    -webkit-transition: -webkit-transform 0.6s ease-in-out;
    transition: transform 0.6s ease-in-out;
    outline: 1px solid transparent;
  }

img:hover {
    -webkit-transform: rotate(360deg);
    transform: rotate(360deg);
  }

Logo becomes pixelated when it spins


How to get the section and sub-section a table belongs to?

Question: I have the following code which loops through all the tables in a word document and pull all row and cell information. I also need to know which section and sub-section this table belongs too and record it in a cell in Excel. Can I access this information ?
<code> 'On Error Resume Next
  Application.ScreenUpdating = False
  Application.EnableEvents = False
 'Application.Calculation = xlCalculationManual
  Application.DisplayAlerts = False
  Dim intChoice As Integer
  Dim strPath As String
  Dim objWord As Object
  Dim objdoc As Object
  Dim ChartObj As ChartObject
 'Dim objShape As InlineShape
  Dim tTable As Word.Table
  Dim wb As Worksheet
  Dim wb1 As Worksheet
  Set wb = Worksheets("Pull Images")
  Set wb1 = Worksheets("Results")
  wb1.Pictures.Delete
  Set objWord = CreateObject("Word.Application")
  objWord.Visible = True
  Application.FileDialog(msoFileDialogOpen).AllowMultiSelect = False
  intChoice = Application.FileDialog(msoFileDialogOpen).Show
  'On Error Resume Next
  'if the user selects a file
  If intChoice <> 0 Then
  'get the path selected
  strPath = Application.FileDialog( _
    msoFileDialogOpen).SelectedItems(1)
  'opens the document
  Set objdoc = objWord.Documents.Open(strPath)
  With objWord.Documents(objdoc)
  Set Rng = wb.Range("A1")
  N = 3
  For Each tTable In objdoc.Tables
    tTable.Range.Copy
    Rng.Select
    Rng.Parent.PasteSpecial Format:="Text", Link:=False, _
                DisplayAsIcon:=False
    With Rng.Resize(tTable.Rows.Count, tTable.Columns.Count)
        .Cells.UnMerge
        .Cells.ColumnWidth = 14
        .Cells.RowHeight = 14
        .Cells.Font.Size = 10
    End With

    Set Rng = Rng.Offset(tTable.Rows.Count + 2, 0)

 Next tTable
 End With
 End If
 'objWord.ActiveDocument.SaveAs ThisWorkbook.Path & "\" &
  ActiveSheet.Range("E3").Value & "_MVR"
 'objWord.ActiveDocument.Close
  objWord.Quit
  Set objdoc = Nothing
  Set objWord = Nothing
  Set myrange = Nothing
  Set myrange1 = Nothing
  Application.DisplayAlerts = True
  Application.EnableEvents = True
  'Application.Calculation = xlCalculationAutomatic
  Application.ScreenUpdating = True
  end sub
</code>
Comment: [Yes, you can](http://vba.relief.jp/word-vba-get-current-section-number/).  What have you tried?
Comment: I have tried selection.Information(wdActiveEndSectionNumber), but it does not work.
Comment: `it does not work` is not a description of a problem
Comment: are you trying to get information about the `selection` object or about the `tTable` object ?
Answer: This code will return the section number of each table in your document.
<code>Dim Tbl As Table

With ActiveDocument
    For Each Tbl In .Tables
        Debug.Print Tbl.Range.Sections(1).Index
    Next Tbl
End With
</code>


Area of intersecting circles

Question: Find all points of intersection $(r,\theta )$  of the curves $r=5\cos(\theta),\  \ r= 4 \sin(\theta)$.
In this problem the curves intersect at the pole and one other point. Only enter the answer for nonzero $r$ in the form $(r,\theta)$ with $\theta$ measured in radians.
Find the area inclosed in the intersection of the two graphs.
I understand that if I make them equal I can find the intersection in radians which is $4/5*tan\theta=1$ but I dont understand the $r$ part or how to find the area.
Comment: You can try rewriting the equations in the fashin of $r^2=5r\cos(\theta)$, then transform back into $(x,y)$ coordinates to have a better geometric intuition of what's going on.
Answer: Once you have $\frac 45 \tan \theta=1$ you can determine $\theta$.  You can plug that $\theta$ into either of your original equations to get $r$.
For the area, have you plotted the two graphs?  You have two circles.  You are to find the area of the intersection of the two circles.  You can do that with geometry, knowing the radii and distance between the centers.  Draw a line through the two intersection points and you get two circular segments.
Answer: To complement the answer by @Ross Millikan.
It is very important for such a problem to have a figure; I have displayed one below.
The unique non zero solution modulo $2\pi$ to equation $5 \cos(\theta)=4\sin(\theta)$ is
$$\tag{1}\theta_0:=\arctan(5/4).$$
Using the usual formula $\frac12\int r^2(\theta)d\theta$ for the area enclosed by a curve described by a polar equation (take care that we use the red curve first, then the blue curve):
$$A=\frac12\int_{0}^{\pi/2} min (4 \sin(\theta),5 \cos(\theta))^2d\theta$$
$$A=\frac12\int_0^{\theta_0} (4 \sin(\theta))^2d\theta+\frac12\int_{\theta_0}^{\pi/2} (5 \cos(\theta))^2d\theta.$$
Using formulas $cos^2(\theta)=\frac12(1+\cos(2\theta))$ and $sin^2(\theta)=\frac12(1-\cos(2\theta))$, one obtains:
$$A=\frac{16}{4}\int_0^{\theta_0} (1-\cos(2\theta))d\theta+\frac{25}{4}\int_{\theta_0}^{\pi/2} (1+\cos(2\theta))d\theta.$$
$$A=\frac{16}{4}(\theta_0-\frac12 \sin(2\theta_0))+\frac{25}{4}(\frac{\pi}{2}-\theta_0-\frac12 \sin(2\theta_0)).$$

$$A=\frac{25\pi}{8} - \frac{9}{4}\theta_0-\frac{41}{8} \sin(2\theta_0)
.$$
Using (1): $A \approx 3.6888$

Area of intersecting circles


Qual componente utilizar no PyQt4 para exibir registros de uma consulta SQL na tela?

Question: Estou desenvolvendo uma aplicação gráfica para Desktop em Python utilizando o PyQt4 (Qt Designer), a minha duvida é sobre qual componente usar para exibir na minha tela frmTelaPesqAluno registros de uma consulta SQL, e como usa-lo? É um componente parecido com o DataGridView do Visual Studio ou um Grid que eu preciso utilizar para listar os registros da consulta.
Segue imagem da tela que estou desenvolvendo:
Answer: Procurou na documentação? É a QTableWidget. Existem outros widgets auxiliares a ele.
Um exemplo simples em Python.
Tutorial mais completo para C++ utilizando Model-View.
Outro sobre o assunto.
Uma aplicação completa usando ele.
Comment: Estou em duvida se devo usar um QTableWidget ou QTableView, nunca utilizei um grid no Python.
Comment: A `QTableView` é mais simples e na verdade ela é usada internamente pela `QTableWidget`. Quase sempre você ficará com a mais completa e que permite melhor organização do código. A mais simples não se comparação com a existente no .Net.
Comment: Tem um exemplo do QTableView?
Comment: Tá cheio de exemplos mas não é o que você queria na pergunta: https://www.google.com.br/search?q=qtableview+example
Comment: Eu quero mandar o resultado de uma consulta SQL para o QTableView.
Comment: Aí você tem que fazer outra pergunta mas tem que mostrar o que já tentou fazer.
Comment: Vou tentar usar o QTableWidGet a partir do exemplo que vc me passou.


How to access a dynamic value from JavaScript to Php

Question: Is it possible to used the value of the centerId and pass it to my php variable?
const data = {
        action: 'ft-add-member',
        maritalStatus: $('.ft-entry-relationship-info .ft-marital-status ul li.current a').data('dropdown'),
        biological: $('.ft-entry-relationship-info .ft-biological .ft-circle-check').hasClass('checked'),
        caregiver: $('.ft-entry-relationship-info .ft-caregiver .ft-circle-check').hasClass('checked'),
        whangai: $('.ft-entry-relationship-info .ft-whangai .ft-circle-check').hasClass('checked'),
        deceased: $('.ft-entry-relationship-info .ft-deceased .ft-circle-check').hasClass('checked'),
        ex: $('.ft-entry-relationship-info .ft-ex-partner.unhide .ft-circle-check').hasClass('checked'),
        relationship,
        relatedTo: familyTree.selectedPerson.relatedTo,
        centerId: familyTree._person.id,
    };

Comment: Javascript is processed in the browser (Client side in this case), PHP runs on the server. To send something from the client to the server you have to use POST. In your case, you probably want to do this with AJAX, but the older method of setting a form action to POST works fine too.
Comment: You wouldn't pass it to a PHP variable because by that point PHP had already executed. Do you mean send it to your server through ajax? Then your PHP could pick it up
Comment: that means I need to bridge it with the post method via ajax? correct me if im wrong. just new to js ajax and php.
Answer: You would need to send that data through AJAX to a PHP endpoint.
Javascript:
const data = {
  action: 'ft-add-member',
  maritalStatus: $('.ft-entry-relationship-info .ft-marital-status ul li.current a').data('dropdown'),
  biological: $('.ft-entry-relationship-info .ft-biological .ft-circle-check').hasClass('checked'),
  caregiver: $('.ft-entry-relationship-info .ft-caregiver .ft-circle-check').hasClass('checked'),
  whangai: $('.ft-entry-relationship-info .ft-whangai .ft-circle-check').hasClass('checked'),
  deceased: $('.ft-entry-relationship-info .ft-deceased .ft-circle-check').hasClass('checked'),
  ex: $('.ft-entry-relationship-info .ft-ex-partner.unhide .ft-circle-check').hasClass('checked'),
  relationship,
  relatedTo: familyTree.selectedPerson.relatedTo,
  centerId: familyTree._person.id,
};

fetch('/your-php-endpoint.php', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify(data)
});

your-php-endpoint.php:
<?php
$data = json_decode(file_get_contents('php://input'), true);
print_r($data); // print to debug


Erro básico em JavaScript

Question: bom dia galera, alguem sabe me dizer por que estou recebendo o erro

SyntaxError: missing ) after argument list , na linha 14 no codigo a seguir:

$(document).ready(function() {

      var $div_novospedidos = $('#div_novospedidos');
      $div_novospedidos.empty(); //Limpando a tabela
      // setTimeout(function(){ location.reload(); }, 5000);
      $.get("database3.php", function(data, status) {
          //if (data == null) {console.log("sem pedidos hoje")} else {console.log("teste")};
          var dados_pedido = data;
          console.log(dados_pedido);
          const obj = dados_pedido;
        }

        var dados = $.parseJSON(data); console.log(dados); $.each(dados, function(i, item) {
          numeroPedido = item.doc_number;
          nomeCliente = item.client_name;
          horarioPedido = item.clock;

          var newReq = $('
');
          var cols = "";

          cols += '
HORARIO: ' + horarioPedido + '

';
          cols += '
PEDIDO: ' + numeroPedido + '

';
          cols += '
Cliente: ' + nomeCliente + '

';

          newReq.append(cols);

          $("#div_novospedidos").append(newReq);

        });
      }

Comment: Tens um `}` perdido no código antes de `var dados`. E esse `. ready(` não está fechado.
Answer: Faltou fechar o trecho onde você faz o $.get('database3.php', function (data, status) {.
Apenas adicione o ) antes do trecho que tem var dados. Também falta fechar com o ) a última linha, ou seja, depois do último  } da função usada em $(document).ready(function () {;
Sugiro utilização de alguma IDE que você possa, em tempo real, verificar os erros de sintaxe, para que evite perda de tempos ou embarassos como esse.
Eu descobri fácil por estar usando o VS Code.
Veja:

Erro básico em JavaScript


WSO2 ESB about dbreport and performance

Question: I have a testing in WSO2 ESB dbreport mediator.
When I send a "BIG message" (100 or 1,000 or 10,000 ... 500,000 rows) from Database A to WSO2 ESB.
And WSO2 ESB split the message as rows by Iterate mediators,
then use the DBReport mediator write row by row into Database B (by Data source pool).
When write 100 rows, it spent 5 seconds,
when write 1,000 rows, it spent 188 seconds,
and then write 10,000 rows, it need spent 19163 seconds.
How efficient use of DBReport mediators?
thanks.
Comment: below is my configuration...
Answer: DBReport mediator is executed synchronously in WSO2 ESB which means the thread that execute this DBReport mediator is stucked until the Database operation is completed. That means the performance is lower than when the execution happens asynchronously.
Therefore to gain the maximum performance, please use WSO2 Data Services server and use DSS to execute the insert Database operation. That way you can gain the maximum performance.
Comment: Hi Harshana,
 Thanks your answer.

 I have try to test insert data into RDBMS from WSO2 ESB through WSO2 DSS, but have no idea how to implement it.
Comment: In order to implement that, you need to generate the message that you want to send to DSS inside ESB. For that you can use "PayloadFactory" mediator or "Enrich" mediator. Once you have generated the Request to be sent to DSS from ESB, you can use "Send" mediator to send the generated message to the DSS server. For that you can add an "Address Endpoint" to the Send mediator. Please follow the http://blog.afkham.org/2012/05/wso2-esb-service-chaining.html article and http://blog.afkham.org/2012/05/scatter-gather-eip-with-wso2-esb.html to get a better understanding.
Comment: You can implement your ESB messaging flows as well as DSS configurations with WSO2 Developer Studio. You can download it from http://wso2.com/products/developer-studio location. It will help you to learn WSO2 suite quickly. Hope this helps!
Comment: Hi Harshana, Thanks again.
I was created a WSO2 ESB proxy service, it be receive SOAP message that contain rows data, and use Iterate mediator split as data row by row. Then send row data to WSO2 DSS.
WSO2 DSS write the data into RDBMS table.

The WSO2 ESB proxy service content in next comment....
Comment:

.
.
.
.
Comment: WSO2 DSS configuration is ...

TEMP_DB_esb_test

insert into ld_report_his (emfact_nm, document_nm, history_time) values (:emfact_nm,:document_nm,to_date(:history_time,'yyyy/mm/dd hh24:mi:ss'))

         <with-param name=" . . .
      </call-query>

Comment: But, it is not work!

It has the error message in WSO2 ESB and WSO2 DSS :
ERROR - AxisEngine The endpoint reference (EPR) for the Operation not found is /services/test and the WSA Action = null

I can not fixed the error.
Comment: Are you sure you have provided the correct endpoint URL for the data service? As i can see in the configuration it is   which is wrong. It should be  Please try with that change.
Comment: Hi, Harshana,

It's maybe the "stackoverflow" system bug.
When I post content in edit field, it is no semicolon. But when I click "Add Comment" button and complete post comment, the semicolon was appear.

So, the URI is correct. I was tried the WSO2 DSS service by "try it" successful.
Comment: Ok. I think the reason is that you are sending an incorrect request to your Data Service. Before sending the request using Send mediator you need to construct the message with Payload Factory. So if you could attach your Data Service Request you have tried with try-it tool, i can help you with the ESB configurations.

WSO2 ESB about dbreport and performance


su: /bin/bash: Resource temporarily unavailable

Question: Unable to switch user to postgres.
Ulimit settings for postgres user is set with reasonable limits. We are not hitting max.
No errors in /var/log/messages.
Error:
BETA -bash-4.2# sudo su - postgres
su: /bin/bash: Resource temporarily unavailable

Settings:
BETA -bash-4.2# ps -auxww | grep -i postgr | wc -l
503
BETA -bash-4.2# lsof | grep -i postgr | wc -l
35225
BETA -bash-4.2#

Ulimit for postgres process.
BETA -bash-4.2# cat /proc/26230/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             256580               256580               processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       256580               256580               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Netstat
BETA -bash-4.2# netstat -plan | grep -i post | grep ESTABLISHED | wc -l
496
BETA -bash-4.2#

Ulimit settings
BETA -bash-4.2# cat /etc/security/limits.d/postgres_limits.conf
# Limits settings for postgres

postgres soft nofile 4096
postgres hard nofile 4096

postgres soft nproc 400
postgres hard nproc 400

After I restart postgres, I was able to get in.
Resource util for postgres user.
BETA -bash-4.2# netstat -plan | grep -i post | grep ESTABLISHED | wc -l
1
BETA -bash-4.2#

BETA -bash-4.2# lsof | grep -i postgr | wc -l
309
BETA -bash-4.2# ps -auxww | grep -i postgr | wc -l
8
BETA -bash-4.2#

Comment: Your postgres max file limits is 4 times what your bash ulimit for open files is. Honestly I'm not super sure of how they interact, but the error you received could be caused by your max open files being exceeded.
Comment: try strace -ff su - postgres from root without using sudo from root :)
Comment: @c4f4t0r http://pastebin.com/U6dKxUgv
Answer: Just restarting postgres is not a long term solution, you will hit the limit again, unless you have physical resource constraints on the server such as memory. During the issue the number of processes opened(nproc) by postgres user was 503 and the estimated number of open files(nofile) was 35225 and yet your postgres_limits.conf shows that you have set nproc to only 400 and nofile to only 4096. Based on your data, you will need to increase those two parameters.


Balancing dataset for survival analysis

Question: So I'm using survival analysis to help identify customers at risk of churning. As of now, I've been using a hugely imbalanced dataset that consists of 99.5% of customers in the majority class (i.e. not churned) and 0.5% churned. To improve my model I've randomly undersampled the majority class such that the ratio between the minority and majority classes is only 1/3, which resulted in improvements when compared to using the full dataset.
However, in order to avoid removing data points which can be important, I would like to try oversampling techniques such as SMOTE, etc but I don't know how well this would suit survival models, because the duration (target variable) depends on a churn date/time which is the result of the difference of two variables in the dataset (date_client_churned + date_client_joined). Isn't generating new examples going to mess with the distribution of these two variables and as such produce erratic results? Has anybody dealt with this kind of imbalance in the dataset for survival analysis, and if so what was your approach? Is it really necessary to balance the dataset for survival analysis?
For reference, the survival models that I've used are: Cox Proportional Hazards, Random Survival Forests and Gradient Boosted Trees with Cox loss.
Just to clarify how I compute the duration variable: For time t = 0 I'm using the date a customer signs a contract with the company, given by t_born, and for the end date I'm using either a) the churn date t_churn if t_churn < t_study, where t_study represents the end of the study period (t_study=t_born+6, t_study=t_born+12, t_study=t_born+24 months); b) t_study otherwise. The duration column will then be given by the difference t_churn - t_born if t_churn < t_study; t_study - t_born otherwise.
Comment: [Misconceptions abound when it comes to class imbalance, and statistics sees little problem.](https://stats.meta.stackexchange.com/q/6349/247274) Why do you think the class imbalance is problematic in your case?
Comment: Any apparent improvement that comes from disrespecting the sample sizes is an illusion.  And don't use the c-index to compare models.  It's not good at that.
Comment: It seems at first glance that the `duration` should be the _difference_ of two dates, `date_client_churned - date_client_joined`, to represent the length of time that the individual was a client. Please edit the question to explain what you are using for the reference `time = 0` for individual customers and why you are using the sum of those two dates for the `duration` survival time. Please do that by editing the question, as comments are easy to overlook and can be deleted. Also, note that this might better be handled by a cure model that accounts for individuals that never have events.
Comment: The fact that you would create a new topic when this issue has been covered extensive on the site.  Thinking class imbalance is a problem is equivalent to not understanding the field of statistics.
Comment: @EdM Thank you for the suggestion, I've added an explanation on the OP.
Comment: @Dave Thank you for the article! I'm not saying it's problematic per se, but I've seen a significant boost in performance and lower training time when I used random undersampling compared to a model using the full dataset.
Comment: @FrankHarrell Not so much in the case of survival analysis. I'm not saying it's a problem, but as I said in the above comment, I've seen performance improvements when using random undersampling, hence why I wanted to explore other techniques, I just didn't find much information online on balancing techniques in survival models other than some adapted models.
Comment: Improvement of what metric(s)?
Comment: lift score (the first 10 percentiles) and the c-index
Comment: Particularly on the c-index I'm getting around 30% improvement.
Answer:
I would like to try oversampling techniques such as SMOTE, etc but I don't know how well this would suit survival models...

As the comments on this question and many pages like this one demonstrate on this site, it's not clear that SMOTE or any type of biased over/under-sampling suits any type of model well. You then necessarily bias the intercept of the model and thus all probability estimates (unless you back-correct for the biased resampling). It's not clear that even relative log-odds or log-hazard estimates are improved, as you don't have any more cases than you started with.
Unbiased resampling by bootstrapping at the original sample size, however, is a very useful technique for validating and calibrating models, including survival models. See the validate() and calibrate() functions of Frank Harrell's rms package for example. Provided that you resample by case so that each set of t_born, t_churn and t_study is kept together, there is no problem.
I suspect that your apparent improvement with SMOTE would disappear if you evaluated performance of the modeling process by bootstrap resampling or (with tens of thousands of cases) repeated train/test splits.
Finally, it's not clear that a continuous-time survival model like a Cox model is the best choice for data like yours, where there is apparently a very large ratio of cases to observation times. A discrete-time model--a set of binary regressions, including time as a predictor, on appropriately structured data--could be preferable. In particular, it would naturally handle your large proportion of censored cases that might be considered "cured" in a survival model. At the least you should consider a cure model that combines survival over time with a binary "cured" outcome, as the types of model you have used implicitly assume that every case eventually experiences the event in question.

Balancing dataset for survival analysis


Error: At line 463 of file custom.f90: Memory allocation failed

Question: I am facing an issue when scaling up my optimisation model. Here is a minimal example, which results in the stated error, when given enough entities to process. You can adjust the number of entities to be processed at the top of the code snippet.
Ideally, this script should run for tens of thousands of entities.
I appreciate the help! Thank you a lot :-)
import numpy as np
import pandas as pd
import datetime
from tqdm import tqdm
from scipy.stats import gaussian_kde
import math
import os
import matplotlib.pyplot as plt
from scipy.stats import norm, poisson
from gekko import GEKKO

NUM_ENTITIES = 50 # 10 crashes already... just a few work out fine

np.random.seed(321)
di_kde = dict()

for entity in range(NUM_ENTITIES):
    arr = np.random.normal(14.75, 6, 500)
    kernel = gaussian_kde(arr, bw_method='scott')
    di_kde[entity] = (kernel, int(np.amax(arr)), int(np.amin(arr)))

def normcdf_gekko(yi, mean, sd):
    y = 0.5*(1 + m.erf((yi-mean)/(sd*m.sqrt(2.0))))
    z = m.if3(1-y, 1, y)
    return z

def cdf_gauss_nogecko(kdes, maxi):

    bw = kdes.neff**(-1./(1+4))
    res = np.divide(np.cumsum([sum([norm.pdf(i, val, bw) for val in kdes.dataset.flatten()]) for i in np.linspace(0, maxi, 100, endpoint=True)]), max(np.cumsum([sum([norm.pdf(i, val, bw) for val in kdes.dataset.flatten()]) for i in np.linspace(0, maxi, 100, endpoint=True)])))
    return res

def cdf_gauss_singular_gecko(kdes, yi):

    bw = kdes.neff**(-1./(1+4))
    res = m.sum([normcdf_gekko(yi, val, bw) for val in kdes.dataset.flatten()]) / kdes.dataset.flatten().size
    return res

def cdf_gauss_singular_nogecko(kdes, yi):

    bw = kdes.neff**(-1./(1+4))
    res = np.sum([norm.cdf(yi, val, bw) for val in kdes.dataset.flatten()]) / kdes.dataset.flatten().size
    return res

def pdf_gauss_gekko(kdes, yi):

    bw = kdes.neff**(-1./(1+4))
    res_gecko = m.sum([normpdf_gekko(yi, val, bw) for val in kdes.dataset.flatten()])

    return res_gecko

def normpdf_gekko(x, mean, sd):
    var = float(sd)**2
    denom = m.sqrt((2*math.pi*var))
    num = m.exp(-(x-float(mean))**2/(2*var))
    return num/denom

cost = 0.1
revenue = 1

print(f'{datetime.datetime.now().strftime("%H:%M:%S ")}Running optimisation under constraint...')

if not os.path.isdir(os.path.abspath(r'.\Logging')):
    os.mkdir(os.path.abspath(r'.\Logging'))

m = GEKKO(remote=True) # Initialize gekko
#m.time = np.linspace(0,20,100)

m._path = os.path.abspath(r'.\Logging')

m.options.SOLVER = 3  # APOPT (1) is an MINLP solver
m.options.IMODE = 3 # 3 or 6
m.options.MAX_MEMORY = 10

m.solver_options = ['minlp_maximum_iterations 500', \
                    # minlp iterations with integer solution
                    'minlp_max_iter_with_int_sol 10', \
                    # treat minlp as nlp
                    'minlp_as_nlp 0', \
                    # nlp sub-problem max iterations
                    'nlp_maximum_iterations 50', \
                    # 1 = depth first, 2 = breadth first
                    'minlp_branch_method 2', \
                    # maximum deviation from whole number
                    'minlp_integer_tol 0.05', \
                    # covergence tolerance
                    'minlp_gap_tol 0.01']

print(f'{datetime.datetime.now().strftime("%H:%M:%S ")}Adding variables...')

qi = [m.FV(value=di_kde[k][1] , lb=di_kde[k][2], ub=di_kde[k][1], integer=True, name=f'qi_{k}') for idx, k in enumerate(di_kde.keys())]

print(f'{datetime.datetime.now().strftime("%H:%M:%S ")}Adding constraint equations...')

inter = m.Intermediate(m.sum([cdf_gauss_singular_gecko(di_kde[k][0], qi[idx4]) for idx4, k in enumerate(di_kde.keys())]) / len(di_kde), name='InterConstr')
m.Equation(inter >= 0.9)

print(f'{datetime.datetime.now().strftime("%H:%M:%S ")}Adding maximisation function...')

for ii, kk in enumerate(di_kde.keys()):
    m.Maximize(m.sum([(-cost * (m.max3(0, qi[ii]-j) + m.max3(0, j-qi[ii]) + qi[ii]) + revenue * m.min3(qi[ii], j)) * pdf_gauss_gekko(di_kde[kk][0], j) for j in range(di_kde[kk][2], di_kde[kk][1])]))

print(f'{datetime.datetime.now().strftime("%H:%M:%S ")}Start solving...')

m.open_folder()
m.options.DIAGLEVEL=10
output = m.solve(disp=True)

print(f'{datetime.datetime.now().strftime("%H:%M:%S ")}End solving...')
print('Results')
print('qi: ' + str(qi))
print('constraint: ' + str(inter.value))
print('Objective: ' + str(-m.options.objfcnval))

Comment: The stack trace is not accessible to the audience and myself, since the error results from an executable and is passed forward to Python and truncated strongly.

I actually was encouraged by the maintainer to post here instead of on GitHub to have a greater audience :-) https://github.com/BYU-PRISM/GEKKO/issues/133
Comment: Welcome to SO! But we are not the maintainers of `gekko`, so this may not be the proper forum for a bug report. In any case, your question should include the full stack trace of the error, not just the message. You should not expect us to work out for ourselves where line 463 is.
Answer: The size of the model scales with NUM_ENTITIES as 55486*NUM_ENTITIES-2008 with 53,224, 109,470, 164,195 lines for NUM_ENTITIES=1,2,3 in gk0_model.apm in the m.path run directory.
Model Lines versus NUM_ENTITIES

Inspecting the model files shows a lot of simple equations that could be reduced. For example, about 30% of the model lines are these equations:
    v54786=((exp(-951.5365149722506))/(sqrt(0.5231086080192687)))
    v54787=((exp(-1699.0345902817821))/(sqrt(0.5231086080192687)))
    v54788=((exp(-1123.400950839389))/(sqrt(0.5231086080192687)))
    v54789=((exp(-2752.8539627322))/(sqrt(0.5231086080192687)))
    v54790=((exp(-4449.244551756981))/(sqrt(0.5231086080192687)))

These could be replaced by a simple evaluation in Python to remove the variables and the equations. You may also see other opportunities as you inspect gk0_model.apm.
There is m.options.REDUCE=1 that automatically removes the simple assignment equations or m.options.REDUCE=2 or higher to successively scan and remove simple equations. However, you may want to do it in Python to avoid a large .apm file write and speed up the model compile time.


How does the universal set apply to the truth set?

Question:
If the truth set is identical to it's universal set, then the statement is logically true statement and it's called a tautology.

Can someone please explain the universal set to me in this context? I thought the universal set has to be defined for each problem, so how is it defined, for, say,
$\neg (p\wedge q)\iff (\neg p) \vee (\neg q)$
I'm not taking a class, just interested in learning more about math. Thanks
Comment: It would help if you gave a definition of truth set and universal set, I don't think it is common terminology.
Comment: @Calvin, that certainly is attractive! I will take some time to learn this mark-up  thanks for the great resource
Comment: Hi, I edited a typo (I think!) and made the symbols more pretty. You can learn how to do that yourself here: https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference . Please check if the symbols are correct now :)
Comment: The notions of truth sets or universal sets rarely if ever come up in mathematics. There are only a few basic rules of logic that you need to remember, e.g. if A implies B then not B implies not A (the contrapositive rule). To learn them, you might Google "software to learn the basic methods of proof."
Answer: The terms "truth set" and "universal set" aren't really common, but from context it sounds like they mean the following:

Fixing a propositional language $P$, the universal set associated to $P$ is the set of all valuations of propositional formulas built from $P$ (or essentially equivalently, the set of all maps $P\rightarrow\{True, False\}$), and for a formula $\varphi$ built from $P$ the truth set of $\varphi$ is the subset of the universal set consisting of all valuations making $\varphi$ true.

Now you are correct that $\varphi$ alone can't tell us what $P$ is - e.g. if $\varphi$ is $p\rightarrow q$, then $\{p,q\}$ and $\{p,q,r\}$ would each make sense. However, $\varphi$ does have a minimal language associated to it, namely the set of all propositional atoms which actually occur in $\varphi$, so we often default to that language.
Moreover, in this context - and indeed in many contexts - the choice of language doesn't matter:

Suppose $P_1,P_2$ are two languages each of which contains every propositional atom occurring in $\varphi$. Then $\varphi$ is a tautology in the sense of $P_1$ iff $\varphi$ is a tautology in the sense of $P_2$.

This is a good exercise: the key point is that whether or not a valuation $v$ makes a sentence $\varphi$ true is determined entirely by the restriction of $v$ to the propositional atoms actually occurring in $\varphi$. So we don't have to worry about the choice of language issue here.

That said, sometimes the choice of language does matter. For example, in the context of first-order logic it is important to specify the language when talking about the decidability or completeness of a theory: Presburger arithmetic, for example, is complete and decidable as a $\{+\}$-language but not as a $\{+,\cdot\}$-language.


Identifying features using two-sample t-test - resulting in over-fitting?

Question: Let's say that I have a binary classification problem, thousands of features. If I were to run a two-sample t-test on some of these features to figure out which ones differ significantly between classes (correcting for multiple-comparisons), and then use the significant features in an SVM model, will this result in substantial over-fitting? If so, what are alternative ways I can sub-select a group of features that best predict classes while maintaining generalization ability?
Answer: Selecting features by univariate comparisons is not the best way to do things. If you're not careful you can end up selecting a terrible model. Even if you don't do that, it is often the case that the best combinations of features are not the best features individually.
Here's an example to consider to show how these $t$-tests can lead you astray:
Suppose we have features $X_j = (X_{1j}, ..., X_{nj}) \sim \ \text{iid} \ \mathcal N(\vec 0, I_n)$ and labels $Y_i \sim \ \text{iid} \ Bern({1 \over 2})$, with $X_{j} \perp Y_i$ for all $i,j$. Say $i = 1, ..., n$ and $j = 1, ..., p$. Note that everything is independent so there should be no good model here.
Let's say that we perform $p$ two-sample $t$-tests and take the 10 predictors with the most significant tests. Just by chance there will be some predictors that seem fairly related to $Y$ (to be more precise, every null hypothesis is false but if we do enough tests we'll make a lot of type I errors). But clearly the variables that we've selected this way still aren't related to the response. If we found the cross-validated error rate (CVER) for this model AFTER doing the variable selection it would look pretty good, but this model would do horribly for a new observation because there isn't actually any relationship here. We've just found features that by chance seemed to be related in this sample. You could avoid this by cross-validating the entire model selection procedure. If you did that then you would see that you haven't found any good features.
Now here's an example where the features marginally are useless but together perfectly separate the response. Let $X_1, X_2 \sim \ \text{iid} \ Bern(1/2)$ and let $Y = I(X_1 \neq X_2)$, i.e. $Y$ is the indicator that $X_1$ and $X_2$ are not equal. Now we'll find that either $X_j$ alone is not helpful but together they perfectly predict $Y$. In R:
<code>set.seed(1)
n <- 1000
x1 <- rbinom(n, 1, .5)
x2 <- rbinom(n, 1, .5)
y <- factor(as.numeric(x1 != x2))

library(rpart) ## simple classification tree
m1 <- rpart(y ~ x1) ## doesn't do well
m2 <- rpart(y ~ x2) ## doesn't do well
m12 <- rpart(y ~  x1 + x2) ## does perfectly
</code>
How do we then properly select features?
One of the simplest ways is to use a stepwise regression. For SVMs there's recursive feature elimination which is similar in spirit. Another popular method is the lasso. You can also employ the grouped lasso to select or exclude multiple variables together (for instance, instead of selecting individual levels of a factor you could select or exclude the entire variable at once). You can probably find out more about these elsewhere on the site.
Comment: Thank you for the in-depth answer. I was able to get RFECV in SKLearn (recursive feature elimination with cross-validation) working. I gave it my classifier (SVC) and features, then used the resulting features in an SVM classifier. The accuracy is extremely high so I'm wondering how I can check for over-fitting? One issue I have is that I have orders of magnitude more features than samples.
Comment: You definitely don't want to look at the training accuracy as you can trivially get this to be 100%. Avoid over-fitting by only evaluating accuracy via CV or on a held out set. If the accuracy is really high on an independent test set then you've got a great classifier. If you've got $p >> n$ then you'll probably want to stick to simpler models, like maybe only a linear kernel SVM rather than radial kernel.
Comment: Would this pipeline make sense (based on what you've said):  1. Split my full data set into train and test.  2. Run recursive feature elimination with cross-validation on the training set.   3. Fit a linear-kernel SVM model with the newly identified features.   4. Test this model on my test set.


Reinforcement Learning: Actor Critic - Why is weight sharing possible?

Question: I was looking at Open Ai's actor-critic code and noticed that some of the neural network's weights are shared
class Policy(nn.Module):
    def __init__(self):
        super(Policy, self).__init__()
        self.affine1 = nn.Linear(4, 128)
        self.action_head = nn.Linear(128, 2)
        self.value_head = nn.Linear(128, 1)

        self.saved_actions = []
        self.rewards = []

     def forward(self, x):
        x = F.relu(self.affine1(x))
        action_scores = self.action_head(x)
        state_values = self.value_head(x)
        return F.softmax(action_scores, dim=-1), state_values

namely affine1.
I am not sure why this is possible, because, theoretically,

The actor should calculate a function from states to actions
The critic should calculate a function from states to reward

These are two different functions, thus I would expect separate weights.
The only reason I came up with was a projection of exterior information on the problem, which is "Embedding the state".
If that is the case, then who is to say it makes sense to use he same state embedding for both networks?
Why does weight sharing in actor critic make sense?
Answer: You can use weight-sharing almost any time you might want to make more than one prediction given some input. The shared layer(s) construct some representation which is useful for all the prediction tasks. This is effective because the loss from all down stream prediction tasks backpropagates to this layer, allowing faster and more efficient learning.

These are two different functions, thus I would expect separate
  weights.

The last layers "head" have different weights and thus compute different outputs.


Silent push notification with Firebase in iOS10

Question: I can't call method in background using silent push notifications.

Checked "Background Modes" > "Background fetch" and "Remote notification".
Add method to send data to Firebase.

func application(_ application: UIApplication, didReceiveRemoteNotification userInfo: [AnyHashable : Any], fetchCompletionHandler completionHandler: @escaping (UIBackgroundFetchResult) -> Void) {
        let appDelegate = UIApplication.shared.delegate as! AppDelegate
        appDelegate.viewController.sendData()
    }

Send push notification with content_available: true.

When app is in foreground, sendData() is called. But when in background, it isn't called.
Comment: can you copy-paste a FCM data template for silent notification?
Comment: Did you implement applicationReceivedRemoteMessage? It's more of a foreground thing but it needs to be implemented either way. Oh, and what is the implementation of sendData()?
Comment: I resolve this problem with using Firebase cloud messaging!
Answer: Try to set time_to_live = 0. It works for me.


joint PMF in a class of n students

Question: A class of n students takes a test in which each student gets an A with probability p, a B with probability q, and a grade below B with probability 1 − p − q, independently of any other student. If X and Y are the numbers of students that get an A and a B respectively, calculate the joint PMF pX,Y .
I found this answer online:
Let r = 1 − p − q
Then,
Prob( X = i, Y = j) = n!/(i!j!(n-i-j)!)(q^j)(r^(n-i-…....so on)
for j=0,1,2..., j=0,1, 2..., 0<=i+j<=n
However I'm not able to figure out how to get to this answer. Any help is appreciated!
Answer: The corresponding distribution is known as multinomial distribution.
The rationale is as follows:
Consider we want the first $i$ people  to get A, followed by the next $j$ people to get B and the rest get the other grades. The corresponding probability is $p^iq^jr^{n-i-j} $.
However, the original problem doesn't specify the first $i$ must get A and the next $j$ to get B. We just have to pick $i $ of them to get A, $j$ of them to get B. Hence, we multiply the term by $\frac{n!}{i!j! (n-i-j)!} $.

joint PMF in a class of n students