When should I balance classes in a training data set?

Question: I had an online course, where I learned, that unbalanced classes in the training data might lead to problems, because classification algorithms go for the majority rule, as it gives good results if the unbalance is too much. In an assignment one had to balance the data via undersampling the majority class.
In this blog however, someone claims that balanced data is even worse:
https://matloff.wordpress.com/2015/09/29/unbalanced-data-is-a-problem-no-balanced-data-is-worse/
So which one is it? Should I balance the data or not? Does it depend on the algorithm used, as some might be able to adept to the unbalanced proportions of classes? If so, which ones are reliable on unbalanced data?
Answer: The intuitive reasoning has been explained in the blogpost:

If our goal is Prediction, this will cause a definite bias. And worse,
  it will be a permanent bias, in the sense that we will not have
  consistent estimates as the sample size grows.
So, arguably the problem of (artificially) balanced data is worse than
  the unbalanced case.

Balanced data are good for classification, but you obviously loose information about appearance frequencies, which is going to affect accuracy metrics themselves, as well as production performance.
Let's say you're recognizing hand-written letters from English alphabet (26 letters). Overbalancing every letter appearance will give every letter a probability of being classified (correctly or not) roughly 1/26, so classifier will forget about actual distribution of letters in the original sample. And it's ok when classifier is able to generalize and recognize every letter with high accuracy.
But if accuracy and most importantly generalization isn't "so high" (I can't give you a definition - you can think of it just as a "worst case") - the misclassified points will most-likely equally distribute among all letters, something like:
<code>"A" was misclassified 10 times
"B" was misclassified 10 times
"C" was misclassified 11 times
"D" was misclassified 10 times
...and so on
</code>
As opposed to without balancing (assuming that "A" and "C" have much higher probabilities of appearance in text)
<code>"A" was misclassified 3 times
"B" was misclassified 14 times
"C" was misclassified 3 times
"D" was misclassified 14 times
...and so on
</code>
So frequent cases will get fewer misclassifications. Whether it's good or not depends on your task. For natural text recognition, one could argue that letters with higher frequencies are more viable, as they would preserve semantics of the original text, bringing the recognition task closer to prediction (where semantics represent tendencies). But if you're trying to recognize something like screenshot of ECDSA-key (more entropy -> less prediction) - keeping data unbalanced wouldn't help. So, again, it depends.
The most important distinction is that the accuracy estimate is, itself, getting biased (as you can see in the balanced alphabet example), so you don't know how the model's behavior is getting affected by most rare or most frequent points.
P.S. You can always track performance of unbalanced classification with Precision/Recall metrics first and decide whether you need to add balancing or not.

EDIT: There is additional confusion that lies in estimation theory precisely in the difference between sample mean and population mean. For instance, you might know (arguably) actual distribution of English letters in the alphabet $p(x_i | \theta)$, but your sample (training set) is not large enough to estimate it correctly (with $p(x_i | \hat \theta)$). So in order to compensate for a  $\hat \theta_i - \theta_i$, it is sometimes recommended to rebalance classes according to either population itself or parameters known from a larger sample (thus better estimator). However, in practice there is no guarantee that "larger sample" is identically distributed due to risk of getting biased data on every step (let's say English letters collected from technical literature vs fiction vs the whole library) so balancing could still be harmful.
This answer should also clarify applicability criteria for balancing:

The class imbalance problem is caused by there not being enough
  patterns belonging to the minority class, not by the ratio of positive
  and negative patterns itself per se. Generally if you have enough data, the "class imbalance problem" doesn't arise

As a conclusion, artificial balancing is rarely useful if training set is large enough. Absence of statistical data from a larger identically distributed sample also suggests no need for artificial balancing (especially for prediction), otherwise the quality of estimator is as good as "probability to meet a dinosaur":

What is the probability to meet a dinosaur out in the street?
1/2 you either meet a dinosaur or you do not meet a dinosaur
Comment: I think besides the explanation of the issue, the important take-away from this answer is that one should try unbalanced first and check its results and only if necessary do the balancing and check its result. +1
Comment: @ChristosK. Thanks for some clarifications. It's not the same what I meant but the approach is very similar. Usual recommendation for applicability of k-fold is to do it when your initial sample is "kinda small". Not sure, but folding shouldn't hurt anyways - it's just it takes more runs, and the less you care about prediction, the less you care about generalization/performance as tautological as it sounds :). But overall - k-fold means less bias essentially.
Comment: @ChristosK. Oh, and as a warning, spam/not-spam ratio might be a non-stationary random variable on its own. With all those "fake news", "russian trolls" and other stuff I'd be careful about such assumptions - ratio could be biased too. You might want to estimate PrecisionRecall on your classifiers first, if something is under-sampled - I'd rather collect/generate(?) more data.
Comment: So, in other words, with evenly distributed classes to the training subset the model will loose its accuracy in unseen data, right? But, in the opposite case, where you try to randomly extract entries of a dataset for your training/testing subsets, will your classifier perform better?
Comment: @ChristosK. As many stated, when you see the problem as classification, it's hard to reason about prediction. In any case, if you remove bias (sample "randomly") - you need a bigger sample to improve performance. It's just "usually" sample is big enough to preserve semantics, so overbalancing would only hurt and act like regularizing hammer that "flattens" everything without proper consideration. Also, as dinosaur metaphor suggests, "balanced" doesn't mean "even" - you do proper balancing only when you know that some probabilities are misrepresented in a "random" sample.
Comment: @ChristosK. Worth mentioning that I took accuracy as a metric only for clarity, it's sensitive to noise as much as to useful "signal" content - loss is more appropriate in practice (especially for classification). With that in mind I'd say that overbalancing ("too even") hurts generalization in terms of underfitting.
Comment: @ChristosK. Oh and if you need a direct answer - logically (assuming axiom of excluded middle) it's in your question already :), given that "too even" is not "random".
Comment: @dk14 Thank you for the nice explanations. When I wrote evenly what I had in my mind wasn't a totally flat trainset with 50%spam-50%legit, for the example of a spam detector, but rather a trainset that maintained the spam-legit ratio of the initial dataset. So, if I get it right, in case someone wants to stay away from predictions and just want to compare the performance of classifiers it's better to try k-fold instead of splitting since the k-fold will make sure that there will be no bias in the trained classifiers, right?
Answer: Consistent with @kjetil-b-halvorsen's  comment, the rapid adoption of machine learning has confused researchers about prediction vs. classification.  As I described in more detail here, classification is only appropriate in a minority of cases.  When the outcome is rare (or too common), probabilities are everything because in that case one can only reasonably speak about tendencies, not about predicting individual occurrences.
In statistics, we learned a while back that any method that requires one to exclude some of the data is highly suspect.  So the goal of balancing outcomes is misplaced.  Prediction of tendencies (probabilities) does not require it.  And once you estimate a probability you can make an optimal decision by applying the utility/cost/loss function to the predicted risk.
Answer: Depends on what you want to achieve from the classification?
Say it is cancer v/s non cancer, then detecting cancer is vital. However since non-cancer will form majority of your data the classifier can essentially send all cases to non-cancer class and get very high accuracy. But we can't afford that, so we essentially down sample non-cancer cases, essentially moving the decision boundary away from cancer region into the non-cancer region.
Even in use cases where accuracy is our only aim, balancing can be essential if the test time balance is expected to be different from train time.
For example say you want to classify mangoes and oranges, you have a training dataset with 900 mangoes and 30 oranges, but you expect to deploy it in a marketplace with equal mangoes and oranges, then ideally you should sample in the expected sample ratio to maximize accuracy.
Comment: But the problem with the cancer example *is to view it as classification*, it should be treated as *risk estimation*. Then the apparent problem with unbalanced classes disappears, see https://stats.stackexchange.com/questions/127042/why-isnt-logistic-regression-called-logistic-classification
Comment: That is what I understood from the lectures I had. However, I don't understand when balancing can be bad, as this blog post suggests. Why would it ever be bad to balance, if sufficient data points remain for each class?
Comment: Sorry, but in your analogy, what does the market fruit distribution has to do with model accuracy? You either learned to separate mangoes from oranges, or not. In other words, you should be able to deploy the *same model* on a orange-only or mangoes-only market.
Answer: When your data is balanced you can prefer to check the metric accuracy. But when such a situation your data is unbalanced your accuracy is not consistent for different iterations. You need to concentrate more metrics like Precision(PPR), Recall(sensitivity). This two metrics should be balanced when compare. Also you should have to check F1-Score which is harmonic mean of Precision and recall. This is applicable for all the machine learning algorithms


Spivaks Calculus. Conics Section. Change of coordinates system

Question: I have a question regarding the conics section of Spivak calculus and  the way he derives the final equation. I understand everything that he is doing up until:
Now we have to choose coordinate axes in the plane P. We can choose L as the first axis, measuring distances from the intersection Q with the horizontal plane (Figure 5); for the second axis we just choose the line through Q parallel to our original second axis. If the first coordinate of a point in P with respect to these axis is x, then the first coordinate of this point with respect to the original axes can be written in the form
$$\alpha x+\beta$$
for some $\alpha$ and $\beta$
Now, I understand everything before that, and why he is changing coordinates from the 'standard' ones, xyz, to coordinates within the plane, so he can express the intersection of the cone and the plane in a equation. But the thing is how does he justify that there is some $\alpha$ and some $\beta$ that satisfy this condition for all x ? He just claims that such values exist, with no proof.
For anyone not familiarized with this chapter, i leave it down here so you can read it.
One of the simplest subsets of this three-dimensional space is the (infinite) cone illustrated in Figure 2; this cone may be produced by rotating a "generaling line," of slope C say, around the third axis.
For any given first two coordinates x and y, the point (x,y,0) in the horizontal plane has distance $$\sqrt{x^2+y^2}$$ from the origin, and thus
$$\tag1 (x,y,z)\text{ is in the cone if and only if }z=\pm C\sqrt{x^2+y^2}$$.
We can descend from these three-dimensional vistas to the more familiar two-dimnensional one by asking what happens when we intersect this cone with some plane P (Figure 3).
If the plane is parallel to the horizontal plane, there's certainly no mystery--the intersection is just a circle. Otherwise, the plane P intersects the horizontal plane in a straight line. We can make things a lot simpler for ourselves if we rotate everything so that this intersection line points straight out from the plane of the paper, while the first axis is in the usual position that we are familiar with. The plane P is thus viewed "straight on," so that all we see (Figure 4) is its intersection L with the plane of the first and third axes; from this view-point the cone itself simply appears as two straight lines.
In the plane of the first and third axes, the line L can be described as the collection of all points of the form
$$(x,Mx+B)$$,
where M is the slope of L. For an arbitrary point (x,y,z) it follows that
$$\tag2 (x,y,z)\text{ is in the plane }P\text{ if and only if }z=Mx+B.$$
Combining (1) and (2), we see that (x,y,z) is in the intersection of the cone and the plane if and only if
$$\tag{$*$} Mx+B=\pm C\sqrt{x^2+y^2}.$$
Now we have to choose coordinate axes in the plane P. We can choose L as the first axis, measuring distances from the intersection Q with the horizontal plane (Figure 5); for the second axis we just choose the line through Q parallel to our original second axis. If the first coordinate of a point in P with respect to these axis is x, then the first coordinate of this point with respect to the original axes can be written in the form
$$\alpha x+\beta$$
for some $\alpha$ and $ \beta$. On the other hand, if the second coordinate of the point with respect to these axes is y, then y is also the second coordinate with respect to the original axes.
Consequently, (*) says that the point lies on the intersection of the plane and the cone if and only if
$$M(\alpha x+\beta)+B=\pm C\sqrt{(\alpha x+\beta)^2+y^2}.$$
Although this looks fairly complicated, after squaring we can write this as
$$\alpha^2C^2y^2+\alpha^2(M^2-A^2)x^2+Ex+F=0$$
for some E and F that we won't bother writing out. Dividing by $\alpha^2$ simplifies this to
$$C^2y^2+(C^2-M^2)x^2+Gx+H=0.$$
Answer: You're correct that his presentation is a bit confusing. He's using $x$ both for the original coordinate (in the $xyz$ coordinate system) and to represent the new coordinate along $L$ in the plane $P$. So to clarify, let's use $x'$ as the indicated (signed) distance from the point $Q$ to the point (which I'll call $R$ for now) on $L$. Then Spivak's claim is that the original $xz$-coordinates of $R$ are linear expressions in $x'$. Let $S$ be the point on the $x$-axis directly below $R$.
If you'll allow me to introduce a little trigonometry for a moment, let $\theta$ be the angle between the two lines, and let $Q=(\beta,0)$. Then $\alpha=\cos\theta = QS/QR$, so $QS = \alpha(QR) = \alpha x'$. Since $S=(x,0)$, we have $x-\beta = \alpha x'$, so $x=\alpha x' + \beta$. Of course, we do not need the definition of the cosine function to observe that as $R$ moves along the line $L$, the ratio of the signed distances $QS/QR$ stays constant; this is just similar triangles.
Now proceed with his argument, substituting $x=\alpha x'+\beta$. The $x'y$-coordinates in the plane $P$ are, of course, not the usual $xy$-coordinates coming from $3$-space.
Last comment: Since the equation of $L$ is $z=Mx+B$, then we have $\beta = -B/M$ and $M=\tan\theta$, so $\alpha=\cos\theta = (\pm)\frac1{\sqrt{M^2+1}}$.
Comment: You're right. I didn't pay careful enough attention. I will edit.
Comment: LOL, you're welcome. Depending on what edition you have, I am to blame for lots of exercises and a bit more.
Comment: Oh so basically alpha and beta come from the equation of the plane, but wouldn't that alpha and beta be the M and B from the ecuation of L? If I understood correctly your answer, these alpha and beta give the equation of the plane, and it relates the x coordinates and the z coordinates, I don't understand how it relates the "new coordinates" that spivak introduces. I think what you are saying is that the plane can be written as z = Mx + B , but I have already understood that part. If I'm not understanding your answer please explain, I'm kind of lost here
Comment: I taught a course out of Spivak's book 15 years, but I never taught this section. I would have complained about the notation if I had :P
Comment: Oh god you are Ted Shifrin , the one from the Preface. I never realized. Thanks so much for your help !!!!
Comment: @hiroshin The usual equation of the ellipse we're trying to derive is a cartesian equation in the plane $P$.
Comment: Thanks! I finally understood!
Comment: Great explanation, I wish it had been included into the appendix.
Comment: @TedShifrin But what do we gain from introducing coordinates along the plane P?
Comment: Thank you very much!


installing simplejson on the google appengine

Question: Super nub question time! I am trying to use simplejson on the google appengine. In a terminal on my machine I have simplejson installed and working. But my when I try to import it in a script running on the appengine I get an error saying no such library exists. If open the interactive console on my machine (from the link on http://localhost:8080/_ah/admin) and type "import simplejson" I get:
Traceback (most recent call last):
  File "/home/chris/google_appengine/google/appengine/ext/admin/init.py", line 210, in post
    exec(compiled_code, globals())
  File "", line 1, in
ImportError: No module named simplejson
Any thoughts?
Comment: nevermind, here it is:

from django.utils import simplejson
Answer: Look in django package:
from django.utils import simplejson as json
obj = json.loads(json_string)

Since Sdk 1.4.2 Json can be imported with the following statement:
import simplejson

Note that on Python 2.7 runtime you can use the native Json library.
Comment: Django is part of the SDK so i don't see problem to use it.
You could also drop in simplejson in your project directory and use it directly.
Remember that c speedup won't work, App Engine doesn't allow python
extensions written in c
Comment: I've seen this answer given frequently to this question, but it seems a bit odd to me to import anything from the django package if the user is not using Django.  Should non-Django users just add the generic simplejson directly to their project?  Or is the version supplied by Google in the django package somehow better tuned to GAE then the generic version of simplejson?  Since I use json so heavily I want to be sure that I am using the best version.
Comment: I haven't looked specifically at how Django packages simplejson, but importing only part of Django often leads to importing _all_ of Django.  I don't know whether this is the case here, but if so, you might be better off just using simplejson in your project directory.
Answer: You no longer need to use the django package for simplejson on Google App Engine.
import simplejson as json

This is expecially handy for avoiding the flurry of warnings about django versions in your log file.
Comment: Note that with python 2.7 you have to use `json` because `simplejson` is c-accelerated.


How to find two repetitive elements in a unsorted array on python

Question: In this problem I had to find duplicates from 1-N integers in an array given that N+2 size array with N no of integers, below is my code and error it is showing,
Please tell me what is the wrong
def find_duplicates(a,n):
    for i in range(n+2):
        if(a[abs(a[i])]>0):
            a[abs(a[i])]= - a[abs(a[i])]
        else:
            print(abs(a[i]), end= " ")
    print("\n")

if __name__ =='__main__':
    cases=int(input())
    for _ in range(cases):
        n=int(input())
        a=list(map(int,input().strip().split()))[:n+2]
        find_duplicates(a,n)

error is
Wrong Answer. !!
Possibly your code doesn't work correctly for multiple test-cases (TCs).
The first test case where your code failed:
Input:
2
1 1 2 2

Its Correct output is:
1 2

And Your Code's output is:

1 2 4 2 8 15 3 6 2 1 7 6 19 5 7 16 9 2 2 4 35 30 30 2...

Comment: I did `find_duplicates([1, 1, 2, 2], 2)` and it gave me `1 2`, so it works for me
Answer: Your code already works, you can see the first two outputs are "1 2" which are correct. However, the issue is you print all outputs of testcases in one line which may possibly cause the error. Please try to go to a new line before solving a new testcase as below:
if __name__ =='__main__':
    cases=int(input())
    for _ in range(cases):
        n=int(input())
        a=list(map(int,input().strip().split()))[:n+2]
        find_duplicates(a,n)
        print()

Comment: Thanks it worked.. :) but I forgot to add print("\n") at the end of the function find_duplicates I used before, can't it done using print("\n") because when I was using print("\n) for new line it was again showing error of no output
Comment: Try to use print() instead of print("\n")


Placing Drupal core/modules code outside public_html

Question: I've been trying to improve my sites security as I build a Drupal 8 update for my page. One thing I've seen recently is keeping your core code outside the web accessible docroot folder. I've also heard about keeping a copy of the code setup on git so you can easily bring down a clean copy if somehow someone changes something.
What is the current best practice for such a thing? On my server I have an account for the page, within which is public_html where anything actually being sent out to the world is kept. Is it possible to have the Drupal code one level down and then have things run from public_html?
Comment: Nah I wouldn't . there are 1000s of sites that live in public domain .. unless your site deals with extremely sensitive data  then dont worry just keep an eye on security patches / updates.
Answer: There is some work being done to move as much of core out of DOCROOT, but this is complicated by assets that need to be web accessible, mainly JS, CSS, and images.  So, moving core/modules outside DOCROOT is tough because many of those have all three.
Current thinking is to place the vendor directory outside of DOCROOT to help reduce attack surface area.  Since using composer for all site building is also essentially a best practice now, the drupal-composer/drupal-project template does this, and also will place modules, themes, libraries, and other Drupal stuff in the proper place, while keeping vendor non-web accessible, as it contains the non-Drupal specific dependencies.
Answer: You just move your code into under public_html folder with the same permissions.


How to commit the last un-commit action in git

Question: Most of the previous threads are asking how to undo the last commit. But I was wondering if is possible to undo the last uncommit. I ran git reset --hard HEAD^ but it didn't uncheckout all the files that were committed and now they are deleted.
How to commit the last un-commit action?
Comment: what is a uncommit action?
Comment: What do you mean by "uncheckout all the files"? Files are not checked out. Branches are. Please explain exactly what the results you saw, and how that differs from your desired result.
Answer: Just do git checkout - . without resetting anything


Comunicação TEF

Question: Preciso integrar o TEF no meu sistema. Alguém faz ideia de como posso começar isto? Existe algum software de código aberto para fazer esta integração?
Encontrei umas informações na internet que falava sobre a PayGo, esta seria uma opção viável?
Meu sistema é Web, uso AngularJS e meu BackEnd em PHP, tenho um módulo de Vendas/Frente de loja.
Comment: Não é necessário desenvolver uma classe JS para comunicação com o PayGo.
A [NTK Solutions](http://www.ntk.com.br) disponibiliza uma API para TEF, POS e e-commerce. Chama-se ControlPay.
Com essa API você pode acionar o TEF diretamente da sua aplicação, via web service.
Quando a transação termina no PDV você recebe um callback na URL de retorno cadastrada no ControlPay.
Answer: Faça o cadastro na PayGo e baixe a documentação com o roteiro de testes, documentação e o Gerenciador Padrão para testes. Resumindo: a implementação do TEF consiste basicamente na troca de arquivos com o Gerenciador Padrão (A documentação da PayGo explica como funciona). Você precisará desenvolver uma classe em Javascript para se comunicar com o GP. Você pode pesquisar por classes desenvolvidas em JAVA e converte-las para Javascript (Foi o que eu fiz).
Você precisará desenvolver ou ter uma versão do seu Frente de Loja em desktop desenvolvido em NodeJS para criar os respectivos arquivos nas pastas do Gerenciador Padrão. Recomendo que vc utilize o Electron para desenvolver a versão desktop. https://electron.atom.io/

Comunicação TEF


With the use of Apex code, how can I select all ''OpportunityProducts' objects which are belongs to particular 'Opportunity Id'

Question: I am way to customizing 'Sales' application that belongs to 'salesforce.com' platform.
Is there any way to select all the 'OpportunityProducts' objects which are belongs to particular 'Opportunity Id' ?
[SELECT Id FROM  OpportunityProduct WHERE Opportunity =:opportunitId];

When I execute above code for select those 'OpportunityProduct', I got following error. If any one have some idea please update me. Thanks.
Save error: sObject type 'OpportunityProduct' is not supported. If you are attempting to use a custom object, be sure to append the '__c' after the entity name. Please reference your WSDL or the describe call for the appropriate names.

Comment: I have get done this with the following Apex code.
Internally they have used identifier 'OpportunityProduct' name as a 'OpportunityLineItem'. Therefore for this query I have to use 'OpportunityLineItem' as a object name.

List opProDucts = [SELECT Id FROM OpportunityLineItem WHERE OpportunityId =:opportunitId];
Answer: Another way to get this done when you need the actual products, not just the line items, is as follows.  First get your opportunities:
List<Opportunity> opps = [SELECT Id, Name FROM Opportunity LIMIT 1000];

Then loop through to create a list of opportunity Ids
List<Id> oppIds = new List<Id>();
for(Opportunity o : opps)
{
   oppIds.add(o.Id);
}

Now get your actual products that belong to your opportunities...
List<OpportunityLineItem> oppProds = [SELECT Id, PricebookEntry.Product2.Name, PricebookEntry.Product2.Family
   FROM OpportunityLineItem
   WHERE OpportunityId IN :oppIds];

Hope that helps.


"Major" progressions & "Modal" voicings (tonal vs modal) when building chords & progressions

Question: im not asking "what i can or cannot play" as all music is subjective, but im asking for "theory" or "technical" reasonings.
people often say that they replace the 4/11 over a Major chord, primarily with the tonic Major chord, with the #4/#11, so to avoid the tension between the M3 guide tone. so what they are really saying, in technical terms, is that they prefer a Lydian sound over an Ionian sound?
My confusion comes from the fact that if we build a tertian harmony sequence of diatonic notes, over a CMaj scale chord, in the key of C, you will have the "avoid note" of [F] as the diatonic 11th (ie; C-E-G-B-D-F). Its as if the chord is its own worst enemy, if that makes any sense lol The tonic chord is meant to be "at home", yet if built up until the 11th is reached naturally, it creates this conflicting sound.
This is mostly regarding the tonic Major, as the other true Major chord of the scale is the subdominant (IV), which is naturally the Lydian chord of the scale. So technically, the (IV) cannot be voiced in such a way to naturally create this "avoid" note scenario, if one were to use just the notes of the corresponding parent scale. (ie; using the notes of C Ionian or F Lydian, etc.)
However, if in the key of CMaj, you decided to play [F Ionian] over the [IV] chord, you would then be intentionally creating the "avoid" note 4/11 scenario?
So, if you have a typical [1-5-6-4] pop or rock progression in CMaj & you voice the 1 chord as having the [11], that means you are strictly implying an "Ionian" sound? However, if you voiced it to have a [#11], you would imply a Lydian sound, just over that 1 tonic chord, although the entire progression or piece would still be considered a CMaj key?
If a progression was "modal" on the other hand, such as a [C Ionian] progression, then that means you would SPECIFICALLY use notes like the [4/11] & the [7]... not only over the [I] chord, but as often as needed over/under other chords in the progression as well? For example, a pure Ionian progression could be [CMaj(add11)-> FMaj-> CMaj-> bdim]?
So when other sites & sources describe the [4/11] as an avoid note, they are perhaps speaking from a primarily "tonal" chord voicing or progression standpoint & are simply saying, in reality, they prefer a Lydian sound, rather than an Ionian voicing?
I know nothing is stopping us from using a CMaj(add11) chord, in a "tonal" progression, as mentioned above, though... if it sounds good to you, that's all that matters, but Im simply trying to understand things from a theory, technical or "historical" point.
thanks
Comment: There's an entire book called *The Lydian Chromatic Concept* which attempts to reconcile your concerns here.
Answer: I'm not fully sure what you're asking here, but I think I've considered some of these problems myself. First of all, from my understanding, avoid notes aren't referring to chords built on every degree of a scale. The avoid note for C major is only for the C major chord. These are notes to avoid when soloing over specific chords, rather than applying to the whole key. I came across this concept on my own sort of. I wanted to figure out what makes some harmonies sound bad (to me). I ended up finding that I can put every note of a key into a "chord" and it sounds good, up until the F (in C major or A minor). That note only sounded good to me as a passing note or a suspension note resolving down. That's essentially the avoid note concept, I think. I started trying to rationalize it in other way too. Those notes are related by 5ths built up from the root. For example, C-G-D-A-E-B. The next note is not F but F#. In fact, even though F looks close to C on the circle of 5ths, it's actually the very furthest note when you go up by 5ths like that. To me, this helps explain why a #4 sounds much better than the natural 4, because the relation by consonant intervals (5th being more consonant than a 4th) is much more distant. Another "side effect" of this that I notice is that if the 4th is emphasized too much, it can sound like it's the actual root of the chord, thus making it sound less like a I chord and more like a IV chord, because that creates the 5th relation with C, but now C isn't the lowest note in the chain of upward 5ths.
This might be inaccurate or unhelpful, but this is basically my personal experience with the concept from my own perspective. It's actually a huge part of my concept of harmony, because it has helped me learn to build the chords I like and what notes go well over them.


How to visualize Bayesian goodness of fit for logistic regression

Question: For a Bayesian logistic regression problem, I have created a posterior predictive distribution. I sample from the predictive distribution and receive thousands of samples of (0,1) for each observation I have. Visualizing the goodness-of-fit is less than interesting, for example:

This plot shows the 10 000 samples + the observed datum point (way in the left one can make out a red line: yea that's the observation). The problem is is that this plot is hardly informative, and I'll have 23 of them, one for each data point.
Is there a better way to visualize the 23 data points plus there posterior samples.

Another attempt:

Another attempt based on the paper here
Comment: See [here](http://pymc-devs.github.com/pymc/modelchecking.html#goodness-of-fit) for an example where the above data-vis technique works.
Comment: So you have 23 data points, and how many predictors?  And is your posterior predictive distrubtion for new data points or for the 23 you used to fit the model?
Comment: Your updated plot is close to what I was going to suggest. What is the x-axis representing though? It appears you have some points super-imposed - which with only 23 seems unnecessary.
Comment: @AndyW the x-axis is temperature, and like you guessed, the same temperature was recorded for possibly more than 1 observation.
Comment: @probabilityislogic The only predictor is temperature, shown on the  x-axis above (second plot). The posterior predictive distribution is currently for the observed data points, though it is possible to sample it for arbitrary temperature.
Comment: That is alot of wasted space IMO! Do you really only have 3 values (below 0.5, above 0.5, and the observation) or is that just an artifact of the example you gave?
Comment: It's in fact worse: I have 8500 0s and 1500 1s. The graph just pushes these values to make a connected histogram. But I agree: lots of wasted space. Really, for each data point I can reduce it to a proportion (ex 8500/10000 ) and a observation (either 0 or 1)
Comment: Thanks @Cam.Davidson.Pilon I came here following the link on your book. One question I have is, why is there no mention of the [**marginal likelihood**](http://en.wikipedia.org/wiki/Marginal_likelihood) (model evidence) $p(D|m)$ when discussing if the model is a good fit? Wouldn't also that help with the visualization? (in particular in the comparison between possible models for predicting defects that you have in the book).
Comment: @user023472 Yes, that is good suggestion. Marginal likelihoods, and specifically Bayes factor, are both good tools of goodness-of-fit. For V2.0 perhaps =)
Comment: hey @Cam.Davidson.Pilon one question about this, using [ml classification metrics][https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics] might provide you with other ways of visualizing the precision of the model, such as AUC(PR). is there a reason why not to use those? thanks !
Answer: I have a feeling your not quite giving up all the goods to your situation, but given what we have in front of us lets consider the utility of a simple dot-plot to display the information.

The only real thing to not here (that aren't perhaps default behaviors) are:

I utilized redundant encodings, shape and color, to discriminate between the observed values of no defects and defects. With such simple information, placing a dot on the graph is not necessary. Also you have a problem when the point is near the middle values, it takes more look-up to see if the observed value is either zero or one.
I sorted the graphic according to observed proportion.

Sorting is the real kicker for dot-plots like these. Sorting by values of proportion here helps easily uncover high residual observations. Having a system where you can easily sort by values either contained in the plot or in external characteristics of the cases is the best way to get the bang for your buck.
This advice extends to continuous observations as well. You could color/shape the points according to whether the residual is negative or positive, and then size the point according to the absolute (or squared) residual. This is IMO not necessary here though because of the simplicity of the observed values.
Comment: I do like this solution and content, I'm just waiting on other submissions. Thanks Andy.
Comment: @Cam.Davidson.Pilon - I'm waiting on other submissions too! Because your model only has one predictor - sorting by the predicted proportion of defects would be synonymous as sorting by temperature  (assuming a monotonic effect - as it appears in your graph). Perhaps someone will come along though with another solution that effectively allows one to see both the predicted proportion and original temperature (or something completely different). This display is good for seeing bad predictions, but is not very good for things like seeing non-linear effects.
Comment: I'm happy to award the bounty to you. Sorting is the key to presenting it, and the paper linked from your previous post is what I'll be using. Thanks!
Answer: The usual way to visualising the fit of a Bayesian logistic regression model with one predictor is to plot the predictive distribution together with the corresponding proportions. (Please, let me know if I understood your question)
An example using the popular Bliss' data set.

Code Below in R:
library(mcmc)

# Beetle data

ni = c(59, 60, 62, 56, 63, 59, 62, 60) # Number of individuals
no = c(6, 13, 18, 28, 52, 53, 61, 60) # Observed successes
dose = c(1.6907, 1.7242, 1.7552, 1.7842, 1.8113, 1.8369, 1.8610, 1.8839) # dose

dat = cbind(dose,ni,no)

ns = length(dat[,1])

# Log-posterior using a uniform prior on the parameters

logpost = function(par){
var = dat[,3]*log(plogis(par[1]+par[2]*dat[,1])) + (dat[,2]-dat[,3])*log(1-plogis(par[1]+par[2]*dat[,1]))

if( par[1]>-100000 ) return( sum(var) )
else return(-Inf)
}

# Metropolis-Hastings
N = 60000

samp <- metrop(logpost, scale = .35, initial = c(-60,33), nbatch = N)

samp$accept

burnin = 10000
thinning = 50

ind = seq(burnin,N,thinning)

mu1p =   samp$batch[ , 1][ind]

mu2p =   samp$batch[ , 2][ind]

# Visual tool

points = no/ni
# Predictive dose-response curve
DRL <- function(d) return(mean(plogis(mu1p+mu2p*d)))
DRLV = Vectorize(DRL)

v <- seq(1.55,2,length.out=55)
FL = DRLV(v)

plot(v,FL,type="l",xlab="dose",ylab="response")
points(dose,points,lwd=2)

Comment: @Cam.Davidson.Pilon I am sorry, my reputation does not allow me to include plots. But the idea is to plot the whole dose-response curve together with the observed proportions.
Comment: I've added the picture. You assume a different structure for the data in which the OP's does not directly extend to your example. The OP's data would be like if your `ni = 23` and `no = 7` and each of the 23 individuals has a different `dose`. You could make a similar plot for the OP's data though, (points are either placed at 0 or 1 on the Y axis, and you plot the function). See some examples of similar plots for logistic regression in the references [I give on this answer](http://stats.stackexchange.com/a/45099/1036).
Comment: @AndyW ah the papers you link are quite useful! I'll have to take a closer look at those to see if I can apply them.
Answer: I am responding to a request for alternative graphical techniques that show how well simulated failure events match observed failure events.  The question arose in "Probabilistic Programming and Bayesian Methods for Hackers " found here.  Here's my graphical approach:

Code found here.
Comment: Interesting -- can you offer any arguments on why to use this technique? Thanks for sharing!
Comment: This is a probabilistic, not a deterministic result.  Therefore, I looked for a representation that conveyed several things: 1) the range of observed and predicted events; 2: the probability distribution of the predicted failures; 3) the probability distribution of predicted non-failures; and 4) ranges where failure is more likely, non-failure is more likely, and ranges where failure and non-failure likelihoods overlap.  This graph shows does all that to my eyes.
Comment: A few more additions/clarifications: 1) the **temperature** range of observed and predicted events; 5) actual observed failures and non-failures
Answer: I arrive here after reading "Probabilistic Programming and Bayesian Methods for Hackers". Disclaimer not sure if I'm committing a crime here, but as we have a "prediction" model with a classification problem, one way to evaluate and visualize its goodness is to apply some standard ML classification eval plots. I usually use AUC and AUCPR plots, but there's also DET curve as well. we base these plots on the posterior_probability given by the sampling procedure.

Another alternative (the one that I like the most) is the "precision-recall threshold curves" you can clearly see here how "good" your prediction probability is, given a different probability threshold. ( I think is the closest to the one that is in the book and publication)

if you also define some probability threshold you can evaluate the confusion matrix among other metrics. At least for me, it is clearer when I have the AUC score, but again, maybe I'm getting too ahead of myself using this method for a bayesian model.
code here


between-group multinomial test

Question: I'm having some trouble to decide what is the best method to analyze a between-groups / multinomial problem.
For simplicity, I'll use an example:
Let's say I want to compare basketball shooting abilities between boys and girls.
Each child has one minute to throw as many balls as possible to the basket.
Scoring is given for each throw as -1, 0 or 1: did not reach basket, reached but missed, or made the shot - respectively. So in the end I have a table such as this one:

subject
group
-1
0
1

1
boy
3
8
2

2
girl
2
6
9

...

My initial though was to sum all outcomes for each group, and then use a chi^2 test. I wonder if there's another method which is preferable?
Thanks!
Answer: If you want to test whether there is an overall difference in shooting abilities between boys and girls you could use the improved Wilcoxon-Mann-Whitney test by Brunner & Munzel that allows tied values. In R you can do this with the function bmp from the WRS package:
# sample dataset:
> basket <- list("boy" = c(rep(-1, 3), rep(0, 8), rep(1,2)),
                 "girl" = c(rep(-1, 2), rep(0, 6), rep(1, 9)))

# load the package
> source("https://dornsife.usc.edu/assets/sites/239/docs/Rallfun-v38.txt")

# test:
> bmp(basket$boy, basket$girl)
$n1
[1] 13

$n2
[1] 17

$test.stat
[1] 2.113882

$phat
[1] 0.6923077

$dhat
[1] -0.3846154

$s.e.
[1] 0.09097372

$p.value
[1] 0.04356367

$ci.p
[1] 0.5059565 0.8786589

$df
[1] 27.99999

$summary.dval
        P(X<Y)    P(X=Y)    P(X>Y)
[1,] 0.5294118 0.3257919 0.1447964

You see this test tells us that the distributions of shooting capabilities between girls and boys do indeed differ. Specifically, we are more likely to observe lower values among boys because P = 0.69 and the confidence interval does not contain 0.50 so we can reject H0: P = 0.5.
Another approach of testing general difference is to use Cliff's improvement of the Wilcoxon-Mann-Whitney test again allowing tied values. This test has a better control over Type 1 error probability than the former test and keeps it below 5%.
> cidv2(basket$boy, basket$girl)
$n1
[1] 13

$n2
[1] 17

$d.hat
[1] -0.3846154

$d.ci
[1] -0.671372675  0.002307866

$p.value
[1] 0.06

$p.hat
[1] 0.6923077

$p.ci
[1] 0.4988461 0.8356863

$summary.dvals
        P(X<Y)    P(X=Y)    P(X>Y)
[1,] 0.5294118 0.3257919 0.1447964

With this test we would fail to reject H0 that the distributions are identical (P = 0.5) because although P = 0.69 the confidence interval for P contains in this case 0.5 and the confidence interval for the difference between the probability of observing higher values among boys and the probability of observing higher values among girls contains 0.

between-group multinomial test


Show block in /user/* page, not /user/*/edit

Question: From the Drupal 8 administration, I configured a block (block created from a custom module) for display it in /user/* pages.

It works; I see this block in /user/* and /user/*/edit pages.
Now, I want to display my block only from /user/* pages (the view mode), not from /user/*/edit pages (the edit mode).
Can I do that from the Drupal Administration page? I tested it adding "!/user/*/edit" but this has no effect.

If I can't, how can I do that from my custom module?
Comment: You can write your own condition plugin class. It has a method that returns true if the block should be visible.
Comment: Try using /user/*/view instead.
Comment: Not work ;) with "/user/*/view", the block not showed from /user/4 or /user/4/edit by example
Comment: It could be that this condition plugin is too simple for your use case. Fortunately it is fairly easy to create a custom condition plugin in drupal 8.
Comment: I think too..what is a good way for manage this case from a custom module ? The block is already defined from my custom module in a Block Class
Answer: If you use this module then the exclamation prefix would work to exclude paths: https://www.drupal.org/project/block_exclude_pages
Answer: For Drupal 8 install the Route Condition Module and use the entity.user.canonical condition.
Answer: For drupal 7 users you can use Context module and simple add page to path /user/* and ~/user/*/edit/ to exclude edit page.
Answer: If it's your own custom block plugin, you can have your build function return null on node edit pages.
Answer: There is a module called Route Condition that allows you to specify exact routes (pages) to display blocks on. So for the user profile page the route would be user.entity.canonical (which is actually the example on the module page.
Answer: sometimes block display settings are all you need
Block displays are set per theme. You can sometimes simply disable the block on the theme used for /*/edit pages and leave it enabled on the theme used for /user/ pages.
If your site is set up this way, you can avoid extra modules, custom code and complex workarounds.
example
On a site using Claro as the admin theme (for other themes just substitute the name in the path):

visit /admin/appearance to confirm there's a check in the box labeled "Use the administration theme when editing or creating content" (if not, see note below)
visit /admin/structure/block/list/claro and disable the unwanted blocks
visit /admin/structure/block (the default theme block settings page) to confirm that the same blocks are still enabled

With this configuration they show at /user/ but not at /user/edit.
note
If your site doesn't already use the admin theme when editing or creating content, checking the box will change the display of all add/edit pages (e.g. node/add/page and node/123/edit). Have a look at all add/edit pages if you make this change, as it may cause some changes you'd need to address separately, like adjusting theme templates.
Answer: Why don't you try just to hide on the pages you don't want it to be displayed instead of using "Show for the listed pages"? Change the option with "Hide for the listed pages" and add there user/*/edit and every other page where you don't want that block to be displayed.
Comment: _and every other page_: that would be a lot of pages!


Why does my program return the first inputted value without the decimal every run?

Question: #include <stdio.h>

int main (void) {

    int count, value;
    double avg, sum;
    count = 0;
    sum = 0;
    printf("please input how many integers you have \n");
    scanf("%d", &count);

    for (int i = 0; i < count; i++) {
        printf("please input your values \n");
        scanf("%d", &value);
        sum = sum + value;
    }
    avg = sum / count;
    printf("your average is " "%f", avg);
}

Example: count input is 4. input values is 7.6, 1, 2, 3.
I understand that in the for loop scanf sees 7.6 first, but disregards the decimal point as it is not a valid form of input, and passes it along every subsequent scanf in the loop though they never truly accept an input. This results in the only inputted value as 7, but then the program should continue to divide 7 by 4 to retrieve the "Expected" average, but that is not the case. I end up with 7.000000, which I can't figure out why is happening.
Disregard the fact that I am prompting the user to input integer values even though floating point values were inputted because it is part of my homework assignment. Any hints or references to what I should study would be great
Comment: `scanf("%d", &value);` --> `scanf("%lf", &value);`
Answer:
I understand that in the for loop scanf sees 7.6 first, but disregards the decimal point [..]

No. That's a matching failure as the input you enter (7.6) doesn't match the format specifier %d. Hence, scanf() fails. That's why you should always check the return code of all the standard functions. See 220.127.116.11 The fscanf function.
If you want to be able to read floating point values then you should read (change your code) to read floats (or doubles).
For example to read a double:
double value;
if (scanf("%lfd", &value) != 1) {
   /* handle failure */
}

A general suggestion: Don't use scanf() if at all possible. Please read Why does everyone say not to use scanf? What should I use instead? for further explanation.
Comment: I'm not a language lawyer, but I'm pretty sure that's not undefined behaviour. If the type of the variable didn't match the format specifier, then sure, but the format specifier was `%d` and the type was `int`. The input not matching the format given should just result in a matching failure, and make `scanf` return early with a smaller return value, as far as I can see - and in this case, on the first time, it should match up to the period. (If you disagree, I would very much like a pointer to the part of the standard that supposedly makes it undefined)
Answer: Your reasoning seems to be correct, except for the assumption that scanf will set value to zero on the subsequent iterations. Instead, after reading up to the period on the first iteration and assigning 7 to value, on subsequent iterations scanf will see the period, conclude that the input doesn't match the format, and not touch value at all, leaving it as 7 on every iteration. (It should return 1 on the first iteration and 0 on subsequent ones, to indicate the number of items matched and assigned)
So, the loop will add 7 on every iteration, and then divide by the number of iterations, giving a result of 7.
Comment: Thanks for the comment and it's indeed a matching failure (which is not undefined behaviour). I stand corrected. +1!


Help with $\int_{0}^{\pi}\int_{x}^{\pi}\frac{\sin y}{y}\,dy\,dx$

Question: For one of my Calculus 3 practice problems we are given
$$\int_{0}^{\pi}\int_{x}^{\pi}\frac{\sin y}{y}\,dy\,dx$$
I am having difficulty trying to integrate in terms of $y$, should I switch the order of integration from $dy\,dx$ to $dx\,dy$? Thanks for any help.
Comment: You absolutely should.
Comment: Formatting tips [here](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference).
Comment: Yes...................
Comment: @DougM would it also work if I split this function up? Or is changing the order of integration the easiest way to go about it. Thanks
Comment: Change the order of integration and the problem is a snap.  $\int \frac {sin y}{y} dy$ is not integrable into elementary functions.  Unless you are comfortable with exotic functions you must find a way to turn that into something integrable.
Answer: $$\int_0^\pi\int_x^\pi \frac{\sin y}{y}dydx=\int_0^\pi\int_0^y \frac{\sin y}{y}dxdy$$
$$=\int_0^\pi \left(\int_0^y dx\right)\frac{\sin y}{y}dy$$
$$=\int_0^\pi y\frac{\sin y}{y}dy$$
$$=\int_0^\pi \sin y dy$$
$$=\left[-\cos y\right]_0^\pi$$
$$=2$$
Answer: $$\int_0^\pi\int_x^\pi \frac{\sin y}{y}dydx=\int_0^\pi\int_0^y \frac{\sin y}{y}dxdy = \int_0^\pi \sin y~dy = 2$$


How are concurrent updates handled in Memcached?

Question: My understanding is that Memcached is lockless (in its default configuration), such that no client can impede any other client's operation.  If that is the case, then how does Memcached handle two or more clients attempting to update the same object at the same time?
How does Memcached handle multiple versions of an object?  Simply take the one whose update finished last and evict the other versions from the cache?  Is the database actively invalidating objects cached on a Memcached server?
Answer: Memcached hold only last version of object. If you get, modify and put you have no guarantee that the put applies to the same value as the get. You could use CAS operations (gets, cas) if you need to synchronize.
Answer: Adding some details on how it works. CAS - Compare and Swap or Check and set.
Memcached 1.2.5 and higher versions provide the gets and cas commands, which can solve the above problems. If you use the gets command to query the cache of a key, it will return you the unique identifier of the item's current value. If you overwrite this item and want to write it back to Memcached, you can send the unique ID to Memcached through the cas command. If the unique identifier of the item stored in Memcached is consistent with the one you provided, your write operation will be successful. If another process also modifies the item during this period, the unique identifier of the item stored in Memcached will change, and your write operation will fail.
The key/value saved by Memcached has a unique identifier casUnique. When performing incr/decr operations, first obtain casUnique, execute incr, and check whether the return value is casUnique+1. If it is, update it, otherwise, it fails and does not update!
https://developpaper.com/question/why-does-incr-of-memcache-not-have-concurrency-problems/


propensity matching question regarding dropped treatments

Question: I am reviewing a paper that uses propensity matching.  I think i get how it works but i am an amateur at this and was looking for some help.
The paper has 150 in the treatment group and 650 in the control group.  They do the matching and state that they found 102 matches.  So they toss out the remaining 48 in the treatment group.  But matching works if they have identified all the confounding variables and included them in constructing a propensity score.  Now, say I strongly suspect there is another variable that should have been included but was not.  Would it be valid to say that the reason they were having difficulty finding matches was because there were ignored variables that were causing variation.  Or does that have nothing to do with it?
Answer: The basic intuition for the common support problem is that if you are going to estimate the counterfactual for a treated observation by someone matched to that person, then you need to find someone similar in terms of the propensity score in the counterfactual state.
Typically you will want to match on all variables that affect both participation and the outcome to satisfy the CIA. While it is possible that the authors left something out the PS mode, it is also possible that they matched on an instrument (a variables that affects participation, but not outcomes). This will will make a mess of the common support, leaving lots of unmatched treated observations. See Bhattacharya J and Vogt WB, "Do Instrumental Variables Belong in Propensity Scores?" International Journal of Statistics & Economics 9(A12) (2012).
Answer: There are a few reasons why members of the treated group were dropped. One is that common support adjustment was used. There are several ways to do this; one is by using the convex hull, and another is to estimate propensity scores and then remove units outside the region of overlap. Another reason may be that calipers were used in matching (this is more likely). In this way, treated units that don't have a close enough control unit will be discarded. Closeness is typically assessed by using the distance between each unit's propensity scores.
Even if the propensity score was estimated perfectly, units may still be dropped because of the lack of overlap between the groups or the lack of of control units close to the treated units. So it has little to do with having the correct propensity score and more about the degree of similarity between the treated and control groups to start. Of course, measuring that similarity requires propensity scores, and well-estimated propensity scores (i.e., including all relevant variables) will do a better job at characterizing the true degree of similarity between the groups, but won't inherently lead to better matches. In fact, a poor propensity score model might lead to more matches than one would expect, because the degree to which units are similar is being mismeasured, and thus possibly overestimated.
Comment: Well what I am saying here is that the 48 dropped subjects had a propensity score that was calculated.  Now the paper doesn't state why the 48 were dropped.  I, personally agree with Noah that the groups were not entirely comparable to start because the (participation only) variables they used to construct the propensity score were not the relevant confounders.
Comment: @Ssb: From the healthcare perspective, the purpose of propensity methods (intuitively) is to approximate randomization via adjustment or matching on factors that determine treatment selection. Confounders can be adjusted for after treatment selection (group assignment) is handled via either matching or inclusion of the propensity score as a regressor. The confounders can be and hopefully are a completely different set of variables than those contained in the propensity score.


Polymer - icon with paper-badge

Question: I tried use the paper-badge.I wrote this code:
<iron-icon id="star" icon="icons:star-border" style="fill:blue;height: 38px;
        width: 58px;"></iron-icon>
<paper-badge label="6" for="star"></paper-badge>

My problem is that the 6 is not close to the star.

I want to make it more like

I dont know if their is attribute in polymer for doing it.
Answer: You can modify paper-badge's custom css properties to achieve this. Note that I changed the width of the iron-icon to 38px.

    <iron-icon id="star" icon="icons:star-border" style="fill:blue;height:38px;width:38px;"></iron-icon>
    <paper-badge label="6" for="star"></paper-badge>

Check it out here.
Answer: paper-badge has two simple properties
<paper-badge for="btn" label="♥"></paper-badge>

Default width and height:
--paper-badge-width The width of the badge circle   22px
--paper-badge-height    The height of the badge circle  22px
Custom CSS Styling can be changed in

Regarding iron-icon
Default width and height's are
--iron-icon-width   Width of the icon   24px
--iron-icon-height  Height of the icon  24px
and you can customize in your own way by changing in styles
if you could use same default height and width values for booth paper-badge and iron-icn , you can get the view you want.
Please go through the polymer elements docs for more reference
iron-core , paper-badge

Polymer - icon with paper-badge


Which is greater , $7\cdot \tan(10^\circ)$ or $10\cdot \tan(7^\circ)$

Question: My direction is turning the problem into
$$\frac{\tan(10^\circ)}{10}  \space\space\space\space?\space\space\space\space  \frac{\tan(7^\circ)}{7}$$
then differentiating $f(x)=\frac{\tan(x)}{x}$, which is $$f'(x)=\frac{\frac{x}{cos^2(x)}-tan(x)}{x^2}$$
but I don't know how to continue from here.
Comment: Your title and body seem to be different...
Comment: @SaketGurjar edited, for future viewers. will keep that in mind next time :)
Comment: @MohammadZuhairKhan its the same...just cross multiply
Comment: Then why the negative sign?
Comment: @MohammadZuhairKhan it is a hyphen, not a minus sign i think.....if it was a minus sign then the answer will be quite obvious....one will be negative and the other positive.
Comment: My bad, I would probably use a colon in cases like that.
Comment: True...colon would be clearer
Answer: $f'(x)=[x-\sin x \cos x]/[x^2\cos^2 x]$ And $\sin x\cos x\leq \sin x\leq x$ for $x\in (0,\frac{\pi}{2})$.
So, $f'(x)>0$ in that interval, i e. $f$ is increasing in that interval. Your angles in radians are in that interval, so you can compare the two expressions that you want.
Answer: $$f'(x)=\frac{\frac{x}{\cos^2(x)}-\tan(x)}{x^2}=\frac{x-\sin x \cos x}{x^2\cos^2x}=\frac{2x-2 \sin x \cos x}{2x^2\cos^2x}=\frac{2x-\sin 2x}{2x^2\cos^2x}$$
Now to prove that $f'(x)>0$ you need to prove that $2x\gt\sin 2x$ or $\alpha>\sin\alpha$. And this is really trivial. For example, check a few answers given here: Why $x<\tan{x}$ while $0<x<\frac{\pi}{2}$?


Detectar configuração de proxy no webrequest

Question: Boa tarde meus caros, preciso realizar uma requisição para uma api que fica em um domínio externo a rede local, realizo isto através de uma webrequest que foi criada deste maneira:
WebRequest myWebRequest = WebRequest.Create("urldestino");

Porém, a rede possui um proxy, que barrava minhas solicitações informando que a autenticação do proxy era necessária. Para testar o funcionamento, consegui realizar a requisição setando o proxy manualmente da seguinte maneira:
WebProxy myProxy = new WebProxy();
myProxy = (WebProxy)WebProxy.GetDefaultProxy();
myProxy.Credentials = new NetworkCredential("meuusuario", "minhasenha", "meudominio");
myWebRequest.Proxy = myProxy;

Detalhe: após a execução de "myProxy = (WebProxy)WebProxy.GetDefaultProxy();"
o myProxy.Credentials ficou null, por isto setei manualmente.
Desta maneira, a request passa pelo proxy, porém, fui orientado a obter a sessão do proxy já autenticada pelo windows, da mesma maneira que os browsers e o postman faz, no postman por exemplo eu realizo a requisição sem a necessidade de informar os dados do proxy e ele não me retorna um proxy authentication is required.
Existe alguma maneira, de eu obter a sessão do proxy autenticada, sem que precise passar manualmente a matrícula e senha dos usuários logados?
Desde já muito lhes agradeço.
Comment: O proxy está configurado no WIndows e está como padrão?
Comment: Acredito que sim, está setado no browser para "usar script de configuração automática" e abaixo está setado o endereço dele.
Comment: Então não precisa de código nenhum, o padrão do WebRequest é usar o proxy configurado no SO. Não vou colocar como resposta porque não tenho como testar agora, mas testa aí e deixa um feedback nos comentários =D
Comment: Ele não está obtendo automático, acredito que ele obteria no myProxy = (WebProxy)WebProxy.GetDefaultProxy(); porém o myproxy não recebe as credenciais já configuradas, e ao nao passar as credenciais recebi um "{"The remote server returned an error: (407) Proxy Authentication Required."}"
Answer: Para evitar de passar as credenciais manualmente você pode obter as credenciais da seguinte maneira:
WebProxy myProxy = new WebProxy();
myProxy = (WebProxy)WebProxy.GetDefaultProxy();
myProxy.Credentials = System.Net.CredentialCache.DefaultCredentials;
myWebRequest.Proxy = myProxy;

Com o System.Net.CredentialCache.DefaultCredentials;  ele retornará as credenciais já setadas no IE por default, logo abaixo voce seta ao webrequest.proxy as credenciais.
Comment: Funcionou da forma como precisava, autenticou no proxy com sucesso obtendo as credenciais sem precisar passar os dados do usuário. Muito obrigado.


how to display all content for the people table

Question: How does one display all the data in the people table in a SQL or other statement? I would like to get a complete listing of the table(s).
Answer: There is no table {people} in the database.  However, the way I understand this question, you want to display everything that is recorded about all user entities (i.e. you want the list of user entities that is shown under People in the admin interface expanded to include all the fields attached to such entities).
In that case, you can install the Views project and enable the Views and Views UI modules (both part of the Views project). The recipe below assumes you're using Views 3 on Drupal 7. Older versions are different.
Navigate to Home » Administration » Structure » Views and click Add a new view. Give the view the name "Users", and select "Users" from the Show pull down menu.  Click Save & Exit.  This will give you a page view showing all users, but no fields.
Now, return to Home » Administration » Structure » Views, locate the view named "Users", and click Edit.
Next, locate the "Fields" widget, and click Add.  Filter on "User".  This will show all fields defined for the user entity (except pass), with a checkbox for each.  Check all boxes and click Apply (all displays).  Now you've get to configure how to show each of the fields.  You can of course do a lot of fine tuning here, but I find it better to do that later, after the view is fully populated with fields. So just click Apply (all displays) to each of these.  AFAIK, this gives you access to all the fields in the user entity except pass (which holds the user's hashed password, and is excluded for security purposes).
After you've added all, there will be a preview at the bottom of the page.  At this stage, the listing will not be pretty, but you can fine-tune how this appears later, by editing the format of the view.
When done editing, remember to click Save (near top of screen) to save the view.
If your objective is to export this data, there are deveral modules let you export lists produced by Views. One of these is Views Data Export.
Comment: How do I do that and (potentially) download it to a CSV or other file?
Comment: How do I edit the format of the view? And, by the way, Thank you very much.


Gostaria de saber como faço para mudar a cor dos itens selecionados, ActionBar?

Question:

<resources>

</resources>

<?xml version="1.0" encoding="utf-8"?>

<resources>

    <color name="colorPrimary">#93662f</color>
    <color name="colorPrimaryDark">#3a1a0b</color>
    <color name="colorAccent">#93662f</color>

    <color name="orange">#FF9900</color>
    <color name="green">#009900</color>
    <color name="blue">#000099</color>
    <color name="red">#ff0015</color>

    <color name="background_color">#e2e2e2</color>
    <color name="divider">#f2f2f2</color>
    <color name="text_color">#525252</color>
    <color name="white">#ffffff</color>

</resources>

Answer: Você pode mudar a cor de um View ao ser pressionado(exemplo: textView, button etc) criando Selectores:
drawable/selector.xml:
<shape xmlns:android="http://schemas.android.com/apk/res/android"
    android:shape="rectangle">
    <solid android:color="@color/custom_color"/>
    <corners android:radius="8dp"/>
</shape>

color/custom_color.xml:
<selector xmlns:android="http://schemas.android.com/apk/res/android
    <item android:color="@color/blue_bt_pressed" android:state_pressed="true" />
    <item android:color="@color/blue_bt"/>
</selector>

E na sua View adicione:
android:background ="@drawable/selector

Não sei como você pode implementar isto no ActionBar mas espero que isto possa o ajudar.


error updating CloudFront Distribution (E32RNPFGEUHQ6J): InvalidWebACLId: Web ACL is not accessible by the requester

Question: I am using terraform to create a web-acl in aws and want to associate that web-acl with CloudFront distribution.
So, here's how my code looks like:
provider "aws" {
  alias  = "east1"
  region = "us-east-1"
}

# -------------------------------------------
# -------------------------------------------
# Cloud Front
module "front_end_cloudfront" {
  source = "./modules/front-end/CF"

  # CF_ALIASES = ["terraformer-frontend.dev.effi.com.au"]
  CF_LAMBDA_ARN = module.frontend_lambda.cf_lambda_qualified_arn
  CF_BUCKET_DOMAIN_NAME = module.front_end_bucket.website_endpoint
  CF_BUCKET_ORIGIN_ID = module.front_end_bucket.website_domain
  CF_TAGS_LIST = { "Name" : "terraformer-front-end-cloudfrontv2" }
  CF_CERTFICATE_ARN = var.CLOUDFRONT_US_EAST_1_ACM_ARN
  # WEB_ACL = module.waf.web_acl_id
  WEB_ACL = module.waf_cf.web_acl_id

  depends_on = [module.waf_cf]
}

# -------------------------------------------
# -------------------------------------------
# WAF for CF
module "waf_cf" {
  source = "./modules/waf"

  providers = {
    aws = aws.east1
  }

  WAF_NAME  = "terraform-web-acl-cf"
  WAF_DESCRIPTION = "terraform web acl-cf"
  WAF_SCOPE = "CLOUDFRONT"
  WAF_RULE_NAME_1 = "AWSManagedRulesCommonRuleSet"
  WAF_RULE_NAME_2 = "AWSManagedRulesAmazonIpReputationList"
  WAF_RULE_NAME_3 = "AWSManagedRulesLinuxRuleSet"
  WAF_RULE_NAME_4 = "AWSManagedRulesKnownBadInputsRuleSet"
  WAF_VENDOR = "AWS"
  WAF_METRIC_1 = "aws-waf-logs-terraformer-metric"
  WAF_METRIC_2 = "aws-waf-logs-terraformer-metric"
  WAF_METRIC_3 = "aws-waf-logs-terraformer-metric"
  WAF_METRIC_4 = "aws-waf-logs-terraformer-metric"
  WAF_TAG_LIST = {
    "Tag1" : "Name"
    "Tag2" : "terraformer-rule-cf"
  }
  WAF_METRIC = "aws-waf-logs-friendly-metric-name"
  CLOUDWATCH_METRICS_ENABLED = false
  SAMPLE_REQUESTS_ENABLED = false
}

These are terraform modules I have wrote, the specific resource files for above modules are below respectively.
# CF
resource "aws_cloudfront_distribution" "aws_cloudfront_distribution" {
  # aliases = var.CF_ALIASES

  default_cache_behavior {
    allowed_methods = ["GET", "HEAD"]
    cached_methods  = ["GET", "HEAD"]
    compress        = "true"
    default_ttl     = "0"

    forwarded_values {
      cookies {
        forward = "none"
      }

      query_string = "false"
    }

    lambda_function_association {
      event_type   = "origin-response"
      include_body = "false"
      lambda_arn   = var.CF_LAMBDA_ARN
    }

    max_ttl                = "0"
    min_ttl                = "0"
    smooth_streaming       = "false"
    target_origin_id       = var.CF_BUCKET_ORIGIN_ID
    viewer_protocol_policy = "redirect-to-https"
  }

  enabled         = "true"
  http_version    = "http2"
  is_ipv6_enabled = "true"

  origin {
    custom_origin_config {
      http_port                = "80"
      https_port               = "443"
      origin_keepalive_timeout = "5"
      origin_protocol_policy   = "http-only"
      origin_read_timeout      = "30"
      origin_ssl_protocols     = ["TLSv1", "TLSv1.1", "TLSv1.2"]
    }

    domain_name = var.CF_BUCKET_DOMAIN_NAME
    origin_id   = var.CF_BUCKET_ORIGIN_ID
  }

  price_class = "PriceClass_All"

  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  retain_on_delete = "false"

  tags = var.CF_TAGS_LIST

  viewer_certificate {
    acm_certificate_arn            = var.CF_CERTFICATE_ARN
    cloudfront_default_certificate = "false"
    minimum_protocol_version       = "TLSv1.2_2018"
    ssl_support_method             = "sni-only"
  }

  web_acl_id = var.WEB_ACL
}

# WAF
resource "aws_wafv2_web_acl" "aws_wafv2_web_acl" {
  name        = var.WAF_NAME
  description = var.WAF_DESCRIPTION
  scope       = var.WAF_SCOPE

  default_action {
    allow {}
  }

  rule {
    name     = var.WAF_RULE_NAME_1
    priority = 1

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = var.WAF_RULE_NAME_1
        vendor_name = var.WAF_VENDOR

        # excluded_rule {
        #   name = "SizeRestrictions_QUERYSTRING"
        # }

        # excluded_rule {
        #   name = "NoUserAgent_HEADER"
        # }
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = var.CLOUDWATCH_METRICS_ENABLED
      metric_name                = var.WAF_METRIC_1
      sampled_requests_enabled   = var.SAMPLE_REQUESTS_ENABLED
    }
  }

  rule {
    name     = var.WAF_RULE_NAME_2
    priority = 2

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = var.WAF_RULE_NAME_2
        vendor_name = var.WAF_VENDOR
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = var.CLOUDWATCH_METRICS_ENABLED
      metric_name                = var.WAF_METRIC_2
      sampled_requests_enabled   = var.SAMPLE_REQUESTS_ENABLED
    }
  }

  rule {
    name     = var.WAF_RULE_NAME_3
    priority = 3

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = var.WAF_RULE_NAME_3
        vendor_name = var.WAF_VENDOR
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = var.CLOUDWATCH_METRICS_ENABLED
      metric_name                = var.WAF_METRIC_3
      sampled_requests_enabled   = var.SAMPLE_REQUESTS_ENABLED
    }
  }

  rule {
    name     = var.WAF_RULE_NAME_4
    priority = 4

    override_action {
      count {}
    }

    statement {
      managed_rule_group_statement {
        name        = var.WAF_RULE_NAME_4
        vendor_name = var.WAF_VENDOR
      }
    }

    visibility_config {
      cloudwatch_metrics_enabled = var.CLOUDWATCH_METRICS_ENABLED
      metric_name                = var.WAF_METRIC_4
      sampled_requests_enabled   = var.SAMPLE_REQUESTS_ENABLED
    }
  }

  tags = var.WAF_TAG_LIST

  visibility_config {
    cloudwatch_metrics_enabled = var.CLOUDWATCH_METRICS_ENABLED
    metric_name                = var.WAF_METRIC
    sampled_requests_enabled   = var.SAMPLE_REQUESTS_ENABLED
  }
}

But I am getting the below error

error updating CloudFront Distribution (E32RNPFGEUHQ6J): InvalidWebACLId: Web ACL is not accessible by the requester.

Here the cloudfront is created in ap-southeast-2 region and the waf is created in us-east-1 region.
Can someone please help me on this?
Answer: When using WAFv2, you need to specify the the ARN not the ID to web_acl_id in aws_cloudfront_distribution.
See the note here https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudfront_distribution#web_acl_id
or this GitHub issue https://github.com/hashicorp/terraform-provider-aws/issues/13902
Comment: Had this issue with CDKv2.
Comment: this doesnt seems to be working for me, any suggestion from anybody?
Comment: I was also using CDK v2 instead of Terraform, and encountered this issue. Thanks!
Answer: I know this is tagged as terraform but I got the same error with just Cloudformation and could not find any helpful posts.
I was able to resolve my issue by updating permissions for the codebuild. The error message doesn't feel like permissions but this could be it for you too.
Disclaimer: I'm using a corporate shared WAF so you may also need create permissions. Good reference here: https://aws.permissions.cloud/iam/wafv2
          Action:
            - wafv2:ListWebACLs
            - wafv2:GetWebACL
            - wafv2:GetWebACLForResource
            - wafv2:AssociateWebACL
          Resource: '*'

Comment: Looks like you repeated "wafv2:GetWebACLForResource" twice?


Transcendental Galois Theory

Question:
Is there a good reference on transcendental Galois Theory?

More precisely, if $K/k$ admits a separating transcendence basis (or maybe if it is a separably generated extension) it seems to me that many of the usual theorems of Galois theory go through.  Moreover, the group $\text{Aut}(K/k)$ seems to have additional structure; namely it should be an algebraic group over $k$.
For example, it seems to me that $k(x_1, ..., x_n)/k$ has automorphism group $GL_n(k)$.  (EDIT:  As Qiaochu Yuan points out, this is incorrect; the automorphism group at least must contain $PGL_{n+1}(k)$, acting via its action on the function field of $\mathbb{P}_k^n$.)  This sort of thing must be well-studied; if so, what are the standard references on the subject?
I have seen Pete L. Clark's excellent (rough) notes on related subjects here but they seem not to address quite these sorts of questions.
Comment: Isn't the automorphism group more something like $\text{PGL}_{n+1}(k)$?
Comment: How do you figure?  I could just be being silly but I don't even see an action of $PGL_{n+1}$...
Comment: I think the action is by generalized fractional linear transformations, e.g. when $n = 1$ we can send $x_1$ to any $\frac{ax_1 + b}{cx_1 + d}$ and this has inverse the corresponding inverse fractional linear transformation.
Comment: I'm pretty sure you're right; as I've remarked in my edit, this action should arise via the natural action of $PGL_{n+1}$ on the function field of $\mathbb{P}^n_k$.
Answer: For every $n \geq 1$, there is a natural effective action of $\operatorname{PGL}_{n+1}(k)$ on $k(x_1,\ldots,x_n)$.  In fact $\operatorname{PGL}_{n+1}(k)$ is the automorphism group of $\mathbb{P}^n_{/k}$, the action being the obvious one induced by the action of $\operatorname{GL}_{n+1}(k)$ on the vector space $k^{n+1}$ in which $\mathbb{P}^n$ is the set of lines.
However, no one said this was the entire automorphism group of $k(x_1,\ldots,x_n)$!  It is when $n = 1$ -- for instance because every rational map from a smooth curve to a projective variety is a morphism ("valuative criterion for properness").  However, $\operatorname{PGL}_{n+1}(k)$ is known not to be the entire automorphism group of $k(x_1,\ldots,x_n)$ when $n > 1$.  Rather, the full automorphism group is called the Cremona group.  For $n = 2$ we have a problem in the geometry of surfaces, and it was shown (by Max Noether when $k = \mathbb{C}$) that the automorphism group here is generated by the linear automorphisms described above together with a certain set of simple, well-understood birational maps, called quadratic maps or indeed Cremona transformations.  But even when $n = 2$ this automorphism group is not an algebraic group: it's bigger than that.
When $n \geq 3$ it is further known that the linear automorphisms and the Cremona transformations do not generate the whole automorphism group, and apparently no one has even a decent guess as to what a set of generators might look like.  I had the good fortune of hearing a talk by James McKernan on (in part) this subject within the last few months, so I am a bit more up on this than I otherwise would be.  Anyway, he gave us the sense that this is a pretty hopeless problem at present.  For instance, see this recent preprint in which a rather eminent algebraic geometer works rather hard to prove a seemingly rather weak result about finite subgroups of the three dimensional Cremona group!
So, yes, this is a different sort of question from the ones considered in my rough note on transcendental Galois theory.  To all appearances it's a much harder question...
Comment: Thanks!  I should have realized that we could allow birational automorphisms of $\mathbb{P}^n$ as well; I wonder if there is some way of functorially identifying an algebraic subgroup (e.g. look at automorphisms that fix some basepoint on $\mathbb{P}^n$...).
Comment: Recently I have started updating links containing `www.math.uga.edu` (which is no longer working) to `math.uga.edu`. I do not know whether this edit is substantive enough for the author of the post to receive the notification - if yes, I apologize for lost of pings. But I should be able to finish this in a few installments. Anyway, here I changed it not to http://math.uga.edu/~pete/galois.pdf - instead I have used http://math.uga.edu/~pete/transgalois.pdf (I suppose that was the intention of giving the "redirect pdf" there.)

Transcendental Galois Theory


Can't create table for Azure SQL Server database in VS 2015

Question: I created a SQL Server database on Azure and connected to this database (from SQL Server Object Explorer in Visual Studio 2015).
But when I try to create new table, I got endless:

"Loading... The design surface is loading"

How can I fix it?
Answer: I suppose the loading info is the result of high CPU and high memory.  You could close other useless pages in project and other apps. Try to restart the VS. Then open the Sql server object explorer slowly(wait a while).  Or reconnect to Azure in VS to try again.
Close high CPU and high memory apps in Task Manager:

Besides, you could also create table in Azure sql in Server Explorer.
Comment: If my answer is helpful, you could vote it up or mark it as answer to help other community members. Thanks!
Comment: I don't think many people searching for an answer to this issue are having a simple performance problem. Most of them would just wait a few seconds, instead of googling. We are having a different problem where it fails to load despite fast hardware and no previous issues with it loading quickly.


Is every Baire space Cech-complete?

Question: I can't think of any example of a topological space satisfying the Baire Category Theorem which is not Cech-complete. (A space is Cech-complete if it is a Tychonoff space and if it is a $G_\delta$ subset of its Stone-Cech compactification, or equivalently of any other compactification). There isn't even an example of one in $\pi$-base. However, it's very well known every Cech-complete space is Baire, and I've never seen it mentioned that the reverse is true as well.
Are these properties equivalent? Is there a counterexample?
Comment: There are metrisable but not completely metrisable Baire spaces: https://mathoverflow.net/questions/150472/completely-metrizable-space-and-baire-theorem
Comment: Very good, thank you.
Answer: The space $\mathbb{N}^{\omega_1}$ is Baire (as any product of Čech-complete spaces is, Engelking 3.9.J(e), due to Oxtoby or Bourbaki), but it is not a $k$-space (see Engelking 3.3.E(a)) so not Cech-complete.
A metrisable example: $X = \mathbb{Q}\times \{0\} \cup \mathbb{R} \times (0,\infty) \subseteq \mathbb{R}^2$ (subspace topology) is Baire as $\mathbb{R} \times (0,\infty)$ is open and dense and Baire (as a locally compact space, e.g.).
But $X$ cannot be Čech-complete as the closed subset $\mathbb{Q} \times \{0\} \simeq \mathbb{Q}$ would be Čech-complete (the class is closed under closed subsets and $G_\delta$ subsets) which it is not.
Comment: @HennoBrandsma...you are one of the kings of general topology  in MSE..;)
Comment: I still like my old field... and glad to help @MariosGretsas
Comment: @HennoBrandsma..I also love this field and i want to master it (when i say master ,maybe it sounds naive..i mean to become very good at it)because it will help me in analysis..Now that i found you,i would like your opinion about something if you have the time of course..:)
Comment: @MariosGretsas read good topology books, lots of papers and go to conferences for 6 years or so...
Comment: Is it a good move to take a book and solve it all?..At my level i was thinking to read and solve topology without tears..Munkres would be very difficult for me at this level i am
Comment: @MariosGretsas practice is very good. Reading good exposition too.
Comment: Can you suggest some books?
Comment: Thank you! This is quite counter-intuitive for me.
Comment: @MariosGretsas If Munkres is too hard, topology without tears is fine,and freely available. http://at.yorku.ca/i/a/a/b/23.htm is also basic and accessible.
Comment: Ok.. thank you for the conversation and your time..:)

Is every Baire space Cech-complete?


TaskCanceledException while invoking AWS Lambda

Question: I am hitting an unhandled TaskCanceledException every time my code invokes an AWS Lambda. The code runs on an Android device. (It's written in C# with Xamarin.Android and references AWSSDK.Core, AWSSDK.Lambda).

Why is the task timing out? [Update: this has been figured out]
Why isn't the exception handled?
Why can't I see any diagnostics from AWS SDK for .NET in the logs?

Code:
public class SomeActivity: Activity
{
    private AmazaonLambdaClient mAWSLambdaClient;

    protected override void OnCreate(Bundle savedInstanceState)
    {
        base.OnCreate(savedInstanceState);

        SetContentView(...);
        FindViewById(...).Click += ButtonClickAsync;

        // System.Diagnostics.Trace redirects to Log.Debug with TAG="System.Diagnostics.Trace"
        System.Diagnostics.Trace.Listeners.Add(new MyAndroidTraceListener("System.Diagnostics.Trace"));
        System.Diagnostics.Trace.TraceInformation("Android trace listener installed");

        // AWS logs to System.Diagnostics
        AWSConfigs.LoggingConfig.LogTo = LoggingOptions.SystemDiagnostics;
        AWSConfigs.LoggingConfig.LogResponses = ResponseLoggingOption.Always;
    }

    protected override void OnStart()
    {
        base.OnStart();

        var idToken = ...
        var awsCredentials = new CognitoAWSCredentials("IdentityPoolID", AWSConfig.RegionEndpoint);
        awsCredentials.AddLogin("accounts.google.com", idToken);

        mAWSLambdaClient = new AmazonLambdaClient(awsCredentials, AWSConfig.RegionEndpoint);
    }

    protected override void OnStop()
    {
        base.OnStop();

        mAWSLambdaClient.Dispose();
        mAWSLambdaClient = null;
    }

    private async void ButtonClickAsync(object sender, System.EventArgs e)
    {
         await DoSomethingAsync();
    }

    private async Task DoSomethingAsync()
    {
        var lambdaRequest = ...
        try
        {
            var lambdaInvokeTask = mAWSLambdaClient.InvokeAsync(lambdaRequest);
            invokeResponse = await lambdaInvokeTask; <= VS breaks here after ~30 to 60 seconds
        }
        catch (TaskCanceledException e) // also tried catching Exception, no luck
        {
            Log.Error(TAG, "Lambda Task Canceled: {0}, {1}", e.Message, e.InnerException);
            return;
        }
    }
}

Visual Studio breaks on the await line, telling me I have an unhandled TaskCanceledException: a task was canceled. Weird I do handle that exception.
After the unhandled exception, I check the Device Log in Visual Studio. I filter by TAG="System.Diagnostics.Trace" and all I find is:
base apk Information 0:
Android trace listener installed

Where is the AWS SDK log I should have gotten according to logging-with-the-aws-sdk-for-net?
UPDATE:
I've figured out question 1, why it times out. It was due to a lambdaRequest with a bad PayloadStream set to a MemoryStream whose position had not been reset to 0 after JSON serializing an object to the stream.
I have not figured out why 2, the exception wasn't handled by the try/catch, and 3, why AWS SDK did not log as requested.
Comment: @Nkosi, I figured out #1, a bug in my code that creates the lambdaRequest, code that wasn't even shared in the question. My bad but I did not think that code was relevant. Updated the question. Question #2 and #3 still puzzling me. Thanks.
Comment: How and where is `DoSomethingAsync()` being called/invoked? You may have a blocking call somewhere that is causing a dead lock, resulting in time expiring and cancellation of the task
Comment: I've updated the question to show more specifically how DoSomethingAsync() gets called. It's called by a button click handler.
Comment: Ok why are you beginning the task outside of the try catch. try just `await mAWSLambdaClient.InvokeAsync(lambdaRequest);` within the try/catch and see if the exception is caught.
Comment: That would make no difference (also tested right now, same thing repros), exceptions thrown inside functions that returns async Task capture their exception and store it inside the task. The exception bubbles up when you await or Wait() the task.

The reason I keep the task creation outside the try block, is that I sometimes try to do more work that can be done immediately between creating the task, and awaiting the task. I prefer to keep my try block as close to where the exception bubbles so that I catch only what I think I will catch.
Comment: That makes a lot of sense.
Answer: I'm guessing either the TaskCanceledException instance is not from the same namespace your code is expecting in the catch statement, or it is being thrown from the line just above your try-catch, i.e mAWSLambdaClient.InvokeAsync(lambdaRequest). What happens if you move that line and possibly more lines inside the try-catch block?
If this doesn't help, please post the stack trace.
Comment: Good point about the TaskCanceledException possibly being thrown in a different namespace, I need to check if this is what's happening thanks. I know the InvokeAsync statement isn't throwing because I've double checked for that my moving it inside the try block and it didn't change.
Comment: Unhandled exception is of type System.Threading.Tasks.TaskCanceledException, same as what I'm catching. I updated the code to not only move the try to include everything including the invokeAsync, but also catching general System.Exception. Still not catching it...


toggling sibling buttons in bootstrap

Question: http://jsfiddle.net/s89a4/3/ shows how I have placed yes/no/na buttons for each row of a table. I want to click on a value such that its siblings get toggled.
How can I achieve this? Tried out the sibling jquery methods but can't pull it through.

<button id="yesbutton" type="button" class="btn btn-success">Yes</button>
<button id="nobutton" type="button" class="btn btn-danger">No</button>
<button id="nabutton" type="button" class="btn btn-info">N/A</button>

I can toggle the individual buttons by clicking twice on each. However I need the siblings to be toggled for that particular row. (my fiddle does not show even the toggling somehow but I think it puts across the point) thanks.
Comment: Where's the yes/no/na buttons in the jsfiddle?
Comment: Check out the boostrap page: http://getbootstrap.com/components/#btn-groups
Comment: None of your code here is in your fiddle.
Comment: I would suggest shortening the delay on your dropdown.  It is not the "expected behavior" and may be confusing or weird or make someone think something is wrong with your site.  When I first hovered the nav, nothing happened, so I clicked  and it seemed like my click is what opened it.
Comment: sorry about the wrong link folks - here's my fiddle http://jsfiddle.net/s89a4/1/
Comment: corrections on the button IDs -> http://jsfiddle.net/s89a4/2/ As I said, the individual buttons are toggling on clicking them, but I need only one button in a row to be "active" at any time since this is a yes/no/na scenario.
Comment: please ignore the earlier fiddles - this one is the perfect example for my problem -> http://jsfiddle.net/s89a4/3/
Comment: This the sort of thing you're after? http://jsfiddle.net/Brv5A/
Comment: you nailed it @patrick - it now sounds too obvious :) http://jsfiddle.net/Brv5A/1/ <-- modified to my needs
Comment: Good to hear : ) posted it as an answer for closure's sake, if you wouldn't mind accepting it.
Comment: @Patrick thanks again! should have done this long ago - marked your answer as accepted.
Answer: You can achieve this by toggling the needed class on your element, while removing the unwanted class(es) from its siblings. Simple example would be:
$('#translateTable').on( 'click', '#yesbutton', function (event) {

    $(this).toggleClass("active");
    $(this).siblings().removeClass("active");

});

Demo for you.

toggling sibling buttons in bootstrap


Migrating Terminal Server profiles from 2003 to 2008 R2 "V2"

Question: I've done some searching and there doesn't appear to be a method of migrating old 2003 based profiles to the new "V2" profiles automatically.  Specifically, we're interested in retaining the <Profile>.NK2 file in Outlook, the Favorites folder, and the Signatures folder.
I'm in the process of writing a fairly complicated migration script that'll be executed at login, but seems like this is something that I shouldn't have to do.
The User Migration Tool appears to only work for Vista, not 2008 R2.  Any other ideas?
Comment: Could you send me the script you used for this? I am trying to migrate users from win 2k3 to 2k8R2 for Citrix. I have an RDP app on Citrix 2k3 server and I want to move everyone to our new farm while keeping the cached computer names on their new profiles. I am trying to do this with microsoft office as well. Thanks!
Answer: Nope. There isn't a way to migrate profiles from W2K3 to W2K8, they're not interchangeable and they're not interoperable. It's a headache but there's no way around the fact that you're going to have to manually copy the components you want (NK2 files, signature files, Favorites, etc.) from the old profiles to the new profiles.
Comment: Caused me physical pain to mark that as the accepted answer.  Gotta love Microsoft sometimes.
Comment: Yeah, I've looked high and low for a way around this but haven't found anything. I've got 600 W2K3 profiles to migrate to W2K8.
Comment: I can post up my ghetto script when I'm done if you're interested.  Will be going live with 60 profiles later this week.
Comment: Did yourscript work? Im also in the process of moving 200 users from a TS2003 farm, to 2008 R2. It'll be painful to migrate without the favorites, signature files etc :\
Comment: Yes it did.  Let me clean it up and I'll pastebin it.


Is there any easy way to replace a saddle?

Question: The "saddle" on my acoustic is broken (like it looks shattered/cracked) where the G-String makes contact with it. This makes the string sound dead. Is there an easy way for someone like me, who has no experience with guitar repair, to replace the saddle? Or should I try to find a professional to do it?
Also what would the price range be for replacing it? I only spent $50 to buy the guitar, so maybe it's not even worth it...
Comment: So, Jeff, how goes your saddle?
Comment: it's not an emergency, but the g string is completely dead
Answer: Stewart-MacDonald is a good source for saddles and the like. A precut saddle will run $15 or so and might be less at your local guitar pusher. Whether your guitar is worth it is up to you.
Answer: Buying a saddle blank is easy and cheap.  Making one out of a piece of bone is somewhat more difficult but do-able.
It's fitting the saddle to the bridge and adjusting the action that takes a bit of experience and patience.
The saddle must sit absolutely square in the bottom of the slot, and snugly enough in the slot so as to have no side-to-side play.
It must (or should) be configured to match the curvature of the fretboard if the fretboard is not flat.  (most have some radius)
As well, it may need to be "compensated" to correct minor intonation adjustments.
Best, if you have no experience, to just have a luthier do it. It's relatively cheap.
Answer: If your guitar is pretty cheap, then it might not be worth it to have it professionally repaired.  May as well try it yourself by either repairing the existing saddle with "gel" type super glue or buying a blank and cutting it to size (probably more work).


How to choose between SGD with Nesterov momentum and Adam?

Question: I'm currently implementing a neural network architecture on Keras. I would like to optimize the training time, and I'm considering using alternative optimizers such as SGD with Nesterov Momentum and Adam.
I've read several things about the pros and cons of each methods (SGD with Nesterov momentum: very sensitive to initial value of the learning rate, and requires learning rate scheduling --I'm not sure what it means...), but I still do not know how they compare, and how to choose between them.
Can someone help me clarifying this points? Thanks!
Comment: this is a nice article: http://sebastianruder.com/optimizing-gradient-descent/

but in general, which is best depends on the network architecture and how much time you want to spend tuning the optimizer
Comment: Why not use both? https://web.stanford.edu/~tdozat/files/TDozat-CS229-Paper.pdf
Comment: [downhill](http://downhill.readthedocs.io) does AdaM etc. with or without Nesterov momentum (wraps theano). Could you run a problem similar to yours on that -- try both ?
Answer: In general, there aren't definitive results on one learning algorithm being "better" than another. The common wisdom (which needs to be taken with a pound of salt) has been that Adam requires less experimentation to get convergence on the first try than SGD and variants thereof. However, this is highly dataset/model dependent. More recently some groups have made the claim that despite training faster, models trained with Adam generalize worse to the validation or test sets. See https://arxiv.org/abs/1712.07628 for more information.