Categorizing modes of scales other than major scale

Question: Modes of major scale are often ordered such that each consecutive scale can be obtained  by altering one step of the previous one:
lydian     1  2  3 #4  5  6  7
ionian     1  2  3  4  5  6  7
mixolydian 1  2  3  4  5  6 b7
dorian     1  2 b3  4  5  6 b7
aeolian    1  2 b3  4  5 b6 b7
phrygian   1 b2 b3  4  5 b6 b7
locrian    1 b2 b3  4 b5 b6 b7

This way one can easily see how they gradually change from "bright" and "open" (lydian), towards "dark" and "closed" (locrian).
Not all scales however have modes which follow such simple dependence. In particular not the commonly used melodic minor and harmonic minor. Modes of these scales are typically presented as modes of major scale with alterations (see e.g. Melodic Minor Modes) but this doesn't explain much about how these modes relate to each other.
It would seem natural that since modes of a scale consist of the same intervals, just ordered differently, they should share some sonic properties. Is there some way to categorize modes of melodic minor and harmonic minor (and perhaps some other scales) in a way that shows connection between them?
Comment: Great question. And by the way, with this I am reminded of the Lydian Chromatic Concept that puts the Lydian scale as the root of the series.
Answer: I've never encountered something like this, and my guess is that this is untenable for at least two reasons:

Since the major scale is a conjunct subset of the circle of fifths (C major is F–C–G–D–A–E–B rearranged), this allows the clean "add one accidental for the next mode" that we see (since adding an accidental is equivalent to moving around the circle of fifths). This feature is not true for the melodic or harmonic minor scales.
The augmented second of the harmonic minor really messes with the ensuing scale constructions.

But because I'm a theorist, I tried to come up with something like your system. In order to fill in the blanks, I decided to combine the modes of both harmonic and melodic minor. I've ordered it to maximize common tones from one scale to the next, but as you see, there's no way to do it as cleanly as we do it with the major scale. The parenthesized numbers at the end of each mode show the number of common tones with the mode above, and only three times does the next mode have six tones in common with the prior one (as they all do in the modes of the major scale).
Melodic  Mode 3     1   2   3  ♯4  ♯5   6   7   1
Harmonic Mode 6     1  ♯2   3  ♯4   5   6   7   1   (5 tones in common)
Melodic  Mode 4     1   2   3  ♯4   5   6  ♭7   1   (5)
Harmonic Mode 3     1   2   3   4  ♯5   6   7   1   (4)
Harmonic Mode 4     1   2  ♭3  ♯4   5   6  ♭7   1   (6)
Melodic  Mode 1     1   2  ♭3   4   5   6   7   1   (5)
Harmonic Mode 1     1   2  ♭3   4   5  ♭6   7   1   (6)
Melodic  Mode 5     1   2   3   4   5  ♭6  ♭7   1   (5)
Melodic  Mode 2     1  ♭2  ♭3   4   5   6  ♭7   1   (4)
Harmonic Mode 5     1  ♭2   3   4   5  ♭6  ♭7   1   (5)
Melodic  Mode 6     1   2  ♭3   4  ♭5  ♭6  ♭7   1   (4)
Harmonic Mode 2     1  ♭2  ♭3   4  ♭5   6  ♭7   1   (5)
Melodic  Mode 7     1  ♭2  ♭3  ♭4  ♭5  ♭6  ♭7   1   (5)
Harmonic Mode 7     1  ♭2  ♭3  ♭4  ♭5  ♭6 ♭♭7   1   (6)

(Note that, for the purposes of creating this chart, I didn't consider enharmonic equivalence. For instance, I treated "♯2" and "♭3" as distinct entities.)
I'm sure it's possible to flesh this chart out so that every succeeding mode only differs by one pitch, but it would necessitate many more scales/modes than just the rotations of these two forms of minor. As a sample of this idea, I add in the diatonic modes in the chart below; now look at how many scales have six tones in common with their predecessor! (I've asterisked the new sixes.)
Ionian              1   2   3   4   5   6   7   1
Lydian              1   2   3  ♯4   5   6   7   1   (6)*
Melodic  Mode 3     1   2   3  ♯4  ♯5   6   7   1   (6)*
Harmonic Mode 6     1  ♯2   3  ♯4   5   6   7   1   (5)
Melodic  Mode 4     1   2   3  ♯4   5   6  ♭7   1   (5)
Mixolydian          1   2   3   4   5   6  ♭7   1   (6)*
Harmonic Mode 3     1   2   3   4  ♯5   6   7   1   (5)
Harmonic Mode 4     1   2  ♭3  ♯4   5   6  ♭7   1   (6)
Dorian              1   2  ♭3   4   5   6  ♭7   1   (6)*
Melodic  Mode 1     1   2  ♭3   4   5   6   7   1   (6)*
Harmonic Mode 1     1   2  ♭3   4   5  ♭6   7   1   (6)
Aeolian             1   2  ♭3   4   5  ♭6  ♭7   1   (6)*
Melodic  Mode 5     1   2   3   4   5  ♭6  ♭7   1   (6)*
Melodic  Mode 2     1  ♭2  ♭3   4   5   6  ♭7   1   (4)
Phrygian            1  ♭2  ♭3   4   5  ♭6  ♭7   1   (6)*
Harmonic Mode 5     1  ♭2   3   4   5  ♭6  ♭7   1   (6)*
Melodic  Mode 6     1   2  ♭3   4  ♭5  ♭6  ♭7   1   (4)
Locrian             1  ♭2  ♭3   4  ♭5  ♭6  ♭7   1   (6)*
Harmonic Mode 2     1  ♭2  ♭3   4  ♭5   6  ♭7   1   (6)*
Melodic  Mode 7     1  ♭2  ♭3  ♭4  ♭5  ♭6  ♭7   1   (5)
Harmonic Mode 7     1  ♭2  ♭3  ♭4  ♭5  ♭6 ♭♭7   1   (6)

Comment: Standing slow clap for this one!
Comment: Interesting, to put all several scales together! But this also means you're not in favor of an idea of finding a connecting element of modes of an arbitrary scale (other than that they are modes of the scale)?
Comment: @user1079505 It's not that I'm not in favor of that idea; in fact, I think it's a good one. But I can't see a way to organize these modes like you do in your question, so I decided to include other collections.
Comment: I wonder if you could construct some analogue of a [de Bruijn sequence](https://en.wikipedia.org/wiki/De_Bruijn_sequence) restricted to 1-2(3) semitone steps.
Comment: Given the system you've devised here, you might be interested in Ricardo Javier Rademacher Mena's "[The Universal Accidental](https://muse.jhu.edu/article/777100/pdf)". The author proposes a mathematical model for transformations between modes based on the changes in accidentals between them.


Why no uniform testing for doping at major sporting events?

Question: It seems that doping scandals are still very much a part of big sporting events. There are the recent news about the Russian team at the Olympics, the Tour de France scandals, and I'm sure there are more. In the Russian case, for example, we're finding - years after the event - officials that are testifying to the widespread use of doping, including by gold medalists, and we're finding evidence that the analysis lab (run by Russian officials) was falsifying results.
I do not understand why this is still a part of sports. It seems that there's a simple solution: just test every participant in a sporting event, right on the day of the event, in a uniform/standardized way, by the same independent laboratory, with supervisors from the public (perhaps even broadcast to the public), instead of just selective testing by country-specific secretive and closed labs as seems to be the case now.
If for some reason the tests are prohibitively expensive, then make it a rule that tests will always be taken from people who score one of first 3 places in a given competition. Or run basic tests on every participant, and very extensive/thorough tests on the winners.
Why is this type of thing not done to put an end to doping once and for all? Or am I misunderstanding something?
Answer: The answer is simple, although you won't hear it in the media: it's financial.
Imagine a league like the MLB suddenly decides to test every one of its players on a given day. Immediately they end up having to suspend 5-15% of the league including many of their star players. This hurts the credibility of the league, and the credibility of every team in the league, and the only ones that lose are the MLB and its members themselves.
This means that there is no financial incentive for a league or organization to take drug testing seriously. There's only enough incentive to give the facade that they do so the credibility of the league or whatever stays in tact.
The only alternative is to set a hard deadline in the future in which players need to be clean by, and then implement consistent testing by that time, but in doing so the organization admits that their are dopers present now, and the league isn't testing them. So again they lose credibility.
And when a good number of athletes stand to gain huge benefits by doping the incentive is going to be there for them to continue doing so. A good example comes to mind, Jhonny Peralta of the Detroit Tigers was suspended for doping and then afterward signed a 50 million dollar contract.
With money like that available for players, and no viable way for leagues to maintain their integrity and effectively deal with the problem, it's not going away any time soon save outside intervention by a third party.
Comment: contract for 7 years at age 32.  So now owners care a bit.  But in actuality if only 5-15% of the players used an illegal substance - from enhancing to coke, uppers, pot, whatever... then the 85% would get substantial drug testing passed.  As it stands that number has to be some where near 50% because the unions keep pushing testing off.  And really there doesn't need to be a financial impact if the rules are written right - players don't get paid if caught but owners have to still distribute money (in capped sports).
Answer: It is not enough to test on the day of the competition, even if a 100% foolproof test for all illegal substances were possible.
A doping athlete can use a drug during the training period leading up to competition, then stop use in time for any traces to disappear from the body before the test is taken. But the effects on the body (e.g. increased muscle mass) will remain longer, giving an unfair advantage.
This is the reason that out-of-competetion testing is routinely carried out too, but this raises the problems of resources, of trusting local testing authorities, and of athlete rights and privacy, so it is very difficult to stop doping completely.
Regarding the top 3 places testing, I seem to remember an anecdote of some cyclist in the Tour de France making certain to finish 4th on a day that he had taken a stimulant due to testing the 3 podium positions, and only racing 100% for stage wins on clean days. I'm not sure if this is a true story or not, but it reinforces the point: if dopers know when the tests are going to be taken they will find a way to cheat without getting caught.
Answer: There are two good answers here that raise good points:

There is no uniformity across sports because the incentives are different in different sports. Canadian Coder's answer about MLB is a good explanation of why MLB's testing structure is much different from that of, say, swimming or cycling.
In the sports which do test extensively, as Fillet explains, in-competition testing is carried out in exactly the way you describe, actually even more extensively (e.g. there have been world championships and Olympics where literally every competitor has been tested at the venue), but because of the way the benefits of doping are accrued, in-competition testing is not sufficient by itself to detect doping. Out-of-competition testing - at times which are unpredictable to the athlete - is essential for detecting doping. (See John Brant's excellent piece on Eddy Hellebuyck for details on how a doper evaded testing for years.)

A lot of what you're seeing in the news these days is driven by two problems which aren't solved by comprehensive in-competition testing.
One is accessibility of athletes for out-of-competition testing, which is an issue in e.g. East Africa, where many elite distance runners train. Maintaining the infrastructure needed to get samples and test them in an unpredictable way is expensive, and despite some significant investment (e.g. by the World Marathon Majors) there still needs to be more funding to do this properly. Much of the negative news about Kenya from WADA has to do with the lack of testing infrastructure.
The other problem - and this is what we're seeing from Russia - is the presence (or rumored presence) of state-sponsored structures which facilitate doping on a national level. If anti-doping "agents" (for lack of a better word) are simply testing individual athletes, that's one thing; if they're contending with an entire system which is coordinating evasion of out-of-competition testing, that's entirely another. And the stories coming out of Russia about evasion of testing are real James Bond stuff: local police detaining testers in order to give athletes time to get away, or to invalidate the samples either environmentally (keeping them from being refrigerated) or by breaking the chain of custody, athletes giving their whereabouts as in closed "military cities" to prevent surprise visits from testers, etc. When the full apparatus of the state is employed in protecting dopers, the "arms race" between cheaters and anti-doping officials becomes very expensive, underlining Canadian Coders' point about finances.
So, the short answer to your question: in-competition testing happens and is pretty comprehensive, but it's not enough and the implications of closing the gaps are all expensive.


How to convert food-101 dataset into usable format for AWS SageMaker

Question: I'm still very new to the world of machine learning and am looking for some guidance for how to continue a project that I've been working on. Right now I'm trying to feed in the Food-101 dataset into the Image Classification algorithm in SageMaker and later deploy this trained model onto an AWS deeplens to have food detection capabilities. Unfortunately the dataset comes with only the raw image files organized in sub folders as well as a .h5 file (not sure if I can just directly feed this file type into sageMaker?). From what I've gathered neither of these are suitable ways to feed in this dataset into SageMaker and I was wondering if anyone could help point me in the right direction of how I might be able to prepare the dataset properly for SageMaker i.e convert to a .rec or something else. Apologies if the scope of this question is very broad I am still a beginner to all of this and I'm simply stuck and do not know how to proceed so any help you guys might be able to provide would be fantastic. Thanks!
Answer: if you want to use the built-in algo for image classification, you can either use Image format or RecordIO format, re: https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html#IC-inputoutput
Image format is straightforward: just build a manifest file with the list of images. This could be an easy solution for you, since you already have images organized in folders.
RecordIO requires that you build files with the 'im2rec' tool, re: https://mxnet.incubator.apache.org/versions/master/faq/recordio.html.
Once your data set is ready, you should be able to adapt the sample notebooks available at https://github.com/awslabs/amazon-sagemaker-examples/tree/master/introduction_to_amazon_algorithms


Use ssh-agent with SSH-FS and VSCode, without password prompts

Question: I want to edit server file using SSH-FS that is extensions of Visual Studio Code.
I want to connect server without to input password using ssh-agent.
How should I set?
I tried following settings.
I can connect server without password prompts by following setting but I cannot use ssh-agent:
<code>"sshfs.configs": [
    {
        "name": "<server_name>",
        "host": "<server_ip_address>",
        "port": 22,
        "username": "user",
        "privateKeyPath": "<path_to_private_key>",
        "root": "/",
        "sftpSudo": "root",
        "passphrase": "<passphrase_of_private_key>",
        "password": "<password_of_sudo>",
    }
]
</code>
I can connect the server using ssh-agent by following setting but I should input the password:
<code>"sshfs.configs": [
    {
        "name": "<server_name>",
        "host": "<server_ip_address>",
        "port": 22,
        "username": "user",
        "agent": "//./pipe/openssh-ssh-agent",
        "root": "/",
        "sftpSudo": "root",
    }
]
</code>
I cannot connect the server by following setting:
<code>"sshfs.configs": [
    {
        "name": "<server_name>",
        "host": "<server_ip_address>",
        "port": 22,
        "username": "user",
        "agent": "//./pipe/openssh-ssh-agent",
        "root": "/",
        "sftpSudo": "root",
        "password": "<password_of_sudo>",
    }
]
</code>
Here is why I want to set it up like this:

I want to connect multiple servers in this workspace but vscode supports only one password prompt. Two or more prompts are not displayed and I cannot enter that passwords.
I want not write passphrase of the private key to any files.

Thank you.
Answer: This is quite a late response, but if this is still an issue, I suggest making an issue on GitHub.
Currently, the extension can act a bit weird if you try to have multiple authentication methods, especially if you involve agent-based authentication. The underlying library (ssh2) just doesn't like to play too nice in that regards. Although the library has improved it lately with a big refactor, this includes (temporary) loss of previously exposed APIs, thus making it hard to port right now.
As for VS Code only supporting one password prompt, that's something I didn't know. If that's (still) the case and causes issues, I could make the extension queue up password prompts.
Disclaimer: I am the developer of the SSH FS extension


Monitor a service from a remote streaming server?

Question: My client uses a 3rd party to stream their radio station. Recently then have complaints about service outage in different global locations.
Does anyone know of a service I can use to monitor the remote audio stream coming from the streaming server?
Answer: You could use something like Nagios or Zabbix to periodically (every minute or two) connect to the port the streaming services uses.  Connection ok = good result. Connection refused or timed out = bad result and you can throw an error with whatever action... email, text message, or just logging it.
Comment: I only have a URL like this, http://subdomain.example.com/jw.php?channel=mymusic - can I use this... or is there a way to tell what port is being used?
Comment: When the URL is up does it return a 200 OK, and when it's down does it report a 500, or something other than 200?  If yes then you can absolutely use it.
Comment: Thanks - sounds good. I might give one of those a go... although they look rather complex. I just found a php script that will retrieve headers for a remote file. If I stick it on cron I think that might do the trick. I found the script at http://bit.ly/9oDhX0 if anyone is looking for something similar.


Trying to use historical data to make an event 'non-rare' in a logistic regression model

Question: I am working on a logistic regression model that attempts to predict failure events in a population of devices using the previous 10 days of data (60 features from sensor data). The failure event is rare; I would expect to find approximately 20-30 on a given day, and the total population size is > 10,000.
Here is where I may be mistaken: I have access to years-worth of historical data, so I thought that I would be slick and make the event 'non-rare'.  That is, collect 1000 failure examples and 1000 non-failure examples, then estimate the model on that. This, I thought, would give the model enough information to determine the relationships between sensor readings and class membership.
However, I am starting to think that my intercept term (and others?) may be problematic because the real initial class probabilities are not respected.
Is the model corrupted by this disregard of the real event probabilities?
Comment: If I'm reading this correctly, this sounds akin to a case-control study in epidemiology/biostatistics -- that is, sampling on event status (event/non-event) and then considering differences in exposure status. This is useful for modelling odds ratios and answering causal questions (e.g. is paste exposure to Substance X more common amongst brain cancer patients than controls?) but less useful if you're looking at predicting event status based on other features (and it sounds like your data structure is more complex than this simple example anyway.)
Comment: See [Does down-sampling change logistic regression coefficients?](http://stats.stackexchange.com/q/67903/17230).
Answer: What you do is 'over sample' on the 'failures' and as you conclude, this biases your estimates.  When you do not draw a random sample, but a sample based on the values of the binary outcome, then you are doing a so-called 'choice based sampling'.
The correction of King and Zeng is one option, another option is to use the weights  parameter in glm: If $\pi$ is the fraction of failures in your population and $p$ the fraction in the sample, then each observation gets a weight which is $\frac{\pi}{p}$ for the failures and $\frac{1-\pi}{1-p}$ for the non-failures.  For details see e.g. W.H. Greene, Econometric Analysis.  Note that in the latter case you should use White's sandwich estimator for the var-covar matrix as was mentioned by @Scortchi in the comments.
Comment: @Scortchi: You are right about the var-covar estimator, The reference that I menstion (Greene) also says that you should use the ''White's robust sandwich estimator'', Greene refers to White (1982a), Maximum lilekihood estimation of misspecified models, Econometrica 53, 1982, p 1-16. But I think you can no longer say that my answer is wrong ?
Comment: (-1) The `glm` manual says "For a binomial GLM prior weights are used to give the number of trials when the response is the proportion of successes". So these are frequency weights, **not** sampling weights: standard errors for odds ratio estimates will be hugely inflated. (BTW I hope it doesn't seem like I'm picking on you today - you should've also noticed a few up-votes.)
Comment: @Scortchi: no hard feelings, if I am wrong I am wrong, but in this case I am not so sure whether I am, did you check the formula in the reference ? Because if you look at it, these are the weights of the observations in the likelihood function that is maximised ?
Comment: Which reference?
Comment: @Scortchi: W.H. Greene, Econometric Analysis
Comment: I don't have it to hand. I've just looked at King & Zeng: they mention this method of weighting - in fact they say it's more robust under model mis-specification than prior correction -, but note that "the usual method of computing standard errors is severely biased" & recommend using White's standard errors, wh. I think means [White (1980). "A heteroskedasticity-consistent covariance matrix estimator & a direct test for heteroskedasticity", *Econometrica*, **48**, 4](https://www.jstor.org/stable/1912934).
Comment: Important to mention that in the answer, I think.
Comment: @Scortchi: well if you look at the question I think my answer answers that question, moreover I have a reference 'for detail see ...' where that is mentioned ? But as I said, no hard feelings :-)
Comment: True but it's hardly a detail. (And I fear not many will check the reference.) Just one little edit ... :(
Comment: @Scortchi: you're right, I changed it
Answer: This approach definitely does corrupt the model.  The output of logistic regression is meant to convey the probability of an event given some configuration of independent variables.  Explicitly removing large amounts of data because of the value of the dependent variable will definitely harm this aspect of the model's output.
It sounds like your goal is to classify as opposed to using these probabilities, so if the skew is leading to two many false negatives, you should produce an ROC curve and determine if there is some threshold which produces an acceptable amount of false positives/negatives.
The problem that you discuss has been studied greatly and is often referred to as one-class classification.  When the vast majority of the data is "normal" and the goal is to detect "anomalies" (for example flight malfunction) there are many methods which essentially focus on understanding normality as opposed to understanding anomaly.
Comment: From what I am reading, it appears I may be able to estimate the model using my 'enhanced' dataset created by selecting on Y, then correct the intercept manually using the King and Zeng method (base-rate correction). Can you speak to that at all?
Comment: While I've heard of the King and Zeng method, I do not know it.  I may look into it, but can't make any promises.  I do strongly suggest you look into methods of "one class classification" though.


There are five boxes consisting of several identical coins.

Question: There are five boxes consisting of several identical coins. The first box contains 4 coins, the second box contains 9 coins, the third box contains 7 coins, the fourth box contains 10 coins, and the fifth box contains 3 coins. Some coins are taken from the boxes such that at least a coin taken from each box. In how many ways in taking the coins, such that the numbers of the coins taken are 12?
My attempt was solving it using repetition combination of which, there are only seven coins remaining considering that at least a coin taken from each box. By using the repetition combination, i.e. taking 7 from 9 (five boxes and four borders), resulting in 36 ways. However, there is no 36 ways in the choices.
Comment: Note that you cannot take all seven of the extra coins beyond the initial one coin from each from the fifth box... there were only three coins available to take.
Comment: Are you familiar with [stars and bars](https://en.wikipedia.org/wiki/Stars_and_bars_(combinatorics))?
Comment: @Fimpellizieri clearly, yes (*whether or not the OP calls it that*), as evidenced by the OP saying "*five boxes and four borders.*"  That was not the source of the OP's mistake.  The mistake was in that some of the outcomes the OP counted were some outcomes which are impossible due to the limited number of coins available in certain boxes.
Comment: @Fimpellizieri, yes, now I just realized my fault by seeing the comment of Jmoravitz.
Comment: If you're not doing formal power series, there is a modification one can apply to stars and bars when dealing with this kind of question, where the variables have both a lower and an upper limit. It uses the principle of inclusion-exclusion, and is a bit of work, but produces closed formulas. [This answer](https://math.stackexchange.com/questions/1649039/how-many-solutions-for-an-equation-with-simple-restrictions/1649090#1649090) explains it.
Answer: You will need to find the coefficient of $x^{12}$ in the series $(x+x^2+x^3+x^4)(x+x^2+x^3+...+x^9)(x+x^2+x^3+...+x^7)(x+x^2+x^3+...+x^{10})(x+x^2+x^3)$
i.e. the coefficient of $x^{7}$ in the series $(1+x+x^2+x^3)(1+x+x^2+x^3+...+x^8)(1+x+x^2+x^3+...+x^6)(1+x+x^2+x^3+...+x^8)(1+x+x^2)$
which can be further simplified to the coefficient of $x^{7}$ in the series $(1-x^4)(1-x^9)(1-x^7)(1-x^{10})(1-x^3)(1-x)^{-5}$
Well, it can be furthur simplified leaving all the power greater than 7 and power of sevel itself will cancel.
$(1-x^3-x^4...)(1-x)^{-5}$
=$C(5+7-1,5-1)-C(5+4-1,5-1)-C(5+3-1,5-1)$
=$C(11,4)-C(8,4)-C(7,4)$
Comment: why there is no series for the fifth box?
Comment: You are missing the fifth box.  There is also the question of if leaving the answer in this form is acceptable in whatever context the OP is working.  I can see arguments both for and against this.  If we were to assume the OP has a simple calculator and was required to give a numerical answer, this doesn't help that much.  It isn't wrong per se however, and is arguably just as simplified as a solution as one that leaves things as unsimplified binomial coefficients...
Comment: When converting to binomial coefficients, you seem to have made a mistake, namely with the second term, these should each have been $5-1$ rather than $5$.
Comment: Thanks! Changed it. I missed it while solving
Comment: Thanks! yeah it should be n-1. :) i always seem to goof up formula
Answer:
Num available: 4,9,7,10,3  -  Num needed to take: 12

Take one from each box ahead of time to handle the condition that we need to take at least one from every box

Num available: 3,8,6,9,2   -   Num needed to take: 7

Now... if we were to ignore the upper limits on each box, the number of ways to take 7 objects from 5 boxes would be:
$$\binom{7+5-1}{5-1}=330$$
You said something about $36$, which implies to me that you have made a mistake or learned stars and bars incorrectly.  With $k$ distinct boxes and $n$ identical balls there are $\binom{n+k-1}{k-1}$ ways to place the balls in the boxes (or as in our case, remove balls from the boxes/coins who have an unlimited supply)
Of these outcomes however, you counted some impossibilities, such as where you took too many coins from the first box, so let us remove those impossibilities.  If we took too many from the first box, that means we took at least four more from the first box, putting us at the situation of:

Num available: -1, 8,6,9,2 - Num needed to take: 3

Still five boxes, and needing to take only three more, there are $\binom{3+5-1}{5-1}=\binom{7}{4}=35$ bad outcomes where we took too many coins from the first box.  We similarly count how many bad outcomes there were for having taken too many from the third or fifth box.
Note however, that in counting the number of bad outcomes as a result of taking too many from the first box and counting the number of bad outcomes as a result of taking too many from the fifth box, we accidentally counted one of these outcomes twice... the one where we took too many from both the first and fifth box simultaneously.  Correctly applying inclusion-exclusion then we arrive at a final answer of:
$$\binom{7+5-1}{5-1}-\binom{3+5-1}{5-1}-\binom{0+5-1}{5-1}-\binom{4+5-1}{5-1}+\binom{0+5-1}{5-1}$$
$$ = \binom{11}{4}-\binom{7}{4}-\binom{4}{4}-\binom{8}{4}+\binom{4}{4} = 225$$
Answer: After taking one coin out of each box the boxes have contents $2,3,6,8,9$, and we have to pick $7$ more coins. From the first box we can take $0,1,2$ coins, and from the second box $0,1,2,3$ coins. This gives $1,2,3,3,2,1$ ways to take $0,1,2,3,4,5$ coins from these two boxes. The remaining $7,6,5,4,3,2$  coins to pick can be taken arbitrarily from the three other boxes, with one exception: you cannot take all $7$ coins from the $6$-coins box. Using stars and bars for these pickings we therefore obtain the following total number of admissible cases:
$$N=1\cdot\left({9\choose2}-1\right)+2\cdot{8\choose2}+3\cdot{7\choose2}+3\cdot{6\choose2}+2\cdot{5\choose2}+1\cdot{4\choose2}=225\ .$$


Topology and countable base

Question: I wish to show that if the topological space X has a countable base, then every base contains a countable base. I know if if a space has a countable base, then it has countable dense subset but they are not the same statements apparently~
Comment: No, it's not the same.
Comment: Yaa so I am wondering how to approach the problem of proving every base contains a countable base~
Comment: @Homaniac: Under what conditions? That a countable base exists?
Comment: Hint: if $\mathcal B$ is countable, then $\mathcal B\times\mathcal B$ is countable.
Comment: and the latter is a subset of the former?
Answer: No, it is not true that a topological space $X$ has a countable base if and only if it has a countable dense set (only the forward implication holds).  Consider $\mathbb R$ with the topology generated by the basis $\{[x,x+\varepsilon):x\in\mathbb R, \varepsilon>0\}$ (sometimes called the lower limit topology).  Then, as with the standard topology, $\mathbb Q$ is a countable dense set.
However, there is no countable base.  For if $\mathcal B$ is a base, then for each $x\in\mathbb R$ there is some $B_x\in\mathcal B$ such that $x\in B_x\subset[x,x+1)$.  If $x\neq y$, then $B_x\neq B_y$, and therefore $\mathcal B$ is uncountable.

Here is how you can get started with your question.  Suppose $\mathcal B=\{B_n\}$ is a countable base for $X$, and let $\mathcal C$ be a base for the topology on $X$.  Whenever it is possible, choose some $C_{n,m}\in\mathcal C$ such that $B_m\subset C_{n,m}\subset B_n$.
Comment: I only wish to show that the topological space X has a countable base implies every base contains a countable base.

Topology and countable base


In a meta-analysis, do you report the full N of each study or just the experimental n?

Question: I just noticed that one meta-analysis (Jeon & Day, 2016) reported only the size of the experimental group. For example, when I look at the particular N sizes for the Robb & Kano (2013) studies, they report, in Table 3, the the economics majors as N = 543, which is the experimental cohort size (Robb & Kano Table 5), but this excludes the 555 students in the control cohort.
The effectiveness of ER on reading proficiency: A meta-analysis. Reading in a Foreign Language, 28(2), 246–265 by Jeon, E.-Y., & Day, R. R. (2016)
Effective extensive reading outside the classroom: A large-scale experiment. Reading in a Foreign Language, 25(2), 234–247 by Robb, T., & Kano, M. (2013)
Comment: If a study has equal sample sizes for both the experimental and the control group, I guess you could report just one of these but mention why. If a study has unequal sample sizes, reporting just the sample size for the experimental group seems incomplete to me (unless you know the ratio of the two sample sizes and can work your way back to the sample size of the control group). It's poor practice, in my view, to not report both sample sizes for reproducibility purposes in a meta-analysis.
Answer: Short answer: They should have reported the N for both arms.
Long answer: Reporting is often a weakness of any research project. The Equator network provides suggestions on what to report based on the study design. For systematic reviews and meta-analyses in healthcare there are a few guidelines that can be followed (e.g. Methodological Expectations of Cochrane Systematic Reviews MECIR, PRISMA and its extensions, etc.).
In general, it better to report all data used in a meta-analysis at the study level for both arms rather than cherry-pick what is reported or only reporting a summary effect estimate for each study (or just the results of the meta-analysis). This provides transparency.
Hope this helps.


Coloco uma imagem no meu link com bootstrap, e a imagem não carrega de jeito nenhum... Alguém sabe o que está acontecendo?

Question:

Ta aí o código, ja tentei colocar um botão, não vai, não chega nem a aparecer um espaço clicável na tela...
Comment: então tem que mostrar o código pra gente tentar ver o que está provocando o erro.
Comment: O endereço da imagem deve estar errado. Coloca o endereço absoluto. Se mesmo assim o erro continuar deve ser proveniente de outra parte do código que não está na postagem.
Comment: Tentei colocar o caminho absoluto e ainda nada... Pode ter algo haver com a versão do Bootstrap? Por que eu não sei o que pode haver de errado com o resto do código
Comment: posta o código completo ai pra gente ver
Answer:
Lazy loading é um padrão de projeto de software, comumente utilizado
em linguagens de programação, para adiar a inicialização de um objeto
até o ponto em que ele é necessário. Isso pode contribuir para a
eficiência no funcionamento de um programa, se utilizado
adequadamente. Wikipédia

Por que você quer colocar carregamento preguiçoso na sua logo se ela será a primeira a aparecer? isso não faz sentido e não foi criado para isso, agora que a Microsoft acabou com o edge e foi para o chromium, você pode usar a imagem no formato .webp que é mais leve e resolverá o seu problema de baixo desempenho.
Retire loading="lazy" que irá funcionar, além disso, escreva algo no atributo alt="" para você saber quando o link da imagem não está funcionando


Microsoft Windows Web Server 2008 + Microsoft SQL Server 2008 Web Edition

Question: I am a totally rookie in this area.
I am about to rent a Dedicated Server, that has Microsoft Windows Web Server 2008 installed in it.
The plan is to put a web application that we have developed in it. But for this web application, we need Microsoft SQL Server 2008 Web Edition. I have read somewhere that both are compatible… but I am not sure. So I would like to have some confirmation on this subject if possible.
Also, in the description of this server, it says: "Windows Server 2008 Web Edition (1 processor)". This obviously means that it only uses 1 processor… I didn't know that could be applied to Windows itself… I thought it was just the SQL Server (since the license is Per Processor). Is this some typo? Or the Windows Web Server is Per Processor too?
Also, performance wise, am I choosing the right Windows Server and SQL Server versions? (I remember you, that this is a Web Application developed by us). I think that other distributions of Windows Server and SQL Server are only a matter of features, and not performance… right?
PS: I know, I know… too many questions on a single post. But if you think that is more correct to create multiple questions…. I can do that.
Answer:
They are compatible.
No, it means that the price is for one processor. These are licensed per processor. You HAVE to license as many processors as the server has. For both.
All are per processor. This is how SPLA licensing works. For that they are - cheap ;) And no CAL.
Try using expres edition if money is an issue. otherwise - you will see when you run into an issue. The next REAL step up would be Enterprise server for SQL - vey expensive. Performance depends a LOT on hardware (disc subsystem - do NOT go with less than 2 drive SETS to start with) and exact programming / requirements.

I have seen servers vomit on 2 requests per second - because idiot programmers put in no indices. It really depends. SQL Server is really high performane ,b ut SQL Server REALLY depends on how you use it and what you need IO wise.
NOTE: If this is a start for you, check MS BizSpark initiative - quite a lot of low cost stuff for the first three years.
Comment: thank you for your answer. But you said: "You HAVE to license as many processors as the server has". And on their website the processor says: "1x Intel Quad Core Xeon X3210", and the software says: "Windows Server 2008 Web Edition (1 processor)". Does this means that the other 3 cores won't be used?!!!!
Comment: It says PROCESSOR, not CORE. Ms was the first to say "a physical processor is one SOCKET, regardless how many cores".
Comment: Great! Thank you a lot for your comments. I would give you extra points but I need 15 points to "mark has usefull". Once again, thank you.


Write to multiple text files, grouped by attribute value using XSLT-2.0

Question: Given XML like this:
<?xml version="1.0" encoding="UTF-8"?>
<tmx version="1.4">

    <prop type="x-acme-db-name">test_deu</prop>

    <tu srclang="en">
      <prop type="x-acme-source-ipath">English/GUID-asdf.xml</prop>
      <prop type="x-acme-target-ipath">German/GUID-qwer.xml</prop>
      <prop type="x-acme-tm-uda-SID">Index</prop>
      <tuv xml:lang="en">
        <seg>text:editing</seg>
      </tuv>
      <tuv xml:lang="de">
        <seg>Text:bearbeiten</seg>
      </tuv>
    </tu>
    <tu srclang="en">
      <prop type="x-acme-source-ipath">English/GUID-lkjh.xml</prop>
      <prop type="x-acme-target-ipath">German/GUID-poiu.xml</prop>
      <prop type="x-acme-tm-uda-SID">Index</prop>
      <tuv xml:lang="en">
        <seg>text:finding</seg>
      </tuv>
      <tuv xml:lang="de">
        <seg>Text:suchen</seg>
      </tuv>
    </tu>
    <tu srclang="en">
      <prop type="x-acme-source-ipath">English/GUID-zxcv.xml</prop>
      <prop type="x-acme-target-ipath">German/GUID-vbnm.xml</prop>
      <prop type="x-acme-tm-uda-SID">Index</prop>
      <tuv xml:lang="en">
        <seg>text:replacing</seg>
      </tuv>
      <tuv xml:lang="de">
        <seg>Text:ersetzen</seg>
      </tuv>
    </tu>

</tmx>

I'm trying to generate one file per xml:lang (2 in this case), where each file contains the  text for the appropriate lang.
I started with the code below, but I see the variable is being set to all of the lang values at once ("en de en de en de").  Any help on how to fix this to achieve the desired output would be greatly appreciated.
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
    <xsl:output method="text" encoding="UTF-8" />
    <xsl:template match="/tmx/body">
        <xsl:for-each-group select="tu/tuv/seg" group-by="//@xml:lang">
            <xsl:variable name="lg" select="//@xml:lang" />
            <xsl:result-document href="foo_{$lg}.txt" method="text">
                <xsl:copy-of select="current-group()" />
            </xsl:result-document>
        </xsl:for-each-group>
    </xsl:template>
</xsl:stylesheet>

Answer: Using the absolute path //@xml:lang as you did will return the attributes of a node set consisting of all languages found in your file.
The group-by expression is usually given as relative path to the expression selecting the group. In your case this should simply be ../@xml_lang since the grouping criterion resides in the parent node. This will give you exactly one language per <seg>.
Comment: @dbl, I think instead of `` you simply want ``, to have items in each group output on a new line.
Comment: Excellent!  Thanks, Marcus!  That did the trick.  (Although I now have to figure out how to get newlines between each segment in the output files...)
Comment: This is usually (safely) done by adding newlines coded as `
`.
Comment: Right, but I don't think inserting that in the result-document block will help.
Comment: Oh, then I misinterpreted your term _between each segment in the output files_. Where again do you want to put the newlines exactly? Outside the result documents? I'm confused...
Comment: Sorry...  I implemented your fix of ../ for //, and I get two files, as desired.  However, the contents of foo_de.txt looks like this:
    Text:bearbeitenText:suchenText:ersetzen

I'd like a newline between each segment, but adding the newline text within the result-document block (after copy-of) doesn't get it.
Comment: This surprises me. Are you Windows based? Maybe `
` will help?
Comment: I work both in Linux & Win.  On Win now, but I think the issue is that the newline is added after the group, rather than between each member of the group.  Switching from copy-of to value-of at least gets me whitespace between each segment...
Comment: Aha!  Within the result-document, I changed to for-each over the current-group, with a block of value-of select=".", then the newline text.  Thanks again for all your help!
Comment: Nice, Martin.  Much more concise (and less wasteful) than my solution.  Thanks!
Answer: Solution, thanks to Marcus & Martin:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
    <xsl:output method="text" encoding="UTF-8" />
    <xsl:template match="/tmx/body">
        <xsl:for-each-group select="tu/tuv/seg" group-by="../@xml:lang">
            <xsl:variable name="lg" select="../@xml:lang" />
            <xsl:result-document href="foo_{$lg}.txt" method="text">
                <xsl:value-of select="current-group()" separator="
" />
            </xsl:result-document>
        </xsl:for-each-group>
    </xsl:template>
</xsl:stylesheet>


Azure pipeline on PR validate file before merging to master branch

Question: I have master branch which has multiple yaml files. Every file on update will be pushed to feature/{filename.yaml} and with PR will be merged to master branch.
But before merging I need to validate that file from feature has valid YAML syntax.
If syntax is not valid build will fail and not be merged to master branch.
But if everything is ok I need to send content of yaml file to API endpoint, and then merge it to master branch.
Thanks
Comment: At which step did you encounter problems?
Answer: You can use Files Validator task to validate file.
[


Changing Azure User Access Administrator?

Question: The root User Access Administrator that is inherited by all our subscriptions is assigned to an account of an ex-employee.  We're keeping that account alive so that we can continue to make changes as needed, but it's a less than ideal setup.  Is there some way we can go about removing that account/reassigning those privileges?  Or are we stuck with that ex-employee account forever (even the Azure consultants we enlisted were unable to change it)...
Answer: There are several terms:

Azure AD tenant (something.onmicrosoft.com) that is directory of all users.
In properties you can set that all users with "Global Admin" role have access to all Azure subscriptions.
Azure subscription, has owner of user, this can be changed using portal.azure.com or via support ticket. It is typically user from organization (Azure AD tenant) in charge of the project / payments.

You can add additional users (free) in Azure AD tenant and provide access permissions to Azure subscription or resource groups (using IAM)
Comment: Has not enabled me to solve my problem yet, but is helpful information in my search
Comment: You can simply use Azure AD roles and Azure AD Security groups to achieve permissions on Azure subscription or resource group level. You should not reuse one account for multiple persons due to security concerns.
Comment: That's the problem.  I can't find where this role is being assigned to change it the proper way...
Comment: portal.azure.com > suscriptions > IAM
Comment: That screen does not allow changing this attribute at the root level
Answer: You can remove this by logging in as the old user, going to the Azure Portal, Azure Active Directory.
Then select the Properties screen in the left menu.
https://portal.azure.com/#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Properties
The bottom Yes/No is for Access Management for Azure Resources, this will control the user access administrator.
Automated
Through Powershell or CLI you can also remove this as a Global Admin, with the User Access Administrator role active(root level Azure Subscription access is required):
az role assignment delete --assignee "email@example.com" --role "User Access Administrator" --scope "/"

Or AZ CLI:
 az role assignment delete --assignee firstname.lastname@example.com --role "User Access Administrator" --scope "/"

Background info
The User Access Administrator is a temporarily solution to gain access to the Azure Subscriptions which are tied to the same Azure Active Directory.
The Global Administrator is the only one allowed to do so, and after gaining access, new/direct permissions can be applied to the Azure Subscription, after which the User Access Administrator role needs to be disabled again. (principle of least privilege)
Access to Subscriptions tied to a different Azure AD can not be resolved this way.


How to get the header, the same style and full width as the footer here

Question: I am playing around with bigcommerce at the moment and I am trying to recreate the footer structure for the header. You can see here:
http://thespeedfactory.mybigcommerce.com/
If you look at the footer, how it is full width but the content is central within it.
I want the header to be exactly the same, black with pink/white highlights.
Ive tried moving around the structure within bigcommerce, but I am having a brain failure in getting it to do and look how I want despite knowing it is based around containers and margins.
Any guidance is appreciated.
Comment: If you inspect the website you'll see that under the body there are two divs. One for the Header and content and one for the footer. Try to replicate that on your conditions.
Answer: If I understand you correctly, you want:

the header (#Header) to span the entire width of the page
the footer (#ContainerFooter) to span the entire width of the page
the header (and footer to have the same styling (colors, etc.)
the content area (#Wrapper) to stay a fixed width and centered on the page

To do this, add the following css:
#Container {width:100%;}
#Header {width:100%; margin:0, auto;}

The above css allows the header (by way of its parent container) to stretch the width of the browser page. You'll notice #Wrapper is shifted to the left. Add this:
#Wrapper {margin:auto;)

This centers the #Wrapper.
Your structure should be in place and now you can add your colors, etc. to the #Header to make it match the footer.
Answer: This is pretty basic html/css.
 Just create a div, place a container in it and start styling.
HTML:

content

CSS:
#header {
    width: 100%;
    height:400px;
    background:black;
    position:absolute;
    border-top:3px solid #ff25a7;
}
#container {
    width:90%;
    height:300px;
    margin:0 auto;
}
#container p {
    font-size:30px;
    padding:10px;
    color: #ff25a7;
}

Here's a jsFiddle to help you get started.
Answer: You can try giving the header the same class as the footer and afterwards (if the footer's position is absolute bottom), set the position to absolute top:0px;


How to return a boolean in a promise?

Question: Inside submitButton event, I would like to check if there are any users registered or not.
Suppose there are 5 users registered:
 getUsers = function () {
    return new Promise(function (resolve, reject) {
      resolve(5);
      reject(0);
    });
  }

Inside the checkregisteredUsers, I call getUsers and then I check if there are any users. If there are any users, then return true, otherwise false. I set state of UsersFound.
  checkRegisteredUsers = () => {
    return this.getUsers().then(result => {
      return this.setState({ UsersFound: result > 0 ? true : false };
    }).catch(reason => {
      console.log(reason);
    });
  }

Now, inside the function submitButton, I would like to check for true or false, if there were users registered.
submitButton = () => {
      this.checkRegisteredUsers().then(result => {
      this.setState({ UsersFound: result > 0 ? true : false });
    });

    if(!this.state.UsersFound){
    return; **//exit code here and the rest of the code below should not execute and show the dialog box**
    }
    // run some other code
  }

When I call submitButton, the state of UsersFound is not set yet. If I click a second time then it is set.
How can I set it inside a promise? Or How can I check for true/false inside the submitButton using promises?
EDIT:
The question is: how can I return a boolean from a promise as I would like to check something like this:
submitButton = () => {
 if(this.checkRegisteredUsers()){
  return; //the rest of the code below should not execute
}
 //if check users is true then the code below should not execute
 // some other code
}

I don't want a promise returned, I want a boolean.
Comment: Put the logic inside then
Comment: remove the `then`  and `catch` callback from the `checkRegisteredUsers` method.
I recommend using `asyn/await` statements to handle async functions as well.
Comment: are you suggesting like this:   `checkRegisteredUsers = () => {
    return this.getUsers();
  }`

I still can't set the state of userChecks. it still returns false the first time I click on the button
Comment: Does this answer your question? [How do I return the response from an asynchronous call?](https://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call)
Answer: Long story short: You cannot make a function that returns a promise return a boolean. If you have some asynchronous action (like a fetch to get users from some service), that is inherently asynchronous. You cannot have such a function return a boolean, because by the time the function itself ends, you don't have the information to return true or false.
The closest you can get to that is using async/await. But async functions still only return Promise objects. You can make it look different using async/await, but it is just syntactic sugar. You are still effectively running all of the code after the await as though it were a function passed to then.
As an example, this:
//Some function that will resolve or reject.
const getUsers = () {
  return new Promise(function (resolve, reject) {
    resolve(5);
  });
}
const foundUsers = async () => {
  const users = await getUsers();
  return Boolean(users);
}
const LoggedIn = () => {
  const [usersFound, setUsersFound] = useState(false)
  const onSubmit = async () => {
    const found = await foundUsers();
    if (found) {
      //do stuff
    }
    setUsersFound(found)
  }
  return (
    <>
      <button onClick={onSubmit}>Submit</button>
      { usersFound ? <Dialog /> : <WhateverElse /> }
    </>
  )
}

Is the same as this:
const LoggedIn = () => {
  const [usersFound, setUsersFound] = useState(false)
  const onSubmit = () => {
    foundUsers().then((found) => {
      if (found) {
        //do stuff
      }
      setUsersFound(found)
    })
  }
  return (
    <>
      <button onClick={onSubmit}>Submit</button>
      { usersFound ? <Dialog /> : <WhateverElse /> }
    </>
  )
}

Comment: the intention is to not execute the rest of the code insde submit button if users were found. How can I do that? That is why I am trying to use the true false here.
Comment: I replaced the answer to something I think better answers what you're asking "how can I return a boolean from a promise". Hopefully this makes more sense.

How to return a boolean in a promise?


jquery ui resizable on click of buttons

Question: I am trying to do the resizable control on click of button.
I have used the resizable plugin in jquery ui to resize the image.
But now i need to do image increase or descrese size using plus and minus buttons.
Do anyone know how to do it??
Comment: Code snippet or jsFiddle please ...!
Comment: without seeing any of your code - you will probably need to set up the buttons so that they increase or decrease the size when clicked, you will most likely end up using the `.width()` and `.height()` jquery functions to get and set the image values. here is the example for width...http://api.jquery.com/width/
Comment: a fiddle would be helpful :)
Answer: I don't think you want to use jQuery UI for this.  That provides a standard set of features for the end-user to interact with.  If you want to mess with the height and width of a div, do something directly like this...

-

+

$("#more").click(function() {
    var h = $('.box').height();
    h = h + 20;
    $('.box').css({height: h, width: h});
});

$("#less").click(function() {
    var h = $('.box').height();
    h = h - 20;
    $('.box').css({height: h, width: h});
});

Working fiddle here... http://jsfiddle.net/TUwV2/

jquery ui resizable on click of buttons


I'm trying to use pyplot.scatter() but I'm getting an invalid syntax error

Question: I'm trying to use pyplot.scatter() but I'm getting the following error:
<code> plt.scatter(y=temperature_planet,x=t,s=2, color="red")
   ^
 SyntaxError: invalid syntax
</code>
You can see the issue is on the last line of the code. I'm not sure what's going on
My code:
<code>import numpy as np
import matplotlib.pyplot as plt
import time
import decimal

t = 0
temperature_planet = 200
temperature_atmosphere = 250

epsilon = decimal.Decimal(0.25)
dt = 60*10
heat_capacity = decimal.Decimal(1E5)
insolation = decimal.Decimal(1370)
sigma = decimal.Decimal(5.67E-8)
planet_radius = decimal.Decimal(6.4E6)

circle = decimal.Decimal(np.pi)*planet_radius**decimal.Decimal(2.0)
sphere = 4*decimal.Decimal(np.pi)*planet_radius**decimal.Decimal(2.0)

plt.scatter(y=temperature_planet,x=t,s=2, color="red")
plt.ion()
plt.xlabel("Time (S)")
plt.ylabel("Temperature (K)")
plt.legend(loc='upper left')
plt.show()

print(temperature_planet)
while True:
   temperature_planet += dt*(circle*insolation + sphere* epsilon*sigma*
   temperature_atmosphere**decimal.Decimal(4.0) -
   sphere*sigma*temperature_planet**decimal.Decimal(4.0)/heat_capacity
   t += dt
   plt.scatter(y=temperature_planet,x=t)
   plt.pause(0.005)
   time.sleep(0.05)
</code>
Comment: You're missing the closing `)` at the end of the previous line
Answer: Check second last line sphere = 4*(np.pi*planet_radius**2.0
The Parenthesis are missing at the end.
Comment: I changed my code to look as it does now, but now I get an issue with
t += dt. where the error is: invalid syntax


How to translate minimization problem containing a min function into a Mixed Integer Problem (MIP)?

Question: I have the following optimization problem:
$\min J = \sum_{i=0}^{N} k_i = k_0 + k_1 +\dots+ k_N$
s.t:
$k_i = f_i(z_i)$
where
$f_i(x) = \begin{cases}
    x,& \text{if } x\leq P_i\\
    P_i,              & \text{otherwise}
\end{cases} = \min(x, P_i)$
and $z_i$ is a linear combination of other constants and decision variables $z_i = \sum_{j=0}^{M} K_{ij}(Y_{ij} - y_{ij})$

$P_i >0, \, K_{ij}>0, Y_{ij}>0$ are constants
$y_{ij} \geq 0,\, k_i \geq 0$ are continuous decision variables

I do not know how to write the minimization problem containing the Objective with the summation over min functions into a Mixed Integer programming problem suitable for Pythons pulp package for example.
I do not have a mathematics background and did my research for many days on this topic, reading papers or articles, but it is difficult to correctly identify the problem when not coming from a mathematics background.
Does anyone know how to solve this and/or how to translate this into a MIP formulation?
Answer: You want to minimize
$$\sum_{i=0}^N \min\left(\sum_{j=0}^M K_{ij}(Y_{ij}-y_{ij}), P_i\right).$$
Introduce a continuous variable $s_i$ to represent the outer summand and a binary variable $b_i$ to indicate which minimand is active.
Let $M^1_i$ be a small constant upper bound on $\sum_{j=0}^M K_{ij}(Y_{ij}-y_{ij}) - s_i$,
and let $M^2_i$ be a small constant upper bound on $P_i - s_i$.
The problem is to minimize $\sum_{i=0}^N s_i$ subject to linear constraints
\begin{align}
s_i &\le \sum_{j=0}^M K_{ij}(Y_{ij}-y_{ij}) \tag1 \\
s_i &\le P_i \tag2 \\
s_i &\ge \sum_{j=0}^M K_{ij}(Y_{ij}-y_{ij}) - M^1_i b_i \tag3 \\
s_i &\ge P_i - M^2_i (1-b_i) \tag4 \\
\end{align}
Constraint $(3)$ enforces $b_i=0 \implies s_i \ge \sum_{j=0}^M K_{ij}(Y_{ij}-y_{ij})$.
Constraint $(4)$ enforces $b_i=1 \implies s_i \ge P_i$.

Given $0 \le y_{ij} \le Y_{ij}$, you can take
$$M^1_i = \sum_{j=0}^M K_{ij}(Y_{ij}-0) - P_i = \sum_{j=0}^M K_{ij}Y_{ij} - P_i$$
and
$$M^2_i = P_i - \sum_{j=0}^M K_{ij}(Y_{ij}-Y_{ij}) = P_i.$$
Comment: Thank you for the quick answer. I will try and implement this approach. How do you have to choose the $M_i^1$ and $M_i^2$ values?
Comment: Do you have a good upper bound on $y_{ij}$?
Comment: Yes, it is $Y_{ij} - y_{ij} \geq 0$, so the upper bound is $y_{ij} \leq Y_{ij}$.
Comment: Thank you very much for the quick answers! This makes so much sense now. I have another question about the constants: Can i choose any constant as long as it does fit with the whole system, e.g. icould also choose $M_i^1$ to be $M_i^{1} = \sum_{j=0}^{M} K_{ij}Y_{ij}$, is that correct?
Comment: You can choose a larger value than what I recommended, but doing so will weaken the strength of the linear programming relaxation.
Comment: And the binary decision variables $b_i$ are optimized to indicate the active minimand - Why does this happen? Why are they going to have the correct values after optimization? This is the only thing i cannot grasp at the Moment, it feels like magic (but surely it is not)
Comment: @MarkusU The solver will explore a binary decision tree, where at each node a binary variable $b_i$ will be selected and two child nodes will be created, one with $b_i=0$ and the other with $b_i=1$. (3) and (4) will then assert one choice for minimand in each of the children, and that selection remains in force for the subtree descending from the child node. So the decisions are the values of the $b_i$, and activation of the correct minimands is the consequence of those decisions, rather than the other way round (selecting the minimands and getting the binary values as consequences).
Comment: @RobPratt I implemented the solution and it worked. But I have another constraint, that makes the solution infeasible. I want to implement and add the additional constraint $\sum_{i=0}^{N} y_{ij} = S_j$ with $S_j$ being  supply values. How can i include this to be optimal? It seems like the additional constraint and $y_{ij} \leq Y_{ij}$ is not possible for the solver to be both true.
Comment: To help diagnose the issue, I recommend solving with a constant zero objective function, with your new constraints, and without (1) through (4).


how to model counfounding variable in mixed models?

Question: subject <- factor(rep(seq(1,28),each=4))
y <- c(5,4,9,3,4,4,2,1,10,7,8,7,1,2,1,1,5,10,1,7,3,2,1,4,3,8,7,3,1,1,2,1,15,10,20,11,2,2,1,3,11,12,9,7,2,3,1,2,11,9,8,9,3,4,2,1,5,4,8,3,7,6,27,18,1,7,7,6,13,5,8,1,7,11,8,9,3,2,5,3,1,9,7,4,2,1,7,4,15,9,6,7,15,23,1,23,12,1,1,17,2,13,21,23,11,9,2,3,7,6,5,4)

gender <- factor(rep(c(rep("Female",times=16),rep("Male",times=12)), times=2))
day <- factor(rep(c("day1","day2","day3","day4"),times=28))
score <- factor(rep(c("A","B","A","C","A","C","B","B","C","C","B","B","A","A"), each=4, times=2))
data <- data.frame(sub=subject, y=y, gender=gender, day=day, score=score)

gender is a confounding variable. What is the best model to account for gender when my interest is in the score:day interaction? Should I interact it with other predictors of interest:
m <- lmer(y ~ score*day*gender + (1|sub), data=data)

or simply add it to the model like this:
m <- lmer(y ~ score*day + gender + (1|sub), data=data)

Answer: If gender is a confounder then all you typically need to do is add it to the model, as per your 2nd model formula:
m <- lmer(y ~ score*day + gender + (1|sub), data=data)

Comment: You're welcome. If you have a good theoretical reason for why it should be interacted then go ahead and do that. I just said that typically all you need to do is to include the variable itself. If you are going to interact it with `score*day` then you are saying that the `score:day` interaction varies depending with `gender`. You will also have to make sense of the 2-way interactions `day:gender` and `score:gender`. But if that all makes sense in your study, that's fine.
Comment: Thank you @Robert. Would you mind explaining to me why that is it the case that I only need to add it to the model? What are the consequences of making gender interact with the other predictors? Could that be beneficial/correct in some cases?


Why the onProgressUpdate() of AsyncTask<> interface is never called?

Question: The onProgressUpdate() is never called, can you tell me why?
private class LoginMe extends AsyncTask<Context, Void, Boolean> {

    @Override
    protected Boolean doInBackground(Context... arg0) {
        doSomething();
        return true;
    }

    @Override
    protected void onProgressUpdate(Void... v) {
        super.onProgressUpdate(v);
        Log.d("Dev", "Im in onProgressUpdate()");
    }

    @Override
    protected void onPostExecute(Boolean result) {
        super.onPostExecute(result);
        if (result) {
            doOne();
        } else {
            doTwo();
        }
    }
}

Answer: You have to call publishProgress from within doInBackground manually.

doInBackground(Params...), invoked on
  the background thread immediately
  after onPreExecute() finishes
  executing. ... This step can also use
  publishProgress(Progress...) to
  publish one or more units of progress.
  These values are published on the UI
  thread, in the
  onProgressUpdate(Progress...) step.

http://developer.android.com/reference/android/os/AsyncTask.html


SAN booting Oracle VM from a UCS blade

Question: We have an Oracle VM Cluster that we are provisioning on a Cisco UCS B series blade chassis. It has 2 fabric interconnects (FC0 and FC1) provisioned to the (3) Oracle VM service profiles and it is set to boot from san with no local storage configured at all. The issue is that each time we reboot the Oracle VM Server, there is about an 80% chance that it will result in the below errors. I have been able to boot all three service profiles up at the same time with no issue at least three times throughout this process.

Setting up Logical Volume Management: /dev/sdq: open failed: No medium found    /dev/sdp: open failed: No medium found
Checking filesystems
fsck.ext3: Unable to resolve 'UUID=39b6935d-5c93-45aa-b54d-344fa171c40c'

This drops me to a 'Repair filesystem' prompt where a fsck gives me the same error. I am able to mount this uuid and view the data (from withing the Repair filesystem). Running 'blkid' shows that the boot partition and erroring partition's UUIDs match and they are mapped exactly between grub.conf and /etc/fstab.
As a test, I just zoned the fabric switches back to Single initiator, single target for this service profile, but this has provided no positive movement. Are there zoning or specific UCS service profile requirements that need to be met in order to boot from san with Oracle VM? Any help would be appreciated.
Don
Comment: Good question- I had the same problem with a linux VM during an eval of UCS.  We never bothered going forward (for a host of other reasons) and never figured out why boot from san worked under windows and redhat but not oracle vm.  I look forward to seeing the answers.
Answer: It looks like multi pathing boot from SAN is not supported in Oracle VM 3.0.3.
It seems like there are ways to get booting form SAN without multipathing working using iSCSI and I believe FC should work as well.
Here is a thread on Oracle's forumns that talks about this issue.

SAN booting Oracle VM from a UCS blade


arrays in PHP multiple choosing from checkboxes

Question: I have an array from checked checkboxes named with numbers which can be true or false
i want to get only then numbers that are 1.

For example:
Array ( [8] => 1 [13] => 1 [14] =>  [15] => 1 [16] =>  )

i want an array with numbers 8,13,15 because they are 1 in previous array.
Thanks for any help
Comment: where is your code or attempts to do anything?
Answer: Use array_filter
<?php
$result = array_filter($input, function($var){return $var==1;});

Comment: Thanks, it works, but im wondering now, how to work with the keys now, i just want to simply delete keys as id from database i tried foreach, but i cant get the key value, just the value "1" thanks
Comment: @Zelí I'm not sure what you are asking. Can you rephrase it?
Comment: `Array
(
    [8] => 1
    [13] => 1
)` i have this array, and i want to delete id 8 and 13 from database with query, but i cant get the values 8 and 13 :(
Comment: @Zelí Use manual, show at least minimal effort. There is ready function for this.
Answer: Just try with array_filter:
$input  = array(8 => 1, 13 => 1, 14 => null, 15 => 1, 16 => null);
$output = array_filter($input);

var_dump($output);

Output:
array (size=3)
  8 => int 1
  13 => int 1
  15 => int 1


Flux Integral using Calculus

Question: How do I do this question:

Let $\mathbf F=3x\mathbf i+5y\mathbf j$ and let $\mathbf n$ be the outward unit normal vector to the positively oriented circle $x^2+y^2=4$. Compute the flux integral $\int_C\mathbf F\cdot\mathbf n\,\mathrm ds$.

Also how does $n$ affect how I write my integral?
I am really confused because there are so many formulas, theorems, and variations.
Answer: Your equation of the Surface is given by $G(x,y) = x^2+y^2-4=0$. You obtain the unit normal vector by plugging this equation into the formula: $\vec{n} = \frac{grad G(x,y)}{|grad G(x,y)|}$. Then you can compute the scalar product $\vec{F} \vec{n}$.
Now you can Change to polar coordinates since you integrate over a circle with radius $2$. Set $x=2cos \phi,y=2sin \phi$ and the Surface element is given by $ds = 2 d \phi$.
Comment: This would be a double integral?
Comment: This will be a single integral because you integrate only over all possible angles $\phi$ of the circle. The radius (=2) of the circle remains constant on the "Surface" of the circle.
Comment: How did you find the surface element? Also doesn't the magnitude of the gradient get cancelled with ds?
Comment: $gradG = (2x,2y)^T,|gradG|= 2 \sqrt{x^2+y^2}$. $\vec{F} \vec{n} = 3x* \frac{x}{\sqrt{x^2+y^2}} + 5y* \frac{y}{\sqrt{x^2+y^2}} = \frac{3x^2+5y^2}{\sqrt{x^2+y^2}}$. Now you can use polar coordinates with Surface element $ds = rd \phi$ (is the length of an infinitesimal circle arc).

Flux Integral using Calculus


R - Treemap Package not sorting by size

Question: This has never been an issue before, but recently when I try to produce treemaps (using the R treemap package) that have rectangles ordered by value I'm unable to do so. Rather rectangles are ordered alphabetically by index name. Have I misunderstood the recent documentation at any point or is this a non-reproducible issue (one to do with my package install)?
Reproducible example below. The resulting treemap is alphabetically ordered when I want it ordered by 'vals' size
<code>library(treemap)

names <- as.character(c("a", "c", "b", "d"))
vals <- c(10, 20, 65, 5)
tree_ex <- as.data.frame(cbind(names, vals), )
tree_ex$vals <- as.numeric(as.character(tree_ex$vals))

treemap(tree_ex, index='names', vSize = "vals",
        algorithm = "pivotSize",
        sortID = "vals")
</code>
Comment: very strange indeed. Even the [documentation](https://cran.r-project.org/web/packages/treemap/treemap.pdf) says "**By default, large
rectangles are placed top left**". But even when leaving everything as default it doesn't appear to work
Comment: The documented behavior held true for me previously so I suspect this must relate to a recent update. If others are getting the same issue I may raise it with the package author.
Comment: Are you able to down-grade to a previous version to test it?
Answer: The package is broken. Reproducible example below. See also Issue #30 which I have just filed on github for treemap package
Code to reproduce the broken behaviour:

<code># create data set
set.seed(1234)
df <- data.frame(A=LETTERS[1:12],size=seq(12,1),sort=sample(12))

# install the last version with sortID respected and plot treemap
devtools::install_github("treemap", user="mtennekes", subdir="pkg",ref="5ca7c0b562e4a60bab111752210d6aba8995f55a")
library(treemap)
treemap(df,index="A",vSize="size",algorithm="pivotSize",sortID="sort")

# install the version which breaks sortID and plot treemap
devtools::install_github("treemap", user="mtennekes", subdir="pkg",ref="d8817db2805058f6a36db9f882790681961bc052")
library(treemap)
treemap(df,index="A",vSize="size",algorithm="pivotSize",sortID="sort")
</code>

UPDATE 2016-03-23
Package fixed Commit 0224354

R - Treemap Package not sorting by size


Gostaria de saber como faço para deixar meus vídeos nessa configuração idêntica no ffmpeg?

Question: Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 202 MiB
Duration : 22 min 9 s
Overall bit rate : 1 278 kb/s
Writing application : Lavf57.2.100
Video
ID : 1
Format : AVC
Format/Info? : Advanced Video Codec
Format profile : High@L4
Format settings : CABAC / 3 Ref Frames
Format settings, CABAC : Yes
Format settings, ReFrames? : 3 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 22 min 8 s
Bit rate : 1 146 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Variable
Frame rate : 23.976 (24000/1001) FPS
Minimum frame rate : 23.974 FPS
Maximum frame rate : 23.981 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.023
Stream size : 182 MiB (90%)
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709
Audio
ID : 2
Format : AAC
Format/Info? : Advanced Audio Codec
Format profile : LC
Codec ID : mp4a-40-2
Duration : 22 min 9 s
Bit rate mode : Constant
Bit rate : 126 kb/s
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 kHz
Frame rate : 43.066 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 19.9 MiB (10%)
Language : Portuguese
Default : Yes
Alternate group : 1
Comment: Como assim `ffmpeg`, seja mais claro com sua pergunta. Pois você somente deu uma ficha técnica mas não específicou por qual meio deseja obter esses ajustes sobre o aquivo, a não ser o próprio `ffmpeg`. Você esta usando alguma distro GNU/Linux???
Comment: Não sou expert em codificar vídeos. sou bem leigo.. Esse foi um vídeo que peguei e tinha essas configurações, e gostaria de fazer um vídeo com as mesmas configurações. Não sei usar direito o ffmpeg ainda estou aprendendo. Por isso fiz a pergunta, gostaria de saber o comando que posso usar para obter esse resultado.
Comment: obrigado amigo... uso windows 7 64bits
Comment: obrigado amigo... uso windows 7 64bits  Tenho o ffmpeg instalado e só faço    o basico mesmo.
Comment: Saiba como executar comandos Linux no Windows. Em paralelo a isso, é comum que muitos usuários (você ou outra pessoa conhecida) pensem algo assim: "Estou querendo aprender sobre o Linux, mas não quero que o Windows saia da minha máquina".
Para tentar atingir boa parte desse público de usuários foi criado ferramentas que emulam um ambiente Linux no sistema Windows;
[Cygwin](https://www.cygwin.com/) e [Cash](https://github.com/dthree/cash) são dois programas para esta finalidade.
Comment: O **Cygwin** é uma DLL que emula o Linux dentro do Windows. O ***Cash**, não fugindo muito disso, promove tudo isso e mais um pouco.
Comment: Então para coseguir meu obejtivo preciso ter esses programas instalados na minha máquina?
Comment: instalei o ffmpeg de acordo com um tutorial e uso ele no Terminal CMD do windows
Comment: Tudo bem, compreendo! Veremos no que posso ser útil. Lhe darei um exemplo prático pra você realizar  um teste em seu CMD. Ok!? Aguarde ...
Comment: ok. vou aguardar.
Answer: Com base nas informações dada na pergunta. Temos a conclusão que:
Vídeo

Resolução: 1920x1080p
Frame rate :  23 FPS
Bit rate : 1146 kb/s
Format : AVC
Codec : mpeg4

Áudio

Bit rate : 126 kb/s
Format : aac
Codec: mp4a-40-2
Channel(s) : 2 channels
Sampling rate : 44.1 kHz
Frame rate : 43.066 FPS (1024 SPF)

.. pode-se elaborar o seguinte pseudo-código no terminhal CMD:
<code>ffmpeg -y -i "video_original.mpg" -vcodec mpeg4 -r 23.976 -b 1146k -s 1920x1080 -acodec mp4a-40-2 -ab 126k -ar 44100 -map 0.0 -map 1.0 "video_convertido.mp4"
</code>
Explicando cada parâmetro:
Vídeo
<code>-vcodec - Faz cópia do vídeo original
-b      - Transferência do bitrate do vídeo
-r      - fps framerate do vídeo[Frames por segundo]
-s      - resolução do vídeo[tela]
</code>
Áudio
<code>-acodec - Faz cópia do áudio original
-ab     - Transferência do bitrate do áudio
-ar     - Taxa de amostragem frequência em Mega Hertz
-map    - Definir os canais 1 ou 2 [L R] (mono ou estereo)
</code>

Nota - Os codec mais comuns são MPEG4 e H264 para vídeos geralmente no formato mp4.
Comment: @kaputtz O "MP4", que é o melhor formato de vídeo. O MP4 tem um padrão de container de áudio e vídeo que é parte da especificação MPEG-4 desenvolvido pela ISO/IEC 14496-14. É similar ao popular AVI, mas traz algumas vantagens: Suporte nativo a legenda embutida. Melhor suporte ao padrão de codificação de vídeo H.264, também conhecido como **MPEG4 AVC**. O arquivo MP4 pode ter sido criado com diversos tipos de codec diferentes, incluindo DivX e XviD. É comum você encontrar um MP4 que seja constituído por streams criados por codecs incomuns.
Comment: bom dia... Ficou diferente..  ainda os comandos estavam dando erro,, na parta -b 1146k ele estava dizendo que seria -b:a ou -b:v e tambem na parte  -acodec informando que o parâmetro estava errado, ai coloquei -acodec aac ai deu certo.. mais ficou assim..

Format                      : MPEG-4
Format profile              : Base Media
Codec ID                    : isom (isom/iso2/mp41)
File size                   : 247 MiB
Duration                    : 21 min 40 s
Overall bit rate mode       : Constant
Overall bit rate            : 1 593 kb/s
Writing application         : Lavf58.2.103
Comment: @Alex Para mais informações, só posso passar via [Chat](https://chat.stackexchange.com/rooms/72342/diego-henrique). Uma vez que, a pergunta pode ter que aplicar várias modos ajustes experimentais[é um trabalho árduo]. Que no seu requisito têm terá de ter uma abordagem ampla, mais profunda.
Comment: Olá Diego não consigo falar com vc via chat, não abre nenhuma opção para eu digitar ou falar com vc... Mais respodendo sua pergunta o vídeo original é em mp4 também, mais com configuração diferente.. E te faço uma pergunta, quais app para android que podem fazer isso?
Comment: Show de resposta, só queria entender essa parte: Format : AVC


NFS Share with root for anonuid / anongid

Question: I've exported a share from my server and set anonuid and anongid to be 0 (root).  However, when I mount the share on the client, it doesn't appear my settings are working as I'm getting permission denied to folders within the share owned by root.
Server CentOS 5.7 / Client CentOS 6.4 using NFS version 3.2.29.
Here is my /etc/exports on SERVER:
/STORAGE 10.0.5.10(rw,sync,no_subtree_check,no_root_squash,anonuid=0,anongid=0)

Basically, when the client (10.0.5.10) connects to the server, I need it to behave as if it were root on the server.  Thank you in advance for the help!!
Comment: Check this - http://bugs.centos.org/view.php?id=6345, it should be a bug.
Comment: Thank you!  I made a small mistake in my original post.  The SERVER is CentOS 5.7 and the CLIENT is 6.4.  Assuming the same bug still applies?  Thanks!
Answer: If you want any user on 10.0.5.10 to appear as root you want to do this:
/STORAGE 10.0.5.10(rw,sync,no_subtree_check,all_squash,anonuid=0,anongid=0)
all_squash tells NFS that for any user connecting from 10.0.5.10, ignore their actual UID/GID and instead treat them as if UID=anonuid and GID=anongid. Since you set anonuid=0,anongid=0 that gives all users on 10.0.5.10 root access privileges on /STORAGE, effectively bypassing all security on /STORAGE and leaving it wide open to abuse from anyone appearing to come from the 10.0.5.10 IP address.
FWIW, this is a terrible idea from a security point of view.
If you can use NFSv4 on the server, you can enable UID/GID mapping and add a static map to /etc/idmapd.conf on the server, telling it that a specific user on 10.0.5.10 should be given root access on the NFSv4 server. man idmapd.conf for details on setting up the config file. Once the config file is set up on the NFSv4 server, update your export:
/STORAGE 10.0.5.10(rw,sync,no_subtree_check,no_root_squash)
Then you just want to enable mapping, clear the idmap cache, and restart the map service:
echo N > /sys/module/nfs/parameters/nfs4_disable_idmapping
nfsidmap -c
service rpcidmapd restart

If you do that, you're only giving one user root access, not all users.
Comment: Although `anonuid=0` is a bad idea, I found this helpful for setting to a different value (default Ubuntu first user added, 1001).
Answer: First of all, you should upgrade to NFS4, as things behave slightly different on each version.
Instead of no_root_squash you will need to use root_squash or all_squash - this is the only relevant parameter regarding this question.
all_squash makes any client connected to that share to use the ID given in the anonuid/anongid parameters.
Comment: Thank you so much! `all_squash` along with `anonuid=33,anonguid=33` is exactly what I was looking for (executing CLI PHP scripts with my "webdev" user, but still writing the files as www-data), but I couldn't (easily, at all) find an answer to my interrogation. Thanks :)


Which version of the extension builder is working for TYPO3 9.5 LTS?

Question: It is recommend to use a recent version from Github https://github.com/FriendsOfTYPO3/extension_builder.
The latest version of the master branch does not work for the current LTS version of TYPO3 and I don't find any tags 9.x at all on Github.
In the extension manager the master branch identifies already as 9.10.0 so I have hope there is a working version for TYPO3 9.5 LTS. Which version is it?
If there is no working version for 9.t LTS, what is your workaround?
Answer: As of today a working approach for TYPO3 9.5 LST is this:
Latest commit of master:
ad22d356546794cecfe436ad2d3e3290933f3693
Manual fix of the save bug as given here by the user qbadev:
https://github.com/FriendsOfTYPO3/extension_builder/issues/178


Negative regex pattern matching in python for JSON

Question: I believe this may have been asked before many times, but i could not find a way make it working for json content. The result negative pattern is matching for all json strings (even if the substring exists). Im sure, i might be doing something wrong.
Idea is to match the json string which has no "key" string in it, and not match the one with "key" string in it.
Note: I do need to achieve this via "re.match" with negative regex (and not with matching it and negating in python), as im doing this in bulk with many expression, and cant really change the way  of the function for one expression alone.
For example, below is my two json strings
'{"key": "success", "name": "peter"}'
'{"name": "sam"}'

And Im using the below regex pattern to negative match
((?!key).).*

Result is
Python 3.9.5 (default, May 11 2021, 08:20:37)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> pattern = r"((?!key).).*"
>>> jsonstring = '{"key": "success", "name": "peter"}'
>>> re.match(pattern, jsonstring)
<re.Match object; span=(0, 35), match='{"key": "success", "name": "peter"}'>

>>> jsonstring = '{"name": "sam"}'
>>> re.match(pattern, jsonstring)
<re.Match object; span=(0, 15), match='{"name": "sam"}'>

Am I doing anything terribly wrong here? was trying different pattern, but without success so far.
Answer: ((?!key).).* matches a positive sequence of characters ..* (that is equivalent to .+) which does not start with "key" (more precisely, the beginning must not be followed by the word "key"). Indeed both the strings do not start with the word "key", so both of them match the pattern.
Notice that the brackets are useless here.
You may want to use (?!.*"key").*:
>>> import re
>>> pattern = r"(?!.*\"key\").*"
>>> jsonstring = '{"key": "success", "name": "peter"}'
>>>

>>> jsonstring = '{"name": "sam"}'
>>> re.match(pattern, jsonstring)
<re.Match object; span=(0, 15), match='{"name": "sam"}'>

which works in this case although it is not a good way of parsing a JSON string.
The best way is to use a JSON parser:
>>> import json
>>> jsonstring = '{"key": "success", "name": "peter"}'
>>> obj = json.loads(jsonstring)
>>> "key" not in obj
False
>>> jsonstring = '{"name": "sam"}'
>>> obj = json.loads(jsonstring)
>>> "key" not in obj
True

Comment: @Danny Ok. Please notice that this will exclude also the strings in which `"key"` appears as a value (e.g. `{"foo" : "key"}`) or as array item (e.g. `["foo", "key"]`). In order not to run into eventual false positives, you can use `r"(?!.*\"key\"\s*:).*"` instead.
Comment: Nice!! This looks good, and working perfect. Thanks!!   I agree, this is not a most appropriate way for json content. Though im using some patterns inside a function, which will do for all contents, including text, csv and json. So sticking to something common consistent. Plus, the dictionary might be nested, and huge, and i may not know where exactly the substring exists.
Comment: Yes, I modified that a a bit to fit a bit global, and the answer gave me, what i really needed to start with. Thanks for that. And, oh yeh, I forgot accepting this as answer (just did!!) Thanks a heap


What is the difference between the probabilistic and non-probabilistic learning methods?

Question: What is the difference between the probabilistic and non-probabilistic learning methods for different situations? When is one considered more appropriate than the other? Can you give me examples?
Answer: The task of classification enables a simple comparison:
A probabilistic approach (such as Random Forest) would yield a probability distribution over a set of classes for each input sample.
A deterministic approach (such as SVM) does not model the distribution of classes but rather separates the feature space and return the class associated with the space where a sample originates from.
Yet it is possible for every probabilistic method to simply return the class with the highest probability and therefore seem deterministic.
Further, the other way around, based on the distance to the separating hyperplane in SVMs a probability can be computed and returned for each class.
Generally, the probabilistic approaches are better suited to incorporate uncertainty regarding the answer and can provide information about how safe a prediction is. Further, probabilistic methods might be able to incorporate prior information about the class distribution.
Example:
If you would for example try to predict gender based on body height, a deterministic approach would simply chose a separating point and say that:
(completely made up numbers)
height >= 175 --> man
height < 175 --> woman
so for a sample with the height of 175 it would yield woman as well as it would for a height of 155.
A probabilistic approach in this case could yield a probability such as for a height of
155 ---> man(0.1), woman(0.9) and for
175 --> man(0.49), woman(0.51)
Comment: The example was very useful, thank you for that. But if a classifier is using fixed set of rules in determining the class of an input (as in the deterministic approach) would it still be considered as a machine learning algorithm? I am new to ML, so apologies if this is not a correct question to ask
Comment: The question is not whether the rules are deterministic or not but rather whether the rules were "learned" (~parameters adapted) from data.
Comment: @NikolasRieble Since all models can turn probabilities into predicted classes and scores into probabilities, what is the fundamental distinction? I would argue (I am not sure) that in the probabilistic inference we directly model a probability distribution whereas in non-probabilistic we just learn decision boundaries.


registrar novos usuários usando o Ionic e o parse

Question: Eu to trabalhando em um app hibrido usando o ionic e o parse.
Quando eu vou no terminal e uso o comando ionic serve, e depois tento criar um novo usuario ou fazer o login, eu consigo sem problema nenhum, ou seja, no localhost está funcionando.
O problema é que, quando eu faço o build pro android e testo no celular, eu não consigo fazer o login e nem registrar novos usuários.
Alguém sabe me dizer o por que disso?
Obrigado!
Answer: A melhor coisa a se fazer é subir o ionic em modo debug e você acessar pelo celular, vai ser possível você ver os logs no console.
ionic run --consolelogs --serverlogs

Nas chamadas pro backend coloca alguns "alert" no sucesso e no erro, vai facilitar bastante.
Eu já tive alguns problemas com isso, porém estava utilizando outro framework, e era questões de permissões configuradas no meu backend.
Comment: Obrigado! vou tentar isso.
Comment: Consegui resolver o problema, vlw pela ajuda. No meu caso, eu estava chamando um script hospedado na parse.com. Só precisei baixar o script e adiciona-lo com os outros arquivos js.


Using Selenium and Python pull the list of specific search results based on site name

Question: I am attempting to visit specific sites provided in the search results page.  I can pull back an entire list of sites on the page but 1) it includes sites which are not specific to the one I want to visit and 2) will error out.
driver.get('https://www.Google.com')

search = driver.find_element_by_name('q')
search.send_keys( sites[site_index] + " " + phones[phone_index] +   " review" )
search.send_keys(Keys.RETURN)
time.sleep(4)
my_search_list = driver.find_elements_by_xpath("//div[@class='rc']")
myRandomNumber = random.randint(0, len(my_search_list))
print("this is len list ", len(my_search_list))
print(myRandomNumber)
my_search_list[myRandomNumber].click()

Please notice in the above that search.send_keys is specifically entering this text "The Verge Google Pixel 3 review"
Here is the page I am trying to pull from.  I want all the websites with The verge.
https://www.google.com/search?source=hp&ei=zbwKXubiFua1tgW0sKHACQ&q=The+Verge+Google+Pixel+3+review&oq=The+Verge+Google+Pixel+3+review&gs_l=psy-ab.3...23.169..180...0.0..0.0.0.......0....1..gws-wiz.1kE53ioW9rU&ved=0ahUKEwim_ZK79d7mAhXmmq0KHTRYCJgQ4dUDCAg&uact=5
Answer: I added for loop to filter the my_search_list array and append the verge results to a new array. Maybe this will get you going in the direction you are trying to go.
driver.get('https://www.Google.com')
time.sleep(5)
search = driver.find_element_by_name('q')
search.send_keys("The Verge Google Pixel 3 review")
search.send_keys(Keys.RETURN)
time.sleep(4)
my_search_list = driver.find_elements_by_xpath("//div[@class='rc']")
    save_links = driver.find_elements_by_xpath("//div[@class='rc']//div//a[contains(@href, 'theverge.com')]")
the_verge_links = []

for link in my_search_list:
    if 'theverge.com' in link.text:
        the_verge_links.append(link)
        time.sleep(2)
        print(link.text)

print("____________________________________________________")
print("____________________________________________________")
myRandomNumber = random.randint(0, len(the_verge_links))
print("this is len list ", len(the_verge_links))
print(myRandomNumber)
# the_verge_links[myRandomNumber].click()

link_text = []
for verge_link in save_links:
    link_text.append(verge_link.get_attribute("href"))

for saved_link_text in link_text:
    driver.get(saved_link_text)

Comment: @Normandy to visit each site in the list you can either do the google search again so the link will be able to be clicked on again or you can save all the links you are interested in and use `driver.get()` to get to them.

I'll edited the answer to provide an example.


Serializar Objeto para JSON

Question: Como faço para converter um objeto em JSON?
Até achei alguns códigos na internet que até funcionou, porém meu objeto possui alguns campos que são outros objetos e então a conversão não chegou até o último nível.
Exemplo:
class Pessoa(object):
    codigo = 0
    nome = ''
    endereco = Endereco()

class Endereco(object):
    codigo = 0
    logradouro = ''
    bairro = ''
    cidade = Cidade()

class Cidade(object):
    codigo = 0
    nome = ''
    uf = ''

Esses são meus objetos, porém fazendo a conversão que encontrei, o resultado JSON vem apenas com os valores contidos em Pessoa em Endereco, ficando então Cidade de fora do JSON:
{
    'codigo': 1,
    'nome': 'Oliveira',
    'endereco': {
        'codigo': 5,
        'logradouro': 'Rua A',
        'bairro': 'Campos'
    }
}

Ou seja, acredito que o código não percorreu todos os níveis.
Estou usando o pyMongo e vi que possui uma biblioteca bson que possui algumas conversões, mas eu não soube usa-la ou então ela não faz o que eu preciso.
Comment: Eu estou meio confuso - o que são esses campos definidos na classe? Você sabe que quando define campos desse jeito, eles são estáticos, certo? Pra definir campos de objetos, você faz dentro do `__init__`. Talvez seja por isso que os métodos de conversão objeto/json que você experimentou não estejam dando o resultado esperado.
Answer: Em primeiro lugar, você precisa definir os seus campos dentro do __init__, e não assim na classe, pois do contrário eles serão compartilhados por todas as instâncias da classe (i.e. serão "estáticos") em vez de ter um para cada objeto:
class Pessoa(object):
    def __init__(self):
        self.codigo = 0
        self.nome = ''
        self.endereco = Endereco()

class Endereco(object):
    def __init__(self):
        self.codigo = 0
        self.logradouro = ''
        self.bairro = ''
        self.cidade = Cidade()

class Cidade(object):
    def __init__(self):
        self.codigo = 0
        self.nome = ''
        self.uf = ''

(pode colocar parâmetros - opcionais ou obrigatórios - no construtor se quiser, em vez de iniciar tudo com um valor padrão; só tome cuidado com os valores padrão que também são objetos, colocá-lo na assinatura do método trará efeitos indesejados)
Muitas vezes você não precisa de classes - usar literais para dict, ou criá-los à mão mesmo (como sugerido por Christian Felipe) pode ser suficiente. Entretanto, as classes podem ser úteis se você pretende lhes dar métodos, então não digo para não usar, use sim se for necessário/conveniente.
Feito isso, as bibliotecas de conversão objeto/json que você experimentou devem dar o resultado correto. Mas caso queira um método simples de fazer à mão, você pode se aproveitar do fato que o campo __dict__ de um objeto transforma suas propriedades em campos de um dict:
import json

def para_dict(obj):
    # Se for um objeto, transforma num dict
    if hasattr(obj, '__dict__'):
        obj = obj.__dict__

    # Se for um dict, lê chaves e valores; converte valores
    if isinstance(obj, dict):
        return { k:para_dict(v) for k,v in obj.items() }
    # Se for uma lista ou tupla, lê elementos; também converte
    elif isinstance(obj, list) or isinstance(obj, tuple):
        return [para_dict(e) for e in obj]
    # Se for qualquer outra coisa, usa sem conversão
    else:
        return obj

p = Pessoa()
s = json.dumps(para_dict(p))

Isso deve funcionar para os casos mais comuns, e o uso do json.dumps garante que ao menos nada inválido será ecoado para a saída.
Comment: Valeu pela ajuda cara, ajudou bastante!
Answer: Então cara, em Python existe o tipo de dado Dictionary que é tipo o Map de outras linguagens, é uma estrutura de (key, value), ou seja, quase parecido com o problema que você está precisando, porque o formato Json segue esse padrão de (key, value). Segue um exemplo abaixo de um uso dessa estrutura que já vem na linguagem e não precisa de outros pacotes e módulos, porém você deve adaptar ao seu uso aí, obviamente:
Pessoa = {}
Endereco = {}
Cidade = {}

Endereco['Logradouro'] = "Rua A"
Endereco['Numero'] = 80
Endereco['Bairro'] = "Centro"

Cidade['Nome'] = "Belo Horizonte"
Cidade['UF'] = "Minas Gerais"
Cidade['Codigo'] = 0

Pessoa['Nome'] = "Meu nome é esse"
Pessoa['Codigo'] = 1
Pessoa['Endereco'] = Endereco
Pessoa['Cidade'] = Cidade

print(repr(Pessoa))

Resultado
{
    "Nome": "Meu nome é esse",
    "Codigo": 1,
    "Endereco": {
        "Numero": 80,
        "Logradouro": "Rua A",
        "Bairro": "Centro"
     },
    "Cidade": {
        "Nome": "Belo Horizonte",
        "Codigo": 0,
        "UF": "Minas Gerais"
     }
}

Então você pode fazer uma classe que monta esses dicionários separadamente, utilizando os métodos de acesso aos seus objetos, e, com o método repr(obj) que já te retorna o json que você precisa, está resolvido seu problema.
Comment: Tem certeza que o resultado de `repr` é sempre json válido? Se você já tem uma estrutura só com `dict`s, `list`s e primitivos, pode usar `import json` e `s = json.dumps(obj)` pra produzir um json válido - e sem o risco de serializar coisas que não são válidas, como funções. Como o módulo `json` é nativo do Python, não precisa de bibliotecas externas.
Comment: Um contra-exemplo: `>>> repr({'teste com aspas':'"'})` saída: `'{\'teste com aspas\': \'"\'}'`. Usando `json.dumps`: `'{"teste com aspas": "\\""}'`
Comment: Mas nesses casos seus, você já está jogando uma ``str`` para o método ``repr`` e não um ``dict`` como sugerido para ele ir montando. A questão é que, pelo que vi por aí, a estrutura ``dict`` implementa um método default para todos objetos que é o ``obj.__repr__()``, que aí quando eu chamo o ``repr(obj)`` internamente é chamado esse método, que pega os pares ``(key, value)`` e reproduz eles do tipo ``json``. No seu caso apresentado são ``str``'s e aí a reprodução é diferente mesmo. Porém, a forma que apresentei é apenas uma das formas, utilizando o módulo ``json`` também é uma boa.
Comment: Eu passei um `dict` pro `repr`, não uma string. De todo modo, pra usar esse método a pessoa tem que tomar cuidado pro objeto não ter funções, tuplas, datas... Não estou dizendo que não dá pra usar, só acho o `json` mais seguro/robusto.
Comment: Bom, nisso eu concordo!
Comment: Valeu pela ajuda cara, ajudou bastante!

Serializar Objeto para JSON


Is there a way to show the Google Analytics panel inside a drupal site?

Question: I'm using the Google Analytics module to track the visits.
Is there a way to show the Google Analytics panel inside the Drupal site?
Answer: Google analytics reports Provide you a default report page. You can use the Query hooks to make your own query and show it as a block or panel.
They provide common queries like visits, Top pages viewed etc.
Comment: Just what I was looking for.


Returning custom array in lazy builder callback

Question: I have to return a custom array from a lazy builder callback and then loop through that array in a Twig template for a custom block. Here is my code so far:
Custom block:
public function build() {

return [
  '#theme' => 'custom_theme',
  '#variables' => [
    '#lazy_builder' => [static::class . '::lazyBuilder', []],
    '#create_placeholder' => TRUE,
  ],
];
}

The callback
  public static function lazyBuilder() {

  //code above
      foreach ($vars as $var) {
        $arr[] = [
          'title' => $var->getTitle(),
          'body' => $var->body->value),
          'summary' => $var->summary->value,
        ];
      }
    }

    sleep(3); // For testing only
    return $arr;
  }

In my twig template
{% for var in variables %}

{{ var.title }}

{% endfor %}

For each item I want to display its title, body and summary. Everything works fine if I return from the callback an array which has #markup inside, but I need to return my custom array.
If that's not possible it would be also OK if a lazy loaded the whole block not just the content inside it.
Thank you.
Answer:
I have to return a custom array from a lazy builder callback

You can return multiple #markup elements nested in the render array, also other render elements like #plain_text, which is more safe to output a title containing user input:
  'title' => ['#plain_text' => $var->getTitle()],

For the body field you need a more complex render element, see How to safely render node body on a custom variable?

and then loop through that array in a Twig template

You can't loop through the array in the theme because you have set-up the variable to be lazy-built. The array doesn't exist at this point, only the placeholder. The array needs to be a proper render array which can be rendered recursively as explained above.

If that's not possible it would be also OK if a lazy loaded the whole
block not just the content inside it.

Blocks are lazy-built by default and placeholdered when meeting the auto-placeholder conditions. See https://www.drupal.org/docs/8/api/render-api/auto-placeholdering