Testing significant differences between regressions in R

Question: I am running several phylogenetic least squares analyses in R, where I'm taking an existing data set for several species, and adding two new species for which I have data. I want to do is test whether the new all-species regression is significantly different than the old regression. But I am unsure of the appropriate test to use.
The existing data set establishes a regression that should be applicable to all species. I can add data on two species, and what I find is that one of the two species falls more or less on regression predicted by the original data set. The other species fall way below the regression predicted by the original data set. So I guess what I'm trying to determine is if there are significant differences in both slope and intercept, as well as significant differences in the strength of the regression.
Comment: Can you be more specific regarding how the new regression might differ from the old one? Eg, are you wondering if the slope of the relationship b/t X & Y differs by species, if the intercept does, or if both do? Something else?
Comment: Hi @Nate and welcome to the site. Am I correct that the dependent variable (outcome) is the same in the all-species and in the old regression? To test whether the model with all species is an improvement over the old regression you could use a likelihood ratio test. See [this post](http://stats.stackexchange.com/questions/59085/how-to-test-for-simultaneous-equality-of-choosen-coefficients-in-logit-or-probit/59093#59093) for a worked example in `R`.
Comment: COOLSerdash, thanks for the help, ill work through that example and code.

@gung Sorry, So what im trying to accomplish is the following. The existing data set establishes a regression that should be applicable to all species. I can add data on two species, and what I find is that one of the two species falls more or less on regression predicted by the original data set. The other species fall way below the regression predicted by the original data set. So I guess what im trying to determine is if there are sig. differences in both slope and intercept, when adding the new data.

Thanks!
Comment: There are two general approaches. One (the most common approach, I think) is to combine into a single regression with dummies for the different sets, which looks like the approach COOLSerdash is pointing to). The other is to construct tests based on separate regressions; there are advantages and disadvantages either way. The first requires somewhat more restrictive assumptions (like equality of variance), but is both more convenient and should - if the assumptions are true - have slightly better power.
Answer: The most basic way to do this is to run a new, larger regression with all of your data.  (That is, the data that had been used before, plus the data for the two new species.)  You should create a dummy variable to indicate each species (obviously with one species as the reference level--I've explained dummy coding here).  You will also want to add product terms for the interaction between whatever your $X$ variable is and the species indicators (I've explained what an interaction is here).
After having run this larger multiple regression model, test to see if the interaction terms are 'significant'.  (I have discussed this here.)  If you believe that the interactions are contributing something meaningful, then that means the slope of the relationship between $X$ and $Y$ differs by species.  You can also test to see if the species dummy variables are statistically significant.  This will tell you if the intercepts differ by species.  (A couple of notes here: If the interaction is significant, the significance of the dummies / intercepts probably isn't very meaningful, and should definitely be kept in the model either way--see here & here.  If the interaction is sufficiently non-significant that you don't believe it, but the dummies are, that means the lines run parallel to each other but are shifted vertically.  My answer here puts a lot of this information together in one place.)
Answer: This question is concerned with whether a new data point is an influential observation in a PGLS model. I think the easiest way to do this would be to examine diagnostic plots in a regression of phylogenetically independent contrasts. You can do this with help from the 'caper' package. I will demonstrate below.
First I'll simulate a phylogeny, a pair of correlated traits, and make one data point a big outlier.
library(phytools)

# Simulate example data with an influential data point
set.seed(23)
tree <- rcoal(12)
data <- sim.corrs(tree, vcv=matrix(c(1,0.7,0.7,1),2,2))
data <- data.frame(data, rownames=rownames(data))
names(data) <- c("x", "y", "species")
data[4,2] <- -1.8

# Visualize raw data
par(mfrow=c(1,2))
plot(tree)
nodelabels("18", 18)
plot(y~x, type="n",data=data, pch=16, xlab="x", ylab="y")
text(data$x, data$y, labels=data$species)

I highlight node number 18 on the phylogeny, because that is the node ancestral to tip $t2$, which is the influential observation as seen in the scatter plot.
Now, I use caper to do phylogenetically independent contrasts and look at the model diagnostic plots. Note that in these diagnostic plots, data points correspond to independent contrasts, not raw data points. Thus, for a tree with $n$ tips, there are $n - 1$ phylogenetically independent contrasts corresponding to each of the nodes in the tree.
library(caper)

# Compute independent contrasts regression
comp.data <- comparative.data(tree, data, species)
fit <- crunch(y ~ x, data=comp.data)

# Diagnostic plots
par(mfrow=c(2,2))
plot(fit)

As expected, all four of these plots point towards node number 18 as an influential observations.


table request in db returns date like this: 2023-07-20T21:39:00.000Z I need to format this data to dd/MM/yyyy HH:mm

Question: In the database the format is correct, but in the ts request it is arriving this way, how can I correct it?
cols: any[] = [
  {
    field: 'dataHoraInicio',
    header: 'Início',
    fieldback: 'dataHoraInicio'
  },
  {
    field: 'dataHoraFim',
    header: 'Final',
    fieldback: 'dataHoraFim'
  }
]

The only thing I found was formatting for fixed codes
Comment: Does this answer your question? [converting dates to a desired format](https://stackoverflow.com/questions/55851656/converting-dates-to-a-desired-format)
Answer: Create a TypeScript (TS) function that returns the formatted date as a string:
function formatDate(inputDate) {
  // Convert the input string to a Date object
  const date = new Date(inputDate);

  // Get the day with leading zeros
  const day = String(date.getDate()).padStart(2, '0');

  // Get the month with leading zeros
  const month = String(date.getMonth() + 1).padStart(2, '0');

  // Get the year
  const year = date.getFullYear();

  // Get the hours with leading zeros
  const hours = String(date.getHours()).padStart(2, '0');

  // Get the minutes with leading zeros
  const minutes = String(date.getMinutes()).padStart(2, '0');

  // Format the date as dd/MM/yyyy HH:mm
  const formattedDate = `${day}/${month}/${year} ${hours}:${minutes}`;

  return formattedDate;
}

Now use the above function to get the formatted date like this:
const inputDate = '2023-09-18T12:52:45.000Z';
const formattedDate = formatDate(inputDate);
console.log(formattedDate); // Output: "18/09/2023 18:22"

function formatDate(inputDate) {
  // Convert the input string to a Date object
  const date = new Date(inputDate);

  // Get the day with leading zeros
  const day = String(date.getDate()).padStart(2, '0');

  // Get the month with leading zeros (Note: Months are zero-based)
  const month = String(date.getMonth() + 1).padStart(2, '0');

  // Get the year
  const year = date.getFullYear();

  // Get the hours with leading zeros
  const hours = String(date.getHours()).padStart(2, '0');

  // Get the minutes with leading zeros
  const minutes = String(date.getMinutes()).padStart(2, '0');

  // Format the date as dd/MM/yyyy HH:mm
  const formattedDate = `${day}/${month}/${year} ${hours}:${minutes}`;

  return formattedDate;
}

const inputDate = '2023-09-18T12:52:45.000Z'; // Replace this with desired date
const formattedDate = formatDate(inputDate);
console.log(formattedDate);


A basic doubt on sigma algebra generated by a random variable

Question: Why do we need the concept of sigma algebra generated by a random variable ?
Answer: Let $(\Omega,\mathcal F,\mu)$ be a probability space.There is a natural definition of independence of two $\sigma$-sub-algebras $\mathcal A$ and $\mathcal B$ of $\mathcal F$, namely
$$\forall A\in\mathcal A,B\in\mathcal B, \quad \mu(A\cap B)=\mu(A)\mu(B).$$
Now we wood like to formalize the fact that "a realization of the random variable $Y$ does not affect one of $X$", and this is done defining "$X$ is independent of $Y$" if and only if the $\sigma$-algebra generated by $X$ is independent of those of $Y$.


How is the fourier series of $\frac{\pi-x}2$ derived?

Question: $$S = \sum_{n=1}^{\infty} \frac{\sin(n)}{n} $$
I seem to have found on the web:
$$\frac{\pi-x}{2}=\sum_{n\geq1}\frac{\sin\left(nx\right)}{n} \space, x \in(0, 2\pi)$$
Then:
$$x = \pi - 2\sum_{n=1}^{\infty} \frac{\sin(nx)}{n}$$
How is this fourier series derived? Only hints please, no complete answers.
Answer: $\newcommand{\angles}[1]{\left\langle\, #1 \,\right\rangle}
 \newcommand{\braces}[1]{\left\lbrace\, #1 \,\right\rbrace}
 \newcommand{\bracks}[1]{\left\lbrack\, #1 \,\right\rbrack}
 \newcommand{\dd}{{\rm d}}
 \newcommand{\ds}[1]{\displaystyle{#1}}
 \newcommand{\dsc}[1]{\displaystyle{\color{red}{#1}}}
 \newcommand{\expo}[1]{\,{\rm e}^{#1}\,}
 \newcommand{\half}{{1 \over 2}}
 \newcommand{\ic}{{\rm i}}
 \newcommand{\imp}{\Longrightarrow}
 \newcommand{\Li}[1]{\,{\rm Li}_{#1}}
 \newcommand{\pars}[1]{\left(\, #1 \,\right)}
 \newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
 \newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,}
 \newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}}
 \newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}$
Besides the Brian Fitzpatrick nice answer, you can use the Abel-Plana Formula:
\begin{align}\color{#66f}{\large\sum_{n=1}^{\infty}\frac{\sin\pars{nx}}{n}}
&=-x + \bracks{\int_{0}^{\infty}\frac{\sin\pars{tx}}{t}\,\dd t
+\half\,\lim_{t \to 0}\frac{\sin\pars{tx}}{t}}
=-x + {\rm sgn}\pars{x}\,\frac{\pi}{2} + \half\,x
\\[5mm]&=\color{#66f}{\large\,{\rm sgn}\pars{x}\,\frac{\pi}{2} - \half\,x}
\end{align}
In the present case $\ds{\pars{~x \in \pars{0,2\pi}~}}$:
\begin{align}\color{#66f}{\large\sum_{n=1}^{\infty}\frac{\sin\pars{nx}}{n}}
&=\color{#66f}{\large\frac{\pi - x}{2}}\,,\qquad\qquad x \in \pars{0,2\pi}
\end{align}
Comment: From the Wikipedia page. Where did the:

$$i\int_{0}^{\infty} \frac{f(it) - f(-it)}{e^{2\pi t} - 1} dt$$

Go? Did you consider:

$$f(it) = \frac{\sin(it)}{it}$$

$$f(-it) = \frac{\sin(it)}{it}$$ Because sine is odd.

$$f(t) - f(-it) = 0$$?
Answer: This was worked out by computing to fourier series of the periodic continuation of $\frac{\pi-x}2$ from $(0,2\pi)$ to $\mathbb T$. Note that this function is discontinuous at $0 \equiv 2\pi$.
$$a_k = \frac1{2\pi} \int_0^{2\pi} \frac{\pi-x}2 e^{-ikx} \ \mathrm dx$$
Then by the Fourier-inversion thm.
$$\frac{\pi-x}2 = \sum_{k\in\mathbb Z} a_k e^{ikx}$$
At all points of continuity of $x$, i.e. for all $x\in (0,2\pi)$. Finally note that $\sin(kx) = \frac1{2i} (e^{ikx} - e^{-ikx})$ to bring this into the aforementioned form. What is left for you is to see that
$$a_k = \frac{1}{2ik} \forall k > 0,\quad a_0 = 0$$
Answer: Let $f$ be a $2\,L$ periodic function. The Fourier series of $f$ is defined as
$$
F(t)=a_0+\sum_{k=1}^\infty\left\{
a_k\cos\left(\frac{k\,\pi}{L}t
\right)
+
b_k\sin\left(\frac{k\,\pi}{L}t\right)
\right\}
$$
where the constants $a_0$, $a_k$, and $b_k$ are defined by the equations
\begin{align*}
a_0&=\frac{1}{2\,L}\int_{-L}^Lf(t)\,dt
&
a_k&=\frac{1}{L}\int_{-L}^Lf(t)\cos\left(\frac{k\,\pi}{L}t\right)\,dt
&
b_k&=\frac{1}{L}\int_{-L}^Lf(t)\sin\left(\frac{k\,\pi}{L}t\right)\,dt
\end{align*}
In our case we have the function
$$
f(t)=\frac{\pi-t}{2}
$$
defined on the interval $[0,2\,\pi]$. Here $L=\pi$.
Can you compute the coefficients $a_0$, $a_k$, and $b_k$?


Can't understand the solution of this INMO problem

Question: In any set of $181$ square integers, prove that one can always find a subset of $19$ numbers, sum of whose elements is divisible by $19$.
Someone on AOPS:

Direct checking shows that any square is $0,1,4,9,16,6,17,11,7,5 \mod
[19].$ Thus there are exactly $10$ distinct possibilities. From the
pigeon-hole principle, since you used $181$ numbers at least one of
the class contains at least $19$ of your squares. Adding these $19$
squares from the same class leads to the result.

Can anyone explain me this solution, or some other?
Thanks!
PS: This question is from Indian National Mathematical Olympiad 1994, Problem 3
Answer: If there were at most $18$ of each of those $10$ square congruences being used in the set, then the maximum number of elements in the set would be $18 \times 10 = 180$. However, since there are $181$ elements, this means that, by the pigeon-hole principle, there must be at least one congruence, say $r$, used at least $19$ times. As the solution states, adding these $19$ elements will give a sum with a remainder of $19r$, i.e., a value divisible by $19$.
Note that, in general, you actually only need just a set of $2 \times 19 - 1 = 37$ integers, without any particular restrictions on their congruences, for there to be a subset of $19$ of them that sum to a multiple of $19$. For more information about these types of problems, please see Proving that among any $2n - 1$ integers, there's always a subset of $n$ which sum to a multiple of $n$.
Comment: Oh thank you very much. Silly of me to not get the simple point.
Comment: @AkshatVats You're welcome. I've quite often found I've had trouble initially understanding some fairly simple concept as well.


Which correlation formula should be used when we add up many measurements of the ordinal type?

Question: Let's say, I have two set of questions; one set is to detect depression (var1) where the response of each question is in Likert scale and another set of questions to detect hopelessness (var2) where responses are also in Likert scale. Assume that, I have 50 samples who will be tested for depression and hopelessness. Now. I have to find the correlation between the variables, var1 and var2.
To do that, I want to calculate the depression or hopelessness score for each sample by summing the questions score (i.e. Likert Scale value for each question) of each questionnaire (depression or hopelessness). Then, find the correlation using formula.
What does create problem?
As each question of our questionnaires is in Likert scale, they are ordinal (Likert scale is ordinal).
BISHOP and Herron remarks

If  the  data  are  ordinal,  then  non-parametric statistics are
  typically considered  the  most  appropriate  option  for analysis.

This one also says

For ordinal scales, the correlation coefficient which is usually
  calculated is Spearman's rho.

However, this one in an answer of SE question, says

But when you add up many measurements of the ordinal type, which is
  what you have in your case, you end up with a measurement which is
  really neither ordinal nor interval, and is difficult to interpret.

On the other hand, Muayyad Ahmad remarks as an answer of a question in RG

For Likert scale, the items are ordinal, but usually we do summing for
  the items to get total score, which is considered as continuous
  variable. then we do Pearson correlation

David L Morgan also remarks there

Twenty-five items are a lot for a single scale, which is what you
  would get if you simply added all the items together. That is standard
  practice for creating a continuous variable from a series of ordinal
  Likert items, but it requires careful attention to whether thee items
  do indeed form a single scale.

Therefore, as we are adding many measurements of the ordinal type questions score in case each of the questionnaire (depression and hopelessness scale), will the value (summation of all questions score for each person and questionnaire) still remain in ordinal type? If it becomes like an interval variable and continuous, can you please refer a book or paper which says this one?
Comment: This has been answered at [How to choose between Pearson and Spearman correlation](https://stats.stackexchange.com/questions/8071/how-to-choose-between-pearson-and-spearman-correlation). I'm not sure what else there is to say.
Comment: @gung-ReinstateMonica I am extremely sorry to say you that I can not agree with you. They did not answer what I wanted to know. If you may check my question, you will find that I have already cited that question.
Answer: Have you tried plotting your data?  Both histograms of the 2 summed scores and a scatter-plot of them against each other.
When you start summing up responses (even if they are ordinal to begin with) the Central Limit Theorem comes into play and if you sum enough things together that are not too close to boundaries, then you will start to see something that is approximately normal.  But if many of your answers are at one or the other of the possible extremes or the individual responses are highly correlated, then you may not have enough for the CLT to help yet.  You need to do some data exploration to get a feel for this.
In your case there is not one perfect/correct answer, but there could be many approaches that are good enough, but which are good enough will depend on things that you will see in  the plots (and combined with your knowledge of the science behind the data).
If the histograms look fairly bell-shaped and the scatter-plot looks pretty linear then you will not see much difference between Pearson, Spearman, and other correlations and it is probably fine to use any of them.  But if there are clear clusters on the scatter-plot and/or U-shapes in the histograms then any single measure of correlation will probably over-simplify the relationship and distract more than help understanding.
I am a big fan of simulation for helping me understanding what is happening and what may happen.  You could randomly generate some data under different conditions (uniform across the possible answers, mostly low, mostly high, mostly low or high unlikely in the middle, etc.) and compute the different correlations and plots and compare them under the different situations to see which give you the most meaning compared to the simulation conditions.


Can I use residual as control variable?

Question: Let's say I want to see the effect of X1 on Y. I know X2 also has an effect on Y and I also know X1 has an effect on X2. Therefore, X1 can have an effect on Y directly and through X2. X1 and X2 are both continuous variables.
Normally, I would do Y = X1 + X2 + e. However, due to the high correlation between X1 and X2, I find an insignificant coefficient for X1. Is it proper for me to orthogonalize X1 and X2 by doing X2 = X1 + e and then using the residual of this regression in place of X2, i.e., Y = X1 + residual + e?
Answer: Your two models are equivalent.
First model is:
$Y=\beta_0 + \beta_1 X_1 + \beta_2X_2 + e$
In the second model, residual is $X_2-a - bX_1$.
Then the second model is:
$Y=\alpha_0 + \alpha_1X_1 + \alpha_2(X_2-a - bX_1) + e$
After simplifying the second model, you will find the relationship between $\alpha$s and $\beta$s.

Can I use residual as control variable?


Is there a supervised learning method for classifying graph walks?

Question: I have geolocational data(coordinates and times with device id), I can bucket this down using say 5m by 5m squares to represent a vertice on a graph. Then following the device id and creating edges chronologically creates a walk(or a graph). I also have a label for each walk, either a True or a False on whether they visited a particular location or not.
I want to use a supervised machine learning method to use this data to predict whether future walks would be a True or a False. One idea that I'm currently exploring is to generate a graph kernel with unit edge weight and try logistic regression or support vector machines on that.
Does anyone else have any ideas on other machine learning methods that I could explore? Is there a way of leveraging the time-series data such that I don't have to even create the graph? I should also mention that I have data on walks that I don't know if they correspond to a True or a False as well, so a semi-supervised method would be welcome as well. Thanks
Comment: Do you have an actual graph (edges and nodes) for the paths that these points lay upon? Is the actual point to have the machine learning algorithm infer the geographic location of the particular location that is potentially visited, so that any walk through a proximity around that point triggers a prediction of True?
Comment: @JohnYetter No, all I have are GPS co-ordinates, I plan on creating a graph as a way of mitigating the fact that I don't know how to handle the time-series data. I was thinking of creating a grid of 5m by 5m squares and a measurement in one box followed by another measurement in another box would be an edge connecting two vertices. I'm not sure how two measurements in the same 'box' would be represented.As for your second question, yes that is roughly true but my goal is to predict based on movement behavior of visitors(as opposed to someone who goes past in a straight line).
Comment: I don't think that you need to deal with the data as time series, unless the time of day (or within the week) that they approached the spot is important. The main thing is that they approached that geographic location and stopped there. Does it matter where they approached from? That is, does order matter, or just proximity?
Comment: Order does matter. Proximity is the definitely the easiest way of inferring a "stop", I want to see if I can use the order/or the sequential nature of the measurements as well.
Answer: //// Can't comment because of reputation so adding an answer ///
Please refer to below paper -
http://www.lifewear.gatech.edu/resources/Ashbrook,Starner-_Using_GPS_to_Learn_Significant_Locations_and_Predict_Movement_Across_Multiple_Users.pdf
The main idea of the methodology is to compute probability of each possible path & then model it
Alternately you can use Markov chains if the deviceIds can only move between set destinations (like bus stops or tourist points) -
http://setosa.io/ev/markov-chains/
Comment: I think if you expanded each of these options into a slightly longer description of what's going on "under the hood" you would have a very strong answer that could dispense with the "not enough reputation to comment" disclaimer.
Comment: Hey I accepted because graphical models definitely seem to be the direction to explore. I'm currently looking at Conditional Random Fields as well and trying to think of how to work that with my data.
Comment: @GeneralAbrial - I'll keep that in mind for next answers :). I didn't want to sound naive since I don't have a proper grip on concepts of conditional random fields. Jimbo - If the method helps/doesn't please put a short description here. Would love to know the application of this to your problem !


ColorHighlighter is not taking any effect on SBT3

Question: I go to :

and install :

with these settings :

and have no idea why hex is not my SBT3 still not display as its color form.

I've also tried : CSS

Reboot the IDE & even reboot the Mac OS X ... still not working.
Did I forgot to turn on some settings?
Can someone please help me ?

Edit
ColorHelper works on the first try, right after install. Huge thanks to : @MattDMo

... but I am still curious if someone really knows what I did wrong.
Comment: I would recommend trying out [`ColorHelper`](https://packagecontrol.io/packages/ColorHelper), which at least for me seems to Just Work™. Completely uninstall any other similar packages first, then try ColorHelper. There are some great docs [here](https://facelessuser.github.io/ColorHelper/).
Comment: It works on first try, thank-you!
Comment: Glad to hear it! I'll make it into a full answer if you want, so it doesn't get lost.
Comment: Ok. Please do. I'll make sure to accept it
Answer: Over the past few years I began having more and more problems with the different color highlighting packages available for Sublime, including Color Highlighter, which hasn't been updated in at least 3 years, and Color Highlight at 2½ years. Someone recommended ColorHelper, and I've been happy with it ever since. Make sure you completely uninstall Color Highlighter and/or Color Highlight first, as I think I had issues with it before I did that. ColorHelper is in regular development with the release of the new v3, which apparently is a complete rewrite, and is easy to set up and use out of the box. There are a ton of customization options if you want to get into the weeds, and the author has been very responsive with some config issues I had recently. I would highly recommend it.


variable value is not changing when i am changing the value with a function in js

Question:     function show() {
        let date_ = new Date();

        let sec = String(date_.getSeconds());
        fill_gaps(sec);

        function fill_gaps(candidate) {
            if (candidate.length < 2) {
            candidate = "0" + candidate;
          } else {
          candidate = candidate;
         }
       }

        time_ = String(`${sec}`);
        console.log(time_)
      }

The fill_gaps() function is supposed to change the var sec value; but it isn't changing the value outside the function. My guess is that it is not changing on global scope.
I know this is a basic problem but I can't figure it out as I am still a beginer.
Expected:
console.log(time_) = 0("and the current second")

Actual:
console.log(tiem_) = ("the current second")

Comment: strings are passed as a value inside the function in JS not as a reference.
Comment: Does this answer your question? [Is JavaScript a pass-by-reference or pass-by-value language?](https://stackoverflow.com/questions/518000/is-javascript-a-pass-by-reference-or-pass-by-value-language)
Comment: are you declaring a function inside another function?
Comment: Chris G - am i doing something wrong?
Answer: Edit: You were missing a return statement in your fill_gaps function, you were calling it but you weren't doing anything with it

function show() {
  let date_ = new Date();

  let sec = String(date_.getSeconds());
  sec = fill_gaps(sec); //call your function like this, save its value inside the sec variable

  time_ = String(`${sec}`);
  console.log(time_)
}

//this outside, it is a separate function
function fill_gaps(candidate) {
  if (candidate.length < 2) {
    candidate = "0" + candidate;
  } else {
    candidate = candidate;
  }
  return candidate;//you were missing a return statement
}

show();
Comment: you can mark the answer as accepted instead :)
Comment: yeah this works. but i need to use that function multiple times. p.s- i haven't shared the full code..
Comment: thanks for the help man... i was stuck on this simple prob for hours... thanks a lot
Comment: too bad i can't upvote your answer... i am new to stack and need some reputation first


Não estou conseguindo criar array com chaves em Java

Question: Estou estudando a linguagem Java e para praticar estou criando um programa simples de login. Possuo uma classe chamada Database que deveria guardar um array com as informações (nome e senha) dos usuários.
O problema é que ao tentar compilar o código, o compilador aponta um erro onde está sendo inicializado o array. Veja abaixo o código e o erro:
public class Database{

    private static String[][] data = new String[5][2];

    {
        Database.data[0] = {"Jean", "pato_34"};
        Database.data[1] = {"Maria", "lolzin"};
        Database.data[2] = {"Lucas", "bot3000"};
        Database.data[3] = {"Pedro", "showd3b0la"};
    }

    public static String[] find(String username){

        for (String[] user : Database.data){

            if (username == user[0]){
                return user;
            }
        }

        return {"", ""};
    }
}

.\server\Database.java:8: error: illegal start of expression
        Database.data[0] = {"Jean", "pato_34"};
                           ^
.\server\Database.java:8: error: not a statement
        Database.data[0] = {"Jean", "pato_34"};
                            ^
.\server\Database.java:8: error: ';' expected
        Database.data[0] = {"Jean", "pato_34"};
                                  ^
.\server\Database.java:9: error: ']' expected
        Database.data[1] = {"Maria", "lolzin"};
                      ^
.\server\Database.java:9: error: ';' expected
        Database.data[1] = {"Maria", "lolzin"};
                       ^
.\server\Database.java:9: error: illegal start of type
        Database.data[1] = {"Maria", "lolzin"};
                         ^
.\server\Database.java:9: error: <identifier> expected
        Database.data[1] = {"Maria", "lolzin"};
                          ^
.\server\Database.java:9: error: ';' expected
        Database.data[1] = {"Maria", "lolzin"};
                            ^
.\server\Database.java:9: error: illegal start of type
        Database.data[1] = {"Maria", "lolzin"};
                                   ^
.\server\Database.java:9: error: <identifier> expected
        Database.data[1] = {"Maria", "lolzin"};
                                    ^
.\server\Database.java:9: error: ';' expected
        Database.data[1] = {"Maria", "lolzin"};
                                             ^
.\server\Database.java:10: error: ']' expected
        Database.data[2] = {"Lucas", "bot3000"};
                      ^
.\server\Database.java:10: error: ';' expected
        Database.data[2] = {"Lucas", "bot3000"};
                       ^
.\server\Database.java:10: error: illegal start of type
        Database.data[2] = {"Lucas", "bot3000"};
                         ^
.\server\Database.java:10: error: <identifier> expected
        Database.data[2] = {"Lucas", "bot3000"};
                          ^
.\server\Database.java:10: error: ';' expected
        Database.data[2] = {"Lucas", "bot3000"};
                            ^
.\server\Database.java:10: error: illegal start of type
        Database.data[2] = {"Lucas", "bot3000"};
                                   ^
.\server\Database.java:10: error: <identifier> expected
        Database.data[2] = {"Lucas", "bot3000"};
                                    ^
.\server\Database.java:10: error: ';' expected
        Database.data[2] = {"Lucas", "bot3000"};
                                              ^
.\server\Database.java:11: error: ']' expected
        Database.data[3] = {"Pedro", "showd3b0la"};
                      ^
.\server\Database.java:11: error: ';' expected
        Database.data[3] = {"Pedro", "showd3b0la"};
                       ^
.\server\Database.java:11: error: illegal start of type
        Database.data[3] = {"Pedro", "showd3b0la"};
                         ^
.\server\Database.java:11: error: <identifier> expected
        Database.data[3] = {"Pedro", "showd3b0la"};
                          ^
.\server\Database.java:11: error: ';' expected
        Database.data[3] = {"Pedro", "showd3b0la"};
                            ^
.\server\Database.java:11: error: illegal start of type
        Database.data[3] = {"Pedro", "showd3b0la"};
                                   ^
.\server\Database.java:11: error: <identifier> expected
        Database.data[3] = {"Pedro", "showd3b0la"};
                                    ^
.\server\Database.java:11: error: ';' expected
        Database.data[3] = {"Pedro", "showd3b0la"};
                                                 ^
.\server\Database.java:14: error: class, interface, or enum expected
    public static String[] find(String username){
                  ^
.\server\Database.java:20: error: class, interface, or enum expected
            }
            ^
.\server\Database.java:24: error: class, interface, or enum expected
    }
    ^
.\server\Database.java:9: error: cannot find symbol
        Database.data[1] = {"Maria", "lolzin"};
                ^
  symbol:   class data
  location: class Database
.\server\Database.java:10: error: cannot find symbol
        Database.data[2] = {"Lucas", "bot3000"};
                ^
  symbol:   class data
  location: class Database
.\server\Database.java:11: error: cannot find symbol
        Database.data[3] = {"Pedro", "showd3b0la"};
                ^
  symbol:   class data
  location: class Database
Main.java:6: error: Client(String,String) has private access in Client
        Client client = new Client("Jean", "pato_34");
                        ^
.\server\Session.java:15: error: cannot find symbol
        String[] user = Database.find(username);
                                ^
  symbol:   method find(String)
  location: class Database
35 errors

O que estou fazendo de errado? Sei que é algo relacionado a sintaxe mas não consigo entender onde está o problema.
Answer: A criação de arrays em Java é meio chatinha mesmo.
Essa maneira que você tentou só funciona se for na própria declaração da variável. Ex:
String[] nomes = {"fulano", "ciclano"};

Mas se eu fizer em outra linha, dá erro:
String[] nomes;
nomes = {"fulano", "ciclano"}; // erro!

Nesse caso só funciona se usar new:
String[] nomes;
nomes = new String[] {"fulano", "ciclano"};

Portanto, no seu caso deve ser:
Database.data[0] = new String[] {"Jean", "pato_34"};

Ou então crie todos os valores de uma vez:
private static String[][] data = {
    {"Jean", "pato_34"},
    {"Maria", "lolzin"},
    // etc...
 };

Outro detalhe não relacionado é que em Java deve-se comparar Strings com equals  em vez de ==:
if (username.equals(user[0]))

Comment: Obrigado hkotsubo <3


Hosting an image-hosting website on Amazon Web Services

Question: I am making an image hosting service.  This is my first time building a large site, so I don't have much knowledge about creating a reliable web service, but I THINK I have this figured out.
How should I set up my site hosting to accommodate large amounts of traffic?  This is what I was thinking.  Tell me if this is a good idea:

Pick any cheap provider that has php and mySQL.
Store only the back end stuff on that cheap provider (php scripts, server config files, sql database).
Use Amazon S3 to store all the front end things, like css, js, images, and of course, all the stored images that the users upload (this is an image hosting website).

Does that work?  That means that the cost from large amounts of traffic is all done through Amazon S3, right?  The cheap provider shouldn't get hit with any significant costs because all it's doing is running scripts and updating the database?  Or will that also add up (and run slowly)?
Should I move the database to Amazon SimpleDB?  I also hear I can run the site using Amazon EC2, but it looks like that takes a lot of work to set up (and is expensive).  I guess what I'm asking can be summarized as: what's the most cost-effective way to reliably run an image-hosting website?
Thanks.
Comment: If you're serious about doing this; go work for someone who does this stuff already, get the prerequisite knowledge, then if you're still interesting in doing this you'll know how.
Comment: @Chris - It's just a small project, I just hope I'm not getting in over my head.  I already have almost all of the site built; I think I'll see how it goes and enlist some help if it gets crazy
Answer: The most cost-effective way to run a large-scale image (or any other) hosting website is not to run one -- Bandwidth costs alone will be astronomical if you achieve any level of popularity.
That all being said, I would test the waters on some commodity virtual private server host first (vps.net, linode.com, etc. -- Not endorsements, google around).  If you're growing fast enough & it looks like you might be able to at least break even on costs you can expand to a scenario like you described.

From an architecture standpoint I would suggest that if you're going "into the cloud" you go all the way -- Shoving data back and forth to your back-end systems will be slower if it's not "near" the front-end, and I believe cloud providers are pretty generous with bandwidth that doesn't leave their networks.
Also take a lesson from the recent Amazon EBS-related outages and ensure that you have an appropriate level of redundancy in your cloud-based services.  Customers tend to get whiny when you lose all their family photos :-)
Answer: So, here are a few points that may help you out.

Provider:  For most of the front end side of things, if cost is your main factor then it is up to you to find out what provider suits your needs.  Reliability, Cost, and Scaling are all factors that you will need to consider.
Note, unless you have the user download some kind of client side program (Flash, JS etc) your servers will have to receive the file and then upload it to S3 for them.  This will induce a lot of load as well as bandwidth costs.  However, it will also give you much better control over 'what' can be uploaded and how.  Once you hand control over to the client you will not be able to truly control what gets uploaded.
S3 is great for storing static content and it will be key in creating a site like this and keeping costs in line.  Make sure you properly control who has upload permissions to which buckets.  For example, if you have css and javascript in one bucket, only you should be able to upload to that location, otherwise a malicious user could upload some nasty files to replace your content.  On the other side if you are going to allow the user to upload content directly to save on bandwidth, you will have to make sure that is a separate bucket, ideally per user.  This is not trivial to enforce, and nearly impossible if you provide the client direct upload access.

Depending on your upload configuration (Client Side Client vs Server Side Client) your needs will be different.  Client Side will be cheaper up front for server costs, but be aware that someone will probably find a way to store any kind of file and you will be responsible for moderating that content.  For the Server Side model, be prepared to have your server costs increase with user traffic as you will need to build out more servers to handle upload requests.
Once you have the content hosted you will also want to look into a CDN (Content Delivery Network) such as Amazon's CloudFront (if you want to stay on the Amazon stack) or Akamai Networks.  These will increase your costs at first, but save you money on high usage content.
Amazon SimpleDB is an interesting Database style.  It is 'eventually consistent' which means that data sent to the database may not be immediately accessible, similar to Amazon S3.  If you are going use the database as a way to keep data synced across multiple nodes for many realtime transactions, I would not recommend it.
Comment: Hehe, just saw the other posts/comments.  I have to agree.  It is going to be very expensive once you get big and it is a very complex problem that will require talent and experience.


How can I record as MIDI track a live performance in MainStage 3 (MIDI piano)

Question: I usually use Garageband as piano sequencer but Mainstage 3 has a much better piano so I want to try recording and reproducing my piano MIDI tracks using Mainstage 3.
What is the recommended approach to record my piano performance in Mainstage but not as an audio track?
As an alternative, I have read that I can use Mainstage sounds in Garageband but at least in last versions (Garageband 10.3.2 and Mainstage 3) I can't seem to find Mainstage piano sounds (Steinway and Yamaha grand pianos) in Garageband. Any directions on this?
Answer: You can't record MIDI in Mainstage. I'm not sure that you can even record audio, but I know you can play it back.
I haven't found an easy way to use Mainstage sounds in GarageBand. I believe the intention there by Apple is that you would upgrade to Logic Pro, which will include all the Mainstage sounds and let you record, etc.
Comment: Thanks for your answer. I will try to find a way to run Garageband and Mainstage together so I can at least record the MIDI track in Garageband.


Como identificar o erro no código PHP pelo Chrome?

Question: Eu escrevo meu código no Sublime e quando tento executar a tela do chrome fica em branco. Como uma virgula pode causar um monte de problemas, fico relendo e relendo código à procura de erros.
Porque o meu Chrome não indica os erros?
Comment: Não seria porque php é lido e interpretado pelo **servidor php** e não pelo navegador? Diferente de javascript e de html, PHP não vai pro navegador.
Comment: Você pode instalar esse plugin no Sublime, http://www.sublimelinter.com/en/latest/, ele identifica erros como falta de `;` ou fechamento de chaves, pra você não precisar procurar por isso no código.
Comment: Entendi. Apesar disso, estou acompanhando as aulas do Curso em Video, e no caso dele, o navegador sempre informa os erros, igual ao javascript.
Answer: Coloque no início do seu código:
  ini_set('display_errors',1);
  ini_set('display_startup_erros',1);
  error_reporting(E_ALL);

e caso tenha o error_reporting(0), comente:
//error_reporting(0);

Comment: Obrigado pela dica. Incluí os métodos ao meu código e agora, de fato, o navegador indica alguma coisa ao invés de uma página em branco. Porém preciso de ajuda para entender qual o problema. Vou colar aqui os erros indicados:

Fatal error: Uncaught Error: Call to private carro::__construct() from invalid context in /Applications/MAMP/htdocs/POO - PHP/Teste/index.php:15 Stack trace: #0 {main} thrown in /Applications/MAMP/htdocs/POO - PHP/Teste/index.php on line 15
Comment: me indica uma fonte para eu entender melhor esses métodos.
Comment: Dá uma olhada nesses vídeos: https://www.youtube.com/channel/UCPs8o9v3gVkIYfO1jlbb5jA
Comment: Sobre o erro: https://stackoverflow.com/questions/1997721/fatal-error-call-to-private-myobject-construct-from-invalid-context
Comment: Em continuidade dos meus estudos, encontrei uma forma mais simples de identificar os erros do código PHP. Localize o arquivo php.ini e busque a linha 'display_errors = off'e altere para 'display_errors = on'. Pronto! Com isso os erros aparecerão naturalmente assim que o HTML for carregado.
Answer: O PHP é interpretado pelo servidor web (apache, nginx) e sua resposta é entregue a quem o chamou (navegador, ajax). Para visualizar eventuais erros que possam acontecer você deve habilitar a exibição de erros do PHP, no arquivo php.ini existe uma diretiva chamada display_errors mude o valor dela para 1 ou On. Assim quando acontecerem erros os mesmo serão exibidos na tela.
Answer: Coloque essas duas linhas no início do seu código que fará com que o PHP reporte e mostre TODOS os erros e avisos:
ini_set('error_reporting', E_ALL); // mesmo resultado de: error_reporting(E_ALL);
ini_set('display_errors', 1);

Caso queira que todo seu ambiente exiba tais erros e avisos, altere estas mesmas diretivas na configuração do PHP (arquivo php.ini) e reinicie o servidor HTTP (Apache/IIS/Nginx/etc).
Exemplo de configuração do error_reporting:

Obs.: as linhas iniciadas por ponto e vírgula ; são comentários. A configuração se dá pela linha:
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT

As opções estão disponíveis na documentação.

Mais detalhes, vide documentação:
PHP: error_reporting
PHP: display_errors
PHP: ini_set


SLURM sacct shows 'batch' and 'extern' job names

Question: I have submitted a job to a SLURM queue, the job has run and completed. I then check the completed jobs using the sacct command. But looking at the results of the sacct command I notice additional results that I did not expect:
       JobID                        JobName      State      NCPUS  Timelimit
5297048                                test  COMPLETED          1   00:10:00
5297048.bat+                          batch  COMPLETED          1
5297048.ext+                         extern  COMPLETED          1

Can anyone explain what the 'batch' and 'extern' jobs are and what their purpose is. Why does the extern job always complete even when the primary job fails.
I have attempted to search the documentation but have not found a satisfactory and complete answer.
EDIT: Here's the script I am submitting to produce the above sacct output:
#!/bin/bash
echo test_script > done.txt

With the following sbatch command:
sbatch -A BRIDGE-CORE-SL2-CPU --nodes=1 --ntasks=1 -p skylake --cpus-per-task 1 -J jobname -t 00:10:00 --output=./output.out --error=./error.err < test.sh

Answer: A Slurm job contains multiple jobsteps, which are all accounted for (in terms of resource usage) separately by Slurm. Usually, these steps are created using srun/mpirun and enumerated starting from 0. But in addition to that, there are sometimes two special steps. For example, take the following job:
sbatch -n 4 --wrap="srun hostname; srun echo Hello World"
This resulted in the following sacct output:
       JobID    JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
5163571            wrap     medium      admin          4  COMPLETED      0:0
5163571.bat+      batch                 admin          4  COMPLETED      0:0
5163571.ext+     extern                 admin          4  COMPLETED      0:0
5163571.0      hostname                 admin          4  COMPLETED      0:0
5163571.1          echo                 admin          4  COMPLETED      0:0

The two srun calls created the steps 5163571.0 and 5163571.1. The 5163571.bat+ accounts for the ressources needed by the batch script (which in this case is just srun hostname; srun echo Hello World. --wrap just puts that into a file and adds #!/bin/sh).
Many non-MPI programs do a lot of calculations in the batch step, so the ressource usage is accoutned there.
And now for 5163571.ext+: This step accounts for all resources usage by that job outside of slurm. This only shows up, if the PrologFlag contain is used.
An example of a process belonging to a slurm job, but not directly controlled by slurm are ssh sessions. If you ssh into a node where one of your jobs runs, your session will be placed into the context of the job (and you will be limited to your available resources by cgroups, if that is set up). And all calculations you do in that ssh session will be accounted for in the .extern job step.
Comment: Thanks for the insightful answer Marcus!
Answer: A job is comprised of several job steps. Each job step is shown independently. In your output you have $JOBID, which stands for the overall reservation, $JOBID.batch, which represents the main script you submitted.
Regarding external... I'm not sure, but I guess that you started one job step and you named it as "external". In that case, that is the information of that job step.
If you show us the script you submitted we can clarify some soubts.
Comment: If $JOBID.batch represents the script I submitted, and external is something yet to be resolved what does the job with simply $JOBID represent?
Comment: The $JOBID represents the overall reservation.
Comment: If you run `sacct` with `-l` parameter you will see that the information given by each of the lines is different. Maybe you have a reservation of 8 CPUs and two job steps, each of them using 4 CPUs. That will be shown in `sacct`.
Comment: Unfortunately, I cannot see neither any jop step being started in your job nor any reference to "external". I have no idea about that.
Comment: I think the `.ext` job is common to SLURM, for example you can see in this unrelated question the `sacct` output shows a `.ext` job step.
https://bugs.schedmd.com/show_bug.cgi?id=3461#c2


Some people have several separate policies how can I filter the data so I can only see the most recent contact date for each email address?

Question: My data looks like the below

Policy
ContactDate
Email
Name

POL1
17:02:2021:11:51:41
JON@gmail
Jon

POL2
17:02:2021:15:10:44
JON@gmail
Jon

PO3
13:02:2021:09:41:00
Pete@aol
Pete

POL4
14:02:2021:05:40:15
Kate@yahoo
Kate

POL4
14:02:2021:18:15:06
Kate@yahoo
Kate

Some people have several separate policies how can I filter the data so I can only see the most recent contact date for each email address? So my results would be Row 2,3 & 4.
Answer: Like this
<code>SELECT MAX(ContactDate), Email, Name
FROM Table
GROUP BY Email, name;
</code>
Comment: Just to be clear - original question asks for rows 2,3 & 4 - this is not achievable with the (reworked) solution from ex4 - row 4 has an earlier timestamp than row 5 - So question might need revising to show rows 2,3 & 5 ?
Comment: I meant 5 not 4 , clearly didn't read it after I typed it! Thank you :)


What is the expected value of $E[X^2 Y]$ if X and Y are independent?

Question: Given X, Y independent, how can we compute $E[X^2 Y]$?
Comment: Take $Z = X^2$. Then what is $E(ZY)$? Would $Z$ and $Y$ be independent?
Comment: How can I prove that $[^2 ] = []^2 E[]$ ?
Comment: @lle What you have written would be false in general, as we usually have  $E(X^2) \neq E(X)^2$. Otherwise there wouldn't be any point in the variance $\sigma^2$
Comment: My question was on the back of a comment that got posted and later deleted, thanks :)
Comment: Following your answer, how can i prove that Z and Y are independent?
Answer: The short answer is that $$E(X^2 Y) = E(X^2) E(Y)$$ as independence is preserved under transformations. In general, if $X$ and $Y$ are independent, then $f(X)$ and $g(Y)$ will be independent.
Note however that this does not simply any further. We cannot say that $E(X^2) E(Y) = E(X)^2 E(Y)$ as this is untrue in general.
Comment: Though you can say $\mathsf E(X^2)=\mathsf{Var}(X)+\mathsf E(X)^2$, to use if you have the variance and expectation handy.
Answer: $$\begin{align}\mathsf E(X^2Y)&=\mathsf E(\mathsf E(X^2Y\mid Y))&&\text{Law of Total Expectation}\\&=\mathsf E(\mathsf E(X^2\mid Y)\cdot Y)&&Y\text{ relatively constant to a conditional expectation wrt }Y\\&=\mathsf E(\mathsf E(X^2)\cdot Y)&&\bigstar\\&=\mathsf E(X^2)\cdot \mathsf E(Y)&&\mathsf E(X^2)\text{ is a constant}\\[3ex]&=\big(\mathsf{Var}(X)+\mathsf E(X)^2\big)\cdot \mathsf E(Y)&&\text{by definition for Variance.}\end{align}$$
$^\star:$ $X$ and $Y$ are independent, so the realised value of $Y$ has no influence on the expected value of $X$ nor any monovariate function of $X$ (such as $X^2$).   Therefore the conditional expectation with respect to $Y$ is just the expectation.
Answer: $X$ and $Y$ are independent $\Rightarrow X^{2}$ and $Y$ are independent.
Let us recall some definitions. Let $(\Omega,\mathcal{F},P)$ be a
probability space. Let $\mathcal{G}_{1},\mathcal{G}_{2}\subseteq\mathcal{F}$.
We say that $\mathcal{G}_{1},\mathcal{G}_{2}$ are independent if
$P(A\cap B)=P(A)P(B)$ for any $A\in\mathcal{G}_{1}$ and $B\in\mathcal{G}_{2}$.
Now, if $X$ and $Y$ are random variables, we say that $X$ and $Y$
are independent if $\sigma(X)$ and $\sigma(Y)$ are independent.
Note that $\sigma(X^{2})\subseteq\sigma(X)$ (because $X^{2}$ is
$\sigma(X)/\mathcal{B}(\mathbb{R})$-measurable and $\sigma(X^{2})$
is the smallest $\sigma$-algebra on $\Omega$ such that $X^{2}$
is measurable). Now, it is clear that $\sigma(X)$ and $\sigma(Y)$
are independent $\Rightarrow$ $\sigma(X^{2})$ and $\sigma(Y)$ are
independent.


Build xcframework using Aggregated Targets in xcode 12.4

Question: I was trying to create an aggregate framework in Xcode 12.4  My run script is :-
but getting the error when i compile. can anyone help me on this .
Answer: You could delete the derived data and try once again.
ANSSupportURLConnectionDelegate is implemented in more than one place. You can recheck extensions and verify whether it's repeated in any extensions.


How to validate a form by ajax before submit in jQuery?

Question: I'm doing it this way:
...submit(function() {
    $.post(...,function() {
         validate();
    })
});

But it turns out that the form is submitted directly...
Answer: Have a look at the jQuery Validation Plugin.
Here's an example on how it could be used:
$("#myForm").validate({
    rules: {
        command: 'required'
    },

    messages: {
        command: 'Please enter a command.'
    },

    submitHandler: function(form) {
        $(form).ajaxSubmit({
            success: function(data) { /* ... */ },
            url: form.action,
            dataType: 'json'
        });
    }
});

Comment: No, usually the proper way is to do client-side validation first as it is much faster and *then* submit the form, either returning a success-response (which could e.g. redirect the user) or an error which is then also handled by your success callback.
If most of your clients have JavaScript enabled the server-side validation could be as simple as returning a "Some fields are invalid"-like error and rely on clientside validation for verbose errors
Comment: Does it support server side validation?
Comment: +1 for doing validation on the Client side. 99% of clients do enable Javascript, hence pretty & meaningfull validation errors can be handled Client Side. In some cases, you have no choice though & will have to to complex business logic on the server side (ie. access a Db & so on)
Answer: You'll have to grab each element and run it through your validation, then submit the form later.
Use $(id).value (for id-d elements) or yourForm.someElementName.value (for name-d elements, assuming your form is named yourForm).
Answer: you need to prevent the default form submission
http://api.jquery.com/event.preventDefault/
$("form").submit(function(event){
  event.preventDefault();
  if(valid())
  {
    $.post . . .
  }
});

Comment: I somehow have the feeling he's doing serverside validation.. i.e. submitting the form twice: once for validation and once for actual processing. IMO this is awful (but the struts-jquery plugin does something like that too); see my answer for a better client-side validation solution.
Comment: Yes,I'm doing server side validation.
Answer: If you want to AJAXily do server side validation without posting the form to another page, you could do something like this. Instead of binding the validation action to submit, you could instead make your "submit" button just a "button" type, and then bind your validation to it's click event. Once it passes validation, then you can move on.
For instance:
$('#myPseudoSubmitButton').click(function() {
     // Send the form to the server side for validation
     $.post('validation_script.php', { form inputs here }, function(data) {
          if (data.validated) {
               // Perform some functionality that happens when the form validates
          }
          else {
               // Display an error message to the user
          }
     });
}, 'json');

This of course presupposes that your validation script returns JSON-formatted data with a "validated" attribute.


cancel uploading file using CustomMultiPartEntity in Android

Question: I am using CustomMultiPartEntity for Uploading file with Cancel feature, file upload successfully but when I try to cancel this task to stop uploading it won't, does anybody have implemented or any idea how to stop uploading?
below is my code with cancel
import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.charset.Charset;

import org.apache.http.entity.mime.HttpMultipartMode;
import org.apache.http.entity.mime.MultipartEntity;

import android.util.Log;

public class CustomMultiPartEntity extends MultipartEntity {
    private static final String TAG = CustomMultiPartEntity.class.getSimpleName();
    private final ProgressListener listener;
    private static boolean isUserInterrupt = false;
    public CustomMultiPartEntity(final ProgressListener listener) {
        super();
        this.listener = listener;
        isUserInterrupt = false;
    }

    public CustomMultiPartEntity(final HttpMultipartMode mode,
            final ProgressListener listener) {
        super(mode);
        this.listener = listener;
        isUserInterrupt = false;
    }

    public CustomMultiPartEntity(HttpMultipartMode mode, final String boundary,
            final Charset charset, final ProgressListener listener) {
        super(mode, boundary, charset);
        this.listener = listener;
        isUserInterrupt = false;
    }

    public void cancelUploading(){
        isUserInterrupt = true;
        if(cos!=null){
            try{
                cos.flush();
                cos.close();
                cos = null;
            }catch(Exception e){
                e.printStackTrace();
            }
        }
    }
    private CountingOutputStream cos;
    @Override
    public void writeTo(final OutputStream outstream) throws IOException {
        if(isUserInterrupt){
            if(cos!=null){
                try{
                    cos.flush();
                    cos.close();
                    cos = null;
                }catch(Exception e){
                    e.printStackTrace();
                }
            }
        }

        cos = new CountingOutputStream(outstream, this.listener);
        super.writeTo(cos);
        Log.i(TAG, "write to calling");
    }

    public static interface ProgressListener {
        void transferred(long num);
    }

    public static class CountingOutputStream extends FilterOutputStream {

        private final ProgressListener listener;
        private long transferred;

        public CountingOutputStream(final OutputStream out,
                final ProgressListener listener) {
            super(out);
            this.listener = listener;
            this.transferred = 0;
        }

        public void write(byte[] b, int off, int len) throws IOException {
            if(isUserInterrupt)
            {
                Log.i(TAG, "write interrupt, off="+off+", len+"+len);
                return;
            }
            out.write(b, off, len);
            this.transferred += len;
            this.listener.transferred(this.transferred);
            Log.i(TAG, "write off="+off+", len+"+len);
        }

        public void write(int b) throws IOException {
            if(isUserInterrupt)
            {
                Log.i(TAG, "write interrupt, b+"+b);
                return;
            }
            out.write(b);
            this.transferred++;
            this.listener.transferred(this.transferred);
            Log.i(TAG, "write b+"+b);
        }
    }
}

Cancelling Upload using this way in AsyncTask
public class UploadImageTask extends AsyncTask<HttpResponse, Integer, String> {
@Override
    protected void onCancelled() {
        super.onCancelled();
        Log.e(TAG, "task cancelled");
    }

    public void taskCancel() {
        if(httpPost!=null){
            httpPost.abort();
            httpPost = null;
        }
        if(httpClient!=null){
            httpClient.getConnectionManager().shutdown();
        }
        httpClient = null;
    }
}

Cancel Task using below code
UploadImageTask task = (UploadImageTask)iMap.get(id);
if(task!=null && !task.isCancelled()){
    task.taskCancel();
    task.cancel(true);
}

Comment: How are you trying to cancel the upload?
Comment: I assume you are using an AsyncTask to make your Upload. Why don't you call cancel() on your AsyncTask if the user cancels the upload?
Comment: Upload task performing in AsyncTask, when I cancelling AT it will cancelled but upload file not stop after upload it will called onCancel() and onCancel(String s) methods
Comment: I edited my question with more code
Answer: I found way to cancel the uploading current file. Create one method which abort or shutdown current connection with server
public class MyTask extends AsyncTask<Void, Integer, String>{

    private HttpPost httpPost;
    private CustomMultiPartEntity multipartContent;
    private HttpClient httpClient;
    @Override
    protected String doInBackground(HttpResponse... arg0) {
        httpClient = new DefaultHttpClient();
        HttpContext httpContext = new BasicHttpContext();
        httpPost = new HttpPost(urlStr);

        ....
        // other upload statement here
        return resp;
    }
    @Override
    protected void onCancelled() {
        super.onCancelled();
        Log.e(TAG, "task cancelled");
    }

    public void taskCancel() {
        if(httpPost!=null){
            httpPost.abort();
            httpPost = null;
        }
        if(httpClient!=null){
            httpClient.getConnectionManager().shutdown();
        }
        httpClient = null;
    }
}

Now call taskCancel() first then cancel(true) method to abort current uploading task
Hope this will help you
Comment: Nicely done.  But this code is rather old.  Do you have more recent code?  Perhaps something that uses Volley or some other currently supported library?


Forecasts fail for new period

Question: I am working with prophet library in python where I do some forecasts. While I split to train and test to check, it shows very good performance, but when I forecast for the actual future period it loses everything. See photo below:

Any ideas as to why this would occur?
Also, here is the part in my function that I do the forecasting:
<code>resampled_data = data.resample(granularity, on='created_at').agg({y_col: 'sum', regressor: 'nunique'}).reset_index()

    # Normalize data
    scaler = MinMaxScaler()
    resampled_data['normalized_y'] = scaler.fit_transform(resampled_data[[y_col]])

    # Define train and test data
    train = resampled_data.iloc[-(3*30 + 2*period):-period]
    test = resampled_data.iloc[-period:]

    model = Prophet(interval_width=prophet_intervals)
    model.add_country_holidays(country_name='CY')
    model.add_regressor(regressor)
    model.fit(train.rename(columns={'created_at': 'ds', 'normalized_y': 'y'}))

    future = model.make_future_dataframe(periods=2*period, freq=granularity)
    future = future.merge(resampled_data[['created_at', regressor]], left_on='ds', right_on='created_at', how='left')

    # Fill NaN values in the regressor column
    future[regressor].fillna(method='ffill', inplace=True)

    forecast_out = model.predict(future)
</code>
I think I understand why it does it. I think it's because I use a regressor for which I don't have the data for the forecasted period. No it produces NaN which are then filled with the last observed value. So my regressor has a constant value for the period I want to forecast.
How do I handle this?
The actual data looks like this:
Comment: Welcome to Cross Validated! What do the real data do? I don't see them on your plot.
Comment: Hello, and thank you! They are exactly as the previous periods.
Comment: Could you please plot the actual data? You have a nice line in blue showing the rest of the data. Perhaps continue that with the forecasts plotted on a different color.
Comment: @Dave I edited my question
Answer: It looks like hourly data with a clear daily cycle and weekend effect. Interestingly the last day in your data (Thursday I guess) has a different pattern -> holiday effect? ; did you account for that in your features?
Have you tried the basics first like seasonal decomposition/forecasting? Rob Hyndman has several algorithms in that area for improved forecasting of multi-seasonal time series. See e.g. https://pkg.robjhyndman.com/forecast/reference/msts.html
Also I suggest using sthg else than prophet, especially for the basics like seasonal time series forecasting. Why?  Have a look at Peter Cottons excellent take on prophet at " Microprediction -
Is Facebook's "Prophet" the Time-Series Messiah, or Just a Very Naughty Boy?".
He actually goes into very much detail on failures specifically for cyclic time series.  And offers a better API for prophet and other forecasting tools at microprediction.com / https://github.com/microprediction
Comment: Rob Hyndman is my idol :D I did try the decomposition and some naive methods which would capture it but not as good as prophet with holidays into account, etc. I think the problem was as I described it in my question. My regressor did not have values for the period to be forecasted and was producing NaNs. To fix this I carried the last value forward to fill them and that is how I lost the seasonalisty of it, while the trend was downwards. I did two models instead and I forecast the regressor variable and use those forecasts as the regressor for the main model. Maybe I should answer it?
Comment: Yes, right.  You need to do a one step at a time forecast, concatenate it with data from last k-1 steps, and feed it back in as 'x' .   See also comment in Microprediction blog - their wrapper API does it automatically

Forecasts fail for new period


JQuery Masonry combined w/ Infinite Scroll. Infinite Scroll doesn't retain CSS

Question: I have combined the JQuery Masonry plug in with the Infinite Scroll Plug in and the "infinite scroll" blocks don't retain the CSS of the Masonry "blocks" (see url). I am confused as to why and how this can be remedied. Thank you.
http://helpknow.com/black/
Comment: Are you notifing masonry after infinitescroll has loade new content?
Comment: Oh, and make sure you do that after the images of new content are really loaded.
Comment: Check it out here: http://www.emotiland.com
Comment: I don't know how to notify it after. I will look at that site and hopefully it's innards will show me.
Answer: Masonry loads and formats the DOM objects. Infinite Scroll then adds more elements to the DOM. You need to tell Masonry to then format these new items.