Conflict in the graph of $\{(x,y)|(x-|x|)^2+(y-|y|)^2 \leq 4\}$ directly and by cases

Question: Question

$\text{Plot the graph of}$ $\{$ $(x,y) \in \mathbb{R}$ | $(x-|x|)^2+(y-|y|)^2 \leq 4\}$.

My attempt

Case $1$ : $x,y<0$
$⇒(x+x)^2+(y+y)^2\leq4⇒4x^2+4y^2\leq4⇒x^2+y^2\leq1⇒ $ A shaded circle with center at origin and radius $=1.$

Case 2 : $x>0,y<0⇒ 4y^2\leq4⇒y^2\leq1⇒|y|\le1⇒y=1,-1⇒$ Shaded region from $-1$ to $1$ on $Y-$axis.

Case 3: $x<0,y>0⇒ $ Similar to above shaded from $[-1,1]$ on $X-$axis.

Case 4 : $x,y>0⇒(x-x)^2+(y-y)^2\le4⇒$ All $+\text{ve}$ Real Numbers.

So when I plot this all the region should be the answer.

Correct Answer
But when I plot above inequality directly ,

Problem
Where is it wrong? Because the final graph will be union of all the cases right?
References
Link for interactive graph of my cases (image 1): https://www.desmos.com/calculator/cwhuffantk
Link for the graph of the original inequality (image 2): https://www.desmos.com/calculator/bwc19xvibn
Comment: @ryang The graph cleared my problem! Thanks!
Comment: Assuming that your 'case 2' here is the blue band: why does that extend all the way to the left?
Comment: @StevenStadnicki Consider $y=1$ and $y=-1$. Notice that $\le 1$  is there. This means that this is the whole region where it satisfies the inequality.
Comment: @ryang Updated!
Answer: Why is there a difference in the two solutions?
The issue here is that you haven't considered the restrictions on x and y in each scenario. In each case you assume that x and y are either positive negative or a combination of the two. However, in your graph, you assume that each case holds simultaneously for all values x and y (when they only hold in the case where x and y meet the required restriction).
How to resolve this difference?
Let's look at the four claims you make.
Case 1: you are considering a shaded circle of radius 1. However, we assume that $x<0$ and $y<0$, so clearly we don't have a full circle here, we only have one quarter of a circle (where x and y are both negative). We therefore, must only sketch the circle for the bottom left quadrant.
Case 2: by the same reasoning, we are only consider $|y|\le 1$ in the bottom right quadrant of the graph as this only holds under the assumption that x is positive and y is negative.
Case 3: again, as above, $|x| \le 1$ only applies in the top left quadrant by the same reasoning. We see that this only holds when we assume that x is negative and y is positive.
Case 4: as before, all pairs of real numbers $x,y$ only applies in the top right quadrant (positive x, and positive y). Since we make this assumption to come to that conclusion, it isn't correct to consider this to hold for values outside positive x and positive y combinations.
The main take away here, is that these 4 cases only give you those equations if you make an assumption about the values of x and y. Therefore, if you apply these graphs to the entire cartesian plane, this does not follow on from the cases. Each one applies in a different quadrant and so you need to look at the suitable values of x and y in each case and shade each quadrant appropriately.
If you do this, then you will arrive at the same answer as the one in the Desmos produced graphic that you have included in your question.
Answer:
Case $1$ : $x,y<0$ $⇒(x+x)^2+(y+y)^2\leq4⇒4x^2+4y^2\leq4⇒x^2+y^2\leq1⇒ $ A shaded circle
with center at origin and radius $=1.$

Correct, and note that Case 1 lies only in Quadrant 3. (So, restrict the shaded red circle to just the portion in Quadrant 3.)

Case 2 : $x>0,y<0⇒ 4y^2\leq4⇒y^2\leq1⇒|y|\le1⇒y=1,-1⇒$ Shaded region from $-1$ to $1$ on $Y-$axis.

Yes, and Case 2 lies in Quadrant 4.

Case 3: $x<0,y>0⇒ $ Similar to above shaded from $[-1,1]$ on $X-$axis.

Yes, and Case 3 lies in Quadrant 2.

Case 4 : $x,y>0⇒(x-x)^2+(y-y)^2\le4⇒$ All $+\text{ve}$ Real Numbers.

Note that $$0\le4\kern.6em\not\kern-.6em\implies x>0$$ (for example, notice that $x=-3$ satisfies the LHS, but not the RHS, of the implication); rather, $$0\le4\implies (x,y)\in\mathbb R^2.$$ But since case 4 lies in Quadrant 1, we just want the the entirety of Quadrant 1.
Finally, taking the union of the four cases (and considering the points on the two axes) gives the required region https://www.desmos.com/calculator/spznk8ziya, which matches what Desmos directly gave.


Calculate $\sum\limits_{n\geq 1}\frac{1}{1+2+\ldots+n} $

Question: Calculate  $\displaystyle \sum_{n\geq 1}\dfrac{1}{1+2+\ldots+n} $
My attempts :
\begin{aligned}
 \sum_{n\geq 1}\dfrac{1}{1+2+\ldots+n} &= \sum_{n\geq 1}\dfrac{2}{n(n+1)}=2\sum_{n\geq 1}\left( \dfrac{1}{n}-\dfrac{1}{n+1}\right)=2\left( \dfrac{1}{1}-\dfrac{1}{+\infty}\right)=2(1-0)=2?
\end{aligned}
Comment: Try computing a few values of $$\sum_{n=1}^N\left(\frac1n-\frac1{n+1}\right)$$ just for kicks...
Comment: Both $\infty$ :D
Comment: yes i see now thanks
Answer: The series 'telescopes' with the only remaining terms being $1-\frac{1}{r+1}$ where $r$ is the number of terms. That's $\frac{r}{r+1}$ which tends to $1$ for partial sums. It's not too hard to show that this sum is not commutative because it alternates with decreasing terms.
Answer: \begin{aligned}
\sum_{n\geq 1}\dfrac{2}{n(n+1)}=2\sum_{n\geq 1}\dfrac{1}{n}-2\sum_{n\geq 1}\dfrac{1}{n+1}
\end{aligned}
False. The equality does not hold because the two sums on the right are divergent, while the sum on the left is convergent.

Better go at it this way:
$$\sum_{n=1}^\infty\frac{1}{n(n+1)} = \sum_{n=1}^\infty\frac{1}{n}-\frac{1}{n+1} = \left(\frac 11 - \frac12\right) + \left(\frac12-\frac13\right) + \left(\frac13-\frac14\right) + \cdots$$
Now, remember that $$\sum_{n=1}^\infty a_n = \lim_{N\to\infty} \sum_{n=1}^N a_n$$
In your case, $$\sum_{n=1}^N \frac 1n-\frac{1}{n+1}$$ can be calculated relatively easily.
Answer: $$\sum_{n>1}\frac{1}{1+2+...+n}=2\sum_{n>1}\left(\frac{1}{n}-\frac{1}{n+1}\right)=2\lim_{n\rightarrow+\infty}\left(1-\frac{1}{n+1}\right)=2$$
Answer: I like to look at it like this:
$$\sum_{n=1}^\infty\frac1{n(n+1)}=\sum_{n=1}^\infty\left(\frac1n-\frac1{n+1}\right)$$
$$\begin{align}\require{cancel}&=\left(\frac11-\frac12\right)+\left(\frac12-\frac13\right)+\left(\frac13-\frac14\right)+\dots\\&=\left(\frac11-\color{red}{\cancel{\frac12}}\right)+\left(\color{red}{\cancel{\frac12}}-\color{orange}{\cancel{\frac13}}\right)+\left(\color{orange}{\cancel{\frac13}}-\color{green}{\cancel{\frac14}}\right)+\color{green}\dots\\&=1\end{align}$$
Notably, this is called a telescoping series.
Answer: HINT:
$$\dfrac1{n(n+1)}=\dfrac{n+1-n}{n(n+1)}=\dfrac1n-\dfrac1{n+1}$$
See Telescoping series
Comment: i already did that
Comment: @Yacob, Set a few values of $n$ to find the surviving terms
Comment: @Yacob, See https://math.stackexchange.com/questions/475256/if-this-is-a-telescoping-series-then-how-does-it-collapse-frac3r1rr-1r/475264  OR https://math.stackexchange.com/questions/475256/if-this-is-a-telescoping-series-then-how-does-it-collapse-frac3r1rr-1r/475264 OR https://math.stackexchange.com/questions/2100299/how-do-telescopic-series-work-in-general-and-in-this-specific-problem


Show $T$ is a bounded linear operator

Question: Let $T:C[0,1]\rightarrow C[0,1]$ be defined as:
$$(Tf)(x)=\int_{0}^{1}xyf(y)\,\mathrm{d}y.$$
Show $T$ is a bounded linear operator and calculate its norm.
My idea:
First I know that $\Vert f\Vert=\sup_{y\in[0,1]} |f(y)|$ and $f$ is defined and continuous on $[0,1]$. I feel that the integral is clearly not ill-defined since:
$$\int_{0}^{1}xyf(y)\,\mathrm{d}y\leq x\int_{0}^{1}y\cdot\sup_{z\in[0,1]}|f(z)|\,\mathrm{d}y=x\sup_{z\in[0,1]}|f(z)|$$ (not sure about that)
Now, if $\Vert f\Vert\neq 0$, we can find the norm of the operator by applying the following formula:
$\sup\frac{\Vert Tf\Vert}{\Vert f\Vert}=\sup\frac{\sup\left|\int_{0}^{1}xyf(y)dy\right|}{\sup|f|}$, but where does it lead?
Answer: If $x\in[0,1]$ and $\lVert f\rVert=1$, then\begin{align}\bigl\lvert(Tf)(x)\bigr\rvert&=\left\lvert\int_0^1xyf(y)\,\mathrm dy\right\rvert\\&\leqslant x\int_0^1y\bigl\lvert f(y)\bigr\rvert\,\mathrm dy\\&\leqslant x\int_0^1y\,\mathrm dy\\&=\frac x2.\end{align}Therefore, $\lVert Tf\rVert\leqslant\frac12$. Actuallly, $\lVert Tf\rVert=\frac12$; just see what happens if $f=1$.
Answer: Well, as you (nearly) already noticed
$$
\left| \int_0^1 xy f(y) \,\mathrm{d}y \right|
\le x \|f\|_\infty \int_0^1 y \,\mathrm{d}y
= \frac{x}{2} \|f\|_\infty,
$$
hence
$$\|Tf\|_\infty \le \frac12 \|f\|_\infty.$$
Therefore $T$ is bounded with $\|T\| \le \frac12$.
For $f \equiv 1$ you get $(Tf)(x) = \frac{x}{2}$, hence $\|Tf\|_\infty = \frac12 \|f\|_\infty$, so $\|T\| \ge \frac12$.
Comment: right, forgot about the one-half factor! that's really helpful

Show $T$ is a bounded linear operator


Elasticsearch : How to get all indices that ever existed

Question: is there a way to find out the names of all the indices ever created? Even after the index might have been deleted. Does elastic store such historical info?
Thanks
Answer: Using a plugin that keeps an audit trail for all changes that happened in your ES cluster might do the trick.
If you use the changes plugin (or a more recent one), then you can query it for all the changes in all indices using
<code>curl -XGET http://localhost:9200/_changes
</code>
and your response will contain all the index names that were at least created. Not sure this plugin works with the latest versions of ES, though.


Are off-policy learning methods better than on-policy methods?

Question: I cannot understand what the fundamental difference between on-policy methods (like A3C) and off-policy methods (like DDPG) is. As far as I know, off-policy methods can learn the optimal policy regardless of the behavior policy. It can learn by observing any trajectory in the environment. Therefore, can I say off-policy methods are better than on-policy methods?
I have read the cliff-walking example showing the difference between SARSA and Q-learning. It says that Q-learning would learn the optimal policy to walk along the cliff, while SARSA would learn to choose a safer way when using the epsilon-greedy policy. But since Q-learning have already told us that the optimal policy, why don't we just follow that policy instead of keep exploring?
Plus, are there situations for the two kinds of learning methods that one is better than the other? in which case would one prefer on-policy algorithms?
Answer: As you already said, off-policy methods can learn the optimal policy regardless of the behaviour policy (actually the behaviour policy should have some properties), while on-policy methods require the agent acts with the policy that it's being learnt.
Imagine the situation where you have a data set of trajectories (i.e, data in form of tuples (s,a,r,s')) previously stored. This data has been collected applying a given policy, and you cannot change it. In this case, which is common for medical problems, you can apply only off-policy methods.
This means that off-policy methods are better? No necessarily. We can say that off-policy methods are more flexible in the type of problems they can face. However, from the theoretical point of view, they have different properties that sometimes are convenient. For instance, if we compare Q-learning versus SARSA algorithm, a key difference between them is the max operator used in the Q-learning update rule. This operator is highly non-linear, which can make more difficult to combine the algorithm with function approximators.
When it's better to use on-policy methods? Well, if you are facing a problem with continuous state-space and you are interested in using a linear function approximator (and RFB network, for instance). Then it's more stable to use on-policy methods. You can find more information on this topic in the Section off-policy bootstrapping of Sutton and Barto's book.
Comment: Thank you! So Q learning is more difficult to be approximated by linear functions, but now DQN is invented and tricks like experience replay have made it stable. Is it still a bad part then?
Comment: Experience replay actually is quite old, but yes, this kind of tricks made Q-learning more stable. I cannot say that there is a "bad part", only that they have different theoretical properties and sometimes these properties play a key role in their practical application. Currently, and specially in the deep learning field, Q-learning is likely more popular and widely used.
Comment: Can we say on-policy methods often converge faster, or more stable, compared to off-policy ones? So it can be used when we want to find a fairly good policy in a short time?
Comment: In general, on-policy metfhods often converge faster, however this does not mean that is computationally less demanding. Also, they provide more theoretical convergence properties when combined with linear function approximators. A more specific comparison would depend on the particular algorithms considered and on the problem at hand.
Comment: @DarkZero, please, let me know if you have additional doubts. Otherwise, if the answer was useful, don't forget to accept it.
Comment: Sorry for late comment, so can we say that on default we can choose off-policy methods? Because it seems rare to face a problem with continuous state space, and few people want to use linear approximator now, they all prefer neural networks...
Comment: Well, "rare" and "few people" depends very much on the ambit. But if you are working in a kind of applications where linear approximators are unusual, yes off-policy methods can be the default option. On the other hand, continuous state-space means that the state space if defined by continuous variables, which happens very often. For example in a simply cart-pole problem, where angle and agular velocity are continuous variables.
Comment: Oh I know the cart pole example. So in practice, we often have to try both to determine which is the better choice for the problem, right?
Comment: Yes, in general is difficult to know in advance which exact algorithm will perform better than the rest. However, if you are a wide experience with the domain and/or the algorithms, probably you will have a feeling of which kind of algorithm can fit better the requirements of the problem.
Comment: Ok, I got it. Thank you for all the discussions!


What is the difference between the Cartesian product of two sets with and set of all function from one set to other?

Question: There are two sets A and B with cardanlity |A| and |B| respectively.
First talking about Cartesian product - the cardnality of set is |A||B|
And the cardanility of set of all functions from set A to set B is |A| to the power |B|
but both are formed the same way then why the cardanility are different
Comment: Why do you say that both are formed the same way? A function can be interpreted as a "subset" of a Cartesian product.
Comment: Because they are **not** the same set... Try with a simple example: $A = \{ 1,2 \}$ and $B = \{ a,b,c \}$. $A \times B$ has $2 \times 3 =6$ elements while $A^B$ has $2^3=8$ elements.
Comment: The two sets are most certainly not formed the same way. Why do you think they are formed in the same way?
Comment: A function $\ f:A\to B$ is a subset of the cartesian product $\ A\times B$
Answer: In set theory, the set of functions from X to Y may be denoted X → Y or Y^X. Let X and Y be finite sets | say, |X| = n and |Y| = m. Without loss of
generality, you could let X = {1,2,..., n} and Y = {1,2,..m}. Verify that |Y^X|=|Y|^|X|
i.e., |Y^X| = m^n
*    As a special case, the power set of a set X may be identified with the set of all functions from X to {0, 1}, denoted 2^X.
Cartesian product of two sets
X = {a, b}
Y = {1, 2, 3}
Now,
X×Y = {(a,1), (a,2), (a,3), (b,1), (b,2), (b,3)}
Y×X = {(1,a), (1,b), (2,a),(2,b), (3,a), (3,b)}
Answer: The members of the Cartesian product $X\times Y$ are ordered pairs of the form $(x,y)$ where $x\in X,y\in Y$, but of course in the family $Y^X$ of all functions from $X$ to $Y$ each member is a function from $X$ to $Y$ not an ordered pair $(x,y)$ as above.


Agrupar objeto no javascript por categorias

Question: Possuo um código com Datatable e queria agrupar e somar os valores em situações
  var table = $('#tabelaDadosFinanc').DataTable();

    var data = table
    .rows()
    .data();

    var total = 0;

    for (var i = 0; i < data.length; i++){

        var valor = data[i]['valor'];
        var moeda = data[i]['moeda'];

        removeFirst = valor.replace('','');

            valores = removeFirst.replace('','');

            a = valores.replace('.','');
            b = a.replace(',', '.')

            total +=parseFloat(b);

    }
     console.log(total);

Precisaria Agrupar pelo tipo de moeda e somar os valores,
ex:
 como tenho hoje
Moeda |  Valor
Real  | -56,12
Real  | -48,54
AU    | -46,60
Real  | -46,60
Real  | -46,60
Real  | 46,60
Real  | 48,54
Real  | 56,12

para ficar assim no final
 Real| -46,60
 AU | -46,60

Comment: Vai considerar sigla da moeda e nome?
Comment: considero primeiramente a moeda - e agrupo os valores que pertencem a moeda ex: Real -> -46,60
 AU -> -46,60
Comment: Os valores no final do agrupamento não são esses por exemplo Real | vai dar outro valor?
Comment: o valor do agrupamento é a soma dos valores, total de real, total de Au
Comment: Mas Real é diferente de R$ que, por sua vez, é diferente de RS. A dúvida é, Real, R$ e RS são a mesma coisa?
Comment: Sim são as mesmas coisas
Comment: Felipe, só uma dúvida como converter isso (var valor = data[i]['valor'];
        var moeda = data[i]['moeda'];) em const data = [
  {"moeda": "Real", "valor":-56.12},
  {"moeda": "Real", "valor":-48.54},
  {"moeda": "AU", "valor":-46.60},
  {"moeda": "Real", "valor":-46.60},
  {"moeda": "Real", "valor":-46.60},
  {"moeda": "Real", "valor":46.60},
  {"moeda": "Real", "valor":48.54},
  {"moeda": "Real", "valor":56.12 }
];
Answer: Para fazer isso, basta fazer o seguinte:

const data = [
  {"moeda": "Real", "valor":-56.12},
  {"moeda": "Real", "valor":-48.54},
  {"moeda": "AU", "valor":-46.60},
  {"moeda": "Real", "valor":-46.60},
  {"moeda": "Real", "valor":-46.60},
  {"moeda": "Real", "valor":46.60},
  {"moeda": "Real", "valor":48.54},
  {"moeda": "Real", "valor":56.12 }
];

const resultado = {};

data.forEach(item => {
  if(resultado.hasOwnProperty(item.moeda)) {
    resultado[item.moeda] += item.valor;
  } else {
    resultado[item.moeda] = item.valor;
  }
});

console.log(resultado);

console.log("MOEDA\t| VALOR");
for(var moeda in resultado) {
  console.log(moeda+"\t| "+resultado[moeda]);
}

Caso você queira fazer uma tradução da moeda, você pode fazer da seguinte forma:

const siglaMoedas = {
  "R$": ["Real","RS"],
  "AU$": ["AU", "AUD"],
  "US$": ["Dolar", "USD"]
};

function pegaSigla(nomeMoeda) {
  for(let sigla in siglaMoedas) {
    if(siglaMoedas[sigla].includes(nomeMoeda)) return sigla;
  }
  return nomeMoeda;
}

console.log(pegaSigla('Real'));
console.log(pegaSigla('AU'));
console.log(pegaSigla('USD'));
console.log(pegaSigla('Dolar Canadense'));

E utilizar o segundo script para substituir as partes onde tem item.moeda para pegaSigla(item.moeda)


How to map to add multiple IEnumerable to an IEnumerable property of a model

Question: Let's say I have this code
int? pk=100;
int? ptype1 =51;
int? ptype2 =52;
int? ptype3 =53;

public class SampleDto
{
    public IEnumerable<int> Property1{ get; set; }
    public IEnumerable<int> Property2{ get; set; }
    public IEnumerable<int> Property3{ get; set; }
}

SampleDto sample = new SampleDto();
sample.Property1.Add(11);
sample.Property1.Add(12);
sample.Property1.Add(13);
sample.Property2.Add(21);
sample.Property2.Add(22);
sample.Property2.Add(23);
sample.Property3.Add(31);
sample.Property3.Add(32);
sample.Property3.Add(33);

How do I map SampleDto properties to add to an IEnumerable of SampleModel using Automapper
public class SampleModel
{
    public int? Id { get; set; } //same for all pk
    public int? TypeId { get; set; } //ptype depending on the property. Ex. ptype1 for Property1
    public int? Values { get; set; } //Value of Property
}

With the values above, I want an expected result of IEnumerable of SampleModel with the following data:
Id =100, TypeId =51, Values =11
Id =100, TypeId =51, Values =12
Id =100, TypeId =51, Values =13
Id =100, TypeId =52, Values =21
Id =100, TypeId =52, Values =22
Id =100, TypeId =52, Values =23
Id =100, TypeId =53, Values =31
Id =100, TypeId =53, Values =32
Id =100, TypeId =53, Values =33

Comment: Feels like you're asking Automapper to do this https://stackoverflow.com/questions/12640881/how-to-use-zip-on-three-ienumerables or from the linked duplicate https://stackoverflow.com/questions/5284315/create-items-from-3-collections-using-linq ?
Comment: But the more I read your question, the less I understand what you're trying to do. Please give example data as-is and also desired results. You can use JSON representation
Comment: @CaiusJard ok. editting now
Comment: Added expected result
Comment: So you want automapper to do `sample.Property1.Select(p1=>new{p=p1,pt=ptype1}) .Concat(sample.Property2.Select(p2=>new{p=p2,pt=ptype2})) .Concat(sample.Property3.Select(p3=>new{p=p3,pt=ptype3})) .Select(new SampleModel{Id=pk,TypeId=ptype,Values=p})` ?
Comment: WE kind of want to do it in automapper
Comment: @ByronScott that's not what Automapper is about. There's no mapping involved in this case, you're concatenating sequences, not mapping objects
Comment: @PanagiotisKanavos well, there's always the final select that could be mapped :)
Comment: So, did Automapper saved you time? Based on the question did not ;). Write code to concat collections by hand - written only once, ti will work and you spend much less time then asking question about Automapper ;)
Answer: AutoMapper is the wrong tool for the job, since you are not doing a one-to-one mapping (there is no common 'root' between SampleDto and SampleModel).
Using LINQ, you can however create a rather straightforward transformation function that transforms a list of SampleModel objects to the corresponding list of SampleDto objects. Something like this:
static IEnumerable<SampleDto> ConvertFunction(IEnumerable<SampleModel> sample)
{
    var query = from d in sample
                group d by d.Id into g
                let lookup = g.ToLookup(x => x.TypeId, x => x.Values)
                select new SampleDto
                {
                    Property1 = lookup[51],
                    Property2 = lookup[52],
                    Property3 = lookup[53]
                };
    return query.ToList();
}

Then if you absolutely must, you could integrate this in automapper with a custom conversion function:
var config = new MapperConfiguration(cfg =>
        cfg.CreateMap<IEnumerable<SampleModel>, IEnumerable<SampleDto>>()
        .ConvertUsing(input => ConvertFunction(input))
    );

This allows you to do this:
var mapper = new Mapper(config);
var result = mapper.Map<IEnumerable<SampleDto>>(sample);


Apache not responding while server is online

Question: Since a hour, apache stopped responding. I can reach the server with putty and the average load is only a few percent. Apache is (re)starting without error. If it helps: this morning the server turned itself magically off but in this case it's still on, it worked all day without any further problems.
I have absolutely no idea what to look/search for :(
Comment: You don't need a Q&A site you need education and a consultant.
Comment: Servers don't turn themselves off "magically".
Comment: I know, it probably crashed
Comment: So take a look in /var/log/httpd or /var/log/apache2 or similar and see what the error logs show.
Comment: I encountered a problem with SELinux.   This can cause apache not to work after it has been installed.  You have to configure it correctly for the enabling of web serving.
Answer: If it were me I'd....

start with top - always the first place to look when things are amiss - is the load average too high? Too low?
next, netstat -nap | less - is the webserver listening on the relevant ports?
if apache is running, then check the webserver logs - is any traffic going through? Anny errors logged?
then the messages log - is the firewall blocking traffic? Is there anything else amiss?

this morning the server turned itself magically off

Does the error log show it being shutdown? If not, does the messages log show the oom killer terminating processes?


Programmatically updating a Webform submission with a file element doesn't remove the attached files

Question: I am using Drupal 9.3 and Webform 6.1.7.
I have a Webform that contains a file element and am building a custom form to update the submission data.
I have found that when I submit my custom form I can load the Webform submission and set the file IDs and it will save the data fine with no issues.
However when it comes to removing a file after saving, the file I remove just remains attached.
I know that using the Webform interface the file upload / remove is working fine.  However it seems to be a problem with removing files when updating the submission programmatically.
Is there any extra step I am missing to remove the files?
This is really simple version of the form I am building:
<code><?php

namespace Drupal\example\Form;

use Drupal\Core\Form\FormBase;
use Drupal\Core\Form\FormStateInterface;

/**
 * Provides an example form.
 */
class WebformFileUpdateForm extends FormBase {

  /**
   * {@inheritdoc}
   */
  public function getFormId() {
    return 'example_webform_file_update_form';
  }

  /**
   * {@inheritdoc}
   */
  public function buildForm(array $form, FormStateInterface $form_state) {
    $em = \Drupal::entityTypeManager();

    // Load all webform submissions for the node.
    $nid = 86552;
    $query = $em->getStorage('webform_submission')
      ->getQuery()
      ->condition('webform_id', 'tutorial', '=')
      ->condition('entity_id', $nid)
      ->pager(10);

    $submission_ids = $query->execute();
    $webform_submissions = $em->getStorage('webform_submission')
      ->loadMultiple($submission_ids);

    $form['#tree'] = TRUE;

    $form['files'] = [
      '#type' => 'fieldset',
      '#title' => $this->t('Files'),
    ];

    foreach ($webform_submissions as $submission) {
      $form['files'][$submission->id()]['file'] = [
        '#type' => 'managed_file',
        '#title' => $this->t('File'),
        '#default_value' => $submission->getElementData('attached_files'),
        '#upload_location' => 'private://tutorials/' . $submission->id() . '/',
        '#multiple' => TRUE,
      ];
    }

    $form['submit'] = [
      '#type' => 'submit',
      '#value' => $this->t('Update'),
    ];
    return $form;
  }

  /**
   * {@inheritdoc}
   */
  public function validateForm(array &$form, FormStateInterface $form_state) {
  }

  /**
   * {@inheritdoc}
   */
  public function submitForm(array &$form, FormStateInterface $form_state) {
    $em = \Drupal::entityTypeManager();
    $values = $form_state->getValues();
    foreach ($values['files'] as $sid => $data) {
      $em->getStorage('webform_submission')
        ->load($sid)
        ->setElementData('attached_files', $data['file'])
        ->save();
    }
  }

}
</code>
Answer: This issue replicable for anonymous users only, kind of caching issue
so if you add following cache API function code in custom form , then latest submission data 'll be load that 'll be added/removed by user while interacting with custom form.
<code> $form['#cache'] = [
   'contexts' => ['session']
 ];
</code>
Comment: That's fixed it perfectly thank you!  Is there any specific documention you derived the answer from I could look at, or was it just experience you had?
Comment: Well there is not any documentation for it , I used debugging skills to replicate and trace issue & found the root cause of issue & got the solution ;)
Comment: You did a lot better than me! I debugged the webform post save file handler, saw that it was calling the file delete but never actually deleting the file or reference and would never of thought of the cache.
Comment: no worries, mine pleasure to contribute to the community :)


Complexity - why is RL=NL when omitting the demand for polynomial run-time?

Question: The complexity class RL is described at the complexity zoo as: Has the same relation to L as RP does to P. The randomized machine must halt with probability 1 on any input. It must also run in polynomial time (since otherwise we would just get NL).
The question is - how can we get NL when omitting the demand for polynomial time? I have a solution of my own but it seems strange to me.
My solution: If suffices to solve ST-CON. We can use the same NL algorithm for ST-CON (guessing a path) with one major difference - we count the number of steps we make, and if it suppresses the number of vertices in the graph, we restart the computation, without remembering ANYTHING.
This means we can play this game indefinitely. If the graph is not ST-connected, then we'll never halt, but if it's ST-connected we'll halt in probability 1 (this is the same as saying that a geometric random variable obtains a finite value in probability 1). However, since we do not halt for NO-instances, this solution "feels wrong" to me.
Is there another solution? And is my solution correct?
Comment: Added some tags. However, it is my belief that theoretical computer science IS mathematics...
Comment: @Qiaochu: My guess is that this will be closed there. That site is akin to MathOverflow for CS. This looks like a homework problem to me. btw, Gadi, did you check the wiki for NL?
Comment: @Gadi: having the only tag be computer-science makes it look like this question is not appropriate for a math-based Q&A site.  Could you please remedy this by adding a "mathier" tag?
Comment: @Gadi A: regardless of your beliefs, the fact remains that there are probably more people qualified to answer this question at http://cstheory.stackexchange.com/ than here.
Comment: @Gadi, btw, your proof looks right to me, though I am not sure (rusty regarding the definitions). I believe there is no condition that a NDTM should halt on every computation path.
Comment: Indeed this is a homework-level problem (it's taken from homework in a course I will TA...), and indeed I think that it's not suited for http://cstheory.stackexchange.com but I also guess (from the lack of response) that this is not the place either.
Comment: @Gadi: You could try your luck at stackoverflow.com.
Answer: Your solution is almost correct.
In the definition of RL, we require that the algorithm halt with probability $1$ on any input. As you note, your algorithm runs forever on "No" instances. To fix this, we'll take advantage of the fact that we're allowed a small chance of error. In particular, we're allowed to sometimes reject "Yes" instances, as long as we never accept a "No" instance.
The basic idea is to keep a counter of how many iterations we've used so far. If a path exists, we should find it within $2N^N$ iterations (with probability at least $3/4$). So if we've gone that long without finding a valid path, we'll reject the graph. Note that we'll never accept a bad graph this way, and we'll rarely reject a good graph.
We can't naively count to $N^N$, however, since this would require $N\lg N$ space. Instead, we'll probabilistically "count" that high.
After each iteration, we'll flip $N\lg N + 2$ coins. If every coin lands heads, then we stop (and reject the graph). If at least one coin lands tails, then we begin another iteration. This can be done in log-space, since we only need to count how many flips have occurred.
The probability of getting all heads is $1/2^{N\lg N + 2}=1/4N^N$, so with probability at least $3/4$, this won't happen until at least $2N^N$ iterations have elapsed.
Taking a union bound over both ways the algorithm could fail, we see that everything works with probability at least $1/2$.
Finally, note that this halts with probability $1$, since we'll flip all heads eventually.


Do I need a global test set when using k-fold cross-validation and a small n (n<<p)?

Question: In most of the IS (Information Systems) papers I read, the authors do not use a global test set, but only cross-validation and present these results. They reason this makes sense because they have a very small n, e.g. in Text Classification (n=400,p=3000). Is this approach correct?
This question was partly asked already here, but less specific: Do we need a test set when using k-fold cross-validation?
Comment: Exactly. While they do not necessarily tune hyper parameters, they use in every case the statistical performance results on the validation set to choose a model with standard parameters (e.g. SVM vs Random Forest vs ANN) which will then be deployed within a predictive information system.
Comment: Are they using the same cross validation loop to *both* choose algorithm hyperparameters *and* report an estimate of hold out error?  If that's the case, then yes, that is mistaken.
Answer: I'd just like to add to @cbeleites' answer, which is good, with some quotes from a recent blog post of Frank Harrell's on split-sample validation:

[...] data splitting is an unstable method for validating models or classifiers, especially when the number of subjects is less than about 20,000 (fewer if signal:noise ratio is high). This is because were you to split the data again, develop a new model on the training sample, and test it on the holdout sample, the results are likely to vary significantly.

And

Data are too precious to not be used in model development/parameter estimation.  Resampling methods allow the data to be used for both development and validation, and they do a good job in estimating the likely future performance of a model. Data splitting only has an advantage when the test sample is held by another researcher to ensure that the validation is unbiased.

In other words, in most cases you should prefer a resampling-based validation approach, and especially so when you have few observations. In a $n << p$ situation, the modelling process can have super high variance even if you use high-bias methods such penalized regression. If you don't do resampling you might spend a lot of time trying to interpret noise (speaking from experience).

Edit: so what's resampling-based validation anyway?
There are a few ways to do validation by resampling, the common idea is to make new samples from your original sample. A popular and by now quite well-understood method is to use the bootstrap, which would go like this:

Make a bootstrap sample by drawing $n$ observations from your original $n$-sized sample with replacement
Since you drew the bootstrap sample with replacement, some observations will appear more than once, others won't be in there at all. Roughly about 60% of the observations will be in there, the other 40% won't.
Fit your model to the bootstrap sample
Estimate your validation statistic by predicting on the observations that aren't in the bootstrap sample
Repeat for as long as you have the patience to

This gives you an empirical distribution over the validation statistic which you can use to say something about how well your method will generalize. This could be for eg to estimate uncertainty or to calculate a point estimate (preferably both!).
Other possible methods: repeated holdout set, repeated cross-validation.
Comment: Thank you for your answer. Please specify "resampling-based validation"
Comment: @NKu I have added a brief explanation to my answer above
Answer: There are 2 separate issues here:

"single" vs. nested validation, and
cross validation vs. hold out test set

Single vs. nested
The point is that you have to decide what to use the validation results for:

either model selection (including hyperparameter tuning, deciding training algorithm, etc.)
(x)or as generalization error estimate

So regardless of sample size, if data-driven steps are applied for deciding the final model, generalization error can only be estimated on a data set that was not involved at all in these decision and training procedures. In other words, a nested validation is needed.
Hold out vs. cross validation
If sample size is too small for single (hold out) splits, both levels of validation can be done by cross validation (aka nested cross validation).

do not use a global test set, but only cross-validation

Note that the hold-out test sets I've seen in practice are typically generated by the same random splitting as cross validation sets. So both hold out and cross validation have the same risk of possible confounders violating independence.
Thus, hold out tests are not per se superior than cross validation.
Once you go ahead and plan a validation study covering all the possible confounders you can think of, things look different: it is then typically more efficient (in terms of total no of cases to collect) to obtain a single well-defined independent test set.


Is my time series predictable?

Question: I am new to time series and would appreciate help in this matter.
I have a time series with the following graph as a result of applying the plot_acf in Python. Given this, can it be inferred that this series can be predictable via Machine Learning algorithms? I have already tried LSTM and N-HiTS to predict and the MAE is very large compared to the MAE using a baseline algorithm (the baseline MAE is 20% of the MAE using the previously mentioned algorithms)
I am adding the data that originated this ACF plot, which corresponds to actual sales figures:

324
281
691
281
410
346
86
43
389
43
194
22
0
130
65
173
86
281
0
0
65
43
86
65
0
0
130
65
130
173
151
43
0
43
173
108
173
130
65
0
0
0
86
86
0
22
22
22
22
22
43
43
22
65
130
130
-86
22
173
43
86
43
86
22
65
0
43
43
0
194
0
0
108
108
130
86
86
22
43
86
22
0
0
43
86
22
86
22
0
108
0
65
108
22
86
43
65
43
0
108
86
0
22
65
151
0
43
86
151
43
43
22
108
0
108
0
43
65
43
108
86
0
151
22
0
108
65
65
22
0
43
22
22
65
43
130
151
108
0
130
151
130
65
0
130
43
0
0
22
43
0
65
108
22
65
0
130
86
86
281
216
22
173
108
173
302
410
86
65
65
22
86
22
194
86
130
86
216
108
173
432
238
151
194
194
324
22
367
324
238
367
410
216
497
259
108
281
281
216
108
259
216
130
65
173
86
65
43
43
86
130
194
108
194
238
108
22
43
65
173
86
151
151
130
22
151
86
281
86
259
65
86
173
65
259
173
108
238
130
151
259
259
151
389
65
259
173
238
108
43
65
173
65
65
216
151
302
86
259
130
86
151
65
238
43
86
130
65
130
259
22
432
173
216
108
130
410
324
86
475
130
410
86
216
151
173
85
302
173
259
281
281
86
173
216
302
22
108
43
108
389
245
43
43
216
22
151
302
259
194
346
43
302
194
151
194
173
43
43
281
367
43
367
43
0
65
86
151
108
281
130
22
86
389
324
216
346
43
216
43
130
497
65
22
43
238
324
0
65
43
86
216
238
43
108
194
108
605
302
130
281
86
86
302
173
216
194
86
259
151
389
194
454
65
130
518
216
43
302
22
151
216
65
302
389
281
410
259
238
151
216
389
194
216
130
194
108
65
475
194
216
43
86
86
130
410
324
173
65
86
151
324
475
281
454
130
259
389
216
86
216
194
130
454
194
108
108
43
86
65
238
65
108
86
65
22
108
216
65
65
346
281
86
302
151
324
194
151
151
108
43
0
0
65
324
281
151
0
0
389
302
432
540
108
346
0
194
238
259
130
238
389
238
518
86
281
0
0
108
43
0
0
0
65
0
0
0
0
0
0
108
194
0
216
238
22
43
259
216
22
410
238
86
151
281
259
151
173
432
43
238
65
389
43
173
324
130
367
194
86
65
65
302
367
367
130
216
454
65
0
432
108
0
216
194
108
281
259
108
259
173
238
367
324
259
324
346
346
194
259
151
151
238
65
346
130
562
43
108
173
43
151
65
151
86
43
194
86
22
0
0
65
43
151
43
108
86
0
130
65
65
108
65
22
0
86
0
130
43
108
86
43
108
238
259
324
497
324
346
389
259
324
151
302
389
216
281
216
324
216
194
259
43
108
302
151
22
389
454
194
324
734
0
108
238
22
0
432
108
43
0
22
43
346
43
65
130
86
302
238
475
216
389
281
497
302
65
475
432
259
173
108
216
43
65
173
238
108
86
173
43
173
518
151
281
238
216
151
238
86
173
108
43
108
65
0
259
216
216
22
216
281
259
238
130
86
238
216
173
216
346
648
216
43
302
302
108
259
216
216
194
173
86
194
130
259
194
86
389
65
410

Do you recommend using other methods to determine the predictability of the time series?
Many thanks for the help provided.
Comment: You have given us an ACF plot, but neither the original series, nor any information about what the series measures.
Comment: @StephanKolassa thank you for your quick feedback. I have added the information you mentioned. I hope it helps.
Comment: Are these daily, weekly, monthly sales, or on some other kind of granularity? Do you only have this single time series, or is this just one of many?
Comment: @StephanKolassa These are daily sales figures, which only happen on weekdays. On weekdays with no sales, zero appears. I only have this time series. If necessary I can try to obtain more datapoints, but this will take quite some effort with associated time. Thank you once again
Comment: OK, thanks. Are weekdays for you Monday through Friday, or Monday through Saturday?
Comment: @StephanKolassa, the weekdays are Monday through Friday
Comment: @StephanKolassa Thank you for your answer. I will read the links you provided. In the meanwhile, let me clarify that the low values in the beginning of the series, are due to the COVID period and are clearly understood.
Comment: Very good. Then you might be able to improve your forecasts by either removing these data points (i.e., starting the time series later), or using a predictor and running a regression with time series errors.
Comment: @StephanKolassa Thanks for all your help
Comment: In the name of readability, it would be great if you can edit your question to include the data in a more succinct format and at the end of the question. Perhaps something similar to how StephanKolassa included the data in his answer?
Answer: Your time series does not show any intra-week seasonality (see the seasonplot at the right below). Exponential Smoothing (as per ets() in the forecast package for R) chooses a model with no seasonality, additive errors and an additive trend with strong dampening, so the forecasts are almost flat. Since there is very little time series structure in your data, the prediction intervals are very wide (and actually go below zero, which probably does not make a lot of sense for your data).
Your series does show a number of prolonged periods of low sales. It may be worthwhile figuring out what happened during these times. I believe this has a better chance of improving your forecasts than tweaking the models: How to know that your machine learning problem is hopeless?.
In any case, you have far too little data for a ML method like LSTM to make much of a difference compared to standard time series methods. I recommend the resources here to learn more about forecasting: Resources/books for project on forecasting models

R code:
sales <- c(324L, 281L, 691L, 281L, 410L, 346L, 86L, 43L, 389L, 43L, 194L,
22L, 0L, 130L, 65L, 173L, 86L, 281L, 0L, 0L, 65L, 43L, 86L, 65L,
0L, 0L, 130L, 65L, 130L, 173L, 151L, 43L, 0L, 43L, 173L, 108L,
173L, 130L, 65L, 0L, 0L, 0L, 86L, 86L, 0L, 22L, 22L, 22L, 22L,
22L, 43L, 43L, 22L, 65L, 130L, 130L, -86L, 22L, 173L, 43L, 86L,
43L, 86L, 22L, 65L, 0L, 43L, 43L, 0L, 194L, 0L, 0L, 108L, 108L,
130L, 86L, 86L, 22L, 43L, 86L, 22L, 0L, 0L, 43L, 86L, 22L, 86L,
22L, 0L, 108L, 0L, 65L, 108L, 22L, 86L, 43L, 65L, 43L, 0L, 108L,
86L, 0L, 22L, 65L, 151L, 0L, 43L, 86L, 151L, 43L, 43L, 22L, 108L,
0L, 108L, 0L, 43L, 65L, 43L, 108L, 86L, 0L, 151L, 22L, 0L, 108L,
65L, 65L, 22L, 0L, 43L, 22L, 22L, 65L, 43L, 130L, 151L, 108L,
0L, 130L, 151L, 130L, 65L, 0L, 130L, 43L, 0L, 0L, 22L, 43L, 0L,
65L, 108L, 22L, 65L, 0L, 130L, 86L, 86L, 281L, 216L, 22L, 173L,
108L, 173L, 302L, 410L, 86L, 65L, 65L, 22L, 86L, 22L, 194L, 86L,
130L, 86L, 216L, 108L, 173L, 432L, 238L, 151L, 194L, 194L, 324L,
22L, 367L, 324L, 238L, 367L, 410L, 216L, 497L, 259L, 108L, 281L,
281L, 216L, 108L, 259L, 216L, 130L, 65L, 173L, 86L, 65L, 43L,
43L, 86L, 130L, 194L, 108L, 194L, 238L, 108L, 22L, 43L, 65L,
173L, 86L, 151L, 151L, 130L, 22L, 151L, 86L, 281L, 86L, 259L,
65L, 86L, 173L, 65L, 259L, 173L, 108L, 238L, 130L, 151L, 259L,
259L, 151L, 389L, 65L, 259L, 173L, 238L, 108L, 43L, 65L, 173L,
65L, 65L, 216L, 151L, 302L, 86L, 259L, 130L, 86L, 151L, 65L,
238L, 43L, 86L, 130L, 65L, 130L, 259L, 22L, 432L, 173L, 216L,
108L, 130L, 410L, 324L, 86L, 475L, 130L, 410L, 86L, 216L, 151L,
173L, 85L, 302L, 173L, 259L, 281L, 281L, 86L, 173L, 216L, 302L,
22L, 108L, 43L, 108L, 389L, 245L, 43L, 43L, 216L, 22L, 151L,
302L, 259L, 194L, 346L, 43L, 302L, 194L, 151L, 194L, 173L, 43L,
43L, 281L, 367L, 43L, 367L, 43L, 0L, 65L, 86L, 151L, 108L, 281L,
130L, 22L, 86L, 389L, 324L, 216L, 346L, 43L, 216L, 43L, 130L,
497L, 65L, 22L, 43L, 238L, 324L, 0L, 65L, 43L, 86L, 216L, 238L,
43L, 108L, 194L, 108L, 605L, 302L, 130L, 281L, 86L, 86L, 302L,
173L, 216L, 194L, 86L, 259L, 151L, 389L, 194L, 454L, 65L, 130L,
518L, 216L, 43L, 302L, 22L, 151L, 216L, 65L, 302L, 389L, 281L,
410L, 259L, 238L, 151L, 216L, 389L, 194L, 216L, 130L, 194L, 108L,
65L, 475L, 194L, 216L, 43L, 86L, 86L, 130L, 410L, 324L, 173L,
65L, 86L, 151L, 324L, 475L, 281L, 454L, 130L, 259L, 389L, 216L,
86L, 216L, 194L, 130L, 454L, 194L, 108L, 108L, 43L, 86L, 65L,
238L, 65L, 108L, 86L, 65L, 22L, 108L, 216L, 65L, 65L, 346L, 281L,
86L, 302L, 151L, 324L, 194L, 151L, 151L, 108L, 43L, 0L, 0L, 65L,
324L, 281L, 151L, 0L, 0L, 389L, 302L, 432L, 540L, 108L, 346L,
0L, 194L, 238L, 259L, 130L, 238L, 389L, 238L, 518L, 86L, 281L,
0L, 0L, 108L, 43L, 0L, 0L, 0L, 65L, 0L, 0L, 0L, 0L, 0L, 0L, 108L,
194L, 0L, 216L, 238L, 22L, 43L, 259L, 216L, 22L, 410L, 238L,
86L, 151L, 281L, 259L, 151L, 173L, 432L, 43L, 238L, 65L, 389L,
43L, 173L, 324L, 130L, 367L, 194L, 86L, 65L, 65L, 302L, 367L,
367L, 130L, 216L, 454L, 65L, 0L, 432L, 108L, 0L, 216L, 194L,
108L, 281L, 259L, 108L, 259L, 173L, 238L, 367L, 324L, 259L, 324L,
346L, 346L, 194L, 259L, 151L, 151L, 238L, 65L, 346L, 130L, 562L,
43L, 108L, 173L, 43L, 151L, 65L, 151L, 86L, 43L, 194L, 86L, 22L,
0L, 0L, 65L, 43L, 151L, 43L, 108L, 86L, 0L, 130L, 65L, 65L, 108L,
65L, 22L, 0L, 86L, 0L, 130L, 43L, 108L, 86L, 43L, 108L, 238L,
259L, 324L, 497L, 324L, 346L, 389L, 259L, 324L, 151L, 302L, 389L,
216L, 281L, 216L, 324L, 216L, 194L, 259L, 43L, 108L, 302L, 151L,
22L, 389L, 454L, 194L, 324L, 734L, 0L, 108L, 238L, 22L, 0L, 432L,
108L, 43L, 0L, 22L, 43L, 346L, 43L, 65L, 130L, 86L, 302L, 238L,
475L, 216L, 389L, 281L, 497L, 302L, 65L, 475L, 432L, 259L, 173L,
108L, 216L, 43L, 65L, 173L, 238L, 108L, 86L, 173L, 43L, 173L,
518L, 151L, 281L, 238L, 216L, 151L, 238L, 86L, 173L, 108L, 43L,
108L, 65L, 0L, 259L, 216L, 216L, 22L, 216L, 281L, 259L, 238L,
130L, 86L, 238L, 216L, 173L, 216L, 346L, 648L, 216L, 43L, 302L,
302L, 108L, 259L, 216L, 216L, 194L, 173L, 86L, 194L, 130L, 259L,
194L, 86L, 389L, 65L, 410L)

sales_ts <- ts(sales,frequency=5)

library(forecast)

par(mfrow=c(1,2),las=1)
model <- ets(sales_ts)
plot(forecast(model,h=20))
seasonplot(ts(sales,frequency=5),pch=19)

Is my time series predictable?


Read from large MongoDB, write to JSON

Question: I have a MongoDB with about 70gb of data in it. I need to download the entire DB into one JSON document, preferably with Python. However, when I try and do json.dump(object, file_object) a couple million times, I tend to unintentionally create a malformed JSON. Storing the entire DB in memory isn't an option, because, well, 70gb > 16gb of RAM. How can I write JSON documents, one at a time, to the same file object in Python?
Thanks!
Answer: Use mongoexport to export the data:
mongoexport --db test --collection traffic --out traffic.json

Read from large MongoDB, write to JSON


Send C struct over TCP socket

Question: I have seen a lot of answers that are related to my question, but I really cannot use it under my condition.
I am building a network module using socket programming with C language under Linux domain.
I have to implement a function that could send a struct that consists of int, char, char * and some other structures (nested struct).
struct EnQuery
{
   char  type[6]; // insert, select , update , delete
   char * columns; //note here, it's an array of big, {name, age, sex, position, email}  not in string
   struct Values * values; //an array of values(another struct), {tom, 23, male, student, email@example.com}  is represented by the second struct values, not in string
   struct condition * enCondition; //  array of condition,  "name=tom and age>30" is represnted by the third struct condition
   short len;
   short rn;
};

struct Values
{
    char * doc;
    char  key[2];
    char  s;
};

struct condition
{
     short k;
     struct condition * children;
};

Above is the structure that I am trying to send.
I am declaring the variables and sending over through socket using send() function.
How would I have to send char * through the socket?
Or would there be a way to resize the char array length conveniently?
P.S I can't use external libraries
Comment: Don't use structs as network protocols. Use network protocols as network protocols. First define the protocol in octets, then write yourself an API to send and receive it.
Comment: Your strcts has pointers, and that cannot be transmitted as they will not be valid at the receiving end -- you need to serialize and deserialize the data -- how you do that has a million good answers.
Comment: Everything with a star (`*`) needs to be serialized, not only the `char`s.   If you didn't have any pointers you could just send the struct as is (assuming receiver have the same endianness and memory padding etc).
Comment: Just decide on the wire format you want to use and write code to convert to and from that wire format. Look at existing specification (BER, XML, JSON, etcetera) for ideas.
Comment: I cant use any external libraries..
Comment: Why can you not use external libraries?
Comment: @Soren So do you mean that i would have to change char * to char[] and do serialization before sending over? If that is the case, I have to make the char array length variable and how would I have to to this?
Comment: @Soren Its for my project and the supervisor does not want any external libraries used (due to speed of the program)
Comment: @YongHeeLee You seem to be confusing the issue of what the network protocol should be with how the network protocol is implemented. You need to decide what the protocol will be before you work on the details of implementing it.
Comment: @David decide protocol? as in?
Comment: @YongHeeLee Have you ever read the specification for a network protocol layered on top of TCP such as IRC, SMTP, or HTTP? They specify, at the byte level, exactly how information is exchanged.
Comment: Probably working with FTP, as I have to transfer files and struct
Answer:
How would I have to send char * through the socket?

Obviously, sending the pointer value isn't a useful thing to do, since the pointer wouldn't point to anything valid on the receiving computer.
So instead of sending the pointer, you have to send the data that it points to.  A typical method for doing that would be to first send the number of bytes the pointer points to:
uint32_t sLen = strlen(doc)+1;  // +1 because I want to send the NUL byte also
uint32_t bigEndianSLen = htonl(sLen);
if (send(sock, &bigEndianSLen, sizeof(bigEndianSLen), 0) != sizeof(bigEndianSLen)) perror("send(1)");

.... and then follow by sending the bytes of the string:
if (send(sock, doc, sLen, 0) != sLen) perror("send(2)");

On the receiving side, you'd do the opposite:  first receive the string's length:
uint32_t bigEndianSLen;
if (recv(sock, &bigEndianSLen, sizeof(bigEndianSLen), 0) != sizeof(bigEndianSLen)) perror("recv(1)");
uint32_t sLen = ntohl(bigEndianSLen);

... and then receive the string's data:
char * doc = malloc(sLen+1);
if (recv(sock, doc, sLen, 0) != sLen) perror("recv(2)");
doc[sLen] = '\0';  // paranoia:  make sure the string is terminated no matter what

Note that this example code is a bit naive, in that it doesn't handle correctly the case where send() or recv() return a value other than the byte-count that was passed in to them.  Production-quality code would handle errors properly (e.g. by closing the connection), and would also handle properly the case where send() or recv() send/receive only transferred some of the bytes that were requested (by calling send()/recv() again later for the remaining unsent/unreceived bytes).
Comment: Would this work on char as well? How would I have to cast this ?
Comment: char is not a pointer, so you can send a char directly, similarly to how you send an int (actually even easier than an int, since a char is only 1 byte you do not need to worry about endian-conversion issues for a char).  E.g. char s = 'J'; send(sock, &s, 1, 0);
Comment: would you give me an example like above for char as well? Thank you in advance
Comment: I am trying to implement this, but since you have only specified an example with only one element, how would I add another? let say, send key[2] followed by doc? A specific example would be much appreciated and would contribute in a lot of help. Thank you very much
Comment: I am having trouble sending two elements at the moment.. If you could please help me with this, I would be appreciated
Comment: i still can not understand why converting from bigEndian to litleEndian and vis-versa? network has nothing to do with `sLen`. it is application specific variable.
Comment: @milevyo if the sending program is going to send the string's length across the network as a binary integer, it is important that the receiving program interpret the integer's bytes correctly.  One easy way to ensure that is to have the sending program convert the value to an architecture-invariant format (e.g. big-endian, via htonl()) before sending it, and have the receiving computer convert it back to its own cpu-native format (e.g. via ntohl()) after receiving it.
Comment: Note that htonl() and ntohl() only convert to/from the CPU-native format to big endian if the program was compiled for a little-endian computer.  On a big-endian computer they are both no-ops, since the CPU's native integer format is already big-endian and therefore no conversion is needed.
Comment: @JeremyFriesner, of course, you are right. i forget about ppc and variants
Answer: You need to make a choice : either serialize the data structure in memory on the sending side using a "pack" function, send across the wire and deserialize using an unpack function (straightforward), or dynamically serialize/deserialize straight to the socket (better done afterwards as an optimization of the first version).
You can get some inspiration from this open source code for example:
https://code.google.com/p/protobuf-c/
Comment: would you be able to make some example with the specified struct? Thanks in advance
Comment: How about trying to write a very simple version of pack that copies only the fixed length items into a char[], and then extend that with code to copy an array of condition.  Once you can do that you can use similar code to add a char* to the end of the char[], so now you can handle a Values struct.  then, extend that code to handle an array of Values, and finally you have all the tools to do an EnQuery struct.  By starting small, and debugging as you go, you can figure out how to do the big task.
Comment: Serializing directly to the socket is hardly an optimization -- it makes performance much worse.
Comment: This dates me unfortunately.  The last time I wrote directly to a socket was when 10 Base-T was first coming on line and wasn't yet prevalent.
Answer: define an enum with "type numbers".  Using your example:

Tchar
Tstring
Tshort
Tint
TEnQuery
TValues
Tcondition
Tpop
Define a few flag bits that get or'ed with the type:Tarray -- array of base type
Now output the type number.  Follow it by the actual data.  For example, to output a Values struct:Tvalues
Tstring,data for doc (inc EOS)
Tchar | Tarray,2,key data
Tchar,s value
Tpop  Any aggregate (e.g. struct) does an implied "Tpush" that is paired with the Tpop
Now you have a byte stream that you process recursively with a state machine of sorts.  You'll need recursive functions for export/import for all the above types.  On import, the key thing is that type number is what you dispatch on
Answer: A struct is a data type available with the (ANSI) C language. I believe you are looking for the following functions to perform sending of the data within your struct.

socket()
read()
write()

Keep in mind these are the bare minimum requirements for IPC (also referred to as signals), TCP/IP and/or UDP communications.
For a simple implementation of a client/server architecture (although thoroughly outdated these days) reference; client / server. You might serve better to focus your studies on peer to peer communication providing you with a more robust set of fail overs as each peer holds connection information for each other; see here:

Send C struct over TCP socket


Determinant of matrix exponential?

Question: Suppose $A$ is a $n \times n$ constant matrix. How can I prove
$\det(e^A) = e^{\displaystyle \sum_{\lambda_i\in\sigma(A)} \lambda_i}$,
where $\sigma(A)$ is the multiset of eigenvalues of $A$?
The following matlab code shows this is true in $n = 3$ case:
<code>A = rand(3)
detA = exp(sum(eig(A)))
detmA = det(expm(A))
</code>
Comment: prove it for diagonal matrices and use denseness of diagonalizable matrices. notice that the trace of $A$ is the sum of the eigenvalues
Answer: If you're afraid of the density of diagonalizable matrices, simply triangularize $A$. You get $$A=P^{-1}UP,$$ with $U$ upper triangular and the eigenvalues $\{\lambda_j\}$ of $A$ on the diagonal.
Then
$$
\mbox{det}\;e^A=\mbox{det}(P^{-1}e^UP)=\mbox{det}\;e^U.
$$
Now observe that $e^U$ is upper triangular with $\{e^{\lambda_j}\}$ on the diagonal.
So
$$
\mbox{det} \;e^A=\mbox{det} \;e^U=e^{\lambda_1}\cdots e^{\lambda_n}=e^{\lambda_1+\ldots+\lambda_n}.
$$
Comment: Anyone trying to understand this solution: The main idea is that if the eigenvalues of $A$ are $\lambda_i$, then the eigenvalues of $e^A$ are $e^{\lambda_i}$, which you can see by using the series definition of $e^A$ and multiplying nonzero eigenvector $v_i$ as in $e^A v_i = \lambda_i v_i$. Distribute the $v_i$ through the series. Finally, the determinant of a matrix is the product of the eigenvalues, and the trace of a matrix is the sum of the eigenvalues. This explains the second to last equality in the proof above.
Comment: Your comment was super useful
Answer: You  just use the Jordan normal form, $A$ is your matrix, $D$ is the Jordan normal form of $A$ and $S$ is the transformation matrix, for the second equal just remember the definition of matrix exponential, and that
$$(SAS^{-1})^n = S A  S^{-1}S A S^{-1} S A \dots S A S^{-1}=S A^n S^{-1}$$
\begin{align*}
\det(\exp(A))&=\det(\exp(S D S^{-1}))\\
&=\det(S \exp(D) S^{-1})\\
&=\det(S) \det(\exp(D)) \det (S^{-1})\\
&=\det(\exp (D))\\
&=\prod_{i=1}^n e^{d_{ii}}\\
&=e^{\sum_{i=1}^n{d_{ii}}}\\
&=e^{\text{tr}D}
\end{align*}
As the trace is invariant this works.
Answer: For an analytic method, using differential equations:
Let $f(t)= \det(e^{tA})$. Then $f'(t)=D \det(e^{tA}) \cdot Ae^{tA}=\text{tr} \left(^t \text{com}(e^{tA})Ae^{tA} \right)$. But $A$ and $e^{tA}$ commute, and $^t\text{com}(e^{tA})e^{tA}=\det(e^{tA}) \operatorname{I}_n$. Therefore, $f'(t)=\text{tr}(A)f(t)$ and $f(0)=1$, hence $f(t)=e^{\text{tr}(A)t}$. For $t=1$, $\det(e^{A})= e^{\text{tr}(A)}$.
Comment: Hello from 2016! Could you please give a reference to this proof?
Comment: $\mathrm{com}(A)$ is the matrix of cofactors associated to $A$. Possibly, it is a French notation, where $\mathrm{com}$ stands for *comatrice*.
Comment: What kind of commutator is $^t\text{com}(e^{tA})$? Never seen this notation.
Answer: You can reduce problem for diagonal matrix using the fact that every matrix can be approximated with any given precision and both functions: $e^X$ and $\det X$ are continuous. The problem with diagonal matrix is obvious.
P.S. It seems that lyj was faster than me.

Determinant of matrix exponential?


Alinhar duas divs verticalmente depois de usar float em uma delas

Question: Estou criando um rodapé para uma página html. Ele é bem simples e composto por uma ÚNICA linha. Gostaria de que uma parte, que contém as informações de Copy, ficasse alinhada à margem esquerda e outra, que contém um menu, ficasse alinhada à margem direita do site.
Separei o texto em duas divs e dei um float: right na segunda. Acontece que não estou conseguindo alinhar as duas partes verticalmente de modo que pareçam estar na mesma linha.
Seguem os códigos:
HTML

Copyright © 2017. Todos os direitos reservados.

Documentos Legais

Início

Blog

Facebook

Twitter

Linkedin

Segurança

Privacidade

Termos de uso

CSS
section#rodape-legal {
margin: auto;
padding-top: -20px;
text-align: left;
}

div.rodape-legal {
display: inline-block;
}

div#rodape-legal-menu {
text-align: right;
display: inline-block;
float: right;
}

nav.doc-legais li {
display: inline-block;
margin-left: 10px;
}

nav.doc-legais h1 {
display: none;
}

nav.doc-legais a {
transition: color, 1s;
text-decoration: none;
color: black
}

footer h6 {
font-size: 12px;
font-weight: 300;
}

Answer: Você precisa retirar a margin da
 dentro da div da direita:
div#rodape-legal-menu ul {
   margin: 0;
}

Essa margin nativa da
 está criando um espaço vertical na div.
Precisa também definir um line-height do mesmo tamanho da fonte, para ajustar a linha:
nav.doc-legais {
   line-height: 12px;
}

Execute o snippet abaixo em tela cheia para ver o resultado:

section#rodape-legal {
margin: auto;
padding-top: -20px;
text-align: left;
}

div.rodape-legal {
display: inline-block;
}

nav.doc-legais {
   line-height: 12px;
}

div#rodape-legal-menu ul {
   margin: 0;
}

div#rodape-legal-menu {
text-align: right;
display: inline-block;
float: right;
}

nav.doc-legais li {
display: inline-block;
margin-left: 10px;
}

nav.doc-legais h1 {
display: none;
}

nav.doc-legais a {
transition: color, 1s;
text-decoration: none;
color: black
}

footer h6 {
font-size: 12px;
font-weight: 300;
}

Copyright © 2017. Todos os direitos reservados.

Documentos Legais

Início

Blog

Facebook

Twitter

Linkedin

Segurança

Privacidade

Termos de uso

EDIT
Para ficar responsivo, eu sugeria definir width: 50%; em cada div e mais outros ajustes:

section#rodape-legal {
margin: auto;
padding-top: -20px;
text-align: left;
display: block;
}

div.rodape-legal {
display: inline-block;
width: 50%;
}

div.rodape-legal ul,
div.rodape-legal li,
div.rodape-legal h6 {
   margin: 0;
}

div#rodape-legal-menu {
text-align: right;
display: inline-block;
float: right;
}

nav.doc-legais li {
display: inline-block;
margin-left: 10px;
}

nav.doc-legais {
   line-height: 12px;
}

nav.doc-legais h1 {
display: none;
}

nav.doc-legais a {
transition: color, 1s;
text-decoration: none;
color: black
}

footer h6 {
font-size: 12px;
font-weight: 300;
}

Copyright © 2017. Todos os direitos reservados.

Documentos Legais

Início

Blog

Facebook

Twitter

Linkedin

Segurança

Privacidade

Termos de uso

Answer: De margin: 0 na tag ul. As ul vem por padrão nos navegadores com margin, assim como as tags mais utilizadas p e h. E também com a chegada do -webkit está vindo com -webkit-padding-start, que da padding da esquerda para a direita, como se fosse um padding-left.


RealmSwift: Unexpected non-void return value in void function

Question: I'm trying to query for the token and user_id from my realm database and then pass to my private var token and userId but I keep getting thrown this error

Unexpected non-void return value in void function

at return particular.id and

Cannot assign value of type '()' to type 'Int!'

at _userId = getId()
This is my code:
import Foundation
import RealmSwift

class Users {
  private var _userId: Int!
  private var _token: String!

var userId: Int {
    _userId = getId()
    return _userId
}

func getId () {

    guard let userName = Data.sharedInstance.userName  else { print("Empty username"); return }

    do {
        let realm = try Realm()
        if let particular = realm.objects(Particulars).filter("username == '\(userName)'").first {
            return particular.id
        }

    } catch let error as NSError {
        // handle error
        print(error)
    }

}
}

So there are two things that I want to know:

Why am I being thrown this error?
Is there a better way of doing so? (I.e. more efficient way of querying)
Answer: Your function is defined as func getId () which means it doesn't return a value. Then you try and return something at return particular.id. You need to change your function definition to func getId() -> Int? and then also add a return nil as the last line for when your fetch fails (same for your guard statement).
Of course, you'll also have to handle that nil value when calling getId() in the var userId, or change userId to be optional as well.


Weak compactness in measures implies compactness in the underlying metric space via the Dirac's delta

Question: Suppose that $(M,d)$ is a separable and complete metric space. Define $C_b(M)$ to be the space of all bounded continuous functions $f\colon M\to\mathbb R$ and denote by $\mathcal P(M)$ the space of all Borel probability measures on $M$.
Endow $\mathcal P(M)$ with the weak topology, that is, the coarsest topology that makes $\mu\mapsto \int_M f d\mu$ continuous for all $f\in C_b(M)$.
Suppose that $K\subset \mathcal P(E)$ is relatively weakly compact.
Suppose that $L\subset M$ satisfies that $\delta_x\in K$ for all $x\in L$, where $\delta_x$ denotes the Dirac's delta.
My question is: Is it true that $L$ is relatively compact in $(M,d)$?
Comment: I suggest you use https://en.m.wikipedia.org/wiki/Prokhorov's_theorem
Answer: YES. Convergence of $\delta_{x_n}$ to some $\mu$ implies that $\mu$ is degenerate. (This can be proved by decomposing $M$ into countable many sets of arbitraily small diameter).  From this it follows that if $(x_n)$ is a sequence in $L$ then $\delta_{x_n}$ has a subsequence converging to  $\delta_x$ for  some $x$ which proves that $L$ is relatively compact.


Box-Cox transformation in R

Question: I am relatively new to R and I am interested in performing a Box-Cox transformation. However, I am a little lost as to the step-by-step process of doing this. I have been searching the web for a number of days and have tried various code to no avail.
Overall, within my dataset (data=fw) I want to test for differences in a dependent variable (x) between between two locations (north and south). However, my data is not normally distributed and has unequal variance. Therefore, I need to transform my data (log and square root have not worked). Is the 'MASS' package the only package that I need to use to perform the Box-Cox transformation, or do I need some others? Any advice on the step-by-step process on Box-Cox transformation of the data would be much appreciated.
Comment: Your data need NOT necessarily  be normal to apply a model. Only your **residual** from the model should be normal. even if your residual are non  normal, This could be caused by outliers and other anomalies. Even after correcting for outliers if your residuals are non normal then you could use an appropriate box-cox transformation.
Comment: Read the manual `?boxcox`, & see [Faraway (2002), *Practical Regression and Anova using R*, Ch.8](http://cran.r-project.org/other-docs.html) for some examples.
Comment: I think this may be off-topic because it is about how to use R without a reproducible example.
Answer: First, I would review some introductory resources on data transformations, in general, and on Box-Cox transformation, in particular. For example, see my relevant answer and this webpage.
For more details on performing Box-Cox transformation in R, check this excellent discussion. In addition to MASS package, some other R packages can be used for Box-Cox transformation, also consider using car package, which offers several types of power transformations and somewhat more general than in MASS functions, for example this one.
In regard to selecting the optimal parameter for the transformation, see this answer on StackOverflow as well as the AID R package (see page on CRAN).
Answer: You can use function "preProcess" of caret package  for univariate CB transformation and "bcPower" function of "car" package for multivariate CB transformation.
Regards,
Jüri.

Box-Cox transformation in R


Simular recebimento de ligação com aparelho no usb (adb shell Android)

Question: Gostaria de saber como fazer para simular o recebimento de uma ligação (Receber uma ligação) com o aparelho conectado via usb no adb shell do Android.
Preciso realizar alguns testes no aparelho físico, porém não pode ser através do emulador.
Não pode ser  pelo emulador,pois a máquina de desenvolvimento é muito limitada.
Não posso ligar para o aparelho , pois não tem número, o chip de teste é só 3g.
Comment: Simular uma ligação? você diz "chamar" o seu telefone? com que finalidade?
Comment: Porque não pode ser através do emulador?
Comment: Não bastaria ligar de um celular fixo para ele?
Comment: Esqueci de informar mais um detalhe, o aparelho só tem um chip 3g.
Answer: Quando necessitei testar uma chamada em um aparelho físico eu apenas fiz uma ligação para o mesmo. Exemplo:

O aplicativo está instalado e "debugando" no aparelho A.
No aparelho B fiz uma ligação para o aparelho A.

Porem os teste era apenas para ver como o app se comporta quando houver uma ligação. Agora não sei se no seu caso é o mesmo, ou seu app é para trabalhar diretamente com a chamada, como gravar, pegar número de quem está ligando etc.
Comment: Esqueci de informar mais um detalhe, o aparelho só tem um chip 3g.
infelizmente o cenário é esse.
Answer: Não é possível realizar ligações simuladas para um aparelho real.
Para isto, infelizmente você precisará utilizar o emulador.


SASS Map - map-get no longer supporting interpolated values?

Question: Previously in SCSS (version 3.4.21), I can use variable interpolation to get a specific item from a map:
<code>$colors: (
  color-1: #aaa,
  color-2: #bbb,
  color-3: #ccc
);

@mixin color($shade) {
  color: map-get($colors, #{color-}$shade );
}

.element {
  @include color(2);
}
</code>
Compiles to:
<code>.element {
  color: #bbb;
}
</code>
Playground Link
In sass 4 (alpha), it doesn't work (or i'm screwing something up):
<code>#{color-}$shade
</code>
It doesn't allow me to do that. Is there a reason this was taken out?
Answer: I think your syntax is just a bit backwards.
<code>@mixin color($shade) {
  color: map-get($colors, gray-#{$shade} );
}
</code>


How to retrieve content from a website with VS 11

Question: I am trying this in Visual Studio 11 in Windows 8. As I see I can't use WebClient to do this, so I tried using HttpClient:
var client = new HttpClient();
var response = client.Get("http://google.com");
var result = XDocument.Parse(response.Content.ReadAsString());

My problem is that I always get result to be null. In response I get a status code 200 (OK) but I can't see the content.
What am I doing wrong ?
Thanks.
Comment: Also, Google.com is not valid XML.
Comment: Why can't you use `WebClient`?
Comment: @SLaks: I include Syste,.Net but it doesn't find WebClient and it doesn't work with any website, I am trying to get the xml from a page
Comment: You'd better use GetAsync() in Metro apps.
Answer: Most webpages are not valid XML.
You should use HTML Agility Pack instead.


Override OpenAPI Generator BasePath on Generation

Question: I am creating a TypeScript Axios REST API client with OpenAPI Generator. I'm doing this in my terminal using npx @openapitools/openapi-generator-cli generate -i ../swagger.json -g typescript-axios -o ./rest/.
Since my host is depending on my .env variables, I want to read my API_URL from my env variables and pass it to the generating process of the generator.
I found in the docs that changing the BasePath is in some way possible, but I can't figure out the way to do it. Adding --basePath=184.108.40.206:8080 does not work, and I don't understand what "Defined by ToolingExtension" means.
Comment: Unfortunately nothing clean. As Robert says, localhost seems to be hardcoded within the tool. But you could write a bash script, that triggers the api generator and afterwards overrides the host in the resulting code by string replacement. It's not beautiful, but it works.
Comment: For those who are stuck on this, I ended up switching to openapi-typescript following this tutorial:
https://blog.logrocket.com/generating-integrating-openapi-services-react/

Now I can set baseUrl, headers and more easily.
Comment: i'm stuck exaclty at same situation as you, did you found any solution?
Answer: In the latest openapi-generator-cli 5.4.0 the typescript-axios generator respects the servers setting and uses the first server object in the list as the basePath - i.e. if you specify
{
   "openapi": "3.0.2",
   "info": { ... },
   "paths": { ... },
   "components": { ... },
   "servers": [ { "url" : "https://testurl.com" } ]
}

then the generated base.ts file should contain
export const BASE_PATH = "https://testurl.com".replace(/\/+$/, "");

Comment: You can also pass in `--server-variables` to specify different variables to use in the server object. See https://stackoverflow.com/a/72182063/2892375
Answer: For anyone that's arriving here from search, I've been able to set the basePath variable during generation. This was for the Angular generator but I think the outcome would be the same for the Axios generator. I set this variable by using basePath=<YOUR_PATH> in the additional-properties flag in the CLI generate command. An example would be:
npx --yes @openapitools/openapi-generator-cli \
    generate -i ./src/openapi.yaml \
    -g typescript-angular \
    -o ./src/generated-client \
    --additional-properties=fileNaming=kebab-case,basePath=''

You'll now see in your Angular classes the following:
    protected basePath = '';

Comment: it works with `-g typescript-angular` but `-g typescript-axios` ignores it :(
Answer: Unfortunately the "localhost" is hardcoded and as far as I can tell (after digging into the source code for the OpenAPI Generator) there is no way to change it. The documentation that mentions it is, I believe, referring to to the fact that the client that gets outputted allows that value to be overridden.
Answer: Not sure is this relevant, but I've had the same issue today and I wasn't able to override it in any way I found online. I tried passing basePath as addition to the rest of additional paramas, but didn't worked.
Command I used in the end:
openapi-generator-cli generate -g typescript-axios -i https://api.mysite.com/swagger/v1/swagger.json --skip-validate-spec --additional-properties=useSingleRequestParameter=true,withSeparateModelsAndApi=true,modelPackage=models,apiPackage=api -o ./src/"

What worked for me is creating an axios instance, where I have defined my base url:
import axios from 'axios';

const axiosInstance = axios.create({
  baseURL: import.meta.env.VITE_API_BASE_URL
});

axiosInstance.interceptors.response.use(
  response => {
    return response;
  },
  error => Promise.reject(error)
);

export default axiosInstance;

and then passing that to the actual request:
 const getUserData = async (id: string): Promise<UserDataDTO> => {
    const api = UserDataApiFp();
    const executeRequest = await api.apiUserDataIdGet(id);
    const response = await executeRequest(axiosInstance);
    return response.data;
  };


Change Camera Face Direction

Question: I am just starting with 3D in Delphi, and am trying to understand what the camera RotationAngle is used for.  I had presumed it was to specify the direction and angle of the camera.  I have set up a small test bed for this.  It has a camera with 6 cubes around it.  I assume that the camera is initially pointing toward the back of the scene.
Changing RotationAngle X should make it turn to the bottom, front, top and back, in that order.  However in my test bed it goes bottom, top and back, missing the step for front.
Changing RotationAngle Y should make it turn to the right, front, left and back, in that order. However in my test bed it goes right, left and back, again missing the step for front.
Have i got the use for RotationAngle wrong? If so, what should I be using.
This is the code:
<code>    unit MainForm;

    interface

    uses
      System.SysUtils, FMX.Forms, FMX.Materials, System.Math.Vectors, FMX.Types3D, FMX.Objects3D, FMX.Controls3D, FMX.Viewport3D,
      System.Classes, FMX.Types, FMX.Controls, FMX.Layouts, FMX.MaterialSources, System.Types, System.UITypes,
      FMX.Controls.Presentation, FMX.StdCtrls, FMX.Edit;

type

  TForm1 = class(TForm)
    Viewport3D: TViewport3D;
    LayoutMain: TLayout;
    LightMaterialSource1: TLightMaterialSource;
    Panel1: TPanel;
    grpPosition: TGroupBox;
    Camera1: TCamera;
    CubeLeft: TCube;
    CubeRight: TCube;
    grpAngle: TGroupBox;
    ColorMaterialSource2: TColorMaterialSource;
    CubeBottom: TCube;
    CubeTop: TCube;
    CubeBack: TCube;
    CubeFront: TCube;
    ColorMaterialSource1: TColorMaterialSource;
    ColorMaterialSource3: TColorMaterialSource;
    ColorMaterialSource4: TColorMaterialSource;
    ColorMaterialSource5: TColorMaterialSource;
    ColorMaterialSource6: TColorMaterialSource;
    AngleX: TEdit;
    Button1: TButton;
    Button2: TButton;
    Button3: TButton;
    AngleY: TEdit;
    Button4: TButton;
    Button5: TButton;
    AngleZ: TEdit;
    Button6: TButton;
    Button7: TButton;
    PositionZ: TEdit;
    Button8: TButton;
    Button9: TButton;
    PositionY: TEdit;
    Button10: TButton;
    Button11: TButton;
    PositionX: TEdit;
    Button12: TButton;
    txtStep: TEdit;
    procedure ChangePosition(Sender: TObject);
    procedure ChangeAngle(Sender: TObject);
    procedure FormCreate(Sender: TObject);
    procedure txtStepExit(Sender: TObject);
    procedure ChangeAngleText(Sender: TObject);
    procedure ChangePositionText(Sender: TObject);
  private
    Step: Single;
    procedure UpdateCameraPosition;
    procedure UpdateCameraAngle;
  public
  end;

var
  Form1: TForm1;

implementation

{$R *.fmx}

procedure TForm1.ChangeAngle(Sender: TObject);
begin
  case TButton(Sender).Tag of
    0: Camera1.RotationAngle.X := Camera1.RotationAngle.X - Step;
    1: Camera1.RotationAngle.X := Camera1.RotationAngle.X + Step;
    2: Camera1.RotationAngle.Y := Camera1.RotationAngle.Y - Step;
    3: Camera1.RotationAngle.Y := Camera1.RotationAngle.Y + Step;
    4: Camera1.RotationAngle.Z := Camera1.RotationAngle.Z - Step;
    5: Camera1.RotationAngle.Z := Camera1.RotationAngle.Z + Step;
  end;
  UpdateCameraAngle;
end;

procedure TForm1.ChangeAngleText(Sender: TObject);
begin
  case TEdit(Sender).Tag of
    0: Camera1.RotationAngle.X := StrToFloat(AngleX.Text);
    1: Camera1.RotationAngle.Y := StrToFloat(AngleY.Text);
    2: Camera1.RotationAngle.Z := StrToFloat(AngleZ.Text);
  end;
end;

procedure TForm1.ChangePosition(Sender: TObject);
begin
  case TButton(Sender).Tag of
    0: Camera1.Position.X := Camera1.Position.X - Step;
    1: Camera1.Position.X := Camera1.Position.X + Step;
    2: Camera1.Position.Y := Camera1.Position.Y - Step;
    3: Camera1.Position.Y := Camera1.Position.Y + Step;
    4: Camera1.Position.Z := Camera1.Position.Z - Step;
    5: Camera1.Position.Z := Camera1.Position.Z + Step;
  end;
  UpdateCameraPosition;
end;

procedure TForm1.ChangePositionText(Sender: TObject);
begin
  case TEdit(Sender).Tag of
    0: Camera1.Position.X := StrToFloat(PositionX.Text);
    1: Camera1.Position.Y := StrToFloat(PositionY.Text);
    2: Camera1.Position.Z := StrToFloat(PositionZ.Text);
  end;
end;

procedure TForm1.FormCreate(Sender: TObject);
begin
  Step := StrToFloat(txtStep.Text);
end;

procedure TForm1.txtStepExit(Sender: TObject);
begin
  Step := StrToFloat(txtStep.Text);
end;

procedure TForm1.UpdateCameraAngle;
begin
  AngleX.Text := Format('%3.2f', [Camera1.RotationAngle.X]);
  AngleY.Text := Format('%3.2f', [Camera1.RotationAngle.Y]);
  AngleZ.Text := Format('%3.2f', [Camera1.RotationAngle.Z]);
end;

procedure TForm1.UpdateCameraPosition;
begin
  PositionX.Text := Format('%3.2f', [Camera1.Position.X]);
  PositionY.Text := Format('%3.2f', [Camera1.Position.Y]);
  PositionZ.Text := Format('%3.2f', [Camera1.Position.Z]);
end;

end.
</code>
This is the form:
<code>object Form1: TForm1
  Left = 0
  Top = 0
  Caption = 'Form1'
  ClientHeight = 592
  ClientWidth = 713
  FormFactor.Width = 320
  FormFactor.Height = 480
  FormFactor.Devices = [Desktop]
  OnCreate = FormCreate
  DesignerMasterStyle = 0
  object LayoutMain: TLayout
    Align = Client
    Size.Width = 713.000000000000000000
    Size.Height = 592.000000000000000000
    Size.PlatformDefault = False
    TabOrder = 2
    object Viewport3D: TViewport3D
      Align = Client
      Camera = Camera1
      Size.Width = 533.000000000000000000
      Size.Height = 592.000000000000000000
      Size.PlatformDefault = False
      UsingDesignCamera = False
      object Camera1: TCamera
        AngleOfView = 45.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
      end
      object CubeRight: TCube
        Position.X = 4.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
        TwoSide = True
        MaterialSource = ColorMaterialSource2
      end
      object CubeLeft: TCube
        Position.X = -4.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
        TwoSide = True
        MaterialSource = ColorMaterialSource1
      end
      object CubeBottom: TCube
        Position.Y = -4.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
        TwoSide = True
        MaterialSource = ColorMaterialSource5
      end
      object CubeTop: TCube
        Position.Y = 4.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
        TwoSide = True
        MaterialSource = ColorMaterialSource6
      end
      object CubeBack: TCube
        Position.Z = -4.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
        TwoSide = True
        MaterialSource = ColorMaterialSource3
      end
      object CubeFront: TCube
        Position.Z = 4.000000000000000000
        Width = 1.000000000000000000
        Height = 1.000000000000000000
        Depth = 1.000000000000000000
        TwoSide = True
        MaterialSource = ColorMaterialSource4
      end
      object ColorMaterialSource2: TColorMaterialSource
        Color = claSpringgreen
        Left = 344
        Top = 280
      end
      object ColorMaterialSource1: TColorMaterialSource
        Color = claRed
        Left = 272
        Top = 288
      end
      object ColorMaterialSource3: TColorMaterialSource
        Color = claYellow
        Left = 360
        Top = 296
      end
      object ColorMaterialSource5: TColorMaterialSource
        Color = claPink
        Left = 376
        Top = 312
      end
      object ColorMaterialSource6: TColorMaterialSource
        Color = claOrange
        Left = 384
        Top = 320
      end
      object ColorMaterialSource4: TColorMaterialSource
        Color = claViolet
        Left = 368
        Top = 304
      end
    end
    object Panel1: TPanel
      Align = Right
      Position.X = 533.000000000000000000
      Size.Width = 180.000000000000000000
      Size.Height = 592.000000000000000000
      Size.PlatformDefault = False
      TabOrder = 0
      object grpPosition: TGroupBox
        Position.X = 5.000000000000000000
        Position.Y = 134.000000000000000000
        Size.Width = 169.000000000000000000
        Size.Height = 123.000000000000000000
        Size.PlatformDefault = False
        Text = 'Position'
        TabOrder = 1
        object Button7: TButton
          Tag = 5
          Position.X = 139.000000000000000000
          Position.Y = 87.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 41
          Text = '>'
          OnClick = ChangePosition
        end
        object PositionZ: TEdit
          Tag = 2
          Touch.InteractiveGestures = [LongTap, DoubleTap]
          TabOrder = 38
          Text = '0.00'
          Position.X = 35.000000000000000000
          Position.Y = 87.000000000000000000
        end
        object Button8: TButton
          Tag = 4
          Position.X = 9.000000000000000000
          Position.Y = 87.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 40
          Text = '<'
          OnClick = ChangePosition
        end
        object Button9: TButton
          Tag = 3
          Position.X = 139.000000000000000000
          Position.Y = 55.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 44
          Text = '>'
          OnClick = ChangePosition
        end
        object PositionY: TEdit
          Tag = 1
          Touch.InteractiveGestures = [LongTap, DoubleTap]
          TabOrder = 39
          Text = '0.00'
          Position.X = 35.000000000000000000
          Position.Y = 55.000000000000000000
        end
        object Button10: TButton
          Tag = 2
          Position.X = 9.000000000000000000
          Position.Y = 55.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 43
          Text = '<'
          OnClick = ChangePosition
        end
        object Button11: TButton
          Tag = 1
          Position.X = 139.000000000000000000
          Position.Y = 23.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 45
          Text = '>'
          OnClick = ChangePosition
        end
        object PositionX: TEdit
          Touch.InteractiveGestures = [LongTap, DoubleTap]
          TabOrder = 42
          Text = '0.00'
          Position.X = 35.000000000000000000
          Position.Y = 23.000000000000000000
          OnExit = ChangePositionText
        end
        object Button12: TButton
          Position.X = 9.000000000000000000
          Position.Y = 23.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 46
          Text = '<'
          OnClick = ChangePosition
        end
      end
      object grpAngle: TGroupBox
        Position.X = 5.000000000000000000
        Position.Y = 5.000000000000000000
        Size.Width = 169.000000000000000000
        Size.Height = 124.000000000000000000
        Size.PlatformDefault = False
        Text = 'Angle'
        TabOrder = 0
        object AngleX: TEdit
          Touch.InteractiveGestures = [LongTap, DoubleTap]
          TabOrder = 42
          Text = '0.00'
          Position.X = 34.000000000000000000
          Position.Y = 24.000000000000000000
          OnExit = ChangeAngleText
        end
        object Button1: TButton
          Position.X = 8.000000000000000000
          Position.Y = 24.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 46
          Text = '<'
          OnClick = ChangeAngle
        end
        object Button2: TButton
          Tag = 1
          Position.X = 138.000000000000000000
          Position.Y = 24.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 45
          Text = '>'
          OnClick = ChangeAngle
        end
        object Button3: TButton
          Tag = 3
          Position.X = 138.000000000000000000
          Position.Y = 56.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 44
          Text = '>'
          OnClick = ChangeAngle
        end
        object AngleY: TEdit
          Tag = 1
          Touch.InteractiveGestures = [LongTap, DoubleTap]
          TabOrder = 39
          Text = '0.00'
          Position.X = 34.000000000000000000
          Position.Y = 56.000000000000000000
          OnExit = ChangeAngleText
        end
        object Button4: TButton
          Tag = 2
          Position.X = 8.000000000000000000
          Position.Y = 56.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 43
          Text = '<'
          OnClick = ChangeAngle
        end
        object Button5: TButton
          Tag = 5
          Position.X = 138.000000000000000000
          Position.Y = 88.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 41
          Text = '>'
          OnClick = ChangeAngle
        end
        object AngleZ: TEdit
          Tag = 2
          Touch.InteractiveGestures = [LongTap, DoubleTap]
          TabOrder = 38
          Text = '0.00'
          Position.X = 34.000000000000000000
          Position.Y = 88.000000000000000000
          OnExit = ChangeAngleText
        end
        object Button6: TButton
          Tag = 4
          Position.X = 8.000000000000000000
          Position.Y = 88.000000000000000000
          Size.Width = 22.000000000000000000
          Size.Height = 22.000000000000000000
          Size.PlatformDefault = False
          TabOrder = 40
          Text = '<'
          OnClick = ChangeAngle
        end
      end
      object txtStep: TEdit
        Touch.InteractiveGestures = [LongTap, DoubleTap]
        TabOrder = 37
        Text = '2.00'
        Position.X = 43.000000000000000000
        Position.Y = 263.000000000000000000
        OnExit = txtStepExit
      end
    end
  end
  object LightMaterialSource1: TLightMaterialSource
    Diffuse = claWhite
    Ambient = xFF202020
    Emissive = claNull
    Specular = xFF606060
    Shininess = 30
    Left = 436
    Top = 56
  end
end
</code>
Answer: Using your test application and keeping the position at (0, 0, 0), the rotation seems to work as you expect, contrary to what you claim it does.
Rotating around x-axis (button 2):
violet - pink - yellow - brown - violet
Rotating around y-axis (button 4):
violet - green - yellow - red - violet
Rotating around z-axis (button 6):
rotates the violet square.
Zooming out (third position setting) -15

violet becomes hidden by the yellow cube, all others are visible
at rotation of 15 the cube constellation image is rotated -15

So, when rotating the camera around the selected axis the view changes according to expectations.

Change Camera Face Direction


Time Series analysis ARIMA

Question: I am trying to predict "daily_cases" using time series analysis.
The time series plot looks like -

The ACF plot of original series is given below-

The above ACF plot suggests that trend is present which needs to be removed using  appropriate differencing.
The ACF and PACF plots of 1-differenced series are given below-

The above plots suggest an ARMA(1,1) process which was also validated by "arma_order_select_ic()" function in python on the basis of "aic" and "bic" scores.
But, after fitting the model I get weird fitted values ,which were way different from observed values(differing by 10000!).
Please help.
Thanks in advance
Comment: I'm not familiar with estimating models in Python but you might be fitting the differenced model so, to get the actual predictions, you have to undifference the predictions of the differenced model ? Just a guess.
Comment: @mlofton The 'I' in ARIMA stands for integrated, and the order of integration is same as the order of differencing. Btw, I passed the original series to function.
Comment: Hi: I know that the I in ARIMA stands for integrated. What I'm saying is that, when you make the call to the ARIMA function in python, it may be giving you the fitted model AFTER THE SERIES IS DIFFERENCED. If you want to predict the original data, you need to figure out what the predictions about the differenced series imply about the predictions of the not differenced series. Say it was ARIMA(1,1,0). Then the estimated modelis $Y_t - Y_{t-1} = \phi (Y_{t-1} - Y_{t-2})$ but this is the differenced series. So, fitted's will be differences.
Comment: @mlofton the python library automatically adjusts the calculation like shown here- https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/
Comment: Okay. So, it sounds like my guess was wrong. I'd have to look at the details to have any hope of coming up with another guess. I have no time so  hopefully someone else with Python experience will help you. You may want to do the same thing in R in order to ensure consistency of result.
Answer: I would recommend using a unit root test like Augmented Dicky Fuller Test to check if your time series is really stationary after differencing, if not you might need to do a second differentiation.
Comment: You can also look at the box ljung test which suggest if serial correlation (of any type) exists. The problem with tests like ADF is that they have notorious low power. Some suggest, as a partial correction to this, that you use a test like ADF which as the null of non-stationarity and KPSS which has the opposite null and see if the results are logically the same. To me it looks like your data was non-stationary, but then became stationary at a later point.

Time Series analysis ARIMA


Debian + ProFTPD + LDAP Incorrect Password Issue

Question: I have the LDAP configuration configured for ProFTPD and I have modified the modules.conf file  to include the LDAP module. However, every time I login with FileZilla I get 530 Login Incorrect. It does this for all users except those whose passwords are defined locally as well as in LDAP. The exact same setup works fine on my CentOS server and I've already tried re-installing it after purging the configuration files.
Answer: Do you have the appropriate /etc/pam.d file for proftpd to tell it to use LDAP?
The directory /usr/share/doc/libpam-ldap/examples/pam.d/ has examples. Normally you can just copy them all over to /etc/pam.d/
However on squeeze you have to change all occurances of "pam_unix_" to "pam_unix" because all /lib/security/pam_unix_ files are gone and it now is just /lib/security/pam_unix.so. If you don't make these changes and log out you can't log back in, neither log in through the console or anything. You'll have to boot a rescue CD and change them that way.
So, for example in /usr/share/doc/libpam-ldap/examples/pam.d/ssh change:
auth       required     /lib/security/pam_unix_auth.so try_first_pass
account    required     /lib/security/pam_unix_acct.so
session    required     /lib/security/pam_unix_session.so

to:
auth       required     /lib/security/pam_unix.so try_first_pass
account    required     /lib/security/pam_unix.so
session    required     /lib/security/pam_unix.so

The file for ftp (/usr/share/doc/libpam-ldap/examples/pam.d/ftp) looks like this:
#%PAM-1.0
auth       required /lib/security/pam_listfile.so item=user sense=deny file=/etc/ftpusers onerr=succeed
auth       required /lib/security/pam_shells.so
auth       sufficient   /lib/security/pam_ldap.so
auth       required /lib/security/pam_pwdb.so shadow nullok
account    sufficient   /lib/security/pam_ldap.so
account    required /lib/security/pam_pwdb.so
#session   sufficient   /lib/security/pam_ldap.so
session    required /lib/security/pam_pwdb.so


Migrating to a Field Collection

Question: I'm working on a project that requires a CSV of speaking engagements to be migrated to Drupal. With Migrate, I've been able to migrate over basic fields, like name, title, etc.
However, the speaker field is a field collection, which is driving me nuts. While the field collection allows unlimited speakers, the CSV only has one per event. I've tried using the node_insert hook, as described here but to no avail. Is there a better/easier way to migrate a single speaker into the speaker field collection?
Answer: There is a Field Collection issue with a Field Collection Migrate destination (in the patch in comment #222). The added field_collection.migrate.inc file contain a bit of documentation on how to use it.
In you case, I guess you have a speaking engagements migration class which load your CSV file and create Speaking engagement nodes. You will then need a speaker migration class to load the same CSV file and create Speaker field collections. This second migration class depends on the first one, so when a Speaker is created, it will be linked to its Speaking engagement.
Something like
class SpeakerMigration extends Migration {

  public function __construct($arguments) {
    parent::__construct($arguments);
    $this->dependencies = array('SpeakingEngagementMigration');
    $this->source = new MigrateSourceCSV('path/to/speaking/engagement.csv', array(), array());
    $this->destination = new MigrateDestinationFieldCollection('field_speakers',array('host_entity_type' => 'node'));
    $this->map = new MigrateSQLMap($this->machineName, array(
      'id' => array(
        'type' => 'char',
        'length' => 40,
        'not null' => TRUE,
        'description' => t("Speaking Engagement's identifier"),
      )), MigrateDestinationFieldCollection::getKeySchema());
    $this->addFieldMapping('host_entity_id', 'id')
      ->sourceMigration('SpeakingEngagementMigration');

    // ... Add field mapping here.
  }
}

Answer: Another option is the implement the complete method of the Migration class and populate the field collection programmatically there.
public function complete($entity, $row) {

  // load the newly created node (just in case entity isn't fully populated).
  $node = node_load($entity->nid);

  // set the values of all field collection fields
  $values = array(
    'field_name' => 'FIELD_COLLECTION_ITEM_FIELD_NAME',
    'field1'     => array(LANGUAGE_NONE => array(array('value' => 'VALUE'))),
    'field3'     => array(LANGUAGE_NONE => array(array('value' => 'VALUE'))),
  );

  // create the field collection item entity
  $item = entity_create('field_collection_item', $values);

  // attach it to the node loaded above
  $item->setHostEntity('node', $node);

  // save the entity
  $item->save();
}

NOTE: If you are adding files to the field collection item, they will not be removed on a rollback unless you explicitly do that.

Migrating to a Field Collection