Mathematical odds of picking letters

Question: What are the odds of picking two pre-selected non-sequential letters in order out of a $26$ letter alphabet, i.e. $D$ then $S$, not $S$ then $D$? One chance only.
Comment: Are we sampling with replacement or without replacement? And are the two letters specified in advance?
Comment: $26^{-2}$, it's exactly one out of $26^2$ many possible outcomes.
Comment: @user155124 Nope, he said non-sequential, so after d is picked, c and e could not be picked
Comment: @user155124 or maybe he means that the second letter must come after the first, any amount later, but not before. Im not 100% sure
Comment: Hello, yes, the letters are specified in advance, but their position in the alphabet is not important. I think non-sequential was misleading. They can be ANY two letters. They happen to be D and S, and it happens that D is picked before S.
Comment: First are the selection of the two letters the same event? Or separate, and are you done selecting letters, or will you continue to select letters. If you continue to select letters that needs to be specified or the answer would assume you selected wrong (1/26)(1/26), or 1/26^2 but that wouldn't be correct for your scenario is suggesting either two separate events (like above) however selecting from a different amount of letters than 26. What exactly do you mean sequential? Picking D before the next event of selecting letters? That's wouldn't be 1/26 as mentioned by others because selecting an
Answer: if I understand the question correctly, what's the probability that you select a two letters from the English alphabet, such that the second one comes after the first. Use the law of total probability:
$$
P(X) = \sum_{k=1}^{26}P(X|Y=k)P(Y=k) = \frac{25}{26} \cdot P(X=A) + \frac{24}{26} \cdot P(X=B) +\ldots
$$
of course $P(X=j) = \frac{1}{26}$. Can you handle from here?
EDIT: you may want ot use the identity $\sum_{k=1}^{n} k = \frac{n(n+1)}{2}$
Comment: Hello Alex, I'm not a mathematician, and need some more help. Specifically the letters are D and S, and they are pre-specified, i.e. first D, then S. These are the only two letters picked out of the English alphabet in that order in a single try. What are the 'odds'? 1 in 600?, 1 in 6000? etc.
Comment: if I understand your question correctly, there are two options: 1)with replacement, then this probability is $\frac{1}{26^2}$ because after sampling 'D' 'you put it back' and the trials are independent, 2) without replacement, this probability is $\frac{1}{26} \cdot \frac{1}{25}$, because you don't 'put' 'D' back. This is if the order is important (i.e. DS, not SD).

Mathematical odds of picking letters


Contagem de letras em um string

Question: Preciso fazer um programa em Python que receba um string e retorne a soma das letras desse string em uma lista de tuplas, da seguinte maneira:
Exemplo:
banana -----> [('b', 1), ('a', 3), ('n', 2)]
Meu programa está assim:
for n in txt:
      c =  txt.count(n)
      if n != ' ':
          if n not in lista_txt:
              tupla = n, c
      lista_txt.append(tupla)

Porém ele retorna a soma de cada letra mais de uma vez:
[('b', 1), ('a', 3), ('n', 2), ('a', 3), ('n', 2), ('a', 3)]

Como posso resolver isso?
Answer: Você está verificando apenas se n está na sua lista de tuplas, quando era para você estar verificando se a tupla (n, c) existe na sua lista
Outro detalhe é que você está adicionando a tupla na sua lista mesmo quando ela não passa pelas condições != ' ' e (n, c) not in lista_txt, você precisa indentar esse seu append para ele ficar dentro do if, dessa forma:
for n in txt:
    c =  txt.count(n)
    if n != ' ' and (n, c) not in lista_txt:
        tupla = n, c
        lista_txt.append(tupla)

Veja o código funcionando aqui
Answer: Ainda que não respondendo diretamente à sua questão acho interessante mencionar que você consegue chegar ao seu objetivo de uma forma muito direta usando funções e estruturas já existentes no python.
Para tal basta servir-se de Counter para gerar um dicionário com as contagens, e da função items obter as contagens como uma lista de tuplas:
>>> from collections import Counter
>>> texto = "banana"
>>> list(Counter(texto).items())
[('b', 1), ('a', 3), ('n', 2)]

Veja a funcionar no Ideone
Como aparte, para além de ser mais direto é também mais efeciente pois a sua solução tem complexidade O(n²), ou seja quadrática, ao passo que com Counter a solução fica em O(n), linear.
Answer: Você pode também usar um dicionário e ir setando as chaves e os valores, a medida que vai contando as vezes em que a letra contida na palavra se repete na sequência:
txt = 'banana'
dict = {}

for n in txt:
    c = txt.count(n)
    if not (n,c) in dict:
        dict.__setitem__(n,c)
print(dict)

Contagem de letras em um string


Scipy odeint giving lsoda warning

Question: I am totally new to coding and I want to solve these 5 differential equations numerically. I took a python template and applied it to my case. Here's the simplified version of what I wrote:
import numpy as np
from math import *
from matplotlib import rc, font_manager
import matplotlib.pyplot as plt
from scipy.integrate import odeint

#Constants and parameters
alpha=1/137.
k=1.e-9
T=40.
V= 6.e-6
r = 6.9673e12
u = 1.51856e7

#defining dy/dt's
def f(y, t):
       A = y[0]
       B = y[1]
       C = y[2]
       D = y[3]
       E = y[4]
       # the model equations
       f0 = 1.519e21*(-2*k/T*(k - (alpha/pi)*(B+V))*A)
       f1 = (3*B**2 + 3*C**2 + 6*B*C + 2*pi**2*B*T + pi**2*T**2)**-1*(-f0*alpha/(3*pi**3) - 2*r*(B**3 + 3*B*C**2 + pi**2*T**2*B) - u*(D**3 - E**3))
       f2 = u*(D**3 - E**3)/(3*C**2)
       f3 = -u*(D**3 - E**3)/(3*D**2)
       f4 = u*(D**3 - E**3)/(3*E**2) + r*(B**3 + 3*B*C**2 + pi**2*T**2*B)/(3*E**2)
       return [f0, f1, f2, f3, f4]

# initial conditions
A0 = 2.e13
B0 = 0.
C0 = 50.
D0 = 50.
E0 = C0/2.

y0 = [A0, B0, C0, D0, E0]       # initial condition vector
t  = np.linspace(1e-15, 1e-10, 1000000)   # time grid

# solve the DEs
soln = odeint(f, y0, t, mxstep = 5000)
A = soln[:, 0]
B = soln[:, 1]
C = soln[:, 2]
D = soln[:, 3]
E = soln[:, 4]

y2 = [A[-1], B[-1], C[-1], D[-1], E[-1]]
t2  = np.linspace(1.e-10, 1.e-5, 1000000)
soln2 = odeint(f, y2, t2, mxstep = 5000)
A2 = soln2[:, 0]
B2 = soln2[:, 1]
C2 = soln2[:, 2]
D2 = soln2[:, 3]
E2 = soln2[:, 4]

y3 = [A2[-1], B2[-1], C2[-1], D2[-1], E2[-1]]
t3  = np.linspace(1.e-5, 1e1, 1000000)
soln3 = odeint(f, y3, t3)
A3 = soln3[:, 0]
B3 = soln3[:, 1]
C3 = soln3[:, 2]
D3 = soln3[:, 3]
E3 = soln3[:, 4]

#Plot
rc('text', usetex=True)
plt.subplot(2, 1, 1)
plt.semilogx(t, B, 'k')
plt.semilogx(t2, B2, 'k')
plt.semilogx(t3, B3, 'k')

plt.subplot(2, 1, 2)
plt.loglog(t, A, 'k')
plt.loglog(t2, A2, 'k')
plt.loglog(t3, A3, 'k')

plt.show()

I get the following error:
lsoda--  warning..internal t (=r1) and h (=r2) are
   such that in the machine, t + h = t on the next step
   (h = step size). solver will continue anyway
  In above,  R1 =  0.2135341098625E-06   R2 =  0.1236845248713E-22

For some set of parameters, upon playing with mxstep in odeint (also tried hmin and hmax but didn't notice any difference), although the error persists, my graphs look good and are not impacted, but most of the times they are.
Sometimes the error I get asks me to run with the odeint option full_output=1 and in doing so I obtain:
A2 = soln2[:, 0]
TypeError: tuple indices must be integers, not tuple

I don't get what this means when searching for it.
I would like to understand where the problem lies and how to solve it.
Is odeint even suitable for what I'm trying to do?
Answer: In that lsoda warning, t refers to the current time value and h refers to the current integration step size. The step size has become so close to zero that the current time plus the step size evaluates as equal to the current time due to rounding error (i.e. r1 + r2 == r1). This sort of problem usually occurs when the ODEs you are integrating are badly behaved.
On my machine the warning message only seems to occur when computing soln2. Here I've plotted the values of each of the parameters in the region where the warnings are occurring (note that I've switched to linear axes for clarity). I've marked the time step where the lsoda error first appeared (at r1 = 0.2135341098625E-06) with a red line:

It's no coincidence that the appearance of the warning message coincides with the 'kink' in E!
I suspect that what's happening is that the kink represents a singularity in the gradient of E, which is forcing the solver to take smaller and smaller steps until the step size reaches the limits of floating point precision. In fact, you can see another inflection point in D which coincides with a 'wobble' in B, probably caused by the same phenomenon.
For some general advice on how to deal with singularities when integrating ODEs, take a look at section 5.1.2 here (or any decent textbook on ODEs).

You were getting an error with full_output=True because in this case odeint returns a tuple containing the solution and a dict containing additional output information. Try unpacking the tuple like this:
soln, info = odeint(..., full_output=True)

Scipy odeint giving lsoda warning


Error uploading image

Question: Since I changed my drupal to my backup that it's giving me some errors, but overall it stills working, but today I was about to add a new content and when I try to upload new images it shows me this error

"The file could not be uploaded."

And in log messages I've this message errors:

Location: http://localhost/node/add/imovel?_wrapper_format=drupal_ajax&ajax_form=1&element_parents=field_image%2Fwidget
Message: The upload directory public://2017-02 for the file field field_image
  could not be created or is not accessible. A newly uploaded file could
  not be saved in this directory as a consequence, and the upload was
  cancelled.

In the same location this 2 other errors:

Recoverable fatal error: Argument 1 passed to Drupal\Core\Form\FormState::setError() must be of the type array, null given
Notice: Undefined index: value in Drupal\Core\Field\Plugin\Field\FieldWidget\NumberWidget->errorElement()

Now I can't upload images and that don't allow me to add new content.
I already try to create a new field but that didn't work.
PS: I think the error has to do something about Image Crop because before this error happen a box to crop my image would appear after I upload and know I upload and no box to crop the image.. And when I press save and publish content the error pops

The specified file temporary://filekrGdEp could not be moved/copied because the destination directory is not properly configured. This may be caused by a problem with file or directory permissions.
Comment: your first log message might be a permission issue or your public folder found in `sites/default/files` was deleted. Kindly double check.
Comment: I have the folder in "/home/user/public_html/sites/default/files/2017-02"
Comment: I see... There is a missing field that is required to be filled in the manage form display of your content type that is set to disabled/hidden.
Comment: What does any of that have to do with the image problem?
Answer: This sounds very much like a permission issue. Please check the permissions of

/home/user/public_html/sites/default/files/2017-02

The directory must be writeable for Drupal. Set its permissions at least to 710. You might want to set them for testing purposes to 755 or even 777 to see if your error messages disappear.
Comment: I already put my folder 'files' with permissions 777 and everything inside it.
Didn't work, I'm going for a fresh start..
Comment: Please check permissions of your temporary directory also.
Comment: I also did that, I ask a buddy help and we were in that for over an hour and then he told me it was better to start over. Too many bugs, you fix one -> one more appears
Answer: Try setting private files path in settings.php to:

"../FOLDER_WITH_PRIVATE_FILES"
Answer: I was having this same error.
I didn't have important files in the folder. So I just deleted. Try to upload the file again, and it works, it created the folder and upload the files correctly.
If you already have files in the folder, you may just rename it, try the upload, and if it succeeds and creates the folder anew, you just move the files from the old folder to the new one.
Makes sense?
Answer: The fact that i had to change the file permission of settings.php to 755 or above first then after mentioning all the things change it back to 555 then everything worked fine and i would suggest keeping the private directory on the same level as web and vendor directories and mention file path  as ../private in settings.php.
Comment: Welcome to Drupal Answers! You could also have the *private* directory in a different directory and use the absolute path to access it. It would not make any difference, as long as the system account used for the web server and PHP has write access permission to that directory. It isn't clear what the first part of the answer is trying to say; I guess that is one of the reasons for the down-vote. (The other one could be that the suggested place for the *private* is irrelevant for the permission issue.)
Comment: By putting the directory outside the web folder you make it more easy to manage and permissions dont get all messed up.I had to go through all the permissions in my cpanel instead of doing it in my sftp

Error uploading image


Avoiding collision with a plane

Question: I am developing the software to control a common fly's flight in Opensim simulator, although I am facing some troubles determining when the fly must maneuver to avoid hitting a wall.
Questions

How do I get the point of collision between the fly and the wall?
How do I know if I have to turn clockwise or counterclockwise?

Data
I have access to the following data:

Position of the center of wall in world coordinates.
Two additional points representing the nearest end of the wall and the furthest, in local coordinates of the wall.
Position of the fly in world coordinates.
Vector representing the direction (force) of the fly in local coordinates.
The difference, in radians, of the position of the local axis of the fly with regard to the global axis.

Example: a fly moving from left to right, and a wall of 14m wide and 8m tall.

Coordinates
There are several axis in the world: one axis per object, situated at its center and initially with the same orientation as the global axis; and one unique global axis, at one end of the world.
The global axis is used to place objects in the world.
Algorithm
My idea so far is to calculate the intersection of the fly's predicted path with the wall, and then, with the point of collision, calculate the distance and maneuver to avoid that collision. The problem is that I have not been able to calculate that point with the information neither the direction of the rotation (clockwise or counterclockwise).
Valid assumption

The wall is an infinite plane.
Fly's rotation only occur around z-axis.
I am a bit geometry newbie.
Comment: If you know the position of the fly and his velocity, can't you use that to define a line of infinite length and then find where that line hits the wall. It doesn't matter if you're using world or wall coordinates, as long as use the same for both. As for direction, the sign of the determinant will tell you if one vector is to the right or left of another, though I am sure there are better ways.
Comment: @AnonSubmitter85 that's the problem, how do I get a line with a vector and an angle? How do I interset that line with a plane defined by 3 points? I don't really know what a determinant is (I've updated the assumptions ;) )
Answer: You know where the fly is, $\mathbf{r}_{fly}$, and its velocity $\mathbf{v}_{fly}$. For $\alpha \in \mathbb{R}$, we have the following line that goes the through fly oriented with its velocity:
$$
\mathbf{r} = \mathbf{r}_{fly} + \alpha \cdot \mathbf{v}_{fly}.
$$
Convince yourself that as $\alpha$ varies from $-\infty$ to $+\infty$, all points on this line are covered, which means that there must exist some value of $\alpha$ that tells us where the line hits the wall -- assuming that the line hits the wall at all.
For a point to be part of the wall, the vector from the wall's center (actually it's any other point on the wall, but it helps to choose a fixed reference point), $\mathbf{r}_{wall}$, to said point must be orthogonal to the wall's normal vector, $\mathbf{n}$:
$$
\mathbf{n}^{\mathrm{T}}(\mathbf{r} - \mathbf{r}_{wall}) = 0.
$$
We can combine these two relations to get
$$
\mathbf{n}^{\mathrm{T}} \left( \mathbf{r}_{fly} + \alpha \cdot \mathbf{v}_{fly} - \mathbf{r}_{wall} \right) = 0.
$$
Everything in the above equation except for $\alpha$ is known, which is easily solved for:
$$
\alpha = {-{\mathbf{n}^{\mathrm{T}} (\mathbf{r}_{fly} - \mathbf{r}_{wall})   } \over {\mathbf{n}^{\mathrm{T}} \mathbf{v}_{fly}}    }
$$
If the denominator is $0$, then we have an undefined equation. However, note that the denominator is only zero if the velocity is orthogonal to the wall's normal vector, that is, the fly is moving parallel to the wall. In which case, the line never hits the wall and there is no solution to the equation. Otherwise, plugging the above value of $\alpha$ back into the equation for the position of the fly will tell you where he hits the wall. Another case of note is when $\alpha < 0$. This would mean that fly is moving away from the wall.

Avoiding collision with a plane


Why is it wrong to stop an A/B test before optimal sample size is reached?

Question: I am in charge of presenting the results of A/B tests (run on website variations) at my company. We run the test for a month and then check the p-values at regular intervals until we reach significance (or abandon if significance is not reached after running the test for a long time), something I am now finding out is a mistaken practice.
I want to stop this practice now, but to do that, I want to understand WHY this is wrong. I understand that the effect size, the sample size (N), the alpha significance criterion (α) and statistical power, or the chosen or implied beta (β) are mathematically related. But what exactly changes when we stop our test before we reach the required sample size?
I have read a few posts here (namely this, this and this), and they tell me that my estimates would be biased and the rate of my Type 1 error increases dramatically. But how does that happen? I am looking for a mathematical explanation, something that would clearly show the effects of sample size on the outcomes. I guess it has something to do with the relationships between the factors I mentioned above, but I have not been able to find out the exact formulas and work them out on my own.
For eg., stopping the test prematurely increases Type 1 error rate. Alright. But why? What happens to increase type 1 error rate? I'm missing the intuition here.
Help please.
Comment: might be useful http://www.evanmiller.org/how-not-to-run-an-ab-test.html
Comment: Yes I went through this link, but I just did not understand the example given.
Comment: Can you explain what you don't understand. The maths/intuition seems pretty clear: its not so much stopping before the required sample size, but repeatedly checking.  $P(\cup _{i \in 1\dots N}  x_i>\theta) \ge P( x_N>\theta)$, so you can't use a test designed for single checks multiple times.
Comment: sorry Gopalakrishnan - hadn't seen that your first link pointed to that already.
Comment: @GopalakrishnanShanker mathematical explanation given in my answer
Comment: Maybe you should look into control charts:  http://marketing-bulletin.massey.ac.nz/V17/MB_V17_T3_Burk.pdf
Answer: If the null hypothesis is true, then people often expect the p value to be very high. This is not true. If the null hypothesis is true, then p is a uniformely distributed random variable. Meaning, that from time to time is will be below 0.05 just randomly. If you look at a lot of different subsamples, sometimes the p value will be below 0.05.
To make that easier to grasp, here is a small simulation in R:
This will throw a coin 10,000 times and we know, it is a fair coin:
set.seed(1)
n=10000
toss <- sample(1:2, n, TRUE)

Starting from the 5th toss, this will perform a binomial test for fairness after every toss and save the p values:
p.values <- numeric(n)
for (i in 5:n){
     p.values[i] <- binom.test(table(toss[1:i]))$p.value
}

And this will plot the p-values one after the other:
plot(p.values, type="l")
abline(h=0.05)

As you can see, the p value dips below 0.05 a couple of times just to recover and finally end up being far above p=0.05. If we stopped the trial any time p was "significant", we would come to the wrong conclusion. One might argue "We have a sample of roughly over 4000 trials i.i.d. and p was below .05. Shurely we can stop sampling any further". The more frequently you check the p-value, the more likely you are to check at a random dip. In this case we generated the data under the $H_0$ and know, that $H_0$ is true.
(Just to be perfectly open, I've tried more than one seed for the number generator before it was as clear as this example, but that is fair for educational purposes. If you have Rinstalled and running, you can easily play with the numbers.)
Comment: "finally end up being far above p=0.05" sounds misleading to me: shouldn't we expect the curve to keep reaching down to 0.05 level after any arbitrarily large $n$? Is there any guarantee that the curve will "end up" somewhere? I don't think so.
Comment: Thanks for the simple experiment.

But say I stopped the test at one such stage (when p-value < 0.05), what will my results mean? (other than the fact that it is wrong). Is it possible for me to compensate by reducing the p-value threshold?
Comment: +1 Note the correlated tests and related multiple testing problem. See my extended answer with adjustment options below, based on your (very good) example.
Comment: You can not quite compensate it by reducing the p-value threshold $\alpha$. Choosing an $\alpha$ is always a compromise in the area of conflict between type I error and type II error, also known as alpha error and beta error. Of course you can compensate for one of them totally or for each of them partially, but finally this conflict grows increasingly difficult the more often you test.   I have read that Baysians claim they have less of a problem with that but with to little knowledge I think this holds only for the modelling of believe in the value of a statistic and not in yes/no decisions.
Comment: My main point is that of controlling family wise error (FWER) rate or false discovery rate (FDR) both targeting type-1 error. Controlling type-2 error is less of an issue in a/b tests due to usually very large samples.
Comment: @GopalakrishnanShanker if you stopped at the first $p=0.05$ it means you have a false positive. See answer below, also for an adjustment
Comment: @tomka If sample size was large enough anyways, why do interim tests ? Just take the large sample an be gone. Obviously they try to sample as shortly as possible, which means, they are waiting for the sample to be "big enough" and can not rely on "big enough anyway".
Comment: @amoeba Ok, may be misleading, as I was only speaking of this diagram, which was designed to display 10,000 tosses and happend to end far above .05 but could be misunderstood as a general expectation. For arbitrarily large samples, one will have to differentiate between simulation and real life. In real life $H_0$ is never true, so with arbitrarily large sample sizes p can be expected to come close to zero. In simulation, where $H_0$ holds, at any given step, p is a random number from a uniform distribution, so I do expect it to be below .05 in one out of 20 cases, but I am no mathematician.
Comment: @Bernhard this is of course then just a classical statistical test. Nothing to say against that. Testing once has most power here.
Answer: A/B tests that simply test repeatedly on the same data with a fixed type-1 error ($\alpha$) level are fundamentally flawed. There are at least two reasons why this is so. First, the repeated tests are correlated but the tests are conducted independently. Second, the fixed $\alpha$ does not account for the multiply conducted tests leading to type-1 error inflation.
To see the first, assume that upon each new observation you conduct a new test. Clearly any two subsequent p-values will be correlated because $n-1$ cases have not changed between the two tests. Consequently we see a trend in @Bernhard's plot demonstrating this correlatedness of p-values.
To see the second, we note that even when tests are independent the probability of having a p-value below $\alpha$ increases with the number of tests $t$ $$P(A) = 1-(1-\alpha)^t,$$ where $A$ is the event of a falsely rejected null hypothesis. So the probability to have at least one positive test result goes against $1$ as you repeatedly a/b test. If you then simply stop after the first positive result, you will have only shown the correctness of this formula. Put differently, even if the null hypothesis is true you will ultimately reject it. The a/b test is thus the ultimate way of finding effects where there are none.
Since in this situation both correlatedness and multiple testing hold at the same time, the p-value of the test $t+1$ depend on the p-value of $t$. So if you finally reach a $p< \alpha$, you are likely to stay in this region for a while. You can also see this in @Bernhard's plot in the region of 2500 to 3500 and 4000 to 5000.
Multiple testing per-se is legitimate, but testing against a fixed $\alpha$ is not. There are many procedures that deal with both the multiple testing procedure and correlated tests. One family of test corrections is called the family wise error rate control. What they do is to assure $$P(A) \le \alpha.$$
The arguably most famous adjustment  (due to its simplicity) is Bonferroni. Here we set $$\alpha_{adj} = \alpha/t,$$ for which it can easily be shown that $P(A) \approx \alpha$ if the number of independent tests is large. If tests are correlated it is likely to be conservative, $P(A) < \alpha$. So the easiest adjustment you could make is dividing your alpha level of $0.05$ by the number of tests you have already made.
If we apply Bonferroni to @Bernhard's simulation, and zoom in to the $(0,0.1)$ interval on the y-axis, we find the plot below. For clarity I assumed we do not test after each coin flip (trial) but only every hundredth. The black dashed line is the standard $\alpha = 0.05$ cut off and the red dashed line is the Bonferroni adjustment.

As we can see the adjustment is very effective and demonstrates how radical we have to change the p-value to control the family wise error rate. Specifically we now do not find any significant test anymore, as it should be because @Berhard's null hypothesis is true.
Having done this we note that Bonferroni is very conservative in this situation due to the correlated tests. There are superior tests that will be more useful in this situation in the sense of having $P(A) \approx \alpha$, such as the permutation test. Also there is much more to say about testing than simply referring to Bonferroni (e.g. look up false discovery rate and related Bayesian techniques). Nevertheless this answers your questions with a minimum amount of math.

Here is the code:
set.seed(1)
n=10000
toss <- sample(1:2, n, TRUE)

p.values <- numeric(n)
for (i in 5:n){
  p.values[i] <- binom.test(table(toss[1:i]))$p.value
}
p.values = p.values[-(1:6)]
plot(p.values[seq(1, length(p.values), 100)], type="l", ylim=c(0,0.1),ylab='p-values')
abline(h=0.05, lty="dashed")
abline(v=0)
abline(h=0)
curve(0.05/x,add=TRUE, col="red", lty="dashed")

Comment: This works for me. I will have to translate this to business-speak to get my point across to my seniors now, but that's my own problem. Thank you very much


Emails not adding DKIM signature: Milter service - Connection refused

Question: I have generated and added a DKIM record with a selector: 2020.
The DKIM record seems to have been added successfully when I used a few DKIM checkers online but GMAIL (mailed-by is displayed whereas signed-by isn't when I check the info of the email that I sent from the server) and the MxToolBox report can't seem to detect a DKIM signature in the emails sent.
What could be causing this?
Comment: But you signed with the selector 2020? That's getting really odd, then. I'd post the headers from such a message.
Comment: One of the sites where I checked the DKIM record at is [this one](https://protodave.com/tools/dkim-key-checker/).
Comment: MXToolbox being broken could cause it. The other site works, as does a manual query of the DNS.
Comment: @MichaelHampton Hey hello again man! So I ended up using one of those "send an email here and we'll check your records" website and even those sites are saying I don't have a DKIM record even though I'm sure I do and the above site indicated that I did as you noticed too.
Comment: Even gmail is not recognising the DKIM certificate and not showing the `signed-by:` in the email header.
Comment: Do you have a dummy email I can send a message using the server email from? Or should I just copy-paste the header here? And do I need to obfuscate anything from the header or it's safe to copy it as it is?
Comment: You can just use a message you've already tested with. And no, [we generally frown on obfuscation](https://meta.serverfault.com/q/963/126632).
Comment: @MichaelHampton Header added to question. I have only obfuscate the gmail account and left everything else unchanged.
Comment: Your message doesn't have a DKIM signature!
Comment: @MichaelHampton Yes. And I don't understand why because the DKIM checkers are saying I do have a DKIM record.
Comment: And I used virtualmin to generate the private and public keys btw. Not sure if that could be the reason for this though.
Comment: You have a DKIM record in the DNS. That doesn't mean you're signing your email. It means that DKIM signatures can be verified.
Comment: In Virtualmin, I have made sure to checked Yes to `Signing of outgoing mail enabled?`, `Reject incoming email with invalid DKIM signature?`and `Force generation of new private key?` when I initiated the DKIM request for all domains in my server. The only other option in the DKIM section is `Size of new DKIM key` which I manually wrote 2048. Is there something else that I should do after generating the keys?
Comment: Well, if you're using virtualmin you're beyond our ability to help here.
Comment: @MichaelHampton I understand. I really appreciate the help though. Thanks man.
Comment: `DKIM` is not an actual record type, it is a `TXT` with a specific structure. If you query for `2020._domainkey.shillong.work TXT` you will see it. The URL of your tool seems to give incorrect result, if you put this record name, it shows it properly. And validates it. You see it by using this link instead https://mxtoolbox.com/SuperTool.aspx?action=dkim%3Ashillong.work:2020&run=toolpage. DNSViz is also happy at https://dnsviz.net/d/2020._domainkey.shillong.work/Xuf4dA/dnssec/ so there is no problem on the DNS side of things.
Answer: So I have figured out the problem. On checking the error logs in /var/log/mail.log, I noticed multiple lines with the same error:
warning: connect to Milter service inet:localhost:8891: Connection refused

On googling this, I found numerous other people going through the same problem and finally came across a forum discussion about this issue (this is an issue with opendkim btw and not with virtualmin as I initially thought) and figured out that opendkim was not using the correct port (8891) and all I needed to do was:

Un-comment the following line (remove the trailing #) from /etc/opendkim.conf:

Socket                  inet:8892@localhost

On that same line, change 8892 to 8891:

Socket                  inet:8891@localhost

Comment the following line (Add a trailing #) from /etc/opendkim.conf:

#Socket                 local:/var/run/opendkim/opendkim.sock

You can now simply restart opendkim by running the following command (Ubuntu):
service opendkim restart

And also restart apache by running the following command (Ubuntu):
service apache2 restart

And your outbound emails should start including the DKIM signature in your email header.


Como codar esse layout?

Question: 1 - Como dar este aspecto da "div" estar em um plano mais alto que o body, gerando a sombra?
2 - Neste caso, a section tem uma certa transparência e a div é sólida?
Comment: Da pra fazer via css, mas esses tipos de layout a galera faz com imagem recortada mesmo.
Comment: Beleza, mas caso eu escolha com css, o modo que citei está correto para chegar a este resultado?
Comment: Bem vinda ao StackOverflow em Português, talvez queira fazer uma **Tour**: https://pt.stackoverflow.com/tour ou dar uma olhada na **Central de Ajuda**: https://pt.stackoverflow.com/help e verificar como efetuar perguntas de uma forma que sejam respondidas rapidamente, tente informar o que você já tentou e poste o seu código, dificilmente alguém fará o trabalho por você.
Answer: Isso pode ser feito com um efeito de sombreamento bem simples:

.teste1 {
  width: 100%;
  height: 100px;
  background-color: gray;
  display: flex;
  justify-content: center;
}
.teste2 {
  width: 100px;
  height: 100px;
  background-color: gray;
  box-shadow: 0 0 2px 0 rgba(0,0,0,0.12), 0 5px 5px 0 rgba(0,0,0,0.24);
}

Repare que as duas div´s tem a mesma cor, a direfença é que em uma apliquei um leve sombreamento, logo o nível de profundidade fica a seu critério
Box shadow css tricks
Answer: basicamente você precisa brincar com o background-color, um rgba(r, g, b, 0.5) deve resolver, ja o efeito de revelo, você consegue com o box-shadow

html, body {
  position: relative;
  height: 100%;
  width: 100%;
  padding: 0px;
  margin: 0px;
  background-color: rgb(0,150,136);
}

#header {
  position: absolute;
  height: 30px;
  width: 100%;
  background-color: rgb(238,238,238);
  box-shadow: 0px 0px 5px silver;
}

#subheader {
  position: absolute;
  left: 0px;
  right: 0px;
  height: 60px;
  width: 400px;
  margin: auto;
  background-color: rgb(224,224,224);
  box-shadow: 0px 0px 10px black;
}

#content {
  position: absolute;
  left: 0px;
  right: 0px;
  bottom: 0px;
  width: 400px;
  margin: auto;
  height: calc(100% - 120px);
  background-color: rgba(224,224,224,0.5);
  box-shadow: 0px 0px 10px black;
}

#box {
  background-color: rgb(224,224,224);
  height: 60px;
}

Como codar esse layout?


Como criar uma função jQuery para puxar o menu pra cima?

Question:
Gostaria de usar jQuery para subir e descer este menu. Podem me ajudar? O resultado final deve ser este ... Eu comecei o código mas não sei mais como continuar.
$("#esconder_menu").on("click", function(){
     // #header-main <---- esse é o id do menu a subir.
});

Comment: Basta diminuir o `height` do elemento. Se quiser escondê-lo completamente defina `0`, senão deixe uns 10 ou 15 `px`.
Answer: Você não deu muitos detalhes, mas se só estiver faltando a animação de recolher o menu, o slideUp() deve resolver:
$("#esconder_menu").on("click", function(){
     $('#header-main').slideUp();
});

Comment: Sem conhecer o HTML desse trecho não dá pra responder mais que isso...
Comment: O SlideUp some totalmente com o bloco, não gostaria que sumisse, simplesmente subisse até a altura da segunda figura.
Comment: Basicamente você precisa aplicar o `slideUp` só na parte que é pra sumir. Se ela não está separada no seu HTML, embrulhe em uma div.


Como ordenar uma lista pela quantidade de ocorrências? Em Python

Question: Tipo, eu tenho uma lista assim:
lista = [1,1,2,3,3,3]

E quero ordená-la pra ficar
lista = [3,3,3,1,1,2]

Ou seja, os valores mais ocorrentes ficariam no começo da lista, enquanto os valores menos ocorrentes no final. Já tentei várias funções, mas não deu certo.
Comment: A lista é assim: lista = [1,1,2,3,3,3] e quero que seja ordenada por quantidade de ocorrência, então ela ficaria: lista = [3,3,3,1,1,2]. Não tenho ideia de como fazer isso
Answer: Caso não queira usar bibliotecas, aqui vai uma ideia usando dicionários:
Primeiro crio um dicionário com as ocorrências
lista = [1,1,2,3,3,3]
ocorrencias = {}

for numero in lista:
    if numero in ocorrencias: #Se o numero ja esta no dicionario
        ocorrencias[numero] += 1
    else: #Se não está, adiciono em ocorrencias
        ocorrencias[numero] = 1
        #OU ocorrencias.update({numero:1})

>>> print ocorrencias
{1: 2, 2: 1, 3: 3}

Depois crio a lista ordenada a partir do dicionário de ocorrências
def pega_maior_ocorrencia(dicionario):
    maior_ocorrencia = 0
    chave = None
    for key, value in dicionario.iteritems(): #Python2.X
    #for key, value in d.items(): #Python3.X
        if value > maior_ocorrencia:
            maior_ocorrencia = value
            chave = key
    del dicionario[chave] #deleta do dict mesmo dentro da função
    return [chave]*maior_ocorrencia

lista_ocorrencias = []

for i in range(len(ocorrencias)):
    lista_ocorrencias += pega_maior_ocorrencia(ocorrencias)

>>> print lista_ocorrencias
[3, 3, 3, 1, 1, 2]

Answer: Tente desta forma:
from collections import Counter

lista = [1,1,2,3,3,3]

print [item for items, c in Counter(lista).most_common() for item in [items] * c]

# [3, 3, 3, 1, 1, 2]

Comment: [Edite](https://pt.stackoverflow.com/posts/266231/edit) sua resposta e coloque nela sua explicação. Você não testou esse código para dizer "**Se não resolver...**" ?
Comment: Poderia explicar sua resposta ?
Comment: Este "for items, c in Counter(lista).most_common()" retorna os valores da sua lista e a quantidade de vezes que ele se repete, o restante trata de preencher esta lista com item * c ou seja valor x a quantidade de vezes que o valor se repete em sua lista.
Vai desculpando a explicação apressada, mas estou saindo aqui.
Se não resolver desta forma avise o erro que poderei te ajudar assim que retornar.
Comment: Deu super certo Raphael! Muito obrigado amigo, serviu demais! Ainda não conhecia essa biblioteca Counter. Agradecido pela ajuda!


Just curiosity about virtualization

Question:
Possible Duplicate:
How to run VMWare ESX or ESXi in a virtual machine?

If I install an EsxI, and then I try to virtualize an EsxI on my EsxI host, is there a chance that might work?
Just wondering, that's all.
Comment: indeed, didn't see this post before. I can delete this question if you want.
Answer: Yes, it can.
How to run VMWare ESX or ESXi in a virtual machine?

Just curiosity about virtualization


Matrix theory, use Kronecker delta

Question: I been stuck on this problem for a while.
Use Kronecker delta to show when $AE_{i,j}$ is equal to $E_{i,j}A$, where $A$ is a square matrix and $E_{i,j}$ is the elementary matrix, that is $1$ on $(i,j)^{th}$ entry and zero elsewhere.
I know the answer as $A$ needs to be zero on all elements except the intersection  where these two meet. I just cant figure out how to show this with the Kronecker delta.
I forgot to say, in the book they refer to
$E_{i,j}$ $E_{k,l}$ = (delta)(j,k)$E_{i,l}$
I believe the book want the reader to use this in the exercise.
Answer: Here's an attempt.  Let me know how I do.
$$(AE(i,j))_{lk} = \sum_{m}A_{lm}E(i,j)_{mk}$$
$$ = \sum_{m}A_{lm}\delta_{im}\delta_{jk}$$
$$ = A_{li}\delta_{jk}$$
$$(E(i,j)A)_{lk} = \sum_{m}E(i,j)_{lm}A_{mk}$$
$$ = \sum_{m}\delta_{il}\delta_{jm}A_{mk}$$
$$ = \delta_{il}A_{jk}$$
We require:
$$ A_{li}\delta_{jk} = \delta_{il}A_{jk}$$
Suppose $j = k$ and $i = l$.  Then $A_{ii} = A_{jj}$.
Suppose $j \ne k$ and $i = l$.  Then $0 = A_{jk}$.
Suppose $j = k$ and $i \ne l$.  Then $A_{li} = 0$.
So the off-diagonal terms of $A$ must be $0$.  The $i$'th and $j$'th terms along its diagonal must be equal.
Comment: Thank you, yesterday when you wrote your solution it made little sense to me. But after writing it down and expanding on some things it made allot of sense. I think my mind gets confused in how to keep track with all the different labels, but will def practice much more with index notation. I was also able to prove Ei,j Ek,l = (delta)(j,k)Ei,l by inspiration of your solution, so i am making progress.
Comment: This is correct, although i believe this is a different way to solve it than the book teach. Thank you, and wow this is a real brain scratcher.   The book teaches also about E(i,j)E(k,l) = (delta)(j,k)E(i,l).  I believe they want the reader to use this in the exercise. I will study your solution and compare
Comment: In words, my solution is brute force.  I write out what each element of the matrix product would be for both $AE(i,j)$ and $E(i,j)A$ and then I set them equal to each other.  Search "index notation for matrix multiplication" if that wasn't clear.  I also used  $E(i,j)_{mk} = \delta_{mi}\delta_{jk}$, which is true because $E(i,j)_{mk}$ is zero unless it is $E(i,j)_{ij}$, when it is one.
Comment: Thank you, the book i am studying is very dry, and each page takes me a long time to comprehend. It is used in bachelor program in mathematics in Stockholms university. Supposed to be a proof based linear algebra course. I used to be enrolled years ago, but dropped out, now i am just trying to study the material on my own.  For some reason multi variable analysis, proof based multivaraible calculus is so much easier for me to comprehend for some reason. Any books or material that can aid my understanding i would love to be introduced too.
Comment: I recommend you focus on something familiar: matrix multiplication.  You know how to multiply matrices (row by column etc.).  But are you comfortable with matrix multiplication in index notation?  This is key.  If not, study that first.  Compare it to the method you already know.  Think about it for a few hours at least.  Practice multiplying three matrices in index notation.  Do not use the same letter more than twice on one side of an equation.  Once all of that makes perfect sense, you are ready to think about matrix products involving $\delta_{ij}$.  That's what worked for me.
Answer: I will use Einstein's notation and drop the $i,j$ indices of $E$ to avoid confusion.
We have that
$$(AE)_{kl}=A_{kp}E_{pl}=A_{kp}\delta_{pi}\delta_{lj}=A_{ki}\delta{lj}$$
While
$$(EA)_{kl}=E_{kp}A_{pl}=\delta_{ki}\delta{pj}A_{pl}=A_{jl}\delta_{ki}$$
We want them to be equal, i.e.:
$$A_{ki}\delta_{lj}=A_{jl}\delta_{ki}$$
Can you continue?
Comment: Thank you that helps allot.

Matrix theory, use Kronecker delta


Click add button does not show the data

Question: After I input the data and click add button, it shows blank data on the list instead of showing the data on the list. there is an error that says:  expected 0 argument but got 3. I created a const newMovie so it can pass the input value and added it to the .addMovie(newMovie). It shows the data though in the console but does not show the data in the browser. Any help? Thank you.
      onSubmit(formData: NgForm) {
                console.log(formData);
                  const value = formData.value;
                // having problem in here,  it says expected 0 argument but got 3
                 const newMovie = new Movies(value.title, value.actors, value.releasedDate);
                 this.moviesService.addMovie(newMovie);
                }

            //moviesService
             movies: Movies[] = [
                  {
                        title: 'Mr & Mrs Smith',
                        actors: ['Brad Pitt',  ' Angelina Jolie'],
                        releasedDate: '2010-03-18'
                    },
                    {
                        title: 'Titanic',
                        actors: ['Leonardo deCaprio', ' Cate Winslet'],
                        releasedDate: '2008-03-20'
                    }
                ];

                moviesChanged = new BehaviorSubject<Movies[]>(this.movies);

                getMovies(): Movies[] {
                    return this.movies.slice();
                }

                addMovie(movie: Movies) {
                    this.movies.push(movie);
                    this.moviesChanged.next(this.movies.slice());
                }
            }

      // movie-list.component
        export class MovieListsComponent implements OnInit {
        movies: Movies[];

        constructor(private moviesService: MoviesService) { }

          ngOnInit() {
             this.movies = this.moviesService.getMovies();
            this.moviesService.moviesChanged.subscribe(
              (movies: Movies[]) => {
            this.movies = movies;
              }
            );
          }

            movies.component.html

{{message}}

                  <form (ngSubmit)="onSubmit(f)" #f= "ngForm">

                      Title
                      <input type="text" class="form-control" id="title"
                      ngModel
                      name="title">

                        Actors
                        <input type="text" class="form-control" id="actors"
                        ngModel
                        name="actors">

                          Released Date
                          <input type="text" class="form-control" id="releasedDate"
                          ngModel
                          name="releasedDate">

                      <button class= "btn btn-primary" type="submit">Add</button>
                  </form>

    // movie-lists.component

              <app-movies  message = "Add your favourite movie: " ></app-movies >

 Title: {{movie.title}}

                   Actors:  {{ movie.actors }}

                      Released Date:  {{movie.releasedDate}}

Comment: Can you include the code for the `Movies` class?
Answer: Welcome to stackoverflow!
This line of code:
const newMovie = new Movies(value.title, value.actors, value.releasedDate);

Is calling a parameterized constructor in the Movie class, passing in three values.
What does your Movies class look like? Does it have a parameterized constructor with 3 parameters? The error message is suggesting that you don't.
Try instead using this:
const newMovie = new Movies();
newMovie.title = value.title;
newMovie.actors = value.actors;
newMovie.releaseDate = value.releaseDate;

It is constructing the movie with no parameters and then setting the properties individually.

Click add button does not show the data


Return variable when call from nested backticks with @ARGV

Question: In visudo Ubuntu I whitelist this program (I doing this way for security purpose, parameterized all commands)
myuser ALL=(root) NOPASSWD:/App/Filter_Parameters_Wrap.pm *

In program.pl
my $capture = qx("/usr/bin/sudo /App/Filter_Parameters_Wrap.pm kernel_version");

In the module Filter_Parameters_Wrap:
my $fuction  = $ARGV[0];
print filters_dispatch($fuction) if defined $fuction;

sub filters_dispatch {
    my $filter = shift;

    my $dispatch = {
      kernel_version    => \&filter_kernel_version,
      };

    return $dispatch->{$filter}->();

}

sub filter_kernel_version {
   my $command = '/bin/uname -a';
   my $sudo = App::Sudo::Main_Sudo->root($command);
   utf8::decode($sudo);
   return $sudo;
}

This approach is working , but I have to do print in print filters_dispatch (print directly a variable string), so I can get the output of return of function filter_kernel_version in the variable $capture
In some cases inside the function filter_kernel_version I want to create a hash and return as anonymous hash without print directly, but this way is not working
can you recommend a better approach?
Comment: If you need to run a different Perl (because of sudo), this is probably a good approach. Just return a JSON or Data::Dumper of the hash and deserialize it back to a has.
Comment: You can't return anything from backticks. You can print, and capture the output. If you need to return a data structure, consider serialising it. JSON would probably work well for that.
Answer: No matter what option you use to communicate between processes, you'll be limited to sending a sequence of bytes. This means that you will need to serialize your hash somehow. Encoding it using JSON (e.g. using Cpanel::JSON::XS) might be a simple way of doing that.


jQuery/ajax help required

Question: I have below list of links

USA
Canada
France

Cars
Houses
Machines

What I want to do is when a user click on USA then it(The USA link) gets bold. The user then click on any element in list 2. After clicking both the links, the values of selected element in link1 and link2 are passed to an ajax function. Can anyone please tell me about this.
Comment: What about if the user first clicks on a link in the second list and then one in the first?
Comment: Did you try anything so far? Maybe post a code snippet. Don't just expect others to do the work for you
Comment: Post a little bit of you javascript and we can help you
Comment: Alp I can make a button which trigger the ajax function then both the clicked links values will be submitted.
Comment: For others, I know how to get values of a multiple select box using .val() and id but I dont know how to get values from a href link. Any suggestions
Answer: I didn't check it, but you can use smth like that:

USA
Canada
France

Cars
Houses
Machines

Answer: Do something like:
Create a hidden input element for storing selected country value,

$("#list1 a").click(function(e) {
  e.preventDefault();
  $(this).addClass("blodAnchor");// may be font-weight bold in this class
  $("#selectedCountry").val('');
  $("#selectedCountry").val($(this).text());
});

//then
$("#list2 a").click(function(e) {
  e.preventDefault();
  var selectedVal = $(this).text() || "";
  var selectedCountry = $("#selectedCountry").val() || "";
  if(selectedCountry != "" && selectedVal != "") {
     $.ajax({
       type: "POST",
       data: {country: selectedCountry, someVal : selectedVal},
       url: ".....",
       success: function(resp) {
          ........
       }
     });
  }
});

jQuery/ajax help required


How well-sourced should answers be?

Question: MeNoTalk's answer to this question gave me pause before thinking about voting for or against it.  It's anecdotal, and so low on verifiability, but possibly of use to someone asking the question.
But as much as I'd like to say that answers should be well-sourced, being an expat is a life experience and thus likely to be impacted by many subjective issues.  What guidelines can we infer from this?
Answer: That's first hand experience, and unless it's challenged with verifiable evidence, I'd be content to consider the author credible. If he's on the ground there, living and working, then his observations are of course valuable - and I don't know how we'd ask him to 'back it up' any more than he has.
I almost answered the same question because I know people that have worked there and heard the stories that they told. I was going to answer that inclusive of the anecdotal bit, but with more links to how the Philippine government preps people to go live and work there, but I didn't have time.
First hand accounts are good. First hand accounts with a little support from things that reinforce what you're saying are great. But I think we should give the benefit of the doubt in cases where people basically say "source: my two eyes, I live and work here."
I can tell you how crowded the buses in Manila tend to be, in great detail. Should I have to take a picture of it too? I don't want to put too much of a barrier in front of folks that have this sort of rare knowledge that might otherwise leave questions unanswered completely.
Comment: On the other hand that answer is somewhat dangerous, as from the other answer you'll know that doing that is considered illegal in that country.
Comment: @SztupY Well, that's why we have voting. There are _plenty_ of answers on Stack Overflow that are insanely dangerous if used, and thankfully they are voted down with specific comments as to why in most cases.
Answer: From what I've seen so far, I'd roughly divide questions into 3 categories:

visa/legal/tax questions — IMO should always be sourced, preferably to primary source (i.e. official government page) and/or reputable source (eg. immigration law firm or tax advisory office).
services/goods questions (where do I get X?) — preferably concrete answer (like "you can buy it in store X"), if possible sourced to business' webpages.
personal experience based  — possibly unsourced if writing out of own experience, optionally sourced to blogs etc. by people with similar experiences.
Answer: Personal experience answers are great, is some cases even much better than theoretical answers based on various sources.
For example I've asked that question: Avoiding pitfalls when transfering money between banks within EU (Germany -> Poland)

in theory, no fee should be taken (it can be found in official sources on bank sites)
in praxis, an extreme fee was taken (citing theory would be useless, since the facts obviously contradict the theory)

In that case I wouldn't trust any answer based on sources, and not on personal experience.

How well-sourced should answers be?


Como fazer update do Apache2 no Dedian 9 (Stretch)?

Question: Estou usando a versão 2.4.10 do Apache, no Debian 9. No entanto, essa versão do Apache, que eu estou usando, é do Debian 8.
Gostaria de saber como faço para atualizar o Apache2 para a versão 2.4.25.
Comment: Como esta o seu source.list?
Comment: não existe versão 2.10.10 do apache, na verdade 2.10 seria maior que 2.4, então, mas não lançado nenhuma versão acima do 2.4 ainda. Você deve estar fazendo alguma confusão.
Comment: Atualizei a pergunta.
Comment: @EduardoAlmeida A resposta respondeu a sua pergunta?
Comment: @gfleck Desculpa não ter respondido. Sim, respondeu. Obrigado!
Answer: Digite no seu terminal como ROOT:
apt update
apt upgrade

Ou se quiser atualizar somente o apache veja os pacotes que precisam ser atualizados no seu sistema com o comando apt list --upgradable e depois instale somente os pacotes relacionados ao apache, exemplo:
apt install apache2-bin/stable


$f \in \mathcal{R}[a, b]$, $g$ is a bounded real-valued function on $[a, b]$ s.t. $f(x) = g(x)$ almost everywhere. Is $g \in \mathcal{R}[a, b]$?

Question: Suppose that $f \in \mathcal{R}[a, b]$ and $g$ is a bounded real-valued function on $[a, b]$ such that $f(x) = g(x)$ almost everywhere.
Does this mean that $g \in \mathcal{R}[a, b]$?
I think that if $g = f^\mathrm{1/2}$, then it would not be true that $g \in \mathcal{R}[a,b]$ as a consequence.
On the other hand, it would be true for $g = f^\mathrm{1/3}$
Of course, these are not necessarily good examples for the given: $f(x) = g(x)$ almost everywhere
If this is not true, is there an "classic" example of why it doesn't work?
Answer: No. Let $f:[0,1]\rightarrow \mathbb{R}$ be defined by $f=0$ and $g:[0,1]\rightarrow \mathbb{R}$ be the Dirichlet function, i.e., $g(x)=0$ if $x$ is irrational and $g(x)=1$ if $x$ is rational. Clearly, $f$ is Riemann integrable, $g$ is bounded, and $f=g$ a.e.. However, $g$ is not Riemann integrable.
Comment: I had not considered the rational-irrational divide. (Not relevant for the question itself, but does your example hold true for broader intervals like $[0,10] \rightarrow \mathbb{R}$)


Safely merge data frames by factor columns

Question: Factors can help preventing some kinds of programming errors in R: You cannot perform equality check for factors that use different levels, and you are warned when performing greater/less than checks for unordered factors.
a <- factor(letters[1:3])
b <- factor(letters[1:3], levels=letters[4:1])
a == b
## Error in Ops.factor(a, b) : level sets of factors are different
a < a
## [1] NA NA NA
## Warning message:
## In Ops.factor(a, a) : < not meaningful for factors

However, contrary to my expectation, this check is not performed when merging data frames:
ad <- data.frame(x=a, a=as.numeric(a))
bd <- data.frame(x=b, b=as.numeric(b))
merge(ad, bd)
##   x a b
## 1 a 1 4
## 2 b 2 3
## 3 c 3 2

Those factors simply seem to be coerced to characters.
Is a "safe merge" available somewhere that would do the check? Do you see specific reasons for not doing this check by default?
Example (real-life use case): Assume two spatial data sets with very similar but not identical subdivision in, say, communes. The data sets refer to slightly different points in time, and some of the communes have merged during that time span. Each data set has a "commune ID" column, perhaps even named identically. While the semantics of this column are very similar, I wouldn't want to (accidentally) merge the data sets over this commune ID column. Instead, I construct a matching table between "old" and "new" commune IDs. If the commune IDs are encoded as factors, a "safe merge" would give a correctness check for the merge operation at no extra (implementation) cost and very little computational cost.
Comment: IIUC, you'd prefer an error in this case because the levels don't match?
Comment: @Arun: In many use cases I would prefer an error, or at the very least a warning.
Comment: I see. Out of curiosity, in what way do you consider this unsafe?
Comment: Got it, thanks. I'm guessing you're looking for a "cleaner" solution than adding an if-statement to check for the levels.. (on by-columns)?
Answer: The "safe guard" with merge is the by= parameter. You can set exactly which columns you think should match. If you match up two factor columns, R will use the the labels for those values to match them up. So "a" will match with "a" regardless of how the hidden inner working of factor have coded those values. That's what a user sees, so that's how it will be merged. It's just like with numeric values, you can choose to merge on columns that have complete different ranges (the first column has 1:10, the second has 100:1000). When the by value is set, R will do what it's asked. And if you don't explicitly set the by parameter, then R will find all shared column names in the two data.frames and use that.
And many times when merging, you don't always expect matches. Sometimes you're using all.x or all.y to specifically get unmatched records. In this case, depending on how the different data.frames were created, one may not know about the levels it doesn't have. So it's not at all unreasonable to to try to merge them.
So basically R is treating factors like characters during merging, be cause it assumes that you already know that two columns belong together.
Answer: Well, with much credit (and apologies to) MrFlick:
> attributes(ad$x)
$levels
[1] "a" "b" "c"

$class
[1] "factor"

> attributes(ad$a)
NULL

> attributes(ad$b)
NULL

> adfoo<-merge(ad,bd)

> attributes(adfoo$x)
$levels
[1] "a" "b" "c"

$class
[1] "factor"

So in fact  the merged column $x is a factor, although only levels common to both ad and bd are merged.   The other columns  were coerced via as.numeric long ago.
Comment: @MrFlick  My apologies -- hoist by my own petard.  I'll rewrite my answer.
Comment: I don't see where `data.frame` is removing the `factor` attribute from the initial values. What is the evidence of that?
Comment: @MrFlick  try running `attributes(a)` and `attributes(ad)` .  Please don't dump on posts without investigating first.
Comment: I wasn't dumping; I did investigate first which is why I was confused. I had compared `attributes(a)` and `attributes(ad$x)` which were identical. I was surprised to see you looking at `attributes(ad)`. A `data.frame` is just a collection of vectors. The `data.frame` itself doesn't take on the attributes of any one of its columns nor does attributes() recursively investigate the sub-elements. A `data.frame` can hold factors with different levels: `dd<-data.frame(x=letters[1:2], y=letters[11:12]);levels(dd$x); levels(dd$y)`
Comment: Thanks. I wouldn't have expected that. So, the behavior is to create a new factor that contains the levels found in both data frames (modulo `all.x` or `all.y`). Unfortunately, this doesn't seem to be documented (a coarse search in [the docs](http://stat.ethz.ch/R-manual/R-patched/library/base/html/merge.html) for `factor` yields zero hits), but that's just another minor issue. The original question remains: Why??? (And is there an alternative that will do more strict checking?)


Use "dotnet" or ".NETPlatform5.0" in NuGet nuspec?

Question: I have a NuGet .nuspec created with with various framework dependencies.
Based on this article, one of the dependencies is "dotnet":
<group targetFramework="dotnet">
[dependencies here]
</group>

When I pack the .nuspec file using NuGet 3.3, the nuspec file that gets packed up has "dotnet" replaced by ".NETPlatform5.0":
<group targetFramework=".NETPlatform5.0">
[dependencies here]
</group>

What's the impact here, if any?
Should I use "dotnet" or ".NETPlatform5.0" going forward?
Answer: Eventually figured this out myself by reading the docs at NuGet and GitHub
NuGet looks at the "dotnet" moniker and finds the best platform to use, in my case "dotnet5.0", which is actually .NETPlatform5.0.
Confusing as hell, but that's RC life I guess :)


How to check that an object is empty in PHP?

Question: How to find if an object is empty or not in PHP.
Following is the code in which $obj is holding XML data. How can I check if it's empty or not?
My code:
$obj = simplexml_load_file($url);

Comment: possible duplicate of [PHP object holding XML data](http://stackoverflow.com/questions/9411681/php-object-holding-xml-data)
Comment: possible duplicate of [How to check if object is empty in PHP?](http://stackoverflow.com/questions/1389431/how-to-check-if-object-is-empty-in-php)
Answer: You can cast to an array and then check if it is empty or not
$arr = (array)$obj;
if (!$arr) {
    // do stuff
}

Comment: The typecasting doesn't work for me, because I get the error:
    **PHP Parse error:  syntax error, unexpected '(array)' (array) (T_ARRAY_CAST) in ...**

I use PHP version 5.4.28 and the first option with two lines of code works for me.
Comment: empty doesn't actually check if an array is empty, `empty($var)` equivalent to `(!isset($var) || !$var)`. You can replace your `empty($arr)`s with `!$arr` since `array() == FALSE`
Comment: The reason `empty((array) $obj)` does not work in PHP 5.4+ has nothing to do with typecasting on one line. You get the error because `empty()` takes a reference to the variables passed to it, and it cannot get a reference when typecasting unless you first store that typecasted variable in another variable. It does this because of what I described in my previous comment. It's rather frustrating that this is the accepted answer since it can lead people to believe that `empty()` checks if an array is "empty" which is not true - that's just a side effect of what it actually does.
Comment: Had the same error when uploading a WordPress plugin, seems that they also use an older version of PHP.
Comment: Will it work in php 7+, Or is it the right way to fix it.
Answer: Edit: I didn't realize they wanted to specifically check if a SimpleXMLElement object is empty. I left the old answer below
Updated Answer (SimpleXMLElement):
For SimpleXMLElement:
If by empty you mean has no properties:
$obj = simplexml_load_file($url);
if ( !$obj->count() )
{
    // no properties
}

OR
$obj = simplexml_load_file($url);
if ( !(array)$obj )
{
    // empty array
}

If SimpleXMLElement is one level deep, and by empty you actually mean that it only has properties PHP considers falsey (or no properties):
$obj = simplexml_load_file($url);
if ( !array_filter((array)$obj) )
{
    // all properties falsey or no properties at all
}

If SimpleXMLElement is more than one level deep, you can start by converting it to a pure array:
$obj = simplexml_load_file($url);
// `json_decode(json_encode($obj), TRUE)` can be slow because
// you're converting to and from a JSON string.
// I don't know another simple way to do a deep conversion from object to array
$array = json_decode(json_encode($obj), TRUE);
if ( !array_filter($array) )
{
    // empty or all properties falsey
}

Old Answer (simple object):
If you want to check if a simple object (type stdClass) is completely empty (no keys/values), you can do the following:
// $obj is type stdClass and we want to check if it's empty
if ( $obj == new stdClass() )
{
    echo "Object is empty"; // JSON: {}
}
else
{
    echo "Object has properties";
}

Source: http://php.net/manual/en/language.oop5.object-comparison.php
Edit: added example
$one = new stdClass();
$two = (object)array();

var_dump($one == new stdClass()); // TRUE
var_dump($two == new stdClass()); // TRUE
var_dump($one == $two); // TRUE

$two->test = TRUE;
var_dump($two == new stdClass()); // FALSE
var_dump($one == $two); // FALSE

$two->test = FALSE;
var_dump($one == $two); // FALSE

$two->test = NULL;
var_dump($one == $two); // FALSE

$two->test = TRUE;
$one->test = TRUE;
var_dump($one == $two); // TRUE

unset($one->test, $two->test);
var_dump($one == $two); // TRUE

Comment: @juha-untinen The solution with `->count()` is specifically for instances of `SimpleXMLElement` and not for instances of `stdClass`. I believe all of this still works in PHP7.2.
Comment: Doesn't work in PHP7.2: ``PHP Warning:  Uncaught Error: Call to undefined method stdClass::count()``
Answer: You can cast your object into an array, and test its count like so:
if(count((array)$obj)) {
   // doStuff
}

Answer: Imagine if the object is not empty and in a way quite big, why would you waste the resources to cast it to array or serialize it...
This is a very easy solution I use in JavaScript. Unlike the mentioned solution that casts an object to array and check if it is empty, or to encode it into JSON, this simple function is very efficient concerning used resources to perform a simple task.
function emptyObj( $obj ) {
    foreach ( $obj AS $prop ) {
        return FALSE;
    }

    return TRUE;
}

The solution works in a very simple manner: It wont enter a foreach loop at all if the object is empty and it will return true. If it's not empty it will enter the foreach loop and return false right away, not passing through the whole set.
Answer: Using empty() won't work as usual when using it on an object, because the __isset() overloading method will be called instead, if declared.
Therefore you can use count() (if the object is Countable).
Or by using get_object_vars(), e.g.
get_object_vars($obj) ? TRUE : FALSE;

Answer: Another possible solution which doesn't need casting to array:
// test setup
class X { private $p = 1; } // private fields only => empty
$obj = new X;
// $obj->x = 1;

// test wrapped into a function
function object_empty( $obj ){
  foreach( $obj as $x ) return false;
  return true;
}

// inline test
$object_empty = true;
foreach( $obj as $object_empty ){ // value ignored ...
  $object_empty = false; // ... because we set it false
  break;
}

// test
var_dump( $object_empty, object_empty( $obj ) );

Answer: there's no unique safe way to check if an object is empty
php's count() first casts to array, but casting can produce an empty array, depends by how the object is implemented (extensions' objects are often affected by those issues)
in your case you have to use $obj->count();
http://it.php.net/manual/en/simplexmlelement.count.php
(that is not php's count http://www.php.net/count )
Answer: in PHP version 8
consider you want to access a property of an object, but you are not sure that the object itself is null or not and it could cause error. in this case you can use Nullsafe operator that introduced in php 8 as follow:

$country = $session?->user?->getAddress()?->country;

Answer: If you cast anything in PHP as a (bool), it will tell you right away if the item is an object, primitive type or null. Use the following code:
$obj = simplexml_load_file($url);

if (!(bool)$obj) {
    print "This variable is null, 0 or empty";
} else {
    print "Variable is an object or a primitive type!";
}

Comment: wrong. $x = new \stdClass(); var_dump((bool)$x); prints 'true';
In your case, simpleXml may return false in some cases
Comment: Correct. new stdClass() is a valid object, that's why it's true. PHP manual says, simplexml_load_file(), "Returns an object of class SimpleXMLElement with properties containing the data held within the XML document, or FALSE on failure." So, if you cast (bool)simplexml_load_file(), it will be true (because the function returned an object) but false (because the function returned false).
Answer: Based on this answer from kenorb, here's another one-liner for objects with public vars:
if (!empty(get_object_vars($myObj))) { ... }

Edit: Thanks to @mickmackusa's comment below - below is a more succinct one-liner, since this converts the object to an associative array (of accessible properties), and an empty array is falsy in PHP.
if (get_object_vars($myObj)) { ... }

Edit 2: In the event that your object is not a true object, get_object_vars() may produce warnings or alerts. This checks that the var is an object before ensuring that it's populated:
if (is_object($myObj) && get_object_vars($myObj)) { ... }

Just to reiterate - this is for objects with public/accessible variables. Objects with static, private, or protected vars will render false, which may be unexpected. See https://www.php.net/manual/en/function.get-object-vars.php
Comment: `!empty()` is also checking the existence of the passed in argument -- which is unnecessary.  The simper way of doing the same thing is `if (get_object_vars($myObj)) {`
Answer: If an object is "empty" or not is a matter of definition, and because it depends on the nature of the object the class represents, it is for the class to define.
PHP itself regards every instance of a class as not empty.
class Test { }
$t = new Test();
var_dump(empty($t));

// results in bool(false)

There cannot be a generic definition for an "empty" object. You might argue in the above example the result of empty() should be true, because the object does not represent any content. But how is PHP to know? Some objects are never meant to represent content (think factories for instance), others always represent a meaningful value (think new DateTime()).
In short, you will have to come up with your own criteria for a specific object, and test them accordingly, either from outside the object or from a self-written isEmpty() method in the object.
Answer: I was using a json_decode of a string in post request. None of the above worked for me, in the end I used this:
$post_vals = json_decode($_POST['stuff']);

if(json_encode($post_vals->object) != '{}')
{
    // its not empty
}

Answer: Simply check if object type is null or not.
if( $obj !== null )
{
     // DO YOUR WORK
}

Comment: You can't simply check for `null`, `$obj = new \stdClass()` is empty, but not `null`.
Answer: count($the_object) > 0  this is working and can be use for array too
Answer: $array = array_filter($array);
if(!empty($array)) {
    echo "not empty";
}

or
if(count($array) > 0) {
    echo 'Error';
} else {
    echo 'No Error';
}

Comment: While this may answer the question it's always a good idea to put some text in your answer to explain what you're doing. Read [how to write a good answer](http://stackoverflow.com/help/how-to-answer).


barman + postgresql not start after barman retore

Question: I use the barman ( http://www.pgbarman.org/ ) , for backup and restore on my linux redhat machine
after restore , we start the postgresql as the following
 systemctl start postgresql.service

but postgresql not startup
from the log:
  /var/lib/pgsql/backups/data/pg_log

we seen the following errors
 requested recovery stop point is before consistent recovery point

please advice what is the meaning of this FATAL error?
 more pg_log

 LOG:  database system was interrupted; last known up at 2017-08-06 07:30:49 UTC
 LOG:  restored log file "00000008.history" from archive
 LOG:  starting point-in-time recovery to 2017-08-06 07:30:49.813508+00
 LOG:  restored log file "0000000800000000000000C1" from archive
 LOG:  redo starts at 0/C1000080
 LOG:  recovery stopping before commit of transaction 96139, time 2017-08-06 07:30:49.855629+00
 LOG:  redo done at 0/C1001B90
 FATAL:  requested recovery stop point is before consistent recovery point
 LOG:  startup process (PID 55858) exited with exit code 1
 LOG:  terminating any other active server processes

example - when I start the postgres service:
  systemctl start postgresql.service
  Job for postgresql.service failed because the control process exited with
  error code. See "systemctl status postgresql.service" and "journalctl -xe"
  for details.

  ls
  backup_label.old     pg_hba.conf                   pg_hba.conf.2017-08-
  03_13:08  pg_snapshots  pg_xlog
  postgresql.conf.2017-08-03_13:03
  barman_xlog          pg_hba.conf.2017-08-03_12:07  pg_ident.conf
  pg_stat_tmp   postgresql.conf                   postgresql.conf.2017-08-
  03_13:08
  base                 pg_hba.conf.2017-08-03_12:22  pg_log
  pg_subtrans   postgresql.conf.2017-08-03_12:07  postgresql.conf.origin
  global               pg_hba.conf.2017-08-03_12:24  pg_multixact
  pg_tblspc     postgresql.conf.2017-08-03_12:22  postmaster.opts
  pg_clog              pg_hba.conf.2017-08-03_12:28  pg_notify
  pg_twophase   postgresql.conf.2017-08-03_12:24  recovery.done
  pg_hba_bak.conf.old  pg_hba.conf.2017-08-03_13:03  pg_serial
  PG_VERSION    postgresql.conf.2017-08-03_12:28

Comment: You still have the recovery file in the pg data.
Comment: sorry - what you mean about that I still have this file?
Comment: Maybe I misunderstood the timing of the error. Does the restore finish correctly or...? That error sounds like you are trying to do Point In Time Recovery with an older point than the base backup...
Comment: Try removing recovery.conf from the pg data folder.
Comment: @Florin I remove the file and start the postgres , and postgres service was start successfully , so I just asking if by remove this  file ( recover.conf ) is ok and not damage the DB ?
Comment: That file is only required for the Point in Time Restore procedure. When that file is present, the DB will start in recovery mode, not in normal operation mode.
Comment: Do not just remove `recovery.conf`. For some reason you have  recovery target that's too soon after crash recovery / snapshot. What is the exact restore command you used? What's the content of the generated `recovery.conf`? This *should not* happen and it's not necessarily safe to use this database, there could be corruption due to an incomplete restore.
Comment: I use this syntax from barman for restore -  barman recover --target-time "$BEGIN_TIME"  --remote-ssh-command "ssh postgres@$IP"   main-db-server   $ID  /var/lib/pgsql/data
Comment: @shalom You cannot recover to the time you start the backup, so there's your problem. If you want the soonest possible recovery point you can use recovery target 'immediate'.
Answer: You forgot to remove the recover.conf file after restoring.
This file is only required for PiTR and not for normal DB operation. The solution is to simply remove it.
Comment: in other case , after I removed the recovery.conf , still postgres not start , see my update question in the last , do you think I missed something ?
Comment: No, that's not it. This error indicates that a recovery target has been set that's too old for this backup.
Comment: @CraigRinger, I couldn't agree more. But I think that the problem is actually because the transaction logs were already rolled and the OP didn't remove the file. Since the transaction logs were already rolled, of course, the transactions in the DB are newer than the recovery point specified in the recovery file.


laravel 5 show blank page on server error (and no laravel log), running with hhvm and nginx

Question: I followed Fideloper post about running laravel with hhvm, step by step. There are two projects hosted on my server, one is my blog which is based on wardrobe (laravel 4.1), and another project with laravel 5.
My blog has no issue with server errors and laravel log file; But the other project cannot create log files, and it won't show error page because of that.
I double checked storage folder permission. It is 777.
When I run php artisan serve (native php, not hhvm) and browse to it, the error page showed up, and log file was created. So something is wrong with hhvm I think.
How can I fix blank page, when running laravel 5 with hhvm, and an error occurred? How can I bring back the error page (or whoops page)
P.S. I have read related question like this and still have the problem.
P.S.S. If any information needed ask me and I will update the question.
Comment: I figured out what is going on: http://stackoverflow.com/questions/29789945/when-i-turn-on-hhvm-on-homestead-i-dont-get-any-syntax-error-or-missing-clas/30226587#30226587
Comment: You will probably have better luck asking this question on [Server Fault](http://serverfault.com/) which is for system administration questions.
Comment: some grammar fixes.
I think what's needed from the OP is clarification as to what kind of help is sought for.
Comment: HHVM (unfortunately) logs fatal errors to `/var/log/hhvm/error.log`.
Comment: Have you solved this problem yet? I think I've got the same problem. I'm in Homestead and Lumen with HHVM. I get a blank page, no errors show. I can see the errors in `/var/log/hhvm/error.log` though.
Comment: @DavidGraham please post it as an answer
Thank you by the way
Comment: Done. I put up a detailed answer, and made a few adjustments, see answer below
Comment: Updated answer to be more Laravel specific. Also reported this issue/fix on the Laravel github project.
Answer: I suffered the same problem when I installed Laravel Homestead with HHVM. If you enter some random junk like sdfkjl into your routes file you'll get a blank page (however, if add a semi-colon sdfkjl; you get error output). The errors are being logged at /var/log/hhvm/error.log but they don't go to the browser, instead you just get a blank page. It appears this is intentional behavior for HHVM. It also appears Laravel tries to handle these but doesn't catch some of the fatals sent by HHVM. Thanks to clues from this github issue I decided to make some slight changes to Laravel's Foundation\Bootstrap\HandleExceptions.php to see if I could get it to catch all of these fatals:
First, update your /etc/hhvm/php.ini and add these settings:
hhvm.server.implicit_flush = true
hhvm.error_handling.call_user_handler_on_fatals = true

Before modifying a package source, let's remove the vendor\compiled.php with this artisan command:
$ php artisan clear-compiled

And let's set the environment sessons to array:
in your .env
SESSION_DRIVER=array

(You may also need to clear all the random-looking session files in storage/framework/sessions)
Now any changes we make to the Laravel package source will immediately reflect. Let's update a few things in the HandleExceptions class:
in vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php
// **** Add this to hold fatal error
protected static $fatalError = null;

...

public function handleError($level, $message, $file = '', $line = 0, $context = array())
{

   // **** Add this, loads fatal error
   if ($level & (1 << 24)) {
        self::$fatalError = array(
            'message' => $message,
            'type' => $level,
            'file' => $file,
            'line' => $line
        );
    }

    if (error_reporting() & $level)
    {
        throw new ErrorException($message, 0, $level, $file, $line);
    }
}

...

// *** Update this function so it can handle the fatal
public function handleShutdown()
{
    $error = error_get_last();

    if(self::$fatalError){
        $error = self::$fatalError;
    }

    if ( ! is_null($error) && $this->isFatal($error['type']))
    {
        $this->handleException($this->fatalExceptionFromError($error, 0));
    }
}

...

protected function isFatal($type)
{
    // *** Add type 16777217 that HVVM returns for fatal
    return in_array($type, [16777217, E_ERROR, E_CORE_ERROR, E_COMPILE_ERROR, E_PARSE]);
}

...

Now type the random junk (no semi-colon) in your routes folder and you'll see the fatal display. I have now reported this issue to Taylor on the Laravel github. If you are in Lumen, I've got a solution here that will work until Lumen gets fixed as well.
Comment: Thanks alot! After surfing the whole websites that made the debugging to a new level
Answer: I also got the same issue. Give 755 permission to entire laravel project it's work for me
anyway you can check hhvm error log, run
$tail -n 50 -f /var/log/hhvm/error.log

Comment: You're better of either using ACL or chown to allow the webserver to write to the `storage` directories.
Comment: 777 on everything is both dangerous and unnecessary. There are much better ways to set permissions.
Comment: @ceejayoz sorry it is not 777 it should be 755, is it ok?


Deserialize string into values using format pattern

Question: I'm trying to get a good solution for a (de-)serializer.
I've got a format pattern and all the values to put in.
The format pattern is as follows:
msg = '$bla,%d,%02d,%02d %02d:%02d:%02d.%03d' % (kwargs['...'], ...)

When I serialize the values, I get the following string:
bla,1990,12,24 13:37:11.001

But I also have to deserialize it. The pattern can strongly vary in length and types.
I'd like to deserialize the string based only on the format pattern.
Any ideas how this is achievable?
EDIT: I'm using Python 2.7.6
Answer: If you can fully control the protocol, or saying format after serialization, I suggest using some existing solution, e.g Pickle provided by Python standard library, Json which is very popular in web, or Protobuf which is cross-language provided by Google
Pickle:
>>> import pickle
>>> formattuple = (1990,12,24,13,37,11,1)
>>> s = pickle.dumps(formattuple)
>>> s
'(I1990\nI12\nI24\nI13\nI37\nI11\nI1\ntp0\n.'
>>> pickle.loads(s)
(1990, 12, 24, 13, 37, 11, 1)

Json:
>>> import json
>>> formattuple = (1990,12,24,13,37,11,1)
>>> s = json.dumps(formattuple)
>>> s
'[1990, 12, 24, 13, 37, 11, 1]'
>>> json.loads(s)
[1990, 12, 24, 13, 37, 11, 1]

Be aware that json has some limitation, such as a bit more difficult to serialize and de-serialize objects other than dict, list and tuple. And would NOT de-serialize to a totally identical format, because some data structure such as tuple is not existed in json
Protobuf is a more powerful but more complex solution. You need to define the data schema first.
Comment: Unfortunately I don't have control over the protocol. The incoming string always looks similar to the one I posted.
Answer: You might be able to do this with regex, with some assumptions. Here's a partial example, you'd probably need to add to it for a full solution. Basically, we convert each print format, to a regex that matches it, by pieces.
import re
formattuple = (1990,12,24,13,37,11,1)
formatstr = 'bla,%d,%02d,%02d %02d:%02d:%02d.%03d'

def rep_format(fmt):
    fmt = fmt.group(0)
    if fmt[0] != '%':
        return fmt
    if fmt == '%d':
        return r'(\d+)'
    ftype = fmt[-1]
    if ftype == 'd':
       fwidth = int(fmt[1:-1])
       return r'(\d{%d})'%(fwidth)
    else:
       return fmt

scanstr = re.sub(r'%\d+[df]', rep_format, formatstr)
scanstr
'bla,(\\d+),(\\d{2}),(\\d{2}) (\\d{2}):(\\d{2}):(\\d{2}).(\\d{3})'

fstr = formatstr%formattuple
fstr
'bla,1990,12,24 13:37:11.001'

match = re.match(scanstr, fstr)
match.groups()
('1990',
 '12',
 '24',
 '13',
 '37',
 '11',
 '001')
mtuple = tuple(int(x) for x in match.groups())


Scoring Errors in Defensive Judgement

Question: In Baseball, do errors in judgement factor in scoring?
Example: Nobody on base, batter hits single to center field but fielder throws home instead of 2nd
allowing runner to advance to second. Both center fielder and catcher catch the ball.
Example: Runner on 2nd, ground ball to third baseman who runs to touch third base before throwing to first. (Runner on 2nd stays, third baseman just assumed a force) The throw is late and runner is safe at first. First baseman catches the ball.
These appear to be lapses in judgement not in throwing or catching. In the second example it cost the team an out.
Does all the scoring credit go to offense on defensive mental mistakes that are not fielding or throwing errors?
Answer: Example 1 is a bit tricky for me, only because I think it's difficult for a scorer to tell the difference between a CF that is intentionally throwing home versus one that has misplayed a throw to second.  Especially with no one on base, I would not assume that it was an intentional throw home.  I would assume it was a misplayed attempt to throw to second and would award an error.   Maybe if the player were well in and well away from center so that it was obviously a deliberate throw to the plate.
But "throwing to the wrong base" is explicitly called out as not an error.  The MLB Rulebook has a couple of comments on rule 9.12(a)(1).  One is quoted below.  (My emphasis)

The Official Scorer shall not score mental mistakes or misjudgments as errors unless a specific rule prescribes otherwise.  A
fielder's mental mistake that leads to a physical misplay—
such as throwing the ball into the stands or rolling the ball to
the pitcher's mound, mistakenly believing there to be three
outs, and thereby allowing a runner or runners to advance—
shall not be considered a mental mistake for purposes of this
rule and the Official Scorer shall charge a fielder committing
such a mistake with an error. The Official Scorer shall not
charge an error if the pitcher fails to cover first base on a play,
thereby allowing a batter-runner to reach first base safely. The
Official Scorer shall not charge an error to a fielder who incorrectly throws to the wrong base on a play.

The second scenario is clearer to me.  Except for the fact that the fielder ran the ball to the wrong base rather than threw it, the spirit seems the same.  There was no physical misplay, so no error.
I would assume a hit in the second scenario, and in the first if I were able to tell that the throw was intentional.

Scoring Errors in Defensive Judgement


Finding % of remaining students in the class would be girls?

Question: In class of $120$ students, boys constitute $40$% of total. If $\dfrac 13^{rd}$ of boys and $4$ girls drop out of class to join a camp , what % of remaining students in the class would be girls ?
Answer: Total number of students, $ T = 120 $. Number of boys, $ N_B = 40\% \text{ of } T = 0.4 * 120 = 48 $. Thus, number of girls, $ N_G = 120 - 48 = 72 $.
Now, $ \frac{1}{3}^{rd} $ of boys are leaving, $ \implies (\frac{1}{3} * 48) + 4 = 16 + 4 = 20 $ students are leaving.
New count $ N = 120 - 20 = 100 $. New percentage of girls population:
$$ P_G = \dfrac{72 - 4}{N} * 100 \% \ = 68 \% $$
Comment: @SSK There is no name for this formula, AFAIK. The percentages are calculated using the following: $$ \dfrac{\text{Number of girls}}{\text{Total number of students}} $$ Originally, there were $72$ girls of which $4$ left for the camp. Hence, $ 72 - 4 $ in numerator.
Comment: can explain Pg Formula , how did you have 72 ? what is name of this formula ?
Comment: why you took N ]
Comment: @SSK Because the percentage is to be calculated for new number of students, which is $ N $.


VMware Host CPU Usage Climbing

Question: This is somewhat of a longshot, but I am wondering if anyone might be able to explain the following:
I have a VMware Host Server running Ubuntu Server 8.04 LTS, and VMware Server 2.0. The User CPU usage on the host keeps climbing, until I reboot the server when it goes back down to almost nothing - and starts to climb again. This is ever since I swapped a Server 2003 VM for a Server 2008 one (upgraded my DC, and migrated to 2008). I can't find any problem with the Windows server though, or any abnormally high CPU usage on the VM itself.
I have been keeping up-to-date on the host, so I've been through about 3 kernel upgrades, numerous recompiles of VMware, and a whole new VMware Server version when the last one was released not too long ago. I just can't figure this one out.
Any suggestions would be very well appreciated, I'm now just looking for things to try!
Comment: What process is the cpu usage listed under? That seems like the most obvious place to look.
Comment: i have the same.. caused by 32bit w2003 & 64 bit w2008 guests. other [ vista, 32 & 64 bit linux vms ] do not cause this. will be interesting to see the answers.
Comment: Same here.  Noticed it with Windows Server 2008 (32-bit) and three Ubuntu Server 8.04.3 guests. Using Ubuntu Server 8.04.3 as the host with VMware Server 2.0.1.  When I reboot the Ubuntu guests it goes back to nothing then keeps climbing...
Comment: Presumably you have a vmware support contract, what do they say?
Comment: It's a 64-bit host to allow me to address all the RAM, but all the guests are 32-bit. I plan to upgrade the CPU to one supporting VT, and then upgrade the DC to Server 2008 R2 - which means it would have to be a 64-bit guest. That might sort the problem out, as suggested by some answers below.
Answer: I can't answer the question, but I can add an extra little bit of anecdotal evidence. I have noticed this too, with a Linux hosts (Debian/Etch and Debian/Lenny).
After some mucking around when I first noticed the issue, I came to the conclusion that the issue is VMWare, not the guests themselves. Upon stopping all services in the VMs on a particular machine the CPU use remained high despite the OSs in the VMs doing nothing. On shutting down each of the four VMs the excessive host CPU use dropped by about 25% per VM (I didn't measure this by any scientific means, but certainly no one VM seems to be imposing most of the load). After restrting the VMs the CPU use remained where it used to be, even with the services in the VMs active, and the load has begun to slowly rise over time with no related rise in apparent useful activity.
In both the cases that I have noticed this happen, the host OS has been 32-bit Linux and the guest OSs have also been 32 bit Linux.
I have not seen it in all cases though. On my home server (64 bit Linux kernel with 32-bit user-land running one large and two small 32-bit Linux VMS and occasionally Windows VMS for testing) and the main dev/test host at work (64-bit Linux (both kernel and userland) host running mainly Windows VMs, some 32-bit and some 64) this aberrant behaviour doesn't seem to be present. All the above are running VMWare Server 2.
So to cut a long story short: it isn't just you, and it isn't just Windows based guests, but it doesn't seem to be a consistent problem (as many arrangements don't see it as do, in my limited experience). Though unfortunately I can't help any further than that as I've not had the time to look into the problem in more detail.
Comment: Thanks for the help, it makes be feel a bit better that it's VMware and not the guests themselves - though it makes the problem harder to solve!

Hopefully an Intel-VT CPU will help ease the issue, especially if I start moving guests to 64-bit. The new E6300 is a new chip that supports VT, and it's the cheapest out there at the moment.
Comment: My main concern when I first noticed the unexpected CPU use my first fear was that something malicious had managed to get past my defences (I've since ruled this out to my satisfaction). At least when it happens it is not difficult to resolve (unless you are providing a HA service and so rebooting VMS occasionally is a problem)
Answer: First, I've seen this with VMware Server, too - on both Windows and Linux. In my experience it was related to running both 64-bit and 32-bit VMs at the same time.
Though it may not be an option for you, I'd suggest ESXi - the lightweight, free edition of ESX.
Comment: @warren - ESXi will not work with software raid. vmware is quite picky when it comes to hardware compatibility list, they support only 'real' raids / nfs / iscsi.
Comment: I would love to try ESXi, but I am using mdadm software RAID, so I can't get rid of the host linux O/S.
Comment: I haven't tried running ESXi on software RAID, but would expect it would work OK since it's running linux underneath :)
Comment: good to know, since that was how I was planning to deploy it at home
Answer: I experience the same issue on a production server. Host OS is Debian Linux 64 bit. Guests are 4 Linux machines and 1 Windows XP. All guests are 32 bit. When i restart all VM services on the host, the CPU scheduling is fine and host CPU per VM is on par with real guest activity. However after some weeks the CPU scheduling increases until more or less the maximum available CPU time is consumed over the running VMs. The CPU usage on the host at that time is more or less 10x the initial CPU usage of the moment where the VMs just got started.
To me it seems a leak in the CPU scheduling on the host. Instaed of leaking RAM, it's leaking CPU cycles :-). How am I sure this doesnt happen on ESXi?
Thomasgg
Comment: I have yet to hear of this issue in ESXi and if it was an issue, I am sure it would be fixed very quickly as its an enterprise level hypervisor.

VMware Host CPU Usage Climbing


directing a mass of enemies at once

Question: I am working on a simple 2d game where many enemies continually spawn and chase the player or players in python + pygame. A problem I ran into, and one many people that have programmed this type of game have run into is that the enemies converge very quickly. I have made a temporary solution to this problem with a function that pushes any two enemies randomly apart if they are too close to each other.  This works well but is about an O(n^2) algorithm which is run every frame and at high enemies the program starts to slow down.
When my program runs with this function the enemies seem to form round object I nicknamed a "clump".  The clump seems to usually ecliptic but may actually be more complex (not symmetrical) because as the player moves the enemies are being pulled in different directions.  I do like the way this clump behaves, however I am wondering if there is a more efficient way to calculate it.  Currently every enemy in the clump (often >100) is first moved in the direction of the player, and then pushed apart.  If there was instead a way to calculate the figure that the clump creates, and how it moves it would save a lot of computation.
I am not exactly sure how to approach the problem.  It may be possible to calculate where the border of the figure moves, and then expand it to make sure the area stays the same.
Also my two functions currently being used to move enemies:
<code>def moveEnemy(enemy, player, speed):
    a = player.left-enemy.left
    b = player.top-enemy.top
    r = speed/math.hypot(a,b)
    return enemy.move(r*a, r*b)

def clump(enemys):
    for p in range(len(enemys)):
        for q in range(len(enemys)-p-1):
            a = enemys[p]
            b = enemys[p+q+1]
            if abs(a.left-b.left)+abs(a.top-b.top)<CLUMP:
                xChange = (random.random()-.5)*CLUMP
                yChange = ((CLUMP/2)**2-xChange**2)**.5
                enemys[p] = enemys[p].move(int(xChange+.5), int(yChange + .5))
                enemys[p+q+1] = enemys[p+q+1].move(-int(xChange+.5),-int(yChange+.5))
    return enemys
</code>
Edit: some screen shots of how the clump looks:
http://imageshack.us/photo/my-images/651/elip.png/
http://imageshack.us/photo/my-images/832/newfni.png/
http://imageshack.us/photo/my-images/836/gamewk.png/
The clump seems to be mostly a round object just stretched (like an eclipse but may be stretched in multiple directions), however it currently has straight edges due to the rectangular enemies.
Comment: Also, don't do this every frame but rather every X frames. I solved this same problem in a similar way, however I allowed for some overlap. It makes the mass swarm look more dangerous!
Comment: Depending on the quality of answers you get here, you might also like to try asking this on http://gamedev.stackexchange.com/.
Comment: How about the code for range?  Is that doing the distance checking?  That can be very expensive for a large number of units.
Comment: Could you provide some images?
Comment: Is this about ... Zombies? :)
Answer: There are several ways to go about this, depending on your game.  Here are some ideas for improving the performance:

Allow for some overlap.
Reduce your distance checking to be done after a fixed number of frames.
Improve your distance checking formula.  If you are using the standard distance formula, this can be optimized in many ways.  For one, get rid of the square root.  Precision doesn't matter, only the relative distance.
Each unit can keep track of a list of nearby units.  Only perform your calculations between the units in that list.  Every so often, update that list by checking against all units.
Depending on how your game is setup, you can split the field up into areas, such as quadrants or cells.  Units only get tested against other units in that cell.

EDIT: When the units get close to their target, it might not behave correctly.  I would suggest rather than having them home-in on the exact target from far away, that they actually seek a randomized nearby target.  Like an offset from their real target.
I'm sure there are many other ways to improve this, it is pretty open ended after all. I should also point out Boids and flocking which could be of interest.
Comment: Thanks for the response, in answer to your advice: 1. there is already some overlap allowed because when enemies are 5 units apart they are actually overlapped because they are 24 by 24 squares.  2. A good option but then the distance moved must be bigger in order to offset this but then the enemies seem to be jumping around.  3. point taken I will take out the square root.  4.  Seems to be a good option.  5.  I might try this out but I think I would only need one of 4 or 5.  What I am really looking for (if possible) is a way to calculate the whole "clump" at once.
Comment: @user1125600 I could come up with some theoretical algorithms for calculating the entire clump at once.  It would be similar to maximizing the amount of objects in a container.  But I don't think that is a good idea for a real-time game.
Answer: You could define a clump as a separate object with a fixed number of spacial "slots" for each enemy unit in the clump. Each slot would have a set of coordinates relative to the clump center and would either be empty or would hold a reference to one unit.
A new unit trying to join the clump would move towards the innermost free slot, and once it got there it would "stay in formation", its position always being the position of the slot it occupied. Clumps would have a radius much larger than a single unit, would adjust position to avoid overlapping other clumps or loose units that weren't trying to join the clump, etc.
At some point, though, you'd need to deal with interactions for the individual units in the clump, though, so I'm not sure it's worthwhile. I think Austin Henley's suggestion of splitting the field up into cells/regions and only testing against units in nearby cells is the most practical approach.
Answer: I think you're looking for flocking.
Answer: The best intro to flocking / steering behavior movement: http://www.red3d.com/cwr/steer/ . See the attached paper red3d paper. And the related OpenSteer

directing a mass of enemies at once