Calculate power of 2 sided Z A/B test - pooled version

Question: I want to calculate the statistical power of a 2 sided A/B test where I have 2 groups of coin tosses (that is, from 2 different coins) and their respective successes. I want to test whether the two coins have different probabilities.
I'm following page 6 of https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Tests_for_Two_Proportions.pdf, which follows the paper Chow et al. (2008) (I haven't found it online), and states that:
$$1 - \beta = Pr(Z <  \frac{z_\alpha \sigma_{D,p} + (p_1 - p_2)}{\sigma_{D,u}})$$
where:
$n_1$ and $n_2$ are the respective number of coin tosses.
$s_1$ and $s_2$ are the respective successes.
$p_1 = s_1/n_1$, $p_2 = s_2/n_2$.
$q_1 = 1 - p_1, q_2 = 1 - p_2$.
$\sigma_{D,u}$ is the standard unpooled error:  $\sqrt{\frac{p_1 q_1}{n_1} + \frac{p_2 q_2}{n_2}}$
$\bar{p} = \frac{n_1 p_1 + n_2 p_2}{n_1 + n_2}$, $\bar{q} = 1 - \bar{p}$.
$\sigma_{D,p}$ is the standard pooled error: $\sqrt{\bar{p} \bar{q} (\frac{1}{n_1} + \frac{1}{n_2})}$
and $z_\alpha$ seems to be the critical value.
I have no problems with the standard errors and I know I'm calculating them correctly. However, I know I'm not getting a correct statistical power as I compared my results to this tool and I'm getting other values.
I suspect I'm calculating $z_\alpha$ incorrectly. The book states the following:
"Find the critical value (or values in the case of a two-sided test) using the standard normal distribution. The critical value is that value of z that leaves exactly the target value of alpha in the tail."
I'm using $\alpha = 0.05$ (I want a confidence level of $0.95$). What value for $z_\alpha$ should I be using and why? I don't understand how to calculate the $z_\alpha$ that this book describes. I don't understand if, it being a 2 sided test, I should be doing 2 calculations.
For the record, let's assume:
$$n_1 = 81000$$
$$n_2 = 80000$$
$$s_1 = 1600$$
$$s_2 = 1696$$
where $n_1$ and $n_2$ are the respective number of coin tosses with each coin, and $s_1$ and $s_2$ are the respective number of successes.
In this example, as far as I know, the correct power should be (according to an online tool), 83.13%.
This it not for homework; I'm implementing a specialized open source tool and want to share it online.
Comment: Please define symbols that you are using and show formulas for $\sigma_{D,p}, \sigma_{D,u}.$ Questions here should be as self-contained as possible, not relying on users here to find and read relevant parts of a linked (possibly unreviewed) paper.
Comment: you are right, done
Answer: Your notation, and thus details of your Question, are unclear.
Are $S_1$ and $S_2$ observed numbers of successes? If so, I
assume you have $n \approx 80\,000$ and want to to know power
of a test of $H_0: p = 0.20$ vs. $H_1: p\ne 0.20,$ at the 5% level
of significance, to detect differences of about $\pm 0.02.$
Result from Minitab statistical software. If all of that is true, then the following output from a 'power and sample size'
procedure in a recent release of Minitab may be relevant. This procedure
assumes equal sample sizes in the two groups.
Power and Sample Size

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.02
α = 0.05

              Sample
Comparison p    Size     Power
       0.022   80000  0.796665

The sample size is for each group.

Note: There are many versions of tests for differences in binomial success probabilities in various statistical software packages, depending on
whether a pooled or unpooled standard error is used, whether
a continuity correction is used for a normal approximation, and
other considerations. Especially using procedures in online sites,
you need to know the details of the test for which the power
computation is made.

Simulation in R. In R, I used a two-sided prop.test without a continuity correction (on account of
large sample sizes) to distinguish between
numbers of successes from $\mathsf{Binom}(81000, 0.019)$ and
$\mathsf{Binom}(80000, 0.021)$ at the 5% level. In 100,000 such tests
there were $81,769$ rejections. So the power of the test is about 81.8%.
set.seed(1117)
pv = replicate(10^5, prop.test( c(rbinom(1,81000,.019),
               rbinom(1,80000,.021)) , c(81000,80000), cor=F)$p.val)
mean(pv <= 0.05)
[1] 0.81769

The first of these 100,000 tests led to rejection of $H_0$ with P-value $0.014 < 0.05 = 5\%,$ as follows:
set.seed(1117)
x1 = rbinom(1, 81000, 0.019)
x2 = rbinom(1, 80000, 0.021)
x1; x2
[1] 1500
[1] 1663

prop.test(c(x1,x2), c(81000,80000), cor=F)

        2-sample test for equality of proportions
        without continuity correction

data:  c(x1, x2) out of c(81000, 80000)
X-squared = 10.759, df = 1, p-value = 0.001038
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.0036252315 -0.0009127315
sample estimates:
    prop 1     prop 2
0.01851852 0.02078750

Comment: Hi, I really appreciate your answer, but I'm not looking for the statistical power of this particular example. My question (as I stated) is  the meaning of $z_\alpha$ at page 6 of the link and how to calculate it in the pooled version of a 2-sided A/B test.
Comment: I edited my question in an effort to make it clearer
Comment: Moreover, please notice that in my scenario I have an unequal sample size
Comment: Yes. You may have trouble finding software or an online calculator that accepts different $n$'s as input. The most powerful configuration is a balanced design. If the two $n$/s are nearly the same, then use the smaller for such procedures. If you can find a formula that works for the style of test you're using, then you can use different sample sizes. And (as above) you can use different sample sizes in a simulation.
Comment: This tool: https://abtestguide.com/calc/ accepts both n's, I think - and for what I am doing I would like to do the same


A mixed problem !!!

Question: Let $p=2021$ be a prime number. Let E be the expected value of the expression$$ 3□3□3□⋯□3□3$$ where there are $p+3$ threes and $p+2$ boxes, and one of the four arithmetic operations {${+,−,×,÷}$} is uniformly chosen at random to replace each of the boxes. If $E=\frac{m}{n}$ , where m and n are relatively prime positive integers, find the remainder when m+n is divided by p.
My Approach:
I used python to answer the question as I am not getting the question. My code,
from random import choice
from fractions import Fraction as f
a = ['+','-','*','/']
b = 3
for i in range(1,2024):
 c = choice(a)
 if c == '+':
     b += 3
 elif c == '-':
     b -= 3
 elif c == '*':
     b *= 3
 else:
     b /= 3

The code gives different answers, but none is the correct, can anyone give me hint to solve the problem??
Comment: Your first supposal is not correct: Let $p=2021$ be a prime number.
Comment: This seems likely to come down to ambiguities in expressions of the form $3\div 3\times 3$ and the like. b But what do you mean the code yields "different answers"?
Comment: Your code considers all operators as the same level of precedence, so for example $3+3\times 3 + 3 := (((3+3)\times 3)+3$. Which may not be what the question expects.
Comment: What is the relation between the choosing of the arithmetic operations and the values of m and n? It seems they are independents.
Comment: That $p$ is prime (or not) is anyway utterly irrelevant for this problem.
Comment: If we have each choice for every box, the problem is incredibly complicated. If we have the choice only once, how do we interprete the expression if every box has a ":" ? I guess we just divide repeatedly by $3$. In this case, we just have to consider the $4$ possible results.
Answer: Hint: Condition on the last box that is not $\times $ or $ \div$.
There is an equal chance that it is $ + $ or $ - $.
Condition on sequences suitably to show that the expected value of the tail-end terms cancels out.
Keep on going.
Show that we just need to consider the first block of terms which only use $ \times $ or $ \div$, then weight by the probability of the length.
If there were $n$ boxes originally, then what is the probability that the first block of terms which only use $\times$ or $\div$ has $k$ boxes?
If there are $k$ boxes with only $ \times $ or $\div $ involved, then what is the expected value? (Use your favorite order of operations interpretation, which I hope is left to right for only $\times$ and $\div$ involved.)
Hence, what is the expected value for $n+1$ boxes?

A mixed problem !!!


Centering AChartEngine Dial chart

Question: I'm totally lost here. It should be dead simple but I can't center a Dial chart from AChartEngine in my layout.
I am following the Demo example to the letter but it still does not work for me and I'm out of ideas!
My last layout test follows:
<LinearLayout
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:layout_weight="1"
    android:gravity="center"
    android:orientation="horizontal" >
<TextView
    android:id="@+id/txt"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:layout_weight="1"
    android:gravity="center"
    android:text="TEST-TEST" />

<LinearLayout
    android:id="@+id/graph1"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:layout_weight="1"
    android:gravity="center" />
</LinearLayout>

I get a perfectly centered "TEST-TEST" text and the dial to the right of that but around 55% of the screen size, clearly to the left of where it should be centered. It's actually not getting clipped to the second layout because of the margins (I am using the demo ones too).
If I change the graph for another textview I get it perfectly centered as expected.
Ideas anyone?
Comment: I can add a new bit of interesting evidence here. I migrated BudgetChart demo into my app. Same code AND same layout as used in the demo. The chart shows completely to the left, half chart get's clipped as its drawn outside from the screen. The only difference here is that the layout lives in a frame shown with a SectionsPagerAdapter, could aChartEngine have problems finding it way around on a layout inside a fragment?
Comment: More potentially relevant info. The call used in the demo to add the view 'layout_eeg1.addView(view_eeg1); //, new LinearLayout.LayoutParams(LayoutParams.FILL_PARENT, LayoutParams.FILL_PARENT));' gets flagged as the constants are deprecated. I'm building against Android 4.2. IF I use something like LinearLayout.LayoutParams(200, 200) it does get centered. This is not a solution though as I'm fix the layout size and making it impossible to work for several devices.
Answer: Try adding this call:
renderer.setInScroll(true);

This will help in cases when the chart view is added to a custom layout.

Centering AChartEngine Dial chart


How to use @Valid annotation on return type of a internal method

Question: I have a REST API, inside which I'm transforming the request passed. Now I want to validate the entity which was transformed using @Valid annotation.
<code>public RequestResponse create(EntityV1 entityV1){
    Entity entity = transform(entityV1)
}

private Entity transform(EntityV1 entityV1){
    Entity entity = new Entity();
    //Transformation logic
    return entity;
}
</code>
Please suggest me a way through which the return type of transform can be validated using my custom annotation.
Constraint:
I cannot change the Input of my rest api from EntityV1 to Entity
Answer: You will need to programmatically validate the bean as follows:
<code>@Autowired
Validator validator;

public RequestResponse create(EntityV1 entityV1, BindingResult bindingResult){
    Entity entity = transform(entityV1);

    validator.validate(entity, bindingResult);
    if (bindingResult.hasErrors()) {
        // validation errors
    } else {
        // no errors
    }
}
</code>


Derivation of the balance of kinetic energy from the momentum balance using the continuity equation

Question:
Derive the balance of kinetic energy from the momentum balance using
the continuity equation.

The momentum equation in 3D reads as follows:
$$ \frac{\partial\rho \mathbf{v}}{\partial t} + \nabla\cdot(\rho \mathbf{v}\otimes\mathbf{v}) = \nabla\cdot\sigma + \mathbf{f_b}. $$
The continuity equation is given by:
$$\frac{\partial \rho}{\partial t} + \nabla\cdot(\rho\mathbf{v}) = 0.$$
In the exercise there is a hint to take the inner product of the velocity $\mathbf{v}$ with both
sides of the momentum equation. Then to rewrite that with the help of the continuity equation to obtain the balance equation for
the kinetic energy density:
$$\frac{\partial }{\partial t}\left(\frac{1}{2}\rho v^2\right) = -\nabla\cdot\left(\frac{1}{2}\rho v^2\mathbf{v}\right) + \mathbf{v}\cdot(\nabla\cdot\sigma) + \mathbf{v}\cdot\mathbf{f_b}.$$
Here $v^2 := \mathbf{v}\cdot\mathbf{v}$.
What I've tried is to take the inner product with $\mathbf{v}$ and apply the product rule for derivatives:
\begin{align}
  \mathbf{v}\cdot\left(\rho\frac{\partial\mathbf{v}}{\partial t}\right) + \mathbf{v}\cdot\left(\frac{\partial\rho}{\partial t}\mathbf{v}\right) + \mathbf{v}\cdot(\rho\mathbf{v}\cdot\nabla\mathbf{v}) + \mathbf{v}\cdot(\mathbf{v}\nabla\cdot(\rho\mathbf{v})) = \mathbf{v}\cdot(\nabla\cdot\sigma) + \mathbf{v}\cdot\mathbf{f_b}
\end{align}
Now the second and fourth term on the LHS should cancel out due to the continuity equation, right? But how will I now ever arrive at the correct equation? I don't get where the halfs are coming from. Should I maybe split the second term into half and substitute $\frac{\partial \rho}{\partial t} = - \nabla\cdot(\rho\mathbf{v})$ into only one of them? I still don't see how some of the other terms cancel out then. Help would be much appreciated.
Answer: Taking the dot product of $\mathbf{v}$ and the left-hand side of the momentum equation and using Cartesian components with the Einstein convention for summation over repeated indexes, we get
$$\tag{1}\mathbf{v} \cdot \left[\frac{\partial \rho \mathbf{v}}{\partial t} + \nabla \cdot (\rho \mathbf{v}\mathbf{v}) \right]= v_i \frac{\partial}{\partial t}(\rho v_i)+ v_i\frac{\partial}{\partial x_j}(\rho v_j v_i) \\= v_i^2 \frac{\partial\rho}{\partial t}+ \rho v_i\frac{\partial v_i}{\partial {t}}+ v_i^2\frac{\partial}{\partial x_j}(\rho v_j) + \rho v_j v_i\frac{\partial v_i}{\partial x_j}$$
By the product rule it follows that
$$ \tag{2}\frac{\partial }{\partial t}\left(\frac{1}{2}v_i^2\right)=\frac{1}{2} \frac{\partial v_i^2}{\partial t} = \frac{1}{2}\left[2 v_i\frac{\partial v_i}{\partial t}\right] =  v_i\frac{\partial v_i}{\partial t}, \\ \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right)=\frac{1}{2} \frac{\partial v_i^2}{\partial x_j} = \frac{1}{2}\left[2 v_i\frac{\partial v_i}{\partial x_j}\right] =  v_i\frac{\partial v_i}{\partial x_j}$$
Substituting into (1) with (2), we obtain
$$\mathbf{v} \cdot \left[\frac{\partial \rho \mathbf{v}}{\partial t} + \nabla \cdot (\rho \mathbf{v}\mathbf{v}) \right]\\= v_i^2 \frac{\partial\rho}{\partial t}+ \rho \frac{\partial }{\partial {t}}\left(\frac{1}{2}v_i^2\right)+ v_i^2\frac{\partial}{\partial x_j}(\rho v_j) + \rho v_j \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right)\\ = v_i^2 \frac{\partial\rho}{\partial t}+ v_i^2\frac{\partial}{\partial x_j}(\rho v_j) + \rho \frac{\partial }{\partial {t}}\left(\frac{1}{2}v_i^2\right)+ \rho v_j \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right)\\ = \frac{1}{2} \left[v_i^2 \frac{\partial\rho}{\partial t}+ v_i^2\frac{\partial}{\partial x_j}(\rho v_j)  \right]+  \frac{1}{2} \left[v_i^2 \frac{\partial\rho}{\partial t}+ v_i^2\frac{\partial}{\partial x_j}(\rho v_j)  \right]+  \rho \frac{\partial }{\partial {t}}\left(\frac{1}{2}v_i^2\right)+ \rho v_j \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right)\\ = \frac{v_i^2}{2} \left[ \frac{\partial\rho}{\partial t}+ \frac{\partial}{\partial x_j}(\rho v_j)  \right]+  \frac{1}{2} v_i^2 \frac{\partial\rho}{\partial t}+  \rho \frac{\partial }{\partial {t}}\left(\frac{1}{2}v_i^2\right)+  \frac{1}{2}v_i^2\frac{\partial}{\partial x_j}(\rho v_j) +\rho v_j \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right)$$
By the continuity equation, the first term on the RHS vanishes and we get
$$\tag{3}\mathbf{v} \cdot \left[\frac{\partial \rho \mathbf{v}}{\partial t} + \nabla \cdot (\rho \mathbf{v}\mathbf{v}) \right]\\=\frac{1}{2} v_i^2 \frac{\partial\rho}{\partial t}+  \rho \frac{\partial }{\partial {t}}\left(\frac{1}{2}v_i^2\right)+  \frac{1}{2}v_i^2\frac{\partial}{\partial x_j}(\rho v_j) +\rho v_j \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right),$$
By the product rule we can combine the second and third terms on the RHS of (3) as
$$\frac{1}{2} v_i^2 \frac{\partial\rho}{\partial t}+  \rho \frac{\partial }{\partial {t}}\left(\frac{1}{2}v_i^2\right) = \frac{\partial}{\partial t}\left(\frac{1}{2}\rho v_i^2\right), $$
and similarly we can combine the fourth and fifth terms on the RHS of (3) to get
$$\frac{1}{2}v_i^2\frac{\partial}{\partial x_j}(\rho v_j) +\rho v_j \frac{\partial }{\partial x_j}\left(\frac{1}{2}v_i^2\right) = \frac{\partial}{\partial x_j} \left[ \left(\frac{1}{2} \rho v_i^2\right) v_j \right]$$
Substituting into (3) yields
$$\mathbf{v} \cdot \left[\frac{\partial \rho \mathbf{v}}{\partial t} + \nabla \cdot (\rho \mathbf{v}\mathbf{v}) \right]\\=\frac{\partial}{\partial t}\left(\frac{1}{2}\rho v_i^2\right)+\frac{\partial}{\partial x_j} \left[ \left(\frac{1}{2} \rho v_i^2\right) v_j \right] \\ = \frac{\partial}{\partial t}\left(\frac{1}{2}\rho |\mathbf{v}|^2\right)+\nabla \cdot \left(\frac{1}{2} \rho |\mathbf{v}|^2 \mathbf{v} \right),$$
where under the Einstein convention $v_i^2 = v_1^2+v_2^2 + v_3^2 = |\mathbf{v}|^2$ (the same as $v^2$ in your notation).


Are there conflicting requirements for technique between large and small brass instruments

Question: I want to know some practical details on what to consider if you want to play different brass instruments of various size. Specially differences between the extremities like trumpet and tuba. Is there any reason to not mix because trumpet requires too different muscle training in the lips compared to playing tuba?
I am most interested in focusing on air use and mouth techniques and physics, not so much fingering.
Edit:
What I'm really looking for is if it hurts technique for my tuba playing to also start playing a small instrument like cornet or trumpet. This is only for hobby, not professionally, so it would not be a big amount of practice involved.
Comment: Do mean practice techniques of playing both in the same day? Do you mean learning to play both instruments?  Air, pressure, embouchure technique, and physicality for each instrument is entirely different, though you can learn to play both.  I play both. :)
Comment: What is it you really want to do?  Unless you're going pro, it's purely a matter of how much practice time you have available, and what instruments you *want* to play.
Answer: I'm a professional tubist, and I also play trumpet as well - among other noise makers.  I can tell you first-hand that playing trumpet or other instruments will generally not affect your ability to play tuba.  It is good you are staying within the brass family as it is the least discouraging.
When moving through brass instruments of different sizes, it is best to go from largest mouthpiece to smallest.  So, in a given day, it's better to play tuba first before heading into trumpet land.
Due to the nature of each instrument's technique, it will take you a long time to develop a high register on the trumpet (though your low register will be great!) So keep that in mind.  It will also take a long time to develop stamina - even if your stamina for tuba is great.
What can be damaging to brass embouchure is moving from woodwinds to brass within the same day.  Saxophone and clarinet are the worst.  Flute is probably the best.
In summation: Play trumpet and have fun, just make sure you get your tuba practice in first.
Comment: Because playing saxophone or clarinet causes you to curl your lower lip inward over your teeth.  Your teeth rub against the lip and make it sore / raw.  Tuba players are useless without a solid lower lip. In addition, those two instruments use completely different muscles than for tuba, and can often leave a player with sore muscles that need to be otherwise floppy and relaxed for proper tuba performance.  Though flute is a woodwind, it is actually a great instrument to play *before* tuba practice as it engages the lungs in a special way.
Comment: Good answer. One question though, why do you say it is damaging to go from woodwind to brass in the same day?
Comment: Yes, the lower lip touches the teeth.  No, this shouldn't make it sore or raw.
Answer: Most brass band players I know can play more than one instrument: after all, this is the main point of the "instruments in different keys" system for three-valved instruments. That said, when a player ends up playing different instruments in different bands (so they have to switch over the course of each week), they usually find that a little uncomfortable and try to resolve the situation by switching seats in one band.
There are two big differences between instruments. One is the amount of air. It takes a lot more air to fill a tuba than a cornet, but I think this shouldn't be a problem going from large to small. Sometimes players of cornets have trouble going the other way, because they're used to stretching the air out for long, twiddly passages, rather than pushing lots of air out for one long note.
The other main difference is in embouchure. Different brass instruments require completely different embouchure to each other. This, I think, is the main difficulty in switching. It's not that one instrument makes your technique worse at the other, or makes you forget how to play: more that, if you've been practising with one embouchure all week, and you sit down to play, you have to remind your lips to do the other embouchure, and it can make them tire quickly.
Thinking of it like changing car from a manual (stick) to an automatic is probably a helpful analogy. Driving an automatic doesn't make you forget how to drive a manual, but you have to get back into the habit of working the gears and giving it more or less gas in different situations.The gears are the embouchure and the gas is your breathing! A one-off change is easy to get used to, but if you keep switching, it's easier to get confused and stall at a junction.
One more thing to bear in mind is that the style of music is quite different. Cornets and trumpets often have very "nimble" music: fiddly, with lots of short notes, double-tonguing, and interesting rhythms. Tubas tend to have boring music, with lots of long notes. (Maybe this is why you want to try something new!) Unless you already play other instruments too, you may find that learning to read faster and change fingerings more quickly is a bigger hurdle for you than the change in instrument.
In summary, I don't think you should be afraid that learning a new instrument will hurt the old one. In the long term, the new perspective, and opportunities to play different music, will probably help your playing overall. Just be prepared for your tuba playing to feel harder until you get used to switching back and forth, and be ready to do just as much "lip practice" on trumpet or cornet as if it were your first instrument.
Comment: About the "style of music is quite different", I would just see as a bonus to expand the repertoire on my tuba playing. As to "boring" is more what you make of it. The conductor in the wind band where I play often like us to play the parts that are written for string/EL bass if those has parts that is missed in the tuba arrangement. So that often includes more interesting play. Also I often encounter interesting parts originally arranged for tuba that are both difficult and/or cool passages.
Comment: I am ***not*** thinking *switching* to another instrument. I love to play tuba, and although I am thinking of trying other instruments, I am probably not going to go away from tuba as my main instrument.
Comment: All the tuba players I know complain about their parts being boring. Of course, there are exceptions (tuba solos tend to be quick), but for the most part you're anchoring the chords and rhythm with long, repetitive notes. That's why I say an inexperienced player might find the faster-paced music for smaller instruments tricky, but rewarding in terms of broadening your musical understanding (and making those tricky solos easier).


I have proved that $[a,b]\sim[0,1]$. How similar is the proof with open sets instead of closed?

Question: To prove the original statement, I invented (or discovered?) a function $f:\mathbb [a,b]\to[0,1]$ such that $f(x)=\frac1{b-a}x-\frac{a}{b-a}$ and proved that the function is bijective. Now I'm being asked to prove a similar relation, $(a,b)\sim(0,1)$. Would the same function, with similar ideas work in this case?
(I'm not sure if my tag is very adequate for this.)
Comment: Look at the graph of your function, or simply compute $f(a)$ and $f(b)$; you'll see that the same function works.
Answer: As your bijection $f \colon [a,b]\to [0,1]$ maps $a$ to $0$ and $b$ to $1$, its restriction to $(a,b)$, that is the map $f|_{(a,b)}$ is a bijection $(a,b) \to (0,1)$.
Comment: Hmm, is it really that simple? I can just change the codomain like that and not justify it? What if I wanted to restrict $f$ to $(a,\frac{a}2)$, I could just say $f|_{(a,\frac{a}2)}$ is a bijection $(a,\frac{a}2)\to (0,\frac12)$?
Comment: Yes, it is - you did an error in the calculation of the domain though, instead $\frac a2$, it's $\frac{a+b}2$, the midpoint - for every $I \subseteq (a,b)$, $f|_I$ is a bijection $I \to f(I)$.
Answer: Consider $f(x) = \dfrac{b-x}{b-a}$ which is almost identical to the one you have, but it maps $(a,b)$ to $(0,1)$ in the opposite way.Another way to prove this is: the function $f(x) = \arctan \left(\dfrac{x-a}{b-a}\right)$ maps $(a,b)$ to $(0,\frac{\pi}{4})$ bijectively, and the map $g(x) = \dfrac{4x}{\pi}$ also maps $(0,\frac{\pi}{4})$ to $(0,1)$ bijectively. Hence the function $h(x) = (g\circ f)(x)$ maps $(a,b)$ to $(0,1)$ bijectively, proving that $(a,b) \sim (0,1)$
Comment: I used something like this to prove that $(0,1)\sim \mathbb R$, but thanks for the info!


foreach to build an HTML table from data in an array

Question: The following code tests to see how certain values come out against PHP functions:
//array to hold test values
$values = array
(
    'empty string' => '',
    'space' => ' ',
    'false' => false,
    'true' => true,
    'empty array' => array(),
    'null' => null,
    '0 as string' => '0',
    '0 as integer' => 0,
    '0 as float' => 0.0,
    '$var declared but without a value' => $var
);

foreach($values as $key => $value):
    $result['isset'][$key] = isset($value);
    $result['empty'][$key] = empty($value);
    $result['is_null'][$key] = is_null($value);
    $result['is_numeric'][$key] = is_numeric($value);   //added this line on 10/7/13
    $result['strlen'][$key] = strlen($value);   //added this line on 12/23/13
endforeach;

//view results by function
echo "
view results by function

" . print_r($result,1) . "
";

The array outputted from this code is as follows:
Array
(
    [isset] => Array
        (
            [empty string] => 1
            [space] => 1
            [false] => 1
            [true] => 1
            [empty array] => 1
            [null] =>
            [0 as string] => 1
            [0 as integer] => 1
            [0 as float] => 1
            [$var declared but without a value] =>
        )

    [empty] => Array
        (
            [empty string] => 1
            [space] =>
            [false] => 1
            [true] =>
            [empty array] => 1
            [null] => 1
            [0 as string] => 1
            [0 as integer] => 1
            [0 as float] => 1
            [$var declared but without a value] => 1
        )

    [is_null] => Array
        (
            [empty string] =>
            [space] =>
            [false] =>
            [true] =>
            [empty array] =>
            [null] => 1
            [0 as string] =>
            [0 as integer] =>
            [0 as float] =>
            [$var declared but without a value] => 1
        )

    [is_numeric] => Array
        (
            [empty string] =>
            [space] =>
            [false] =>
            [true] =>
            [empty array] =>
            [null] =>
            [0 as string] => 1
            [0 as integer] => 1
            [0 as float] => 1
            [$var declared but without a value] =>
        )

    [strlen] => Array
        (
            [empty string] => 0
            [space] => 1
            [false] => 0
            [true] => 1
            [empty array] =>
            [null] => 0
            [0 as string] => 1
            [0 as integer] => 1
            [0 as float] => 1
            [$var declared but without a value] => 0
        )

)

The results would be much more friendly to view in an HTML table.  So far I have only had success building the row that contains th tags:
$html = "\n
\n
\n\ttest	\n";
//table header row
foreach ($result as $key => $value):
    $html .= "\t$key	\n";
endforeach;
$html .= "
\n";
//add detail rows here
$html .= "
\n";

I am having much difficulty figuring out how to build the detail rows properly.  The goal is to display one test name per row ().  The tests are the keys in the $values array.  The s would be the test name, answer to isset, answer to empty, answer to is_null, answer to is_numeric, answer to strlen.  How can this be achieved using foreach loops?
Answer: I think this is much easier to pull off if you rearrange the array. Here's what I did.
$values = array
(
    'empty string' => '',
    'space' => ' ',
    'false' => false,
    'true' => true,
    'empty array' => array(),
    'null' => NULL,
    '0 as string' => '0',
    '0 as integer' => 0,
    '0 as float' => 0.0,
    '$var declared but without a value' => @$var
);
$tests = array('isset','empty','is_null','is_numeric','strlen');
?>

        test
        <?=implode("	\n",$tests)?>

<?php foreach($values as $key=>$value): ?>

        <?=$key?>
        <?php foreach($tests as $test): ?>

            <?php if($test == 'isset'): ?>
                <?=(int)isset($value)?>
            <?php elseif($test == 'empty'): ?>
                <?=(int)empty($value)?>
            <?php else : ?>
                <?=@(int)$test($value)?>
            <?php endif; ?>

        <?php endforeach; ?>

<?php endforeach; ?>

Now you can add more tests if you want to the tests array easily and you don't need to use \n and \t to format the markup except in the implode. the @ symbols are to suppress warning the code generates in 2 place - one when using an undefined variable and 2 - when trying to use strlen on an array.
The reason for the conditionals in the inner foreach is because isset and empty are language constructs and not functions, so they cannot be called dynamically.
I know that's not really what you asked for, but I learned something playing around with this so figured I would share.
Answer: Add the following lines at the end of your code:
print("

    isset
    empty
    is_null
    is_numeric
    strlen

".PHP_EOL);
foreach($values as $key => $value):
print("

    $key
    {$result["isset"][$key]}
    {$result["empty"][$key]}
    {$result["is_null"][$key]}
    {$result["is_numeric"][$key]}
    {$result["strlen"][$key]}

".PHP_EOL);
endforeach;
print("
");

Answer: Here is the complete code. I think it should work for you.
<?php
$var="";
$values = array
(
    'empty string' => '',
    'space' => ' ',
    'false' => false,
    'true' => true,
    'empty array' => array(),
    'null' => null,
    '0 as string' => '0',
    '0 as integer' => 0,
    '0 as float' => 0.0,
    '$var declared but without a value' => $var
);

foreach($values as $key => $value)
{
    $result['isset'][$key] = isset($value);
    $result['empty'][$key] = empty($value);
    $result['is_null'][$key] = is_null($value);
    $result['is_numeric'][$key] = is_numeric($value);   //added this line on 10/7/13
    if(!(is_array($value)))
    $result['strlen'][$key] = strlen($value);   //added this line on 12/23/13

}
//echo "
view results by function

" . print_r($result,1) . "
";
?>

        Key
        Value

<?php
foreach ($result as $key => $value):

    foreach ($value as $k => $val)
    {
?>

<?php echo $k ?>	<?php echo $val ?>

<?php
    }
endforeach;
?>

Answer: I would suggest you to "rotate" your array structure around first. From "result -> columns -> rows" to "result -> rows -> columns"
foreach($values as $key => $value):
    $result[$key]['isset'] = (int)isset($value);
    $result[$key]['empty'] = (int)empty($value);
    $result[$key]['is_null'] = (int)is_null($value);
    $result[$key]['is_numeric'] = (int)is_numeric($value);
    $result[$key]['strlen'] = (int)strlen($value);
endforeach;

$html = "
\n";

$html .= "
";
$first_row = reset($result);
foreach ($first_row as $key => $value):
    $html .= "$key	";
endforeach;
$html .= "
\n"

foreach ($result as $row):
    $html .= "
";
    foreach($row as $key => $value):
        $html .= "$value	";
    endforeach;
    $html .= "
\n";
endforeach;

$html .= "
\n"


Getting a laravel run time exception when doing a video upload from post

Question: The error is SplFileInfo::getSize(): stat failed for C:\wamp\tmp\php2625.tmp
Controller action:
public function store() {
<code>$file = Input::file("video");

$title = Input::get("title");
$name = $file->getClientOriginalName();

$file->move('../../public/videos/' . $name);

$size = $file->getSize();

$post = Post::create(["title" => $title, "user_id" => Auth::id()]);

Video::create(["name" => $name, "post_id" => $post->id ]);

return Redirect::route("my_videos");
</code>
}
$size = $file->getSize(), apparently I cannot retrieve the size of the file, according to laravel debugger this seems to be causing the error.
I am uploading a 200kb file and upload_max_filesize = 64M, post_max_size = 10M in php.ini
How do I resolve this?
Comment: Does the file exist? Does your form have `enctype="multipart/form-data"` to support uploads?
Comment: Sorry for the delay.

Yes the file does exist and I've used entype, I'll attach my form code.
Comment: Cant seem to paste in blade syntax html without it getting very messy, but again, I did use enctype. The file gets read properly, I've checked that, for some reason I cannot retrieve the file size though.
Answer: Put the code for
                    <code>$size = $file->getSize();</code>
before move operation
<code>          $size = $file->getSize();
          $file->move('../../public/videos/' . $name);
</code>


Why does this program output different answers when using floats vs. ints?

Question: I did a program that will find factorial of a number. But for the integer data type answer is up to 12 and for float, it's 13. But sizeof both integer and float are 32. Why am I seeing different answers?
<code>#include<stdio.h>
int main()
{
    int n,i=1,fact=1;

    scanf("%d",&n);
    while(i<=n)
    {
        fact=fact*i;
        i=i+1;
    }
    printf("fact is %d\n",fact);
    return 0;
}
</code>
Answer: They're two completely different number representations. There is no reason to expect them to be the same. They're giving different answers because they're totally different.
Arabic numbers can count up to 999 with three places. Roman numerals can't count past III. Why the difference? Well, because the two number systems are completely different ways of representing numbers and use the same amount of space in completely different ways.
Answer: I think the problem lies in that when you used floats, you didn't change the scanf and thus tried to read an integer representation into a floating point variable, as they are the same size it will fit but as David rightly points out they use the 32 bits in completely different ways.
If you read the value correctly into the Float the answer would have been the same.
You would also need to change the printf too.


How should I evaluate the flux?

Question: I have the following vector field
$F= \frac{11x}{\left(x^2+y^2+z^2\right)^{3/2}} \vec{i} + \frac{11y}{\left(x^2+y^2+z^2\right)^{3/2}} \vec{j} + \frac{11z}{\left(x^2+y^2+z^2\right)^{3/2}} \vec{k}$
I need to calculate the flux, $\iint_{S} F \cdot \vec{n}\ \text{dS}$, through the surface $S: x^2+y^2+z^2 = 16$ (sphere). As $F$ has a singularity at origin, I can't use the Divergent Theorem (right?). So I try to calculate using the surface integral of a vector field, $\iint_{S} F \cdot \vec{n}\ \text{dS} = \iint_{D} F(\mathbf{r}(\mathbf{u},\mathbf{v})) \cdot (\mathbf{r_u} \times \mathbf{r_v})\ \text{dA}$.
I've used the parametric representation $\mathbf{r}(\phi,\theta) = (4 \sin\phi\cos\theta, 4 \sin\phi\sin\theta, 4 \cos\phi)$. When I perform $F(\mathbf{r}(\mathbf{u},\mathbf{v})) \cdot (\mathbf{r_u} \times \mathbf{r_v})$, I'm getting 0. With this result, my flux would be 0, but according to my teacher answer, the flux is $44\pi$. What I'm doing wrong?
Comment: @Arthur, I've changed my question to be clearer.
Comment: $\boldsymbol F$ and $d\boldsymbol S$ are collinear, clearly the dot product cannot be zero. We have $\boldsymbol F = 11 \boldsymbol r/r^3$ and $\hat {\boldsymbol n} = \boldsymbol r/r$, so
$$\iint_{r = R} \boldsymbol F \cdot d\boldsymbol S =
\frac {11} {R^2} \iint_{r = R} dS.$$
Answer: In spherical coordinates,
$x = r \cos \theta \sin \phi, y = r \sin \theta \sin \phi, z = r\cos \phi$
$F = \frac{11x}{\left(x^2+y^2+z^2\right)^{3/2}} \vec{i} + \frac{11y}{\left(x^2+y^2+z^2\right)^{3/2}} \vec{j} + \frac{11z}{\left(x^2+y^2+z^2\right)^{3/2}} \vec{k}$
$F = (\frac{11\cos \theta \sin \phi}{r^2}, \frac{11\sin \theta \sin \phi}{r^2}, \frac{11 \cos \phi}{r^2})$
Outward normal vector $\hat{n} = \frac{1}{r}(x, y, z) = (\cos \theta \sin \phi \, , \sin \theta \sin \phi \, , \cos \phi \,)$
In spherical coordinates, $dS = r^2 \sin \phi \ d \theta \, d \phi$
$Flux = \displaystyle \iint_S \vec{F} \cdot \hat{n} dS$
$\vec{F} \cdot \hat{n} dS = (\frac{11\cos \theta \sin \phi}{r^2}, \frac{11\sin \theta \sin \phi}{r^2}, \frac{11 \cos \phi}{r^2}) \cdot (\cos \theta \sin \phi \, , \sin \theta \sin \phi \, , \cos \phi \,) dS$
$ = 11(\cos^2 \theta \sin^2 \phi + \sin^2 \theta \sin^2 \phi + \cos^2 \phi) \sin \phi d\theta d\phi = 11 \sin \phi d\theta d\phi$
So, $Flux = \displaystyle \iint_S \vec{F} \cdot \hat{n} dS = 11 \int_0^\pi \int_0^{2\pi}\sin \phi d\theta d\phi = \fbox {44 $\pi$}$
Comment: Surface integral is directly calculating flux over the _surface_ of the sphere so it works. Divergence theorem says this surface integral (flux) is same as volume integral of the divergence _inside_ the surface. Now you can see why continuity issue at origin matters when we want to use  divergence theorem method to find flux.
Comment: Thank you for your answer. Could you, please, explain why $\hat{n} = \frac{1}{r}(x, y, z)$?
Comment: An outward normal vector to a spherical surface with center at the origin will be $(x-0)i + (y-0)j + (z-0)k$ where $(x, y, z)$ is point on the surface of the sphere. To find unit normal vector, you need to divide it by its magnitude which is $\sqrt{x^2+y^2+z^2} = r$
Comment: I got it, thank you. Regarding my approach, why am I getting the flux equals to 0? Isn't it right what I'm doing?
Comment: @Magela you should not get dot product as zero. What is the normal vector you are getting? That must be wrong.
Comment: $(r_u \times r_v)$ should come to $16 (\sin^2\phi \cos \theta, \sin^2 \phi \sin \theta, \sin \phi \cos \phi)$
Comment: Yes, you're right. I found my mistake. About continuity, $F$ isn't continuous at (0, 0, 0). Doesn't this matter in this case?
Comment: Yes it does matter and we cannot apply Divergence theorem to find flux.
Comment: Right. We cannot apply Divergent theorem for this reason, but why can we calculate the surface integral so?
Answer: There is no real need to use divergence theorem here. The polar coordinate representation should do.
\begin{gather*}
Transferring\ to\ polar\ coordinates,\ we\ see\ that\ \\
F( r) =\frac{11}{r^{2}} \ \\
Since\ the\ field\ is\ spherically\ symmetric,\ \\
we\ can\ directly\ calculate\ the\ flux\ as\ \\
\oint F( r) dr=\int 4\pi r^{2} F( r) dr=\int 44\pi dr=176\pi \\
\\
\end{gather*}
I am not sure whether your final answer is correct, because that would mean that the radius of the sphere should be 1 in that case.
Comment: Thank you for answering. I'm afraid that I did not understand how you can get $F(r) = \frac{11}{r^2}$. Did you see that the vector field has the coordinates $ F = \left\{ \frac{11x}{\left(x^2+y^2+z^2\right)^{3/2}},\frac{11y}{\left(x^2+y^2+z^2\right)^{3/2}},\frac{11z}{\left(x^2+y^2+z^2\right)^{3/2}} \right\}$

How should I evaluate the flux?


Long established installation of postfix has become an open relay; how can I figure out how and stop it?

Question: This server system has been serving email services since Red Hat v 1.1 (circa 1997 I think) and now is on Fedora Core 37; through many hardware and OS updates along the way it has been kept current. And, we chose postfix and dovecot early on, and are still using them. And I've been the system mangler all this time.
A week ago tomorrow we had our /var tree wiped out due to a bug in a backup script - doah! And it required a full rebuild of the OS, "from scratch", to the same version. All the software not included in the Fedora Server 37 distribution was loaded fresh via dnf install. And, we got our full config back from our good backups.
On Monday, two days ago, I noticed the system was noticeably sluggish but didn't have time to look into it. And I also noticed our link to the internet seemed to have performance problems. That was a clue...
Then, I decided to get spamassassin working again - it takes time to configure a mature environment like this! And SA had been disabled before the loss of /var, so it didn't just start back up. And anyway, when I went to check /etc/var/log/maillog to see if it was doing its job, I found all these mail being relayed messages?! Whisky Tango Foxtrot?!
I then checked the mail queues - hundreds of thousand to Gmail alone! WOW!
For now, I've turned off ALL outbound emails with:
default_transport = error: Sorry spammers, we're not sending your email! So sue us!

And began trying to figure out what went wrong.
I DID find SOME were getting through claiming to be 127.0.0.1, so I closed that down. And I methodically went through all the various (and copious) postfix configuration options and couldn't find a thing wrong...
So, I went to use one of these script-testing open-relay testing web sites that try a dozen or so different hacks that spammers use to convince otherwise well-configured servers to relay their mail, but I couldn't find any - the last time I looked, there were a half-dozen or so such web sites! (What happened to 'em?! If you know of one, please tell me!)
And so I used nmap. It does NOT do a comprehensive job, or if it can, I'm not familiar with how. But I turned to send back on and tested. In testing, it says:
Host is up (0.00027s latency).
rDNS record for <ip-addr>: <reverse-lookup-map>

PORT    STATE    SERVICE
25/tcp  open     smtp
|_smtp-open-relay: Server doesn't seem to be an open relay, all tests failed
465/tcp filtered smtps
587/tcp open     submission
|_smtp-open-relay: Server isn't an open relay, authentication needed
MAC Address: [its mac address] (controller's mfg name)

Nmap done: 1 IP address (1 host up) scanned in 22.03 seconds

The ONLY two websites I could find to look at it and report were non-responsive - the one loaded but didn't respond and then when I tried reloading the page, it wouldn't reload, and the second kept saying it was busy, try again later.
So ... back to figuring it out "by hand."
OK, so NOW what do we do?
ALL requests for setting information will be gladly honored, but the config file is first of all huge, and secondly it contains a lot of private information we don't want out there.
More information - at anx's request, the output of postconf -n:
alias_database = hash:/etc/aliases
alias_maps = hash:/etc/aliases
broken_sasl_auth_clients = yes
command_directory = /usr/sbin
compatibility_level = 3.6
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
debug_peer_level = 10
debug_peer_list = <past-not-current-external-ip>
debugger_command = PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin ddd $daemon_directory/$process_name $process_id & sleep 5
default_transport = error: <our-middle-finger-to-spammers>
disable_vrfy_command = yes
html_directory = no
inet_interfaces = all
inet_protocols = all
local_recipient_maps = unix:passwd.byname $alias_maps
mail_owner = postfix
mailbox_size_limit = 1073741824
mailq_path = /usr/bin/mailq.postfix
manpage_directory = /usr/share/man
message_size_limit = 536870912
meta_directory = /etc/postfix
milter_default_action = accept
mydestination = $myhostname, localhost.$mydomain, localhost, <list-of-60ish-domain-names>
mydomain = <primary-domain>
myhostname = mail.<primary-domain>
mynetworks = <list-of-5-internal-ips>
mynetworks_style = subnet
newaliases_path = /usr/bin/newaliases.postfix
proxy_interfaces =
queue_directory = /var/spool/postfix
readme_directory = /usr/share/doc/postfix/README_FILES
relay_domains = $mydestination, <list-of-11-internal-ips-most-don't-exist-now>
sample_directory = /usr/share/doc/postfix/samples
sendmail_path = /usr/sbin/sendmail.postfix
setgid_group = postdrop
shlib_directory = /usr/lib64/postfix
smtp_tls_CAfile = /etc/pki/tls/certs/ca-bundle.crt
smtp_tls_CApath = /etc/pki/tls/certs
smtp_tls_security_level = may
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks, check_helo_access hash:/etc/postfix/helo_access, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_pipelining, reject_non_fqdn_recipient, reject_unknown_recipient_domain, reject_unauth_destination, check_sender_access hash:/etc/postfix/sender_access, check_client_access hash:/etc/postfix/pop-before-smtp, permit_mynetworks
smtpd_sasl_auth_enable = yes
smtpd_sasl_path = /var/spool/postfix/private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_tls_security_options = noanonymous
smtpd_sasl_type = dovecot
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated, check_client_access hash:/etc/postfix/pop-before-smtp, reject_non_fqdn_sender, reject_unknown_sender_domain
smtpd_tls_cert_file = /etc/letsencrypt/live/<primary-domain-name>/fullchain.pem
smtpd_tls_dh1024_param_file = $config_directory/dh2048.pem
smtpd_tls_dh512_param_file = $config_directory/dh512.pem
smtpd_tls_key_file = /etc/letsencrypt/live/<primary-domain-name>/privkey.pem
smtpd_tls_security_level = may
soft_bounce = no
strict_mailbox_ownership = no
unknown_local_recipient_reject_code = 550

More information - most also at the request of anx:

Postfix uses ports 25 (smtp) and 587 (submission or msa).

Dovecot uses ports 993 (imaps) and 995 (pop3s) while it listens on 143 & 110 (imap / pop) which are blocked by (multiple) firewalls.
postconf -M
smtp       inet  n       -       n       -       -       smtpd
submission inet  n       -       n       -       -       smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes
pickup     unix  n       -       n       60      1       pickup
cleanup    unix  n       -       n       -       0       cleanup
qmgr       unix  n       -       n       300     1       qmgr
tlsmgr     unix  -       -       n       1000?   1       tlsmgr
rewrite    unix  -       -       n       -       -       trivial-rewrite
bounce     unix  -       -       n       -       0       bounce
defer      unix  -       -       n       -       0       bounce
trace      unix  -       -       n       -       0       bounce
verify     unix  -       -       n       -       1       verify
flush      unix  n       -       n       1000?   0       flush
proxymap   unix  -       -       n       -       -       proxymap
proxywrite unix  -       -       n       -       1       proxymap
smtp       unix  -       -       n       -       -       smtp
relay      unix  -       -       n       -       -       smtp -o syslog_name=postfix/$service_name
showq      unix  n       -       n       -       -       showq
error      unix  -       -       n       -       -       error
retry      unix  -       -       n       -       -       error
discard    unix  -       -       n       -       -       discard
local      unix  -       n       n       -       -       local
virtual    unix  -       n       n       -       -       virtual
lmtp       unix  -       -       n       -       -       lmtp
anvil      unix  -       -       n       -       1       anvil
scache     unix  -       -       n       -       1       scache
postlog    unix-dgram n  -       n       -       1       postlogd

Information yet to fetch:

What's in the "locally added received header"s.
Analysis of postfix log "stack" for a given queue ID.

HOLD THE PROGRAM!
Simply HAVING to send a few vital emails out, but not wanting to give the spammers a single use of our systems, I decided to try just turning off dovecot... I found that turning off dovecot and restoring the default_transport setting allowed outbound to work normally and Postfix did NOT become an open relay! YAY!
SURE, it won't work for our non-local users (which number > 1 and < 100), but hey, "you gotta do what you gotta do..."
I think this shifts the focus considerably; dovecot is THE issue.
Comment: What does the locally added Received header say how the sender became authorized to submit? What do the postfix logs say about a queue ID that was found in the stack of unauthorized submissions? Any idea why it say "permit_mynetworks" twice in two quite meaningfully different positions in your restriction list? Are both 587 and 465 served by postfix, or is one a dovecot proxy (usually visible in the output of `postconf -M`)?
Comment: @anx As for your first question, IDK yet... About the received headers content, it's not something I've investigated before, or even realize I should! I will! Uh... I'll find that in the logs, or do I have to look at an example? The second, about queue ID, I don't really understand the question, unfortunately. I presume the "queue ID" is the inbound mail ID, easily found in the logs, and I guess by "stack" you mean the collective messages about an individual message? Presuming so, that's something I'll do soon. As for your question about "permit_mynetworks", I didn't realize that and...
Comment: @anx ...the reason is lost to history! Advice on that? As for the ports, ONLY through your asking did I learn that 587 WAS (not in config now) from dovecot - that's in the ".rpmsave" version of 10-master.conf! I take it I should put that back? So, I therefore presume it's coming from Postfix, but I don't see the number in the config, and neither it's /etc/services equivalent, "msa".  BUT, I have to get back to you with more - it;s late!
Comment: I don't recommend making such change now - that should only be executed after careful deliberation and review of affected options, I just know that if you had already opted into putting a dovecot proxy in front of postfix, from such changes a few incomplete configurations could arise that allow arbitrary public submissions to an xclient-enabled service. Such just don't exist in a configuration where exclusively postfix is handling submissions (`submission` and `smtps` are the commonly used port aliases in the first column of the postfix master configuration).
Comment: Look at the dates of configuration files, did you maybe put some old config back.. which is now wildcard-`include conf.d/*.conf` from a common file on top of your known-good files? Look at your postfix logs or the output of `postqueue -p`, the queue identifier looks something like `5F54EFCA0A`, that is what you can `grep` your logs for or ask `postcat -qhbe QUEUEID` to learn more about what is known about one particular message. In the postfix default configuration such ID is only assigned *after* accepting at least one recipient, but that is fine, your story sounds like this is the case.
Comment: Adding sample headers and the output of `postconf -M` might allow a clearer answer instead of just pointers & ideas.
Comment: @anx OK, update of question above done. Now on the hunt for the mail header data and "stack". We do not even have a conf.d - unique to Fedora Server? IDK. And so, there are no include statements - and yes, I checked. And I forgot to say that in the update. As for the queues, I didn't save them! I ran postsuper -d ALL - poof! - gigs of spam, gone in a flash! ... I can reopen and catch a few, if that's a good idea.
Comment: @anx, Just alerting you to the new realization that it's most likely a pure dovecot issue, though if I should fix the dual permit_mynetworks entries, I'd appreciate your thoughts on that. I can devote a good bit of attention to this today but also have some other vital work I must get to as well - so, I'll be "back and forth" at it. THANKS for your help!
Comment: Turning off dovecot solving anything smells like its the `permit_sasl_authenticated` method that allows submission, so possibly valid (just stolen/misused) credentials.
Comment: @anx At the time it was all coming inbound, I was looking at maillog and it DID talk about authentication being successful ... for accounts that clearly aren't ours! I kept the log. I also had some problem getting Dovecot restarted following the reinstallation. In short, I couldn't find the correct auth entry, so I took it from an older configuration file. MAYBE I should create a new question for this? ...Meanwhile, I've got some other critical work to do now - back at it soon. And I'll start investigating dovecot's auth issues ASAP.
Comment: @RohitGupta Hi editor Rohit, I approved the edits and wanted to comment:  Know that as someone who literally helped create the language we use here, first using the internet in 1978, LONG before web sites, your merging those words isn't in my vocabulary and maybe never will be, my use of commas is more old-school than yours, perfectly valid and neither is better here, and your other two edits, well, just style differences; however, if all that helps reach people, SURE! And thanks for caring!
Answer: I'm now convinced it was not a Postfix issue but a Dovecot issue. However, this answer may well be helpful to others.
In particular, how to determine it was likely not a Postfix issue was a bigger learning curve than I expected for being someone who's used the same system (as a system admin) literally for two decades and then some.
What I hadn't realized was that with this kind of configuration, Postfix itself "gives up authority control" to Dovecot. And, well, at this point, Dovecot's ability to receive valid connections from those of us who are not in the internal network except when we're on-site has become an important feature of our functioning.
Thankfully, it's less vital for us than for many other organizations as we have other alternatives, but the reality is that most don't want to bother to log-in via command line just to deal with mail - I suspect this is the most popular position! - in this circumstance! But that takes nothing away from recognizing what I had not:
You cannot just stop Dovecot, once configured, and expect to be able to send WITHOUT reconfiguring Postfix to not use Dovecot. And, you can't even just block the ports to Dovecot from the outside, because Postfix will just call Dovecot directly anyway!
Ultimately, as per my understanding (glad to be proven wrong!) once configured, Postfix gives up darned near all security authority for any connection that pretends to be a valid user to Dovecot, though all "internal" sends and receives function normally with Dovecot running and Postfix prevented from sending (as described in the question).
I figure it's both fruitless and inappropriate to use this question to ask about Dovecot's issues, and while I think these comments constitute a valid answer, I will need to ask another question about Dovecot because "I'll be damned if I can find what's wrong!"
I strongly suspect either I'm making a (likely not uncommon) gaff OR, there's a bug in the version we have now, post rebuild, as the config before reload was NOT sending spam, and that's what we had been using for several days before the spammers began using our system! Notably, when we didn't stop Postfix from sending, NO external system we found / used to test our system thinks our system is an open mail relay but there's NO denying it relayed hundreds of thousand of emails improperly! Further, opening it back up (permitting Postfix to send) promptly allowed
Worth citing here is that I have checked the logs and, with the limited time I have, given all the demands on me at present, I have with 90%+ certainty confirmed that there's no "stolen credentials," but that has yet to be proven. I sure don't see the evidence in the logs, though I'm not keen to open up to the spammers again with new logging configuration to gain more data. PERHAPS it's worth always having maximum authorization logging for just this reason!


semidirect product $\implies$ exact sequence

Question: Hey so I´m given a semidirect product $G=NH$, where $N$ is normal in $G$ and $N\cap H=1$. I have to show that the sequence below is exact.
$$1\xrightarrow{}N\xrightarrow{\alpha}G\xrightarrow{\beta}H\xrightarrow{}1$$
I am not sure how or where to start. Please help me.
Comment: What are $\alpha$ and $\beta$?
Comment: Looking at the history of your questions in the past week, it looks like you are asking this site to write your thesis for you. I can't say that that particularly motivates me to give an answer.
Comment: im sorry that i ask a lot... wish i was as smart as you.
Comment: I second Matt's comment, and would like to ask what you have tried.
Comment: @MattN. Are you asking your question because you are not sure, or because you are trying to give a hint? I mean, the question can be re-phrased as `what are $\alpha$ and $\beta$?' One he knows this, he knows the answer (mod the fact that $G/N\cong H$).
Comment: @user1729 Exactly. But he clearly states that he doesn't even know where to start.
Comment: @user1729 And to answer your question: I was thinking aloud.
Answer: I'm assuming that you need to pick the appropriate $\alpha$ and $\beta$ yourself. Then here is how you can do it:
Define $\alpha$ to be the natural inclusion: $\alpha(n)=n$ for every $n \in N$. Define $\beta$ by saying that $\beta(g) g^{-1} \in N$ (for every $g\in G$ there is exactly one such $\beta(g) \in H$). All you need to do now is check that $\alpha$ and $\beta$ are homomorphisms and that the sequence is exact. You know what $\alpha$ and $\beta$ are, so this is a pretty straightforward task.
If you already know this, then I've told you nothing new. If that's the case then it would really help to see what you've tried yourself.


What is the inverse of $ f(x)= \sqrt{\log_{3} (x)}$. Why is $g(x)= 3^{x^2}$ not properly $f$'s inverse?

Question: $ f(x)= \sqrt{\log_{3} (x)}$.
Suppose $g(x)= f^{-1} (x) = I $.
In that case
$(f\circ g)(x)) = f(I) = x$
( I use I - for "inverse" - to denote the expression in $x$ defining function $g$).
But
$f(I) = x$
$\iff \sqrt{\log_{3} I} = x$
$\iff (\sqrt{\log_{3} I})^2 = x^2$
$\iff \log_{3} I = x^2$
$\iff 3^{\log_{3} I} = 3^{x^2}$
$\iff I = 3^{x^2}$
So $g(x)= I = 3^{x^2}$ is a suitable candidate
If function $g$ is actually the inverse of $f$, then
$g\circ f$ should be identical to $f\circ g$.
But it is not the case :

$(f\circ g)(x)= |x|$

and

$(g\circ f) = \{(x, y)| x\geq 1 \land y=x\}$.

Hence my question : how should I restrict function $g$ in order $g$ to be exactly identical to $f^{-1}$?
Comment: Do you know what a _domain_ and a _codomain_ of a function is?
Answer: In general the inverse of a one-to-one function $f$ is defined on the range of $f$. $\sqrt{\log_3(x)}$ with its maximal domain (i.e. $[1,\infty)$) is a one-to-one function with range $[0,\infty)$, so its inverse is defined on $[0,\infty)$. That inverse is equal to $3^{x^2}$ where it is defined, but it has a different domain from the domain of the "usual" function $x \mapsto 3^{x^2}$ (which has domain $\mathbb{R}$).
One can tweak definitions slightly and require a function to be bijective in order to have an inverse rather than merely injective. In this case the inverse is defined on the codomain of $f$, but in this situation the codomain has to be chosen to be equal to to the range in the first place in order for the inverse to exist.


Distribution of Residuals and Response - When to use which?

Question: I would like to analysze the relation between my continuous response 'soil moisture content [%]' and 2 categorical and 1 continuous predictors.
I fit a linear mixed model and checked the distribution with the check_distribution() function of the performance package.
library(performance)

lmer(soil_moisture ~ weed_coverage+distance  +
    treatment + (1 | kettlehole/plot) +
    (1|date), data = BFK_comp) -> lmmod

check_distribution(lmmod)

I got this result and plot
Predicted Distribution of Residuals

 Distribution Probability
       normal         88%
      tweedie          9%
      weibull          3%

Predicted Distribution of Response

  Distribution Probability
 beta-binomial         34%
       tweedie         22%
         gamma         16%

As I understand it, the predicted distribution of the residuals need to be normally distributed, so I can use a linear regression, right? I am not talking about the other assumptions here, just normality.
After that I did a test with the check_normality() fuction, also from the performance package and it says
Warning: Non-normality of residuals detected
           (p < .001).

These results seem to be contradictory to me or am I missing something here?
Also, I want to test if there is a significant difference between soil moisture content and distance (factor with 2 levels). Do I need to look at the distribution of the response or at the distribution of the residuals to find out if I need to use a t-test or wilcoxon signed rank test?
Comment: I suspect what is going on is that you have a sample size that's large enough that the hypothesis test is returning a significant result ---- there is good evidence that the residuals deviate from a normal distribution ---- even though the residuals may be reasonably approximately normal.  This is one reason why using a hypothesis test to assess model assumptions isn't the best approach.  You are better off looking a plot of the residuals, as a histogram, density, or q-q plot.
Comment: Your response is only defined in the inverval [0, 1]. Therefore, the residuals cannot be normal distributed. However, the normal distribution can still be a sufficient approximation. I would consider using beta-regression. The only issue is that this does not allow observed values of exactly 0 or 1. But then, you could argue that soil moisture cannot be exactly 0 or 1.
Comment: With the two-sample problem you raise at the end of the question:  For me, I'd just as soon use a regression model with a single categorical variable on the right hand side, and then look at the residuals.  But you could also look at the distribution and spread of the two groups.  P.S. You might have meant the Wilcoxon rank sum test.
Comment: Oh, I might have misunderstood that question about comparing between levels of `distance`.  If that variable is part of the larger model, assess the larger model for appropriateness.  And then use the *emmeans* package to compare among the terms you are interested in.  Model types supported by emmeans are listed here: [cran.r-project.org/web/packages/emmeans/vignettes/models.html](https://cran.r-project.org/web/packages/emmeans/vignettes/models.html)
Comment: Thanks for your answers. I still dont understand in which cases I need to look at the predicted distribution of the *residuals* and in which cases at the predicted distribution of the *response*.
Comment: Ultimately, the answer to your question is afforded by the model you are using.  Almost any regression model makes assumptions about the *conditional* response distribution, but not about the marginal response distribution.
Comment: ***a)*** Yes, the ultimate answer is to look at the model you are using.  If it is a type of general linear model (*lm*), you'll see that the model specifies that the *errors* are normally distributed.  One method to assess this is to look at the *residuals*.  As I mentioned, don't use hypothesis tests to look at this assumption. ***b)*** Other tests like anova, ancova, *t*-test, linear regression can be thought of as an *lm*.  It makes life simple to think of these tests this way, and not worry if the *data* should be normally distributed. (Cont').
Comment: (Cont') ***c)*** It's good to start with a textbook.  Montgomery, *Design and Analysis of Experiments* is a common one.  Sometimes websites use imprecise language about assumptions like the normality assumption.  ***d)***  Once you are happy with the model you have fit, you can use something like *estimated marginal means* (*emmeans*) to make any post-hoc comparisons or test other contrasts.
Comment: So in order to see if I have to use a wilcoxon signed rank test or a paired t test, I should create a linear model and then check the distribution of the **residuals** for normality?
Comment: Yes, that's one way to approach the question.  But with something as simple as a *t*-test, you *can* look at the distribution of values to assess if it's likely they come from a population with a conditionally normal distribution.  That is, look at the distribution of each group.
Comment: Okay, thanks a lot for your help! :) I really appreciate!
Answer: Answer mostly from comments:

(Roland) Your response is only defined in the interval [0, 1]. Therefore, the
residuals cannot be normal distributed. However, the normal
distribution can still be a sufficient approximation.

(Roland) I would consider using beta-regression. The only issue is that
this does not allow observed values of exactly 0 or 1. But then,
you could argue that soil moisture cannot be exactly 0 or 1.

(WHuber) Ultimately, the answer to your question is afforded by the model you are using. Almost any regression model makes assumptions about the conditional response distribution, but not about the marginal response distribution.

(Sal) I suspect what is going on is that you have a sample size that's large enough that the hypothesis test is returning a significant result ---- there is good evidence that the residuals deviate from a normal distribution ---- even though the residuals may be reasonably approximately normal.

(Sal) This is one reason why using a hypothesis test to assess model assumptions isn't the best approach. You are better off looking a plot of the residuals, as a histogram, density, or q-q plot.

(Sal) If you want to compare among levels of a variable that is part of the larger model, assess the larger model for appropriateness. And then use the emmeans package to compare among the terms you are interested in. Model types supported by emmeans are listed here: cran.r-project.org/web/packages/emmeans/vignettes/models.html

(Sal) If your model is a type of general linear model (lm), you'll see that the model specifies that the errors are normally distributed. One method to assess this is to look at the residuals. As I mentioned, don't use hypothesis tests to look at this assumption. Other tests like anova, ancova, t-test, linear regression can be thought of as an lm. It makes life simple to think of these tests this way, and not worry if the data should be normally distributed.

To give an example of this final comment, we can look at two independent groups to assess the distribution of values and equal variance before a t-test.
set.seed(sum(utf8ToInt("Salvatore")))

A = rnorm(50, 10, 1)
B = rnorm(50, 12, 1)
Y = c(A, B)
Group = c(rep("A", length(A)), rep("B", length(B)))
Data = data.frame(Group, Y)

library(lattice)

histogram(~ Y | Group, data   = Data,
           layout = c(1,2))

From these histograms, the distribution in each group is relatively bell-shaped, but with some skew.  The spread of the data is each group is relatively similar.
As an alternative, we could formulate the t-test as a general linear model, and then assess the residuals.
 model =lm(Y ~ Group, data=Data)

 hist(residuals(model))

 plot(predict(model), residuals(model))

Here, the residuals are somewhat right-skewed.

The residuals are fairly homoscedastic when plotted against predicted values.


Does it make sense to log-transform the dependent when using Gradient Boosted Trees?

Question: When we are modeling non-negative data with linear regression we often log-transform the dependent, thereby ensuring that all the predictions made by the model will be positive as well.
Does it make sense to do the same when using Gradient Boosted Trees (GBTs)? For Random Forests the output is an average of the training labels in the "winning" leaf of each tree, thereby ensuring that the prediction will be non-negative.
Is it safe to make the same assumption for GBTs? I haven't been able to follow the derivation from the original paper (section 4.3).
Comment: It surprises me that you can get negative values from GBTs; my understanding of this was that you couldn't predict outside the range of the observed (all positive) data.
Comment: You can definitely get negative predictions from a GBT even if all the training data is positive.
Comment: @MatthewDrury is there some way to show this mathematically? Even a toy example would work.
Comment: A single tree, or ensemble of trees, trained only by splitting the observed data alone would not be able to predict outside the range of the data (e.g. CART or random forests). However, boosting creates trees fitted to the residuals of previous model predictions.
Comment: @Bar I got a chance to write an answer for you.  I hope this helps.
Answer: As I mentioned in the comments:

You can definitely get negative predictions from a GBT even if all the training data is positive.

Here's a simple example of this in practice:
x = np.array([0, 1, 2, 3]).reshape(-1, 1)
y = np.array([0, 100, 0, 100]).reshape(-1, 1)

gbr = GradientBoostingRegressor(learning_rate=1.0, n_estimators=2, max_depth=1)
gbr.fit(x, y)

Of course, the parameters here are extreme so as to make the point in the simplest possible situation (no one should use a learning rate of 1.0 in practice).  The predictions from this model produce negative values at the second stage:
for i, preds in enumerate(gbr.staged_predict(x)):
    print("Preds at stage {}: {}".format(i, preds))

which produces the following output:
Preds at stage 0: [  0.          66.66666667  66.66666667  66.66666667]
Preds at stage 1: [ -11.11111111   55.55555556   55.55555556  100.       ]

To understand what happened, we need the gradient from the first stage:
for i, preds in enumerate(gbr.staged_predict(x)):
    grad = y.reshape(-1) - preds
    print("Gradient at stage {}: {}".format(i, grad))

Which produces:
Gradient at stage 0: [  0.          33.33333333 -66.66666667  33.33333333]
Gradient at stage 1: [ 11.11111111  44.44444444 -55.55555556   0.        ]

Recall the a regression tree is fit to the gradient at stage zero, and the average residual is calculated in each of their terminal regions of this true to form the update step of the booster.  This tree resulted in two terminal regions, the first consisting of the first three observations, and the second of only the last observation.  The average gradient (residual) in this first region is 11.111..., which is then subtracted from the prediction of 0.0 for the first observation, resulting in -11.111....


How can I replace all my internal links automatically?

Question: I have changed the domain of my website from www.mysite.com to mysite.net. I upload the site to the new domain and all is fine, but I have more than one thousand nodes, and in almost all the nodes I have a link to another node on the site.
The link are absolute so they have this form: www.mysite.com/node/22 and I need to change all nodes to the new domain.
Changing 1,000 links manually is too much. How can I do this automatically??
Comment: Redirect all traffic from www.domain.com to domain.com with your .htaccess is an option?
Answer: Why are you using absolute links? If you were using relative links, of the form /node/22, the problem would not arise.
I am assuming you are using Drupal 7. The answer for D6 would be a little more complex.
If the links are in the node body, and if you have direct access to your database, you can update the text by running a query on the field_data_body and field_revision_body tables.
First, back up your database in case this goes wrong.
Then run the following query on your tables:
UPDATE field_data_body SET body_value = REPLACE( body_value, 'a href="http://www.mysite.com/','a href="http://mysite.net/');

Repeat with the field_revision_body table.
This should change all references to the old domain with the new domain.
If you have links in the comments, you might also want to do field_data_comment_body and field_revision_comment_body, scanning for references in the comment_body_value field.
In the event the links are being generated automatically by a text filter, change the REPLACE patterns accordingly.
Comment: 1) Without `http://` before domain name browsers should treat links as a relative. If they are absolute, there is supposed to be protocol in them. 2) It could be better to just get rid of the domain name instead of updating it, for future. 3) The way you wrote it, it will not work for things like images and embedded media, and might fail for the links created with any kind of wiki-like or bb-like syntax, nor for the ones dynamically created from URLs plainly pasted into node/comment body.
Comment: Well, 2 and 3 was just suggestions I believe could help future readers, not bugreport on your answer ;)
Comment: @Mołot You are right about the protocol (thank you - was in a hurry). I have updated the answer. The OP specifically asked about links to pages. I assume if he wants to change other links, the pattern would change accordingly. I agree about the relative links.
Answer: You can use the Pathologic module to correct links in your contents.
Comment: Pathologic is a great module and certainly a solution but it doesn't solve the problem of incorrect paths being stored, it only rewrites them when they are displayed.
Comment: Pathologic can rewrite your old domain to the new one if you configure your old domain as an "Also considered local" domain.
Answer: A really nice module to help you do this is Search and Replace Scanner
Or if you prefer something simpler (as a drush command) without the interface you could use Drush Search & Replace
Answer: What you could do if you don't want to touch the db is using hook_process_html, so that you use relative path's for all content, scripts and stylesheets:
function MY_MODULE_process_html(&$vars) {
  //use relative path
  $vars['scripts'] = str_replace("http://www.mysite.com/", "/", $vars['scripts']);
  $vars['styles'] = str_replace("http://www.mysite.com/", "/", $vars['styles']);
  $vars['page'] = str_replace("http://www.mysite.com/", "/", $vars['page']);
}

Comment: This seems like a MASSIVELY inefficient way of doing things.
Answer: Finally I use this http://interconnectit.com/products/search-and-replace-for-wordpress-databases/
Thanks a lot!!
Comment: Hello. Could you elaborate a bit more? Link-only answers usually stops being useful when the linked site goes down or relocates - and your own questions shows it can happen ;)
Comment: May you expand your answer to say exactly what you did? Link-only answers are not considered answers, on Stack Exchange. Providing more details would help making the question helpful for future readers.
Comment: You are in a drupal Q&A page and link to wordpress solutions.


Azure Service Bus Topic/Subcription Quotas and Scaling

Question: My application is required to publish messages to thousands of clients with high throughput requirements (hundreds of, maybe 1000+ messages per second) in order to keep local client databases in sync with the server database. Every server database operation would potentially require a message to be published to a location of 20 or less clients. Clients only have database information for their particular location. Thousands of locations exist.
My idea was to use Azure Service Bus topic/subscription model for this. Each client would establish a subscription (and connection) to the database sync topic and receive filtered messages for their location.
However, according to the Azure Service Bus Quota page, I am limited to the following:

Number of concurrent connections on a namespace - 1,000
Number of concurrent connections on a queue/topic/subscription entity - Capped by the limit of concurrent connections per namespace.
Number of subscriptions per topic - 2,000

With these limitations now in mind, my solution does not seem usable. What is the best approach to solving this problem using Azure Service Bus? To get around the subscriptions per topic limit I could just create a topic per location. However, the killer really is the number of concurrent connections on a topic/namespace (I think). All clients would be subscribed and long-polling for sync messages to come down from each topic and that would be thousands. Would I need to create a new namespace and topic per location? This seems problematic since I would have to open a new connection each time I want to publish from the server which is very frequent.
Is Azure Service Bus not the correct solution for this problem?
Comment: Have you looked into using Azure Queues?  A quick check here: https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx) says that you can create an unlimited number of queues with an unlimited number of concurrent clients.
Comment: I hadn't. Thanks for the recommendation. The major problem I see with Azure Queues is it only supports HTTP which does not allow for long-polling operations from the client receivers. Routing is also a major advantage in Service Bus Topics (deliver once and done), whereas with Azure Queues I would have to deliver the message N number of times for each location (where N is the number of clients at the location).
Comment: Yes - unless you look at some kind of location-based logical partitioning?  It seems that the core limitation is the concurrent connections per namespace.
Answer: Not an expert, but a few points.
Azure service bus does support long-polling:
Azure Documentation
.  "As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports."
Also, you are correct: Event hubs are the reverse of your problem.
Lastly,
Azure notification hubs is actually the closest analog to your use case (it's designed for mobile push to thousands of endpoints, which is an extremely similar scenario). I am not sure if it's feasible to be used for non-mobile applications, but I have not yet found a reason why not. I am considering switching from topics when I have the time to research and POC notification hubs.
Answer: Have you considered using Event Hubs? https://msdn.microsoft.com/en-us/library/dn789973.aspx
Comment: Please correct me if I am seeing this at the wrong angle, but Event Hubs seems like it solves the reverse of my problem -- pushing data from a large number of different senders (devices) into a small number of readers (cloud server). The problem I am trying to solve is pushing messages from a small number of senders (cloud server) to a large number of readers (devices).


I'm trying to make custom formatting that highlights when a number is more than and less than a specific formula

Question: I work for a brokerage and we profit in the margin.  In order to keep these margins at the forefront of everyones mind we want to format it so that if the bid is over our 15% goal it highlights in order to make the broker aware he is outside of the target zone.
In the example below the rate we are charging is $1,000.00 and we want to find a provider for $850.00 or less in order to meet margin.  At $851.00 and higher I have the amounts being highlighted, I managed this by putting =sum(A12*0.85) in F and put a conditional formatting into E for =E14>F14, I then hide F to make it look clean.
I am by no means an experienced Docs user so I'm hoping I'm missing an easy fix but my problems are:

That I need to manually enter a lot of this (thousand of lines, always need more), inserting new lines at the bottom of that E14:E23 string does not carry over the F function and causes even a blank cell to highlight orange since the F cell is blank.
I need to have this same process done for many loads, this is one load, the next one might be $2,000.00 and I don't want to manually have to do each one, a simple copy paste method is beyond my ability to figure out.
Answer: In column E for example.
Conditional Formatting > New Rule > Use a formula
<code>=IF(AND(A1<>"",E1<>"",E1>=A1*0.85),1,0)
</code>
Comment: I changed the "A1" in that formula to "A12" and "E1" to "E14", pasted it in, so far it will only highlight the first cell in the string I want to have this formatting, none of the rest.  Do I need to use some $ signs?  Thanks for the help.


Use MikroTik Bridge as a Public IP Firewall for Public Hosted Servers

Question: We want to do this:
Internet<->MikroTik in Bridge Mode with Firewall Filter<->Hosted Server

The primary objective is to allow RDP and FTP in from the outside but block everything else from the outside. From the inside everything must go out.
The problem we are running into is we add these rules and blocking outside to inside is working, but now the hosted server cannot access anything to the outside. The return TCP/IP from the outside is not port 3389 or port 80, but random.

/interface bridge filter> pr
Flags: X - disabled, I - invalid, D - dynamic
0   ;;; Accept ICMP for PING
     chain=forward action=accept mac-protocol=ip dst-address=196.x.x.x/32 ip-protocol=icmp
1   ;;; Accept FTP Transfer Port
     chain=forward action=accept mac-protocol=ip dst-address=196.x.x.x/32 dst-port=20 ip-protocol=tcp
2   ;;; Accept FTP Control Port
     chain=forward action=accept mac-protocol=ip dst-address=196.x.x.x/32 dst-port=21 ip-protocol=tcp
3   ;;; Accept RDP
     chain=forward action=accept mac-protocol=ip dst-address=196.x.x.x/32 dst-port=3389 ip-protocol=tcp
4   ;;; Log everything that is about to get dropped
     chain=forward action=log mac-protocol=ip dst-address=196.x.x.x/32 ip-protocol=tcp log-prefix="firewall_drop"
5   ;;; Drop everything
     chain=forward action=drop mac-protocol=ip dst-address=196.x.x.x/32 ip-protocol=tcp
Just FYI bridge is set to use firewall and connection tracking is on.
Comment: You enabled connection tracking? Needed - mark connection as ok. Second, why bridge mode? Get an extenrnal transfer IP, then route properly internally, like most do.
Comment: Don't bridge, route.
Comment: Yes I enabled connection tracking. I don't understand "Needed - mark connection as ok". Do you have an example of "externally transfer IP, then route properly internally". Please bear in mind that I have public IPs externally and hosted server with public IPs "internally".
Comment: It's been a year and we never succeeded in getting this working. In addition the MikroTik bridge/firewall occasionally stopped working (due to 100% CPU) completely blocking off our network. We're very close to abandoning this configuration.
Comment: Consder hiring someone who knows what he is doing. Also consider whether your mikrotik simply is too small. I run that stuff - in a much larger and more complex scenario - without any issues.
Comment: Hi @TomTom, thanks for the suggestion we are considering using David Savage to assist. In the meanwhile the reason why we're doing this in bridge mode is because we have 100s of hosted IPs and we can't change client's systems from external to internal IPs, we don't even have access to the operating systems.
Comment: @TomTom, what if I segment my public router, e.g. create a /30 and then use the one part as public and the other as private? What will just introduce one more hop on the network and surely if I separate them logically and physically I create this internal / external scenario?
Comment: It will also make thigns easier as you can do firewalling on the routing level. It is standard to do so - I always use a routing step for that. The network hop is totally irrelevant if you have the processing power to handle it. Always had that step in - i.e. use a router as firewall, not a bridge.
Answer: I suggest to use local ip subnet for your server and have private LAN between mikrotik (router) and server. Also have mikrotik directly on public ip. Then do the ip src-nat for outgoing traffic (from server to internet). All incoming traffic should be port forwarded (dst-nat). Also you can use normal layer3 ip firewall and disable layer3 firewall for layer2 / bridge network.
Answer: This has something to do with connection tracking, but I don't know exactly how it works in a bridge. I would just add another rule to allow everything from inside with src-address or interface:
<code>add chain=forward action=accept mac-protocol=ip src-address=196.x.x.x/32 ip-protocol=tcp place-before=4
</code>
Comment: The problem with a bridge is that you don't really have a src-address or interface, you only have two bridged interfaces which are really all the same thing. It's not like a normal interface that actually has source and destination. Imagine a bridge like a single Ethernet cable but you can't really say which side is which.
Comment: You can use src-address. There is also in-interface available but I'm not sure it works, because, as you've said it's like running an ethernet cable. Have you tried the above command? I think it should work.
Answer: I agree with @Matt that bridging was a stupid idea and that we should have routed from the start. The entire reason why we bridged was because our border router was a legacy Cisco VXR 7206 with two interfaces, a WAN (ATM) and LAN (used for public) interface. We wanted to firewall stuff on public without re-sub-netting our network. Also, although @DJ_Kukky's suggestion of using DST-NAT is plausible that would have led to much more configuration and also we can't (or won't) give our clients private IPs as we're a public facing ISP.
We should have either implemented the firewall rules on the Cisco or replaced the technology (ATM). In the end we replaced the ATM with MetroEthernet and we were able to establish the outside interface on an Ethernet interface and as such use MikroTik. It's working really well.


Promises - this.fileLoad.then is not a function

Question: Boa tarde.
Estou tendo esse erro - this.fileLoad.then is not a function, alguem poderia por favor me explicar o motivo de isso estar ocorrendo?
<code>fileLoad(){

    return new Promise((resolve, reject) =>{

        let fileReader = new FileReader();

        let fieldFile = [...this.formID].find(element =>{

            if(element.type === "file"){

                return element.files;

            }
        });

        fileReader.onload = () =>{

            fileReader.result;
            resolve(fileReader);
        }

        fileReader.onerror = () =>{

            reject("erro");

        }

        fileReader.readAsDataURL(fieldFile.files[0]);
    }

)};

this.fileLoad.then(
        (content)=>{
            console.log(content);

        },
        (e)=>{

            console.log(e);

 })
</code>

Uncaught TypeError: this.fileLoad.then is not a function
      at userController.addUser (userController.js:76)
      at userController.loadFiles (userController.js:57)
      at HTMLDocument.document.addEventListener.event (userController.js:20)
Comment: Bruno, se puder, marque a pergunta como respondida amigo.
Comment: Já tentou sem o **this**?
Comment: Obrigado encontrei o erro, esqueci de definir a chamada para uma function this.fileLoad.then - this.fileLoad.then();
Answer: Você precisa chamar a função com  parêntesis, this.fileLoad().then(...
Comment: Não precisa não man.


Relational Algebra Division sql equivalent

Question: I am having trouble understanding the translation of the sql equivalent of relational algebra division. I am trying to find the sids of suppliers who supply every part. In the nested query is it essentially stating that it is looking "for all suppliers who does not supply every part" and do a not exist?
But wouldn't the condition WHERE C1.sid = C.sid AND C1.pid = P.pid) also target the sid's of suppliers who actually supply every part?
Suppliers(sid: integer, sname: string, address: string)
Parts(pid: integer, pname: string, color: string)
Catalog(sid: integer, pid: integer, cost: real)

SQL Translation
SELECT C.sid
FROM Catalog C
WHERE NOT EXISTS (SELECT P.pid
                  FROM Parts P
                  WHERE NOT EXISTS (SELECT C1.sid
                                    FROM Catalog C1
                                    WHERE C1.sid = C.sid
                                    AND C1.pid = P.pid)
                  )

Answer: Before answering your questions directly, let me walk you through how you should approach translating the query. Let's look at the outer subquery as a whole first:
SELECT P.pid
FROM Parts P
WHERE NOT EXISTS (
     SELECT C1.sid
     FROM Catalog C1
     WHERE C1.sid = C.sid
     AND C1.pid = P.pid)

This snippet gets "the parts that are not supplied by C.sid". Note that the column in the select clause in the subquery here actually has no meaning. We can essentially just write the query like below and still convey exactly the same meaning, so you should not worry about what C1.sid means in the innermost query.
SELECT P.pid
FROM Parts P
WHERE NOT EXISTS (
     SELECT *
     FROM Catalog C1
     WHERE C1.sid = C.sid
     AND C1.pid = P.pid)

Now with the main query involved
SELECT C.sid
FROM Catalog C
WHERE NOT EXISTS (SELECT *
                  FROM Parts P
                  WHERE NOT EXISTS (SELECT *
                                    FROM Catalog C1
                                    WHERE C1.sid = C.sid
                                    AND C1.pid = P.pid)
                  )

This means we want the suppliers (C.sid) for which there does not exist "the parts that are not supplied by them (C.sid)" (Note I copied the translation of the subquery from the paragraph above). This essentially means that we want suppliers who supply every part.
Now back to your two questions:

In the nested query is it essentially stating that it is looking "for
  all suppliers who does not supply every part" and do a not exist?

No. Just see the interpretation above.

But wouldn't the condition WHERE C1.sid = C.sid AND C1.pid = P.pid)
  also target the sid's of suppliers who actually supply every part?

It's not the right approach to view the inner most query in relation to the main query. This innermost query is there for the first subquery to choose the parts we want. The suppliers being selected here are not relevant as shown by substitution with * above.
Comment: This is not the same. This means just the supplier that supplies at least one part. Having not exists twice does not always mean double negation since you are doing them to a different table/column.

Have you tried rewriting C ÷ P with the basic relational algebra operations (projection, selection, difference, cartesian product, rename)? You'll see that the result has two difference operators. You cannot collapse two different operators into one or zero. This is equivalent to how you cannot collapse two consecutive Not Exist queries.
Comment: It seems that there is a double negation occurring. Why not just change all the NOT Exist to Exist?
Example:
SELECT C.sid
FROM Catalog C
Where EXIST (SELECT *
                  FROM Parts P
                  WHERE EXISTS (SELECT *
                                    FROM Catalog C1
                                    WHERE C1.sid = C.sid
                                    AND C1.pid = P.pid)


Using the same hook more than one times

Question: I have several custom modules and each one of them serves different functionalities. Each module contains from few to many hook implementations and inevitably there are cases where I have to use a hook more than one times between the modules, hook_menu for example.
What are the downsides (if any) of using the same hooks more than one times in different modules ? Is that recommended ? What about performance?
Thanks!
Comment: The more modules you have installed the worse your performance.
Answer: That's exactly how it is supposed to work. Hooks are functions defined by Drupal for you to add functionality to the system through your modules. Hooks are like open doors for you to add things.
From Drupal's documentation:

To extend Drupal, a module need simply implement a hook. When Drupal wishes to allow intervention from modules, it determines which modules implement a hook and calls that hook in all enabled modules that implement it.

Obviously, the more hooks defined in different modules, performance gets hit. But you can't avoid that. You shouldn't stop adding functionality to your modules through hooks because of this.
Comment: I was excepting a more in depth answer. I am aware of the drupal's documentation, but there should be some downsides on the hooks re-usage, or at list a way to follow in order to avoid most of those downsides, thank you either way!
Comment: It's not 'hook re-usage' if its used in different modules. The modules are declaring their functionality and hooking into system events. Obviously if you install 150+ modules, there are going to be performance hits.
Comment: What if you have ten modules and [hook_form_alter](https://api.drupal.org/api/drupal/modules!system!system.api.php/function/hook_form_alter/7) implemented to each one of them ? I don't think that this is recommended.
Comment: There's nothing wrong with that. If each of those ten modules contains it's own logic, then it's fine. If it is necessary for each of those ten modules to, for example, alter a form, they **should** implement the corresponding hook.
Comment: Up-vote , but that does not convince me...

Using the same hook more than one times


Is there a generalization of Pillai trace and the Hotelling-Lawley trace?

Question: In the setting of multivariate multiple regression (vector regressor and regressand), the four major tests for the general hypothesis (Wilk's Lambda, Pillai-Bartlett, Hotelling-Lawley, and Roy's Largest Root) all depend on the eigenvalues of the matrix $H E^{-1}$, where $H$ and $E$ are the 'explained' and 'total' variation matrices.
I had noticed that the Pillai and Hotelling-Lawley statistics could both be expressed as
$$\psi_{\kappa} = \mbox{Tr}\left(H\left[\kappa H + E\right]^{-1}\right),$$
for, respectively, $\kappa = 1, 0$. I am looking at an application where the distribution of this trace, defined for the population analogues of $H$ and $E$, is of interest for the $\kappa = 2$ case. (modulo errors in my work.) I am curious if there is some known unification of the sample statistics for general $\kappa$, or some other generalization that captures two or more of the four classical tests.  I realize that for $\kappa$ not equal to $0$ or $1$, the numerator no longer looks like a Chi-square under the null, and so a central F approximation seems questionable, so maybe this is a dead end.
I am hoping that there has been some research on the distribution of $\psi_{\kappa}$ under the null (i.e. the true matrix of regression coefficients is all zero), and under the alternative. I am interested particularly in the $\kappa = 2$ case, but if there is work on the general $\kappa$ case, I could, of course, use that.
Comment: Wait, $H$ is the 'E'xplained variation and $E$ is the 'T'otal variation? Just checking my mnemonics.
Comment: @cardinal, that is correct. When $\hat{B}$ is the multivariate least squares fits to the correlation coefficients, we have $H=\hat{B}^{\top} \left(X^{\top}X\right)\hat{B}$ and $E=\left(Y - X\hat{B}\right)^{\top}\left(Y - X\hat{B}\right).$
A (literally) big picture overview from Michael Friendly has been pretty useful for me: http://www.psych.yorku.ca/lab/psy6140/lectures/MultivariateRegression2x2.pdf
Comment: Thanks! I'll have a look. (By the way, I was kind of just teasing based on the choice of letters, 'h' for 'explained' and 'e' for 'total'.) Interesting question, by the way; (+1) from me.
Comment: @cardinal I was insufficiently caffeinated to notice the joke. Yes, the mnemonics are bad, but the choice of $H$ and $E$ (and $T=H+E$) is rather standard.
Comment: The joke was sufficiently bad that it would have required lots of caffeine to notice.
Answer: I imagine that productive generalizations would come out from observations that

some of these tests are norms of the vector ${\rm spec}[HE^{-1}]=\{\lambda_1, \ldots, \lambda_p\}$, so Hotelling-Lawley's trace is the $l_1$ norm, $\| \{\lambda_1, \ldots, \lambda_p\} \|_1$, and Roy's largest root is the $l_\infty$ norm, $\| \{\lambda_1, \ldots, \lambda_p\} \|_\infty$.
some of these tests may be a norm of the matrix $HE^{-1}$, e.g., Roy's largest root is the spectral, or $l_2$, norm $\| H E^{-1} \|_2$.
some of the tests may be of the generalized entropy form, e.g., Hotelling-Lawley's trace is GE(1), Roy's largest root is GE($\infty$), and Wilks' $\Lambda$ is GE(-1) on $\{1+\lambda_1, \ldots, 1+\lambda_p\}$, up to a monotone transformation each.

When other norms or other generalized entropy parameters are entertained, other statistics may be arrived at that might be meaningful. I doubt that any of them would produce your $\psi_2$, though.
Comment: I believe we have $\psi_{\kappa} = \sum_i \frac{\lambda_i}{1 + \kappa\lambda_i}$, where $\lambda_i$ are the eigenvalues of $HE^{-1}$. But that doesn't seem to get me anywhere. I guess I don't know enough about the distribution of sums of the eigenvalues ...