Sum of last digits of a sum

Question: Let $d_n$ be the last digit of $S_n$ where $S_n = \left(1 + 2 + 3 + .... + n\right)$. Find the remainder when $\sum_{i=1}^{2017}d_i$ is divided by 1000.
My attempt : I found the following
$$S_1 =1   ,    d_1 =1
\\S_2 =3   ,    d_2 =3
\\S_3 =6   ,    d_3 =6
\\S_4 =10   ,    d_4 =0
\\S_5 =15  ,    d_5 =5
\\S_6 =21   ,    d_6 =1
\\S_7 =28   ,    d_7 =8
\\S_8 =36  ,    d_8 =6
\\S_9 =45  ,    d_9 =5
\\S_{10} =55   ,    d_{10} =5
$$
I was expecting to get a sequence, but did not find any. Please help me to solve this question.
Comment: I believe $\sum^nd_i$ is tabulated at https://oeis.org/A111072 and for $n=2017$ the table in the link at that page gives 7069.
Comment: Finding the last digit of $S_n$ is the same as calculating $S_n \mod 10$, and $d_1 \mod 10 + d_2 \mod 10 = (d_1+d_2) \mod 10$.  Does that help?
Answer: The $S_n$ are the triangular numbers and their last digit is given by
$$\frac{n(n+1)}2\bmod10,$$ which is
$$\left(\frac n2\bmod10\right)((n+1)\bmod10)\bmod10$$ for even $n$ and $$\left(\frac {n+1}2\bmod10\right)(n\bmod10)\bmod10$$ for odd $n$. Hence the sequence has period $20$, with a sum of $70$.
Then the sum
$$\sum_{i=1}^{2017}\frac{n(n+1)}2\bmod10$$
has $100$ complete periods and $17$ remaining terms, $7069$ in total.
Comment: This is excellent. I found the period by inspection and then constructed the solution around it. +1
Answer: Ok So note the following things for $n=20m$
$$d_{n+1}+d_{n+2}...+d_{n+4}=10 \\
d_{n+5}+d_{n+6}...+d_{n+15}=50  \\
d_{n+16}+d_{n+16}...+d_{n+20}=10$$
Can you use this to calculate the result?

Note that $n=20m\implies \mod(S_n, 10)=0$. Now $$S_{n+1}=S_n+20m+1 \\ S_{n+1}=1\mod(10) \\ S_{n+2}=3\mod(10) \\ S_{n+3}=6\mod(10) \\ S_{n+4}=0\mod(10)$$
Note that $n=20m+15\implies\mod(S_n,10)=0$, so similar to above argument we can find the sequence form $S_{n+5}$ to $S_{n+15}$.
Same argument work for $S_{n+16}$ to $S_{n+20}$.

So to answer the question $\sum_1^{2000}d_i=70\times\frac{2000}{20}=7000$, $\sum_{2001}^{2015}d_i=60$, $d_{2016}=6, d_{2017}=3$
Hence $T=\sum_{2001}^{2017}d_i=7069$, $T\%1000=69$
Answer: continuing
$$0,1,3,6,0,5,1,8,6,5,5,6,8,1,5,0,6,3,1,0,0,1・・・$$
You can find something.
Answer: In general, if $f(x)\in\mathbb{Q}[x]$ is an integer-valued polynomial of degree $d$ and $m$ is a positive integer, then
$$f\big(n+d!\,m\big)\equiv f(n)\pmod{m}$$
for every $n\in\mathbb{Z}$.  Hence, $d!\,m$ is a (not necessarily minimal) period of $\big(f(n)\big)_{n\in\mathbb{Z}}$ modulo $m$.  To prove this, it is well known that
$$f(x)=\sum_{k=0}^d\,t_k\,\binom{x}{k}$$
for some $t_0,t_1,t_2,\ldots,t_k \in\mathbb{Z}$.  Thus, $F(x):=d!\,f(x)$ is a polynomial with integral coefficients.  It is also commonly known that, for $g(x)\in\mathbb{Z}[x]$, $g(n+N)\equiv g(n)\pmod{N}$ for any $n\in\mathbb{Z}$ and $N\in\mathbb{Z}_{>0}$.  That is,
$$F(n+d!\,m)\equiv F(n)\pmod{d!\,m}\,.$$
Dividing both sides by $d!$, noting that $d!\mid F(n)$ and $d!\mid F(n+d!\,m)$, we obtain the desired result.
Now, since $S_n=\dfrac{n(n+1)}{2}$ is an integer-valued polynomial of degree $2$, we see that $S_n$ in modulo $10$ is periodic with period $2!\cdot 10=20$.  The rest is just as other users have concluded.

Sum of last digits of a sum


How to get Django admin.TabularInline to NOT require some items

Question: class LineItemInline(admin.TabularInline):
    model = LineItem
    extra = 10

class InvoiceAdmin(admin.ModelAdmin):
    model = Invoice
    inlines = (LineItemInline,)

and
class LineItem(models.Model):
    invoice = models.ForeignKey(Invoice)
    item_product_code = models.CharField(max_length=32)
    item_description = models.CharField(max_length=64)
    item_commodity_code = models.ForeignKey(CommodityCode)
    item_unit_cost = models.IntegerField()
    item_unit_of_measure = models.ForeignKey(UnitOfMeasure, default=0)
    item_quantity = models.IntegerField()
    item_total_cost = models.IntegerField()
    item_vat_amount = models.IntegerField(default=0)
    item_vat_rate = models.IntegerField(default=0)

When I have it setup like this, the admin interface is requiring me to add data to all ten LineItems.  The LineItems have required fields, but I expected it to not require whole line items if there was no data entered.
Answer: That's strange, it's supposed not to do that - it shouldn't require any data in a row if you haven't entered anything.
I wonder if the default options are causing it to get confused. Again, Django should cope with this, but try removing those and see what happens.
Also note that this:
item_unit_of_measure = models.ForeignKey(UnitOfMeasure, default=0)

is not valid, since 0 can not be the ID of a UnitOfMeasure object. If you want FKs to not be required, use null=True, blank=True in the field declaration.
Answer: Turns out the problem is default values.  The one pointed out above about UnitOfMeasure isn't the actual problem though, any field with a default= causes it to require the rest of the data to be present.  This to me seems like a bug since a default value should be subtracted out when determining if there is anything in the record that needs saving, but when I remove all the default values, it works.
In this code,
     item_unit_of_measure = models.ForeignKey(UnitOfMeasure, default=0)
it was a sneaky way of letting the 0th entry in the database be the default value.  That doesn't work unfortunately as he pointed out though.
Comment: Odd. Are you using an old version of Django? This very bug was fixed about a year ago (http://code.djangoproject.com/ticket/7975), and tests were included in the fix, so I'd be surprised if it regressed.


JAVA: How do you convert a String variable to a Double variable?

Question: I'm writing a program that is dealing with about 150 different variables, which I have extracted off a webpage. I was looking at how to change them into a double variable, so that I can compare them later in the program, and found this code to do so:
String text = "12.34"; // example String
double value = Double.parseDouble(text);

This works fine for some of them, but a lot of the numbers that I'm extracting from the webpage as a String variable come in the form '1,200.000', with the comma (,) to separate the number into thousands.
As a result, the program will not allow me to run the above code on them, because of that comma (,). How do I get rid of the comma, and hence change these Strings, like 1,200.000, into a double variable, like 1200.000?
Comment: You need locale aware parsing. See http://stackoverflow.com/questions/888088/how-do-i-convert-a-string-to-double-in-java-using-a-specific-locale
Answer: You can easily get rid of the comma :
double value = Double.parseDouble(text.replace(",",""));

Note that this will only work if comma's are used for the thousand separator. Other locales may switch the command and dot around.
Comment: Will I  be able to use that even if some of the numbers do not have commas, or will it return an error?
Comment: @Eran In French (and most of EU if I'm not mistaken) the thousand separator and the decimal separator are switched.
Comment: @owlstead What makes you think the OP cares about French locale? They only asked about getting rid of the comma.
Answer: First just delete the commas from the String and then parse it into Double:
String text = text.replace(",", "");
double val = Double.parseDouble(text);

Note that this will only work if comma's are used for the thousand separator. Other locales may switch the command and dot around.


Settings Autoack true in Rabbitmq and celery

Question: I am using celery and rabbitmq , but due to pushing several task in queue my server memory  utilization becomes more than 40% , so that rabbit further will not accepting any task . so i want to delete those message which are already executed , but due to durable behavior of rabbitmq those message not automatically delete, so i want to set some configuration like autoAck=True , so that if message is consumed from celery ,it will delete from rabbitmq queues and also from my server memory. please explain how can we do that .
Comment: If you never remove a message from the queue, then that's exactly what is going to happen. Why aren't you acknowledging the messages when you are done processing them?
Comment: I voted to close, because you didn't provide enough information for us to help you. From the description you gave, messages are not being removed from the queue, but there are many reasons why this might be. Please provide details as to the code you are using.
Comment: 1.One of the reason for this issue that i will pushing more messages in the queue and rate of consuming message from queue is less...so that messages in queue is continuously increasing and Rabbitmq keep messages in memory , as memory usage increase rabbitmq not taking any further messages after vm_memory_high_watermark ..
Comment: 2. i just want to celery configration . so that once message is consumed by celery or consumer , that message automatically deleted from rabbitmq.
Comment: But what is your question? The behavior described I the default behavior.
Comment: want to know why memory utilization becomes increase day by day for rabbitmq.
Answer: OK, so while I don't fully understand why you have the problem you have, it is clear what is going on.

A publisher puts a message task in the queue
Your worker process pulls the message and processes it
The message is never actually removed from the queue

This behavior happens when a consumer fails to acknowledge the processing of a message.  To confirm, if you look at the RabbitMQ management plug-in, you'll see a whole bunch of unacknowledged messages. These will be unavailable for consumption, but will continue to be held on the server and taking up disk space and memory.
Further, if you do a Basic.Recover, all of these messages will then get dumped back into the queue to be processed again.
This problem is due to incorrect configuration of your consumer. There are two ways to address this:

You can configure the consumer to auto-ack (i.e. acknowledge the message automatically upon receipt). This is done when you declare the consumer (using Basic.Consume).  Edit: It looks like this may be the default behavior of Celery.
You can configure your worker process to submit an acknowledgement (using Basic.Ack). Edit: this is done via the acks_late property in Celery.
Comment: Can u please tell us how i configure python Celery ( from consumer side) so that once message  is consumed , i will remove from queue.how i apply auto-ack true in celery.
Comment: I'm sorry, I don't use Celery. Perhaps search for "celery auto ack" or ask another question on SO so someone who knows can answer. Or maybe someone else can post another answer here.
Comment: please let me know how i process again unacknowledged message from rabbitmq.
Comment: Kill and restart the Celery process.


Vector operation of addition

Question: According to Law of Chasles, we know that:
$\vec{AB}+\vec{BC} = \vec{AC}$
However, for this question, it makes me confused about this law:
The question is to calculate $\vec{AB}-\vec{CD}+\vec{AD}+4\vec{BA}-\vec{BC}$
So if we calculate this according to the law, the procedure would be following:

$\vec{AB}+\vec{DC}+\vec{AD}+4\vec{BA}+\vec{CB}$
$\vec{AB}+\vec{AC}+4\vec{BA}+\vec{CB}$
$\vec{AB}+4\vec{BA}+\vec{AB}$
$\vec{AB}$ (because $4\vec{BA}+\vec{AB} = 4\vec{BB} = 0$ )

However, the right answer is $2\vec{BA}$, which I obtained using another way of calculation:

$\vec{AB}+\vec{DC}+\vec{AD}+4\vec{BA}+\vec{CB}$
$\vec{AB}+\vec{AC}+4\vec{BA}+\vec{CB}$
$\vec{AB}+4\vec{BA}+\vec{AB}$
$2\vec{AB}+4\vec{BA}$
$2\vec{BA}$ (because $2\vec{AB}+4\vec{BA} = 4\vec{BA} - 2\vec{BA}$ )

So the question here is why the first method is not right?
Comment: The first one is right down to the last step. I don't mean to be unkind, but I honestly don't know why you've come up with $\vec{AB}$, how you believe the parenthetical explanation is valid, or how the parenthetical explanation actually helps get to the presented answer.
Comment: Oh no worries. I just thought, according to the Chasles' law, that we can connect two vectors "head-to-end" as I showed at the beginning. So I just thought that in this case it will work as well
Comment: Perhaps you were missing the _order of operations_: $4\vec{BA}+\vec{AB}$ means $(4\vec{BA})+\vec{AB}$, not $4(\vec{BA}+\vec{AB})$ as you seemed to think.
Answer: If im not mistaken your conclusion is wrong in the last bullet is where it fails. Notice that:
$$\vec{BA}=-\vec{AB}$$
So:
$$4\vec{BA}+\vec{AB} \neq 4\vec{BB}$$
It is nothing but $-3\vec{AB}$. Instead we van calculate and get:
$$\vec{AB}+4\vec{BA}+\vec{AB}=2\vec{AB}-4\vec{AB}=-2\vec{AB}=2\vec{BA}$$
As required.
Comment: Okay. It is clear now to me now. I was just stuck at that step where I cannot really understand why. But I am glad that you provided me a very clear explanation. Thank you !
Answer: $$\vec{AB}-\vec{CD}+\vec{AD}+4\vec{BA}-\vec{BC}=$$
$$ AB+DC+AD+4BA+CB=$$
$$ AB + ( AD+DC+CB)+4BA=$$
$$AB+AB+4BA=2BA$$
Comment: Thank you for your explanation!

Vector operation of addition


Cycles of equally spaced points on a circle

Question: Take $n$ equally spaced points on a circle. Connect them by a cycle(circuit) with $n$ line segments. Two cycles are considered equivalent if same when rotated or reflected. How many cycles are there?

It can also be viewed as integer sequence.
Take an integer sequence $a_i(1 \leq i \leq n, \: 1 \leq a_i \leq n, \: a_i \neq a_j)$. Two sequences $a_n, \: b_n$ are considered equivalent if there exists some integers $k, \: l$ such that $a_i \equiv b_{i+l \bmod n}+k(\bmod n)$ or $a_{i+l} \equiv -b_{i+l \bmod n}+k(\bmod n)$
Comment: Have you tried finding the numbers for say $n=1$ to $n=15$ and putting the sequence at o.e.i.s.? Maybe your sequence is there.
Comment: What are your thoughts? What do you get for $n = 6$?
Comment: @coffeemath: Using a naive approach I had trouble even for $n=9$, but fortunately the numbers up to $n=8$ were enough to uniquely identify the sequence. And the description was enough to know that it's the right one.
Comment: The $+l$ in the left hand side of the second congruence seems wrong to me. Since you are talking about the equivalence of *two* sequneces, shouldn't these be $a$ and $b$ or similar? And how about an order reversal? Shouldn't you also include a case for $a_i\equiv b_{l-i\bmod n}+k\pmod n$ or similar?
Answer: This is A000940. The numbers grow fairly quickly, so for many applications, looking up a number in that list might be enough. I'll quote the terms up to the 20-gon:
<code> 3                1
 4                2
 5                4
 6               12
 7               39
 8              202
 9             1219
10             9468
11            83435
12           836017
13          9223092
14        111255228
15       1453132944
16      20433309147
17     307690667072
18    4940118795869
19   84241805734539
20 1520564059349452
</code>
I used the following naive python code to enumerate the first few items and identify the sequence:
<code>import itertools

def f(n):
    r = set()
    for p in itertools.permutations(range(n)):
        # cyclical shifts, correspond to a change in starting point:
        s = set(p[i:] + p[:i] for i in range(n))
        # include the reversed tuples, traverse path in opposite direction:
        s |= set([tuple(reversed(i)) for i in s])
        # modulo-add an integer to all elements, represents a rotation:
        s = set(tuple((i + k) % n for k in j) for i in range(n) for j in s)
        # also modulo-negate the elements, represents a reflection:
        s |= set(tuple(n - j - 1 for j in i) for i in s)
        s = frozenset(s)
        r.add(s)
    return r

for n in range(1, 10):
    print("{} {}".format(n, len(f(n))))
</code>
Hmmm. Come to think of it, that Maple code given there could indeed be read as a formula:
$$f(n)=
\frac1{4n^2}\left(
\sum_{d\mid n}\left(\left(\varphi\left(\frac nd\right)\right)^2
\cdot d!\cdot\left(\frac nd\right)^d\right)
+\begin{cases}
2^{\frac{n-1}2}\cdot n^2\cdot\left(\frac{n-1}2\right)!
& \text{for $n$ odd} \\
2^{\frac{n}2}\cdot\frac{n(n+6)}4\cdot\left(\frac{n}2\right)!
& \text{for $n$ even}
\end{cases}
\right)$$
Comment: I think OP wants to count two the same if they differ by an element of the dihedral group (which includes the reflections). There's an oeis entry for this also, http://oeis.org/A000940 (I havden't checked more into it...)
Comment: @coffeemath: You are right. I thought I had that covered, while in fact I only had changes in path traversal direction accounted for. (Interestingly, without those either, one ends up in [http://oeis.org/A002619](A002619). I wonder whether there is an easy way to see that connection.) Will have to edit my answer.
Comment: @coffeemath: One could of course assume that my original answer is still correct, because you get those same counts of you consider reflected versions the same, but don't allow reversing the path direction. (The two are essentially equivalent, it depends on how you map index and value to position on circle and on path.) As it stands now, the question doesn't mention path reversals, and the attempted formalization clearly only expresses a change of starting point, not a reversal. So it's currently asking for oriented cycles. But somehow I doubt that's intentional, so I keep my edited answer.


Are there any scales (in any musical culture) that can contain non-fixed members (ones that can be sharped or flattened at will)?

Question: I am wondering if there are any scales (that are in actual use) that also contain non-fixed pitches. By that I mean that in addition to having several fixed pitches, also have some "flexible" pitches that can be raised or lowered at will?
For example, are there any scales that can have either a raised or lowered scale-degree 3, where the descision whether to play a raised or a lowered 3rd is left at the discretion of the performer, and not prescripted by the scale itself?
Some examples of what I AM NOT after:

In the melodic minor, scale-degrees 6 and 7 are raised on ascent and lowered on descent. Yet, in the context of either ascending or descending they DO have a fixed pitch.
Similarly, in Indian music, the Arohana and Avarohana often contain different pitches, but those pitches are fixed in either context.
Chromatic passing notes, which by definition are not intrinsically part of the scale.
The use of modal mixture, that through some harmonic alteration may impact the notes of a melody.
In the Klezmer scale Adonai malach (on C), there's an E natural when playing within the 1st octave of the scale, and an Eb when playing in the 2nd octave of the scale (see below). So the scale sometimes has E natural and sometimes Eb, but those too have very clear contexts, and in that sense those pitches are fixed as well.

Purely, to illustrate my point, I am attaching an example (below) of such scale (which I invented for the sake of explaining my question). This C scale, can make use of either E natural or Eb, as well as A natural and Ab interchangeably (at the discretion of the performer). In that sense, the scale itself contains some non-fixed pitches.

Are there such scales in wide use anywhere?
UPDATE
The Hindustani Raag Pilu (or Piloo) seems to fit the bill. Any other examples?
Comment: How close does musica ficta get to what you want? This involves unnotated, often optional accidentals.
Comment: As far as I know, musica ficta doesn't really leave any room for choice. I may be mistaken, but musica ficta MUST be enacted to abide by counterpoint rules and so, it an example of a non-flexible note that is inferred by context, but I wouldn't really call it a flexible note. What do you think?
Comment: A scale is just a set of notes which has been contrived by us. So chromatic would just about cover this.
Comment: @Tim A scale is much more than a contrived set of notes. It's a role granting musical framework that in some form or another is used ubiquitously across the world. As far as I know, the chromatic collection of notes is not used as a scale anywhere.
Comment: Exam boards would disagree with you there.
Comment: @Tim Ok, thank you. I'll let my PhD committee know that the last 5 years of my research should be disregarded because of a comment a guy wrote to me on the internet. But seriously, if you are curious, I'm happy to provide you with some hundreds of citable sources by reputable scholars from across the world and throughout the centuries.
Comment: Isn't it mostly a matter of how we define a scale? If your non-fixed example had an optional b7 as well, it would have all the notes of a major scale as well as the notes of the parallel minor. Using both in the same piece [isn't uncommon](https://youtu.be/Cf5iPS3-Oug?t=309). We could call it modulating between different fixed scales, or say that we are exercising our performer/composer discretion to use different subsets of a flexible scale that contains both sets of notes. Amounts to the same thing, doesn't it?
Comment: Likewise, anything that can be described as borrowing notes or chords from elsewhere should have an equivalent description as using a flexible scale that includes the borrowed parts as optional elements.
Comment: @j-g-faustus what you are referring to is a case of modal mixture, where two fixed modes  are used interchangeably. This is usually the result of borrowing harmonies (and then melodies follow suit). In this sense, this isn't exactly what I'm after, but I should have been more explicit in my original question. Let me update. Thanks for taking the time to contribute some thoughts!!
Comment: I keep on saying "Don't believe a lot of what you see on the 'net" so why be sarcastic? As it stands, the question is (to me) vague.  And what constitutes a scale, in your opinion, anyway? A set of notes which ..?
Comment: @Tim A scale (unlike a set), is a collection of notes that are bound by hierarchical relations, where usually there's a central "tonic" or "final" note, and a "dominant" or "recitation" note. And where, certain notes have certain tonal tendencies (like, say, the way a leading tone pulls up). In that, and in many other senses, a scale is not just arbitrary collection of notes you can pull out of a hat. Some combination of notes can bind to form scales, and others, no matter how much you try, will never be able to form those intrinsic relationships.
Answer: I would be willing to bet that there are many throughout different cultures but what immediately comes to mind for me is the way I sometimes improvise over blues. I use an unofficial composite scale made up of combining parallel major and minor pentatonic scales and adding the #4/b5 blues note as well. It's possible or even likely this scale has a name, I've never researched it. It looks like this in the key of C:

This scale allows for playing either major or minor pentatonic lines and also chromatic lines utilizing notes from both scales combined. It is completely chromatic from the D to the G. I even sometimes add extra chromatic passing notes between the ^5 and ^6 and also the ^b7 and ^1. The bottom line is you can use whichever notes you like at your discretion depending on whether you want a major or minor tonality or a more chromatic style.
Comment: @MichaelSeltenreich There are so many but the intro to "Straighten up and Fly Right" by the Nat Cole Trio immediately comes to mind. ^b3 ^3 ^1 etc. There are also countless examples of blues/rock guitarists playing major pentatonic scales on the I chord then switching to a minor pentatonic on the IV chord.
Comment: One great example I found so far is the Pilu Raga from the hindustani musical tradition. In it, E, A, and B, can appear both as natural or flattened notes. https://en.wikipedia.org/wiki/Pilu_(raga)
Comment: I had thought about something like this as well. Can you think of any recorded or notated examples? I know I definitely heard that E and Eb played interchangeably in the context of blues, but an obvious example doesn't come to mind.
Comment: @MichaelSeltenreich If you combine the blues scale with the bebop scale, then the available pitches are ^1, ^2, b^3, ^3, ^4, #^4, ^5, b^6, ^6, b^7, ^7, ^8. If you want recorded examples, jazz. "Giant Steps", perhaps. But that's just a fixed scale that has a lot of flexibility. There are contexts wherein a musician can use all 12 notes in a piece. Is that what you're looking for?
Comment: Yes, this is a 'scale' that I use a lot. And I'm certain a lot of blues/jazz players do too. Basically every chromatic note, bar the two either side of the root. But even those come out to play sometimes! B as part of the Bebop, and C# as a kind of tts. I tell students that any note in any piece at any time is ripe for use, they just have to know how to play it. +1.
Comment: @Tim you are right in that every note in the chromatic scale can be useful in certain situations. Some may think that some of the notes will only function as passing tones and many times they do but even the C# (in C) can be the leading tone of the V/ii and the B can be the leading tone of the V chord.
Comment: So that's it then. The chromatic scale. And for those who can play notes 'in the cracks', include them too!
Comment: And - C# is the Blue note for V of  key C !  Let alone B being the major third note of the same. So that's all covered. That's if one decides a scale is comprised of notes used in a piece...
Comment: Also, the blue note is often "bent," giving a range of options for exact intonation
Comment: This is a wonderful discussion. Just want to chime in and say that I don't consider a passing chromatic note as belonging to the scale. So while non-scale notes may appear in the context of music, as long as they are non-scale notes, then they are.... non-scale notes.
Answer: A scale, by definition, is fixed.  But there's this thing called 'chromatic notes'.   A jazz player may interpret the chord 'C major' by basically using notes of the C major scale, but making frequent additional use of E♭ and G♭.  It isn't a new scale, E♮ and G♮ are still used, but other notes are too.


Suppose $H$ and $K$ are normal in $G$ and $H \bigcap K = \{e\}$. Show that $HK = G$

Question: Suppose $H$ and $K$ are normal in $G$ and $H \bigcap K = \{e\}$.   Show that $HK = G$
Since both $H$ and $K$ are normal, we have the property that for every $g \in G$, $g^{-1}Kg = K$ and $g^{-1}Hg = H$.  Further we have $H \bigcap K = \{e\}$.  So $H(H \bigcap K) = H\{e\}$ i.e. $H \bigcap HK = H$.  Similarly $K \bigcap HK = K$.  This means that $H \bigcup K \subseteq KH$.  Now for each $g \in G$ we have $g^{-1}(H \bigcup K)g = g^{-1}Hg \bigcup g^{-1}Kg \subseteq g^{-1}KHg$, so $HK$ is normal.
Now I know that if $HK$ is normal, it's order divided by the order of the intersection of $H$ and $K$ which is the trivial group of order 1 will equal the order of $G$.  Since the order of $HK$ = order of $G$, we have that $HK = G$.
Is this correct?  If not, please help me find the right steps.
Comment: What if $H=K=e$?
Comment: Is $G$ is a finite group?
Comment: Yes, $G$ is finite.
Comment: This is false without some further assumptions -- take $H = K = \{e\}$ subgroups of any nontrivial $G$.
Comment: Can you clarify your question?  As it stands, the statement you seek to prove is false.  Are you implicitly making some assumptions regarding $G,H,K$ that you didn't mention?  Or does the counterexample end the question?
Comment: As $|HK| = |H||K|/|H \cap K| = |H||K|$, you won't have $G=HK$ unless $|G| = |H||K|$. For a nontrivial counterexample, take $G = \mathbb Z_2 \times \mathbb Z_2 \times \mathbb Z_2$ and let $H$ and $K$ be any two distinct subgroups of order $2$.
Comment: I think $H$ and $K$ are meant to be nontrivial
Answer: This is not true: let $G=S_3 \times S_3$, $H=A_3 \times \{1\}$ and $K=\{1\} \times A_3$. Then $|G:HK|=4$.
What you can conclude is that $H$ and $K$ commute.
Comment: Yes typo, thanks, edited.


Pseudo random number generation for sum of normal-distributed values

Question: Les us give N distribution functions $f_{1}, f_{1},... f_{N}$ each for normal distributed random value (with shifted from zero mean vector) on n-dimensional space $\mathbb{R}^{n}$. We can construct an arithmetical mean of them which also will be a distribution function for some vector random value. How can I generate random values based on this new random value from uniform distribution?
For N=1 this task can be solved by Box–Muller transform, but what if I need to generate random vectors based on sum of two, three or more normal distributions?
Comment: Are you asking about how to select from a distribution which is the mean of $N$ other distributions, $f(x) = \frac{1}{N}\sum_{m = 1}^N f_m(x)$ ?
Comment: Yes. The general purpose is to develop algorithm for random values generation for such sums that give me two or more "mean" values, so the calculated values will concentrate near and between them.
Comment: It should be pretty simple, then. To generate a sample from the "total distribution" $f(x)$, just select one of the $N$ "sub-distributions" $f_m(x)$ at random with equal probability, and then use Box-Muller to generate a sample from $f_m(x)$. The overall distribution of such samples should be that of the mean distribution.
Comment: I don't quite understand why. Where is actually a "sum" of values. What if I change arithmetical mean to arbitrary convex sum with weights $\alpha_{m}$?
Comment: I've figured out. In this case I must select sub-distributions with $\alpha_{m}$ probabilities. Please, answer this question, and I will mark it as correct one. Thanks!
Answer: Suppose we have $N$ probability distribution functions (pdfs) $f_m(x)$.  Since they are pdfs, each of the $f_m(x)$ satisfies $f_m(x) \ge 0$ and $\int_{-\infty}^{\infty} dx\, f_m(x) = 1$. It is assumed that we have a method for generating samples from each of these pdfs. Define their weighted mean $f(x)$ as
$$
f(x) = \sum_{m=1}^N \alpha_m\, f_m(x)\, ,
$$
with weights $0\le \alpha_m \le 1$ such that $\sum_{m=1}^{N}\alpha_m = 1$. $f(x)$ is then also a pdf, since clearly $f(x)\ge 0$ and
\begin{align}
\int_{-\infty}^{\infty}dx\, f(x) &= \sum_{m=1}^N \alpha_m\, \int_{-\infty}^{\infty}dx\, f_m(x)\\
&= \sum_{m=1}^N \alpha_m \times 1\\
&= 1\, .
\end{align}
To generate a random sample $X$ from the pdf $f(x)$, first pick a random integer $1\le m\le N$ from the discrete distribution with probabilities $\alpha_m$. (This link describes methods for this.) Then set $X$ equal to a sample generated from the distribution $f_m(x)$.
To see heuristically why this works, imagine generating many samples in this fashion, and filling in a histogram to describe their distribution. Each sample will have a probability $\alpha_m$ of populating the "sub-histogram" associated with $f_m(x)$. Clearly, then, the resulting distribution will be a weighted mean of the "sub-distributions" $f_m(x)$.


Reloading option in a select list with ajax

Question: So here is my problem:
I have a form so users can create orders. There is a 'Client' select list where they can select the client of the order. When we select a client, we use ajax to load client info (address, tax rate, discount ...) and populate some field. This works fine.
I am now adding a feature to be able to create directly a client from the order, in a modal.
So I have a button in the order form, that triggers ajax to open a modal and render the new client form:
  $account_form = drupal_get_form('mymodule_creation_account_custom_form',$form['field_client']);
  $output = drupal_render($account_form);

  $commands = array();
  $commands[] = ajax_command_invoke('div#createClientModal','foundation',array('reveal', 'open'));
  $commands[] = ajax_command_replace('#new-account-form-placeholder',$output);
  return array(
    '#type' => 'ajax',
    '#commands' => $commands,
  );

Then I fill in the info, and submit the account from the modal:
  $account = my_module_new_account_form_submit($form, $form_state);

  $form['client_hidden_field']['und']['#options'][$account->nid] = $account->title;
  $form['client_hidden_field']['und']['#default_value'] = $form['client_hidden_field']['und']['#value'] = $account->nid;

  $commands = array();
  // Closing the modal
  $commands[] = ajax_command_invoke('div#createClientModal','foundation',array('reveal', 'close'));
  $commands[] = ajax_command_replace('.field-name-field-client', render($form['client_hidden_field']));
  $commands[] = ajax_command_invoke('.field-name-field-client select','change');

  // Replace the form with an empty div
  $commands[] = ajax_command_replace('div#account-form-wrapper','

');

  return array(
    '#type' => 'ajax',
    '#commands' => $commands
  );

So what I am doing in this last submit, is creating the new account (that works), it returns me the nid. Then, as I have stored in a variable ($form['client_hidden_field']) the client field, I update it with the new option, select the option, replace the old client select list with the new one, and trigger change to call the old callback that loads a client info when it is selected.
I have tried lots of thing and each time I select the new client created, I get an 'Illegal choice'. I have tried without default value, without triggering a 'change', in JS with a command_invoke and nothings seems to work.
The only success I had was :
I do all these step, I manually select a client (one already existing before), and then select the new one. After that I didn't have the 'Illegal choice' and was able to save. But this is not a way to make it work.
If someone has any idea, I would love your help !
Thanks
Ed
Comment: I think this is a good place to understand a bit better where I am wrong: http://drupal.stackexchange.com/questions/15989/resolve-the-error-an-illegal-choice-has-been-detected

But because the second ajax callback is about the second $form in the modal, it returns me the new refreshed field, in the modal, not replacing the previous one!
Answer: Have you tried with $form_state['rebuild']=TRUE. It will help you to update your changes while adding new client.
Comment: Where would you add it? It should be when the modal close I guess, but at this point, my $form and $form_state are the one from new client form in the modal.
Comment: You should right it in hook_submit. your current submit hook hook as mention above. you code put it after this line:`$form['client_hidden_field']['und']['#default_value'] = $form['client_hidden_field']['und']['#value'] = $account->nid;`
Answer: Ok, so I was finally able to succeed this. I didn't manage to return the new field to the form properly (cf. Resolve the error "An illegal choice has been detected...") because I am working with two forms (and maybe I am missing something).
So skipping the validation for that field in form.inc seems to be the only solution that I have. And the only way I found was to add a function in #after_build to unset '#needs_validation':
In form_alter:
array_unshift($form['field_client']['und']['#after_build'],'remove_client_validation');

And the function:
function remove_client_validation($form_element) {
  unset($form_element['#needs_validation']);
  return $form_element;
}

Thanks to: I use jQuery to alter the exposed form select options. Got error: "An illegal choice has been detected." How to bypass validation?, where I found that solution.


Understanding Confidence Intervals to Reach Significance

Question: My question is in regards to whether you can use confidence intervals to claim statistical significance?  For example, if I calculate the CI at 95% for weekly sales to be between $100–$200 and one week I reach sales of $250, is this a significant result?  If not, how exactly would you describe the effect in relation to the CI we calculated?
Answer: Sometimes you can interpret CI coverage in terms of significance testing, and sometimes you cannot depending on the specific test and null hypothesis. There's a literature on the issues that arise which is sometimes termed 'visual hypothesis testing', a brief bibliography of which is presented here (I might start with the article by Geoff Cumming):
Afshartous, D. and Preston, R. (2010). Confidence intervals for dependent data: Equating non-overlap with statistical significance. Computational Statistics & Data Analysis, 54(10):2296–2305.
Browne, R. H. (1979). On visual assessment of the significance of a mean difference. Biometrics, 35(3):657–665.
Cumming, G. (2009). Inference by eye: reading the overlap of independent confidence intervals. Statistics In Medicine, 28(2):205–220.
Goldstein, H. and Healy, M. J. R. (1995). The graphical presentation of a collection of means. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158(1):175– 177.
Payton, M. R., Miller, A., and Raun, W. (2000). Testing statistical hypotheses using standard error bars and confidence intervals. Communications in soil science and plant analysis, 31(5):547–551.
Payton, M. E., Greenstone, M. H., and Schenker, N. (2003). Overlapping confidence intervals or standard error intervals: What do they mean in terms of statistical significance? Journal of Insect Science, 3(34):1–6.
Smith, R. W. (1997). Visual hypothesis testing with confidence intervals. In Proceedings of the Twenty-Second Annual SAS® Users Group International Conference.
Tryon, W. W. and Lewis, C. (2008). An inferential confidence interval method of establishing statistical equivalence that corrects Tryon's (2001) reduction factor. Psychological Methods, 13(3):272–277.
Comment: The term "significant" often gets in the way of clear statistical thinking.. Try rewriting your question without using that word and with more specifics about what question you are trying to answer.
Comment: I used to believe confidence intervals were simply concatenated significance tests. From your answer I now believe it was merely because I was dealing with two-tailed t-distributions.
Comment: @lucasfariaslf Confidence intervals are a form of statistical inference which—speaking loosely—provides a range of plausible estimates (e.g., "estimates may be as low as *lower bound*, or as high as *upper bound*"). By contrast, hypothesis tests are a form of statistical inference which provide *evidence against a null hypothesis*, and are more like procedures for making decisions about an image of the world (e.g., "we found evidence that the world *does not* look like $H_{0}$, so we conclude the world looks like $H_{A}$").


Convergence with even and odd subsequences

Question:
Let $(a_n)_{n=1}^\infty$ be a sequence of real numbers satisfying:

The subsequence $(a_{2k})_{k=1}^{\infty}$ is decreasing;
The subsequence $(a_{2k-1})_{k=1}^{\infty}$ is increasing;
$\displaystyle\lim_{n\to\infty}|a_{n+1}-a_n|=0$.

Prove that $(a_n)_{n=1}^\infty$ is convergent.
Comment: What did you try?
Comment: @JoséCarlosSantos, I know that $\lim |a_{n+1}-a_n|=0$ is not sufficient for $a_n$ to be convergent, but I have no clue how to incorporate the 'increasing and decreasing' subsequences in a proof. I think I need to use the Cauchy criterion, but I don't know how to proceed :(
Comment: Hint:: The sequence $u_k=a_{2k}-a_{2k+1}$ is decreasing and $\to 0$ ; Show that this imply that $u_k\geq 0$ for all $k$.
Answer: Suppose that for two natural numbers $k$ and $l$, $a_{2k}<a_{2l-1}$. Suppose furtermore that $k\leqslant l$. Then $a_{2l}\leqslant a_{2k}<a_{2l-1}$. Therefore, $n\geqslant l\implies a_{2n}\leqslant a_{2l}<a_{2l-1}\leqslant a_{2n-1}$ and so$$n\geqslant l\implies|a_{2n}-a_{2n-1}|=a_{2n-1}-a_{2n}\geqslant|a_{2l}-a_{2l-1}|$$and this contradicts the assumption that says that $\lim_{n\to\infty}|a_{n+1}-a_n|=0$. You'll also reach a contradiction of $k\geqslant l$.
So, you always have $a_{2k}\geqslant a_{2l-1}$. Now, take $\varepsilon>0$. If $N$ is large enough, $a_{2N}-a_{2N-1}<\varepsilon$. But then, if $n\geqslant N$, we have $a_{2n}\leqslant a_{2N}$ and $a_{2n}\geqslant a_{2N-1}$. And we also have $a_{2n-1}\leqslant a_{2N}$ and $a_{2n-1}\geqslant a_{2N-1}$. So, after a certain point, all terms of the seqeunce belong to an inter whose length is smaller than $\varepsilon$. Therefore, $(a_n)_{n\in\mathbb N}$ is a Cauchy sequence, as you supected.


copy kvm image from ceph to other storage

Question: A while ago I created a ceph storage cluster and connected it to proxmox2 according to the method described in http://pve.proxmox.com/wiki/Storage:_Ceph.
We have some kvm images running on the ceph storage that where originaly just for testing purpose but over time they are becoming more and more important for our company. Because I don't consider the ceph storage as production ready yet, I would like to move the kvm images to our ISCSI storage. Does anyone know how I can copy the images from ceph to other storage? Knowing how to copy them to local storage would be sufficient.
Answer: You can easily transfer images in and out of Ceph RADOS with the rbd command. I am currently using libvirt to manage my KVM images, so my paths may be different.
The libvirt configuration for one of my virtual machines uses an RBD disk image located at data/vm-www. To see that RBD pool, I can use the rbd ls -p data command. For any other command, such as getting information about the image, you can use a shortened form: rbd info data/vm-www, instead of the longer rbd info -p data vm-www.
Now that you have verified you can find the image, use this command to save it to a local file: rbd export data/vm-www www.raw. www.raw is now a file that contains your virtual machine's disk, and you can tell your libvirt or qemu-kvm command to use it directly as a raw file. Alternatively, you can use qemu-img to convert it to a different format, such as qcow2.
I do not have experience with RBD snapshots, but the rbd export command lists a "--snap" option, presumably to be used with information that you can get from rbd snap ls data/vm-www.
Hope this helps!


Joint distribution of $(|Z_1|,\alpha \sum_{i=1}^nZ_i^2)$ when $\alpha \ge 0$ is deterministic and $Z_1,\ldots,Z_n \overset{iid}{\sim} N(0,1)$

Question: Let $Z=(Z_1,\ldots,Z_n)$ be a random vector in $\mathbb R^n$ with iid entries from $N(0,1/n)$ and let $\alpha \ge 0$ be a constant. Define $X := |Z_1|$ and $Y_n:=\alpha \|Z\|^2 := \alpha \sum_{i=1}^nZ_i^2$.
Question. What is the joint distribution of $(X,Y_n)$ ? What is the limiting form of this distribution ?
Answer: In general, the joint density of $\left(X,Y\right)$ can be written as the product of the conditional density of $Y$ given $X$ and the marginal density of $X$.
Here, let's work with $X^2$ instead of $X$ as it is a bit easier. Then, the marginal distribution of $X^2 = Z^2_1 = \frac{1}{n}Z^2$ for $Z\sim \mathcal{N}\left(0,1\right)$. Then, $Z\sim Gamma\left(\frac{1}{2},\frac{1}{2}\right)$, also known as a chi-squared distribution, so $X^2 \sim Gamma\left(\frac{1}{2},\frac{n}{2}\right)$ using the scaling properties of the gamma distribution.
Now, $Y_n$ has the form
$$Y_n = \alpha X^2 + \sum_{i>1} \alpha Z^2_i,$$
where the second term is independent of $X^2$ and has the distribution $Gamma\left(\frac{n-1}{2},\frac{n}{2\alpha}\right)$ since you are adding up $n-1$ independent $Gamma\left(\frac{1}{2},\frac{n}{2\alpha}\right)$ distributions. This should give you a form for the conditional density of $Y_n$ given $X^2$, and you can look at the product and see if it simplifies to anything useful.


CSS, JS not loading on AKS

Question: Have an Angular SPA which is dockerized and deployed on aks through helm. I could see the application works as expected when deployed on standalone docker. But when deployed on AKS, dont see the resources loading. When inspecting the logs could see

"GET /" "Mozilla/5.0

where it should have been /abc/a.css, so figured out my ingress is not letting the traffic pass through with the path for my service and tried setting the app-root, multiple rewrite target example. But none seemed to help and below is my ingress.yml
Ingress.yml:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: abc-ingress
  namespace: abc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/app-root: /abc
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: fbmdashboardpoc-service
          servicePort: 80
        path: /abc(/|$)(.*)

Thanks is advance
Comment: Maybe this could help https://stackoverflow.com/questions/56989555/angular-on-kubernetes-ingress-routing
Comment: @JoseGuzman Tried it no luck still doesn't appear to solve my issue
Answer: The issue was with my ingress after configuring deployUrl, also made below changes to my ingress file to get it working
kind: Ingress
metadata:
  name: abc-ingress
  namespace: abc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: fbmdashboardpoc-service
          servicePort: 80
        path: /abc/(.*)

Hope it helps if others are struck with same issue
Comment: This path works, but when we refresh on some other paths , it thows 404 erroe

CSS, JS not loading on AKS


If your son goes to an all-boys school, are his classmates more likely to have a brother or a sister?

Question: Steve's son goes to an all-boys secondary school and Steve is going to collect him from school after his first day of school.
Steve does not know any other parent yet and meets Richard at the gates, who also has a son in the same class. Richard tells Steve that he also has another child.
What is the probability that Richard's other child is also a boy?
Note: this is to try to better understand the Boy-girl paradox (https://en.wikipedia.org/wiki/Boy_or_Girl_paradox), which was pointed out to me in an anwser to one of my earlier questions
Comment: On an empirical note, people might also want to look into [this](http://www.ncbi.nlm.nih.gov/pubmed/18027818). Although I think it is completely unrelated to the the answer that you are seeking.
Answer: With all of the reasonable necessary assumptions (equal probability of boy or girl, probability of sending one child to an all boy's school independent from gender to other child......) the probability that Richard's other child is also a boy is (drum-roll)
$$
\frac{1}{2}.
$$
When you hear these problems a good thing to think is "can I label them in such a way that I know the gender of one of the labeled children?"  In this case I will say the child in Steve's sons class is child A and the other child is child B.  For simplicity later on I'll refer to boy as 1 and girl as 0.  Our assumptions boil down to:
$$
P(A=x,B=y) = P(A=x) \cdot P(B=y) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}.
$$
The question you are asking concerns $P(B=1 | A=1)$.  By independence or direct computation we will easily get $1/2$.
The paradox focuses on the scenario where I know someone has two children, and I know they have at least one boy.  No matter how I try to label the children, (youngest=A, oldest=B) (shortest=A, tallest=B) I don't know the gender of one of the labelled children.  Notice I can not talk about the gender of "the other child" because I don't know of one specific child who is a boy.  This is quite different than the other problem.  Once i pick some arbitrary labelling, the quantity of interest mathematically is
$$
P(A+B=2 | A+B\geq 1) = \frac{1}{3}.
$$
Comment: While the intuitive answer would look 1/2 to me as well, I can't find what's wrong in the below reasoning:
The parents of an all-boys school have all at least one boy. If we suppose that Steve is only interested to speak with families with 2 children, speaking with a parent of that school does not add any new information to him. This is simply a family who has at least one boy. Hence if he would ask the same question to 10 different parents with 2 children, this would be a random sample of families with at least one boy, for which the probability ot having 2 boys is 1/3.
Comment: @user90213 haha.  I love how subtle this problem is.  You said Richard has a son *in Steve's son's class* (I assume you meant exactly one).  This allows me to label the students and I know A=1.  If I walk up to a random two child parent there all I know is A+B is at least 1.
Comment: It would have been more tricky if you just saw a parent at the boy school gates, without knowing whether his son is in your son class or not.
In that case I'd say the probability is somewhere in between 1/2 and 1/3, and it would depend on things like number of classes in the school, spacing distribution between childs etc
Answer: It's 50% (or whatever the actual $p$ of a male vs female set is in the population drawn from, but 50% for simplicity.)
We know for a fact that this concrete child is a boy, so the only open situation is the gender (sex?) of the other child.
Another way to frame this particular question is:
"There is a child of unknown gender. What is the probability that the child is male?"
To illustrate the paradox, it would be better to examine a question like this:

From the families with exactly two children and at least one boy, what is the probability that
  one of these families chosen at random has two boys?
Comment: I'm woondering how the probabilty would change if Steve would have meet Richard in a boys clothing section of a department store. In that case I assume it would be more like 1/3?
Answer: The probability is 1/2 cause P(girl|boy)=P(girl) cause they are independent and the fact that Richard has a boy doesn't give any useful information about the other child.
Comment: "Because they are independent" is precisely what needs to be shown.


.net XML Documentation Comment include directive in sandcastle not adding to help file

Question: Have an interesting one. I'm documenting a method using XML documentation comments in C#. I want to use the  directive to include all the contents of a referenced file. I am documenting a piece of code which receives a string formatted as an xml document, and I want to include the XSD file in the parameter documentation.
This works fine in that the XML generated by Visual Studio is as I expect. Here's a snippet of the XML:
<code>    /// <param name="xml">XML string - see Test.xsd <include file="XSD/Test.xsd" path="/*"/> After file include</param>
</code>
However, when I use Sandcastle to generate a help file from this, the contents of the xsd file are not shown, but Sandcastle does not report any errors. I've tried to use the file name as relative to where the help files are as well, i.e.

but then Visual Studio complains it can't find the file. I put them in both places and still the included file does not appear.
Any ideas?
Answer: The include tag is processed at compiler time, so the generated XML file should contain all of its referred contents. Hence you have to put your XSD folder relative to the C# source file, then the compiler won't show any errors. After this, Sandcastle will be able to create the help file from the single XML document.
Comment: Note: I've found that the included contents end up getting escaped for HTML, rendering you unable to have things like links in your included text. I am yet to find a way around this.


Is Loading Bitmaps for static use bad for Garbage Collection?

Question: I have created a class that loads Bitmap images and stores them in a static context:
public class ImgLoader extends View {

public static Bitmap tree1;

public ImgLoader(Context context) {
    super(context);
    loadImgs();
}

public void loadImgs() {
    tree1 = BitmapFactory.decodeResource(getResources(), R.drawable.tree);
}
}

I then later access this image by doing this:
Bitmap tree = ImgLoader.tree1;

Are there any alternatives to doing something like this? Because it may cause some issues with the GC.
Comment: Load the image when it is required. Don't keep it in memory unnecessarily .
Answer: You can add this method to you class
public void recycleBitmap(){
    if(tree1 != null){
        tree1.recycle();
    }
    tree1 = null;
}

Answer: Yes. Loading bitmap in static way is bad for garbage collection. Since bitmap will remain in memory, it will consume the memory and will not be garbage collected. However there are few benefits.
This is a better way to display images. If you are using same image multiple times, making it static and reusing will surely save some memory. Also If you want to display image on multiple views and do not want to load them, making it static will help you. As we know, inflating UI is a heavy operation so as the loading bitmap from IO. Thus static images are good to use especially if there are several copies needed.
Answer: You can use a SoftReference to store your reference to the image. Then if no-where in your program is using the image the garbage collector is free to clean it up.
You will need to reload the image if it has been garbage collected but otherwise it will stay in memory ready to be re-used.


When to stop refining a model?

Question: I have been studying statistics from many books for the last 3 years, and thanks to this site I learned a lot. Nevertheless one fundamental question still remains unanswered for me. It may have a very simple or a very difficult answer, but I know for sure it requires some deep understanding of statistics.
When fitting a model to data, be it a frequentist or a Bayesian approach, we propose a model, which may consist of a functional form for likelihood, a prior, or a kernel (non-parametric), etc. The issue is any model fits a sample with some level of goodness. One can always find a better or worse model compared to what's currently at hand. At some point we stop and start drawing conclusions, generalize to population parameters, report confidence intervals, calculate risk, etc. Hence, whatever conclusion we draw is always conditional on the model we decided to settle with. Even if we are using tools to estimate the expected KL distance such as AIC, MDL, etc., it doesn't say anything about where we stand on an absolute basis, but just improves our estimation on a relative basis. It seems there is no objectivity as the model error is completely ignored.
Now suppose that we would like to define a step by step procedure to apply to any data set when building models. What should we specify as a stopping rule? Can we at least bound the model error which will give us an objective stopping point (this is different than stopping training using a validation sample, since it also gives a stopping point within the evaluated model class rather than w.r.t. the true DGP)?
Comment: I think you should add other tags to the question than just inference, e.g. some modeling tags and model-selection. I think that might be relevant for this is also [Occam's razor](https://en.wikipedia.org/wiki/Occam's_razor). [Here](http://papers.nips.cc/paper/1925-occams-razor.pdf) is also a paper that discusses it w.r.t. Bayesian modeling.
Comment: +1 for the question. In large part, the concerns are philosophical or epistemiological, i.e., not only "what do we know and how do we know it" but "what *can* we know and how *can* we know it?" As the physicist Richard Feynman said, "It is impossible to find an answer which someday will not be found to be wrong." In other words and unless you are religious, there is reasonable doubt whether a univocal, eternal ground truth exists on which to anchor *anything.*.
Comment: Sometimes you build a particular model because it is particularly good for estimating certain parameters, not because you think the overall distribution is accurate (see M-estimation, generalized estimating equations) etc. So, if you really care about a decent estimate of location, you may be better off with a wrong model but one that is not easily thrown by noise (for your parameter of interest). In general, see Robust Estimation.
Comment: Very interesting question. Just a comment that at least in the Bayesian setting one than also raises the question of averaging over the plausible subset of models rather than choosing one. I am not sure of any theoretical way to the OP's question and I guess practically it is done by whether the picked model is good enough for the problem we are trying to solve. Perhaps we need model selection by MCMC methods or something like that! I can envision a nested MCMC approach for this...
Comment: @Luca This has been done. However the problem remains as the space of models defined by the Bayesian prior may or may not contain the true model. Even if it did the model error is still there, that is the error of the average model with respect to the true DGP.
Comment: That sounds like an information theoretic version of *Laplace's Demon* as applied to Newtonian mechanics (https://en.wikipedia.org/wiki/Laplace%27s_demon). As assumptions go, this new "school's" position is intellectually very imperialistic. Regardless, there are all kinds of cockamie "schools of thought" out there. Are you suggesting that this school's existence somehow dominates or is involved with your decision to award the bounty?
Comment: You've had at least one excellent and thoughtful answer to your query but seem to be holding out for something more. Is it possible you're "toying" with us in not taking the responses your query has produced seriously enough?
Comment: >Now suppose that we would like to define a step by step procedure to apply to any data set when building models.
 [Solomonoff induction](http://lesswrong.com/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/) is all about just that.
Comment: I'm confused as to why @Tim did not answer your question. Your question is whether we can bound the *model* error, as opposed to the *empirical error* (e.g. RMS, Type I or Type II errors). As both Tim and I pointed out, there is no objective measure of model error, so we use the very things you don't want to use (AIC, KL-div, Cross-validation). That is the answer, and Tim (and me to a lesser extent) addressed it.
Comment: (cont'd) even if we assume that there is stable/time stationary DGP out there, we need to quantify two things: The difference between the current model and the "true model" (some such measure have already been mentioned), but we also need to describe the "model generating process" (MGP) that is attempting to discover the true DGP. Your assumption that we could every calculate the "model error" presupposes that the MGP is probabilistic with some, a priori known, distribution.
Comment: For instance, in nonparametric Bayesian inference, we use the Dirichlet Process Prior to perform inference over the entire space of discrete distributions, and there are continuous versions as well. But even with this type of model, you need to specify the mean of the process and you are assuming that the model is either continuous or discrete, or time-invariant.
Comment: So, as you can see, what @Tim and I were getting at is that model risk is a philosophical issue, not a statistical issue. As we will ever know is if the current model works "well enough" not that it is the "correct" model. As a final thought, consider that the correctness of a model can be dependent on the range of your observations. Many distributions are almost indistinguishable from a normal distribution over a sufficiently restricted range about the mode.
Comment: @CagdasOzgenc that's exactly my point...the definition of the hypothesis space is a subjective matter, so just having the data and an assumed hypothesis space will not get you any closer to knowing how close you are to the true DGP, which would require you consider every possible hypothesis.
Comment: I edited the title to make it more specific and search friendly. The original title was probably too general to be informative.
Answer: Unfortunately, this question does not have a good answer. You can choose the best model based on the fact that it minimizes absolute error, squared error, maximizes likelihood, using some criteria that penalizes likelihood (e.g. AIC, BIC) to mention just a few most common choices. The problem is that neither of those criteria will let you choose the objectively best model, but rather the best from which you compared. Another problem is that while optimizing you can always end up in some local maximum/minimum. Yet another problem is that your choice of criteria for model selection is subjective. In many cases you consciously, or semi-consciously, make a decision on what you are interested in and choose the criteria based on this. For example, using BIC rather than AIC leads to more parsimonious models, with less parameters. Usually, for modeling you are interested in more parsimonious models that lead to some general conclusions about the universe, while for predicting it doesn't have to be so and sometimes more complicated model can have better predictive power (but does not have to and often it does not). In yet other cases, sometimes more complicated models are preferred for practical reasons, for example while estimating Bayesian model with MCMC, model with hierarchical hyperpriors can behave better in simulation than the simpler one. On the other hand, generally we are afraid of overfitting and the simpler model has the lower risk of overfitting, so it is a safer choice. Nice example for this is a automatic stepwise model selection that is generally not recommended because it easily leads to overfitted and biased estimates. There is also a philosophical argument, Occam's razor, that the simplest model is the preferred one. Notice also, that we are discussing here comparing different models, while in real life situations it also can be so that using different statistical tools can lead to different results - so there is an additional layer of choosing the method!
All this leads to sad, but entertaining, fact that we can never be sure. We start with uncertainty, use methods to deal with it and we end up with uncertanity. This may be paradoxical, but recall that we use statistics because we believe that world is uncertain and probabilistic (otherwise we would choose a career of prophets), so how could we possibly end up with different conclusions? There is no objective stopping rule, there are multiple possible models, all of them are wrong (sorry for the cliché!) because they try to simplify the complicated (constantly changing and probabilistic) reality. We find some of them more useful than others for our purposes and sometimes we do find different models useful for different purposes. You can go to the very bottom to notice that in many cases we make models of unknown $\theta$'s, that in most cases can never be known, or even do not exist (does a population has any $\mu$ for age?). Most models do not even try to describe the reality but rather provide abstractions and generalizations, so they cannot be "right", or "correct".
You can go even deeper and find out that there is no such a thing as "probability" in the reality - it is just some approximation of uncertainty around us and there are also alternative ways of approximating it like e.g. fuzzy logic (see Kosko, 1993 for discussion). Even the very basic tools and theorems that our methods are grounded on are approximations and are not the only ones that are possible. We simply cannot be certain in such a setup.
The stopping rule that you are looking for is always problem-specific and subjective, i.e. based on so called professional judgment. By the way, there are lots of research examples that have shown that professionals are often not better and sometimes even worse in their judgment than laypeople (e.g. revived in papers and books by Daniel Kahneman), while being more prone to overconfidence (this is actually an argument on why we should not try to be "sure" about our models).

Kosko, B. (1993). Fuzzy thinking: the new science of fuzzy logic. New York: Hyperion.
Comment: If there was "true DGP" and we were able to measure it, we wouldn't need statistics, we would measure it directly. It does not exist, so we create approximations of reality. DGP is abstract and not true and in many cases has not much to do with how the data was *really* generated. I want to point that I'm not saying that stats are subjective and useless, but rather that it is an approximation and there is no such things as "true DGP" or "true $\mu$" etc.
Comment: The claim is true when its assumptions are satisfied (e.g. we are given a fixed sample, which is true in practice). Taken out of context and with violations of assumptions, it can of course be made false.
Comment: @CagdasOzgenc is someone has a methodology to create a model that perfectly reflects reality, than there is no need to for stopping rule, or for measuring the model error - the model is perfect by definition. If you know *the* rules for building such model, there is no need to measure divergence of your model from true DGP, since knowing the true DGP just utilize such knowledge. On the other hand, if your model is a simplification based on the data you have, then the general stats rules apply, as described in my answer.
Comment: @CagdasOzgenc still, if you *know* the "truth", than the stopping rule is simple: stop when your model fits the "truth". If you do not know what is the truth, than "all models are [equally] wrong..." and you have to use statistics. If you don't know, you cannot measure divergence from it.
Comment: @Luca It means *very* much, but it is abstract.
Comment: @Luca We don't quite agree with Tim on this matter. I agree that $\mu$ is abstract, but I think that it is abstract because mean is a concept invented by humans. So it is natural for it to be abstract. On the other hand if you start sampling people from a country, that is a true data generating process. Does this process have a representative distribution? Certainly. Is this distribution necessarily a parametric one with a $\mu$, most likely no. The true distributions will be complex and not comprehensible to humans but easy to AI. We make it easy for ourselves and use simplified versions.
Comment: @CagdasOzgenc people are not generated by any process... People procreate, born and die at some point. No distribution describes *all* the individual births and deaths as they happen, migrations within the population and all the other circumstances (and I do not even consider the imperfections of data in here). Distribution can only approximate to some degree what is happening. I never heard of any real-life AI that is possible to do things like this. AI are possible to do marvelous things but you overrate their power.
Comment: I am awarding the bounty for the time you spent on the issue, but I will not mark it as accepted answer. The bounty also requested authoritative references which are not provided for the main question. And nature is a process, so yes it is a DGP. The fact that it is too complex for human brain is irrelevant.
Comment: I would not contrast *modelling* to *predicting* -- because some kind of modelling is usually done regardless of whether the ultimate goal is explanation, prediction or description. I would rather contrast *explanatory modelling* to *predictive modelling* (to *descriptive modelling*).
Comment: @RichardHardy my intention was not to contrast them, it was just an example because sometimes when you want only to predict the actual model may be of less importance to you as far as it predicts properly.
Comment: The grammatical structure of the sentence indicated contrast; you use "..., while...". I just wanted to point out that modelling is part of predicting, so the distinction does not make sense (to me).
Comment: @RichardHardy Take multiple classifiers, use them to predict, than take the average of the predictions as the final prediction - this procedure does not yield any "model", while can produce accurate predictions (often more accurate that predictions by single classifiers).
Comment: OK, I was perhaps inaccurate. When you say *for modeling you are interested in more parsimonious models*, this does not make sense to me, because here *modelling* does not exclude predictive modelling. I simply think you use a too general term in that sentence.
Comment: I get your point, but let's agree to disagree :)
Comment: @CagdasOzgenc you simply cannot make a "true" model of any non-trivial data since it would have to be infinitely complicated. Every model is a simplification so it diverges from how the reality "really" behaves.
Comment: Let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/31099/discussion-between-richard-hardy-and-tim).
Comment: Stupid question perhaps but can you elaborate on what you mean by "does a population has any $\mu$ for age"?
Comment: @Luca think of any population, say a country, does a country has any "common" age? Is the average age a feature of population as a whole? It describes population, help us to make inferences etc., but age is a feature of individual person not a population.
Comment: Thanks for the reply and the clarification. I think I see what you mean now... You are saying that this parameter is sort of a convenient abstraction for summarizing but may not mean much?
Answer: There is a whole field called nonparametric statistics that avoids the use of strong models. However, your concern about fitting models, per se, is valid. Unfortunately there is no mechanical procedure to fitting models that would be universally accepted as "optimal". For example, if you want to define the model that maximizes the likelihood of your data, then you will be led to the empirical distribution function.
However, we usually have some background assumptions and constraints, such as continuous with finite first and second moments. For cases like these, one approach is to choose a measure like Shannon Differential Entropy and maximize it over the space of continuous distributions that satisfy your boundary constraints.
What I'd like to point out is that if you don't just want to default to the ECDF, then you'll need to add assumptions, beyond the data, to get there, and that requires subject matter expertise, and, yes, the dreaded.....professional judgement
So, is there a guaranteed stopping point to modeling...the answer is no. Is there a good enough place to stop? Generally, yes, but that point will depend on more than just the data and some statistical desiderata, you're usually going to take into account the risks of different errors, the technical limitations to implementing the models, and the robustness of its estimates, etc.
As @Luca pointed out, you can always average over a class of models, but, as you rightly pointed out, that will just push the question up to the next level of hyperparameters. Unfortunately, we seem to live within an infinitely layered onion...in both directions!

When to stop refining a model?


Difference between ramification point and branch point

Question: I'm trying to distinguish between the two. According to wiki:

Let $Ω$ be a connected open set in the complex plane $\mathbb C$ and $ƒ:Ω → \mathbb C$ a holomorphic function. If $ƒ$ is not constant, then the set of the critical points of $ƒ$, that is, the zeros of the derivative $ƒ'(z)$, has no limit point in $Ω$. So each critical point $z_0$ of $ƒ$ lies at the center of a disc $B(z_0,r)$ containing no other critical point of $ƒ$ in its closure.
Let $γ $ be the boundary of $B(z_0,r)$, taken with its positive orientation. The winding number of $ƒ(γ)$ with respect to the point $ƒ(z_0)$ is a positive integer called the ramification index of $z_0$. If the ramification index is greater than 1, then $z_0$ is called a ramification point of ƒ, and the corresponding critical value $ƒ(z_0)$ is called an (algebraic) branch point. Equivalently, $z_0$ is a ramification point if there exists a holomorphic function $φ$ defined in a neighborhood of $z_0$ such that $ƒ(z) = φ(z)(z − z_0)^k$ for some positive integer $k > 1$.

What confuses me are the following:
1) the usage in complex analysis where I think we do not distinguish between the ramification and the branch point.
2) In the wiki definition, what happens if $f(z_0)$ is not defined? Do we still say that $f$ has a ramification point at $z_0$ even though the branch point $f(z_0)$ is undefined?
And another question:
3) Is there an easy computational tool to determine the ramification points? I feel like it has to be related to $f$ having a multiple root at a ramification point, but I don't think that's precise enough.
Comment: If $f(z) = f(a)+ C(z-a)^n+O(z-a)^{n+1}, C \ne 0$ is holomorphic then $f^{-1}$ has a branch point at $f(a)$ and $m =n$ is the least integer such that $f^{-1}(f(a)+s^m)$ is holomorphic around $s=0$
Answer: In GTM081, Lectures on Riemann Surfaces by Otto Forster, p21, they refer to the same thing.

4.3 Definition  Soppose $X$ and $Y$ are Riemann surfaces and $p:Y \to X$ is a non-constant holomorphic map. A point $y\in Y$ is called a branch point or ramification point of $p$, if there is no neighborhood $V$ of $y$ s.t. $p|_V$ is injective.


What are good references for the different methods of combining p-values?

Question: I need a good reference for the methods and/or the difficulties that arise when attempting such a combination.
I've found Loughin, TM, A systematic comparison of methods for combining p-values from independent tests, Computational Statistics & Data Analysis (2004) 47(3):467–485, for example. Is there a better/more precise one?
Comment: I guess I don't know why you are interested in the article. They talk in the abstract about combining p-values for a combined hypothesis. If you are combining independent data set to reach strong conclusion because of the increased sample size. That is all that us meant by a meta analysis. One way to do meta analysis (also called data synthesis) is to combined p-values from independent tests of the individual data sets. Fisher's combination test is one of many that could be tried. Apparently the paper compares 6 such methods.  Meta analysis is in the list of key words in the abstract.
Comment: It sounds like you want to do a meta analysis.  Fisher's combination test is one method for combining p-values.  There are many books on meta analysis that could be useful to you. I will give you an answer with links to several books if you tell me I am on the right track.
Comment: Is this for multiple test correction or something else?
Comment: It's pretty long to explain but basically I have 3 `p-values` obtained through the same process (the `kde.test` function of R's `ks` package http://cran.r-project.org/web/packages/ks/ks.pdf) but applied to three different sets of 2 columns each (all 6 columns are linked because they belong to the same 'observation')
Comment: Well I'm not a statistician so I have no idea what a meta analysis is :) (sorry) I actually do not want to perform such a combination, I just need references for the various methods of doing so and/or its difficulties (like the article I presented, to which I have no access sadly)
Comment: I wouldn't be surprised to find out that Fisher's test is one that they compare.
Comment: @MichaelChernick I mentioned the article because it mentions several methods of combining p-values. As I said, I'm not really interested in performing such a combination, rather I'm looking for sources to reference the various methods for, and/or complications of, doing so.
Answer: One useful entry into this literature is an annotated bibliography by R D Cousins which considers most of the well-known methods for combining $p$-values
and which is freely available from arXiv.
Comment: (+1) What is the benefit of using BibTeX on this forum? Presumably that one can copy-paste it into one's .bib file. How often do you think this is happening? I would guess that many many more people would find a hyper-link more useful; my own preference is to put a hyperlink on a short citation, e.g. `Cousins, 2008, Annotated bibliography of some papers on combining significances or $p$-values` or something like that. Perhaps a BibTeX citation can be provided extra in the end of an answer. I am saying this because this is not the first time you are using BibTeX and I am always wondering...
Comment: I would suggest to follow the advice of @amoeba and not embed BibTex for citations here. Assuming you actually *do* want a BibTeX citation to be available, you could cite via a [Google Scholar hyperlink](https://scholar.google.com/scholar?cluster=12388032574443842317&hl=en&as_sdt=0,44). If you click that link, then click Cite, then BibTex, you can get the same result.
Comment: @amoeba (and other commenters) apologies, I had no idea I was going to perturb so many people. I have removed the formatting.
Comment: Thank you for the article mdewey. I edited your answer to add a link since I also wondered (as @amoeba did) why you left the bibtex instead (or without) the link.


C++ How to handle a change of pointers and references in a function?

Question: <code>void parameterMystery3(int* a, int& b, int* c) {
    *a += *c;
    (*c)++;
    b--;
    cout << b << " " << *a << " " << *c << end1;
}

int main() {
    int w = -4;
    int x = 10;
    int y = 9;
    int z = 13;

    parameterMystery3(&x, x, &w);
}
</code>
If argument 2 (at the end of main()) passes x as a normal variable (not a pointer or a reference), does it only reference x in parameterMystery3? Do the &s at the end of main matter if they get converted to pointers with int* a and int* b or am I thinking about it incorrectly?
Comment: I'm sure that the expected behavior of this gets explicitly specified somewhere. But, personally, I prefer the "Doctor, it hurts when I move my arm this way. -- well, don't move your arm that way, then" approach. Life's too short...
Comment: `end1` should be `endl`
Answer: I personally find c++ reference notation a bit confusing.
The & in a parameter declaration means the argument is passed by reference. You pass x as argument. If you didn't declare the parameter as reference, a copy of x value would be passed. As you declared it as reference, a reference to x is passed. This means that, if you change b value, x value is changed.
Now, if you declare a parameter with a pointer type, a pointer of the declared type must be provided as argument. No conversion done. Passing an int variable as argument is a compile-time error. You must pass either an int* variable or a pointer to the variable.
You get a pointer to the variable with &.
The confusing thing is that this last & doesn't mean the same as the & in parameter b. While int &b means b is a reference to an int (the argument is passed by reference) the & before the variables you provide as arguments mean "a pointer to" the variable.
Therefore, if y is an int variable, &y is an int * pointing to y.
Comment: I didn't realize that the pointers and references also change the thing it references/points to originally. This makes sense now. Thank you!
Comment: Well, if you modify a reference parameter, the value referenced by it is modified.
Comment: Sorry sent by mistake. If the parameter is a pointer type, then a pointer is passed by value.
Comment: If the parameter is int *a, and you have a variable x of type int *, a++ changes the place where a points to, but not the place where x points to. If you do (*a)++, then you are actually incrementing the value both a and x point to. Believe me, when you master C pointers, the secrets of the Universe will have no mistery for you ;)


How does Dart/Flutter execute code concurrently even if it's single threaded?

Question: I'm new to Dart/Flutter and have some Node.js background, I understand that Dart has some similarities with Node.js like event loop and single threaded. But I have a hard time understanding how Dart handles async for blocking I/O operation, especially internal I/O like file system and database operation.
For Node.js, when it handles blocking I/O, let's say read the content of a file. It puts that file system operation in the Thread Pool, Node.js then concurrently (not parallelly) runs the next line of code in the main thread while the file system operation keep running. When it finishes reading the file, it puts a read-file callback event in the Event Queue which will be dequeued by Event Loop and be executed in the main thread. The concept of Node.js is not hard to understand, but how about Dart?
As Dart has no Thread Pool or something similar to Node.js's, it's single threaded and the official site says it runs everything in the main thread, whether it find Future, async, or anything like that, it still runs in the main thread, when it finishes that operation, it then puts the callback in Event Queue waiting for being dequeued by Event Loop and executed in the main thread.
This leads to my confusion that Dart can still run the next line of code while file system operation (blocking I/O) is still running. Please have a look at this piece of code
File myFile = File("some-text-file");
myFile.readAsString().then((content) {
  print(content);
});
print("This is my next line");

output:
This is my next line
Lorem ipsum...

How is that possible by not blocking the I/O operation without utilizing Thread Pool or something similar to Node.js's? If it was Node.js, myFile.readAsString() will be delegated to Thread Pool for concurrency.
Or Dart indeed under the hood simply spawn a new Isolate for that blocking I/O operation?
Thank you in advance for all of your answers.
Answer:
Concurrency is not the same as parallelism.  You don't need multiple threads to be concurrent.

Each Dart isolate is single-threaded, but a Dart program can have multiple isolates.

In most cases, an asynchronous operation ultimately executes some lower-level operation implemented outside of Dart (such as in the Dart VM/runtime or via an FFI binding) where they can spawn OS threads, spawn separate processes, or perhaps perform filesystem operations that performs asynchronous I/O. It depends the implementation of the asynchronous operation.


How to backup postfix?

Question: I am putting together a list of things that I need to backup, on my website.
I am using postfix for handling my mail.
Which files in /etc/postfix (if any) do I need to backup?
My goal is that after I restore from backup files, I should have all my previous mail as well.
So my question can be broken into two parts:

How do I backup postfix (i.e. which files do I take a copy of)?
Once I have a backup, how can I restore so that I have access to my previous mail (is it simply a case of copying the restored files to /etc/postfix)?

My server is running a headless Ubuntu 10.0.4
Answer: Backup everything in /etc/postfix, it is simpler that way.
As for the email. If the email hasn't been touched, then Postfix has no influence on it. It just delivers mail, it doesn't provide access to it. (You need access to the filesystem, or an IMAP, POP3, etc server for that)
If the email has been damaged, then restoring it from backup is trickier.
MTAs, like postfix, don't store the mail. They deliver it somewhere. Where that is is determined by the configuration.
My system, which doesn't use Postfix (although the principles are the same) delivers everything to /home/$userid/Maildir/.
… except that some users have Procmail set up to filter mail, so some of it ends up in different places.
Since I said it depends on the configuration, let's have another example. An older system I used delivered to /var/spool/mail/$userid (a completely different part of the filesystem).

How to backup postfix?


Using indexOf in order to filt an array - ( Next.js app with typescript )

Question: I have this array in my next.js app
arr1
[

  {
    identifier: "60a17722225f2918c445fd19",
    name: "Ben Awad",
    _id: "60c94480b8d43c28d0a6eb73
  },

  {
    identifier: "60a455d11fa62a1510b408f8",
    name: "dev ed"
    _id: "60bf62cede309f1a30fe88ab"
  }

]

And i have this another big array
arr2
[

  {
    name: "Ben Awad",
    _id: "60a17722225f2918c445fd19
  },

  {
    name: "dev ed",
    _id: "60a455d11fa62a1510b408f8"
  },

  {
    name: "Katlyn",
    _id: "60a52500ce96f30c14fdaff9"
  },

  {
    name: "Mike",
    _id: "60c95deeb8d43c28d0a6eb74"
  },

  {
    name: "Kassandra",
    _id: "60c960ddb8d43c28d0a6eb7a"
  }

]

I want a new array who should have all users except for those who have similar ids with arr1
So this is the logic i did (Notice that arr1 and arr2 will change constantly)
Me = arr1
AllUsers = arr2
const LookFriends =
    Me &&
    AllUsers.filter(({ _id }) => {
      return Me.friends.indexOf(_id) === -1;
});

console.log(LookFriends);

The output should be Katlyn, Mike and Kassandra, but the console.log says...
[

  {
    name: "Ben Awad",
    _id: "60a17722225f2918c445fd19
  },

  {
    name: "dev ed",
    _id: "60a455d11fa62a1510b408f8"
  },

  {
    name: "Katlyn",
    _id: "60a52500ce96f30c14fdaff9"
  },

  {
    name: "Mike",
    _id: "60c95deeb8d43c28d0a6eb74"
  },

  {
    name: "Kassandra",
    _id: "60c960ddb8d43c28d0a6eb7a"
  }

]

I'm really having a hard time trying to filter an array based on another array, what can i do ?
Answer: You need to use 'findIndex' in this case, then compare the _id field itself:
const LookFriends =
    Me &&
    AllUsers.filter(({ _id }) => {
      return Me.friends.findIndex(friend => friend._id === _id) === -1;
});

If you use "indexOf", it will compare the entire object to just that _id value.
Comment: It worked beatifully, i was mad and tired of trying to find a way out so i didn't tried this lmao !
Answer: You can extract the identifiers from the first array and filter the second array if the element doesn't match any id in our object or our Set
// with an object
const ids = {}

arr1.forEach(user => {
  ids[user.identifier] = true
})

const filtered = arr2.filter(user => !ids[user._id])

// or with a Set
const ids = new Set(arr1.map(user => user.identifier))

const filtered = arr2.filter(user => !ids.has(user._id))

Comment: I already found a solution, but thanks for your answer man !
Answer: Please have a try with this code.
    Me &&
    AllUsers.filter(({ _id }) => {
      let bExist = false
      Me.friends.map( (friend) => {
        if ( friend._id === _id )
          bExist = true
      })
      return bExist
    });

Let me know if it works or not.
Comment: I would suggest using "reduce" instead of map. You can declare "bExist = friends.reduce", then pass it that way, so it is more declarative.
Comment: Somebody gave me the solution already, but, i tried this and it worked, thanks man !


Question about $\frac{1}{\pi} \int_{-\pi}^{\pi} te^{int} dt$ to $\frac{-2i(-1)^{n}}{n}$.

Question: How do you get
$\frac{1}{\pi} \int_{-\pi}^{\pi} te^{int} dt$
to $\frac{-2i(-1)^{n}}{n}$.
The furthest i come to is:
$\frac{1}{\pi}((\frac{\pi e^{in\pi}+\pi e^{-in\pi}}{in}) -\frac{1}{in}(\frac{e^{in\pi}-e^{-in\pi})}{in}))$
Comment: $e^{in\pi} =(-1)^{n}$ for any integer $n$.
Answer: $$I=\frac{1}{\pi}\int_{-\pi}^{\pi}t e^{int} dt$$
Integration by parts leads to
$$I=\frac{1}{\pi}\left (t \frac{e^{int}}{in}+\frac{e^{int}}{n^2}\right)_{-\pi}^{\pi}$$
$$I=\frac{\pi}{in\pi}(e^{in\pi}+e^{-in\pi})+\frac{1}{n^2 \pi} (e^{in\pi}-e^{-in\pi})$$
$$I=\frac{2\cos(n\pi)}{in}+\frac{2i\sin n\pi}{n^2\pi}=\frac{-2i(-1)^n}{n}.$$
As $\cos n\pi=(-1)^n, \sin n\pi =0$


wildcard name-based provisioning in haproxy.cfg with LE

Question: Trying to redirect certain domain names to specific backends using catchall wildcard entry[A, A1] in haproxy.cfg (HAproxy v1.8.12 Host OS:Alpine Linux v3.9 version details [B] posted at the end in case it has to do with different versions behaving differently[C]):
frontend http
        bind *:80
        reqadd X-Forwarded-Proto:\ http
        redirect scheme https code 301 if !{ ssl_fc }
        ### specific domains ###
        acl host_specificdomain hdr(host) -m reg -i  ^[^\.]+\.specific.domain$
        #Redirect all other domains with all extensions and all ports to:
        acl host_catchall hdr(host) -m reg -i ^[^\.]+\.^[^\.].^[^\.](:[0-9]+)?$
        ### clusters ###
        use_backend specificdomain_cluster if host_specificdomain
        use_backend catchall_cluster if host_catchall

frontend https
        bind       *:443 ssl no-sslv3 crt-list /etc/haproxy/ssl/ssl.lst
        rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains;\ preload
        rspadd X-Frame-Options:\ DENY
        reqadd X-Forwarded-Proto:\ https
        acl app_letsencrypt  path_beg   /.well-known/acme-challenge/
        use_backend bk-letsencrypt if app_letsencrypt

backend bk-letsencrypt
    log global
    mode http
    server srv_letsencrypt 127.0.0.1:63443

backend catchall_cluster
        redirect scheme https if !{ ssl_fc }
        mode http
        balance roundrobin
        option forwardfor
        server server1 192.168.200.101:80 check
        server server2 192.168.200.102:80 check

backend specificdomain_cluster
        redirect scheme https if !{ ssl_fc }
        mode http
        balance roundrobin
        option forwardfor
        server server1 192.168.200:103:80 check
        server server2 192.168.200.104:80 check

haproxy stats shows all servers alive, fyi.
The above posed three issues:

The catchall domains option with wildcards (acl host_catchall hdr(host) -m reg -i ^[^.]+.^[^.].^^.?$) did not redirect all but not specific.domain (acl host_specificdomain hdr(host) -m reg -i  ^[^.]+.specific.domain$).
Even the specific.domain gives 503 error "service unavailable, no server is available to handle this request" when:

$ curl -v --resolve specific.domain:443:127.0.0.1 https://specific.domain
* Expire in 0 ms for 6 (transfer 0x55c2bc8bc4c0)
* Added specific.domain:443:127.0.0.1 to DNS cache
* Hostname specific.domain was found in DNS cache
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55c2bc8bc4c0)
* Connected to specific.domain (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt   CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=specific.domain
*  start date: Jun 17 08:00:34 2019 GMT
*  expire date: Sep 15 08:00:34 2019 GMT
*  subjectAltName: host "specific.domain" matched cert's "specific.domain"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: specific.domain
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* HTTP 1.0, assume close after body < HTTP/1.0 503 Service Unavailable < Cache-Control: no-cache < Connection: close < Content-Type: text/html <
503 Service Unavailable
 No server is available to handle this request.
* TLSv1.3 (IN), TLS alert, close notify (256):
* Closing connection 0

The same domain was readily available over http as follows:
$ curl -v --resolve specific.domain:80:127.0.0.1 http://specific.domain
* Expire in 0 ms for 6 (transfer 0x55cecf76c4c0)
* Added specific.domain:80:127.0.0.1 to DNS cache
* Hostname specific.domain was found in DNS cache
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55cecf76c4c0)
* Connected to specific.domain (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: specific.domain
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Sat, 22 Jun 2019 18:58:31 GMT
< Content-Type: text/html
< Content-Length: 17832
< Last-Modified: Wed, 19 Jun 2019 09:19:57 GMT
< Connection: close
< Vary: Accept-Encoding
< ETag: "5d09fe3d-45a8"
< Accept-Ranges: bytes
<
<!doctype html>
...

Haproxy fails to pull additional domain certs which were pointed by DNS to the specific IP of the machine:

Failed authorization procedure. www.specific.domain (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.specific.domain/.well-known/acme-challenge/APRETTYLONGID [PUBLIC IP]: "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n   \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<ht"

Btw, I am using a script [D] to generate LE certs, fyi.
Any pointers appreciated,
Thanks and cheers,
/z
References:
A. https://serverfault.com/a/909483/528889
A1. https://unix.stackexchange.com/questions/166169/how-to-configure-haproxy-to-redirect-multiple-domain
B. HAproxy version details:
 haproxy -vvvvv
HA-Proxy version 1.8.12-8a200c7 2018/06/27
Copyright 2000-2018 Willy Tarreau <firstname.lastname@example.com>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -Os -fomit-frame-pointer
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1a  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.1b  26 Feb 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace

C. https://superuser.com/questions/1195012/haproxy-1-7-2-refuses-to-match-host-in-acl
D. https://github.com/tisc0/letsHAP
Answer: First of all, your https frontend lack of backend.
Also, my advice is to use a default backend in your frontends, rather than a complicated regex to catch "all the others domains".
And about acls with regex on host header, I've been more successful with hdr_reg(host) than hdr(host) -m reg -i.
I would propose you another piece of advice:
- make things less complex
- comment unnecessary options to facilitate troubleshooting
Ah : I'm not sure your regex lines are read in http frontend, since you used the redirect scheme https before : I would say all your request go to https, which has no backend.
Hope it helps.
Cheers,
Comment: Great, so check it as the right answer ;)
Comment: You whacked the mole precisely. Thanks for your invaluable input.
Answer: A backend to the frontend https is appended to the earlier config as of below, yet all traffic is sent to host_catchall server!? Any clues? Thanks!

frontend http
        bind *:80
        reqadd X-Forwarded-Proto:\ http
        redirect scheme https code 301 if !{ ssl_fc }
        ### specific domains ###
        acl host_specificdomain hdr(host) -i subdomain.specific.domain -
        #Redirect all other domains with all extensions and all ports to:
        acl host_catchall hdr(host) -i subdomain.catchall.domain
        ### clusters ###
        use_backend specificdomain_cluster if host_specificdomain
        use_backend catchall_cluster if host_catchall

frontend https
        bind       *:443 ssl no-sslv3 crt-list /etc/haproxy/ssl/ssl.lst
        rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains;\ preload
        rspadd X-Frame-Options:\ DENY
        reqadd X-Forwarded-Proto:\ https
        acl app_letsencrypt  path_beg   /.well-known/acme-challenge/
        use_backend bk-letsencrypt if app_letsencrypt

        ### specific domains ###
        acl host_specificdomain hdr(host) -i subdomain.specific.domain$
        #Redirect all other domains with all extensions and all ports to:
        acl host_catchall hdr(host) -i subdomain.catchall.domain
        ### clusters ###
        use_backend specificdomain_cluster if host_specificdomain
        use_backend catchall_cluster if host_catchall