Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

Question: I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources.

http://www2007.org/papers/paper570.pdf
https://chrisjmccormick.wordpress.com/2015/06/12/minhash-tutorial-with-python-code/

The data that I am working with consists of interactions between users and items. There are 2.2M distinct users and 440M distinct items. Across all of the data, there are only 905M records, so it is very sparse.
In my approach, I am calculating H minimum hash values for each user by reordering the items (of which there are 440M). Users have a wide range of item interactions. The user with the most interactions as 2.5M interactions, the lowest is 1 interaction, the average is 403, and the median is only 26.
In google's doc about Google News, they recommend concatenating 2-4 keys (LSH) and doing this 10-20 times. I imagine this works well when a user has interacted with a smaller amount of items like news articles, but it is woefully low for what I am doing. When I test this number of keys for users that have 1,000+ interactions, many do not have any concatenated min has matches with another user. This is a problem because I can manually calculate cosine or jaccard similarity for some of these users and see an acceptable amount of similarity for my needs. I have found better results by not concatenating hash keys and using as many as 200.
For most of my hash key groups, there are around 2M distinct hash keys for the 2.24M users. As such, there is a fairly low amount of collision.
Do you guys have any tips for increasing the amount of clustering? I am considering using 1,000 hash keys and pairing users if they match on more than one. Thanks in advance.
Answer: MinHash is probably overkill here.
You have about 2 entries per item. The minhash approach is meant for recognizing near-duplicate web pages, and most words occur on millions of web pages - a very different scenario.
On your data, I would simply use inverted lists.
Comment: Do you have any links I can read about inverted lists? My main reason for going with minhash was because google is using it to cluster users who read news, and there are several similarities between what they are doing and what I am trying to do.
Comment: It's nothing but storing the 1s in every column only. The basic technique that is driving Lucene.
Comment: But I would still need to perform a similarity comparison across all of the items, right?
Comment: You would iterate over all the items, yes. Get a list of all matches each, and join them. For further speed up, process the items with most weight first.


Am-Dm-Am-E songs (usually powerful folk songs) what key could they be in? What diatonic blues harp or penny whistle to use?

Question: Examples include some (best ^_^) interpretations of Katyusha, La Llorona, Bella Ciao, Cariñito etc. They often go into short C-major type sequences during emphasis so everything points to an Am scale.
Examples:

Katyusha: https://tabs.ultimate-guitar.com/m/misc_traditional/katyusha_ver3_crd.htm
La Llorona: https://tabs.ultimate-guitar.com/j/joan_baez/la_llorona_crd.htm
Bella Ciao: https://tabs.ultimate-guitar.com/k/kud_idijoti/o_bella_ciao_crd.htm
Cariñito: https://tabs.ultimate-guitar.com/tab/1532168
aand many more

Yet when I try to play them with my C diatonic harmonica, it just doesn't work, also not much more luck with G & A.
What scale are those songs in? Which scale diatonic harmonica to use? (I'm assuming only holes 2-3 are bendable on the cheap stuff I'm buying ^_^) How about with an irish tin penny whistle?
Comment: @ToddWilcox - dur = major. Opposite to molle = minor. Hard/soft, German.
Comment: I would think this should be 2 questions, there is absolutely no way a 1-size-fits all approach for diatonic instruments.  What is good on a D harp won't be on a D whistle and vice versa
Comment: I've heard about harmonicas in _harmonic minor_, which would be the obvious choice here. But I know close to nothing about harp, so...
Comment: Can you translate "C-dur" into English? I can't remember the translation myself.
Comment: @Some_Guy that's a good point to make. I have been playing with/around tin whistles and harmonicas, but with tin whistle I am a total noobie and with harmonicas I have some years of experience + a bit in music theory, so I was hoping to use as much as I know from the harmonicas and apply to tin whistle playing. It really helps to know when things don't work because they just don't work and where they don't work because of skill.
Answer: What key?
A minor, see Richard's answer (and my comment)
Which Harmonica to use
Harmonica is a diatonic instrument designed to play simple major folk tunes, however that's often not the way it's used in the modern day.
"minor harmonicas" do have some interesting applications, but they are a read herring to this question, for these applications what you want is a standard, run of the mill 10 hole diatonic harmonica.
For harmonica, a great way to play in minor keys is using "third position".
This would mean using a G harp to play in A minor.
disclaimer: the following bit is a bit of a tangent
As a general statement: anyone trying to "guess" how to play harmonicas in various keys based on "general music theory" is going to come up with some wrong answers, because it relies not just on a general knowledge of keys and chords etc. but on specific knowledge of the note layout of the harmonica, and knowledge of playing technique. For example, "theoretically speaking", you could never "guess" that a C harp is great to play in G and absolutely useless in F without knowing how harmonicas work, and are laid out.)
As an example of this, someone theoretically literate might guess that to play a major harmonica in a minor key, you want to play the relative minor. (So playing a C harp in A minor).  However, while that would be a perfectly reasonable thing to guess, harmonicas don't play very well in their relative minor for a couple of reasons:

there is no A minor chord.  Best you get is the top of an Am7 (i.e. a C chord)
the note "A" is not present in the bottom octave straight and is the most difficult bent note to get (bending the 3 down a whole step exactly, but not too far to an A flat).  Even very advanced harp players find it difficult to play it in tune.
Only one G# on the harmonica, in the middle octave (ignoring impossible for 99% of players notes)

Back to the point
Third positions is especially good on low harmonicas, because it's best in the middle and top octaves of the harmonica.  Because of this, A minor sounds FANTASTIC on a G harmonica, seriously (one of personal favourites to play).
As a convention, we talk about all harmonicas as if they're a C harmonica, so that the same instructions can apply to any harmonica.  So, I'm now going to talk as if you are playing a C harmonica in D minor (but if you were playing a G, it would come out in A minor).
Third position is great on low harps because holes 3 through 10 are D F A B D F A so you have 2 minor chords, and major 6th if you like (a GREAT colour tone for most latin and folk songs).
You also have a pentatonic scale requiring only 1 bend for the whole harp (add in the second and major sixth for dorian).
Only real disadvantage to 3rd position is you have to overblow to get a B flat in the middle octave.  But if you can't overblow (and most can't), you don't really miss 1 measly Bb all that much.  And hey, you're playing a diatonic instrument not even designed to play this music, you can't complain!
Pennywhistle
D and C whistles are both good in Am.  If you're gonna get 1 get a D because it's generally the most versatile whistle, cause it's pretty much good for D G A, Em Am and Bm.  So folk tunes, in a nutshell ;)
Tips for D whistle in A minor
XXO XXO gives you G# in lower octave, learn to halfhole XXX HOO for the higher octave or if XXO XXO isn't so good for your whistle.
OXX OOO gives you C, again only in the bottom octave. If it's sharp, then play 0XX X00, or again, half hole it as HOO OOO.  On some whistles 0X0 000 is in tune for a C in the top octave, but not all.  Generally a good rule of thumb is to half hole for slow passages where you want a pretty and in tune note, and to cross-finger if you're going to play something quick where the facility is more important than a note that's bang on.
Comment: good answer on the whistle, just need to add that F natural on a D whistle is XXX XH0 or as a fast passing note you can get away with XXX XOX. Neither is spot-on in tune without a bit of care and practice but will do unless the Fs are pivotal in the tune (in which case consider getting a C whistle)
Comment: @SteveMansfield a welcome and useful addition, thanks!
Answer: I can't speak for what blues harp or penny whistle to use, but I can address the question of key:
Pieces with progressions like Am--Dm--Am--E are unequivocally in the key of A minor. If you've ever heard the joke that "there are only three chords" in popular music, these are the exact three chords you're playing! In music theory parlance, we call this a i--iv--i--V progression.
(And note that having "only three chords" doesn't mean it's bad music!)
Comment: The key of A minor, not the scale of A minor.
Comment: @bbozo most minor songs have a mix of notes played, depending on the context.  So in a minor you usually see A B C D E F/F# G/G# A . You can view each combination as a separate "scale" (natural minor F&G, dorian F#&G, harmonic minor F&G#, melodic minor F#&G#) or just learn the fact you can choose depending on the musical context.  In the examples you gave, you're definitely going to have that G# some of the time, because there is an E chord not an E minor chord.  So most likely you would use a mix of harmonic and melodic minors if you want to stick to a "scale".
Comment: You know maybe which A minor? :) Iirc the natural A minor has all same notes like C major?
Answer: Playing on a blues harp necessitates using a harp that is a fourth above the  key everyone else is playing in. So, in Am, the sequence quoted, a Dm will be best. Not a common beast, though. You could try an F harp, which will play nearly all the same notes, although the draw/blow may vary. A D harp will give , again, nearly all you need.And to play ordinarily, since Am is the relative of C, the C harp will, again, give nearly all you need. Using two different harps in a song isn't unusual for good players.
Comment: This is often called playing "cross harp", except you don't want a "D minor" harp, just a harp in D.
Comment: @ToddWilcox - trouble with using a D cross harp is that the draw will include C#. It is possible to flatten that note to a degree, but when drawing two or three holes, I think it may come out out of tune. It's the old thing of it's o.k. to play minor notes over a major chord, but not vice versa.
Comment: I usually have C & G with me (I like to be able to choose between blow & draw for same notes ^_^), I'll start hauling an A too and use A+C a bit instead.
Answer: If you're playing in second position, ordinarily you should be using a D harp for the key of Am.
Comment: hm, indeed, thank you. I tried it with an A I have, and it doesn't +sound+ natural what I'm trying to do, but maybe it's just me, I'll practice for a while with that in more detail and see where it takes me
Comment: Somehow, can't see an A harp working too well. There are three notes that belong in A major that will clash against A minor - unless it's a melodic minor harp, which I imagine doesn't exist. A chromatic would be a better bet. Although since there's an E major or E7 in there, drawing on an A harp will produce E9.
Comment: @Tim - I have to agree. I rarely played first position (haven't played blues harp at all in about 20 years... I actually mess around with chromatic these days - but not blues)  and when I did it was for country and folk material that was in major keys. I'm removing that part of the answer.


Check for monotonicity, convexity

Question: I have the following function:
$$ f(x) =\sin(x) - \sqrt{3} * \cos(x) $$ and $$ I = [-\pi , +\pi] $$
I should check this function for monotonicity and convexity. So I drawed the $\sin$ and $\cos$ on a paper to see how the curve goes between $-\pi$ and $+\pi$. but the whole function is a sum between $\sin$ and $\cos$, so how do I do that? My brain stops there. I surely can see where the sin or cos for itself is monotonic but don't know how to add them together, and the $\sqrt3$ gets me confused too.
$$ f'(x) = \cos(x) + \sqrt3 * \cos(x) \\
   f''(x) = -\sin(x) +\sqrt3 * \cos(x) $$
I then made first and 2nd derivative. But how will i go further? I know where the sin(x) is monotonic .. but I don't know where $\sin x + \cos x$ is. I'm really having a bad time combining and imagining these 2 together..
Or is there just a 100% mathematical approach without looking at the graphs?
Comment: you need to determine the sign of $f'$ and $f''$. To do that find when they become zero and then check the sign at other values of $x$. How does increasing decreasing co-relate with sign of $f'$. How does convexity correlate with sign of $f''$.
Comment: hm. so i looked at $f' $ at wolframalpha and i see the changing of the sign at the first root. so should i check for roots of f'? to determine at which x my f' goes from negative to positve which means my original function keeps rising, not falling?

i did that and now i have tan $ x = \frac{1}{\sqrt3} $ which is $ x = arctan..$ right?
Comment: It may be easier to first transform the function to the form $2\sin(x+?)$.
Comment: just by calculating roots of the first and second derivative, I get my points where something "happens". but I dont know if this is the principle or way of doing it
Answer: It seems the following.

This is the graph of the function $f$ and its derivatives. So intervals of monotonicity and convexity of the function $f$ are those of constant signs of the first derivative and the second, respectively. To find them we should to solve the equalities $f'(x)=0$ and $f''(x)=0$. For this purpose we use the standard approach.
$$a\sin x+b\cos x=\sqrt{a^2+b^2}\sin (x+y)= \sqrt{a^2+b^2}(\sin x\cos y+\cos x\sin y).$$
From here we have $\cos y=a/\sqrt{a^2+b^2}$ and  $\sin y=b/\sqrt{a^2+b^2}$. In our particular case, $\cos y=1/2$, $\sin y=-\sqrt 3/2$, so we may put $y=-\pi/3$. So we have $f(x)=2\sin (x-\pi/3)$.
Comment: I do understand that we set our derivative to zero. But I am not sure about the a sin x + b cos x. If I really need that? Because I will not remember all kind of formulas like this in the exam.
Comment: @DominiqueLüber I don't remember these formulas too, but the solution of the equality $a\sin x+b\cos x=0$ should be simple. If $a\not=0$, then we have $\tan x=-b/a$, and the roots of this equation should separate the real line into open intervals of alternating constant signs of the function $a\sin x+b\cos x$.
Comment: i had something like tan x = .. to. but then my approach would be the arctan.. would that be not correct?
Comment: @DominiqueLüber The set of solutions of the equation $\tan x=-b/a$ should be $A=\{\arctan(-b/a)+\pi n:n\in\mathbb Z\}.$ So you should consider those of them which belong to the investigated interval $I$ for the function $f$ (in our case it is $[-\pi,\pi]$).

Check for monotonicity, convexity


Is microphone useful when singing pop for a small audience?

Question: When learning to sing pop music (not classic), I have difficulties in taking the highest notes. It seems that I can do this, but much more silently than I normally talk. Would it help to try amplifying these notes with microphone?
I am speaking about the performance for a small audience, where the sound amplification would not be required for talking.
Answer: Amplification is nice for singing because you can add a little reverb (a mild echo quality that makes you sound like you're a professional, singing in a hall). It's really cool; and the day I got a mic and a small amp with reverb was the day I stopped sounding like someone singing in his bedroom, and started sounding like a singer.
Having noted that, I think if you're having trouble hitting the higher notes with the forceful command singing is supposed to entail, then two things might help: One - practice, practice, practice! ..I had trouble with higher notes at first, but I wouldn't accept being unable so I sang the same thing over and over, making funny faces from straining to get it done, until it became easier. But it did get easier. You're breaking in some muscles that have never been used much, and so have little strength..so hours of practice daily may be needed to build them up.
And the second thing I'd suggest is for all of us there are some songs that are "out of our (comfortable) range"; and we just learn which kinds of songs we should avoid because they don't fit our individual range.
But hey, don't take no till you've tried I say!
And as for the volume of singing, there are some songs and types of music that are more quietly done than others, yes, but in general I find that well felt and done singing is usually about as loud as yelling (or so my neighbor tells me! 0_o )
Being in the same room as a professional singing even without an amp is generally enough to make a shiver run up your spine (a good one) from the sound waves going right into your bones! And singers are like that. Big mouths, big lungs.. Of course not everyone is physically a powerhouse, but when those things are lacking then  BIG SPIRIT can get it done! So don't give up, Do what you love. Just practice a lot, and - as my music teacher always told me - "belt it out! Sing it like you mean it!" ;)
Comment: Excellent answer.  You not only answered the question - but delved deeper into the question that was not actually articulated in writing - but is just as important.


Argument 1 passed to Coinbase\Wallet\Exception\HttpException::exceptionClass() must be an instance of Psr\Http\Message\ResponseInterface, null given

Question: When i run the Coin Base Transactions offline it works, but when i put it online it fails.
Here is my code for the transaction.

    $configuration = Configuration::apiKey($apiKey, $apiSecret);
    $client = Client::create($configuration);

    $account = $client->getPrimaryAccount();

    $address = new Address([
        'name' => 'New Address19'
    ]);

    $add = $client->createAccountAddress($account, $address);
    $addressId = $client->getAccountAddresses($account);
    $addresses = $client->getAccountAddress($account, $addressId->getFirstId());
    echo "Your address is: ".json_encode($addresses->getAddress())."
";

ive googled but all in vain, the answers i get from here of updatind the certificate in the etc folder also not working. Someone help me
Comment: Did you find a solution?
Answer: I had teh same problem. The solution for was to download the cacert.pem file from:
https://curl.se/docs/caextract.html
Copy it to the Coinbase API etc folder (same place where you will find ca-coinbase.crt).
Then edit src\COnfiguration.php to refer to this new file:
    public function __construct(Authentication $authentication)
    {
        $this->authentication = $authentication;
        $this->apiUrl = self::DEFAULT_API_URL;
        $this->apiVersion = self::DEFAULT_API_VERSION;
//      $this->caBundle = __DIR__.'/../etc/ca-coinbase.crt'; // Replaced with:
        $this->caBundle = __DIR__.'/../etc/cacert.pem';
    }

Answer: $configuration = Configuration::apiKey($apiKey, $apiSecret);
$client = Client::create($configuration);
$account = $client->getPrimaryAccount();

$address = new Address([
    'name' => 'New Address19'
]);

$add = $client->createAccountAddress($account, $address);
$addressId = $client->getAccountAddresses($account);
$addresses = $client->getAccountAddress($account, $addressId->getFirstId());
echo "Your address is: ".json_encode($addresses->getAddress())."
";

Comment: this doesnot work, its no different from what i posted. seems you did copy and paste


Awk: how to replace text within block, but print everything else

Question: I am able to replace the string AllowOverride None with AllowOverride All in the block , but I am only able to print the block of text that matches.
How do I print the text before and after the match?
awk '
/<Directory \/var\/www\/>/, /<\/Directory>/ {
  sub(/AllowOverride None/,"AllowOverride All")
  print
}
' /etc/apache2/sites-available/default

Answer: Add a next after your existing print and add this after your existing closing curly brace:
{print}

Answer: You can just perform the substitution without printing in the first clause, and then add a second clause to print every line. Like this:
awk '
/<Directory \/var\/www\/>/, /<\/Directory>/ {
    sub(/AllowOverride None/, "AllowOverride All");
}
{
    print;
}
' /etc/apache2/sites-available/default

I did a quickie test of this, and I believe it will work.


Tratar valores monetários R$ XX,YY M / B / K em pandas?

Question: Estou tratando dados de um dataframe que vem com valores com virgulas e % com virgulas, segue exemplos que já estão funcionando:
    # Substituir porcentagens com vírgula por floats
    # 10,08%, 1,06%
    df = df.replace(r'^(\d{1,2}),(\d{1,2})%$', r'\1.\2', regex=True)
    # 10,08, 1,06
    df = df.replace(r'^(\d{1,2}),(\d{1,2})$', r'\1.\2', regex=True)
    # R$ 10,74, R$ 9,46, R$ 166,00
    df = df.replace(r'^R\$ (\d+),(\d+)$', r'\1.\2', regex=True)
    # R$ 2.002,05
    df = df.replace(r'^R\$ (\d+).(\d+),(\d+)$', r'\1\2.\3', regex=True)

Gostaria de resolver números que contem casas com virgula más também contem letras indicando K, M, B para mil, milhoes, e bilhoes, ex:
R$ 99,27 K
R$ 8,80 M
R$ 117,25 K
...

Ao final cheguei a esse código:
# R$ 274,71 K -> 27471
df = df.replace(r'^R\$ (\d+).(\d+) K$', r'\1\2', regex=True)

Gostaria de adicionar os zeros ao final, ou multiplicar, já tentei adicionar o \000 ou \0 etc ao final do replace mas sem sucesso.
df= df.replace(r'^R\$ (\d+).(\d+) K$', r'\1\2\000', regex=True)

Qualquer ajuda sera bem vinda, obrigado!
Answer: Parece que voce esta tentando remover os caracteres "R$:, espaçaos em branco a letra indicadora (K, M, B) dos valores no formato "R$ X,XX K" e converte-los em numeros inteiros multiplicando-os pelo fator adequado. Para adicionar os zeros no final, voce pode utilizar a funçao str.zfill() apos fazer a substituiçao. Aqui esta um exemplo de como fazer isso:
import pandas as pd

# Exemplo de dataframe com os valores
data = {'LIQUIDEZ_DIARIA': ['R$ 99,27 K', 'R$ 8,80 M', 'R$ 117,25 K']}
df = pd.DataFrame(data)

# Mapeamento dos fatores para cada letra indicadora
fatores = {'K': 1_000, 'M': 1_000_000, 'B': 1_000_000_000}

# Função para converter os valores
def converter_valor(valor):
    valor = valor.replace('R$', '')  # Remover 'R$'
    valor = valor.replace(' ', '')  # Remover espaços em branco
    letra = valor[-1]  # Último caractere é a letra indicadora
    numero = float(valor[:-1].replace(',', '.'))  # Converter para float
    fator = fatores[letra]  # Obter o fator adequado
    valor_inteiro = int(numero * fator)  # Converter para inteiro multiplicando pelo fator
    return str(valor_inteiro).zfill(5)  # Adicionar zeros no final

# Aplicar a função ao dataframe
df['LIQUIDEZ_DIARIA'] = df['LIQUIDEZ_DIARIA'].apply(converter_valor)

print(df['LIQUIDEZ_DIARIA'])

os saida sera:
0      99270
1    8800000
2     117250
Name: LIQUIDEZ_DIARIA, dtype: object

Comment: Ola @Collaxd ! É claro que você poderia, por exemplo, fazer um loop `for column in df.columns: df[column] = df[column].apply(converter_valor)`
Comment: Olá, obrigado! Sim isso funciona para coluna específica, porém eu gostaria de fazer para o DF inteiro, sem especificar a coluna.


O que está errado no código jQuery que escrevi?

Question: Olá, amigos! Sou novato em programação, e gostaria de ajuda para identificar um problema na construção de um código para criar um botão "Vá para o topo da página", com animação, em jQuery.
O que escrevi parece correto, mas depois que ele é acionado uma vez, a barra de rolagem fica agarrada no topo e não sai mais.
Seguem os trechos de HTML, CSS e JQUERY, para análise:
HTML
<button id="bt-top" type="submit" title="Vá para o topo da página │ Vaya a la parte superior de la página │ Aller en haut de la page"> ▲ </button>

CSS
#bt-top {
display: none;
position: fixed;
font-size: 18pt;
bottom: 12px;
right: 12px;
color: #fff;
border: solid 3px #fff;
background-color: #00b7ce;
cursor: pointer;
padding-bottom: 6px;
/* - - - - - - - - - */
border-radius: 12px; /* CSS3 */
    -moz-border-radius: 12px; /* Firefox */
        -webkit-border-radius: 12px; /* Chrome, Safari */
/* - - - - - - - - - */
width: 54px;
z-index: 999; }

#bt-top:hover { background-color: #6b6b6b; }

JQUERY
$(document).ready(function(){

$(window).scroll(function() {

    if ($(window).scrollTop() > 100)
    {

        $("#bt-top").css({"display" : "block"});
    }
    else
    {
        $("#bt-top").css({"display" : "none"});
    }

    $("#bt-top").click(function() {
        $("html, body").animate({scrollTop: $("#principal").offset().top}, 2400);

    }); }); });

Desde já, grato pela ajuda,
Alexandre Soares
Comment: Pode usar `$("#bt-top").show()` para exibir e `$("#bt-top").hide()` para esconder.
Comment: Obrigado! Vou testar essa possibilidade! Valeu!
Answer: Acontece que você está criando um novo evento toda vez que a barra de rolagem é utilizada. Como a quantidade de eventos criados é alta, então o código vai ficar repetindo o mesmo processo (de levar a página até o topo) várias e várias vezes.
Exemplo:

$(document).ready(function() {

  $(window).scroll(function() {

    if ($(window).scrollTop() > 100) {

      $("#bt-top").css({
        "display": "block"
      });
    } else {
      $("#bt-top").css({
        "display": "none"
      });
    }

  });
});

/* É necessário deixar o código abaixo fora do evento `scroll` */
$("#bt-top").click(function() {
  $("html, body").animate({
    scrollTop: $("#principal").offset().top
  }, 2400);
});
div {
  height: 1000px;
}

button {
  position: fixed;
  display: none
}

<button id="bt-top">Topo</button>

Comment: Muito obrigado! Agora está funcionando! Valeu!


How do I allow users to see other users' views based on taxonomy fields

Question: I have a bunch of nodes called 'article' with a taxonomy field attached called 'publication'
I have also added the 'publication' field to the user so he/she can choose which publications they want to see - they select these and then there's a view where they can only see their own selection, based on a contextual filter.
Question is, how can I let users select the option to see other users' chosen publication view. Can this be added to their setting page, along with the actual publications - /user/1/edit?destination=admin/people
Answer: It is possible with views relationship and contextual filters. You can add a relationship that references users and publications and then add user id to filter the publication based on the result
Comment: That really doesn't help me at all I'm afraid. What view do I start with, a user view, term view? Thanks.
Comment: Create a publication view and add reference entity relationship User referencing Content from field_publication. new add user id in contextual filter. this will show all the publication a user has linked to.
Comment: I have already created this view, I want to enable a different user to see a view that another user has created.
Comment: So basically you'd like to have users and their publication, right?


SAT Math probability and repeats

Question: A ball's area is divided into two sections. If each section is to be painted using one of 5 different colors, how many differently painted designs are possible?
I know that the first area has 5 options for coloring. The second area has 4 options for coloring so 5x4 = total possibilities for differently painted designs.
I'm not sure if 5x4 would include repeats? Can you explain why the solution for the problem, 5x4, would not include repeats?
Comment: hard to say without knowing exactly what's meant by section
Comment: Amazingly badly worded for a SAT question, ambiguities.
Answer: That is an incredibly ambiguous problem!  Here are four answers.

Suppose the two sections are of different shapes or sizes.  To be specific, let's say one is one octant ($\frac{1}{8}$ of the ball's surface) and the other is the remaining $\frac{7}{8}$.  Then there are $5$ options for colouring each section, total $5^2=25$.
Suppose that the ball is divided as in the previous case, but we insist that it must be possible to tell where the two sections are by looking at the coloured ball.  Then the sections must be painted in different colours, so the number of options is $5\times4=20$.
Suppose that the ball is divided into two equal hemispheres, and that we can look at the ball from any direction we like.  Then colouring the hemispheres, say, red and blue will be the same as blue and red.  So colours must be chosen with order unimportant, giving $C(5,2)=10$ possibilities; plus $5$ more if the hemispheres have the same colour, total $10+5=15$.
As in 3., but the single colour option is disallowed, so there are only $C(5,2)=10$ options.

You could also specify that the ball is divided into two hemispheres, but that it is not moved and is always viewed from the same direction.  This would be the same as case 1.  Or maybe 2.
So to sum up, I would say the answer is $25,20,15$ or $10$, depending what you mean.
Answer: If you paint the first section with colour 1, then there are 4 colors left to paint the second section with another color. You go on like this, if you paint the first section with the other 4 colors-and always different colors in section 2.
Here the sections are never painted with the same color at the same time.
greetings,
calculus

SAT Math probability and repeats


Question on the Friedman test and statistical significance

Question: I have 322 clothing items that have been ordered by our military from 2013 to 2018.   Each military member is either in a Tier 1 or Tier 2 location.  I ran Friedman tests on each individual item over the six year period and adjusted p values through Benjamini-Hochberg.  My results look like this:

Because the UIC (Tiers 1 and 2) have more items that are significant (with p-values mainly in the order of 10^-16), I have highlighted those cells for further analysis. Someone has questioned why don't I consider MOS and Status as well, which I discount since there are more items that are not significant than are significant.
Am I correct in doing this?
Comment: Could you please say more about the purpose of this study, the questions you are trying to answer, and how you structured the Friedman tests? That information might help you get a  better answer.
Comment: The purpose of the study is to determine an optimized system for allocating points for clothing to each military member.  The current system, a Tiered approach, is based on UIC, which refers to the unit where they work.  The Friedman test looks at each item and how many were ordered in each year in each UIC (there are 2900 UICs).
Comment: I should also point out that the number of p values in the order of 10^ (-15) were zero for both tiers 1 and 2 Status;  45 and 62 for tiers 1 and 2 respectively, MOS; and, 61, 109 for tiers 1 and 2 respectively , UIC.
Comment: It's been a few decades since I had an MOS, so please remind me what the "points for clothing" allocated to each military member are supposed to represent. Also, I want to make sure that I understand how you did the Friedman tests. I understand that you did these tests separately for each item of clothing and separately for Tier 1 and Tier 2 locations. So in the terminology of the [Wikipedia page](https://en.wikipedia.org/wiki/Friedman_test) did you use the 6 separate years as "blocks" to evaluate differences among MOSs, between genders,  etc. as multi-category "treatments," or vice versa?
Comment: It would help future visitors to the site if you would edit your question to include the information you provided in comments, as comments can sometimes get lost while there is a permanent record of all versions of questions and answers. It would also help if you could specify the numbers of categories of each of Status, Rank, MOS, and Environment, in addition to the 2900 UICs, as that information could be important in formulating an answer.
Comment: EdM, I've examined the level of significance for each item of clothing and accept that UIC has far more highly significant items than Status, which have zero, and MOS, which have much less than UIC.  Since this translates into effect size, I would state that the effect is greater for UIC then it is for Status or MOS.
Answer: If I understand your analysis correctly, much of what you are finding thus far might have a lot to do with the number of levels (categories) within each of the Factors that you have analyzed separately. Also, don't forget that each UIC level that ultimately gets adjusted for will affect fewer military members than will each MOS or Status level that gets adjusted for. It might be better to formulate your model differently, depending on the overall goal of this project.*
My understanding of the situation
As I understand it, the analysis was done separately for each of the 2 Tiers of Location and for each of the 322 clothing items. Within each of these 644 combinations of clothing and Tier, a separate non-parametric Friedman test was done for each of the 6 remaining Factors: Gender, Status, Rank, Unit Identification Code (UIC), military occupational specialty (MOS), and Environment.
For each combination of clothing item and Factor within a Tier, the Friedman test combined information among the 6 years of data to determine whether there were any significant differences in terms of orders for that clothing item among the levels of the Factor being examined. The Friedman test provides a p-value for whether any differences among the levels of the Factor are greater than would be expected by chance. As there are hundreds of comparisons being done overall, corrections for multiple comparisons were done to control false-discovery rate (FDR) via the Benjamin-Hochberg (BH) procedure.**
A difference for a clothing item within a Factor and a Tier was considered significant if the FDR was no greater than 5%.
UIC was singled out for further analysis as more than half of the items of clothing showed significant differences on that basis among UIC in both Tiers of location.
Prioritization
In terms of prioritizing further analysis, it might seem to make sense to start with the Factor that shows the most clothing items having significant differences, which is UIC. But your cutoff of 50% of items showing significance within a Factor/Tier combination is somewhat arbitrary. With almost 50% of items showing significant differences as a function of Status and about 40% showing differences as a function of MOS, it would seem to be unwise simply to ignore Status and MOS going forward.
Also, the numbers of military members associated with each level of a Factor will differ greatly among the Factors. With about 2 million active duty and Reserve US military members overall, adjusting for an individual UIC level will affect on average fewer than 1000 individuals, while each adjustment for an MOS will affect many more, and I suspect that an adjustment for a Status level will affect hundreds of thousands. So you might want to consider the number of individuals likely to be affected as you prioritize your further work.
One potential problem I see
There are 2900 levels of the UIC factor, about 25-30 levels of Rank, a few hundred levels of MOS, only 2 levels of gender, and probably only a few levels of Status (if Status means something like active/reserve). With at least 10 times more levels of the UIC Factor than for any other Factor, it might not be so surprising that more items of clothing have a "significant" difference for at least one UIC value (which is what the Friedman test reports) than you find for the other Factors.
Although the Friedman test takes into account the number of levels of the Factor being considered, I suspect that some levels of UIC/MOS/etc are substantial outliers for each clothing item, with the particular UIC/MOS/etc outlier value possibly different depending on the clothing item. For example, I was issued hospital whites to wear while performing my duties under my US Army MOS; I doubt that there is much call for such clothing in an Airborne unit.
So to my mind, finding 204 items of clothing having a "significant" difference with respect to UIC levels in Tier 2 isn't necessarily more important than finding 141 items with a difference based on MOS in Tier 2, when there are about 10 times as many levels of UIC as there are of MOS.
Alternative analysis approaches
Doing this type of analysis separately for each of the Factors and Tiers might not be the best approach. It's typically more powerful to consider many predictors together in a combined analysis; see this page for discussion in the context of regression. As your data are presumably units of clothing, you could consider a set of Poisson multiple regressions,*** modeling the numbers ordered of each clothing item as a function of all Factors together: Tier, Gender, Status, Rank, UIC, MOS, and Environment. If the model is adequate, you can test whether any of those Factors is significantly related to orders of the clothing item, and do further tests of which of the 2900 levels of UIC,  100-200 values of MOS, etc., are most responsible for those differences, potentially speeding up further analysis and evaluation.
You also might consider looking for clusters of clothing items whose orders tend to track together. Reducing the analysis from 322 individual items to a few dozen clusters of related items might simplify and better inform your analysis. Similarly, instead of examining all 2900 UICs individually you could consider grouping them according to shared characteristics based on the Table of Organization and Equipment. After the higher levels of organization are taken into account, specifics of particular UIC that don't fit well into the model could be evaluated.

*It's still not clear how this analysis will inform the "system for allocating points for clothing to each military member." Editing the question to make this clearer might allow a better answer.
**Probabilities from the BH correction are usually specified as "q" values, to distinguish the false-discovery rates that they represent from the corrected "p" values, but your meaning is clear from the question.
***If the order numbers are large, you could consider standard multiple regressions, possibly with some transformation (like logarithmic) of the order numbers.
Comment: @Angus I added a thought at the end about examining groups of related UIC together rather than individually.
Comment: Thank you @EdM.  Very insightful.


Unable to download artifact from Azure DevOps "IIS Web App Deploy" Task

Question: Suddenly my Release pipeline were taking so long to download artifact and never completes it so my option is to only cancel it. I'm using Deployment Group Agent to deploy my .Net Core web app to IIS. Does anyone encountered this issue and any fix to this? I can't see any status or logs on whether it is downloading artifacts, so I totally don't know what is happening. I tried to remove the vsts agent on services then redeploy again a new vsts agent but still unable to download the artifact.
OS: Windows Server 2016
.Net Core APP
IIS 10
IIS Result, I cancelled it because I'm not sure if it is downloading the artifact
Release Pipeline for IIS using Deployment Group Agent
Answer: Turns out this is a network issue. Opening firewall rule port 443 or 8080 for vstsagent might help fix the problem.


Least square circle via fixed point iteration

Question: You have a collection of 2d points that you want to fit to a circle. Form the sum of the squares of the distances from the points to a generic circle. The variables are the $x,y$ coordinates of the center, and the radius. Set the gradient to 0, then one equation gives the radius as the average distance from the center to the points. For a given point $C$, let $r(C)$ denote the average distance from $C$ to the points. Using the other two equations, we find the center is a fixed point of the mapping $T(C)$, defined in the following way:
From each point, travel towards $C$ a distance $r(C)$, and then average all of these shifted points to get $T(C)$.
The mapping is not well defined at the points. It is possible in some arrangements of points and some starting point $C_0$ for the fixed point iteration $C_{n+1}=T(C_n)$ to land on one of the points to be fit. With this in mind, I'm trying to find a set on which $T$ is a contraction. Intuitively, we could take disks centered at each point to be fit, and grow them uniformly until they all intersect in a nonempty set, maybe let the disks grow a little more, and assuming the intersection doesn't contain any of the points to be fit, this should serve as a good candidate set to prove $T$ is a contraction. Indeed, it is easy to see that the shifted points will remain in their respective disks about each point to be fit, but it is not at all clear that $T(C)$, the average of the shifted points, will lie in every disk, but it has to be close! Note that the intersection of disks is a compact and convex set, and $T$ is continuous, so I have Brouwer fixed point theorem in mind.
Any ideas how to choose the Set? Is the intersection I described good? I just can't figure out a proof. I have tested the problem fairly thoroughly on a computer and the convergence of the fixed point iteration seems is quite reliable choosing $C_0$ as the average of the points.
In general, I do not expect uniqueness, for suppose there are only 1 or 2 distinct points, then there are infinitely many circles fitting the points exactly. Also note that if the points are distributed on a small arc of the circle, the average of the points is quite far from the center, yet convergence is observed, albeit quite slowly.
Edit
I was asked for formulas, so I give
$T(C) = \frac{1}{n}\sum_{i=1}^n (P_i + r(C)\frac{C-P_i}{|C-P_i|})$ where $r(C) = \frac{1}{n}\sum_{i=1}^n |C-P_i|$. And we want to find $C$ such that $T(C)=C$. I just need a set that $T$ maps into itself.
Comment: I suspect by "math", you mean equations and formulas, which do nothing to help solve the problem of finding a set that maps into itself under $T$.
Comment: Personally, I wish you had used more math and less words.
Answer: There's a pretty extensive body of literature on this problem. Apart from the theoretical interest, there is an important practical problem. When manufactured parts are inspected for quality, you get a bunch of points from a coordinate measuring machine. You often want to fit circles to these points, to estimate sizes of holes, diameters of shafts, etc.
For numerous papers and a book on the subject, look here.
Answer: Probably not an answer to the question but too long for a comment.
Supposing that you have $n$ data points $(x_i,y_i)$ and you want the "best" circle of radius $r$ centered at $(a,b)$, you need to minimize
$$SSQ(a,b,r)=\sum_{i=1}^n \left(r-\sqrt{(x_i-a)^2+(y_i-b)^2} \right)^2$$ which is an highly nonlinear problem. Any minimization method would do it provided goo estimates.
To get such estimates, in a preliminary step, you could write $$f_i=(x_i-a)^2+(y_i-b)^2-r^2$$ and now consider all the $\frac {n(n-1)}2$ equations
$$g_{ij}=f_j-f_i=2a(x_i-x_j)+2b(y_i-y_j)+\left((x_j^2-x_i^2)+(y_j^2-y_i^2)\right)$$ A linear regression (or matrix calculation) based on these $\frac {n(n-1)}2$ data points will provide $a$ and $b$. Using them would give as an estimate $$r=\frac 1n \sum_{i=1}^n \sqrt{(x_i-a)^2+(y_i-b)^2}\tag 1$$ Now, you have all elements for starting the optimization. You could even use Newton-Raphson method to solve
$$\frac{\partial SSQ}{\partial a}=\frac{\partial SSQ}{\partial b}=\frac{\partial SSQ}{\partial r}=0$$  which reduce to
$$\frac{\partial SSQ}{\partial a}=\sum_{i=1}^n \frac{(x_i-a) \left(r-\sqrt{(x_i-a)^2+(y_i-b)^2}\right)}{\sqrt{(x_i-a)^2+(y_i-b)^2}}=0\tag 2$$
$$\frac{\partial SSQ}{\partial b}=\sum_{i=1}^n \frac{(y_i-b) \left(r-\sqrt{(x_i-a)^2+(y_i-b)^2}\right)}{\sqrt{(x_i-a)^2+(y_i-b)^2}}=0\tag 3$$
$$\frac{\partial SSQ}{\partial r}=\sum_{i=1}^n \left(r-\sqrt{(x_i-a)^2+(y_i-b)^2}\right)=0\tag 4$$ The solution of $(4)$ is already given by $(1)$; so, you are left with two nonlinear equations in $(a,b)$.
If you are as lazy as I am, use numerical derivatives.
Comment: @Randy. What does mean *int* ?
Comment: What do you mean int?
Comment: Note that you can rearrange the equations provided above to get the fixed point problem $T(C)=C$ using the notation from the OP, and the formulas I added.


É necessário versionar a pasta node_modules no Git?

Question: Estou criando um projeto em Angular. Quando fui adicionar o projeto ao Git para fazer o commit, o terminal informou que o .gitignore está ignorando a pasta node_modules.
Percebi que a pasta tem muitos arquivos, mas é importante e necessário adicioná-la ao Git para que o projeto rode do mesmo jeito em outra máquina ou quem clonar o repositório irá executar isso ao baixar o projeto em sua máquina?
Comment: não deve, nem é necessário, o que você deve adicionar no git é o `package-lock.json`, pois na hora de executar o `npm install` ele irá baixar as versões exatas dos pacotes.
Comment: Essa é uma ótima pergunta na realidade. Acredito que vá ajudar muitos iniciantes :)
Answer: Essa é uma dúvida muito comum para pessoas que estão iniciando na área de desenvolvimento e vão mexer com algum framework Javascript.
Só para dar um contexto histórico: Antes do lançamento do Node, geralmente era necessário baixar dependências manualmente e colocá-las no projeto. Cada pluguin, font e biblioteca precisava ser baixada e adicionada em alguma pasta e versionada juntamente com o projeto, senão ele não funcionaria. Se fosse preciso usar uma versão mais recente de uma dependência, isso era feito excluindo os arquivos e então baixando a versão mais nova e colocando no lugar(Imagina o caos as vezes haha). Mais tarde, surgiram algumas soluções pra deixar esse trabalho mais fácil, como o Bower. Mas, o que realmente revolucionou a comunidade de Javascript foi o lançamento do NodeJs junto com seu gerenciador de pacotes, o NPM.
Com o node e o npm, ficou muito mais fácil criar e compartilhar código javascript, e com isso surgiram muitas libs para fazer tudo que se possa imaginar. E, além disso as ferramentas de frontend avançaram muito após, ficando com cada vez mais dependências. Mas, diferente do trabalho chato que citei antes de gerenciar manualmente, o npm consegue por meio do package.json salvar todas as dependências que seu projeto necessita, e cada dependência consegue baixar as suas próprias dependências e assim por diante. Além disso, agora é possível fazer update ou remoção de dependências de maneira muito mais fácil, com um simples comando.
Depois, foi criado um arquivo chamado package-lock.json. Esse arquivo é gerado automaticamente a primeira vez que você instalar as dependências. Como eu comentei, o npm cuida de instalar as dependências listadas no package.json e instalar as dependências das dependências, gerando uma espécia de arvore de dependências. Então, antes de existir o lock, algumas vezes alguma dependência podia ter conflito de versão por ser uma dependência do seu projeto em uma versão X e ao mesmo tempo ser uma dependência de outra dependência na versão Y, fazendo com que o projeto não funcionasse. Então, o package-lock.json serve para salvar exatamente a árvore de dependências e evitar que esse tipo de conflito aconteça.
Agora que expliquei o papel do npm e seu package.json e do package-lock.json, acho que deu para perceber que os arquivos de dependência em si não precisam ser versionados, porque o npm cuidará disso para você. Se olhar atentamente para o conteúdo da pasta node_modules, verá que é gigantesco e facilmente pode chegar até 1gb ou mais de tamanho. Seria bem improdutivo salvar todos esses arquivos desnecessariamente, já que o npm cuida exatamente de podermos baixar todos eles, na versão correta com apenas um comando.
Então, quando vamos versionar um projeto no git, seja de Frontend ou de Node, nós precisamos apenas salvar os arquivos do nosso projeto e os arquivos do package.json e do package-lock.json. No caso de um repositório público ou  compartilhado com outros, é uma boa prática criar um arquivo chamado readme.md com uma seção que ensina a instalar as dependências e a rodar o projeto.
Então, não se preocupe, quando alguém for baixar seu projeto, ela irá rodar o npm install e o projeto funcionará perfeitamente.
Comment: Muito bom, Fábio, Obrigado! Estava com essa dúvida, mas a explicação ajudou bastante.


General definition of characteristic function of a random variable and characteristic function of a probability measure

Question: Let $(\mu_n)_{n\geq1}$ be a sequence of probability measures on $\mathbb{R}^d$ and $(\hat{\mu}_n){_n\geq1}$ denote their Fourier transforms (aka characteristic functions). Starting from the definition of characteristic function of a random variable $X$ (defined on a probability space $(\Omega\text{, }\mathcal{F}\text{, }\mu)$) as $$\varphi_X(u)=\hat{X}(u)=\mathbb{E}(e^{iuX})=\displaystyle\int e^{iux}d\mu(x)=\displaystyle\int e^{iux}\mu(dx)\tag{1}$$ one has that $$\hat{\mu}(u)=\displaystyle\int e^{iux}\mu(dx)\tag{2}$$
Two questions:

As to the general definition of characteristic function of a random variable (see $(1)$), why does it hold that $$\displaystyle\int e^{iux}d\mu(x)=\displaystyle\int e^{iux}\mu(dx)$$?  That is, how is it possible to show that $$d\mu(x)=\mu(dx)$$?
As to the definition of the characteristic function of a probability measure (see $(2)$), as far as I understand I am integrating a function $f(x)$ with respect to probability measure $\mu$, with $x$ representing all the possible values of $\mu$. In other terms, I am integrating the function $e^{iux}$ with respect to the probability measure $\mu$ on the domain of the probability measure $\mu$ itself. Is that correct? Sorry if it could seem a bit messy.
Comment: $d\mu(x)$ and $\mu (dx)$ are two notations for one and the same thing.
Comment: @Strictly_increasing Indeed just a matter of notation and not a consequence of anything.
Comment: So, $d\mu(x)=\mu(dx)$ is just a matter a notation and NOT a consequence of a Theorem/result? @KaviRamaMurthy
Answer: By definition,
$$\int_X f(x) \mu(dx) = \int_X f(x) d\mu(x) = \int_X f d \mu$$
are all the same thing: a Lebesgue-integral w.r.t. the measure $\mu$.
Some authors even write things like
$$\int_X \mu(dx) f(x)$$
Answer: $\int f(x)d\mu(x)$ and $\int f(x)\mu(dx)$ both denote the integral of function $f$ wrt probability measure $\mu$.
Using a more neutral notation that avoids the bound variable $x$ we could say that: $$\int f(x)d\mu(x)=\int fd\mu=\int f(x)\mu(dx)$$
For any fixed $u\in\mathbb R$ we have the (nice) function $f_u:\mathbb R\to\mathbb C$ that is prescribed by: $$x\mapsto e^{iux}$$Then:$$\hat{\mu}(u):=\int f_ud\mu$$is well defined for every probability measure $\mu$ on measurable space $(\mathbb R,\mathcal B)$.
If $X:\Omega\to\mathbb R$ is some random variable then it has a distribution $\mu$ on $(\mathbb R,\mathcal B)$ prescribed by $B\mapsto P(X\in B)$.
In that situation the function $\hat{\mu}(u)$ is labeled as "characteristic function of $X$".
This all remains valid if we work in $\mathbb R^d$ and with random vectors.
Comment: @Strictly_increasing Given a random variable $X$, we have the induced measure $\mathbb{P}_X$. The characteristic function of $X$ is by definition $\hat{\mathbb{P}}_X(u)$
Comment: The statement that $\hat{\mu}(u)$ is the characteristic function of probability measure $\mu$ is more basic. It connects the function with a probability measure on $(\mathbb R^d,\mathcal B)$, and we can already speak of probability measures without speaking of random variables. But in practice often probability measures on $(\mathbb R^d,\mathcal B)$ are *induced* by random variables. In that case one also uses the terminology "characteristic function of r.v. $X$". Observe that the correspondence between Fourier transform and probability measure is one-to-one.
Comment: ...This is not so for random variables because we can have rv's $X,Y$ with $X\neq Y$ but still $P_X=P_Y$ and consequently $\hat{P_X}=\hat{P_Y}$.
Comment: My doubt arose since on my textbook it is said that $\hat{\mu}(u)$ is the "characteristic function (Fourier transform) of the probability measure $\mu$", NOT that $\hat{\mu}(u)$ is the "characteristic function (Fourier transform) of the r.v. $X$". Are these two statements indeed equivalent? Why? @drhab
Comment: Thank you a lot to both of you :)
Comment: In return thank you for editing.
Answer: There is a theorem (sometimes known as the change of variables formula) that says that, given a random vector $\mathbf{X}:(\Omega,\mathcal{F},P)\to\mathbb{R}^n$ and a measurable map $g:\mathbb{R^n}\to\mathbb{R^n}$ then

$E(g(\mathbf{X}))=\int_{\Omega}g(\mathbf{X})\text{ d}P=\int_{\mathbb{R}}g\text{ d}(P\circ \mathbf{X}^{-1})$.

So, in particular $\varphi_\mathbf{X}(\mathbf{t})=E(e^{i\mathbf{t}^\top\mathbf{X}})=\int_{\Omega}e^{i\mathbf{t}^\top\mathbf{X}}\text{ d}P=\int_{\mathbb{R}^n}e^{i\mathbf{t}^\top\mathbf{x}}\text{ d}(P\circ\mathbf{X}^{-1})$.
Obviously, $P\circ\mathbf{X}^{-1}$ is the measure over $\mathbb{R}^n$ induced by the random vector $\mathbf{X}$. You can find a proof of the change of variables formula in the theorem 1.6.9 of Rick Durrett's book.


PHP inserir uma array de dados em uma tabela mysql

Question: Eu possuo uma array em php que contém alguns dados, exemplo:
Array
(
    [nome] => Nome
    [sobrenome] => Sobrenome
    [genero] => 1
    [email] => firstname.lastname@example.com
)

E gostaria de inserir esses dados em uma tabela mySql de modo mais automático. No momento em que gero essa array, já é feito o processo para que fique de acordo com a tabela, ou seja, minha tabela é composta pelas colunas nome, sobrenome, genero e email.
Então, ao invés de fazer uma query assim:
INSER INTO tabela (nome, sobrenome, genero, email) VALUES ('$nome', ...)

Gostaria de saber se existe algum método de automatizar essa execução com base na array de dados que possuo.
Pensei em executar algum processo para tratar essa array da seguinte maneira:
foreach ($data as $key => $value){
    $keys   = $keys.$key.",";
    $values = $values.$values.",";
}
$keys   = substr($keys, 0,-1);
$values = substr($values, 0,-1);

INSER INTO tabela ('$keys') VALUES ('$values');

Mas não sei se é o mais adequado/ideal e/ou se existe um método "correto" de se executar essa tarefa.
Comment: @DanielOmine blz, valeu!
Comment: [sugestão](http://pt.stackoverflow.com/a/33528/91)
Comment: @rray parece que com `array_keys` e `array_values` eu consigo simplificar o código que usei como sugestão, correto?
Comment: Sim é uma forma de resolver o problema só não esqueça que as chaves do array devem ter os mesmo nomes das colunas.
Comment: @rray isso eu já estou fazendo justamente para tentar automatizar ao máximo esse processo.
Answer: Bom esta não é nem de longe a melhor solução, só estou lhe dando um norte, porém o ideal seria ter escapes no valor para evitar SQL Injection, então esta é só uma forma reduzida. ah! não testei.
$data = array(
    'nome' => 'Nome',
    'sobrenome' => 'Sobrenome',
    'genero' => 1,
    'email' => 'email@example.com'
);

$sql = sprintf(
    "INSERT INTO tabela (%s) VALUES ('%s');",
    implode(',', array_keys($data)),
    implode("','", array_values($data))
);

Answer: No osCommerce tem uma função que faz justamente isso que você precisa e tem até alguns recursos bem bacaninhas, basta adaptar no seu projeto ou analisar o código e fazer algo parecido:
// função principal
function tep_db_perform($table, $data, $action = 'insert', $parameters = '', $link = 'db_link') {
    reset($data);
    if ($action == 'insert') {
      $query = 'insert into ' . $table . ' (';
      while (list($columns, ) = each($data)) {
        $query .= $columns . ', ';
      }
      $query = substr($query, 0, -2) . ') values (';
      reset($data);
      while (list(, $value) = each($data)) {
        switch ((string)$value) {
          case 'now()':
            $query .= 'now(), ';
            break;
          case 'null':
            $query .= 'null, ';
            break;
          default:
            $query .= '\'' . tep_db_input($value) . '\', ';
            break;
        }
      }
      $query = substr($query, 0, -2) . ')';
    } elseif ($action == 'update') {
      $query = 'update ' . $table . ' set ';
      while (list($columns, $value) = each($data)) {
        switch ((string)$value) {
          case 'now()':
            $query .= $columns . ' = now(), ';
            break;
          case 'null':
            $query .= $columns .= ' = null, ';
            break;
          default:
            $query .= $columns . ' = \'' . tep_db_input($value) . '\', ';
            break;
        }
      }
      $query = substr($query, 0, -2) . ' where ' . $parameters;
    }
    return tep_db_query($query, $link);
}

// Funções chamadas
function tep_db_query($query, $link = 'db_link') {
    global $$link;
    $result = mysqli_query($$link, $query) or tep_db_error($query, mysqli_errno($$link), mysqli_error($$link));
    return $result;
}

function tep_db_input($string, $link = 'db_link') {
    global $$link;
    return mysqli_real_escape_string($$link, $string);
}

function tep_db_error($query, $errno, $error) {
    die('' . $errno . ' - ' . $error . '

' . $query . '

[TEP STOP]

');
}

Utilização:
$data_array = array(
    'nome' => 'Nome',
    'sobrenome' => 'Sobrenome',
    'genero' => 1,
    'email' => 'email@example.com',
    'fone' => 'null',
    'data' => 'now()');

// insert
tep_db_perform('clientes', $data_array);

// update
tep_db_perform('clientes', $data_array, 'update', "cliente_id = '1'");


Why logrotate makes an extra rotation?

Question: Trying to configure logrotate for 10 daily rotation in my dump folder I have 11 rotations (mydump.sql, mydump.sql.1 ... mydump.sql.11), why?
This is my logrotate configuration file in /etc/logrotate.d/ (I'm running Ubuntu server 12.4 LTS with logrotate 3.7.8):
/home/alepac/dumps/mydump.sql {
    rotate 10
    daily
    nomissingok
    create
    nocompress
    nocopy
    prerotate
        mysqldump -u alepac -pThisIsNotMyPwd myDatabase > /home/alepac/dumps/mydump.sql
    endscript
    postrotate
        rsync -avz --partial --delete /home/alepac/dumps/ my-remote-server.no-ip.biz:/home/alepac/db-backups/dbs/customer/ > /tmp/rsync.stdout 2> /tmp/rsync.stderr
    endscript
}

For better explaination, I'm rotating the dumps of a mysql database; I'm doing the dump on prerotate, then I rotate and sync the dump dir with remote machine using rsync on postrotate. Finally the create options makes a new zero sized dump to ensure next rotation.
Comment: Do you have current +10 or current +11 ?
Comment: @latin current +11, I edited the question for better understanding
Comment: This doesn't seem to be a good way to do MySQL backups. Why are you reinventing the wheel?
Comment: @Michael could you suggest another way to do the same (save db dumps of last 10 days to a remote location with a bad connection)
Answer: The problem is caused by rsync. The postrotate script is executed before removing the old log files so in the remote location I always have the mydump.sql.11 file. Moreover if postscript fails (e.g. whene there is a network problem rsync will fail) then the remaining actions are not executed and so the logrotate doesn't remove mydump.sql.11 file.

Why logrotate makes an extra rotation?


Dual list in asp .net mvc 4

Question: I have created a dual list in mvc 4. In post method I am not able to get the model data.
I can drag and drop an item from one list to another and on submit i need data in my post method. But in model.SelectedColumns I m getting null.What am I doing wrong ?
here is my code--
model :
public class Columns
{
    public List<string> SelectedColumns { set; get; }
    public List<string> RemainingColumns { set; get; }
}

view :
@Html.BeginForm("AddRemoveColumns", "DashBoard",FormMethod.Post){

        AddRemoveColumns

        @foreach (var item in Model.SelectedColumns)
        {

@item

        }

        @foreach (var item in Model.RemainingColumns)
        {

@item

        }

<input type="submit" value="Submit"  />
}

script :

controller :
public ActionResult AddRemoveColumns()
{
    Columns columns = DbAccess.getSelectedColumns();
    return View(columns);

}

[HttpPost]
public ActionResult AddRemoveColumns(Columns model)
{
    Columns columns = new Columns();
    columns.SelectedColumns = model.SelectedColumns;
    return View();
}

Answer: You are just displaying your list using html tags and you have not binded your list with any control that's why you are getting null value, If you want to use form posting then you have to bind you model with MVC controls as below:
@Html.ListBoxFor(model => model.SelectedColumns, new { @id = "SelectedColumns" })
  @Html.ListBoxFor(model => model.RemainingColumns , new { @id = "RemainingColumns " })
Thanks
Comment: I dont want to use @Html.ListBoxFor becoz i m using jquery sortable. So is there any way of getting the modified data of my lists in my controller ?...Thanks for reply
Comment: Simply you can use $.ajax for passing list to controller or if any how you want use form posting then maintain 2 hidden @Html.ListBoxFor list & bind it with model & modified their value using jquery on change of your actual list (jquery sortable list)
Answer: the form requires <input> to submit its values to controller
you can use hidden input if you don't want to use list box
<input id="hiddenInput" name="SelectedColumns" type="hidden" value="" />

*input name must be exactly same as model
then update selected values using jquery
$(function () {
    $("#SelectedColumns, #RemainingColumns").sortable({
        connectWith: ".connectedSortable" ,
        update: function() {
            var selectedValues = $("#SelectedColumns").serialize(); //get selected values
            $('#hiddenInput').val(selectedValues); //update hidden input
        }
    }).disableSelection();
});

Dual list in asp .net mvc 4


How to append a n-dimension image to the end of a (n+1)-dimension image?

Question: I have a 4D dataset with dimensions, e.g., 100(width) x 100(height) x 2048(energy channels) x 20 (frames).
Is there any function or method to append a 3D datacube (frame) of the same width, height and energy channels to the end of the 4D data set?
Finally the 4D data set should be 100*100*2048*21.
Answer:
There is no command which expands an image object in this way.

You have to perform this as a manual copy. (Which requires the additional memory, unfortunately.)
number sx = 100
number sy = 100
number sz = 100
number sf = 10

image before := RealImage( "Before", 4, sx, sy, sz, sf )
before = icol
image addFrame := RealImage( "Frame", 4, sx, sy, sz )
addFrame = irow

image after := RealImage( "After", 4, sx, sy, sz, sf + 1)
after.SliceN( 4,4, 0,0,0,0, 0,sx,1, 1,sy,1, 2,sz,1, 3,sf,1 ) = before
after.SliceN( 4,4, 0,0,0,sf, 0,sx,1, 1,sy,1, 2,sz,1, 3,1,1 ) = addFrame
before.showimage()
after.ShowImage()

If this is an issue, you can alternatively use the streaming commands to directly stream data (values only, no meta data) into either a buffer memory or onto the hard disc. Only at the end, you'd stream that data into an apropriately sized image variable.
Finally, depending on your application, you might consider adding new data as a taggroup to a taglist (which can be appended). You can store "images" as tags ( TagGroupSetTagAsArray() ). Again, you could create a 'complete' image form this information at a later time point.
number sx = 100
number sy = 100
number sz = 100
number sf = 10

tagGroup FrameTags = NewTagGroup()
tagGroup FrameList = FrameTags.TagGroupGetOrCreateTagList( "frames" )
FrameTags.TagGroupSetTagAsLong( "X Size", sx )
FrameTags.TagGroupSetTagAsLong( "Y Size", sy )
FrameTags.TagGroupSetTagAsLong( "Z Size", sz )
// Add frames whenever you need .....
for( number f = 0; f < sf; f++ )
{
    image addFrame := RealImage( "Frame", 4, sx, sy, sz )
    addFrame = random()
    FrameList.TagGroupInsertTagAsArray( f, addFrame )
}
// Handle taggroup as you want. (Add to global tags, add to image, store to disc... )
FrameTags.TagGroupOpenBrowserWindow( "frames", 0 )

// Whenever needed, reconstruct image from frames
number nsx, nsy, nsz
FrameTags.TagGroupGetTagAsLong( "X Size", nsx )
FrameTags.TagGroupGetTagAsLong( "Y Size", nsy )
FrameTags.TagGroupGetTagAsLong( "Z Size", nsz )
TagGroup nFrameList
FrameTags.TagGroupGetTagAsTagGroup( "frames", nFrameList )
number nsf = nFrameList.TagGroupCountTags()

image after := RealImage( "After", 4, nsx, nsy, nsz, nsf )
for( number f = 0; f < nsf; f++ )
{
    image addFrame := RealImage( "Frame", 4, sx, sy, sz )
    FrameList.TagGroupGetIndexedTagAsArray( f, addFrame )
    after.SliceN( 4,4, 0,0,0,f, 0,sx,1, 1,sy,1, 2,sz,1, 3,1,1) = addFrame
}

after.ShowImage()

Comment: Thanks for the detailed tutorial. One add-on question: since these 4D data can be very huge and very sparse (many zeros), is there any simple way to compress the data (but optimized for access)? Actually, my purpose is to read the multi-framed EDX datacube from Velox and evaluate the data frame by frame in a automated fashion.
Comment: No, not that I'm aware of. And certainly not when it is open in DM, because then the data is needed "expanded" for slicing and other operations. Practically, I use the "Save workspace" functionality to store STEM SI data, as it does optional ZIP compression.
Comment: I don't know the Velox format. Can you stream in (binary) with it? If so, that might be a nice option - stream in "what you need" for processing - and stream-out the results.
Comment: Velox uses a very compact Stream Event method to store the 4D SI in 1D:  the number in stream indicates the channel of the EDX-event, if the number is 65535 (max 2byte), this means 'go to the next pixel'. Scanning proceeds along X(fast direction) and Y(slow direction), no marker at the end of X-line the size of a frame is therefore width*height + number of events. In this way, 100*100*2048*10 data with typical 5 events per pixel result in data size of 100*100*(5+1)*10 * 2byte = 1.14MB. However in DM will be 100*100*2048*10 * 2byte = 390 MB! BTW, one need the metadata to reconstruct the stream.
Comment: @w4m Thanks for the info! Sure, storing things "small" is comparatively easy. The problem arises when one wants to act on the data (f.e. with the picker tool or slice tool), because then either you need a complicated access-method (slow) or the memory to hold things. In any case, nothing prevents you from storing DM EDS data with the same algorithm, just using streaming commands. Personally, I found that a simple ZIP (or similar) of the DM4 file does the trick quite conveniently as well.


Omega Theme and Superfish Menu

Question: I am trying to use superfish menu with Omega 7.x-3.0-rc3 and so far from bits and pieces on the internet I did the following.

Create sub-theme from the '\omega\starterkits\omega-xhtml' folder.
Add 'regions[superfish_menu] = Superfish Menu' in sub-theme info file.
Add the following in the sub-theme info file: (the paths are correct - sf folder is in my sub-theme folder.
;Superfish Stuff
stylesheets[all][] = sf/css/superfish.css
stylesheets[all][] = sf/css/superfish-navbar.css
scripts[]          = sf/js/superfish.js

Add a Menu called 'Top Navigation' to the region with an expandable Menu Item/Sub-Items.

This is where it seems I got stuck...to add the following to page.tpl.php: (I have renamed the div in the code so that it matches the correct Omega region div - region-superfish-menu)

I cannot seem to make it work as page.tpl.php does not exist in omega theme. Any clues? Thanks for helping!
Kevin
Answer: Why don't you just use the Superfish module that already integrates Superfish into Drupal. It automatically creates a block to add to any region you wish.
Comment: I can use Superfish module which would work as I just tried it now but I am learning Drupal Theming and want to implement a drop-down menu directly into my sub-theme. Thank you very much as I did not know about the ready-made module just the same so more knowledge for me :)
Answer: I would also recommend just using the Superfish module. It handles the block and has a nice admin. That said, 2 problems I see in your approach:
1) $ style jQuery does not work in Drupal 7. You can either replace '$' with 'jQuery' or use a wrapper like
(function ($) {
  // Original JavaScript code.
})(jQuery);

See http://drupal.org/update/modules/6/7#javascript_compatibility
2) The template you want for omega is probably html.tpl.php, not page.tpl.php.
Comment: As in my previous comment I want to learn how to implement superfish directly into my sub-theme for learning purposes. Thank you for the link and tip of what I should probably use for the template. I will try it soon and comment back if it works or not.
Comment: I managed to make it work as required. I have added the following code to html.tpl.php

`code`

Apologies for the messy code paste but could not manage to format the code in the comment the same as in my original question.
Answer: The best way to add code to a jQuery ready function is through Drupal behaviors. You can add a javascript file to your .info, let's call it loadsf.js.
scripts[] = js/loadsf.js

The contents of loadsf.js would be something like this:
(function ($) {

  Drupal.behaviors.superfish = {
    attach: function (context, settings) {
      if ($("#region-superfish-menu ul.menu").length > 0) {
        $("#region-superfish-menu ul.menu").superfish();
      }
    }
  };
})(jQuery);

Omega Theme and Superfish Menu


Apache reverse proxy with self-signed certificate

Question: I run a Unifi hardware appliance which comes with a self-signed certificate, issued on unifi.local. For my current setup, it's not an option to import a certificate on the appliance directly for several reasons, so I tried to get rid of the invalid certificate message of my browser by using an apache2-based reverse proxy which provides access to the appliance under another domain, secured by a Letsencrypt certificate.
My current setup looks like the following:
Laptop <-> Apache Reverse Proxy (2.4.48, Debian, trusted wildcard domain certificate) <-> Unifi appliance (self-signed certificate)

My idea is to provide a secured domain called unifi.mydomain.tld which allows secure access to the appliance.
In my apache reverse proxy, I created and enabled a config file which looks like following:
<IfModule mod_ssl.c>
<VirtualHost *:443>
    Serveradmin firstname.lastname@example.com"
    ServerName unifi.mydomain.tld

    SSLProxyEngine On
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off

    ProxyPass "/" "https://10.0.1.1/"
    ProxyPassReverse "/" "https://10.0.1.1/"
    ProxyPreserveHost Off

    TransferLog /var/log/apache2/proxies/unifi_access.log
    ErrorLog /var/log/apache2/proxies/unifi_error.log

    <IfModule mod_headers.c>
        Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains; preload"
    </IfModule>

    SSLProtocol all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1 +TLSv1.2
    SSLCipherSuite SSL ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384
    SSLCipherSuite TLSv1.3 TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
    SSLOpenSSLConfCmd DHParameters /mnt/certificates/diffie-hellman/dhparam4096.pem
    SSLHonorCipherOrder on
    SSLCompression off
    SSLSessionTickets off

    SSLCertificateFile /etc/letsencrypt/live/mydomain.tld/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mydomain.tld/privkey.pem
</VirtualHost>

# Originally from /etc/letsencrypt/options-ssl-apache.conf
# Written directly here because otherwise SSLProtocol etc is overwritten
# Add vhost name to log entries:
SSLOptions +StrictRequire
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common

</IfModule>

However, if I access unifi.mydomain.tld, my browser returns the certificate for unifi.local and not for unifi.mydomain.tld and hence, it creates an error that the certificate is not trusted. Several tips mentioned to turn SSLProxyVerify to none, and SSLProxyCheckPeerName, SSLProxyCheckPeerCN, and SSLProxyCheckPeerExpire to off, however, none of these tricks worked. It's not an option for me to import Unifi's self-signed snakeoil certificate on my reverse proxy server.
I'm not sure whether apache2 itself complains about the certificate or returns the wrong certificate. How am I able to get access to the appliance by browsing unifi.mydomain.tld without getting this certificate error?
Comment: It's not apache, almost certainly your browser is connecting to the Unifi NOT apache as you wish. Check the name resolution for unifi.mydomain.tld used by the browser. I say 'name resolution' not specifically DNS; name resolution _often_ uses DNS but it can use numerous other methods. If in doubt, try getting a network capture (wireshark, tcpdump, or similar) showing the browser connection(s) and particularly the IP address and MAC address it connects to.
Answer: This worked for me!
Requirements:

Apache Tomcat on :8443 with self-signed key
Apache HTTPD with Reverse Proxy to localhost:8443 Tomcat
Apache HTTPD REQUIRES Client Mutual Authentication. If that is not a requirement for you, then set:  SSLVerifyClient none
Apache HTTPD will pass along the X.509 identity of the caller through the reverse proxy to tomcat.

 ErrorLog logs/ssl_error_log
 TransferLog logs/ssl_access_log
 LogLevel warn
 ProxyRequests On
 ProxyPass /test1 https://localhost:8443/test/
 ProxyPassReverse /test1 https://localhost:8443/test/
# SSL Settings for the DMZ
 SSLEngine on
 SSLProtocol TLSv1.2
 SSLCipherSuite HIGH:!aNULL:!MD5:!SEED:!IDEA
 SSLCertificateKeyFile /home/jdoyle/sample_httpd_server_example_root_ca_.key
 SSLCertificateFile /home/jdoyle/sample_httpd_server_example_root_ca_.cer
 SSLCACertificateFile /home/jdoyle/consolidated_cacerts.cer
# SSL Settings for the Reverse Proxy
 SSLProxyEngine on
 SSLProxyVerify require
 SSLProxyProtocol TLSv1.2
 SSLProxyCheckPeerName off
 SSLProxyCACertificateFile /home/jdoyle/consolidated_cacerts.cer
 SSLVerifyClient require
 SSLVerifyDepth  10
# Pass the SSL Conext on to tomcat

RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
RequestHeader set SSL_CLIENT_I_DN "%{SSL_CLIENT_I_DN}s"
RequestHeader set SSL_SERVER_S_DN_OU "%{SSL_SERVER_S_DN_OU}s"
RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
RequestHeader set SSL_CLIENT_V_START "%{SSL_CLIENT_V_START}s"
RequestHeader set SSL_CLIENT_V_END "%{SSL_CLIENT_V_END}s"
RequestHeader set SSL_CLIENT_M_VERSION "%{SSL_CLIENT_M_VERSION}s"
RequestHeader set SSL_CLIENT_M_SERIAL "%{SSL_CLIENT_M_SERIAL}s"
RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
RequestHeader set SSL_SERVER_M_VERSION "%{SSL_SERVER_M_VERSION}s"
RequestHeader set SSL_SERVER_I_DN "%{SSL_SERVER_I_DN}s"
....

SSLOptions +ExportCertData +StrictRequire
....

Notes:
a. SSLProxyCACertificateFile is for the httpd<->tomcat connection and  contains the Public Key of the Tomcat Server's self-signed cert.
b. SSLCACertificateFile is for the DMZ <-> httpd connection, and must contain and and all of the CA Certs for any inbound connection.
Comment: Thanks, hoped that I do not need to install additional software like Tomcat, however this works.


The Ambiguity in Schwarz Information Criterion Definition

Question: Suppose there are 100 countries, $i = 1, 2, ..., 100$.
Let $b_i$ be the median birth weight of all new born boys in country #i in 2019.
Let $g_i$ be the median birth weight of all new born girls in country #i in 2019.
We want to model data set {$b_1, g_1, b_2, g_2, ..., b_{100}, g_{100}$} with two equivalent models.
Model #1.  Ordinary Least Square
Under this model, we have 200 observations whose dependent variable values are those above and whose independent variable values are 0, 1 dummy variable $d_j$ to indicate if it is a boy or not:
$z_j = x + y * d_j + e_j$ where $e_j$ is IID normal, for $j = 1, 2, ..., 200$.
Model #2. Constrained 0th-Order Vector Auto Regression
Under this model, we have 100 observations, each of which is a 2x1 vector.
$Z_j = (x, y)' + E_j$ where $E_j$ is IID normal, with a diag covariance matrix and equal variances, for $j = 1, 2, ..., 100$.
Ambiguity
Notice that these two models are mathematically equivalent, with the same maximum likelihood estimates for $(x, y)$ and the same maximized log-likelihood function values.
But the "numbers of observations" differ.  So the Schwarz information criterion values differ.  This is troublesome.  What is the proper Schwarz information criterion in this situation?

Thanks Eric, for your detailed response.   Sorry about my late reply.
I was not entirely explicit.   I was not trying to use BIC to choose between "model 1" and "model 2", as they are equivalent.  I should have phrased them as "approaches" rather than "models".
Formally and using the revised proper terms, I intend to, for example, use BIC to choose between two models (for the SAME data), whereas the model 1 is as I described above ("unrestricted"), while model 2 has an extra constraint $y = 0$ ("restricted").   Now for both models, I can parameterized them through either approach 1 (OLS) or approach 2 (VAR).  Under these two different approaches, it is possible that BIC would favor either the restricted or unrestricted model differently.
This bothers me, as these two approaches are equally valid.

Eric, I agree with you, when $d_j$ is stochastic, that the mixed normal would fall out of the exponential family and thus invalidates the key assumption of Schwarz (1978).
But what about the case, when $d_j$ is exogeneous/deterministic, subject to the technicality condition (just as in the classical OLS asymptotic setting), $\frac{\sum_1^N d_j}{N}$ converges to some constant (e.g. 0.5)?  In this case, our data set is $(z_j, d_j)_{j=1,2,...}$ with $d_j$ being degenerate, and the likelihood function would be exactly the same as that for VAR (and thus remains in the exponential family).
The key feature of the exponential family is that data and parameter "mix" in only one place.  Maybe there is an additional implicit "mixing" between $d_j$ and parameters, despite the identical-ness of likelihood functions on the surface?  I need to think more about this.  Love to hear your thoughts.
Comment: I hope I've answered your follow-up question in my revised answer.
Comment: Thanks Eric.  I skipped the Schwab original paper in grad school.  Going over it now.  Will be right back with comments/questions.
Comment: @ericperkerson.  Please see my revised question above as comments/questions to your revised answer.  Thx.
Comment: If $d$ was a degenerate random variable (deterministic), then how could the observations still be iid? Either they would no longer be identically distributed, because the even observations and odd observations are now drawn from different distributions, or they would no longer be independent from one another because $d_i = 1$ determines that $d_{i+1} = 0$. I don't think you can get around this issue to make the data come from a distribution in the exponential family.
Comment: Maybe I should have first pointed out that another assumption that the Schwarz criterion makes is that the data be distributed iid (from an exponential family distribution).
Comment: Even if you thought of the data as being generated by a VAR, this still violates the iid assumption. Successive data points from a VAR are not generally independent because they depend on the previous data point. So even if $d_j$ is made to alternate in a deterministic way, these are certainly not independent of previous values.
Comment: Thanks @ericperkerson.  My gut feelings are that I can relax that IID assumption and modify the proof to get to the same asymptotic result.  But that really is another topic.  In the mean time, I will take your rely as the official answer.  Thanks for your help!
Answer: Edit
Thanks for the clarifying point. In this case, the vector model (Approach #2) is the correct one, and the dummy variable model (Approach #1) is wrong.
This is because Approach #1 doesn't fit the assumptions of the Schwarz criterion, because the distribution of the observations $z$ does not follow a distribution from the exponential family. Because of the dummy variable $d$, the distribution of $z$ would actually be a mixture of two normal distributions, one for boys with mean $x + y$ and the other for girls with mean $x$. Mixture models are not generally members of the exponential family, see the last paragraph of the "Examples" section of the Wikipedia article for the exponential family.
Approach #2, on the other hand, has $z$ distributed according to the normal distribution
$$
\mathcal{N}\left( \left[ \begin{matrix} x + y \\ x \end{matrix} \right] , \left[ \begin{matrix} \sigma^2 & 0 \\ 0 & \sigma^2 \end{matrix} \right] \right)$$
which is a member of the exponential family. So Approach #2 is the correct one, and the correct number of observations is $n = 100$.
Original Answer
The data is not allowed to differ when using the Schwarz criterion to compare two models. To quote from Schwarz's original paper

In a general parameter space, there is no intrinsic linear structure. We therefore assume that observations come from a Koopman-Darmois [Exponential] family, i.e., relative to some fixed measure on the sample space they possess a density of the form
$$
f(x, \theta) = \exp(\theta \cdot y(x) - b(\theta)).
$$
where $\theta$ ranges over the natural parameter space $\Theta$, a convex subset of the $K$-dimensional Euclidean space, and $y$ is the sufficient $K$-dimensional statistic. The competing models are given by sets of the form $m_j \cap \Theta$ where $m_j$ is a $k_j$-dimensional linear manifold embedded in $\mathbb{R}^K$ for each $j$.

The two models in your question don't satisfy the assumptions for this setup, because you are changing the data $x$ between the two models (notice that $x$ does not have a $j$ subscript in the above). If you want to use the Schwarz information criterion (BIC) then you have to compare two models that satisfy these assumptions, which includes using the same data $x$. In particular,
$$
\{ b(1), g(1), b(2), g(2), ..., b(100), g(100) \} \ne \left\{ \left[ \begin{matrix} b(1) \\ g(1) \end{matrix} \right] , \left[ \begin{matrix} b(2) \\ g(2) \end{matrix} \right], ..., \left[ \begin{matrix} b(100) \\ g(100) \end{matrix} \right] \right\}.
$$


MongoDB: dilute collections when mongo is down

Question: as you all must know very well- mongo service is stopped when disk space reaches full capacity.
my question is- if and how can I dilute \ delete a collection while the mongo service is down?
Answer: Not sure what exactly you are looking for but to overcome from this situation you can follow below:

Copy you data dir to another location where you have space and delete any .lock file if exists.
Now you can try to start mongo with pointing to new dir like and repair below
mongod --dbpath  --port
db.repairDatabase()

Once service is up, you can do whatever you want to do.
Comment: thanks for the reply, and sorry for the late response. what I am hoping to accomplish is to delete some collections\data - clearing some space and allowing mongo to come back online without migrating data to another directory.

currently I am thinking somewhere along the line of finding out which .wt file belongs to which collection and deleting it..
Comment: @asktyagi Repair is equivalent to salvage and should not be used as a strategy to reclaim disk space in modern versions of MongoDB. In general, repair should be a last resort where you do not have another viable copy of the data . The WiredTiger storage engine may be able to release unused space using the [`compact` command](https://docs.mongodb.com/manual/reference/command/compact/), but note that this command will also block all other operations for the database containing the collection being compacted.
Comment: @GKman There are integrity checks for the files in your dbpath, so you definitely do not want to directly remove any .wt files. A much better strategy would be to monitor free disk space so you can free space preemptively (eg by dropping unused collections or indexes as your disk usage approaches 100%) rather than waiting until the situation is critical. If you do run out of space on a standalone node, you could restart on an alternative port (so your applications won't try to insert/update data), free up some disk space with collection or index drops, and then restart on the normal port.


Why $\lim_{n\to \infty }\max_{1\leq j\leq 2^n}|f(jt2^{-n})-f((j-1)t2^{-n}|=0$ where $f:\mathbb R\longrightarrow \mathbb R$ is continuous?

Question: Let $f:\mathbb R^+\longrightarrow \mathbb R$ continuous. Why $$\lim_{n\to \infty }\max_{1\leq j\leq 2^n}|f(jt2^{-n})-f((j-1)t2^{-n})|=0\ \ ?$$
Comment: What is $t$? And is the function $f$ defined on $\mathbb{R}$? or $\mathbb{R}^+$?
Comment: Because $f$ is uniformly continuous on $[0,t]$ for every fixed $t\in \mathbb{R}^+$.
Comment: @DanielFischer: I understood Surb answer, but how uniform coninuity can gives you immediately the result ?
Comment: You are looking at $\lvert f(x) - f(y)\rvert$, where $\lvert x-y\rvert = t\cdot 2^{-n}$. By the uniform continuity, for every $\varepsilon > 0$ there is a $\delta > 0$ …, then for all large enough $n$ we have $t\cdot 2^{-n} < \delta$.
Answer: First $jt2^{-n}\in [0,t]$ for all $j\in [0,2^n]$. By the way, since $f$ is continuous on $\mathbb R^+$, it's uniformly continuous on $[0,t]$. Now, let $j_n\in\{1,...,2^n\}$ s.t. $$|f(j_nt2^{-n})-f((j_n-1)t2^{-n})|=\max_{1\leq j\leq 2^n}|f(jt2^{-n})-f((j-1)t2^{-n})|.$$
Let $\varepsilon>0$ and let $\delta>0$ s.t. $|f(x)-f(y)|<\varepsilon$ whenever $|x-y|<\delta$. Let $N$ big enough to have $|jt2^{-n}-(j-1)t2^{-n}|<\delta$ whenever $n\geq N$. In particular, $$|f(j_nt2^{-n})-f((j_n-1)t2^{-n})|<\varepsilon,$$
whenever $n\geq N$, what prove the claim.


Differential equation/indefinitely often differentiable

Question: I have $y'=f(y)$ and $f \in C^{\infty}(\mathbb{R})$
I want to show, that each solution is indefinitely often differentiable in its existence intervall.
Proof: Because of $f \in C^{\infty}(\mathbb{R})$  each solution is unique and exists. Therefore it holds  $y'=f(y)$ and because of $f \in C^{\infty}(\mathbb{R})$  follows $y \in C^{\infty}(\mathbb{R})$
I hope, that's ok.
There is a second question to this: Solution is constant, if it has local max or min.
How can I proof that?
Answer: What you propose in your answer to the first question is circular reasoning, I am afraid: a composition of a $C^1$ and $C^{\infty}$ functions need not be of class $C^{\infty}$.
A proof should proceed by induction.  $t \mapsto y(t)$ is clearly of class $C^1$.  Assume that $t \mapsto y(t)$ is of class $C^k$, so $t \mapsto y'(t) = f(y(t))$ is $C^k$, too, but this means that $t \mapsto y(t)$ is of class $C^{k+1}$.
Regarding your second question, as $f$ is assumed to be $C^{\infty}$, the uniqueness theorem holds.  Let $t_0 \in \mathbb{R}$ be such that a solution $y(\cdot)$ has a local extremum at $t_0$.  Then $y'(t_0) = 0$.  But observe that both $y(\cdot)$ and a function constantly equal to $y_0 := y(t_0)$ are solutions to the initial value problem
\begin{equation*}
\begin{cases}
y' = f(y)
\\
y(t_0) = y_0,
\end{cases}
\end{equation*}
so they must be identical.  (Indeed, without uniqueness the property described in your second question holds, too, but a proof is much more tricky.)
Comment: Thank you for your answer:) You say:"But observe that both $y(\cdot)$ and a function constantly equal to $y_0 := y(t_0)$ are solutions to the initial value problem". What do you mean by a function?
Comment: I define a **number** $y_0$ as the value at $t_0$ of the solution $y(\cdot)$ of the equation, and then consider a function taking a constant value $y_0$.
Comment: I see:) You get, that$ y_0 $and$ y$ are solution ot the IVP, but because of the uniqueness must hold $y_0=y $
Comment: But why should $y(t_0)=y_0$ solve $y'=f(y)$. I mean, why should $f(y_0)=0$
Comment: Because $f(y_0) = y'(t_0) = 0$.
Comment: Thank you very much:)


Can a function and its derivative have different input variables?

Question:
The problem above is from Khan Academy. The correct answer is "C" and I know how to get it. This is what I have trouble with:
Whenever I see a function and its derivative, both the function and its derivative use the same input variable. For example $f(x)$ and $f'(x)$ both use $x$. In the Khan Academy problem, the function is called $f(x)$ so the derivative should be called $f'(x)$. But, since the answer is C, it is impossible for the derivative to be called $f'(x)$ because $f'(\frac{\pi}{2})=\lim_{x\to \frac{\pi}{2}}\frac{sin(x)-sin(\frac{\pi}{2})}{x-\frac{\pi}{2}}$. In other words, you can only write the derivative as $f'($any variable other than $x)$. So, if the function and its derivative in this problem have different input variables, how can you graph them on the same coordinate plane? I think this issue has more to do with how to graph stuff than calculus.
Comment: The reason that $f'(\pi/2)$ is written is that we are looking at the value of the derivative ($f'(x)$) *at $x = \pi/2$*. There's nothing wrong with writing $f'(x)$ in general, it refers to $\lim\limits_{u\to x}\dfrac{f(u) - f(x)}{u-x}$.
Comment: Oh so f(x) is not actually the function. Rather it is the function evaluated at x?
Comment: $f$ is the function
Comment: Because $f'(\frac{\pi}{2})=\lim_{x\to \frac{\pi}{2}}\frac{sin(x)-sin(\frac{\pi}{2})}{x-\frac{\pi}{2}}$ the derivative cannot have the input $x$. Im not really concerned with the derivative being $f'(x)$ my focus is on the fact that in this Khan Academy problem, it's impossible to say both the function and its derivative have the same input variable.
Comment: Is the reason you think the input cannot be $x$ because there is already an $x$ in the limit you wrote? If so, note that that $x$ is just a **dummy variable** for the limit, it could be written with any other symbol too, e.g. $f'(\pi/2) = \lim\limits_{u\to \pi/2} \dfrac{f(u) - f(\pi/2)}{u-\pi/2}$. When we write $f'(x)$, the $x$ here is different to the $x$ in the limit you wrote earlier. In that limit, $x$ was just a dummy variable of the limit, whereas in $f'(x)$, $x$ refers to the point we are evaluating the derivative at, and $f'(x) = \lim\limits_{u\to x}\dfrac{f(u) - f(x)}{u-x}$.
Comment: Yes because the $x$ in the limit stands for a coordinate of an arbitrary point that gets infinitely close to $(\frac{\pi}{2}, sin(\frac{\pi}{2}))$ while the input of the derivative stands for a coordinate of the point that we want to find the slope at.
Comment: Isn't $f(u)$ still the function?
Comment: Is $f(u)$ or $f(x)$ the original function?
Comment: What term represents the original function in your last limit equation?
Comment: The $f'(x)=\lim_{u\to x}\frac{f(u)-f(x)}{u-x}$ equation
Comment: $f(u)$ is "the (original) function evaluated at $u$". $f(x)$ is "the (original) function evaluated at $x$". For example, for the question you posted, $f(u) = \sin u$ and $f(x) = \sin x$.
Comment: Yes. Something like $\sin{x}$ refers to a function (the sine function here) evaluated at $x$.
Comment: I know this sounds really basic but why do graphs have an x-axis then? Isn't that implying you can only graph one value, which is $x$, on that graph?
Comment: You can think of the $x$ as a "variable", allowed to take on arbitrary numerical values. So the graph plots what happens as $x$ varies. If we write something like $f'(x) = \cos{x}$, it means "the derivative of $f$ at any $x$ is the cosine of $x$". So for example, the derivative of $f$ at $x = \pi/4$ would be $\cos(\pi/4)$, i.e. $\frac{1}{\sqrt{2}}$. The derivative of $f$ at $x = 1.2$ would be $\cos(1.2)$. And so on.
Comment: Wait so if I graph a function called $f(x)$ and then I graph its derivative $f'(z)$ should the horizontal axis be called the x, z axis ?
Comment: If you are graphing $y = g(z)$, then you may want to label the horizontal axis as "$z$". It doesn't change the shape of the graph if the actual function $g$ is the same though. It's just a label essentially.
Comment: I'm still confused about what allows someone to graph any 2 functions on the same coordinate plane. I think that's my question essentially, I don't even have to talk about derivatives actually.
Comment: Like what requirements are there for when you write out your 2 functions in function notation.
Answer: Always think of a function as being a conceptual machine that accepts some type of input (specified as its domain) and produces some output. We write "$f : S → T$" to denote that $f$ is a function with domain $S$ such that when you give it any input in $S$ it produces an output in $T$. We write "$f(x)$" to denote the output of $f$ when applied to input $x$.
$
\def\lfrac#1#2{{\large\frac{#1}{#2}}}
$
Now to define a function you must specify both its domain as well as its rule for what input yields what output. For example, we could define a function $g$ with domain $\mathbb{Z}$ such that $g(k) = k^2 - 3·k$ for every $k∈\mathbb{Z}$. Observe that the variable "$k$" here is solely used to define the input-output rule; it does not make any difference if we used any other variable such as "$x$" or "$h$".
Many teachers and textbooks are sloppy in defining functions. For instance, your quoted question from Khan academy appears to define function $f$ by the input-output rule $f(x) = \sin(x)$, but failed to specify its domain, so it is technically an incorrect question. However, we can guess that it wants $f$ to have domain $\mathbb{R}$.
Take any function $f : D → \mathbb{C}$ where $D⊆\mathbb{C}$. We define pointwise differentiability as follows:

Given any $x∈D$, we say that $f$ is differentiable at $x$ iff $\lim_{y→x}^{y∈D} \lfrac{f(y)-f(x)}{y-x}$ exists (in $\mathbb{C}$).

Then we define the derivative of $f$, denoted by "$f'$", as follows:

$\boldsymbol{f'}$ is the function with domain $E = \{ x : \text{$x∈D$ and $f$ is differentiable at $x$} \}$ such that $f'(x) = \lim_{y→x}^{y∈D} \lfrac{f(y)-f(x)}{y-x}$ for every $x∈E$.

Again observe that the variables $x,y,E$" in the above definitions are solely used within the definitions, and we could have substituted them for any three distinct variables.
The whole point is that it is incorrect to say that a function uses an input variable, but it is not your fault because many people teach the concepts wrongly. Rather, you need to use a variable when specifying the input-output rule in defining a function, but that variable is not in any way tied to the function itself. In fact, such variables are called dummy variables, as they only serve to link their occurrences, and do not have meaning outside of the definition.
As to the question in your comments, you have to distinguish between the function and its graph, which unfortunately is another frequent conflation by teachers. Given a function $f : \mathbb{R} → \mathbb{R}$, the graph of $f$ is defined as the set of points $\{ (x,f(x)) : x∈\mathbb{R} \}$, and you can plot the graph of the equation "$y = f(x)$" on the $(x,y)$-plane in the manner you have encountered. You can of course plot the graphs of more than one equation, such as the graph of "$y = f(x)$" and "$y = f'(x)$" on the same $(x,y)$-plane. Note that the choice of variables matters; if you plot the graph of "$x = f(y)$" on the $(x,y)$-plane, you will get a different plot from the graph of $y = f(x)$".
Answer: You're thinking about the derivative incorrectly.
$f^\prime(a) = \displaystyle\lim_{x\rightarrow a}\frac{f(x)-f(a)}{x-a}$. When this limit exist and is finite, we say the function is differentiable at a. So the x value in the limit as approaching the input value $a$.
Comment: In the equation you typed out, how can $f'(a)$ have the input $a$ and $f(x)$ have the input $x$. I don't understand why the derivative can have one input variable while the regular function can have a different input variable. Why is that allowed?