What is the difference between looping fread() over fwrite() vs using them separately?

Question: (Note: Apologies to anyone who read my previous post from half an hour ago- I've realised that I posted the wrong half of the code, and it was actually the second half that was giving me problems. So, if this post seems familiar it probably is!)
I'm trying to make a basic program which copies a .wav file, this was my attempt:
#include <stdio.h>
#include <stdint.h>

int main (void)
{
    FILE* input = fopen("input.wav", "r");
    FILE* output = fopen("output.wav", "w");

    uint8_t header [44];

    fread(header, sizeof(uint8_t), 44, input);
    fwrite(header, sizeof(uint8_t), 44, output);

    int16_t body;

    fread(&body, sizeof(int16_t), 1, input);
    fwrite(&body, sizeof(int16_t), 1, output);

    fclose(input);
    fclose(output);
}

However, after failing to make it work, I looked up how to do it and apparently
fread(&body, sizeof(int16_t), 1, input);
fwrite(&body, sizeof(int16_t), 1, output);

should be
while (fread(&body, sizeof(int16_t), 1, input))
{
    fwrite(&body, sizeof(int16_t), 1, output);
}

Why is it that the first part can copy the header without a problem, but the second part has to use a loop in order to work? To my eyes, they appear to be doing the same thing.
Comment: Both of your `while` loop has an empty body `;`. Voting to close as typo.
Comment: Hm? There is still a loop. Every call to `fread` or `fwrite` can return less than the passed number. In that case you need to repeat with remaining data from buffer. Maybe that is not needed for a single 16 bit value but you should get used to it.
Comment: The first part reads the full header. The second part reads only 1 single bit value of 16 bits. How should that copy the whole file unless the file is very very short? That's the reason for the loop in first place.
Comment: *Why is it that the first part can copy the header without a problem*  Since there's no error checking, the "copy ... without a problem" is based on mere hope.
Comment: Hi @AndrewHenle, thanks for your answer, can you elaborate on what you mean?
Comment: Basically you should also loop over `fwrite` and check if enough data was written.
Comment: Hi @Gerhardh, sorry, my mistake, my original code wasn't supposed to have a loop in it- I copied my code incorrectly. I've edited it now.
Comment: Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/228526/discussion-between-bluekhakis-and-gerhardh).
Answer: I think the fundamental issue here is that your code for reading and writing the header is reading the 44-byte header all at once, whereas the code for reading and writing the body is reading and writing just two bytes.  But the body of the wav file is probably much bigger than two bytes!
Specifically, this code:
fread(header, sizeof(uint8_t), 44, input);
fwrite(header, sizeof(uint8_t), 44, output);

reads and writes a 44-byte header, all at once.  But this code:
fread(&body, sizeof(int16_t), 1, input);
fwrite(&body, sizeof(int16_t), 1, output);

reads and writes two bytes.  If the body is just two bytes long, that's fine.  But if not, it's not going to copy the entire body; it's just going to copy the first two bytes.
This code, on the other hand, reads and writes multiple two-byte objects, as many of them as there are:
while (fread(&body, sizeof(int16_t), 1, input) == 1)
    fwrite(&body, sizeof(int16_t), 1, output);

If you knew how big the body was, you could read and write it all at once, also (that is, just like you did with the header):
uint8_t body[body_size];
fread(body, 1, body_size, input);
fwrite(body, 1, body_size, output);

Or, if you know the body size as 16-bit words, and you want to think about it that way, you could do
uint16_t body [body_size_in_words];
fread(body, 2, body_size_in_words, input);
fwrite(body, 2, body_size_in_words, output);

fread and fwrite can be confusing because they're designed to let you pretend you're reading and writing something other than individual bytes.  There are sort of three different ways to use them:

fread(buf, 1, n, fp)         read n bytes
fread(buf, size, 1, fp)   read one record of size bytes
fread(buf, size, n, fp)   read n records of size bytes

Deep down, fread basically just multiplies n * size, and tries to read that many bytes. Then, however many bytes it reads, it divides that number by size before returning it to you -- that is, it returns the number of records read, not necessarily the number of bytes read.  And the same examples, explanations, and arguments hold for fwrite, of course.
Answer: Have a look at the read and write functions. As you can see, the signatures of both functions are very similiar and use the same arguments. Both are used to operate on blocks of data.
 size_t fread(void *ptr, size_t size, size_t nmemb, FILE *stream);
 size_t fwrite(const void *ptr, size_t size, size_t nmemb, FILE *stream);

Argument
Description

*ptr
Points to the buffer where the data is stored

size
The size of each block in bytes

nmemb
The amount of blocks to read/write

*stream
Points to the structure which describes the outgoing stream

So, the difference in your code is:
// will read ONE block of data with the size of int16_t
fread(&body, sizeof(int16_t), 1, input);
// will read 44 blocks of data with the size of uint8_t
fread(header, sizeof(uint8_t), 44, input);

// will write ONE block of data with the size of int16_t
fwrite(&body, sizeof(int16_t), 1, output);
// will write 44 blocks of data with the size of int16_t
fwrite(header, sizeof(uint8_t), 44, output);

Both methods return a value of type size_t that reflects the amount of bytes which were read or wrote. If they return 0, nothing happened. As you know everything unequal to 0 is considered to be true in C. So to answer your question:
// reads ONE Block of size int16_t at a time and stores it in body
// => will return 0 and stop the loop if nothing was read anymore
while (fread(&body, sizeof(int16_t), 1, input))
{
    // writes ONE block of size int16_t from body to output
    fwrite(&body, sizeof(int16_t), 1, output);
}

The loop will continue as long as there is data left to read, then stop. The header of the file has a fixed size, as it is standardized. The rest of the file has a variable length, hence the loop.
Comment: You should also tell the OP to always check the return value of `fread()` and `fwrite()`: if the header cannot be fully read or written, something fishy is going on an should be reported. Same for the `fwrite` in the loop.  It is also advisable to read and write the samples in large groups to minimize overhead.


Show that $\tau_A$ is a topology on $A$

Question:
Let $(X,\tau)$ be a topological space and $A \subset X$. Let      $\tau_A$={$A \cap U: U \in \tau$}. Show that $\tau_A$ is a topology on $A$.

I know that I need to prove three properties to prove that $\tau$ is a topology, however, I don't know how to prove it. Any help would be appreciated thanks.
Comment: $\emptyset\in\tau$ by definition, so $\emptyset=\emptyset\cap A\in\tau_A$
Comment: Exactly. Don't forget that you can use the fact that $\tau$ is a topology on $X$.
Comment: What have you tried? "Follow your nose" proofs directly from definitions work for all the properties you need.
Comment: I don't know how to prove them. Just know what they are. For the first I was thinking that the null set and $A$ would belong to the topology as the topology is defined as an intersection between $A$ and an open set $U$. Is that right?
Comment: @MikeHaskel How do I prove the other two properties though? I can't just assume any open set and prove that it's union and intersection is in the topology. Or can I?
Answer: By definition $\varnothing,X\in\tau$, so $\varnothing = \varnothing\cap A\in\tau_A$ and $A=X\cap A\in\tau_A$.
If $U\cap A, V\cap A\in\tau_A$, then $U\cap V\in\tau$ so
$$(U\cap A)\cap(V\cap A) = (U\cap V)\cap A\in\tau_A. $$
If $U_\alpha\cap A\in\tau$ then $\bigcup_\alpha U_\alpha\in\tau$ so
$$\bigcup_\alpha (U_\alpha\cap A) = \left(\bigcup_\alpha U_\alpha\right) \cap A\in\tau_A. $$
Hence, $\tau_A$ is a topology on $A$ (called the subspace topology).
Comment: How do I prove intersection?
Comment: Do you mean, why does $(U\cap A)\cap(V\cap A) = (U\cap V)\cap A$? This is a straightforward exercise, just show that the sets are equal if you aren't convinced.
Comment: No, I mean that to prove topology we have to prove that the union and intersection of an arbitrary collection of open sets is in the topology right? You've proven the union? The same thing needs to be done for the intersection?
Comment: That is precisely what I proved.
Comment: Ok, so a similar thing needs to be done for the intersection? I just assume two sets and take it from there?
Comment: For exercises like these you just need to write down the definitions and see exactly what it is you need to show. It really isn't that difficult - it follows directly from the definition of a topology and some basic set theory.

Show that $\tau_A$ is a topology on $A$


Can I use the MIDI In of the Launchpad Pro to send MIDI messages to my computer?

Question: I'm not sure whether this is on topic here. Please direct me elsewhere if it isn't.
I wonder about the MIDI In port of the Launchpad Pro.  From Ableton you can send MIDI messages to the Launchpad, specifically notes to light up the pads.
I wonder if this is also the only use of the MIDI In port.
What I want to do is built a Arduino-based MIDI controller that I can use with my Launchpad. It is supposed to send MIDI messages like Pitch Bend, Modulation or Sustain Pedal.
My question is:

If I send MIDI controller messages to the MIDI In port of the
  Launchpad would they be send to my computer / Ableton?
Answer: Doesn't look it you can 'forward' midi messages from your Arduino to your computer via launchpad pro.  Looks like you can only send sysex messages to light pad / scroll text ect
Can you perhaps multiplex the message to both your computer and the launchpad?
Comment: Well,  it isn't completely clear to me from the document, but you are *probably* right. - I don't actually need to send any messages to the Launchpad, that's why I'm going to build my MIDI controller as a standalone USB device now.


How do I show a set $A = (A\setminus B)\cup (A\cap B)$ for discrete math?

Question: How would I go about showing that for any set A and B,
$$A = (A\setminus B)\cup (A\cap B)?$$
I don't really understand how to show this. Here's my interpretation:
$$A = (A\setminus B)\text{ or } (AB)?$$
$\cup$ means either or both sides? $\cap$ means both $A$ and $B$?
How do I show that
$$A = (A\setminus B) \cup (A\cap B)?$$
Thanks!
Comment: Venn diagram maybe?
Comment: A "truth table" style inclusion/exclusion would work.
Answer: $$\begin{align} x \in (A-B) \cup (A\cap B) &\iff x \in A- B, \;{\bf \text{  or }}\; x \in A \cap B \\ \\ &\iff \underbrace{(x \in A {\bf\text{ and }} x \notin B)}_{\large x \in \,A-B} \text{ or } \underbrace{(x \in A {\bf\text{ and }} x \in B)}_{\large x\in\, A\cap B}\\ \\ &\iff x \in A \text{ and } \underbrace{(x \in B \text{ or } x \notin B)}_{\text{TRUE}}\tag{DL} \\ \\ &\iff x \in A\end{align}$$
This relies only on definitions: set union, set-minus $(A - B = A\setminus B)$, set intersection, along with the distributive law (DL): $$(p \text{ and } q) \text{ or }(p \text{ and } r) \equiv p \text{ and } (q \text{ or } r)$$
Comment: C-D={1,5,6} tells you that {1,5,6} are in C but not in D. D-C={2,8} tells you that {2,8} are in D but not in C. This means we are only missing the values that are in both sets. But these are CnD={3,4,7} so we can 'add' these values to both C and D. So:
C = {1,5,6,3,4,7} and D = {2,8,3,4,7}
Comment: Using this, how would I find sets C and D if: C-D={1,5,6}, D-C={2,8}, and CnD={3,4,7}?
Comment: Awesome, thanks!
Answer: Consider the mechanism of a "truth table" where "true" means "is contained in set X" and "false" means "is not contained in set X":
<code>A B | A\B | A intersect B
-------------------------
T T |  F  |      T
T F |  T  |      F
F T |  F  |      F
F F |  F  |      F
</code>
The remaining statements should be easy to finish.
Note that your interpretation of the symbols is basically correct; $\cup$ is called "union" and means "is in either set", while $\cap$ is called "intersection" and means "is in both sets".  The $\setminus$ "set subtraction" operator is also written as the intersection with the complement of the right-hand set, so we have $A\setminus B=A\cap B^c$.
Answer: You can draw a Venn diagram but it would not be a formal proof, more of a way to visualize the problem for yourself. If you want to proof the equation you should only use the rules of set theory page 5 and 6.
You can then justify each step by saying which rule you apply. For example a first step could be to apply the distributive law then work your way forward, should not be that hard.
Here is the derivation:
$$
(A \setminus B) \cup (A \cap B) = ((A \setminus B) \cup A) \cap ( (A \setminus B) \cup B) )\\
= (A) \cap ( (A \setminus B) \cup B)) \\
= (A) \cap ( A \cup B) \\
= A
$$
Answer: First, to adress your question of what $\cup$ and $\cap$ mean. They are operators joining two sets into one set. An element $x$ is in $A\cap B$ if it is BOTH in $A$ and in $B$. Mathematically written that means
$$x\in A\cap B \Longleftrightarrow x\in A \wedge x\in B$$
where the operator $\wedge$ means (and is read as) "and". Similarly, an element is in $A\cup B$ if it is in $A$ or if it is in $B$:
$$x\in A\cup B \Longleftrightarrow x\in A \vee x\in B$$
Where $\vee$ means "or".
Another operation on sets used in your task is set difference, $A\setminus B$. An element is in thus set if it is in $A$ and it is $not$ in $B$, meaning
$$x\in A\setminus B \Longleftrightarrow x\in A \wedge x\notin B.$$
Now that operations are clear, all you have to do is to prove two sets are equal. For any $2$ sets $C$ and $D$, the sets are equal if they have the same elements, meaning every element of $C$ is in $D$ and every element of $D$ is in $C$. This means that to prove $C=D$, you must prove $$x\in C \Longleftrightarrow x\in D$$ for all $x$. Try it!


How to copy current directory into Drupal profile folder via drush?

Question: I've this yml file in my current folder:
$ cat profile.yml
api: 2
core: 7.x
projects:
  drupal7:
    type: "copy"
    download:
      url: "file://."

and 3 empty files (just for now): drupal7.info, drupal7.install, drupal7.profile.
And I've downloaded fresh Drupal:
$ drush -y dl drupal
Project drupal (7.39) downloaded to drupal-7.39.

Now I'm trying to copy my profile into Drupal directory via:
$ drush make --no-core profile.yml drupal-7.39/
Beginning to build profile.yml.
Non-existent project type copy on project drupal7.

and the same with:
$ drush make -r drupal-7.39/ --yes --no-core profile.yml

This is based on this example:
projects[example][type] = "profile"
projects[example][download][type] = "copy"
projects[example][download][url] = "file://./example"

but I'd like to achieve that in yml format.
What is the correct syntax then? Or how does it work?
Answer: I actually had to specify extra type: "profile" (to follow this example), so type is specified twice (type of project, and type of download), as below:
api: 2
core: 7.x
projects:
  drupal7:
    type: "profile"
    download:
      type: "copy"
      url: "file://."

Then this would work fine:
$ drush make --no-core profile.yml drupal-7.39/
Beginning to build profile.yml.
drupal7 copied from file://..

To avoid copying docroot/ it-self, it's good to actually create a separate folder for profile (as specifying wildcard won't work).


What are the disadvantages of buying a short-scale bass guitar

Question: I'm considering buying my first bass. I already play guitar, and find that even though I'm an adult, stretching to a fifth is a struggle -- so handling a bass with its more widely spaced frets is going to be a challenge.
It occurred to me that a short-scale bass might be easier for me to play. Of course a smaller acoustic instrument is quieter, but for an amplified instrument like a bass, it shouldn't be an issue.
Are there any reasons I should prefer a full size bass?
Comment: One disadvantage that has occurred to me, however, is that as much as I practice on my short-scale instrument, if someone hands me a normal bass, I'll be less able to play it.
Comment: FWIW, when it comes to bass guitar, "full size" simply means "the size Leo Fender picked in 1951." It's not as if a 34" scale length is divinely mandated. Sounds like you like a shorter scale, so I'd say go for it.
Comment: Just for closure, I went to the guitar shop, and there was a Gretsch short-scale bass on display. I had a go, and not only did I find the closer-spaced frets much more pleasant to play, but (to my ears) it sounded much nicer than the other basses in my price range. So I am happy.
Comment: @AlexBasson true, but presumably Leo Fender didn't pluck that size from the air - he'll have experimented with different sizes. Then again, he didn't have the materials we have today. I'm very tempted by the Kala U-Bass, which uses polyurethane strings to get bass pitches from an instrument the size of a baritone ukulele.
Comment: @slim Late to the party perhaps, but as someone who regularly switches between an EB-0 and a standard P-bass, I find that after a while switching between them isn't particularly difficult.  As a matter of fact, I tend to be just a bit faster on the EB-0, although every once in a while I feel clumsy when playing some things higher up on the neck.
Comment: use whatever is most comfortable. You can change the sound and playing experience by using different gauges and types of strings. As noted below, you can always put a capo on fret 3 of a standard bass.
Answer: Volume is obviously not the only aspect of an instrument's sound: frequency distribution, attack characteristics, sustain, tuning accuracy etc. are equally or more important.
Short scales tend to (but don't necessarily) result in

shorter sustain
less pronounced treble frequencies
somewhat "flappier" attack
less well-defined pitch control.

Most of this seems technically disadvantageous, but how good or bad it is musically depends on the context and playing style. The sound of many Beatles songs is influenced quite heavily by McCartney's Höfner bass, in part by the fact that it's short-scale. The same sound would probably not work very well in, say, a funk rock or metal band.
On the other hand, it's of course not only the scale that makes up the sound. If you put bright-sounding strings on a 30" with low-impedance active electronics and tune it up to A, you get much the opposite of the characteristics described above: a glassy piccolo-sound well-suited for playing soloes.
Comment: Shorter scale basses don't allow for the range of bottom end either. The Low `E` gets floppy.
Comment: Nice. Since I'm planning to play reggae, loss of treble doesn't worry me too much. Not sure about loss of attack though. Since these are all questions of sound, I can make a judgement in the shop. Will leave a couple of days to allow for other opinions before accepting the answer.
Comment: @JarrodRoberson: in practice yes, indeed; however not in principle. That's also a matter of not just scale but as well string gauge and -height. A shortscale bass with fat flatwound strings one inch above the fretboard can easily compete with ordinary longscale basses in the low end – nowhere else, though...
Comment: @leftaroundabout That is exactly what I had a short scale Les Paul with flat wound strings, it was great for what it was, I mean "Sunshine of your Love" sounded awesome on it. But it was very limited for other genres, I don't think it would have made a good Reggae instrument, no sustain and no brightness.
Answer: After 15 years of playing around with 34 inch basses, I bought an Ibanez 5 string mikro.
I love it.
Overnight many things which I struggled and failed to be able to play years ago, I can play now with ease. I can't put the bass down and it has totally brought me back into playing music.
Before purchase I did research online, and honestly, I don't know what people are talking about.
The strings ARE NOT FLOPPY. They just are not. I kept on reading this online. All someone needs to do is actually pick up a short scale bass - and this should be clear nonsense.
Are far as the others - I have an old Peavy Forum and a Dean edge 5 string.
All I can compare to is what I have. The Ibanez as a thicker, richer tone, if anything more sustain, and oh... the other basses have dead spots, the Ibanez Mikro does not. It sounds great and for $200 - I paid about $300 for the others and that was 15 or so years ago!
About comments like this:
"Get used to what most people use."
1) play what works for you, not what someone else tell you to  play because it works for them.
"After all, if short scales were superior, why are they relatively rare ?"
2) As far as I can tell, short scales are relatively rare b/c of ignoramuses telling people to not use them... and making up nonsenses about poor tone and floppy strings.
3) "Small hands are no excuse"
Um, if your hands can't reach, they can not reach. PERIOD. You can not magically grow larger hands!
This is what happens in real life. You have to play an F and G note on the E string. Can you reach with your index finger or use your pinky? If you have to use your pinky, then it's sort of big for you. Now what if you have to play F and then 4th fret. Now you have to move your hand where someone with larger hands does not. So now you have to work harder, trying to plan how to move your hand around to play, in order to compensate for small hands. Or your can just get a smaller instrument that actually fits your hand.
Maybe, if people were not bullied into buying the wrong instrument for them, then 2 things would happen:
1) More people actually playing and enjoying bass, because they have an instrument to play
2) More availability of short scale basses!
Answer: It is more difficult to achieve drop tunings while keeping sufficient string tension on short scale basses.
Answer: To answer your question about the disadvantages of buying a 30" scale bass...there are none. For those naysayers...think Hofner for starters... Leo Fender, when he picked 34" was simply using @ 3/4 of the scale length of an upright bass. At one time most of the cheap copies were short scale, which probably contributed to the Short scale bad rep. Now that more people are using their ears instead of their egos, the 30" scale is becoming more popular. IF you look at some of the "Bass Gods" from the 1960s (See old Ed Sullivan shows), you will find that Fenders rarely appear, and Epiphone/Gibson, Vox, Hofner, Framus, Hagstrom are far more prevalent and they are almost exclusively short scale instruments, and yet the SOUND!!! With amplifiers and computers today , if you can pass a signal through a wire, you  can make it sound like a 32-piece orchestra if you are of that mind...And I read some things here that are incorrect, in particular, sustain, which is a function of the materials of the instrument rather than scale length (Think: if it were scale length, guitars would have no sustain at all). Do your research, and buy the bass that you will enjoy playing rather than one bought from silly peer pressure.
Comment: Wonder whether even shorter scale basses would be even better then?
Answer: Want thump on a short scale?  Decent price and playability as well?  FLATWOUNDS.  I have an Epiphone EB-0 (30" scale), an Ibenez Mikro (another 30"), and I think a Memphis, and I've owned Musicmasters, Mustangs, and Corts, and all of them were exceptional, played like champs and gave me their all...as long as I made one slight modification, FLATWOUNDS.  Roundwounds tend to muddy up with time as well as compete with the guitars, flatwounds put me in the missing spot when playing live and in practice, and I get alla the girlies hawt when I give them the lower end thump, rather than the twang.  34" scales are murder on my tiny hands, I've loved 30" scales and once tjey have flatwounds and been set up proper with neck angle, truss rod tweak, and bridge intonation (I'm a huge fan of bolt-on necks, as set necks rob me of adjustments when I change strings), then I can compete with just about any millie long scale out there.  see my kids on my site xanDOTduneDOTnet for details.  The long scale I have is a Jazz copy, and flatwounds make her sweeter.  I also have a Dean EAB, and she's a killer to play, long scale, but again, flatwounds make her stand out of the croud.  I've played her with three acoustics and she was still abke to be heard without an amp.  Flatwounds whatever you choose, will make the difference.  Short scales will make you concentrate on the music, not your stretch.  The Mikro I was playing in a music store while a friend shopped for a tele, and I couldn't leave the store without her, $125 and they threw in a bag, cable, strap, and the whole 9 yards.  Enjoy, just go flatwounds.
Comment: Flatwounds versus roundwound is a question of sound preference. I personally can't live without the 'twang' (as you put it) of roundwounds :-)
Answer: Only just picked up your question.  Probably you've equipped yourself with a bass by now. However if one is used to a standard guitar, the bass will feel like a different beast. That's 'cos it is!  Longer strings will be necessary anyway, so it's worth just playing in a different way. You will get used to it. Longer strings will give more 'body' to the sound - they need to be thicker to produce the same notes.You're not going to be playing chords (unless you have a 5-string high C or a 6-string).  So the stretch you are concerned about won't be a worry.  If it's reggae, then a lovely low sound is in order, so I'd advocate a 5 string in any case - low B may not be much help open, but the extra richness of a thicker string, along with the fact that you've got another 4th (interval) under most people's E is a great bonus.
Bass Collection basses are one of our best kept secrets.  SGC Nanyo versions, '80s and 90s, though, not the newer Chinese versions.  Loads of tonal differences, lovely to play, and not expensive.Even if you're already tooled-up, try to play one - you won't be disappointed.
You've already mentioned getting used to a short scale and then struggling to play a proffered standard.  Don't go there! Get used to what most people use. After all, if short scales were superior, why are they relatively rare?   Small hands are no excuse - I can just stretch an octave on piano, and it doesn't stop me using standard.
Comment: Dv-er - care to explain what you didn't agree with in this answer..?
Answer: I am a newbie bass player. I found that easiest instrument to play is a medium-scaled (32 inches) bass with capodaster set on the 1st or 2nd fret, and tuned to standard EADG. This make the scale equivalent to 30 or 28.5 inches, respectively. Not only frets are shorter and do not require stretching, but also strings are closer to frets, and string tension is less. Sustain and timbre are even better than without capodaster, to my taste.
The only drawback is that the distance between strings become wider, which become noticeable when it is set at 2nd fret. My bass has 17.5 mm distance between strings at bridge which cannot be changed. I'd like to have strings closer to each other.
I am sure capodaster can be applied to 34" scaled basses too (capodaster on 2nd fret turns scale to 30.3 inches), only choose bridge with strings close to each other.
Comment: You bring up a good point: a longscale bass basically _contains_ a shortscale piccolo, if you just don't use the lowest frets! A capo isn't even really necessary for this – many bassists almost never play empty strings anyway.
Comment: Unless you play a lot of open strings, the capo isn't going to make much difference. Once you fret above where the capo is, there could be another foot of neck stretching out behind it, not affecting anything.
Comment: I've never heard "capodaster" before, only "capo." Thanks for the introduction to that word. ;)
Comment: There are 5 string basses available, using 4 string necks. This would solve your problem of string spacing. Heck, if you couldn't cope with 5 strings, one could be taken off!
Answer: There are many reasons for wanting a SS bass. I have a gimp fretting hand /arm from a motorcycle accident and both pronation and supination are limited. I have to play in a more upright position and need a SS for comfort. The bottom line is what works for you doesn't work for me. So get what works for you and I will do the same.
Answer: I was bass player once during my early stage being a string musician, but now I'm a lead guitarist by now. I used both of that short and long scaled basses mentioned here. My own gear is a long scale custom p-bass and sometimes I use a 5 string short scale Ibanez.
Short scale:

Rapid transition of notes is possible because of smaller fret spacing
easy for me to get used to especially when using pick
it almost same size of regular guitar
smaller overall package
strings are not tight compared to full scale

Long scale:

better sustain
richer sound ever for a single coil
more pronounced treble good for slapping
for some players but not me,the tighter strings is a plus. They like
the feel especially when finger picking
Comment: Nobody is mentioning how the higher notes on a SS sound fatter.
Answer: I started bass in 1958 that will be 60 years next October. All that time I have had what someone called a "Gimpy" arm. It wont straighten and I often have to throw my shoulders way left to get across the fret board so I developed my own -Thumb Over- style. I played 34" for 50 years. At 60 I started to play short scale bass and just converted a copy Telecaster and a Strat, guitars to 30" bass. Once I sorted the fretboard and tuning out. I used some old flat wounds fitted Fender Single coil pups and the resultant instrument blew me away! I have helped a lot of people and now I'm 72 years old I would recommend SS to anyone. I see fools on Ebay selling SS bass as "Kids Bass and it saddens me. I have seen SS Bass played in Orchestras in New York London and Helsinki. A 30" or 32" bass is not a kids bass. It's an instrument with its own special place amongst stringed instruments. It not only meets physical needs of some people. It meets very demanding musical needs within the profession. Only a fool knocks a short scale bass!
Comment: One fact overlooked in the answers is that if one plays a SS, and is given a full-sized to play (at a gig, open mic, rehearsal, et al) one looks a bit of a fool saying 'I can only play the SS.' And, because it's so guitar like, the tendency for some is to play it like a guitar.
Answer: There are no disadvantages to playing a short scale guitar or bass. As long as a Short Scale is whats needed. I make and play them, and I play them when they are needed. If you don't need one and cannot find a practical reason for owning one, don't go there! As a bass player of 60 years I have played hundreds of pieces of the years where a short scale was asked for and did its job. Visit YouTube. There are some interesting videos relating to SS Guitars. Good Luck
Comment: Can't think why a SS bass would be required over a standard bass for any job that could not be done equally as efficiently with a standard bass. Enlighten me. I've played for several decades and never come across the differentiation.
Answer: I am in my second life as a bass player.  The first time around I had a '67 Fender Mustang Bass and a `67 Gibson
EB-0 fitted with Guild nylon tape wound strings.  Both of these guitars were 30 inch scale and were great guitars (once I quit using Gibson strings, the flatwounds sounded head and the roundwounds were like kon my ol.d carpanives on my fingers.  I have fat fingers but small hands and short arms and  the 30 inch scale and smaller
bodies worked well for me.  i unfortunately gave up playing and sold my stuff (wish I still had those guitars) so when the bug bit me again I bought what I knew and what I could afford as a retiree, a Squirer
Bass, a PJ Bass (precission neck pickups and jazz bass
bridge pickup.   Nice bass but the stretches  are hard p


k-fold crossvalidation and independent models evaluation

Question: I have a question about using k-fold cross-validation.
From what I have read, there are several steps for using it.

Shuffle my data.
Train my model for
each fold on training data and test it using testing data.
I get k different success-rates for the evaluation of the model so I take a mean and interpret it as the model's average performance.

Now for another model I repeat all steps, I get another average performance and I can compare these two models. Am I right?
Now comes the tricky part. Do I shuffle the data ahead of evaluation for each model?
I've tried searching this and haven't run into any solid mathematical argument. I think that both not-shuffling and shuffling before each model evaluation makes sense. Could you point me to some article?
Answer: Assumption 1: All initial problems orders are uniformly random, and that the only question we have is: how to randomize the problems (i.e. should we randomize them for each model? or should we randomize them once for all models?)? (which I think is your question).
Following assumption 1
If we model $k$-fold x-val as a function, I would suggest the following to represent the average accuracy of model $m$:
$$f(m,k,\mathcal{P}, o)$$
where $\mathcal{P}$ is the set of problems, and $o$ defines the order of the problems in $\mathcal{P}$ before they got partitioned into $k$ folds.
So, in other words, in my view, the score of $k$-fold x-val is only a function of:

Model under evaluation (i.e. $m$).
Total problems set (i.e. $\mathcal{P}$).
Initial order of problems set before partitioning them (i.e. $o$).
And total number of folds (i.e. $k$).

Let's now consider various cases and how they relate to the value $f(m,k,\mathcal{P}, o)$.
Case 1: If $k = |\mathcal{P}|$ (i.e. total number of folds is equal to total number of problems, then for any different orders $o_1$ and $o_2$, the scores are identical:
$$f(m,k,\mathcal{P}, o_1) = f(m,k,\mathcal{P}, o_2)$$
So in the case when $k = |\mathcal{P}|$, clearly the order doesn't matter.
But, in some cases it's impossible to have $k = |\mathcal{P}|$ without causing a class imbalanced in training or testing folds. So I think we should not be too happy about Case 1.
Let's look at another case where the order $o$ does not matter.
Case 2: For any orders $o_1$ and $o_2$, the following is true:
$$
\lim_{|\mathcal{P}| \rightarrow \infty}f(m,k,\mathcal{P}, o_1) = \lim_{|\mathcal{P}| \rightarrow \infty}f(m,k,\mathcal{P}, o_2)
$$
In my view this statement is true because of the law of large numbers.
But similarly to Case 1, I think we should not get too excited about this because usually our problems are not large enough to trigger this equality.
Let's look at another case that is more realistic.
Case 3: For a finite problems set, it is IMO very likely that there exists orders $o_1$ and $o_2$ such that:
$$f(m,k,\mathcal{P}, o_1) \ne f(m,k,\mathcal{P}, o_2)$$
This is an empirical thing. Personally, based on all datasets that I have evaluated against, the probability that this inequality is true is $1$.
Perhaps you can run a Monte-Carlo based simulation to estimate the probability that the inequality above is true as follows:
$$\begin{split}
=&\frac{\text{total num of times }f(m,k,\mathcal{P}, o_1) \ne f(m,k,\mathcal{P}, o_2)}{\text{total number of evaluations}}\\
=&\frac{1}{n} \sum_{i=0}^{n} I(f(m,k,\mathcal{P}, o_{1+i}) \ne f(m,k,\mathcal{P}, o_{2+i}))\\
>& 0 \ \ \ \ \ \text{(I think this is true for most cases)}
\end{split}$$
where $I$ is an indicator function that returns 0 if statement is false, else returns 1.
I think you wouldn't be surprised if it's $> 0$ for all your datasets.
So, clearly, for any models $m_1$ and $m_2$, and any orders $o_1$ and $o_2$, the following test -by itself- is not very meaningful:
$$f(m_1,k,\mathcal{P},o_1) > f(m_2,k,\mathcal{P},o_2)   \ \ \ \ \ \ \ \ (1)$$
because there are two variables: two models $m_1$ and $m_2$, and also two different orders $o_1$ and $o_2$. Therefore, you can't know which one is causing which. Is it due to the order $o_1$ that is very lucky and helping $m_1$ to score better? Or is due $m_1$ being really good? Or a mixture of other factors?
Alternatively, if you choose a single fixed order $o_1$, then (note that order is $o_1$ for both models):
$$f(m_1,k,\mathcal{P},o_1) > f(m_2,k,\mathcal{P},o_1)  \ \ \ \ \ \ \ \ (2)$$
means that model $m_1$ is better than model $m_2$ when tested against $\mathcal{P}$ problems as ordered by the specific order $o_1$.
So clearly $(2)$ is telling us more information about the models than $(1)$.  I.e. at least now we could say something about models $m_1$ and $m_2$ even though it is still somewhat useless as it is speific to the order $o_1$, but at least better than $(1)$ that didn't even isolate the effect of the order!
So, $(2)$ tells us more information about the models, but it is specific to order $o_1$. Do we care? Usually, no. We don't care about the order. So while $(2)$ is better than $(1)$, it is still not very useful.
Therefore, we need to repeat $(2)$ multiple times (forget about $(1)$), each time with a different order $o_i$ (where $i \in \{1, 2, \ldots, n\}$. Then, we need to perform statistical significance tests across all the orders $o_1, o_2, \ldots, o_n$, and then measure the $p$ value. Then we conclude that, for any $i \in \{1,2,\ldots,n\}$, $f(m_1,k,\mathcal{P},o_i) > f(m_2,k,\mathcal{P},o_i)$ is the case only if $p \le \alpha$.
Therefore, we now define another $k$-fold x-val function that is independent of the order $o$ as follows:
$$
g(m,k,\mathcal{P}) = \lim_{n \rightarrow \infty} \frac{1}{n} \sum_{i=1}^n f(m,k,\mathcal{P},o_i)
$$
But since we cannot realistically choose $n \rightarrow \infty$, we choose $n$ that is large enough. How large is large enough? I suggest the following:

Large enough to give us a $p$ value that is less than $\alpha$.

Important note: choosing $n$ that is large enough (maybe $n=1000$) to give you a small enough $p$ value (maybe $p \le \alpha$) does not necessarily mean that model $m_1$ is significantly better than $m_2$. Instead, it only means this:

A1: the order-independent $k$-old x-val process thinks that the score of $m_1$ is higher than that of $m_2$.

Which is a different answer than the following:

A2: $m_1$ is better than $m_2$.

Simply calculating the $p$ value above to show A1 does not show A2.
Unanswered useful questions (beyond your main question)

What if Assumption 1 is violated? This could be interesting for time series problems?
Is the number $n$ independent of the size of the problems set $\mathcal{P}$?
How to show A2?
Comment: Thank you, that was the kind of answer I was looking for.

From my experience, people are only doing one cross validation whereas you advice to do this n=1000 times. Is one cross-validation a good estimate?

A2: I thought the whole point of cross-validation was to get an idea on which model is better. Could you elaborate little bit on this?
Comment: I didn't suggest $n=1000$, I just put it as an example to make my post more readable. I suggest you to choose any $n$ that gives you $p \le \alpha$ (usually you want $n$ to be smallest that satisfies $p \le \alpha$ so that you don't waste your time repeating for too long. As for *A2*, nope, $k$-fold x-val doesn't aim to show statistical significance due to the difference. It only tries to measure some statistic (like accuracy). However, answering whether the score of $m_1$ is better than that of $m_2$ is due to a systematic difference (as opposed to sheer dumb luck) is a different question.
Comment: Sorry I forgot to answer your other question: is doing $k$-fold x-val one time good? Well, I think we can't tell. It depends on how lucky the order $o_i$ was right? One way to find out is to repeat it with different initial orders $o_i$ for $i \in \{1,2,.\ldots,n\}$, then estimate the order-independent variant $g$ good enough that gives you a different that is significant for the $k$-fold x-val process.
Answer: What you're doing here is an example of Adaptive Data Analysis, ie. selecting a model by looking at the CV results of two or more models. This could be dangerous and a lot of research is being carried out right now on how to mitigate it.
So, take extra precaution when you're doing CV for many models. For a demo of the failure of cross validation, see[1].
[1] Rao, R. B., Fung, G., & Rosales, R. (2008, April). On the Dangers of Cross-Validation. An Experimental Evaluation. In SDM (pp. 588-596).
Comment: Thank you. Could you please give a few reasons on why CV fails, and a few examples on how to mitigate it?


Why is this point set a circle?

Question: consider a circle in Euclidean plane $E$ and any point $A$ in the interior of the circle. Now consider all secants $s_A$ to the circle through the point $A$. The claim is now that the set of midpoints to all the secants $s_A$ consitutes an circle.
I'm wondering how to prove this interesting fact? What is the best approach?
Best regards
Comment: You could construct the midpoints $A_1, A_2, A_3, A_4$ so that $A_1A_2A_3A_4$ is a square. Then you form up points "between" $A_1$ and $A_2$, and using method of exhaustion form a one fourth of a circle.
Comment: By "secants" do you mean the segments joining $A$ to the circumference?
Answer:
CD is a chord and it passes trough A. F is its midpoint. Since $\triangle OCD$ is isosceles $OF\perp CD$. Then if $E$ is the midpoint of $OA$, $FE = EA$ . This will be true for every chord trough $A$.
Answer: Without loss of generality, assume

The outer circle is centered at the origin (by translation)
The point $A$ is on the $x$-axis (by rotation)

So the circle has equation
$$x^2 + y^2 = r^2 \tag{T1}$$
If the construction creates an inner circle then:

The leftmost point of the inner circle is halfway between $(-r, 0)$ and $(A, 0)$ : $L = \frac{A - r}{2}$
The rightmost point of the inner circle is halfway between $(A, 0)$ and $(r, 0)$ : $R = \frac{A + r}{2}$
The center point of the inner circle is halfway between $L$ and $R$ : $C = \frac{L + R}{2} = \frac{A}{2}$
The radius of the inner circle is the distance from the center to the edge : $r_2 = R - C = C - L = \frac{r}{2}$

So if an inner circle is constructed, then it has the equation:
$$\left(x - \frac{A}{2}\right)^2 + y^2 = \left(\frac{r}{2}\right)^2 \tag{T2}$$
Consider a line drawn through point $A$ along vector $\begin{bmatrix} m \\ n \end{bmatrix}$ :
$$\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} m \\ n \end{bmatrix} t + \begin{bmatrix} A \\ 0 \end{bmatrix} \tag{T3}$$
Finding the intersection of the outer circle and the line, combining (T1) and (T3) , we get:
$$t_1=\frac{\sqrt{(n^2+m^2)r^2 - n^2A^2} - mA}{n^2+m^2}$$
And finding the intersection of the inner circle and the line, combining (T2) and (T3), we get:
$$t_2=\frac{\sqrt{(n^2+m^2)r^2 - n^2A^2} - mA}{2(n^2+m^2)}$$
So $\frac{1}{2}t_1 = t_2$, so the inner circle of (T2) indeed bisects each line from $(A, 0)$ to the outer circle.

Why is this point set a circle?


Pesquisa com filtro Jquery

Question: Tenho uma um Select, e um campo de pesquisa, que busca tudo dentro de uma tabela.
<select class="custom-select col-12" id="selecionarOrgao">
   <option value="">Escolha o órgão</option>
    <option value="2">portaria</option>
    <option value="3">Vidraçaria</option>
</select>

    <input class="form-control input-search" placeholder="Digite aqui sua pesquisa." id="buscarServico" name="buscarServico" value="" type="text" />

Essa é minha tabela:

            Serviço
            Descrição
            Órgão

               Vidro

               Concerto de porta de vidro

               Vidraçaria

                Concerto de porta

              Concertar portas de todos os modelos, exceto porta de vidro

                Portaria

Tenho esses dois scripts, um é para fazer o filtro pela pesquisa, de acordo com o que o usuário digita, e o outro é para o select, buscar somente os resultados de acordo com o escolhido, no select, busco pelo código, que está no data-cod da
.
Filtrar pela Pesquisa digitada:

Selecionar de acordo com o orgão selecionado:

O problema é que do jeito que mostra ai, depois que eu seleciono o orgão, e faço uma pesquisa, ele me trás todos os resultado, mesmo os que não são do orgão que eu selecionei. ou seja, mesmo com a
 em hide, aparece nos resultados.
como eu faço, para fazer a busca de acordo com o que digito somente nos resultados do orgão selecionado???
Comment: Talvez não seja útil para vc, mais tem uma lib já que faz muitos dessas funcionalidades já pronta, a lib é [DataTable](https://datatables.net/). De uma olhada e vê se pode ti ajudar em algo.
Answer: Tente asssim:
Adicione uma classe à todas as
, para ser mais específico:

    Vidro

    Concerto de porta de vidro

    Vidraçaria

Altere sua função de filtro para fazer com que esconda tudo primeiro e depois só mostre o que você quer/precisa:
var filterTable = function (item) {
  var val = item.val();

  $('.filter-rows').hide();
  $('.fiter-rows [data-cod="' + val + '"]').show();
}

Comment: Obrigado pela resposta, mas ainda assim, continua buscando tudo.
Answer: Troque a função para:
function filterTable (item) {
    var val = item.val();

    $('.filter-rows').hide();
    $('.filter-rows[data-cod="'+val+'"]').show();
}

Testando e funcionando:
https://the-cyberpunk.com/exemplo/
Você também pode adicionar um if, para se a pessoa voltar para o "Escolha uma opção", ele mostrar tudo novamente:
function filterTable (item) {
    var val = item.val();

    if(val.length > 0) {
        $('.filter-rows').hide();
        $('.filter-rows[data-cod="'+val+'"]').show();
    } else {
        $('.filter-rows').show();
    }
}

Pesquisa com filtro Jquery


Django-models: use field from foreign key

Question: I'm working on an image gallery project in DJango, just for the sake of it. And well, I have a class named Gallery and a class named ImageGallery.
The class named gallery would look like this:
class Gallery(models.Model):
 gallery = models.ForeignKey(Gallery, related_name="parent_gallery")
 title = models.CharField(max_length=200)
 folder = models.CharField(max_length=200) # ex: images/galleries/slugify(self.title)

class ImageGallery(models.Model):
 gallery = models.ForeignKey(Gallery, related_name="parent_gallery")
 title = models.CharField(max_length=200)
 image = models.ImageField(upload_to=self.gallery.folder)

Well, the last line of code is what I want to know if its possible or any other fine replacement for it.
In the DJango-admin I want to be able to add records for the table ImageGallery and when selecting the Gallery I would like to use, the images to be saved to the folder specified in gallery.folder field.
What's the best way to go around this? I haven't completed writing the two classes but I doubt they're gonna work like this. Thank you in advance. :-)
Answer: The FileField.upload_to is defined as follows

This attribute provides a way of setting the upload directory and file
  name, and can be set in two ways. In both cases, the value is passed
  to the Storage.save() method. ... upload_to may also be a callable,
  such as a function. This will be called to obtain the upload path,
  including the filename. This callable must accept two arguments and
  return a Unix-style path (with forward slashes) to be passed along to
  the storage system. The two arguments are:

But self.gallery.folder isn't a callable. What you need is to setup a function along the lines given in that example
def get_upload_path(instance, filename):
    return '{0}/{1}'.format(instance.gallery.folder, filename)

And your model will change to
image = models.ImageField(upload_to=get_upload_path)

Comment: Wow, I don't know how I missed that. That aside, the way to go is pretty awesome. yay for Django!


Why $P(A)\cdot P(B \mid A) = P(B)\cdot P(A\mid B)$ for dependent events?

Question: Repeating the question

Why $P(A)\cdot P(B \mid A) = P(B)\cdot P(A\mid B)$?

It makes total sense to me when we consider independent events. However, I struggle to understand why it also holds for dependent events.
If we assume that $P(A)\cdot P(B \mid A) = P(B)\cdot P(A\mid B)$ holds, then we also assume that the order of the events happening does not matter. Why would we?
Let's consider an example. Let $A,B$ be arbitrary two events. Let $P(A) := \frac{1}{3}$ and $P(B) := \frac{1}{3}$.
Suppose that if $A$ happens, then probability of of $B$ happening is $\frac{1}{5}$, or in other words, $P(B|A) = \frac{1}{5}$
And suppose that if $B$ happens, then probability of $A$ happening is $\frac{1}{6}$, or $P(A|B) = \frac{1}{6}$
We have
$$P(A) \cdot P(B\mid A) = \frac{1}{3}\cdot \frac{1}{5} = \frac{1}{15} ≠ \frac{1}{18} = P(B) \cdot P(A\mid B)$$
So in this example, clearly, equality doesn't hold.
I assume it would be possible to come up with some "real world" examples which would also show that equality might not necessarily hold.

So what am I missing?
Comment: Both sides are equal to $P(A \cap B) \equiv P(B \cap A)$.
Comment: It's as if you were to say "Suppose that $x=6$, and suppose that $y=7$, and suppose that $x+y=3$".  Those suppositions can't all be true.
Comment: @NateEldredge, it's also like saying, I don't understand Pythagoras' $a^2 = b^2 + c^2$, because if you take $a=3, b=2, c=1$ it does not hold ;)
Comment: Your example has supposed a bunch of statements that are contradictory, that's all.
Comment: @NateEldredge Can you elaborate on what you mean by contradictory, please?
Comment: Simply that it is not possible to have events whose probabilities and conditional probabilities all have the values that you suppose.  Try to think of a real-life example of a random experiment where they all hold - you won't be able to.  More formally, try to actually define a probability space which has two events for which all those statements hold - you will fail.
Comment: Normally the $\sigma$-algebra that contains all events is not equipped with any order. So there is no such thing as "order of events". You could say that this part of reality is not modeled.
Answer:
"Let's consider an example. Let $A,B$ be arbitrary two events. Let $P(A) := \frac{1}{3}$ and $P(B) := \frac{1}{3}$.
Suppose that if $A$ happens, then probability of of $B$ happening is $\frac{1}{5}$, or in other words, $P(B|A) = \frac{1}{5}$"

From that you can conclude that the probability that both events will occur is $P(A)P(B\mid A)=\frac13\frac15=\frac1{15}$.
With exactly the same reasoning we come to the conclusion that the probability that both events will occur equals $P(B)P(A\mid B)$ so our final conclusion is that:$$P(A)P(B\mid A)=P(A\cap B)=P(B)P(A\mid B)$$
Answer: note, that $P(A) P(B | A) = P(AB)$ is not a theorem, that holds by definition of conditional probability $P(B | A) = \frac{P(AB)}{P(A)}$. Similarly $P(A | B) = \frac{P(AB)}{P(B)}$ by definition, and that implies the bayes formula $P(AB) = P(A) P(B | A) = P(B) P(A | B)$
Answer: Suppose a box contains 5 white and 3 black balls. You take 2 balls without replacement. So, the second ball color depends on the first ball color. Hence:
$$P(W_1\cap B_2)=P(W_1)\cdot P(B_2|W_1)=\frac{5}{8}\cdot \frac{3}{7}=\frac{15}{56}\\
P(B_1\cap W_2)=P(B_1)\cdot P(W_2|B_1)=\frac38\cdot \frac57=\frac{15}{56}$$


Adjusted $R^2$ & F test are not shown in regression with robust standard errors in Stata

Question: The adjusted $R^2$ is not shown when a regression with robust standard errors is calculated in Stata.
This is surprising to me since the value of the $R^2$ is unaffected in regressions with robust standard errors.
Is there any statistical reason for not quoting the adjusted $R^2$ when using robust standard errors in regression?
Furthemore if I add more variables  the F test disappears.(e.g. with 9 variables it shows but not with 13), Is this for the same reason? How can we report such results and deal with this issue?
Answer: If you specify the robust option, you are telling Stata that you don't really believe the errors are homoskedastic. There are several implications:

The sums of squared residuals are still used to drive the estimation (you minimize them).
As every observation has its own variance, the sum of the squared errors is no longer distributed as a $\chi^2$, and the ratios of the model and residual sums of squares are no longer distributed as an $F$.
Since different observations contribute different amount of information, there isn't any way now to correct $R^2$ for the prognostic value the way $R^2_{\rm adj}$ does: one observation $\neq$ one degree of freedom.

Essentially, as the notion of variance is not quite applicable to the dependent variable (do you want to talk about the conditional variance of $Y$, i.e., the variance of the error terms, which, as we said, is not constant any more; or do you want to talk about the total variance, which, inconveniently, depends on the distribution of the explanatory variables), the whole concept of $R^2$ starts breaking down in its meaning.
Comment: Plus adjusted R2 is a horrible thing.
Comment: [According to this post](http://www.stata.com/statalist/archive/2004-05/msg00107.html) which I have just found, "the F statistic, for instance, is no longer based on sums of squares; it becomes a Wald test based on the robustly estimated variance matrix" (is this correct?).  What is puzzling is that the $R^2$  is still shown.
Comment: This is a normative judgement, @JeremyMiles. Obviously, a lot of people expect to see it... and also want to see it in Poisson and logistic regression, as well.
Comment: @StasK On the same logic, the $R^2$ is shown but. Its only the adjusted $R^2$ which is not shown. And so is the F-test.  So technically, when I report the data in this case, show I report the $R^2$ (ignoring the adjusted $R^2$?). How am I to report the data given I'm using the robust se.
Comment: @JeremyMiles Explain why you think so.
Comment: @CesareCamestre - not a horrible thing - I overstated, but not a useful thing. We don't correct most things to make them into population effects. Unless you have a pretty small sample, and/or a really large number of variables, the adjustment is tiny. Adjusted R-squared can be negative, which makes no sense.
Comment: @StasK I have added a further question on the f-test, in this discussion rather than posting a new post because its related.  Would appreciate feedback.
Comment: @StasK I thought that $R^2$ is simply an algebraic description of explained variance -- does it actually require the assumption of homoskedasticity? In deriving $R^2$ I did not think that it's necessary.
Comment: With homoskedastic data, things are clear. With heteroskedastic data, you have to start hesitating whether you have fixed or random regressors, as then your computation of the total and explained variances may differ (the latter making things more complicated). Stata simply protects you from using the numbers that don't have a good justification... unlike Some Alternative Software that would easily produce a chi-square of 10^8 in a non-significant regression in a sample of size 100 if that's what your weights sum up to.
Comment: +1, but I don't understand this: "or do you want to talk about the total variance, which, inconveniently, depends on the distribution of the explanatory variables." Var(Y) doesn't depend on the distribution of the explanatory variables; and Var(Yhat) is pinned down by the point estimates and the distributions of the explanatory variables. Relatedly, your last comment doesn't quite make sense to me, I'm not sure what "fixed vs random regressors" means. Don't regressors have to be looked at as random variables for Var(Yhat) to make sense?


Is it possible having this type error in Flutter even if this part was copied from pub.dev? How can I solve it?

Question: I am trying to make this stopwatch app and on the page:
https://pub.dev/packages/stop_watch_timer
I have found the example but I have two specific lines of code which raise this error, which refers to the third line of the reported code.
The argument type 'int?' can't be assigned to the parameter type 'int'.

builder: (context, snap) {
       final value = snap.data;
       final displayTime = StopWatchTimer.getDisplayTime(value);
...

How can I solve this issue?
Comment: The example probably was never updated for null-safety.  You should file a bug against the package about it.  You need to check if `value` is `null` before you use it.  For example: `...getDisplayTime(value ?? 0)` or, if `value` is guaranteed to never be `null`, ...getDisplayTime(value!)`.
Answer: int and int? is different data type. int? can be null and int not. you can convert int? using ! or ??. example intVar! or intVar ?? 0


Boolean Expression Simplification $A'B'CD + A'B'C'D + ABC'D' + A'BC'D + AB'C'D'$

Question: I want to simplify the below equation but I can't get the same result with an online calculator (the result for the calculator is the correct - I check it with Logisim).
$Y = A'B'CD + A'B'C'D + ABC'D' + A'BC'D + AB'C'D'$
$Y = A'B'(C'D + CD) + A'BC'D + C'D'(AB' + AB)$
--> From Redundancy Laws: $AB + AB' = A$
$Y = A'B'D + A'BC'D + C'D'A$
--> I can simplify until here
The complete simplification is the below, but I can't find a boolean's law to use to remove the $B$.
Does anyone knows which law I have to use in order to remove the variable?
$Y = A'B'D + A'C'D + C'D'A$
Answer: Here is something you can do:
$A'B'D + A'BC'D + C'D'A =$
$A'D(B' + BC') + C'D'A =$
$A'D(B' + B)(B'+C') + C'D'A =$
$A'D1(B'+C') + C'D'A =$
$A'D(B'+C') + C'D'A =$
$A'B'D + A'C'D + C'D'A$
Comment: Many many thanks!


Python regex - why 'e42dsf425' is matched?

Question: pls check the regex below
import re

passwords = ['aaa5aa', 'adsfgg', 'e42dsf425']

regex = r'\w{3}\d{1}\w{2}'

for p in passwords:
    if re.findall(regex, p):
        print(p)

I dont uderstand why this regex match both: aaa5aa and e42dsf425. I expected to see only aaa5aa as we have \d{1} in the pattern.
Thank you very much, have a nice day!
Paweł
Comment: If your goal is to test these against your exact pattern try to change `re.findall` to `re.match` to test against a boolean return on a whole match. That should also forfeit the need for anchors `^` and `$`.
Answer: The short-hand \w includes \d. Use a character class instead:
[a-z]{3}\d[a-z]{2}

In Python:
import re

passwords = ['aaa5aa', 'adsfgg', 'e42dsf425']

regex = r'[a-z]{3}\d[a-z]{2}'

for p in passwords:
    if re.search(regex, p):
        print(p)

See a demo on ideone.com.

\w is a short-hand for [a-zA-Z0-9_], whereas \d is [0-9].
Answer: That is because \w also matches \d
You can match word characters without matching a digit:
[^\W\d]{3}\d[^\W\d]{2}

Regex demo
Note that re.findall returns a list with all matches. The if statement here if re.findall(regex, p): checks if the list has a result.
If you want to match the whole string, you can use re.match.
import re

passwords = ['aaa5aa', 'adsfgg', 'e42dsf425']

regex = r'[^\W\d]{3}\d[^\W\d]{2}'

for p in passwords:
    if re.findall(regex, p):
        print(p)

Output
aaa5aa

If you want to match exactly 6 chars with a digit at the fourth position, you can use anchors:
^\w{3}\d\w{2}$

Regex demo


How to assign values to the elements of an array?

Question: I don't know what's happening. It shows me that abc is as I expected, but when I try to assigns some values of abc to pass, it shows meenter code here an unexpected "4" to the output.
#include <stdio.h>

int main(void)
{
    char abc[52];
    for (int i = 0; i < 52; i++)
    {
        if (i < 26)
        {
            abc[i] = 'a' + i;
        }
        else
        {
            abc[i] = 'A' + i - 26;
        }
    }
    printf("%s\n", abc);
    char pass[2];
    pass[0] = abc[3];
    printf("%s\n", pass);
    pass[1] = abc[1];
    printf("%s\n", pass);
}

I receive:
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
d
db4
Instead of:
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
d
db
Comment: `%s` means string and C strings are null-terminated. `pass` must be of size 3, with the last element being 0. Otherwise, you'll get UB.
Answer: Strings represented as character arrays, need to be null terminated. This includes your abc and pass arrays.  printf doesn't know anything about the size of your array when abc is passed to it. It just prints characters starting at that address until it hits a \0 char.  Ditto for pass
If  abc was printing correctly without a null char appended to it, that that's probably just you getting lucky.  It's technically undefined behavior.
Let's fix your code up....
int main(void)
{
    char abc[52+1]; // 53
    for (int i = 0; i < 52; i++)
    {
        if (i < 26)
        {
            abc[i] = 'a' + i;
        }
        else
        {
            abc[i] = 'A' + i - 26;
        }
    }
    abc[52] = '\0';
    printf("%s\n", abc);

    char pass[2+1]; //3
    pass[0] = abc[3];
    pass[1] = '\0'; // null terminate pass before printing
    printf("%s\n", pass);

    pass[1] = abc[1];
    pass[2] = '\0'; // null terminate pass again before printing
    printf("%s\n", pass);
}

Comment: It looks like a newbie error, it worked. Thank you :)
Comment: @selbie I guess adding `abc[52+1]` is not really needed. You loop starting from `0` to less than `52`(0-51) and set the `abc[52] = '\0'.`
Comment: @JAMESBRYANB.Juventud - not correct.  An array of size `52` has valid indices ranging from `0-51`, which the OP's `for` loop fills in with characters.  Without an increase in the array capacity, there would be no place to append the null char at the end.  Now do you understand why the `+1` is needed?


If the sum and the product of two sets are equal are the sets?

Question: I have the following lemma that I can't seem to disprove or prove but which I am assuming someone here will make short work of.  I will first write a simple case and then a more general case.
Simple case:
Let $\mathbb{F}_q$ be a finite field of prime order $q$.
$\forall a, b, c, d \in \mathbb{F}_q$ where $a$ and $b$ are non zero
if $a + b = c + d$ and $a*b=c*d$
then $(a=c \land b=d) \lor (a=d \land b=c)$
General case:
Let $\mathbb{F}_q^n$ be a vector space of dimension $n$ over a finite field of prime order $q$.
$\forall a,b \in \mathbb{F}_q^n$, where all elements in $a$ are non zero
if $\sum_{i=1}^n a_i = \sum_{i=1}^n b_i$ and $\prod_{i=1}^n a_i = \prod_{i=1}^n b_i$
then there exists a permutation matrix $M$ such that $aM=b$.
Thanks in advance,
Comment: In $\mathbb F_{11}$, $1+4+8+9=2+3+7+10$ and $1\cdot4\cdot8\cdot9=2\cdot3\cdot7\cdot10$.
Comment: If $\mathbb F$ is an $m$-element field, then there are $\binom{m-1}3$ sets $\{a_1,a_2,a_3\}$ of three distinct nonzero elements, while there are only $m(m-1)$ possible pairs $(a_1+a_2+a_3,a_1a_2a_3)$.
Comment: "Are $n$ nonzero numbers determined by their sum and product?" It doesn't seem too likely that $n$ numbers could be determined by only $2$, does it?
Comment: Little tip [irrelevant to the question]: the tags should be correct. So i assume it should be "abstract-algebra"
Comment: The first one is true.  Write $a+b=x=c+d,$ then substitute $b=x-a,d=x-c$ into $ab=cd,$ and forge ahead.
Comment: The title asks "does $a+b=ab$ imply $a=b$. But the body asks whether a set is determined by its sum and its product.
Comment: @bof the question in the body is what I meant.  I didn't now how to concisely phrase it in the title.  I mean $n$ elements from $\mathbb{F}_q$
Comment: In $\mathbb F_{11}$, $1+9+10=2+3+4$ and $1\cdot9\cdot10=2\cdot3\cdot4$.
Comment: The question has now been answered in the comments. Could @saulspatz or bof write it up as a real answer to get the question off the unanswered list?
Comment: @bof, please see my comment above (for some stupid reason it is not possible to notify two users in the same comment)
Answer: It's true for two elements in any field.  We have $a+b=x=c+d,$ so
$b=x-a,\ d=x-c.$  Then $$ab=cd \iff a(x-a)=c(x-c)\iff x(a-c)=(a-c)(a+c)$$  Either $a-c = 0,$ and then $b=d,$ or $$x=a+c\iff a+b=a+c\iff b=c.$$
As @bof has indicated in a comment, the statement is false for $n>2.$


concurrency writings into memcache

Question: Could you help me with my issue. I have some slow mysql queries, so I'm caching its results into memcache.
I do something like this:
<code>$data = get_from_cache();
if (! $data) {
   $data = get_from_mysql();
   set_cache($data);
}
</code>
Problem.
Sometimes I have about 10 requests per second. So, when my cache is expiring, I have 5-10 get-requests which are initiating this slow mysql query at the same time.
Could you recommend a pattern for me to make a mutex in php, or something like this, to do only one slow request to mysql.
Answer: This is known as the dog pile problem. Strategy: Break Up The Memcache Dog Pile  describes two approaches:
Solution 1:

Set the cache item expire time way out in the future.
Embed the "real" timeout serialized with the value. For example, set the item to timeout in 24 hours, but the embedded timeout might be five minutes in the future.
On a get from the cache determine if the stale timeout expired and on expiry immediately set a time in the future and re-store the data as is. This closes down the window of risk.
Fetch data from the DB and update the cache with the latest value.

Solution 2:

Create two keys in memcached: MAIN key with expiration time a bit higher than normal + a STALE key which expires earlier.
On a get read STALE key too. If the stale has expired, re-calculate and set the stale key again.

concurrency writings into memcache


What is the best way to create a form in asp.net mvc 5 for conditional inputs. Using the divs and render partial or Ajax

Question: I have a form in which I want the user to select using the radio button if its a company or individual user. Based on the selection I want to show and hide certain input fields like BirthDate, Gender etc (which is applicable for an individual user but not a company). I have tried different approach but each one has some or the other problem.
Using Html.Partial in the View

        @Html.Partial("Company")

        @Html.Partial("Individual")

The radio button change event:
$("[name=IsIndividual]").on('change', function(){
                var $radio = $(this);

                var $individual = $radio.val();
                if ($individual == "True") {
                    $('.companyDiv').hide();
                   // $('.companyDiv').prop('disabled', true);
                    $('.individualDiv').show();
                  //  $('.individualDiv').prop('disabled', false);

                }
                else {
                    $('.individualDiv').hide();
                  //  $('.individualDiv').prop('disabled', true);
                    $('.companyDiv').show();
                 //   $('.companyDiv').prop('disabled', false);

                }
            });

Using this approach everything loads fine in the View but when I post it the fields which are present in both the partials become null in the controller. The reason to have the same fields in both partial is because I want to use same field in different way. Example - I have a firstName field which I use for first name in case of individual user but in case of company I want to use it to capture name with different labels and validations in both cases.
Using Ajax approach
Main Page

    $(function(){
        $("[name=IsIndividual]").on('change', function(){
            var $radio = $(this);
            $.ajax({
                url:'@Url.Action("GetType", "Customers")',
                data: { IsIndividual: $radio.val() },
                type: 'GET',
                success: function(data){
                    $("#loadpartial").html(data);
                }
            });

        });
    });

Controller
    [HttpGet]
    public ActionResult GetType(bool IsIndividual)
    {
        if (IsIndividual)
        {
            return PartialView("Individual");
        }
        else
        {
            return PartialView("Company");
        }
    }

Partial
@section Scripts {
    @Scripts.Render("~/bundles/jqueryval")
    @Scripts.Render("~/plugins/select2")
    @Scripts.Render("~/plugins/dataPicker")

}

But in this case the controls like datepicker doesnt work. I am not sure what I am doing wrong here since I tried moving the script from main page to the partial.
Also since I am very new to this I am not sure what is the best practice to achieve this. Either of the 2 approach or something else altogether.
Answer: When using partial views, the issue is most likely that the element 'name' attributes are not being set with the correct prefix. Check the rendered html to see if the names being rendered from the partial are correct according to elements that are not part of the partial. If they don't match what the model binder expects then they will be null in the controller.
For the Ajax approach, the issue is that you are dynamically loading your elements so they are added to the DOM after the datepicker function has been called so it is not applied to them. You should call the datepicker function in the success function of your Ajax call rather than the document.ready function. The datepicker should then work.
And to actually answer your question, in this particular situation, I would recommend the ajax route as you will only be loading the content the user selects. But either approach is fine.
Comment: Thanks for your reply. I did what you suggested for Ajax and it worked fine. But when I post my form and if it fails in controller because of Invalid model state then the loaded view doesnt have the partial. I understand that was loaded by Ajax but is there a way that in case of error on the server I can have my same view with data and errors retained?
Comment: You would need to return the view with the model from your action and have some logic based on the user's selection (which would need to be part of the model) that automatically loads the appropriate bit on document.ready, or as a partial.


DB design and optimization considerations for a social application

Question: The usual case. I have a simple app that will allow people to upload photos and follow other people. As a result, every user will have something like a "wall" or an "activity feed" where he or she sees the latest photos uploaded from his/her friends (people he or she follows).
Most of the functionalities are easy to implement. However, when it comes to this history activity feed, things can easily turn into a mess because of pure performance reasons.
I have come to the following dilemma here:
i can easily design the activity feed as a normalized part of the database, which will save me writing cycles, but will enormously increase the complexity when selecting those results for each user (for each photo uploaded within a certain time period, select a certain number, whose uploaders I am following  / for each person I follow, select his photos )
An optimization option could be the introduction of a series of threshold constraints which, for instance would allow me to order the people I follow on the basis of the date of their last upload, even exclude some, to save cycles, and for each user, select only the 5 (for example) last uploaded photos.
The second approach is to introduce a completely denormalized schema for the activity feed, in which every row represents a notification for one of my followers. This means that every time I upload a photo, the DB will put n rows in this "drop bucket", n meaning the number of people I follow, i.e. lots of writing cycles. If I have such a table, though, I could easily apply some optimization techniques such as clever indexing, as well as pruning entries older than a certain period of time (queue).
Yet, a third approach that comes to mind, is even a less denormalized schema where the server side application will take some part of the complexity off the DB. I saw that some social apps such as friendfeed, heavily rely on the storage of serialized objects such as JSON objects in the DB.
I am definitely still mastering the skill of scalable DB design, so I am sure that there are  many things I've missed, or still to learn. I would highly appreciate it if someone could give me at least a light in the right direction.
Comment: I'm not that good with DB optimization, but it seems you probably can need a NonSQL DB (have a look at Cassandra and MongoDB). Actually, they seems to be really faster exactly in that kind of situation. (just think of facebook, using cassandra as main dbms :D)
Comment: Is there a good book on the topic of designing scalable DBs (for social apps and MMOs) ?
Comment: just curious to know, besides scalable DB and retrieval of data, is there any other issue u r experiencing while designing such kind of application..which framework, language you chose to develop such kind of this application...actually I had asked a similar type of question:- http://stackoverflow.com/questions/4842276/designing-a-social-networking-website-db-and-front-end
Answer: If your application is successful, then it's a good bet that you'll have more reads than writes - I only upload a photo once (write), but each of my friends reads it whenever they refresh their feed. Therefore you should optimize for fast reads, not fast writes, which points in the direction of a denormalized schema.
The problem here is that the amount of data you create could quickly get out of hand if you have a large number of users. Very large tables are hard on the db to query, so again there's a potential performance issue. (There's also the question of having enough storage, but that's much more easily solved).
If, as you suggest, you can delete rows after a certain amount of time, then this could be a good solution. You can reduce that amount of time (up to a point) as you grow and run into performance issues.
Regarding storing serialized objects, it's a good option if these objects are immutable (you won't change them after writing) and you don't need to index them or query on them. Note that if you denormalize your data, it probably means that you have a single table for the activity feed. In that case I see little gain in storing blobs.
If you're going the serialized objects way, consider using some NoSQL solution, such as CouchDB - they're better optimized for handling that kind of data, so in principle you should get better performance for the same hardware setup.
Note that I'm not suggesting that you move all your data to NoSQL - only for that part where it's a better solution.
Finally, a word of caution, spoken from experience: building an application that can scale is hard and takes time better spent elsewhere. You should spend your times worrying about how to get millions of users to your app before you worry about how you're going to serve those millions - the first is the more difficult problem. When you get to the point that you're hugely successful, you can re-architect and rebuild your application.
Comment: +1 for get millions of users then worry about scaling, rather than building a scalable app that few people use.
Answer: There are many options you can take

Add more hardware, Memory, CPU -- Enter cloud hosting
Hows 24GB of memory sound? Most of your importantly accessed DB information can fit just in memory.
Choose a host with expandable SSDs.
Use an events based system in your application to write the "history" of all users. So it will be like so: id, user_id, event_name, date, event_parameters' -- an example would be: 1, 8, CHANGED_PROFILE_PICTURE, 26-03-2011 12:34, <id of picture> and most important of all, this table will be in memory. No longer need to worry about write performance. After the records go past i.e. 3 days they can be purged into another table (in non-memory) and included into the query results, if the user chooses to go back that far. By having all this in one table you remove having to do multiple queries and SELECTs to build up this information.
Consider using INNODB for the history/feeds table.

Good Resources to read

Exploring the software behind Facebook, the world's largest site
Digg: 4000% Performance Increase by Sorting in PHP Rather than MySQL
Caching & Performance: Lessons from Facebook
Answer: These kind of problems are why currently NOSql solutions used these days. What I did in my previos projecs is really simple. I don't keep user->wall user->history which contains purely feed'ids in memory stores(my favorite is redis). so in every insert I do 1 insert operation on database and (n*read optimization) insert operation in memory store. I design memory store to optimize my reads. if I want to filter user history (or wall) for videos I put a push feedid to a list like user::{userid}::wall::videos.
Well ofcourse you can purely build the system in memstores aswell but its nice to have 2 systems doing what they are doing the best.
edit :
checkout these applications to get an idea:
http://retwis.antirez.com/
http://twissandra.com/
Answer: I would probably start with using a normalized schema so that you can write quickly and compactly. Then use non transactional (no locking) reads to pull the information back out making sure to use a cursor so that you can process the results as they're coming back as opposed to waiting for the entire result set.  Since it doesn't sound like the information has any particular critical implications you don't really need to worry about a lock of the concerns that would normally push you away from transactional reads.
Answer: I'm reading more and more about NoSQL solutions and people suggesting them, however no one ever mentions drawbacks of such choice.
Most obvious for me is lack of transactions - imagine if you lost a few records every now and then (there are cases reporting this happens often).
But, what I'm surprised with is that no one mentions MySQL being used as NoSQL - here's a link for some reading.
In the end, no matter what solution you choose (relational database or NoSQL storage), they scale in similar manner - by sharding data across network (naturally, there are more choices but this is the most obvious one). Since NoSQL does less work (no SQL layer so CPU cycles aren't wasted on interpreting SQL), it's faster, but it can hit the roof too.
As Elad already pointed out - building an app that's scalable from the get go is a painful process. It's better that you spend time focusing on making it popular and then scale it out.


Why does the lights disapear when ever I make lights near each other unity (spotlight)

Question: I have been making a game in unity and I cant put street lamps because if I put 10 or so near each other  half of them stop making light, How can I fix this.
Thanks!
(project made in URP)
Comment: Number of lights allowed per pixel is a setting, from memory it's either in preferences > quality settings or in the lighting settings. BTW 10 lights near each other is a lot and will affect performance significantly. It's best to try to do more with less - play around with the light settings try try to achieve the same effect with less lights. Consider also area lighting, baked lighting and emissive materials.
Comment: i cant make light lamps for roads :(
Comment: it wont let me cuz they disappear
Comment: @Absinthe there is no quality or lighting in performanc
Comment: OH i opened preferences
Comment: anyway is it possible to change the thing and make light visable?
Comment: it sudently changed the thing now i cant add more than:
Comment: https://drive.google.com/file/d/1wxeBA6mSI3Rdb1pHGwJF6gzf-7-wEnKn/view?usp=sharing the img no need to download wait for load
Comment: Rather than spam comments please update your post to show what you've tried. Also, no one is going to click a random Google Drive link :) Either use an image link or explain in words.
Answer: You have to go to Edit -> Project Settings -> Quality -> Pixel Light Count and you'll see the number 4 by default. Write there the number of light that you want to see.


Query to check number of records created in a month.

Question: My table creates a new record with timestamp daily when an integration is successful. I am trying to create a query that would check (preferably automated) the number of days in a month vs number of records in the table within a time frame.
For example, January has 31 days, so i would like to know how many days in january my process was not successful. If the number of records is less than 31, than i know the job failed 31 - x times.
I tried the following but was not getting very far:
SELECT COUNT (DISTINCT CompleteDate)
FROM table
WHERE CompleteDate BETWEEN '01/01/2015' AND '01/31/2015'

Every 7 days the system executes the job twice, so i get two records on the same day, but i am trying to determine the number of days that nothing happened (failures), so i assume some truncation of the date field is needed?!
Comment: Which version of SQL Server are you using?
Answer: One way to do this is to use a calendar/date table as the main source of dates in the range and left join with that and count the number of null values.
In absence of a proper date table you can generate a range of dates using a number sequence like the one found in the master..spt_values table:
select count(*) failed
from (
     select dateadd(day, number, '2015-01-01') date
     from master..spt_values where type='P' and number < 365
) a
left join your_table b on a.date = b.CompleteDate
where b.CompleteDate is null
and a.date BETWEEN '01/01/2015' AND '01/31/2015'

Sample SQL Fiddle (with count grouped by month)
Comment: This is my prefered solution over the other two because it is closer to what I would do and because it is more flexible (you can easily find which distinct days failed and you can control the time span without being limited to one month).
Answer: You can use the following query:
SELECT DATEDIFF(day, t.d, dateadd(month, 1, t.d)) - COUNT(DISTINCT CompleteDate)
FROM mytable
CROSS APPLY (SELECT CAST(YEAR(CompleteDate) AS VARCHAR(4)) +
                    RIGHT('0' + CAST(MONTH(CompleteDate) AS VARCHAR(2)), 2) +
                    '01') t(d)
GROUP BY t.d

SQL Fiddle Demo
Explanation:

The value CROSS APPLY-ied, i.e. t.d, is the ANSI string of the first day of the month of CompleteDate, e.g. '20150101' for 12/01/2015, or 18/01/2015.
DATEDIFF uses the above mentioned value, i.e. t.d, in order to calculate the number of days of the month that CompleteDate belongs to.
GROUP BY essentially groups by (Year, Month), hence COUNT(DISTINCT CompleteDate) returns the number of distinct records per month.
The values returned by the query are the differences of [2] - 1, i.e. the number of failures per month, for each (Year, Month) of your initial data.

If you want to query a specific Year, Month then just simply add a WHERE clause to the above:
WHERE YEAR(CompleteDate) = 2015 AND MONTH(CompleteDate) = 1
Answer: Assuming you have an Integers table*. This query will pull all dates where no record is found in the target table:
declare @StartDate datetime = '01/01/2013',
        @EndDate datetime = '12/31/2013'

;with d as  (
    select *, date = dateadd(d, i - 1 , @StartDate)
    from dbo.Integers
    where i <= datediff(d, @StartDate, @EndDate) + 1
)
select d.date
from d
where not exists (
    select 1 from <target> t
    where DATEADD(dd, DATEDIFF(dd, 0, t.<timestamp>), 0) = DATEADD(dd, DATEDIFF(dd, 0, d.date), 0)
)

Answer: Between is not safe here
SELECT 31 - count(distinct(convert(date, CompleteDate)))
FROM table
WHERE CompleteDate >= '01/01/2015' AND CompleteDate < '02/01/2015'


Are lmer-coefficients standardized or not?

Question: I have a very basic question; maybe a bit too basic to find a helpful response googeling it.
I am calcultating multi-level-models using the lmer function using this code:
lmer(H1_rirs, data= df_long_cl, REML = T)

Am I right in assuming that the retrieved coefficients are unstandardized? If yes, is there an easy way to standardize them?
Best,
Carolin
Comment: How is `H1_rirs` defined?
Comment: Hi Roland, it is defined like this: `H1_rirs<-commresc ~ 1 + cvar1c + (1 + cvar1c | ID_new) `    both variables are person-mean-centred, although I guess I should not mean-centre my outcome (it functions as a mediator in the bigger model)
Answer: Yes, by default, no standardizing is applied.
If you like to get standardized coefficients, one way would be to standardize the data before fitting your model. There is a robust implementation of such a function in the effectsize-package. Or you can do some post-hoc standardization (also effectsize-package). The latter yields different results, the most accurate would be standardizing the data before model fitting.
Answer: Adding to what @daniel suggested with the effectsize package, there is also the "pseudo-standardized" coefficient (Hoffman, 2015) where the response and the predictor are standardized based on the level of prediction:
library(lme4)

m <- lmer(Reaction ~ Days + (Days | Subject), data = sleepstudy)

effectsize::standardize_parameters(m, method = "pseudo")
#> # Standardization method: pseudo
#>
#> Parameter   | Coefficient (std.) |       95% CI
#> -----------------------------------------------
#> (Intercept) |               0.00 | [0.00, 0.00]
#> Days        |               0.68 | [0.48, 0.88]

Created on 2021-06-07 by the reprex package (v2.0.0)


how to find and install PyVisa with AnacondaNavigator for python3.5 on W10-64b?

Question: The Anaconda Navigator documentation doesnt tell how to do it. Earlier i have used python2.7 where it was easy to install PyVisa1.4, but at the moment I am screwed with Anaconda.
I installed Python3.5 together with Anaconda4.1-64bit, and the documentation says that the Navigator is used to find and install packages but I used several hours to find PyVisa1.8 and still dont know.
Comment: The following solves my PyVisa issue (it is amazing that somebody developed this nice AnacondaNavigator GUI just to make people waste their time with it! The only way to do it is obviously with a CMD terminal and the GUI is totally meaningless): In Anaconda Navigator> Environments, open root> Open Terminal> conda install pyvisa> didnt find anything. anaconda search -t conda pyvisa> finds 9 packages. anaconda show conda-forge/pyvisa> it suggests what to do. conda install --channel https://conda.anaconda.org/conda-forge pyvisa> now it installs PyVisa1.8 and it works.
Answer: I struggled with this issue aswell. Finally i solved it with another work around.. I opend a terminal over the anaconda navigator in the environment where i want the package. Then i used pip. Just type in: pip install pyvisa
Answer: For some reason pyvisa does not seem to be accessible from Anaconda Navigator. To install it you need to open your environment in the terminal:

Then issue this command to install pyvisa from the official Conda-Forge Package Library:
conda install -c conda-forge pyvisa


(547) - Conflito de Foreign Key - Como resolver?

Question: Conflito com uma chave estrangeira existente na tabela GAM_USER - Tentando add um user novo, mas já existe um usuário na mesma tabela que possui o mesmo cpf, sendo que o anterior eu não posso excluinserir a descrição da imagem aquiir. Tenho que manter o novo user. Algúem sabe como posso resolver este issue?
Banco SQL server 2012
aplicação web em .NET
servidor de aplicação IIS
Crystal Report
.Net FWinserir a descrição da imagem aqui
Comment: Um user com o mesmo cpf ... não seria o caso de update no já existente ?
Answer: Se o usuário já existe, o correto seria alterar esse registro, e não tentar adicionar o mesmo registro novamente.


How to remove rows with NAs in a specific column but only if they have a duplicate in another column?

Question: So I have this data frame, in which some rows are duplicated in the "Species" column, and some aren't.
There are NAs in all types of rows, both duplicated and not duplicated.
Species            |  A  |  B  |
--------------------------------
Tilapia guineensis |  1  |  10 |
Tilapia guineensis |  1  |  NA |
Tilapia zillii     |  3  |  23 |
Tilapia zillii     |  3  |  NA |
Eutrigla gurnardus | 18  |  4  |
Caramila artida    |  9  |  NA |
Sprattus sprattus  |  7  |  6  |
Spalili burcant    | 11  |  NA |

I want to remove that rows that have NAs in the B column, but only if they belong to a duplicate row.
My output would be something like this:
Species            |  A  |  B  |
--------------------------------
Tilapia guineensis |  1  |  10 |
Tilapia zillii     |  3  |  23 |
Eutrigla gurnardus | 18  |  4  |
Caramila artida    |  9  |  NA |
Sprattus sprattus  |  7  |  6  |
Spalili burcant    | 11  |  NA |

Basically, if a row is duplicated in the Species column and has a NA in the B column, I want to remove that row with the NA. However, if a row is unique in the Species column, I want to keep it even if it has NA.
Sorry if I'm being confusing, thanks in advance.
Reproducible data format:
df <- read.csv(text="
Species,A,B
Tilapia guineensis,1,10
Tilapia guineensis,1,NA
Tilapia zillii,3,23
Tilapia zillii,3,NA
Eutrigla gurnardus,18,4
Caramila artida,9,NA
Sprattus sprattus,7,6
Spalili burcant,11,NA")

Answer: I think you can avoid using any grouping logic, and get the result in a single pass:
df[!(duplicated(df$Species) & is.na(df$B)),]
#             Species  A  B
#1 Tilapia guineensis  1 10
#3     Tilapia zillii  3 23
#5 Eutrigla gurnardus 18  4
#6    Caramila artida  9 NA
#7  Sprattus sprattus  7  6
#8    Spalili burcant 11 NA

Answer: You can select a row if there is only 1 row for the Species or if it is not an NA value.
library(dplyr)
df %>% group_by(Species) %>% filter(n() == 1 | !is.na(B))

#  Species                A     B
#  <chr>              <int> <int>
#1 Tilapia guineensis     1    10
#2 Tilapia zillii         3    23
#3 Eutrigla gurnardus    18     4
#4 Caramila artida        9    NA
#5 Sprattus sprattus      7     6
#6 Spalili burcant       11    NA

You can also write the same logic in base R and data.table :
#Base R
subset(df, ave(!is.na(B), Species, FUN = function(x) length(x) == 1 | x))

#data.table
library(data.table)
setDT(df)[, .SD[.N == 1 | !is.na(B)], Species]

data
df <- structure(list(Species = c("Tilapia guineensis", "Tilapia guineensis",
"Tilapia zillii", "Tilapia zillii", "Eutrigla gurnardus", "Caramila artida",
"Sprattus sprattus", "Spalili burcant"), A = c(1L, 1L, 3L, 3L,
18L, 9L, 7L, 11L), B = c(10L, NA, 23L, NA, 4L, NA, 6L, NA)), row.names = c(NA,
-8L), class = "data.frame")


How to rebuild solution using NAnt?

Question: I am using following command in NAnt build file
<msbuild project="${appsdest}\${targetname}\${targetname}.sln"/>

appsdest and targetname are variable contain application project path and targetname is the variable contain project name to be rebuild.
but when using this command it build the solution but the time stamp of dlls are not updated.
I need to rebuild the solution and the latest dlls need to be place in debug folder.
can anyone tell me how to rebuild the solution using the msbuild command as written above?
Thanks
Answer: I don't use the MSBUILD tasks in NAnt, I just call the msbuild executable directly and it has worked well for me.
<target name="build">
  <exec program="${MSBuildPath}">
    <arg line='"${SolutionFile}"' />
    <arg line="/property:Configuration=${SolutionConfiguration}" />
    <arg value="/target:Rebuild" />
    <arg value="/verbosity:normal" />
    <arg value="/nologo" />
    <arg line='/logger:"C:\Program Files\CruiseControl.NET\server\ThoughtWorks.CruiseControl.MsBuild.dll"'/>
  </exec>
</target>

I write more about it here: http://enterpriseyness.com/2009/12/continuous-integration-with-cruise-control-net-nant
Answer: You need to use the 'Rebuild' target:
<msbuild project="${appsdest}\${targetname}\${targetname}.sln" target="Rebuild"/>

How to rebuild solution using NAnt?


High entropy random data creating functions?

Question: Are there functions which produce "infinite" amounts of high entropy data? Moreover, do functions exist which produce the same random data (sequentially) time after time?
I kind of know that they exist, but do they have a specific name?
Use case examples:

Using the function to generate 100 bits of random data. (Great!) But while maintaining high values of entropy.
Using the same function to generate 10000 bits of random data. (The first 100 bits generated are the same as the 100 bits of random data generated before). And while still maintaining high values of entropy

Further, how would I go about building these functions myself?
Comment: No deterministic function can produce output with more entropy than the input.
Comment: But perhaps what you want is a streamcipher. Given abot 150 bits of entropy they can output practically unlimited amounts of data that's computationally indistinguishable from truly random data.
Answer: You are most likely looking for Pseudo-Random Number Generators.

They are initialized by a seed, thus taking in a finite amount of entropy.
Good generators have a decent entropy coming out, supposing you judge it only from its output (thus you ignore the seed and/or the algorithm to generate the numbers, otherwise the entropy is obviously 0).

Most PRNG algorithms produce sequences which are uniformly distributed by any of several tests. It is an open question, and one central to the theory and practice of cryptography, whether there is any way to distinguish the output of a high-quality PRNG from a truly random sequence without knowing the algorithm(s) used and the state with which it was initialized.

All PRNGs have a period, after which a generated sequence will restart.

The period of a PRNG is defined thus: the maximum, over all starting states, of the length of the repetition-free prefix of the sequence. The period is bounded by the number of the states, usually measured in bits. However, since the length of the period potentially doubles with each bit of "state" added, it is easy to build PRNGs with periods long enough for many practical applications.

Thus, to have two sequences of different lengths where one is the prefix of the other, you just have to run a PRNG with the same seed both times.
Building them yourself would be pretty tricky, but a rather good and simple one is the Mersenne Twister, which dates back to only 1998 and defined in a paper by Matsumoto and Nishimura [1].
A trivial example would be a linear congruential generator.
[1] Matsumoto, M.; Nishimura, T. (1998). "Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator". ACM Transactions on Modeling and Computer Simulation 8 (1): 3–30. doi:10.1145/272991.272995.
Comment: This is exactly what I was looking for. Although I am not concerned about the cryptographic qualities of the Pseudo-Random Number Generators - having reproducible "high entropic" data functions fits my quest perfectly. Time to change the world!


Oracle SQL Errors

Question: I am very new to SQL and my class is using Oracle cloud as of now. I have created tables and I am trying to insert data into it. I keep receiving the error ORA-02291: integrity constraint violated when I try to insert data into my semester table. Any help would be extremely appreciated. Thank you!
CREATE TABLE SCHOOL (
  SchoolID              char(10)        NOT NULL,
  SchoolName            char(50)        NOT NULL,
  SchoolAddress         char(50)        NOT NULL,
  SchoolCity            char(50)        NOT NULL,
  SchoolState           char(2)         NOT NULL,
  SchoolZIP             char(30)        NOT NULL,
  SchoolPhone           char(25)        NULL,
  SchoolWebsite         char(50)        NOT NULL,
    CONSTRAINT SCHOOL_PK PRIMARY KEY(SchoolID)
);

CREATE TABLE SEMESTER(
SemesterName            char(20)        NOT NULL,
Year                char(4)         NOT NULL,
SchoolID            char(50)        NOT NULL,
    CONSTRAINT SEMESTER_PK PRIMARY KEY(SemesterName),
        CONSTRAINT SEM_SCH_FK FOREIGN KEY(SchoolID)
            REFERENCES SCHOOL(SchoolID)
);

/****************************** School ******************************************/
INSERT INTO SCHOOL(SchoolID, SchoolName, SchoolAddress, SchoolCity, SchoolState, SchoolZip, SchoolPhone, SchoolWebsite)
    VALUES(
        'NIU', 'Northern Illinois University', '1425 Lincoln Hwy', 'Dekalb', 'IL', '60115', '815-753-0446', 'https://myniu.niu.edu/');
SELECT * FROM SCHOOL;

/****************************** Semester ****************************************/

INSERT INTO SEMESTER(SemesterName, Year, SchoolID)
    VALUES(
        'Spring', '2020', 'NIU');
INSERT INTO SEMESTER VALUES(
        'Fall', '2020', 'NIU');
SELECT * FROM SEMESTER;

Comment: What is the actual error message? It should be something like _"integrity constraint  violated – parent key not found"_
Comment: Also, your foreign key data-types should match. At the moment, `SCHOOL.SchoolID` is `char(10)` and `SEMESTER.SchoolID` is `char(50)`
Comment: Hi, the error I am receiving is ORA-02291: integrity constraint (ADMIN.SEM_SCH_FK) violated - parent key not found.
Answer: Check if there are trailing spaces in the SCHOOLID column
select schoolid, length(schoolid) from school;

You would most likely see length as 10 instead of 3 :
|SCHOOLID   |LENGTH(SCHOOLID) |
|NIU        |10               |

A good explanation is below ( quoting Tom Kyte here )

A varchar2 datatype, when stored in a database table, uses only the space allocated to it. If you have a varchar2(1999) and put 50 bytes in the table, we will use 52 bytes (leading length byte).
A char datatype, when stored in a database table, always uses the maximum length and is blank padded. If you have char(1999) and put 50 bytes into it, it will consume 2001 bytes (leading length field is present on char's as well).
In the database -- a CHAR is a VARCHAR that is blank padded to its maximum length.
There are great differences between a char(1999) and varchar2(1999) as the CHAR is always 1999 characters long physically AND logically. the varchar is physically 1999 bytes but logically can be any size in between 0 and 1999. The char is always blank padded, not so for the varchar.

See if the error persists when you define columns as varchar2, as follows ..

CREATE TABLE SCHOOL (
  SchoolID             varchar2(10)        NOT NULL,
  SchoolName           varchar2(50)        NOT NULL,
  SchoolAddress        varchar2(50)        NOT NULL,
  SchoolCity           varchar2(50)        NOT NULL,
  SchoolState          varchar2(2)         NOT NULL,
  SchoolZIP            varchar2(30)        NOT NULL,
  SchoolPhone          varchar2(25)        NULL,
  SchoolWebsite        varchar2(50)        NOT NULL,
    CONSTRAINT SCHOOL_PK PRIMARY KEY(SchoolID)
);

CREATE TABLE SEMESTER(
SemesterName            varchar(20)        NOT NULL,
Year                varchar(4)         NOT NULL,
SchoolID            varchar(50)        NOT NULL,
    CONSTRAINT SEMESTER_PK PRIMARY KEY(SemesterName),
        CONSTRAINT SEM_SCH_FK FOREIGN KEY(SchoolID)
            REFERENCES SCHOOL(SchoolID)
);

Comment: Or replace the data type `varchar2(50)` with `constraint see_sch_fk references school(schoolid)` and let it inherit the type :) But I agree, the problem is `char`.

Oracle SQL Errors