Figure out dependencies of a commit we try to cherry pick

Question: Assume something like the following:
                            HEAD/master
                             |
A<--B<--C<--D<--E<--F<--G<--J
        ^
        official

Where official is a branch.
I wanted to cherry-pick 2 commits to official branch e.g.E and J
Both these commits were fixes affecting the same 3 files.
When I did git cherry-pick E it went fine but when I did git cherry-pick J I got some conflicts.
Looking at the diffs I realized that I needed to also cherry pick commit F which did a change in two of those 3 files which change was basically a method definition change and J was done on top of that.
So it was easy to fix by just doing git cherry-pick F && git cherry-pick J
Question:
If I wasn't aware of the changes done in those files and commit F was a big commit changing many files: Is there another way to figure out on which commit a commit we are trying to cherry pick depends on without manually doing a git log on the file and going commit by commit?
Answer:
Is there another way to figure out on which commit a commit we are trying to cherry pick depends on without manually doing a git log on the file and going commit by commit?

Yes there is - I wrote git-deps to automate precisely that task.  Since it currently accepts a range of commits to analyse, in the case of your example you would need to pass it a singleton range containing just commit J, and also tell it to exclude any dependencies which are already in official:
git deps --exclude-commits official J^!

Behind the scenes the tool will then look at the diff of J^ with J, and then run git blame on any lines which are removed or modified, plus a configurable number of lines of surrounding context.  This would result in it discovering that one or more of the lines J removes or modifies come from commit F, therefore it would output the SHA1 of commit F.
git-deps can not only discover dependencies of a single commit or range of commits, but also recurse a whole dependency tree.  Therefore the entire cherry-picking scenario you describe could be automated into a single command:
git deps --recurse --exclude-commits official J^! | \
    tsort | \
    tac | \
    xargs -t git cherry-pick

Please note that this only guarantees that the cherry-picks will all cleanly apply, not that they are semantically correct.  So the resulting branch head will still need to be carefully inspected and tested for correctness.
More convenient porting of commits between branches is the most obvious (but not the only) use case of git-deps, and for this reason I have dedicated a section of the README specifically to this use case.  It even includes a YouTube video showing you exactly how to use it.
If you would like more details, you may be interested in a couple of talks I have presented on this and other related tools:

a talk given to the Git London User Group
a talk given at an OpenStack developer conference

I also spoke about the tool in
episode #32 of the GitMinutes podcast.
Comment: Thank you for your answer! Do you know if there is a similar cherry picking dependency tracking solution but for Subversion?
Comment: Pretty sure Subversion simply isn't powerful enough to achieve this itself. You could use `git-svn` though.
Answer:
I got some [merge] conflicts ... Is there another way to figure out on which commit a commit we are trying to cherry pick depends on ...

Not really, no.

... without manually doing a git log on the file and going commit by commit?

Even this won't necessarily do it—at least, not without actually thinking about the problem.  Consider the following bit of C code:
void f(int x) {
    int i;

    /*
     * additional lines here, enough to mess with git context
     * since that only looks at +/- 3 lines around each patch
     * segment
     */

    for (i = 0; i < x; i++)
        do_something(i);
}

Suppose that one intermediate commit introduces an additional variable j, and then a later commit changes do_something(i) to do_something(j).  If you cherry-pick only the later commit, Git will happily change the line with no conflicts, but the code will no longer compile, as there is no int j in function f.
That said, it would be possible to write a tool that parses git show output for each intermediate commit, figures out which lines are changed (relative to the version in the work-tree, rather than using the actual numbers in each diff, since those may depend on a previous skipped diff), and points you to commits that modify conflicting regions.  But I don't know of any such tool.
Comment: So what is your workflow in cases like this? What have you found as most efficient?
Comment: @Jim: generally, a lot of inspection of commits—basically, what you ended up doing. Complex cherry-picks should be rare in part because the tools don't deal well with them, and the tools don't deal well with then in part because they're rare, which is sort of catch-22.


GIT - update repo from another repo

Question: Problem:
we have 2 repo's. working on repo A and once each X days (or when repo A is stable) we update repo B to be exactly like repo A (excluding some files from repo A)
Example:
2 repo's (each with its own git)
repoA
+-pom.xml
+-src

repoB
+-pom.xml
+-src

want to run a script that will once every X days take all commits from repoA that didnt affect pom.xml and apply them as 1 commit in repo B
eventually repoB src folder should be exactly the same as repoA src folder
thinking of taking the easy path and just using a bash script
mkdir /tmp/repoA
cp /repoA /tmp/repoA
rm /tmp/repoA/pom.xml
mv /tmp/repoA repoB
git commit -m "updated repo B"
git push

some1 has a better idea doing this using git and not just plain old mv ??
Answer: You could simply specify the working tree (of repoA) when updating repoB:
cd /path/to/repoB
git --work-tree=/path/to/repo/A add -A .
git checkout pom.xml
git commit -m "update repo B"

Comment: @Nimrod007 the work-tree is only set for the `git add` command.
Comment: nice! after doing this I want to continue working on repoB , how can i "unset" the --work-tree ?
Comment: dont u need to do: git reset HEAD pom.xml ??
Comment: @Nimrod007 no reset: checkout, in order to restore the content of pom.xml as in master branch. You want to update the working tree, not just the index: see http://stackoverflow.com/a/3639387/6309

GIT - update repo from another repo


Would the induced tension be approximately equal on an acoustic guitar for electric and acoustic strings of the same gauge?

Question: Consider a set of electric strings and a set of acoustic strings. If their gauge are the same, would the electric ones in general induce less tension than the acoustic ones?
Moreover, if the acoustic ones are nickel wound, say John Pearse nickel wound, would the answer to the above question change?
I tried to find out some relevant information online; you know why I ask here :). Thank you for your time.
Answer: As near as dammit. The difference will be the third (G) string. Generally on an electric set it's plain, but wound on an acoustic set. You probably wouldn't want the plain on an acoustic, although that's what my acoustics usually have - full electric sets. Slightly thinner, but that means less tension.
Comment: When I've gone looking for thicker gauged electric strings, I've actually had trouble finding a plain G string. I played 12s for a while in a band and I had to order those online because in stores there were no 12 sets with plain Gs in my area. So if one wanted to replicate the acoustic feel down to the wound G, it's definitely possible.
Comment: Yes, the individual strings may be differing tension, specifically the wound third, but overall, I think the perception of more or less tension will depend on the scale length than one wound string.
Comment: As noted in my answer, the medium to heavy electric sets often come with a wound G string and the comparison should be between like strings (obviously).  I posted a direct comparison of equal diameter wound G strings - one nickel wound electric vs. one bronze wound acoustic.  Slightly lighter tension with nickel windings due to lower mass (weight).  Inspired by your mention of the prevalence of plain steel G string for electric, I covered that in my answer as well.
Answer: The scale length will be a factor; for a given gauge, the longer the string, the more tension will be required to tune it to the same pitch. The difference between strings of the same gauge will be due to their weight or mass per unit length.
This gauge comparison is only valid for like strings -- that is, wound vs. wound, or unwound vs. unwound. A wound string of a given gauge would be very different from an unwound (solid) string of the same gauge.
Comment: Thanks for bringing the concept of scale length in :); didn't remember to specify it.
Comment: You are correct about mass being the key.  I explained this in great detail to prove that nickel wound has less mass therefore less tension than equal gauge bronze wound.
Comment: The scale length will not make much difference if we're talking about strings on the same guitar, surely?
Answer: The answer is that nickel wound strings (found on electric sets and John Pearse Nickle Wound Acoustic sets) of a given gauge (diameter) will have less overall mass than an equivalent gauge "Phosphor Bronze" or "80/20 Bronze" and therefore tune to a given pitch at slightly lower tension (assuming equal scale length).  See comparison chart at end of this answer.
Let's talk a little about electric string tension vs. acoustic string tension and the reason behind the prevalence of nickel wound strings on an electric guitar.
An acoustic guitar depends on the vibration of the strings to amplify the sound board and produce audible sound.  The heavier the gauge and tighter the tension, the more vibration the strings can produce to vibrate the top (soundboard) and stronger vibrations produce more volume than weak vibrations.  Technically what happens on an acoustic guitar is that the strings vibrate at a given frequency depending on tuning and fretting.  Those vibrating strings in turn vibrate the saddle they sit atop which sits atop the bridge which is fastened to the top.  In simplistic terms - the strings vibrate the saddle which vibrates the bridge and top which then acts in a manner similar to a speaker cone and produces the audible sound we hear at the frequency established by the strings.
Electric guitars rely on magnetic pickups to translate the vibrating string to an electrical signal which eventually feeds into an amplifier through the electronic circuitry -  which translates into a vibration of the speaker through the coil and magnet and we hear the sound emanating from the amplifier's speaker.  The electronics involved permit the volume to be controlled by volume knobs on both the guitar and the amplifier.
Since electric guitars rely on the amplifier to provide the volume, the gauge of string is less important to volume (just turn it up for lighter strings) and electric guitars can use extremely light strings.  The lighter strings allow for easier bending and that fact contributed to the evolution of modern electric guitar styles that use bending of notes to great effect.
Nickel strings are magnetic - meaning they respond well to the magnetic pickups in an electric guitar, whereas the windings on Phosphor Bronze or 80/20 Bronze acoustic strings do not respond well at all to magnetic pickups ("non-magnetic" metal).
But "bronze" wound strings tend to sound brighter on an acoustic and most players prefer the sound of "bronze" wound acoustic strings over "nickel" wound.  There are a few players who like the mellower sound of "nickel" wound on some acoustic guitars which is why John Pearse sells "Nickel Wound Acoustic" strings.
Most electric string sets being much lighter than common acoustic sets, will use a plain steel string on the G string.  The reason for this is that to get the light gauge needed to balance the tension in a light or super light electric set in the G string, the core around which the windings are wound would have to be so thin the string would constantly be breaking.  The tensile strength of the string is in the core itself and the windings provide only mass (no strength).
This is why on acoustic sets with a wound g string, the g string is the weakest string.  The core (where the tensile strength is) is thinner on a wound G string than on the high e string.

Wound string with core
But you can get electric sets with wound G strings.  They are usually found in what most string companies call a "medium" or "heavy" set.  In fact some string sets are sold with both a wound and plain steel G string so the player can use the one they prefer.
So that brings us back to the question which I answered in the opening paragraph.  But let me prove what I said.
The amount of tension required on a given string to cause it to vibrate at a given frequency is determined by the mass of the string and the scale length.  If we keep the scale length constant (same guitar) then the variable will be the mass.  The mass is a factor of the weight per x length of string - usually expressed as a unit weight in pounds per linear inch (lb/in).  The greater the mass (unit weight per linear x) the more tension required to produce a given pitch frequency.
Since the windings used in "nickel" wound strings are made of a material that is lighter in weight than that used in "bronze" wound strings, the mass (unit weight per linear inch) is less - and therefore requires less tension to tune to a given pitch.
Of course there are other factors which can affect the mass such as platings on the core, whether it is flat wound or round wound, etc.  But this discussion is limited to "nickel" wound vs. "bronze" wound of the same gauge and same scale length.
As an example of the difference in tension - below I compare two sets of almost the same gauge D'Addario strings using manufacture's reported tensions (charts from JustStrings.com).  One is an acoustic set of "12s" in "Phosphor Bronze"   and the other is an electric set of "12s" with "nickel" windings.  Both sets are "round wound".  The plain steel strings are identical in mass so there is no difference in tension. All of the wound strings except the low E are of equal diameter.  The comparison is not valid on the 6th strings due to a difference in diameter - but you can see how the other equal gauge strings compare.  Tension ratings shown are based on standard tuning.

Answer: Ever considered Martin Monel acoustic strings? They are nickel plated strings for acoustics and give you a different feel and more flexible response...
Comment: I think the asker mentioned John Pearse nickel wound strings which may be similar. Your answer may get converted to a comment by one of the mods.  Good comment though.  I'm not familiar with the Martin Monel but I may have to try them.  I have an acoustic with magnetic pickup and it responds much better to nickel wound.
Comment: Ok, in general these nickel wound acoustic string give a more "woody" tone to the acoustic guitar. I believe that in de '50 and '60 all string where nickel wound. They don't have that crispy ring to them that bronze does. Bluegrass players like Tony Rice tend to prefer them. They look funny though if youre used to bronze...
Comment: Yes they do look different on an acoustic.  They are making electric strings in many strange colours these days including red, black orange and chartreuse.


Show that $A$ is a nowhere dense subset of $X$.

Question:
Let $A$ be a subset of a metric space $(X,d)$ such that $X$ has no isolated points and each point of $A$ is an isolated point of $A$. Show that $A$ is a nowhere dense subset of $X$.

Attempt:
To show  $(\overline A)^{\mathrm{o}}=\emptyset$ .
Since each point of $A$ is an isolated point of $A$ so for each $a_i\in A$ we have $r_i>0$ such that $B(a_i,r_i)=\{a\}. $Surely $A$ is closed as it does not have any limit points.Hence $\overline A=A$.
Let $a\in A^\mathrm{o}\implies \exists r>0$ such that $B(a,r)\subset A$.
How to show that $A^\mathrm{o}=\emptyset $  from here??Any help.
Comment: A is not necessarily closed in X. As a counterexample, take $A=\{\frac{1}{n} | n \in \mathbb{N} \}$, and $X= \mathbb{R}$. The fact that each point of $A$ is an isolated point of $A$ implies there is, for every $a \in A$ an $r>0$ such that $B(a,r) \cap A = \{a\}$.
Answer: As the comment of M.Van says, $A$ need not be closed.
So consider $p \in \overline{A}$ and suppose there is some $B(p,r) \subseteq \overline{A}$. Then $q \in A \cap B(p,r)$ exists. As $q \in A$, there is some $s > 0$ such that $B(q,s) \cap A = \{q\}$ and as $B(p,r)$ is open we can find $s' \le s$ such that $B(q,s') \subseteq B(p,r)$. But $B(q,s') \neq \{q\}$  and so contains points of $A$ different from $q$. Contradiction.
Comment: @MariosGretsas because the ball is not a singleton and lives inside $\overline{A}$.
Comment: Why is $s'<r$??
Comment: @MariosGretsas there is some $s'$ that works, and then take the minimum of that one and $s$ as the $s'$ we use. A smaller one also works.
Comment: I don't understand something..If $s' \leq s$ then $B(q,s') \cap A \subseteq B(q,s) \cap A= \{q\}$.How does $B(q,s')$ contain different points of $A$ other than $q$..
Comment: A ok...thanks for your clarification..


How to test if a function asserts in a NEAR smart contract (AssemblyScript)?

Question: I have a function in my NEAR smart-contract (AssemblyScript) that I want to test. I want to test if the assertion actually happened.
AssemblyScript
foo(id: string): boolean {
  assert(id != 'bar', 'foo cannot be bar');
  return true;
}

Unit test (as-pect)
describe('Contract', () => {
  it('should assert', () => {
    contract.foo('bar'); // <-- How to test assertion here
  })
});

After running the above test, the console logs says

Failed:
should assert - foo cannot be bar

I know I can return false or throw instead of doing an assert for the above example, and I may do that instead if it makes testing easier.
Answer: use toThrow()
Like this:
describe('Contract', () => {
  it('should assert', () => {
    contract.foo('bar').toThrow('foo cannot be bar');
  })
});

you can also use not.toThrow() to test not trowing:
  it('should assert', () => {
    contract.foo('foo').not.toThrow();
  })
});


Squid connection refused after super simple one row in config

Question: I've just started digging in squid, installed it and run. I suppose it works (from localhost), telnet localhost:3128 works as well as curl https://google.com -x localhost:3182.
I want squid to be accessible from local network. My local network is 192.168.88.0/24 (255.255.255.0). I added the next row to squid.conf just before the last row in http_access section:
http_access allow 192.168.88.0/24

Squid stopped working. I leaved only three rows in http_access block so the full section is:
http_access allow localhost
http_access allow 192.168.88.0/24
http_access deny all

curl https://google.com -x 192.168.88.2:3128 (called from local network) does not work. But the more frustrating is that previous command, curl http://google.com -x localhost:3128 (called from localhost) also stopped working, that means that the second row somehow affects the first one.
What's the right way to write http_access block in squid?
Answer: acl rtfm src 192.168.88.0/24
http_access allow rtfm

Comment: A couple links just in case you need help finding the FM.  http://wiki.squid-cache.org/SquidFaq/SquidAcl http://www.squid-cache.org/Doc/config/acl/


Get information like pitch or amplitude from audio byte in an array

Question: I want to get the pitch in Hz of an audio byte in an byte array.
This is my code now:
        byte[] wav = File.ReadAllBytes("test.wav");
        for (int i = 44; i<wav.Length; i++)
        {
            // wav[i] is an audio byte, channel shifts every 2 bytes (I think)
        }

At first I thought that the wav file is built with hundreds or thousands of chunks, that every chunk contains a sample rate, so I tried to scan the whole array for another byte sequence that represents the word "WAVE" which is a part of a chunk, but the sample rate is only at the beginning of the array, and after place 44, all of the array is just the audio data itself.
The audio byte is just a hexadecimal value, I cant understand how can I get any information from that value.
UPDATE: I have downloaded Math.NET library which has FFT algorithm.
this is the documentation for the FFT: https://numerics.mathdotnet.com/api/MathNet.Numerics.IntegralTransforms/Fourier.htm
I have read all of the methods there but I don't know what method will do what I want (give it a few bytes of the wav file and get their frequency).
UPDATE 2:
Now I am using Accord library for the FFT, I found a tutorial for that in youtube.
This is my code for convert the audio bytes to double array:
            for (int i = 44; i<wav.Length; i+=BufferSize)
        {
            float currentSec = (float) audioLength / wav.Length * i;

            byte[] buffer = new byte[BufferSize];

            for (int j = 0; j < buffer.Length; j++)
            {
                if ((i + j + 1) < wav.Length)
                    buffer[j] = wav[i + j];
            }

            int SAMPLE_RESOLUTION = 16;
            int BYTES_PER_POINT = SAMPLE_RESOLUTION / 8;

            Int32[] vals = new Int32[buffer.Length / BYTES_PER_POINT];
            double[] Ys = new double[buffer.Length / BYTES_PER_POINT];
            double[] Ys2 = new double[buffer.Length / BYTES_PER_POINT];

            for (int k = 0; k < Ys.Length; k++)
            {
                byte hByte = buffer[k * 2 + 1];
                byte lByte = buffer[k * 2 + 0];

                vals[k] = (int)(short)((hByte << 8) | lByte);
                Ys[k] = vals[k];
            }
            Ys2 = FFT(Ys);

            double avgFrq = AverageFromArray(Ys2);

            if(lastSecond < (int) currentSec)
                lastSecond = (int) currentSec;

        }

FFT Function:
       private double[] FFT(double[] data)
   {
        double[] fft = new double[data.Length];
        System.Numerics.Complex[] fftComplex = new System.Numerics.Complex[data.Length];
        for (int i = 0; i < data.Length; i++)
        {
            fftComplex[i] = new System.Numerics.Complex(data[i], 0);
        }

        Accord.Math.FourierTransform.FFT(fftComplex, Accord.Math.FourierTransform.Direction.Forward);
        for (int i = 0; i < data.Length; i++)
        {
            fft[i] = fftComplex[i].Magnitude;
        }

        return fft;
    }

So to check if it works I made a wav file that is just a white noise at the frequency of 5000Hz, but These are the results I get from FFT (values of a 2048 bytes array):
https://pastebin.com/PUq5bQTn
The whole audio file has the same frequency of 5000Hz but my code gives me values like 605.80502914453746 and 4401.1090268930584
Comment: What do you mean "pitch of an audio byte"? The array is a digital capture of an audio stream at a particular sampling rate - each byte represents the amplitude at the moment in time it was sampled - as such the byte does not have a pitch. You need to perform some sort of audio analysis of the entire array (or a portion of it) to get the pitch of that period of time.
Comment: @Eldar: to do that you need to do a series of FFT-transforms for different time intervals of the audiostream. For each interval you can determine the pitch from the frequency spectrum and process them further. Of course the assumption here is that your audiostream contains only a small frequencyband at a time, which is not the case with e.g.music, speech...
Comment: You may want to try the [Signal Processing Stack Exchange](https://dsp.stackexchange.com/) to ask for resources - but as marko said _"A fundamental understanding of the maths involved is essential for any kind of audio analysis"_ - any decent book is going to have some very complex maths involved. I learnt the fundamentals 35+ years ago, so I doubt any books I used are still available - so all I can suggest is google. A quick search brought this page up - https://dspguru.com/dsp/books/favorites/ - I can't vouch for any of the books mentioned though.
Comment: You will need to read and analyze the header.
Comment: @PaulF what is the minimal time period I can analyze and get its pitch? how can I do that?
Comment: @TaW I analyzed the header but it just contains the general sample rate, I want to capture every change in pitch in that file and use it later.
Comment: @EldarAzulay: You need to research audio analysis - the more samples you use the better the frequency resolution.
Comment: Before you can find changes you would need to find pitch. Simple for a sine wave, really really hard for music, impossible for sound.
Comment: @JohanDonne It could be helpful if you send me an example of FFT-transform, I am new to this stuff.
Comment: You will need to find a suitable library - implementing FFT is quite complex. You will probably find the library will have example code. You may find several C# implementations if you try google - these will probably do the job but may not be particularly efficient. The Intel IPP libraries which are available for free are very efficient but may not be the easiest of a beginner to use. Try googling for an FFT tutorial - translating from the time domain (WAV array) to frequency domain needs some understanding before you can progress.
Comment: @PaulF sorry for the trouble, but I've gooled FFT tutorial, but I don't understand how can I use FFT for audio, it could be helpful if you send me a good tutorial that talks about the basics of FFT and how I use audio with it. I am a really dummy on that subject..
Comment: It is very complex but to find the pitch you need to find the dominant frequency of a waveform. A waveform can be represented as the sum of multiple sine waves at a particular frequency and amplitude - the FFT converts the waveform into those individual frequency components. The top part of this tutorial explains it without any real maths : http://dmorris.net/projects/tutorials/fourier_tutorial.pdf. In the example the pitch is likely to be taken as the third peak as it has the highest amplitude.
Comment: Understanding what pitch really is is quite complex there are some explanations here : https://stackoverflow.com/questions/1847633/net-library-to-identify-pitches with a possible library/project that may be useful in the very last answer.
Comment: @PaulF chekc out my updated question about Math.NET. They have FFT algorithm but I just need some explanations about how to use it with my code. currently I just have a byte array of all of the audio data from that file. I want to get changes in frequency from the audio data using that FFT.
Comment: Unfortunately I have not used that library & the documentation appears very limited, so I cannot really be of help. What you are doing is quite complex - you really do need to do some studying on how the FFT works & what the data returned really means. Without a fuller understanding of the FFT you could be getting spurious results - simply taking a chunk of data & applying the FFT can give unexpected frequencies due to the sharp cutoff at each end of the sample unless you apply some form of windowing.
Comment: As you appear to have very limited understanding of digital signal processing - then I do recommend you try to find a complete solution, as doing it yourself requires understanding of some complex things. FFT is not the only solution - auto correlation is another technique - [this library](https://pitchtracker.codeplex.com/) claims to be a completed solution based on that technique. Again I haven't used the library, so cannot really be of any help. To download the library use the Source Code tab on the link.
Comment: Here is a good introduction to the basics of FFT without going into the maths if you are interested : http://download.ni.com/evaluation/pxi/Understanding%20FFTs%20and%20Windowing.pdf
Comment: @PaulF I've made some progress, I found a library for FFT and read a tutorial. The problem is that I get wrong frequency values. I will edit the question with my current code and example for the values.
Comment: A fundamental undestanding of the maths involved is essential for any kind of audio analysis - library documentation is not likely to fill this gap.
Comment: @EldarAzulay - sorry I cannot be of further help - as I said earlier you have a very limited understanding of what you are trying to do & the extra information you have added just goes on to show that. "White noise" is defined as _"noise containing many frequencies with equal intensities"_ - so to speak of _"white noise at the frequency of 5000Hz"_ does not make too much sense. You have not yet grasped the basics of FFT if you do not understand the values being returned & try to average them - that makes no sense at all. I repeat my suggestion you find a ready made solution.
Comment: @PaulF I am really new to this audio programming. I would love to learn it from the basics and if you have a good tutorial you can suggest me to start learning it would be great.
Comment: @PaulF thank you for your help!
Answer: I am afraid your code (and question) is overly naive.

A Wav file is not just a collection of audio samples. Have a look at (e.g.) http://soundfile.sapp.org/doc/WaveFormat/ for a description of the file format and its stucture.
If you want to read, process, write audio files, there are different libraries out there (e.g. NAudio) that will help a lot.
From 1 sample in the audiostream you can never calculate the pitch. To do that you need a (relatively large) number of samples and calculate the frequency spectrum using an FFT transform.
Comment: Nice link, I was searching for such a header description. :-)
Comment: I was just reading that before I posted that question, this figure is so good it helped me a lot to understand the structure of a WAV file.
Answer: WAV is data is just pulse code modulated (PCM). This means that every value represents an actual point of the audio signal.
Wav files have a header, you can find some info about it here. It describes how the file is structured.
If you meant by "pitch" the fundamental frequency of the sample, try an  FFT
Amplitude is the value at a certain point, but beware, you need to take these variables into account:

bits ber sample
byte order
block align
channel count
Comment: See @JohanDonne's answer for a better link to the header structure: http://soundfile.sapp.org/doc/WaveFormat/
Comment: When analysing digital audio - the pitch is normally defined as the fundamental frequency of the sample (ie highest amplitutude frequency)  - _"Pitch is the difference in playback and original sample rate "_ sounds more like a pitch shift.
Comment: I meant frequency analysis, I just dont know how to implement the FFT
Comment: @PaulF: Yes, I indeed misunderstood that part of the question, I'll update it.
Answer:
A single FFT magnitude peak is a poor and often inaccurate way to measure musical pitch, as pitch is a more complicated psychoacoustic phenomena.
There is a time-frequency trade off in estimating frequency, usually proportional to sampleRate/blockLength.  So using 44 sample blocks at a sample rate of 44100, the frequency estimation error will be on the order of 44100/44 or around +-1000 Hz (perhaps depending on stationarity and the signal-to-noise ratio).
Comment: can you define `blockLength`?
Comment: The length of the DFT.


Asymptotic sum of set of natural numbers with positive natural density

Question: Let $d(A)$ denote the natural density of $A\subset \mathbb{N}.$
If $0 < \alpha < 1\ $ and $\ d(A) = \alpha,\ $ then is this enough to imply
$$ \lim_{ N\to\infty } \left( \frac{ \displaystyle\sum_{n\in A}^{n\leq N} \frac{1}{n} }{ \displaystyle \sum_{n\in \mathbb{N}}^{n\leq N} \frac{1}{n} } \right) \equiv \lim_{ N\to\infty } \left( \frac{ \displaystyle\sum_{n\in A}^{n\leq N} \frac{1}{n} }{ \log(N) } \right) = \alpha\ ? $$
I think it should be "obviously" true if, for example, $\ A = \lbrace{ \left\lceil \frac{n}{\alpha} \right\rceil: n\in\mathbb{N} \rbrace},\ $ but to me it is not so obvious for general $A.$
I was hoping to use Stolz–Cesàro, but don't quite see how. Or maybe by looking at lots of intervals like $[N, 1.01N]$ for very large $N,$ but I don't have anything concrete yet. Or maybe there is a counter-example.
Edit: maybe the result is (also?) true for $\ \alpha=0\ $ and/or $\ \alpha = 1.$
Answer: Yes, your statement is true. Abel's partial summation formula states that
$$\sum_{n\le N}{a_n f(n)} = A(N)f(N)\ - \int_1^N{A(t)f'(t)\ dt}$$
where $A(n) := \sum_{k\le n}{a_k}$. This is useful if you already know the partial sums of $a_n$, but need to find the partial sums of $a_n$ weighted by some function $f$. If we let $a_n$ be the indicator function defined by
$$a_n := \begin{cases} 1   &\text{if  } n\in A\\
           0 &\text{Otherwise}\end{cases}$$
and another function $S(n) = \sum_{k\le n}{\frac{a_k}{k}}$, we see that your limit is $S(N)/\log(N)$ and our goal is to show that this approaches $\alpha$. By your assumption $d(A) = \alpha$, we know that $\lim_{N\to\infty}{\frac{A(N)}{N}} = \alpha$. To find $S(N)$, we simply apply the partial summation formula as follows with $f(n)=1/n$ -
$$S(N) = \frac{A(N)}{N}\ - \int_1^N{A(t)\left(-\frac{1}{t^2}\right)\ dt}$$
$$= \frac{A(N)}{N}\ + \int_1^N{\frac{A(t)}{t^2}\ dt}$$
By definition, we know that $A(N)/N$ is bounded by $1$, and will approach zero when divided by $\log(N)$. Furthermore, the integral term can be expressed as so
$$\int_1^N{\frac{A(t)}{t^2}\ dt} = \int_1^N{\frac{\alpha t}{t^2}\ dt} + \int_1^N{\frac{A(t) -\alpha t}{t^2}\ dt}$$
$$= \alpha\log(N) + E(N)$$
This means that for $S(N)/\log(N)$ to approach $\alpha$, it remains to show that $E(N)/\log(N)$ tends to zero. Fix $\epsilon > 0$, and let $N$ be large. By the limit $A(n)/n \to \alpha$, there exists an $n_0(\epsilon)$ such that $\left|\frac{A(n)}{n}-\alpha\right|\lt\epsilon$ for all $n\ge n_0$. Or equivalently, $\left|A(n)-\alpha n\right|\lt\epsilon\cdot n$. Thus, we have
$$|E(N)| \le \int_1^N{\frac{|A(t) -\alpha t|}{t^2}\ dt} = \int_1^{n_0}{\frac{|A(t) -\alpha t|}{t^2}\ dt} + \int_{n_0}^N{\frac{|A(t) -\alpha t|}{t^2}\ dt}$$
$$< \int_1^{n_0}{\max\left(\frac{|A(t) -\alpha t|}{t^2}\right)\ dt} + \int_{n_0}^N{\frac{\epsilon\cdot t}{t^2}\ dt}$$
$$= M\cdot n_0(\epsilon) + \int_{n_0}^N{\frac{\epsilon}{t}\ dt} = c_{\epsilon} + \epsilon\log(N)$$
where $c_{\epsilon}$ is a constant only depending on $\epsilon$. Finally, since $\epsilon$ is arbitrary, we have that
$$\limsup_{N\to\infty}{\frac{|E(N)|}{\log(N)}} = 0,$$
which completes the proof.

Remarks:
The natural density $d(A)$ is an example of the average order or "mean value" of a function. In this case, the function is just an indicator of whether or not a number $n$ is in $A$, but in general, this can be any arithmetic function $f$. More formally, the mean value is defined as
$$M(f) = \lim_{N\to\infty}{\frac{1}{N}\sum_{n\le N}{f(n)}}$$
The logarithmic mean value is also defined as
$$L(f) = \lim_{N\to\infty}{\frac{1}{\log(N)}\sum_{n\le N}{\frac{f(n)}{n}}}$$
In general, one can show that the equality $M(f)=\alpha\ $ implies that $L(f)$ exists and is also equal to $\alpha$. This can be proved using the same steps as shown above. In fact, $\alpha$ can take on any complex value, of course this only applies to mean values and not densities of sets.


Product of $x$ with $x$ and $x+1$ being coprime to $n$

Question:
For an odd positive integer $n \gt 1$, let $S$ be the set of integers $x$, $1 \le x \le n$, such that both $x$ and $x+1$ are relatively prime to $n$. Show that $$\prod_{x\in S} x \equiv 1 \pmod n$$

I know there is already a solution on this site here but I don't understand the solution fully and the thread was created over 3 years ago so that is why I decided to create a new thread. Can anyone retranslate the solution presented in the linked thread. On my approach I thought of using $CRT$ but I couldn' t solve the problem using it, so if you know solution that involves $CRT$ please post it.
Any help appreciated.
Comment: The fact that the previous Question, asking exactly the same thing, was "created over 3 years ago" does not dictate creating a new thread.  You might delve into where a previous solution presented a difficulty, or what exactly you thought might connect this problem with the Chinese Remainder Theorem. Those kinds of context would make your Question better (and help Readers avoid "retranslat[ing] the solution" without overcoming your difficulty).
Answer: Lemma $ $ If $\,S\,$ is a subset of $\,\Bbb Z_n\,$ closed under inverses then the product of its elements equals the product of the set of roots of $\,x^2-1\,$ lying in $\,S$.
Proof $\ S$ partitions into pairs $\,(x,x^{-1})$ with $\,x\not\equiv x^{-1}$ & singletons $(x)$ with $\,x^{-1}\equiv x.\,$ Pairs contribute $\,1\,$ to the product, so it reduces to the product of the singletons = roots of $\,x^2\equiv 1$.

Your $S\,$ is closed under inverses, i.e. $\,(n,x(x\!+\!1))=1\Rightarrow\,(n,x^{-1}(x^{-1}\!+\!1))=1,\,$ since $\,x^2\, x^{-1}(x^{-1}\!+\!1) = 1+x\,$ is coprime to $\,n\,$ hence so too are its factors $\,x^{-1},\ x^{-1}\!+\!1.$
If $\,x\in S\:\!$ and $\,x^2\!-1\equiv (x\!-\!1)(x\!+\!1)\equiv 0\, $ then $\,x\!-1\equiv 0,\,$ by $\,x\!+\!1\,$ is cancellable, being coprime to $\,n.\,$ So the set of roots of $\,x^2\!-1\,$ in $\,S\,$ is $\{1\},\,$ with product $1 = \prod S\,$ via Lemma.
Remark $ $ This is closely related to this elementary proof of a generalization of Wilson's theorerm to $\,\Bbb Z_n,\,$ based on similar reflection (involution) symmetry afforeded by inversion.
Answer: Since $x$ is relatively prime with $n$ there exists some $y$ s.t. $xy  \equiv 1 \mod {n}$. Now assume that $x \ne y \mod{n}$. Then observe that $y+1 \equiv \frac {1}{x} +1 \equiv \frac {1+x}{x}\mod {n}$. Since both $x+1$ and $x$ are relatively prime with $n$, they have muliplicative inverses respective to modulo $n$. Thus, from the previous equality, $y+1$ necessarily has a multiplicative inverse. But this implies that $ y+1 $ and $n$ are relatively prime. Moreover $ y $ and $n$ are relatively prime(since $x$ is the multiplicative inverse of $y$).Hence $y \in S$. If $x = y$, then $x^2 \equiv 1 \mod{n}$ which implies $(x-1)(x+1) \equiv 0 \mod{n}$. But since $x+1$ is relatively prime with $n$, it has an inverse and so this implies that $x\equiv 1 \mod {n}$. Hence, if $x \in S $ and x is not equal to its multiplicative inverse $y$ in modulo $n$, $y \in S$,else $x\equiv 1 \mod {n}$ .
Comment: Are you considering the equation $y+1 = \frac{1}{x} +1 = \frac{x+1}{x}$ in $mod$ $n$? Otherwise it doesnt make much sense for me.
Comment: Exactly. Sorry for the mistake.Fixed it now.


Placing the downbeats in this five-bar passage in Bach's Italian Concerto

Question: I am asking about bars 13-19 in the first movement of J.S. Bach's Italian Concerto (BWV 971;  sheet music). My ear always hears bar 13 plus the first crotchet of 14 as one extended bar of 3/4 and then the bar lines following that shifted forward a crotchet to the middle of each bar for 14-19 with a one-crotchet bar at the end of 19 to get back in sync with the written bar lines. Should I keep hearing it like this or train my ear to hear it as written?
I think one advantage of an "as writ" interpretation is that the pattern in the left hand in bar 15 becomes twice as often in 21-24, increasing the energy prior to the semi-quavers in 25-26, but it doesn't seem so natural to phrase the left hand this way. Should I maybe bring out the right hand in 15-21 more than the left to help with respecting where the bar lines are, even though the left has the more interesting part and the right seems more of an accompaniment?
Comment: Good question.  I have some thoughts about this but no time to answer now.  I'll try to post something in a few hours.
Answer: As a matter of fact, Baroque music does sometimes get a half-bar "out-of-phase" with the barlines. I've addressed this phenomenon in a previous answer here: Can you introduce fugue themes in the middle of a measure? If so, how?. In fact, just last night, I was noticing this happens in the final movement of the 3rd Brandenburg Concerto. After two bars of canonic entries of the main scalar theme, there's a circle-of-fifths progression that takes three half-bars to modulate to the dominant key, and then the main theme comes back in again for two more bars. But this time, the theme begins in the middle of bar four, and straddles the barlines. See timestamp 7:30 here:

. In your case, you have almost the exact same thing: three half-bars used to modulate to the dominant.
So this is certainly something that Bach does not shy away from doing. The two things that make this particular case look odd to me is that the bars are already short (cut time), and the right hand's harmonic rhythm seems to mostly still follow the barlines. Although the later can be interpreted as suspensions. Something to remember, though, is that this was written for harpsichord, which doesn't have much (if any) dynamic control, so the music itself dictates where the accents go, and Bach is quite adept at doing so. If Bach had wanted to emphasize the barlines as written, he could have placed the low pedal C's on the barline, and left the spot where they currently are as a rest (for comparison, try playing that way!). The fact that you hear it the way you do, is likely a good indicator in this case of what Bach had in mind. So my gut feeling is: yes. This is yet another case where the music accents gets out of phase with the barlines.
Answer: Hearing bars 13-19 as shifted by half a bar may be due to the prominent low C's on the offbeats.  But Bach rarely plays games of rhythmic ambiguity like, say, Beethoven.  More important is to preserve the rhythmic structure of the melodic phrases.  Bars 1-8 hammer home a phrase that starts just after the downbeat.  If you hear the bars as written, then when the melody flips to the left hand at bar 13, with the same after-the-downbeat phrasing, then it's clearer to the listener that the hands have exchanged roles, and at bar 13 the right hand is now merely noodling.
In bars 13-19 you needn't overemphasize the downbeat (which risks overemphasizing the noodling).  Instead, underemphasize those offbeat low C's.  Either treat of each one as ending a wide-ranging phrase, or (more plausibly to my ear) treat all those C's as a gentle pedal point, a new voice in the texture, whose continuation builds up suspense and tension (more frequent low C's) until the spring finally snaps at the flourish in bar 25.
Answer: I would think of it as written. The 16th notes at the end of m.14 are a pick-up into the next bar, which would not really make sense if the previous three beats were a 3/4 measure, making the last beat of m.14 a downbeat.
If you are having a hard time hearing the groupings, I recommend that you read and play through this with the 8th note as the beat. This will make each measure have four beats, and I believe the rhythmic patterns will make more sense since you will hear the strong beats of 1 and 3. Much of Bach benefits from making the 8th note the beat. Many of his more complicated rhythms become much easier to count and comprehend.


Mysql insere valor diferente na colula de referência passado na query

Question: Olá, quando executo um insert em qualquer tabela que tem a coluna users_id que faz referência a tabela glpi_users, qualquer valor maior que 126 o banco insere o valor 127, por exemplo:
INSERT INTO `glpi_plugin_actualtime_tasks` (`tasks_id`, `actual_begin`, `users_id`)
  VALUES ('8926', '2019-06-27 19:25:22', '373');

O valor que é inserido é 127 e não 373, não faço ideia do que seja!
Servidor: Localhost via UNIX socket
Tipo de servidor: MariaDB
Versão do servidor: 10.1.40-MariaDB - MariaDB Server
Versão do protocolo: 10

Answer: O tipo de dado TINYINT ocupa um único byte e portanto pode representar valores inteiros de -128 a 127. É impossível armazenar o inteiro 373 em um TINYINT.
Comment: É isso mesmo! Deu certo! Muito obrigado pela ajuda!
Answer: O campo users_id deve estar como NUMERO INTEIRO com auto incremento. Desta maneira, ele sempre irá usar o próximo número em novos registros.
Comment: Na verdade a coluna users_id não está como auto increment ela está como tinyint not null, apenas é uma coluna para referenciar id da tabela glpi_users que é auto increment do tipo int.
Comment: CREATE TABLE atak_glpi.glpi_plugin_actualtime_tasks (
  id INT(1) NOT NULL AUTO_INCREMENT,
  tasks_id INT(11) NOT NULL,
  actual_begin DATETIME DEFAULT NULL,
  actual_end DATETIME DEFAULT NULL,
  users_id TINYINT(11) NOT NULL,
  actual_actiontime INT(11) NOT NULL DEFAULT 0,
  PRIMARY KEY (id)
)
ENGINE = INNODB,
AUTO_INCREMENT = 53,
AVG_ROW_LENGTH = 334,
CHARACTER SET utf8,
COLLATE utf8_unicode_ci;

ALTER TABLE atak_glpi.glpi_plugin_actualtime_tasks
  ADD INDEX tasks_id(tasks_id);

ALTER TABLE atak_glpi.glpi_plugin_actualtime_tasks
  ADD INDEX users_id(users_id);


Caused by: java.lang.IllegalStateException: Could not read class: VirtualFile: Kotlin+ Apache Beam defined runner

Question: I implemented an example using Kotlin + Apache Beam to define the Kotlin properties of the pipes but when I ran the project I got the error:
Caused by: java.lang.IllegalStateException: Could not read class: VirtualFile: /Users/duanybaro/.gradle/caches/modules-2/files-2.1/org.apache.beam/beam-runners-google-cloud-dataflow-java/2.27.0/3e551e54b23441cc58c9d01e6614ff67216a7e87/beam-runners-google-cloud-dataflow-java-2.27.0.jar!/org/apache/beam/runners/dataflow/DataflowPipelineJob.class
    at org.jetbrains.kotlin.load.java.structure.impl.classFiles.BinaryJavaClass.<init>(BinaryJavaClass.kt:122)
    at org.jetbrains.kotlin.load.java.structure.impl.classFiles.BinaryJavaClass.<init>(BinaryJavaClass.kt:34)

This error only occurs in Kotlin because, with the code made in java, it works perfectly. Can you give me any suggestions to solve the error?
Answer: I really recommend you using the latest version of Apache Beam, your version is very old.
You can also use the starter for Beam Kotlin.
I share with your an example of Kotlin Beam project from my Github repo based on Maven.
For the Beam pipeline options, can you try with instead of using DataflowPipelineOptions :
val options = PipelineOptionsFactory
        .fromArgs(*args)
        .withValidation()
        .`as`(TeamLeagueOptions::class.java)

val pipeline = Pipeline.create(options)

Example of PipelineOptions :
import org.apache.beam.sdk.options.Description
import org.apache.beam.sdk.options.PipelineOptions

interface TeamLeagueOptions : PipelineOptions {
    @get:Description("Path of the input Json file to read from")
    var inputJsonFile: String

    @get:Description("Path of the slogans file to read from")
    var inputFileSlogans: String

    @get:Description("Path of the file to write to")
    var teamLeagueDataset: String

    @get:Description("Team stats table")
    var teamStatsTable: String

    @get:Description("Job type")
    var jobType: String

    @get:Description("Failure output dataset")
    var failureOutputDataset: String

    @get:Description("Failure output table")
    var failureOutputTable: String

    @get:Description("Feature name for failures")
    var failureFeatureName: String
}

And pass program argument in the mvn command line :
mvn compile exec:java \
  -Dexec.mainClass=fr.groupbees.application.TeamLeagueApp \
  -Dexec.args=" \
  --project=my-project \
  --runner=DataflowRunner \
  --jobName=team-league-kotlin-job-$(date +'%Y-%m-%d-%H-%M-%S') \
  --region=europe-west1 \
  --streaming=false \
  --zone=europe-west1-d \
  --tempLocation=gs://mazlum_dev/dataflow/temp \
  --gcpTempLocation=gs://mazlum_dev/dataflow/temp \
  --stagingLocation=gs://mazlum_dev/dataflow/staging \
  --inputJsonFile=gs://mazlum_dev/team_league/input/json/input_teams_stats_raw.json \
  --inputFileSlogans=gs://mazlum_dev/team_league/input/json/input_team_slogans.json \
  --teamLeagueDataset=mazlum_test \
  --teamStatsTable=team_stat \
  --jobType=team_league_kotlin_ingestion_job \
  --failureOutputDataset=mazlum_test \
  --failureOutputTable=job_failure \
  --failureFeatureName=team_league \
  " \
  -Pdataflow-runner

Comment: Thanks for your answer. I downgrade the version, trying to resolve the issue but this change didn't solve it.  Currently, I using version 2.44.0 but I have the same issue. The funny thing is that with java it works correctly.
Do you have any other idea of what can be the problem?
Comment: Have you checked the project in my Github repository ?
Are you using Maven ?
Comment: Yes, I reviewed it and also put my code using the starter for Beam Kotlin and I have the same issue.
Comment: I updated my answer with another way of using Beam Pipeline options with `Dataflow`, can you try it please ?
Comment: Thanks a lot, @Mazlum. I was able to successfully run the pipeline using the command
"./gradlew run -Dpipeline="pocBigqueryMongoDbDuany" --args="--runner=DataflowRunner" --stacktrace
"
This was the only way I found but I think it's something.
Comment: You're welcome, if my answer helped you, can you upvote and mark it as solved please ?


Model selection in logistic regression

Question: I am building a logistic multiple regression with 5 potential variables candidates. I read  this post and fit all 2^5 = 32 possible combinations of explanatory variables and chose the best model by AIC. However the 'best' model does not include any significant variable whereas some other models (with higher AIC) do have some significant variables. I am not statistician and do not understand why such situation.
Thank you for any comments, clarification or guidance for another approach if needs be.
Comment: Objective is to look at for an association not for a prediction
Answer: This is just a personal opinion, so take it with a grain of salt.
There are a few ways to approach model selection. One of the ways is to construct all possible models and select the one that is 'the best' according to some criterion, e.g. AIC. This is model dredging and is frowned upon by some people because it doesn't incorporate any existing knowledge and maximizes the chance to find a significant model by chance alone. This is basically what you're doing. That being said, data dredging is probably fine if you are exploring an unknown area of research where there is very little or no theoretical knowledge.
Another way would be to, based on what you know of the phenomenon you're trying to model, construct a few valid hypotheses and then construct a model to test these hypotheses. When you compare these models (for example using AIC), you can say something along the lines of "based on what is already known, out of these hypotheses, this hypothesis (or hypotheses) appears describe my data best". You can also take model parameters of a few best models, average them (weighted average) et voila, you have an "average" model that is a compromise between a few good models. This is called the information-theoretic approach. In biology, a book by Burnham and Anderson (Model selection and multimodel inference) is an advocate of this approach. My memory isn't the best, but I think Gentle introduction to MARK has a chapter on this (with a technical part for this specific program on how to model average).
Comment: This kind of dredging is *NOT* OK.  There are no principles in statistics that back up the approach you have outlined.  A few simulations would expose the damage that variable selection does.  In general, variable selection without penalization is invalid.
Comment: In my experience, if you are just interested in predictive performance, don't perform any feature selection, but use penalised regression (c.f. ridge regression), see my answer to the question referenced in the question above.  It is all too easy to over-fit the feature selection criterion and end up with a model that performs worse than the one with all of the features.  You might find this paper useful http://www.jstor.org/stable/2347628
Comment: Thanks @Roman for this comprehensive review. If I would go for the expertise knowledge as stated in your second point, how should I consider or interpret a best model with lower AIC but with no significant variables whereas another model with higher AIC include significant variables?
Comment: "it doesn't incorporate any existing knowledge and maximizes the chance to find a significant model by chance alone. This is basically what you're doing." It reads as something bad. Computational costs permitting, isn't the approach used by the OP the best one for model selection?
Comment: @user603: thanks. Am I wrong if I then consider the model with at least one significant variable with the lower AIC among the subset of models first selected by expert knowledge (rather than the model with the lower AIC even no significant variables).
Comment: @user38000 you would concluce, that that variable has no statistically significant effect on the outcome.
Comment: @user603 that's why I prefaced the text indicating it's my opinion and others may disagree. I think that dredging is OK if you have no working hypothesis/hypotheses. If there's theory behind your problem, one should use also that. Thanks for pointing it out, I will add this to the original text.
Comment: @RomanLuštrik: my comment was meant as a question, not as a criticism of your answer.
Comment: Just to confirm and to be sure I get your points @user603 and @ Roman: Then if a I have 3 models (M1, M2 and M3) based on expertise knowledge, should I run all the three and select the one with the lower AIC even with any significant variable (for example M1) regardless if M2 for example  included one significant variable but with higher AIC?
Comment: @Frank, could you please more elaborate on penalization for logistic mixed regression?
Comment: Thanks @Dikran. I will have a look and try to see how far it will be possible to implement it with sas glimmix
Comment: But as non statistician, I am a bit confused about statistics. I understood that automated selction is to be avoided. But I  learned so far that AIC is still OK for non nested models. What is then wrong in, based on hypothesis via expert knowledge, stipulating three models and selecting the best one by AIC?
Comment: This is basically what I advocate in my answer. It can go wrong if one of the three models is trivial, thus giving unfair advantage to other two.
Comment: Examples of penalization include lasso, elastic net, ridge regression.  Related to whether AIC is OK for aggressive model selection, just note that AIC is just a restatement of P-values so the severe problems of variable selection based on statistical significance is not solved by AIC.
Comment: I will take into consideration all your valuable inputs. Great

Model selection in logistic regression


Grid not working properly

Question: I am trying to show a grid in an HTML page. The grid has been made using angular. It worked fine when run in isolated environment. But when I merged it with my original angular project, it's layout distorted.
Following is my code :
index.html

    <title> SiteMan</title>

    <link rel="stylesheet" href="http://ui-grid.info/release/ui-grid-unstable.min.css" type="text/css">
    <link rel= "stylesheet" href="stylesheet.css">

Siteman RANN 0.1

            <button> Add Attribute </button>
            <button> Add User </button>
            <button> Add Doccument </button>

                <button class="link" ng-click="attr=1 ; docs=0 ; user=0">All Attributes</button>

                <button class="link" ng-click="user=1 ; attr=0 ; docs=0">All Users</button>

                <button class="link" ng-click="docs=1 ; attr=0 ; user=0">All Document Types</button>

Grid with native pagination controls

            Copyright © Siteman RANN

tree.js
var app = angular.module("myApp", ['ngTouch', 'ui.grid', 'ui.grid.pagination']);
 app.controller('gridCtrl', ['$scope', '$http', function ($scope, $http) {
            $http.get('users.json').success (function(data){
               $scope.myData = data;
               });
                $scope.gridOptions = {
                data: 'myData',
                paginationPageSizes: [05, 10, 15],
                paginationPageSize: 10,
              columnDefs: [

                    { field : 'name', displayName: 'Name' },
                    { field : 'type',  displayName: 'Type' },
                    { field : 'displayName', displayName: 'DisplayName'}
        ]
      };

    }]);

stylesheet.css
#header {
    background-color:black;
    color:white;
    text-align:center;
    padding:5px;
    border: 1px solid black;
}
#nav {
    line-height:30px;
    background-color:#eeeeee;
    color:black;
    text-align:center;
    height:30px;
    padding:5px;
    border: 1px solid black;
}
#aside{
    line-height:20px;
    background-color:#eeeeee;
    height:500px;
    width:20%;
    float:left;
    padding:5px;
    border: 2px solid black;
    overflow: scroll;
}
#section {
    float:left;
    height:150px;
    width:77%;
    padding:10px;
    border: 2px solid black;
    overflow: scroll;
}
a {
    text-decoration: none;
}
#article {
    float:left;
    height:330px;
    width:77%;
    padding:10px;
    border: 2px solid black;
    overflow: scroll;
}
#footer {
    background-color:black;
    color:white;
    clear:both;
    text-align:center;
    padding:5px;
    border: 1px solid black;
}
button.link{
    background: none;
    border: none;
}

 .grid {
      width: 800px;
    }

I could not find the reason of distorted grid. Help..!!
Comment: You might want to add a sample page (e.g link to jsfiddle or jsbin)
Answer: You padding is not working the way you think it is.
Just add box-sizing: border-box to your code.
* {
    box-sizing: border-box;
}

Further explanation on box-sizing:
Box Sizing | CSS tricks
* {box-sizing: border-box}
Comment: This is already mentioned in `http://ui-grid.info/release/ui-grid-unstable.min.css`.
Comment: @PraveenKumar then he is probably getting an error loading that css.
Comment: That's the base CSS right, so may be for us, loading partially might be erroneous. For him, his local computer should load it correctly right?
Comment: @PraveenKumar probably. But I explained the reason why the grid broke. Anyways, deleting this since it doesn't answer OP.
Comment: If this is the only issue and OP gets it sorted, I'll remove the down vote. No offence. `:D`
Comment: No worries. Wait, might be your answer is what the OP is looking for?
Comment: Lets wait and see then.

Grid not working properly


Hypothesis testing for one sample proportions

Question: $H_0: p = 0.1$
$H_a: p \neq 0.1$
So here I have a pretty small sample, and I read somewhere that in order to approximate this binomial distribution with the normal distribution, I have to have $n*p > 5$ and $n*(1-p) > 5$. Since $30 * 0.1 = 3 < 5$, does that mean I cannot use the $Z$ statistic here? If so, what can I do to overcome this?
Comment: [two tailed binomial test](http://en.wikipedia.org/wiki/Binomial_test)
Comment: The Binomial distribution gives you an exact distribution for the number of successes in $n$ trials when these are independent and each trial has success probability $p$.
Comment: @Adrian You don't need the normal approximation. You're doing an exact test and can derive the p-value directly. Use that p-value. Google "Exact test".
Answer: You can do an exact calculation using the binomial theorem. If p=.1 and n=30, what is the probability of 5,6,7 ... 30 cancer cells.  Add these probabilities together (which will be .175).  That's a one-tailed test. Double this probability for two tails.
The individual probabilities can easily be calculated in Excel using the BINOMDIST function.
Comment: @Glen_b you are correct that there is no rejection region at the low end with alpha=.05. With p=.1, n=30 getting 0 events has a probability of .042.  I was following this logic: a two-tailed test is generally appropriate, logically. If so, you would want to locate half the rejection region at the high end, half at the low. So if you have .175 at the high end for a hypothesis test,  that's really a one-tailed test.
Comment: Once you get the probability (say it's 0.175). What do I do next? How do I know if I reject or fail to reject the null?
Comment: If the conventional alpha level for this research is .05, two tailed, you have 2 (.175) = .35 here, which is substantially higher than .05. So don't reject the null, if these are your actual numbers.
Comment: I'm a little confused why you're doubling a one tailed probability here. What's the actual two-tailed rejection region?


How to use TextResponse, HtmlResponse and XmlResponse?

Question: I see there are several types of responses, but how do I signal Scrapy to return an HtmlResponse?
I think the goal would be to implement def parse(self, response: HtmlResponse):. Or is this supposed to be used some other way? Is there an usag example?
This is the example from Scrapy tutorial. How would I use HtmlResponse here instead of the default?
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            'https://quotes.toscrape.com/page/1/',
            'https://quotes.toscrape.com/page/2/',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = f'quotes-{page}.html'
        with open(filename, 'wb') as f:
            f.write(response.body)
        self.log(f'Saved file {filename}')

Comment: can you please add your existing code?
Comment: I added a minimum example. thanks!
Comment: `response.body` returns the HTML source. what's the issue?
Comment: How do you use XmlResponse?
Comment: `XmlResponse` inherits from `Response` class, you use it in the same way.
Answer: Scrapy tries to identify the type of response it gets and calls parse with a specific type. As far as I can tell, parse is never called with the base type Response. The Response identification is done in `scrapy/responsetypes.py by a sort of methods: mimetype, body, headers, etc.
Here's the mimetype identification map:
CLASSES = {
    'text/html': 'scrapy.http.HtmlResponse',
    'application/atom+xml': 'scrapy.http.XmlResponse',
    'application/rdf+xml': 'scrapy.http.XmlResponse',
    'application/rss+xml': 'scrapy.http.XmlResponse',
    'application/xhtml+xml': 'scrapy.http.HtmlResponse',
    'application/vnd.wap.xhtml+xml': 'scrapy.http.HtmlResponse',
    'application/xml': 'scrapy.http.XmlResponse',
    'application/json': 'scrapy.http.TextResponse',
    'application/x-json': 'scrapy.http.TextResponse',
    'application/json-amazonui-streaming': 'scrapy.http.TextResponse',
    'application/javascript': 'scrapy.http.TextResponse',
    'application/x-javascript': 'scrapy.http.TextResponse',
    'text/xml': 'scrapy.http.XmlResponse',
    'text/*': 'scrapy.http.TextResponse',
}

Since parse is called with one of the subclasses, devs have access to it directly in the response parameter. One way to use it is like this:
   def parse(self, response):
       if isinstance(response, HtmlResponse):
           ...


how to combine profiles and content types in a view

Question: I'm trying to filter some content with a view, But view provides access to single entity type. I want to add the fields from profiles and content types. I'm wondering Is it possible to include both contents and profiles in a single view. I have spent lot of hours searching on web to find a solution for this, but didn't helped.
In this image I have selected content and it allows only to access the elements of contents. But I also want to filter the result by matching the contents of a content type and a profile.

Can anyone provide me a better solution to solve this problem....
Comment: in that case you can create two views and embed one view on other using this module, https://drupal.org/project/viewfield
Comment: did you check this post  http://drupal.stackexchange.com/questions/26898/linking-two-different-content-types-with-views
Comment: @shrish But my view only provides access to the fields of the selected profile.
Answer: With Content-type view, you can use the "relationship" feature in Views and have access to the profile fields of the author of a certain node that is listed in the view. The same is valid for the Filters of that view.
To do so, edit your view and expand the "Advanced" settings section on the right. Then click on Relationships -> Add and add the relationship "Content: Author".
Hope that helps.


Question about weak law

Question: This is from Durrett. Let $X_i$ be $iid$ with $P\{X_i = (-1)^k k\} = \frac{C}{k^2\log k}, k\geq 2$ and $C = [\sum_{k=2}^{\infty} \frac{1}{k^2 \log k}]^{-1}$. Show that $E|X_1|=\infty$ but there is a finite constant $\mu$ s.t. $\frac{S_n}{n}\xrightarrow{p} \mu $.
My attempt:
Step 1) Show $nP(|X_1|>n)\rightarrow 0$
Step 2) This means $\frac{S_n-E\tilde{S_n}}{n}\xrightarrow{p} 0$ where $\tilde{S_n} = \sum_{k=1}^{n}\tilde{X_k}$ and $\tilde{X_k} = X_k .1_{(|X_k|<n)}$ which means I have to show $lim_{n\rightarrow \infty} \frac{E\tilde{S_n}}{n}$ is a constant.
Step 1)
$$P(|X_{1}|>n) = \sum_{k=n}^{\infty} \frac{C}{k^2 \log k}\leq \int_{n}^{\infty}\frac{C}{x^2 \log x}dx\leq \frac{C}{\log n}\int_{n}^{\infty}\frac{1}{x^2}dx = \frac{C}{n\log n}$$
$$\therefore nP(|X_{1}|>n) = \frac{C}{\log n}\xrightarrow{n\rightarrow \infty} 0$$
Step 2) I am stuck here. Since $X_i$ are $iid$:
$$E\tilde{S_n} = nE\tilde{X_1} = n E[X_1 .1_{(|X_1|<n)}] = n\sum_{k=2}^{n} (-1)^k k.\frac{C}{k^2\log k} = nC\sum_{k=2}^{n} (-1)^k.\frac{1}{k\log k}$$
So for $lim_{n\rightarrow \infty} \frac{E\tilde{S_n}}{n}$ to be a constant, have to show:
$$lim_{n\rightarrow \infty} \sum_{k=2}^{n} (-1)^k.\frac{1}{k\log k}<\infty$$
Comment: The statement $\frac{S_n}{n} \to \frac{E\tilde{S}_n}{n}$ makes no sense. What I think you mean to say is that $\frac{S_n-E\tilde{S}_n}{n} \to 0$. Then you need to refine that to show that $\frac{E\tilde{S}_n}{n}$ changes slowly enough with $n$ that $\frac{S_n-n\mu}{n} \to 0$ as well, for some $\mu$.
Comment: Yes, I had stated that at the beginning of the sentence $\frac{S_n - E\tilde{S_n}}{n}\xrightarrow{p} 0$ which is the same as $\frac{S_n}{n}\xrightarrow{p} \frac{E\tilde{S_n}}{n}$
Comment: The point is that $\frac{E\tilde{S}_n}{n}$ isn't exactly constant. But if its limit exists, then you can expect that it is the same as the limit in probability of $\frac{S_n}{n}$. So once you know what this number is, you can try to prove what you actually want to prove.
Comment: True, $\frac{E\tilde{S_n}}{n} = C\sum_{k=-n}^{n}\frac{(-1)^k}{k\log k}$, so should I take the limit of this sum?
Comment: I'm not sure that expression is actually correct (even after removing $-1,0$ and $1$ obviously); for instance, does $X$ really have any probability to be equal to $+3$? (Note that if that expression were correct, the sum would actually be zero.)
Comment: To put it another way, your notation suggests to me that the distribution is actually concentrated on $\{ 2,-3,4,-5,\dots \}$. This is especially true because your $C$ has no extra factor of $2$ in it to account for symmetry. In any case you would still need to compute $\lim_{n \to \infty} \sum_{k=2}^n \frac{(-1)^k}{k\log(k)}$, or at least use the value of this sum as a variable.
Comment: Yes, you are right, since $|X_1|<n$, I had the indices from $-n$ to $n$ in haste. But as the support is $\{2,-3,4,-5,..\}$, you are right that I need to find $lim_{n\rightarrow \infty}\sum_{k=2}^{n}\frac{(-1)^k}{k\log k}$.
Comment: ^no, you just need to show that the limit exists. This should be much easier.
Comment: That won't work because actually that integral isn't finite. You need to show conditional convergence.
Comment: True, will try that.
Answer: You should use this theorem which appears in the chapter before the problem:

Theorem 2.2.7. Weak Law of Large Numbers. Let $X_i$ be i.i.d. with $$xP(|X_i|>x)\to 0\qquad as\quad x\to\infty.$$ Let $\mu_n=E[X_1{\bf 1}_{|X_1|<n}]$. Then $S_n/n-\mu_n\to 0$ in probability.

You have already shown $xP(|X_i|>x)\to 0$, so you now can conclude $S_n/n-\mu_n\to 0$. What is $\mu_n$?
$$
\mu_n=\sum_{k=2}^nk(-1)^k\cdot\frac{C}{k^2\log k}.
$$
Note $\lim_n \mu_n$ exists by the alternating series test. Can you conclude?
Comment: Yes, I was applying that theorem. I see, didn't think of alternating series test. Thanks.

Question about weak law


Avoid 'SettingWithCopyWarning:' when setting values with loc and conditions (inside a function)

Question: I'm trying to set some values in one column (col_2) coming from another column (col_3) when values in col_1 fulfill a condition.
I'm doing that with the .loc operator as after all my research looking for alternatives it looks as the best practice.
data.loc[data.col_1.isin(['A','B']),col_2]=data[col_3]
Nevertheless this stills trigger the 'SettingWithCopyWarning:'.
Is there any other way to do this operation without raising that warning?
Comment: This is most likely because your `data` variable is a subset of another dataframe (e.g. you created `data` with some type of `data = some_df.loc[...]`. You can use `data = data.copy()` before you try to change the values in `data` to remedy this warning though.
Answer: Following the advice of Cameron,the line of code was part of a function, where data was passed as argument.
Starting the function with data = data.copy() solves the issue


main menu disappears only on home page

Question: I have a site online engcorrect.net and for some reason my menu on my homepage disappears. On all other pages, it shows. The weird thing is that on my localhost (local machine) it shows correctly but not on the production site and it also shows if I use private browsing. What am I missing here? I am using Drupal 7 and bootstrap 3 theme with a subtheme called paper.
There is a css line that I cannot allocate:
<code>body.front ul.menu { display:none;}
</code>
The inspector shows that it originates from engcorrect.net:1 where is that and who set it?

resizing doesn't fix the issue either.
Who can help me with this?
Answer: Ok. It's resolved. I cleared the browsers cache in Safari under: Preferences/Privacy/Manage Website Data and there I searched for my site name, selected it and hit "remove all". Therefore I presume it was my Safari browser's cache.

main menu disappears only on home page


Deduction in template functions with few args

Question: I have few instances of one template function. Each of them sequentially executes each of given lambdas, accompanying them with specific messages. When I do that with one lambda, everything works fine, but when I try to add more than one, I get
note: candidate template ignored: deduced conflicting types for parameter 'Task'

From clang. Here is my code:

template <class Task> void doTasks(Task task1)  // works fine
{
    if (std::__is_invocable<Task>::value)
    {
        std::cout << "doing first task" << endl;
        task1;
    }
}

template <class Task>
void doTasks(Task task1, Task task2) // deduced conflicting types
{
    if (std::__is_invocable<Task>::value)
    {
        std::cout << "doing first task" << endl;
        task1();
        std::cout << "doing second task" << endl;
        task2();
    }
}

int main()
{
    doTasks([&] (){std::cout << "1" << endl;});

    doTasks([&] (){std::cout << "1" << endl;},
            [&] (){std::cout << "2" << endl;});

    return 0;
}

What's wrong with it? How can I deal with my problem?
Sorry if it's a stupid question, I'm some kind of beginner in C++ and may not understand some template nuances.
Comment: Who told you about `__is_invocable`? Any symbol beginning with double underscore should be considered private and an implementation-detail that should not be used by normal application code (unless specified and mentioned by the C++ specification, like e.g. the `__cplusplus` macro).
Comment: Two different lambdas always have two different types, so the second template won't match. Try `template  void doTasks(Task1 task1, Task2 task2)` etc.
Comment: @Someprogrammerdude compiler told me. In my real code there is just ```is_invocable```, but for some reason Clang couldn't compile example with this and suggested me to add ```__```. Anyway, thanks for explaining what does that mean.
Answer: Even though the two passed lambdas does the same or looks similar, they have completely different types. Hence, you need two different template types there in the doTasks.
Easy fix is providing the template parameter for two lambdas
template <class T1, class T2>
void doTasks(T1 task1, T2 task2)
{
  // ...
}

otherwise, using the variadic template function and fold expression, you could do something like:
#include <type_traits> // std::conjunction_v, std::is_invocable

template <class... Ts>
void doTasks(Ts&&... lmds) {
    if (std::conjunction_v<std::is_invocable<Ts>...>) // maybe if constexpr
    {
        (lmds(), ...);
    }
}

Since the argument is variadic, it will also work for the single/ any arguments of doTasks and hence no code duplication.
Live demo
Comment: @sovesti May be, you pack the intermediate things as a callable object and call the `doTasks`  with them in an order. Here is an example: https://gcc.godbolt.org/z/jvMxhnbM8
Comment: I read about fold expressions and it's very nice feature, but as you can see in my example, I need to print something between lambdas, but I can't find anything about putting few expressions there.
Answer: The lambdas are of different types. You need to add another template parameter for the second one:
#include <type_traits> // for the proper `std::invocable`

template <class Task, class AnotherTask>
void doTasks(Task task1, AnotherTask task2) {
    if constexpr (std::is_invocable_v<Task> && std::is_invocable_v<AnotherTask>) {
        std::cout << "doing first task\n";
        task1();
        std::cout << "doing second task\n";
        task2();
    }
}

Note: Since it's C++17, you can also use constexpr if to deal with the if at compile time only.
Demo


Raspberry Pi NoIR camera hangs

Question: Background Information:
I built a smart door solution with a RPi B+ (Raspbian) which makes use of three sensors (PIR, Gyroscope [I2C] and an Ultrasound). Depending on the sensor status managed by a Python script,  I start recording a video by using the raspivid command and/or play sounds on Bluetooth speakers. I also have a USB WIFI dongle connected to the RPi.
This is how I call the Pi NoIR camera in my .sh file:
raspivid -w 800 -h 600 -t 15000 -o file.h264 -n -rot 270

Problem description:
The camera works fine for a while (I cannot precise for how long [2 hours maybe] nor what triggers the issue), but then it stops working. The funny thing is that I get no error message.
What did I do to identify the root cause:

I tried to manually kill the child process (Raspivid) but it simply doesn't work. If I kill the parent process (.sh file) then my child process gets assigned to the PID 1 (init.d).
I tried recording a new video in a new terminal window but the new process hangs too.
Rebooting the Pi doesn't work neither. It says it is rebooting but it does not (not even with the -f option).
I installed a new Pi Camera ensuring it was not a hardware or ribbon problem.
Camera has enough memory available.
Pi has enough power (Red led is always on)
-v returns no error
I have installed V4L2-CTL but I have the same issue

How can I get it fixed?
Thanks!
** Edit **
dmesg:
[   90.555578] ------------[ cut here ]------------
[   90.555698] WARNING: CPU: 0 PID: 3593 at drivers/media/v4l2-core/videobuf2-core.c:2135 __vb2_queue_cancel+0xf4/0x160 [videobuf2_core]()
[   90.555715] Modules linked in: evdev uinput rfcomm bnep bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd spi_bcm2708 i2c_bcm2708 i2c_dev ecb btusb 8192cu bluetooth rfkill uio_pdrv_genirq uio
[   90.555843] CPU: 0 PID: 3593 Comm: v4l2-ctl Not tainted 3.18.7+ #755
[   90.555915] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
[   90.556012] [<c0012710>] (show_stack) from [<c0555b30>] (dump_stack+0x20/0x28)
[   90.556053] [<c0555b30>] (dump_stack) from [<c0022ef4>] (warn_slowpath_common+0x7c/0x9c)
[   90.556083] [<c0022ef4>] (warn_slowpath_common) from [<c0022fd0>] (warn_slowpath_null+0x2c/0x34)
[   90.556143] [<c0022fd0>] (warn_slowpath_null) from [<bf263a1c>] (__vb2_queue_cancel+0xf4/0x160 [videobuf2_core])
[   90.556252] [<bf263a1c>] (__vb2_queue_cancel [videobuf2_core]) from [<bf265548>] (vb2_internal_streamoff+0x40/0xdc [videobuf2_core])
[   90.556361] [<bf265548>] (vb2_internal_streamoff [videobuf2_core]) from [<bf2656d8>] (vb2_streamoff+0x3c/0x60 [videobuf2_core])
[   90.556436] [<bf2656d8>] (vb2_streamoff [videobuf2_core]) from [<bf26574c>] (vb2_ioctl_streamoff+0x50/0x54 [videobuf2_core])
[   90.556669] [<bf26574c>] (vb2_ioctl_streamoff [videobuf2_core]) from [<bf228658>] (v4l_streamoff+0x28/0x2c [videodev])
[   90.556938] [<bf228658>] (v4l_streamoff [videodev]) from [<bf22baf4>] (__video_do_ioctl+0x254/0x2e8 [videodev])
[   90.557182] [<bf22baf4>] (__video_do_ioctl [videodev]) from [<bf22b49c>] (video_usercopy+0x180/0x560 [videodev])
[   90.557414] [<bf22b49c>] (video_usercopy [videodev]) from [<bf22b898>] (video_ioctl2+0x1c/0x24 [videodev])
[   90.557616] [<bf22b898>] (video_ioctl2 [videodev]) from [<bf2266b0>] (v4l2_ioctl+0x11c/0x14c [videodev])
[   90.557791] [<bf2266b0>] (v4l2_ioctl [videodev]) from [<c014c478>] (do_vfs_ioctl+0x3f0/0x5d4)
[   90.557872] [<c014c478>] (do_vfs_ioctl) from [<c014c6a0>] (SyS_ioctl+0x44/0x6c)
[   90.557917] [<c014c6a0>] (SyS_ioctl) from [<c000e8c0>] (ret_fast_syscall+0x0/0x48)
[   90.557936] ---[ end trace 28c9150399e29093 ]---
[ 3123.544118] ------------[ cut here ]------------
[ 3123.544235] WARNING: CPU: 0 PID: 28629 at drivers/media/v4l2-core/videobuf2-core.c:2135 __vb2_queue_cancel+0xf4/0x160 [videobuf2_core]()
[ 3123.544251] Modules linked in: evdev uinput rfcomm bnep bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd spi_bcm2708 i2c_bcm2708 i2c_dev ecb btusb 8192cu bluetooth rfkill uio_pdrv_genirq uio
[ 3123.544404] CPU: 0 PID: 28629 Comm: v4l2-ctl Tainted: G        W      3.18.7+ #755
[ 3123.544475] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
[ 3123.544570] [<c0012710>] (show_stack) from [<c0555b30>] (dump_stack+0x20/0x28)
[ 3123.544676] [<c0555b30>] (dump_stack) from [<c0022ef4>] (warn_slowpath_common+0x7c/0x9c)
[ 3123.545092] [<c0022ef4>] (warn_slowpath_common) from [<c0022fd0>] (warn_slowpath_null+0x2c/0x34)
[ 3123.545178] [<c0022fd0>] (warn_slowpath_null) from [<bf263a1c>] (__vb2_queue_cancel+0xf4/0x160 [videobuf2_core])
[ 3123.545253] [<bf263a1c>] (__vb2_queue_cancel [videobuf2_core]) from [<bf265548>] (vb2_internal_streamoff+0x40/0xdc [videobuf2_core])
[ 3123.545316] [<bf265548>] (vb2_internal_streamoff [videobuf2_core]) from [<bf2656d8>] (vb2_streamoff+0x3c/0x60 [videobuf2_core])
[ 3123.545377] [<bf2656d8>] (vb2_streamoff [videobuf2_core]) from [<bf26574c>] (vb2_ioctl_streamoff+0x50/0x54 [videobuf2_core])
[ 3123.545562] [<bf26574c>] (vb2_ioctl_streamoff [videobuf2_core]) from [<bf228658>] (v4l_streamoff+0x28/0x2c [videodev])
[ 3123.545803] [<bf228658>] (v4l_streamoff [videodev]) from [<bf22baf4>] (__video_do_ioctl+0x254/0x2e8 [videodev])
[ 3123.546060] [<bf22baf4>] (__video_do_ioctl [videodev]) from [<bf22b49c>] (video_usercopy+0x180/0x560 [videodev])
[ 3123.546309] [<bf22b49c>] (video_usercopy [videodev]) from [<bf22b898>] (video_ioctl2+0x1c/0x24 [videodev])
[ 3123.546483] [<bf22b898>] (video_ioctl2 [videodev]) from [<bf2266b0>] (v4l2_ioctl+0x11c/0x14c [videodev])
[ 3123.546595] [<bf2266b0>] (v4l2_ioctl [videodev]) from [<c014c478>] (do_vfs_ioctl+0x3f0/0x5d4)
[ 3123.546676] [<c014c478>] (do_vfs_ioctl) from [<c014c6a0>] (SyS_ioctl+0x44/0x6c)
[ 3123.546717] [<c014c6a0>] (SyS_ioctl) from [<c000e8c0>] (ret_fast_syscall+0x0/0x48)
[ 3123.546737] ---[ end trace 28c9150399e29094 ]---
[34462.831249] bcm2835_v4l2: error 0 waiting for sync completion
[34465.833783] bcm2835_v4l2: error 0 waiting for sync completion
[34465.833853] bcm2835_v4l2: failed disconnecting src port
[34468.833772] bcm2835_v4l2: error 0 waiting for sync completion
[34468.833818] bcm2835-v4l2: Failed to disable encode component -62
[34471.834219] bcm2835_v4l2: error 0 waiting for sync completion
[34471.834395] bcm2835-v4l2: vidioc_s_fmt_vid_cap: failed to setup mmal components: -62
[34474.873813] bcm2835_v4l2: error 0 waiting for sync completion
[34474.890435] bcm2835-v4l2: queue_setup: capture port not configured
[34495.914162] bcm2835_v4l2: error 0 waiting for sync completion
[34498.916190] bcm2835_v4l2: error 0 waiting for sync completion
[34498.916290] bcm2835-v4l2: vidioc_s_fmt_vid_cap: failed to setup mmal components: -62
[34502.064159] bcm2835_v4l2: error 0 waiting for sync completion
[34502.070076] bcm2835-v4l2: queue_setup: capture port not configured
[36016.330148] bcm2835_v4l2: error 0 waiting for sync completion
[36019.339279] bcm2835_v4l2: error 0 waiting for sync completion
[36019.339327] bcm2835-v4l2: vidioc_s_fmt_vid_cap: failed to setup mmal components: -62
[36022.430218] bcm2835_v4l2: error 0 waiting for sync completion
[36022.441684] bcm2835-v4l2: queue_setup: capture port not configured

syslog:
Feb 20 05:35:02 raspberrypi kernel: [34462.831249] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:05 raspberrypi kernel: [34465.833783] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:05 raspberrypi kernel: [34465.833853] bcm2835_v4l2: failed disconnecting src port
Feb 20 05:35:08 raspberrypi kernel: [34468.833772] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:08 raspberrypi kernel: [34468.833818] bcm2835-v4l2: Failed to disable encode component -62
Feb 20 05:35:11 raspberrypi kernel: [34471.834219] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:11 raspberrypi kernel: [34471.834395] bcm2835-v4l2: vidioc_s_fmt_vid_cap: failed to setup mmal components: -62
Feb 20 05:35:14 raspberrypi kernel: [34474.873813] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:14 raspberrypi kernel: [34474.890435] bcm2835-v4l2: queue_setup: capture port not configured
Feb 20 05:35:35 raspberrypi kernel: [34495.914162] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:38 raspberrypi kernel: [34498.916190] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:38 raspberrypi kernel: [34498.916290] bcm2835-v4l2: vidioc_s_fmt_vid_cap: failed to setup mmal components: -62
Feb 20 05:35:41 raspberrypi kernel: [34502.064159] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 05:35:41 raspberrypi kernel: [34502.070076] bcm2835-v4l2: queue_setup: capture port not configured
Feb 20 06:00:55 raspberrypi kernel: [36016.330148] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 06:00:58 raspberrypi kernel: [36019.339279] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 06:00:58 raspberrypi kernel: [36019.339327] bcm2835-v4l2: vidioc_s_fmt_vid_cap: failed to setup mmal components: -62
Feb 20 06:01:01 raspberrypi kernel: [36022.430218] bcm2835_v4l2: error 0 waiting for sync completion
Feb 20 06:01:01 raspberrypi kernel: [36022.441684] bcm2835-v4l2: queue_setup: capture port not configured

messages:
Feb 19 20:02:10 raspberrypi kernel: [   90.555578] ------------[ cut here ]------------
Feb 19 20:02:10 raspberrypi kernel: [   90.555698] WARNING: CPU: 0 PID: 3593 at drivers/media/v4l2-core/videobuf2-core.c:2135 __vb2_queue_cancel+0xf4/0x160 [videobuf2_core]()
Feb 19 20:02:10 raspberrypi kernel: [   90.555715] Modules linked in: evdev uinput rfcomm bnep bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd spi_bcm2708 i2c_bcm2708 i2c_dev ecb btusb 8192cu bluetooth rfkill uio_pdrv_genirq uio
Feb 19 20:02:10 raspberrypi kernel: [   90.555843] CPU: 0 PID: 3593 Comm: v4l2-ctl Not tainted 3.18.7+ #755
Feb 19 20:02:10 raspberrypi kernel: [   90.555915] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
Feb 19 20:02:10 raspberrypi kernel: [   90.556012] [<c0012710>] (show_stack) from [<c0555b30>] (dump_stack+0x20/0x28)
Feb 19 20:02:10 raspberrypi kernel: [   90.556053] [<c0555b30>] (dump_stack) from [<c0022ef4>] (warn_slowpath_common+0x7c/0x9c)
Feb 19 20:02:10 raspberrypi kernel: [   90.556083] [<c0022ef4>] (warn_slowpath_common) from [<c0022fd0>] (warn_slowpath_null+0x2c/0x34)
Feb 19 20:02:10 raspberrypi kernel: [   90.556143] [<c0022fd0>] (warn_slowpath_null) from [<bf263a1c>] (__vb2_queue_cancel+0xf4/0x160 [videobuf2_core])
Feb 19 20:02:10 raspberrypi kernel: [   90.556252] [<bf263a1c>] (__vb2_queue_cancel [videobuf2_core]) from [<bf265548>] (vb2_internal_streamoff+0x40/0xdc [videobuf2_core])
Feb 19 20:02:10 raspberrypi kernel: [   90.556361] [<bf265548>] (vb2_internal_streamoff [videobuf2_core]) from [<bf2656d8>] (vb2_streamoff+0x3c/0x60 [videobuf2_core])
Feb 19 20:02:10 raspberrypi kernel: [   90.556436] [<bf2656d8>] (vb2_streamoff [videobuf2_core]) from [<bf26574c>] (vb2_ioctl_streamoff+0x50/0x54 [videobuf2_core])
Feb 19 20:02:10 raspberrypi kernel: [   90.556669] [<bf26574c>] (vb2_ioctl_streamoff [videobuf2_core]) from [<bf228658>] (v4l_streamoff+0x28/0x2c [videodev])
Feb 19 20:02:10 raspberrypi kernel: [   90.556938] [<bf228658>] (v4l_streamoff [videodev]) from [<bf22baf4>] (__video_do_ioctl+0x254/0x2e8 [videodev])
Feb 19 20:02:10 raspberrypi kernel: [   90.557182] [<bf22baf4>] (__video_do_ioctl [videodev]) from [<bf22b49c>] (video_usercopy+0x180/0x560 [videodev])
Feb 19 20:02:10 raspberrypi kernel: [   90.557414] [<bf22b49c>] (video_usercopy [videodev]) from [<bf22b898>] (video_ioctl2+0x1c/0x24 [videodev])
Feb 19 20:02:10 raspberrypi kernel: [   90.557616] [<bf22b898>] (video_ioctl2 [videodev]) from [<bf2266b0>] (v4l2_ioctl+0x11c/0x14c [videodev])
Feb 19 20:02:10 raspberrypi kernel: [   90.557791] [<bf2266b0>] (v4l2_ioctl [videodev]) from [<c014c478>] (do_vfs_ioctl+0x3f0/0x5d4)
Feb 19 20:02:10 raspberrypi kernel: [   90.557872] [<c014c478>] (do_vfs_ioctl) from [<c014c6a0>] (SyS_ioctl+0x44/0x6c)
Feb 19 20:02:10 raspberrypi kernel: [   90.557917] [<c014c6a0>] (SyS_ioctl) from [<c000e8c0>] (ret_fast_syscall+0x0/0x48)
Feb 19 20:02:10 raspberrypi kernel: [   90.557936] ---[ end trace 28c9150399e29093 ]---
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544118] ------------[ cut here ]------------
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544235] WARNING: CPU: 0 PID: 28629 at drivers/media/v4l2-core/videobuf2-core.c:2135 __vb2_queue_cancel+0xf4/0x160 [videobuf2_core]()
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544251] Modules linked in: evdev uinput rfcomm bnep bcm2835_v4l2 videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media snd_bcm2835 snd_pcm snd_seq snd_seq_device snd_timer snd spi_bcm2708 i2c_bcm2708 i2c_dev ecb btusb 8192cu bluetooth rfkill uio_pdrv_genirq uio
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544404] CPU: 0 PID: 28629 Comm: v4l2-ctl Tainted: G        W      3.18.7+ #755
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544475] [<c00151fc>] (unwind_backtrace) from [<c0012710>] (show_stack+0x20/0x24)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544570] [<c0012710>] (show_stack) from [<c0555b30>] (dump_stack+0x20/0x28)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.544676] [<c0555b30>] (dump_stack) from [<c0022ef4>] (warn_slowpath_common+0x7c/0x9c)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545092] [<c0022ef4>] (warn_slowpath_common) from [<c0022fd0>] (warn_slowpath_null+0x2c/0x34)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545178] [<c0022fd0>] (warn_slowpath_null) from [<bf263a1c>] (__vb2_queue_cancel+0xf4/0x160 [videobuf2_core])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545253] [<bf263a1c>] (__vb2_queue_cancel [videobuf2_core]) from [<bf265548>] (vb2_internal_streamoff+0x40/0xdc [videobuf2_core])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545316] [<bf265548>] (vb2_internal_streamoff [videobuf2_core]) from [<bf2656d8>] (vb2_streamoff+0x3c/0x60 [videobuf2_core])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545377] [<bf2656d8>] (vb2_streamoff [videobuf2_core]) from [<bf26574c>] (vb2_ioctl_streamoff+0x50/0x54 [videobuf2_core])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545562] [<bf26574c>] (vb2_ioctl_streamoff [videobuf2_core]) from [<bf228658>] (v4l_streamoff+0x28/0x2c [videodev])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.545803] [<bf228658>] (v4l_streamoff [videodev]) from [<bf22baf4>] (__video_do_ioctl+0x254/0x2e8 [videodev])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546060] [<bf22baf4>] (__video_do_ioctl [videodev]) from [<bf22b49c>] (video_usercopy+0x180/0x560 [videodev])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546309] [<bf22b49c>] (video_usercopy [videodev]) from [<bf22b898>] (video_ioctl2+0x1c/0x24 [videodev])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546483] [<bf22b898>] (video_ioctl2 [videodev]) from [<bf2266b0>] (v4l2_ioctl+0x11c/0x14c [videodev])
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546595] [<bf2266b0>] (v4l2_ioctl [videodev]) from [<c014c478>] (do_vfs_ioctl+0x3f0/0x5d4)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546676] [<c014c478>] (do_vfs_ioctl) from [<c014c6a0>] (SyS_ioctl+0x44/0x6c)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546717] [<c014c6a0>] (SyS_ioctl) from [<c000e8c0>] (ret_fast_syscall+0x0/0x48)
Feb 19 20:52:43 raspberrypi kernel: [ 3123.546737] ---[ end trace 28c9150399e29094 ]---

Comment: I would not rely on the red LED to signify that power is sufficient. All it takes is one little glitch to mess things up.
Comment: No messages in `dmesg` or syslog? Does the Pi get enough power? You have quite a power hungry setup that might be prone to power fluctuations. What happens if you re-plug the camera after it fails? What's the `dmesg` output then?
Comment: There is enough power since the red led remains on all the time. Re-plugging the camera does not change anything. -v returns no errors. Which syslog are you talking about? Thank you!
Comment: `/var/log/messages` or `/var/log/syslog`/ and the `dmesg` command.
Comment: Thanks again. I have just checked and no related error (at least not related to camera, raspivid or the driver name). Any keywork? Should I post it here?
Comment: Just after the lockup happens, see what `dmesg` tells you. The last few (~10) lines should be important. Maybe the driver fails and tells you.
Comment: This seems to be a bug in the `bcm2835_v4l2` kernel module. Try reloading it: `rmmod bcm2835_v4l2; modprobe bcm2835_v4l2`.
Comment: I will test and let you know. Camera is actually working fine because I have just restarted :-) . If it works, please post your answer so I reward you with the bounty. Thank you!!!
Comment: Hi nemo. That doesn't seem to solve the issue. Reloading the module doesn't let me record new videos unless I physically reboot my Pi. The process which has the raspivid cannot be killed. If I kill its parent then I gets assigned to the PID 1. I am out of hope I can get this issue fixed.
Comment: libkmod: ERROR ../libkmod/libkmod-module.c:753 kmod_module_remove_module: could not remove 'bcm2835_v4l2': Device or resource busy
Error: could not remove module bcm2835_v4l2: Device or resource busy
Comment: Hi @KagueiNakueka I browsed here after seeing the bounty bacuase I've had a streaming camera working on a Pi (although external USB webcam, not the Pi Cam). I see that you have deleted an answer referring to I2C conflict with another device.  Does that mean that it **was** the root cause, but you still need to use the camera with the other device attached to the same Pi?  Or does it mean that disconnecting the other device did not, in fact, solve the problem?  Can you edit that information into the question?
Comment: your problem seems to stem from the V4L2 driver, there's a pretty active forum on the raspberry Pi site - see [here](https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=62364&hilit=bcm2835_v4l2%3A+error+0+waiting+for+sync+completion&start=375).  Your problem seems to resemble a few of those reported there - definitely worth checking you've got the latest patches talked about there.
Answer: By looking at your kernel messages, specifically the line containing '__vb2_queue_cancel', I think your problem may partially be related to Raspberry's kernel drivers. Have you checked if your kernel contains the fix given at [1]? If not, applying that oneliner may fix your problem.
For some reason your camera application is stopping video capture and due to the aformentioned bug it fails to do so. This may at least give you a chance to restart your capture pipeline.
[1] https://github.com/6by9/linux/commit/f1e870c2c910d0620b1a48ac7165c8f9e9a75527

Raspberry Pi NoIR camera hangs


I am having trouble in Python - from import does not working

Question: I am trying to import from a specific python file (which I wrote) class that in that file and it seems that python just does not recognize my file.
add image to represent the situation more deeply:
that the file I want to import the Ball class from:

the main file and the error:

mylib
the files
Comment: welcome to stackoverflow! please take the [tour](http://stackoverflow.com/tour),
read up on [how to ask a question](https://stackoverflow.com/help/asking). [please do not post code as an image](https://meta.stackoverflow.com/questions/285551/why-not-upload-images-of-code-on-so-when-asking-a-question/285557#285557)
[as they are not helpful](http://idownvotedbecau.se/imageofcode).
Comment: Maybe go read this question https://stackoverflow.com/questions/4383571/importing-files-from-different-folder
Comment: That warning is coming from the IDE, do you get an error when you actually run the script?
Comment: yes they are...
Comment: It's hard to help without knowing the project's structure
Comment: He seems to have two files one called main_Pygame.py which he calls as the script and one called shapes.py that is in the same directory. btw, I did not notice right away, but he has two links in his question next to each other rather than one.
Answer: To import from the same directory, put a dot at the front of the module name like:
from .shapes import Ball

But if you are running main_Pygame.py as the script then this will not work. You should create a subdirectory for shapes.py, say mylib and then make an empty file called __init__.py there and mve shapes.py to that directory. Then import like:
from mylib.shapes import Ball

The __init__.py file tells python that the mylib directory (or whatever else you wish to call the directory) is part of the directory tree it should look for modules under.
Comment: yep! now it's all working, thank you very much for your quick answer and patience.
Comment: Traceback (most recent call last):
  File "C:/Users/idofr/PycharmProjects/FirstPythonProject/Python_3.8_projects/GAME/more_Pygame.py", line 2, in
    from .shapes import Ball
ImportError: attempted relative import with no known parent package
Comment: Ah so you must be running more_Pygame.py as main script?
Comment: yes, the import seems to work but i got this error
Comment: That strange.... I am quite sure that I did it right but now python does not recognize the mylib dict
Comment: added another pic in the question
Comment: I am seeing pics rather than links in the questions now, but they look like the same way as you had it. Just. as a check list 1) The script you call should be in a directory by itself. 2) This directory should have a subdirectory, lets call it mylib. 3) An empty file named `__init__.py` with two underscore characters before and after the init part should be in our mylib subdirectory. 4) The shapes.py file should be in the subdirectory. 5) The import should then be `from mylib.shapes import Ball`
Comment: yeah i did it, look at the bottom of the question its some link or somthing
Comment: they called mylib and files
Comment: Oh I see now. Looks like you have mylib next to the GAME directory. But mylib should be inside the GAME directory as a subdirectory.
Comment: I fixed wording of question to make it more clear I meant mylib to be a subdirectory.


Move tempo signature up from staff

Question: In the code example below, I'd like to move the tempo signature "Slow 80" on the sheet music a little bit up. Is this possible?
\header {
  title = "test"
  subtitle = "123"
  composer = "my self"
  % Remove default LilyPond tagline
  tagline = ##f
}

\paper {
  #(set-paper-size "a4")
}

\layout {
  \context {
    \Score
    \remove "Bar_number_engraver"
  }
}
\relative c''
 {\tempo "Slow" 4 = 80
 a b e (f) f g d2 f g e a b a b1 c d e f \bar"||"}

Answer: Add the following line of code before the \tempo command:
\override Score.MetronomeMark.Y-offset = X
Where X = <number of staff spaces to adjust in vertical direction>
See also: LilyPond Internals Reference — 3.1.76 MetronomeMark

Move tempo signature up from staff