Prevent spawns being placed In the same Xpos on different Y position - Corona SDK

Question: I seem to be having an issue which I cannot solve properly and need some help.
Looking at the image below, I am spawning shapes from a table called enemies and am trying to check if their x positions are occupied by a spawn, if so, then slightly move it left or right so it pops out enough to hit them as they move downwards. The image shows a problem where shapes are stacked behind each other.

Spawn function (called from the game loop)
local xPos = math.random(22,279)
local r = math.random(1, #spawnData)

local sd = spawnData[r] -- get the spawn data for this enemy

local s = display.newSprite(sd.imgSheet, sd.seqData)
s.name = sd.name
physics.addBody(s, {isSensor = true})
s:setSequence(sd.seq)
s:setFrame(sd.frame)
s.y = display.contentHeight - 550 -- Create the enemies using spawnData

local ok = false -- A flag to say whether we found a valid X position
local gap = 2     -- Set the minimum gap in pixels between enemies
local err = 0    -- An error flag to avoid the game slowing down if too many enemies

while ok == false do -- Tell it to loop until a valid X position is found
    xPos = math.random(20,300) -- give a new X position
    ok = true -- Until it has checked against other enemies, assume it is ok for now

    for i = 1, #enemies, 1 do -- Loop through the previously spawned enemies
       local e = enemies[i] -- Link to this enemy

       local dist = mAbs(e.x - xPos) -- Work out the distance between this enemy and the random position chosen
       if dist < (e.width + gap) then ok = false end -- If the enemies are too close, try agian
    end

    err = err + 1 -- Increase the count of attempts to find a valid position
    if err > 50 then ok = true end -- After 50 tries, assume there is no empty space and just spawn wherever
end

s.x = xPos
enemies[#enemies+1] = s

enemyGroup:insert(s)

Answer: You have two snippets of code that aid this error.
In this first snippet, you are looking at every sprite that has existed and trying to find an appropriate x position. With many sprites, the error becomes more prone. Try to only look at the previous Sprite's x position.
for i = 1, #enemies, 1 do -- Loop through the previously spawned enemies
       local e = enemies[i] -- Link to this enemy

       local dist = mAbs(e.x - xPos) -- Work out the distance between this enemy and the random position chosen
       if dist < (e.width + gap) then ok = false end -- If the enemies are too close, try agian
    end

In this next snippet, after 50 errors are made, it places the sprite anywhere. You are allowing this spawn to occur. Try a larger value. However, that is going to only postpone the issue.
err = err + 1 -- Increase the count of attempts to find a valid position
    if err > 50 then ok = true end -- After 50 tries, assume there is no empty space and just spawn wherever

Edit:
Here is what I would do. There may be errors because I did not test it. Hopefully it does give you the idea of what should be done, though.
    local xPos = math.random(22,279)
    local r = math.random(1, #spawnData)

    local sd = spawnData[r] -- get the spawn data for this enemy

    local s = display.newSprite(sd.imgSheet, sd.seqData)
    s.name = sd.name
    physics.addBody(s, {isSensor = true})
    s:setSequence(sd.seq)
    s:setFrame(sd.frame)
    s.y = display.contentHeight - 550 -- Create the enemies using spawnData

    local ok = false -- A flag to say whether we found a valid X position
    local gap = 2     -- Set the minimum gap in pixels between enemies
    local err = 0    -- An error flag to avoid the game slowing down if too many enemies

-- EDITED CODE BELOW
--------------------
    while ok == false do -- Tell it to loop until a valid X position is found
        xPos = math.random(20,300) -- give a new X position

        local e = enemies[#enemies] -- Get the last enemy in the table (should be the previously made)

        if e == nil then
           break  -- Break out of the while loop
        end

        local dist = mAbs(e.x - xPos) -- Work out the distance between this enemy and the random position chosen
        if dist >= (e.width + gap) then
            ok = true
        else
            err = err + 1 -- Increase the count of attempts to find a valid position
            if err > 50 then
                ok = true
            end
        end
    end

    s.x = xPos
    enemies[#enemies+1] = s

    enemyGroup:insert(s)

Comment: I guess trying to check the previous sprites X position would work, however I am not sure how to target a previous spawn in the table and check its position compared to the random x position of the new incoming spawn
Comment: It should be the last index of your table. So just use enemies[#enemies] and you wouldn't need the for loop.
Comment: Im sorry, but Im not sure I understand how you mean. Is there any chance you could show me an example of how you would do it so that I can understand it better? Thanks for helping me also :)
Comment: Edited the answer to include an example.
Comment: Hey thanks for the help. I tried the example but get a 'Attempt to index local 'e' (a nil value) error. Any reason why thats happening? I tried a few workarounds but to no avail.
Comment: That simply means that the index of the table we are trying to access has no value. You first sprite is likely the issue because the table is empty. If e is nil, exit the while loop. I made the edit in the code above.


Function that takes list of GenericType does not accept provided arguments

Question: I am trying to implement a function that takes a list of a generic type. But it does not accept the provided arguments.
interface TheInterface<T extends string> {
    theRecord: Record<T, unknown>
}

const theFunction = <T extends string>(...arrayOfInterfaceType:TheInterface<T>[]) => {

return (key: keyof typeof arrayOfInterfaceType[number]['theRecord']) => key

}

const values1 = {theRecord:{value1: ""}}

const values2 = {theRecord:{value2: ""}}

const values1TheInterface: TheInterface<keyof typeof values1.theRecord> = values1

const values2TheInterface: TheInterface<keyof typeof values2.theRecord> = values2

//Works perfectly
const functionWithOneInterface = theFunction(values1TheInterface)
//Intelisense works for key
functionWithOneInterface("value1")
//Does not work
const functionWithTwoInterfaces = theFunction(values1TheInterface, values2TheInterface)
//Intellisense only works for value1
functionWithTwoInterfaces("value1", "value2")

The code below is simplified but reflects the same behaviour as the actual code.
Playground link to code
Answer: The Heuristics of the compiler prevent you to use multiple different string literals for T here. Once T is inferred by the first element, all the other elements must have the same T.
You probably want something like this:
const theFunction = <T extends TheInterface<any>[]>(
  ...arrayOfInterfaceType: [...T]
) => {
  return (...keys: { [K in keyof T]: keyof T[K]["theRecord"] }) => keys;
};

Instead of using T to store the keys, just let it store the whole array. Also let's use the spread syntax [...T] so that the order of the elements is preserved.
The function you are returning seems to be callable as a rest parameter. For the type, we map over T and extract the key values for each element T[K].

Playground


jquery ui resizable a floating element.

Question: When you float an element to the right, then use jQuery UI's resizable widget to make it resizable from the left edge, weird things start happening.
I've put this together:
http://jsfiddle.net/nicholasstephan/kCduV/2/
If you resize the "sidebar", you'll see immediately what I'm talking about.
What am I doing wrong here?
I'd prefer to keep the float:right so the left column takes care of itself.
Answer: Working demo: http://jsfiddle.net/kCduV/10/ and another solution: http://jsfiddle.net/kCduV/12/
jQuery code:
<code>$(function() {
    // $('.resizable').resizable({'handles': 'w'});
    $('.resizable').resizable({
        handles: 'e,w',
        resize: function (event,ui) {
            ui.position.left = ui.originalPosition.left;
            ui.size.width = (ui.size.width
                - ui.originalSize.width )*2
                + ui.originalSize.width;
        }
     });
});
</code>

OR
<code>$(function() {
    // $('.resizable').resizable({'handles': 'w'});
    $('.resizable').resizable({
        handles:'e,w',
        resize: function (event,ui) {
            ui.position.left = ui.originalPosition.left;
            ui.size.width += (ui.size.width - ui.originalSize.width);
        }
    });
});
</code>
Explanation: jQuery Resizable: doubling the resize width
Comment: makes sense. thanks! Both your fiddles still seem to the do the double transformation thing though. The right edge stays put, but the left edge moves more than the mouse. Using the resize event to modify the sidebar itself is still a better than using event to manipulate the left element's width. Thanks!
Comment: No worries man! Glad it helped you, yep you can try and play around with the ui.position, cheers!

jquery ui resizable a floating element.


If X and Y both have the same pdf, they are identically distributed, but can I say that they are independent?

Question: More specific: I have two variables, X and Y, they are exponentially distributed with parameter a. I know that the distribution of X + Y follows a Gamma(2,a) distribution IF they are independent (I already know that they are identically distributed). But can I safely assume that they are independent?
A yes or no would suffice :)
Comment: Let $X=Y$. Do X and Y have the same PDF? Are $X$ and $Y$ independent?
Comment: @ Matthew Gunn if X = Y and if X (and therefore Y) equals a (particular) constant with probability 1 (i.e., is "deterministic), then X is independent of Y (nevertheless, i upvoted your comment). Of course, this doesn't apply if X 9and Y) are exponentially distributed.
Comment: Though not your example if you take any distribution symmetric about 0 then X and Y could be identically distributed with Y= -X.
Answer: Safety depends on context.
But in general, identicalness of distribution does not imply independence.  In particular situations (applications), random variables may happen to be independent, or in some cases, even if not, they may be close enough to being independent to suit certain practical purposes (calculations of adequate accuracy).
There is a commonly stated specification for random variables known as i.i.d, which stands for independent and identically distributed.  It is a rather strong condition.  In legal documents, you may see such terms as 'cease and desist", both words of which mean pretty much the same thing.  Not so for i.i.d, in which each "i" has a different meaning. Random variables can be 1) independent without being identically distributed or 2) identically distributed without being independent or 3) independent and identically distributed or 4) neither independent nor identically distributed
Comment: Let me summarize with respect to OP's ask of "A yes or no would suffice :)".... No.
Comment: @Eagle The difficulty is that yes-no questions are off limits here.  By offering an extended explanation, Mark is attempting to rescue this question from oblivion.
Comment: It might help to have a simple example illustrating all four possibilities. Draw a card from a standard 52-card deck. Let $X=1$ if the card is a club (which is black) or a diamond (which is red) and $X=0$ otherwise. Let $Y=1$ if the card is a spade or heart, $Y=0$ otherwise. Let $Z=1$ if the card is red and $Z=0$ otherwise. Let $W$ be the card's value from $1$ through $10$; set it to $10$ for the 12 remaining "court cards". Then $(X,W)$ are independent but not identically distributed, $(X,Y)$ are i.d. but not independent, $(X,Z)$ are i.i.d., and $(Z,W+Z)$ are neither i.d. nor independent.


Mix of n normals with known locations

Question: I have data points that are generated with the $n$ normal distributions with the same $\sigma$ and different means.
I do not know $n$, but I know that $1 \leq n \leq 4$. I know the possible set of means, i.e., $\mu = -2,-1, 0, 1, 2$ (so the number of potential means is bigger than expected number of clusters).
Mclust from R package works well and solves the problem in a nice way. The problem is that there are a lot of data points and Mclust works too slow (I guess it would work faster if it could take into account prior information of $\mu$ vector).
Is there a way to solve this problem in a fast and efficient way? I am trying to do mixtools, it is faster than Mclust, the problem is that I do not know number of clusters $n$ and this way of solving the problem is making huge mistakes when $n$ was initially wrong. Should I just apply BIC and do a greedy search for optimal $n$? Or are there more efficient ways?
I would appreciate links to the theory/approach that can help (so I do not expect that it was already implemented).
Comment: Does the answer here http://stats.stackexchange.com/questions/14051/clustering-with-some-cluster-centers-fixed-known?rq=1 help? It is an empirical rather than theoretical approach.
Comment: @mdewey this is a pretty good thing! I will try it, probably it will solve the problem. I do not think that k-means can be correctly applied here (my distributions are mixing due high variances, sorry for not specifiying it), but if it will give good results - it will be super nice.
Answer: Use the initialization option of the mclust command to control the initial settings and give initial hints.
Comment: thanks, finally solved it with mixtools, its way faster somehow

Mix of n normals with known locations


How do you re-learn the basics you already know?

Question: Bit of a story first to explain. Then the question. Sorry for the length, but I'm not even sure how to really ask.
I have a toddler (3 years old) that is showing a strong interest in music, so I decided that I would pick up a "cheap" keyboard and learn to play it properly. In the mean time I already know how to play it "improperly" so I can play small pieces like Twinkle Twinkle etc. And he can see it, and more importantly interact with it by pressing buttons and hearing sounds and listening to some important things (for example these two keys together sound good, and these two sound bad). All in all a good way for him to see, and touch music in the house, that later we can turn into lessons, but for now can give him a place to explore.
For my self, I used to play Trumpet quite well, I enjoyed it, and did several performances before I started to get serious about my business. Still I practice, though more as a hobby, and I have tried to find a way "play" more music but without the need to practice in an ensemble (for example) so that I can have my cake and eat it too.
Several years ago, I decided to learn keyboard because you can play quite a few pieces on keyboard, and there is always the opportunity to preform at a social gathering of nearly any kind. And while I made an honest effort at it, the method I chose really sucked. I don't want to get into a lot of detail here, but while I know where "Middle C" is and so on, music reading was not part (I could already read, so I didn't think I needed a strong focus on that) and left hand play is extremely weak. Chords and what not, were also non-existent.
I realized that I was not really getting anywhere with that so I sold the keyboard (It was actually a very good, and sadly expensive instrument) and stuck with the Trumpet.
This time around however I am trying to learn [more] properly. I have picked up a method book, and I am doing all the exercises, including "learning to read" but there in lies the core of my problem. I can quickly dismiss the years of "messing around" with the old keyboard lessons, because I know they are not correct. This time, I know, and can therefore stay focused on facts like the "old way" never focused on things like finger positioning, but that it's important so pay attention.
But I have a hard time, not focusing on core music principals that I did learn properly and have been practicing for years. As an example, one of the early excersizes has you play a chord, containing a C, D, G on the left hand while playing notes, C, E, G on the right hand one after another. While I know this is to help with finer positioning, and independence, whenever I play it my brain auto flips out and I go from focusing on correct playing to "what did I do wrong".

Now the question
Again, sorry that took so long.
Is there, and how can I focus on learning, like a new student, that wouldn't know that the example "sounds wrong" without ignoring all my current "knowledge". For example I fall back on that same "knowledge" when I am supposed to play a C and some other note comes out. I then know to check my fingers. (again examples)
I do want to learn to play this new instrument, but I keep running into this clash (in several areas, for example I can read both clefs but I read the Trouble clef much stronger, so I have a tendency to "blow though" right hand work, while struggling in left hand work, for no other reason then I have to sit there and go "That is a C not an A"
So again,
How can I focus on learning, like a new student, given that I am not exactly new, but I do need to focus on some areas where I am just like, or worse then, a new student?
Comment: Good teachers will see your level of play and teach you accordingly.
Comment: Take lessons.  Problem solved.  Any other approach is just plain wrong.
Answer: Focus on the basics. You're already coming to that conclusion but you really need to take a strong look at yourself and how you practice. You say your left hand is weak, so practice with only your left hand and the bass clef for awhile. Play up and down the white keys, then the black keys, then chromatically. Do the same for your right hand then combine them. One thing I love to do doesn't even involve playing; take a pencil and some paper and write down the notes, the keys, the fingerings, anything you're studying.

(for example these two keys together sound good, and these two sound bad)

I think this is a mistake in your thinking that's also holding you back. All of the intervals on a keyboard are used between every key. Even though fifths and octaves are more "consonant" in general, there are always situations where dissonance is called for, and even where consonant intervals are combined with dissonant intervals. The exercise is trying to teach you that. If all you do is play normal triad major chords all day your work will sound boring, so most composers write in inversions or suspensions to make it more interesting.
I think one of the biggest ways you can improve your reading is to look at each note not as a single note that has one position (like you would on trumpet) but as a note in comparison to those around it. Don't read the notes, read and hear the intervals. Since you'll often have to change positions it is advantageous to know your intervals and what intervals lead to the notes around your current position. This is a big change from trumpet which has basically only one hand position.
Answer: I would suggest you take a couple of lessons, to receive a proper personalized feedback from a pianist. If you don't have such an opportunity, I would suggest you do the old trick with the metronome: put it on 60 bpm and play the passage carefully as full notes, then go to 70 etc. Finally, when you reach 120, re-do this whole routine as half notes etc. Playing very slowly while concentrated on every note and every movement you make really helps! Think of it not as performing at a slow pace, but rather as engraving moves into your muscle memory. Gradually, you will be able to correct everything. Hope this helps)
As for reading notes, there are plenty of apps that can help you regain/improve your skills, e.g. https://play.google.com/store/apps/details?id=com.binaryguilt.completemusicreadingtrainer for Android. I've managed to learn to read notation and sight read at a decent pace in just a week using a similar app.
Good luck!


Python monkeypatching best practices

Question: I am testing an application that has several external dependencies and I have used monkeypatching techniques to patch the functions of external libraries with a custom implementation to help my tests. It works as expected.
But the problem I currently have is that this makes my test file really messy. I have several tests and each test requires its own implementation of the patched function.
For instance, let us say I have a GET function from an external library, my test_a() needs GET() to be patched so that it returns False and test_b() needs GET() to be patched so that it returns True.
What is the preferred way to handle such a scenario. Currently I do the following:
def test_a(monkeypatch):
    my_patcher(monkeypatch, patch_get_to_return_true = True, patch_get_to_return_false = False, patch_get_to_raise_exception = False)

def test_b(monkeypatch)
    my_patcher(monkeypatch, patch_get_to_return_true = True, patch_get_to_return_false = False, patch_get_to_raise_exception = False)

def test_c(monkeypatch)
    my_patcher(monkeypatch, patch_get_to_return_true = False, patch_get_to_return_false = False, patch_get_to_raise_exception = True)

def my_patcher(monkeypatch, patch_get_to_return_true = False, patch_get_to_return_false = False, patch_get_to_raise_exception = False):

    def patch_func_pos():
        return True

    patch_func_neg():
        return False

    patch_func_exception():
        raise my_exception

    if patch_get_to_return_true:
        monkeypatch.setattr(ExternalLib, 'GET', patch_func_pos)

    if patch_get_to_return_false:
        monkeypatch.setattr(ExternalLib, 'GET', patch_func_neg)

    if patch_get_to_raise_exception:
        monkeypatch.setattr(ExternalLib, 'GET', patch_func_exception)

The above sample has just three tests that patch one function. My actual test file has around 20 tests and each test will further patch several functions.
Can someone suggest me a better way of handling this? Is it recommended to move monkeypatching part to a separate file?
Answer: Without knowing further details, I would suggest splitting my_patcher into several small fixtures:
@pytest.fixture
def mocked_GET_pos(monkeypatch):
    monkeypatch.setattr(ExternalLib, 'GET', lambda: True)

@pytest.fixture
def mocked_GET_neg(monkeypatch):
    monkeypatch.setattr(ExternalLib, 'GET', lambda: False)

@pytest.fixture
def mocked_GET_raises(monkeypatch):
    def raise_():
        raise Exception()
    monkeypatch.setattr(ExternalLib, 'GET', raise_)

Now use pytest.mark.usefixtures to autoapply the fixture in test:
@pytest.mark.usefixtures('mocked_GET_pos')
def test_GET_pos():
    assert ExternalLib.GET()

@pytest.mark.usefixtures('mocked_GET_neg')
def test_GET_neg():
    assert not ExternalLib.GET()

@pytest.mark.usefixtures('mocked_GET_raises')
def test_GET_raises():
    with pytest.raises(Exception):
        ExternalLib.GET()

However, there is room for improvements, depending on the actual context. For example, when the tests logic is the same and the sole thing that varies is some test precondition (like different patching of GET in your case), tests or fixtures parametrization often saves a lot of code duplication. Imagine you have an own function that calls GET internally:
# my_lib.py

def inform():
    try:
        result = ExternalLib.GET()
    except Exception:
        return 'error'
    if result:
        return 'success'
    else:
        return 'failure'

and you want to test whether it returns a valid result no matter what GET behaves:
# test_my_lib.py

def test_inform():
    assert inform() in ['success', 'failure', 'error']

Using the above approach, you would need to copy test_inform three times, the only difference between the copies being a different fixture used. This can be avoided by writing a parametrized fixture that will accept multiple patch possibilities for GET:
@pytest.fixture(params=[lambda: True,
                        lambda: False,
                        raise_],
                ids=['pos', 'neg', 'exception'])
def mocked_GET(request):
    monkeypatch.setattr(ExternalLib, 'GET', request.param)

Now when applying mocked_GET to test_inform:
@pytest.mark.usefixtures('mocked_GET')
def test_inform():
    assert inform() in ['success', 'failure', 'error']

you get three tests out of one: test_inform will run three times, once with each mock passed to mocked_GET parameters.
test_inform[pos]
test_inform[neg]
test_inform[exception]

Tests can be parametrized too (via pytest.mark.parametrize), and when applied correctly, parametrization technique saves a lot of boilerplate code.
Comment: Thanks for your detailed answer. I knew about monkeypatching, fixtures and parameterized tests. But I never thought about combining all these concepts to avoid code duplication. Your answer really helped me. Too bad I can only give you one vote.
Comment: Glad I could help! Reading the official docs helped me a lot when I first started working with `pytest`, lots of great examples there.

Python monkeypatching best practices


An underdetermined system derived for rotated coordinate system

Question: Consider two orthogonal coordinate systems $x_A y_A z_A$ and $x_B y_B z_B$ sharing the same origin. Denote by $\boldsymbol i_A$, $\boldsymbol j_A$, $\boldsymbol k_A$ the set of basis vectors, and assume that $\boldsymbol k_B$ is given in $x_A y_A z_A$. Using directional cosines expansion, I derive the system
$$
\begin{cases} \boldsymbol i_B = c_{xx} \boldsymbol i_A + c_{xy}  \boldsymbol j_A + c_{xz} \boldsymbol k_A \\ \boldsymbol j_B = c_{yx} \boldsymbol i_A + c_{yy} \boldsymbol j_A + c_{yz} \boldsymbol k_A \\ c_{zx} c_{xx} + c_{zy} c_{xy} + c_{zz} c_{xz} = 0 \\ c_{zx} c_{yx} + c_{zy} c_{yy} + c_{zz} c_{yz} = 0 \\ c_{xx}^2 + c_{xy}^2 + c_{xz}^2 = 1 \\ c_{yx}^2 + c_{yy}^2 + c_{yz}^2 = 1 \end{cases}
$$
which is underdetermined, so I need 2 additional constraints.
It is given that each line $AB$ in $x_A y_A z_A$ that is parallel to the $z_A$ axis, is rotated to a line CD in $x_B y_B z_B$, which is parallel to the $z_B$ axis. Moreover, AB and CD intersect.
How to express these constraints in terms of directional cosines? Are there constraints missing from the derived system?
Thank you for your attention and time.
Edit 1
I am not sure, but I think that because of orthogonality of $z_B$ to $x_B y_B$ and $z_A$ to $x_A y_A$, the angle between $x_A$ and $x_B$ is equal to the angle between $y_A$ and $y_B$, so that $c_{xx} = c_{yy}$. This is wrong!
Edit 2
We also need to add
$$
\boldsymbol i_B \cdot \boldsymbol j_B = c_{xx} c_{yx} + c_{xy} c_{yy} + c_{xz} c_{yz} = 0
$$
to the system. So, now we have 7 equations for 8 unknowns:
$$
\begin{cases} \boldsymbol i_B = c_{xx} \boldsymbol i_A + c_{xy}  \boldsymbol j_A + c_{xz} \boldsymbol k_A \\ \boldsymbol j_B = c_{yx} \boldsymbol i_A + c_{yy} \boldsymbol j_A + c_{yz} \boldsymbol k_A \\ c_{zx} c_{xx} + c_{zy} c_{xy} + c_{zz} c_{xz} = 0 \\ c_{zx} c_{yx} + c_{zy} c_{yy} + c_{zz} c_{yz} = 0 \\ c_{xx} c_{yx} + c_{xy} c_{yy} + c_{xz} c_{yz} = 0 \\ c_{xx}^2 + c_{xy}^2 + c_{xz}^2 = 1 \\ c_{yx}^2 + c_{yy}^2 + c_{yz}^2 = 1 \end{cases}
$$
Comment: Never used that method before, or it has a different name in my language... But any reason why you didn't include
$\mathbf i_b.\mathbf j_B=0$ and
$\|\mathbf k_B\|^2=1$ ?
Comment: I think it'd be a bit simpler to parameterize the rotation, and express $\mathbf i_B,\mathbf j_B$ with respect to those parameters... I'll have a quick look.
Regarding "$c_{xx}=c_{yy}$", I think it's wrong, but I don't have a simple example/explanation to give you right now.
Comment: Actually, I would welcome any other approach.

I really did not think about $\boldsymbol i_B \cdot \boldsymbol j_B = 0$. Thanks! On the other hand, $\boldsymbol k_B$ is given, so no need to add your second constraint to the system.

What do you think about $c_{xx} = c_{yy}$?
Answer: Assuming that your two coordinate systems are right-handed, there exists a rotation $r_{L,\theta}$ with axis $L$ through the origin, and angle $\theta$,
such that
\begin{align}
\mathbf i_B &= r_{L,\theta}(\mathbf i_A)\\
\mathbf j_B &= r_{L,\theta}(\mathbf j_A)\\
\mathbf k_B &= r_{L,\theta}(\mathbf k_A)
\end{align}
Expressing the rotation for an arbitrary point is a little annoying in general, but we here have the additional constraint that if you rotate a line parallel to $\mathbf k_A$, its image by $r_{L,\theta}$ will always intersect the original line. I don't feel like doing it right now, but you can show that this condition implies that $L$ must be orthogonal to $\mathbf k_A$.
In other words, consider $\mathbf u$ a direction vector of $L$, and let $\mathbf v$ such that $\mathbf u,\mathbf v,\mathbf k_A$ is a (right-handed) orthonormal basis. There exists
$\varphi\in\mathbb R$ such that
\begin{align}
\mathbf u &~=~&  \cos(\varphi) \mathbf i_A& +\sin(\varphi)\mathbf j_A\\
\mathbf v &~=~& -\sin(\varphi) \mathbf i_A& +\cos(\varphi)\mathbf j_A\\
\mathbf i_A &~=~& \cos(\varphi)\mathbf u& -\sin(\varphi)\mathbf v\\
\mathbf j_A &~=~& \sin(\varphi)\mathbf u& +\cos(\varphi)\mathbf v
\end{align}
Now you get
\begin{cases}
\mathbf i_B &=& r_{L,\theta}(\mathbf i_A) &=& \cos(\varphi)\mathbf u-\sin(\varphi)\Big( \cos(\theta)\mathbf v+\sin(\theta)\mathbf k_A \Big)\\
\mathbf j_B &=& r_{L,\theta}(\mathbf j_A) &=& \sin(\varphi)\mathbf u+\cos(\varphi)\Big( \cos(\theta)\mathbf v+\sin(\theta)\mathbf k_A \Big)\\
\mathbf k_B &=& r_{L,\theta}(\mathbf k_A) &=& -\sin(\theta)\mathbf v+\cos(\theta)\mathbf k_A
\end{cases}
If you plug in the expressions of $\mathbf u,\mathbf v$ in the equation above, you'll obtain an expression of $\mathbf i_B,\mathbf j_B,\mathbf k_B$ in terms of $\mathbf i_A,\mathbf j_A,\mathbf k_A,\varphi$ and $\theta$. Basically if you can determine $\varphi$ and $\theta$, you'll get $\mathbf i_B$ and $\mathbf j_B$. In particular
$$
\mathbf k_B = \sin(\theta)\sin(\varphi)\mathbf i_A -\sin(\theta)\cos(\varphi)\mathbf j_A +\cos(\theta)\mathbf k_A
=c_{zx}\mathbf i_A+c_{zy}\mathbf j_A+c_{zz}\mathbf k_A
$$
so you can  fairly easily obtain $\varphi$ and $\theta$ in the general case.
Note that your problem statement ensures that we are in the general case, because the thing on the lines parallel to $\mathbf k_A$ guarantees that $\sin(\theta)\neq 0$, meaning that you can always find $\varphi$, from which you can always deduce $\theta$, and everything else from there.
I was too lazy to prove the thing on lines parallel to $\mathbf k_A$ then, and I'm still too lazy now. If you really need it and can't figure it out, ping me sometimes next week and I'll have a look.

EDIT: Assuming I didn't mess up somewhere above, you also get
$$
c_{xx}=\cos(\varphi)^2+\sin(\varphi)^2\cos(\theta)
$$
and
$$
c_{yy} = \sin(\varphi)^2+\cos(\varphi)^2\cos(\theta)
$$
so in general $c_{xx}\neq c_{yy}$.
Comment: @AsaturKhurshudyan : I'll be frank, I wouldn't have done it that way. Regarding your "check" question, $$\|\mathbf k_B\|^2=(\mathbf k_B.\mathbf i_A)^2+(\mathbf k_B.\mathbf j_A)^2+(\mathbf k_B.\mathbf k_A)^2=1$$ so the absolute values are equal. If there's a real problem, it's that if you use $\cos^2+\sin^2=1$ you cannot know how the sign behaves wrt to the other equations. Because of sign issues, the actual answer to your "check" is "yes" iff "$\pm\lvert\mathbf k_B.\mathbf j_A\rvert=\mp\mathbf k_B.\mathbf j_A$", otherwise "no". That condition can be re-stated as $\mathbf k_B.\mathbf j_A\le 0$.
Comment: I'd personally rather notice that without loss of generality, you can suppose $-\frac\pi2<\varphi<\frac\pi2$ thus $$\varphi=\text{atan}\left(\frac{\mathbf k_B.\mathbf i_A}{\mathbf k_B.\mathbf j_A}\right)$$ From $\varphi$ you get both $\cos(\varphi)$ and $\sin(\varphi)$ so you obtain $\sin(\theta)$.
Comment: @AsaturKhurshudyan It's just what the computation yields, that's nearly like asking why "$1+1=2$"... If you absolutely want to put some "meaning" behind that you should draw those vectors to understand the geometry of the problem, instead of just looking at equations. To put this in perspective, look at the simpler 2D case. I used $$\mathbf v=-\sin\theta\mathbf i_A+\cos\theta\mathbf j_A$$ but I could have written that out as $$\mathbf v=-(\mathbf u.\mathbf j_A)\mathbf i_A+(\mathbf u.\mathbf i_A)\mathbf j_A$$ Then why is it "$-(\mathbf u.\mathbf j_A)$" ? If you draw it for...
Comment: ... an angle $\theta$ between $0$ and $\frac\pi 2$, it'll come out as "obvious". Your question is basically the same but in 3D.
Comment: @AsaturKhurshudyan You're right, I read your comment too quickly. Sorry about that. But my overall answer is still "yes, that's what the computation yields" and "try drawing it", except that 3D drawings are a bit annoying. If you really aren't confident, just verify that your solutions satisfy $\mathbf i_B,\mathbf j_B,\mathbf k_B$ is an orthonormal basis.
Comment: It is clear that $c_{xx} \neq c_{yy}$. Let me check if I understand your formulas.
Comment: I found:
$$
\sin \theta \sin \varphi = \boldsymbol k_B \cdot \boldsymbol i_A,
$$
$$
-\sin \theta \cos \varphi = \boldsymbol k_B \cdot \boldsymbol j_A,
$$
$$
\cos \theta = \boldsymbol k_B \cdot \boldsymbol k_A.
$$
Thus,
$$
\sin \theta = \pm \sqrt{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2},
$$
$$
\sin \varphi = \frac{\boldsymbol k_B \cdot \boldsymbol i_A}{\sin \theta} = \pm \frac{\boldsymbol k_B \cdot \boldsymbol i_A}{\sqrt{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2}},
$$
Comment: $$
\cos \varphi = - \frac{\boldsymbol k_B \cdot \boldsymbol j_A}{\sin \theta} = \mp \frac{\boldsymbol k_B \cdot \boldsymbol j_A}{\sqrt{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2}}.
$$
Comment: Check,
$$
\cos \varphi = \pm \sqrt{1 - \sin^2 \varphi} = \pm \sqrt{1 - \frac{(\boldsymbol k_B \cdot \boldsymbol i_A)^2}{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2}} = \pm \sqrt{\frac{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2 - (\boldsymbol k_B \cdot \boldsymbol i_A)^2}{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2}}
$$
Is it the same as
$$
\mp \frac{\boldsymbol k_B \cdot \boldsymbol j_A}{\sqrt{1 - (\boldsymbol k_B \cdot \boldsymbol k_A)^2}}
$$
?
Comment: I am a bit lost in the issue with the sign. Assume that any vector $\boldsymbol a$ in $z_A y_A$ that has the same direction as $k_A$, is transformed into $\boldsymbol b$ in the $z_B y_B$ plane and has the same direction as $\boldsymbol k_B$. Moreover, $\boldsymbol a$ and $\boldsymbol b$ intersect. It means that $\boldsymbol k_B \cdot j_A \leq 0$, doesn't it?
Comment: Moreover, any vector $\boldsymbol c$ in $z_A x_A$ plane that has the same direction as $k_A$, is transformed into $\boldsymbol d$ in the $z_B x_B$ plane and has the same direction as $\boldsymbol k_B$. Again, $\boldsymbol c$ and $\boldsymbol d$ intersect. It means that $\boldsymbol k_B \cdot i_A \leq 0$, doesn't it?
Comment: Now I understand the sign issue, thanks.
Comment: The last question. In the expression for $\boldsymbol i_B$ I get $...−(\boldsymbol k_B \cdot \boldsymbol i_A) \boldsymbol k_A$, and for $\boldsymbol j_B$ I get $...−(\boldsymbol k_B \cdot \boldsymbol j_A) \boldsymbol k_A$. Why both are with "minus" sign?
Comment: What makes me confused is that I obtain $\boldsymbol i_B \cdot \boldsymbol j_A = \boldsymbol j_B \cdot \boldsymbol i_A$. Is it really the case?
Comment: @AsaturKhurshudyan That's not what you wrote in your last comments, and that's not the case in general (there are some specific values of $\varphi,\theta$ for which the equality is true).
Comment: I reduce it from your formulas after expressing $\boldsymbol i_B$ and $\boldsymbol j_B$ in terms of $\boldsymbol i_A$, $\boldsymbol j_A$ and $\boldsymbol k_A$.


Wordpress List Nested Pages With Page Content

Question: I am using this code to retrieve a nested list of pages in Wordpress:
<?php wp_list_pages( $args ); ?>

It gives me an output similar to this one:

Page Title 1

Page Title 2

Page Title 3

Page Title 4

Page Title 5

I need an output like this:

Page Title 1 / Page Content 1

Page Title 2 / Page Content 2

Page Title 3 / Page Content 3

Page Title 4 / Page Content 4

Page Title 5 / Page Content 5

Answer: I believe you'll need to use get_pages() instead, which will return an array of page objects. However, this wont give you the hierarchy you need to set up the sub menus. You could overcome this by only grabbing the top level items, then iterating through each and checking for sub pages:
<?php

$pages = get_pages( array(
    'sort_column' => 'menu_order',
    'parent' => 0,
));

?>

<?php foreach( $pages as $page ): ?>
    <?php $title = $page->post_title; ?>
    <?php $content = apply_filters('the_content', $page->post_content); ?>
    <?php $children = get_pages( array( 'child_of' => $page->ID ) );?>

    <?php echo $title . " / " . $content; ?>

    <?php if ( count( $children ) != 0 ): ?>

        <?php foreach( $children as $child ):
            <?php $title = $child->post_title; ?>
            <?php $content = apply_filters('the_content', $child->post_content); ?>

<?php echo $title . " / " . $content; ?>

        <?php endforeach; ?>

    <?php endif; ?>

<?php endforeach;?>


Return results for id if later record meets criteria

Question: Sample data:
date   id   case   type    case_review_date
11/1   1    90     Review  11/1
12/27  1    23     Block
1/1    1    12     Review  1/1
1/2    1    45     Failed
1/3    1    78     Review  1/3
1/3    3    62     Failed
1/1    2    33     Review  1/1
1/10   2    63     Failed

Goal is to only return rows where there is a Failed type for an id within 7 days of a Review date. So for this example, it should only return id 1 case 12. Using MySQL. Having a really hard time thinking through the logic on this one. Would really appreciate any help!
Attempted the following and it does not seem to be working correctly, or isolating it based on ID. Instead, it returns all Review entries if there is any Failed case within 7 days, regardless of if it is for the same ID or not. So it returns the following:
date   id   case   type    case_review_date
1/1    1    12     Review  1/1
1/3    3    62     Failed

select *
from casetable
where exists
   (select 1
    from casetable
    where casetable.type = 'Failed' and DATEDIFF(day, casetable.date, casetable.case_review_date) <= 7)

Comment: Could there ever be more than one review date?  If so, could you update your sample data to reflect this?
Comment: Yes, there can be multiple review dates. Will update to reflect.
Answer: You could use exists logic here:
SELECT date, id, `case`, type
FROM yourTable
WHERE
    type = 'Review' AND
    NOT EXISTS (SELECT 1 FROM yourTable t2
                WHERE t2.id = t1.id AND t2.type = 'Review' AND
                      t2.date < t1.date) AND
    EXISTS (SELECT 1 FROM yourTable t2
            WHERE t2.id = t1.id AND t2.type = 'Failed' AND
                  DATEDIFF(t2.date, t1.date) <= 7);


Returned data not showing in source code, How do I select Ajax Loaded Data?

Question: I am making a Category Tree Selection for eBay Categories.
I have a select box...

    <select id="slct_firstCat" name="slct_firstCat" size="33">
        <option value="353">Antiques</option>
        <option value="550">Art</option> ... and so on..
    </select>

I am using the following JQuery function to post data to my PHP application, which will return another Select box depending on which first category is selected (above).
// First Level Categories
    $("#slct_firstCat").change(function(){ // On change of slct_firstCat
    var actionRequested = "AJAX_getCats";  // My personal PHP Identifer
    var url = "index.php";                 // Post to index.php
    var catId = $("#slct_firstCat").val(); // Get the CatID from the select box.

    // Post this data to PHP and return the data into #div_secondCat
    $.post(url, {AJAX_Action: actionRequested, categoryId: catId, categoryLevel: 1},
        function(data){
            $("#div_secondCat").html(data);
           });
    // End of Function getCats
    });

The PHP File returns this select box, identical to above but different name.
<select id="slct_secondCat" name="slct_secondCat" size="33">
        <option value="353">Sub-Cat of Antiques</option>
        <option value="550">Sub-Cat of Antiques</option> and so on..
</select>

Now when I try to get a option value via JQuery from "slct_secondCat" (The select loaded via AJAX), It doesnt get the option value, so I cant navigate further down the cat tree.
This is the JQuery Function for the second category post.
// Second Level Categories
    $("#slct_secondCat").change(function(){
    var actionRequested = "AJAX_getCats";
    var url = "index.php";
    var catId = $("#slct_secondCat").val();

    $.post(url, {AJAX_Action: actionRequested, categoryId: catId, categoryLevel: 2},
        function(data){
            $("#div_thirdCat").html(data);
           });
    // End of Function getCats
    });

When I look in the source code of the page, after the AJAX loaded data, The div_secondCat doesnt display anything inside the source code, Even though the div has data loaded into it, the JQuery returned data from the post into the div and didnt update the source code of the page.
So Basically, I get categories from my PHP file via eBay web service, I want to navigate to the final category I want and be able to select it via JQuery.
Answer: Try
// Second Level Categories
$("#slct_secondCat").live("change", function(){

Also: Clicking "View Page Source" will only show you the original HTML document. You'll need to check in Firebug or Webkit Inspector to see what's really going on in the DOM.
Comment: Awesome. Glad you're up-and-running. Cheers.
Answer:
When I look in the source code of the page, after the AJAX loaded data

If you're talking about "view source" or similar, that's because it will show you the actual source of the page, not a view of what the DOM now looks like. For that, you need a proper debugger. Fortunately, they're thick on the ground, and free.
Regarding why you're not seeing the change event: When do you run your code to hook it up? Because this line:
$("#slct_secondCat").change(...

...will only hook up the handler if the element already exists. If it doesn't, the handler isn't hooked up. You can either call that function to hook it up after your previous ajax call that creates the element is complete, or use live (or better yet, delegate).
$("#slct_secondCat").live('change', ...

or
$("#div_secondCat").delegate("#slct_secondCat", 'change', ...

(Note in that last one that we're using div_secondCat in the initial selector, then calling delegate on that.)


scala - extracting and concatenating multiple elements from nested arrays of stings

Question: I am new to Scala and this is and could use some help. I have an RDD that is Array[Array[String]]. I need to extract the 1st element (ipaddress) and 4th element in each internal array (timestamp). Sample records look like:
results: Array[Array[String]] =
Array
(Array(126.96.36.199, -, 4712, [15/Sep/2013:23:56:06, +0100], "GET, /ronin_s4.jpg),
 Array(188.8.131.52, -, 28402, [15/Sep/2013:23:42:53, +0100], "GET, /titanic_2200.jpg),
 Array(184.108.40.206, -, 36171, [15/Sep/2013:23:39:33, +0100], "GET, /ronin_novelty_note_3.jpg)
)

I need the results to be:
184.108.40.206/15/Sep/2013:23:56:06
22.214.171.124/15/Sep/2013:23:42:53
188.8.131.52/15/Sep/2013:23:39:33

so I need to some thow replace the "[" with a "/"  and make a concatenated string of the 2 elemeents.  any ideas would be greatly appricated.
Comment: What have you tried so far? Where are you getting stuck?
Comment: I can extract the first element and the 4 element separetly... val IPaddress = results.flatMap(_.headOption)
val results2 = inputJPG.flatMap (x=>x.split(" "))
val Timestamp = results2.filter(x=>x.contains("["))
Answer: results.map { arr => { arr(0) + "/" + arr(3).replace("[", "").replace("]","") }}

This will return Array[String] in the format you state above
Comment: Perfect!  Thank you!
Comment: scala> results.flatMap {
     |   _.map {
     |     nestedArr => nestedArr(0) + nestedArr(3).replace("[", "").replace("]","")
     |   }
     | }
:32: error: value replace is not a member of Char
                  nestedArr => nestedArr(0) + nestedArr(3).replace("[", "").replace("]","")
                                                           ^
:31: error: type mismatch;
 found   : Array[Nothing]
 required: TraversableOnce[?]
                _.map {
                      ^


What is this group called?

Question: Let $X$ denote a set. There's a corresponding group obtained by taking the group freely generated by $X^2$ and then quotienting out by the following families of relations:

$(x,x) = 1$
$(x,y)(y,z) = (x,z)$
$(x,y)(y,x) = 1$
For each quadruple $(x,y,x',y')$ such that $\{x,y\} \cap \{x',y'\} = \emptyset$, we have:
$$(x,y)(x',y') = (x',y')(x,y)$$

Question. What is this group called?

My motivation for considering this group is that it acts on the set $(\{0,1\}^\mathbb{N})^X,$ of $X$-many binary streams, by interpreting $(x,y)$ as the act of taking the first digit of stream $x$, removing it from $x$, and appending it to the beginning of $y$. Each of the four families above can be explained in these terms; for example $(x,x)=1$ is saying that if I take the first digit of stream $x$, and put it back on stream $x$, nothing changes.
Comment: I think rule three is redundant since $(x,y)(y,x)=(x,x)=1$ where the first equality is rule 2 and the second equality is rule 1.
Comment: @ASKASK, good point :)
Comment: For what it's worth, I think I can prove that the action you described is faithful as long as $X$ is infinite.  The abelianization of your group is just the subgroup of the free abelian group on $X$ consisting of elements whose coefficients sum to $0$ (send $(x,y)$ to $y-x$).   Assuming the action is faithful and $|X|>2$, the commutator subgroup can be described as the group of finite-support permutations of $X\times\mathbb{N}$ (think of the elements of the commutator subgroup as permuting the set of all the bits in all your binary streams).
Answer: Not a complete answer, but too long for a comment:
The group you described has a neat equivalent characterization:
It can be described by the presentation $\langle a_1, a_2, ... , a_{|X| - 1}| [a_i, a_j] = [a_k, a_j] \forall i \neq j, j \neq k, k \neq i\rangle$
Proof:
Suppose, $X = \{0, ... , n\}$ ($|X| = n + 1$). Let's define $a_i := (0, i)$. That results in $(i, 0)$ being equal to $a_i^{-1}$ and for $i \neq j, j \neq 0, i \neq 0$ $(i, j) = a_i^{-1}a_j$. Now this change of notation already covers the first three rules, making them redundant.
Now, let's deal with the fourth rule. Without the loss of generality we can assume that one of the following cases takes place:
1)$x' = 0$, $x, y$ and $y' = z$ are pairwise distinct and non-zero. Then $(0, z)(x, y) = (x, y)(0, y')$, which turns into $a_{z}a_{x}^{-1}a_y = a_{x}^{-1}a_{y}a_{z}$ which results in $a_ya_za_y^{-1} = a_xa_za_{x}^{-1}$, or equivalently $[a_y, a_z] = [a_x, a_z]$
2)$x, y, x' = z$ and $y' = t$ are pairwise distinct and non-zero. Then $(x, y)(z, t)=(z, t)(x, y)$ turns into $a_x^{-1}a_ya_z^{-1}a_t = a_z^{-1}a_ta_{x}^{-1}a_y$ which is $(a_ya_z^{-1}a_y^{-1})(a_ya_ta_y^{-1})=(a_xa_z^{-1}a_x^{-1})(a_xa_ta_x^{-1})$ which is a direct corollary of case 1
Thus $[a_y, a_z] = [a_x, a_z]$ for all distinct $x$, $y$ and $z$ are the only non-trivial relations required.

What is this group called?


Volume and area of silids

Question: What should be the shape of elemental part while finding the volume of a given shape formed by revolving a 2-D curve, y = f(x). Should I take it as frustum of cone or a cylinder ? Which and why only that and not other ? If frustum are to be chosen as elemental part then why in some cases like Gabriel's cone, method of choosing cylindrical elemental part also works ? Please explain it in detail I am just a beginner in maths and having only a little knowledge of calculus ?
Answer: The reason we use disks or cylindrical shells to find the volume of a solid of revolution is that they are simple shapes and solids of revolution themselves.  The rules for the volume of a single disk or shell are simple.  Often the limits to use are simple as well.
Presumably when you suggest using frustrums you are thinking about a differentially thin outer shell of the frustrum of a cone, with all the cones having a common angle at the peak.  This is a fine approach if it accounts for all the volume of your solid and only accounts for each part of it only once.  The equation for the volume of an individual frustrum is a little more complicated than disks or cylinders, but if your shape is one that it is much easier to get the limits right because they fit it naturally, that is a fine idea.
Comment: Is simplicity is the only reason, but when I see a small part of curve as a line than it looks like that shape will always be a frustum and as I think there is huge difference in volume of frustum and cylinder , than the answer may not be quite accurate . Please explain.
Comment: You will just use short cylinders.  They are at a particular radius and extend from the point where that is the outer radius to the point where that is the inner radius.  Again the requirement is that you get each piece of the volume once.

Volume and area of silids


Best way of transferring songs that were recorded onto floppy disk using a Clavinova CVP-103

Question: We have a Clavinova CVP-103 which my wife used to record a bunch of music onto floppy discs years ago. I'm trying to work out the best way of ensuring she can still listen to  her music without having to play it through the CVP-103, as we would ideally like to replace it with something lighter, less bulky and more up-to-date.
I have some experience with midi, and own FL Studio and a Scarlett 2i4. Importing the midi files from the floppy discs didn't really work because of the way the piano records the music all as one track. Although I could get it to play in FL Studio, none of the voices were set up and it seemed like it would be a complete nightmare to sort out.
The only thing I can think of that would be remotely practical at this point would be to record them directly from the piano into FL Studio as audio files using an external microphone, but we don't own one and I'd rather not have to purchase something that we're never going to need to use again.
Are there any other options I've not thought about that might work?
Comment: Recording audio would work much better by using a cable connected to any output of the piano (line out, headphone out?), rather than a microphone. Could you provide more details about the files and the tracks, e.g. do the contain multiple tracks played simultaneously with different instrument sounds? Anyway, the very first thing to do is to backup all the floppy discs to some more modern storage space (e.g. your computer), as floppy discs, especially old ones, may not be too reliable.
Comment: It should save a MIDI type 0 file; which is a single track containing all the channels with correct voice/reverb setup at the head of each channel. I've no experience with FL, but a fairly basic requirement of a sequencer is to be able to sensibly spit this data back out with the correct channels.
Answer: If you decide to record the audio, an audio connecting cable would be easier, cheaper and sound better than a microphone.
Or you could persist with deciphering the MIDI files.   If you find it hard to work with a Type 0 MIDI track (single track containing multiple channels) in FL Studio, there are other programs.   MuseScore will do it.  (Don't be put off by it being primarily a notation program.  And it's free.)
Comment: Thanks for the tip about MuseScore. It made me look at Finale, which I have a copy of - that correctly splits the channels into separate staves, which makes the tracks much easier to work with


Pivotal quantities, test statistics and hypothesis tests

Question: We are learning pivot functions, test statistics, and hypothesis testing at university but it makes no sense. I've tried reading my text book/notes, going through examples, etc., but the concepts seem like a random guess and I'm clueless about how to even start guessing what the answer could be.
1st part
Can you please explain how to calculate a pivot function? E.g $X_{1},\dots,X_{n} \sim N(\mu, \sigma^2)$. Pivot function for $\sigma^2$ when $\mu$ is known and when $\mu$ is unknown. Why does $\mu$ being un/known matter?
Also how would you calculate the pivot function for the ratio of two variances ($\sigma_{x}^2$ and $\sigma_{y}^2$)? Is it an F distribution? Assume $\mu_x$ and $\mu_y$ are known and $X_1,\dots,X_n \sim N(\mu_x,\sigma_x^2)$ and $Y_1,\dots,Y_n \sim N(\mu_y, \sigma_y^2)$.
2nd part
Can you please explain how to calculate a test statistic (I get how to show it's a test statistic but don't know how to form one from scratch).
Last, I have a few questions about hypothesis testing. I don't really understand how to calculate the power of a test or even what it means, to be honest. There is a whole bunch of theory and many definitions out there but they are rather abstract so I don't get it at all... I don't understand the notation or how to calculate the size/power of a test (generic form - not just with numbers).
Example: $X \sim N(\mu, \sigma^2)$. $H_0: \mu >= \mu_0$ and $H_1: \mu < \mu_0$. $\mu$ and $\sigma$ unknown. Calculate the power and size of this test. How do I even start? I'm so confused. :(
I'm really stuck with all of this and I hope you can help me! :) if there's a better resource out there to help please do let me know.
EDIT1:
Thanks for your reply.
I did ask my lecturer to clarify… but ended up even more confused. He agrees the notes are unclear but will not rectify them because everyone else seems to get them! :( I also went to my class teacher and read through the Statistical Inference chapter several times – I get the basics but still don't really understand most of it. I have googled around – and read Wikipedia – but it's just more and more theory with no step by step examples explaining what to do. Everything seems randomly chosen and guesswork and hence my massive confusion.
One thing though – I can't read this:
$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$
Is it supposed to be in mathematical notation? How do I view it properly?
Yes- you are correct – the pivot function is used to calculate the confidence interval. The thing is – once I have the pivot it's quite straightforward to calculate the CI. But it's the pivot that's causing the problems.
I still don't get the following:
Pivot function for sigma^2 when mu is known and when mu is unknown. Why does mu being un/known matter?
How would you calculate the pivot function for the ratio of 2 variances (sigmax^2 and sigmay^2)? Is it an F distribution? Assume mu x and mu y are known and X1....Xn - N(mu x,sigmax^2) and Y1...Yn -N(mu y, sigmay^2). Is it 1/Fn-1, m-1 = Fm-1,n-1?
And the hypothesis testing questions above please…. Can you shed some light on this please?
EDIT2:
I did ask my lecturer to clarify… but ended up even more confused. He agrees the notes are unclear but will not rectify them because everyone else seems to get them! :( I also went to my class teacher and read through the Statistical Inference chapter several times – I get the basics but still don't really understand most of it. I have googled around – and read Wikipedia – but it's just more and more theory with no step by step examples explaining what to do. Everything seems randomly chosen and guesswork and hence my massive confusion.
One thing though – I can't read this:
$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$
Is it supposed to be in mathematical notation? How do I view it properly?
Yes- you are correct – the pivot function is used to calculate the confidence interval. The thing is – once I have the pivot it's quite straightforward to calculate the CI. But it's the pivot that's causing the problems.
I still don't get the following:
Pivot function for sigma^2 when mu is known and when mu is unknown. Why does mu being un/known matter?
How would you calculate the pivot function for the ratio of 2 variances (sigmax^2 and sigmay^2)? Is it an F distribution? Assume mu x and mu y are known and X1....Xn - N(mu x,sigmax^2) and Y1...Yn -N(mu y, sigmay^2). Is it 1/Fn-1, m-1 = Fm-1,n-1?
And the hypothesis testing questions above please…. Can you shed some light on this please?
Comment: There are good questions in here but you're asking too much at once.  Focus on one issue at a time and formulate a specific question for it.  Specify what you do know and what efforts you have made towards answering the question.  "How do I even start" doesn't give us anything to go on--we would just refer you to a textbook or Wikipedia (which is not a bad idea, by the way).
Comment: @Probability You don't need to attack me.  Let's be constructive.  The original post shares several characteristics of questions the FAQ specifies should *not* be asked, including "there is no actual problem to be solved" and "open-ended, hypothetical question[s]."  In its original formulation at least four distinct questions were posed, including the hugely general "how to calculate a test statistic".  This needed focusing to achieve appropriate, effective answers.
Comment: @whuber - I did not mean my comment as an "attack" but I could have phrased it a bit better, in hindsight.  I apologise for my poor choice of words in that respect.  What I meant was that the comment you have written would have been much better placed as the start of an answer, in which you focus your response on one particular aspect of the question.  The "how do I even start?" kind of question is a very good one I think, because this is often the kind of situation many statisticians actually face (I know I have).
Comment: @probability Thank you for clarifying.  You make a good point about the "how do I start" question.  It seems to me, though, that there is a fundamental difference between the student and the statistician: the student is supposed to have the the resources at hand to tackle the problem, which is supposed to have a definite answer.  "Resources" = teacher, textbook, etc.  At a minimum, then, the student should be able to describe their initial efforts and articulate the obstacles they are encountering.  If not, what are we supposed to do in response?  Write a textbook?
Comment: @probabiliy All the more power to you!  The vaguer the question, though, the greater the likelihood that all your good work will be of no interest to the OP because you didn't read their mind correctly ;-).  That can be frustrating...
Comment: @Whuber - this is a bit of a cop-out, @LSEactuary is not asking for the answer to everything.  You could easily just provide an answer to one piece of his question.
Comment: @LSEactuary There are too much questions here, and I would suggest to split your initial text in (at least) two parts, as suggested by my edit. The 2nd part deserves to be formulated in a new question (IMHO, but see also @whuber's comment), since only pivotal quantities have been addressed in this thread.
Comment: @whuber - I would suggest that "vague" questions can be given either "vague" answers, or they can specialise to a particular case - noting how it is linked to the "vaguary".  But as you say, one cannot be expected to give the answer to everything.  You could specialise the "how do I start?" question by giving a specific example, and saying how you would start in that particular case.  I had a go in my answer with the notion of "standardising" a distribution.  Presumably there are other ways to "start".
Answer: The first thing you should do is challenge your lecturer to explain these things clearly.  If anything whatsoever seems counter-intuitive or backwards, them demand that he/she explains why it is intuitive.  Statistics always makes sense if you think about it in the "right" way.
Calculating pivotal quantities is a very tricky business - I completely understand your bewilderment in "where should I start?"
For normal variance parameters, The "pivotal quantity" is the sum of squares divided by the variance parameters:
$$T_{X}=\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2} \sim \chi^{2}(N)$$
And a similar expression for $T_{Y}$.  Note that the distribution only depends on $N$, which is known (if $\mu_{X}$ is unknown, replace by $\overline{X}$ and you lose one degree of freedom in the chi-square distribution).  Thus $\frac{T_{X}}{T_{Y}}$ is a pivotal quantity, which has a value of:
$$\frac{\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2}}{\sum_{i=1}^{N}\Big(\frac{Y_{i}-\mu_{Y}}{\sigma{Y}}\Big)^{2}}
$$
Note that because it is a pivotal quantity, we can create an exact confidence interval using the pivot as a starting point, and then substituting in our statistic.  Now because the degrees of freedom are the same for each chi-square, we do indeed have an F distribution.  So you can write:
$$1-\alpha=Pr(L < F < U)=Pr(L < \frac{T_{X}}{T_{Y}} < U)$$
$$1-\alpha=Pr(L < \frac{\sum_{i=1}^{N}\Big(\frac{X_{i}-\mu_{X}}{\sigma{X}}\Big)^{2}}{\sum_{i=1}^{N}\Big(\frac{Y_{i}-\mu_{Y}}{\sigma{Y}}\Big)^{2}} < U)$$
$$1-\alpha=Pr(L < \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}}\frac{\sum_{i=1}^{N}(X_{i}-\mu_{X})^{2}}{\sum_{i=1}^{N}(Y_{i}-\mu_{Y})^{2}} < U)$$
Writing the observed ratio of the sum of squares as $R$ we get:
$$1-\alpha=Pr(L < \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}} R < U)$$
$$1-\alpha=Pr(\frac{L}{R} < \frac{\sigma_{Y}^{2}}{\sigma_{X}^{2}} < \frac{U}{R})$$
As for how this solution comes about, I have absolutely no idea.  What "principles" were followed (apart from being good at re-arranging statistical expression)?
One thing that I can think of is that you need to find some way to "standardise" your sampling distribution.  So for example, normals you subtract mean and divide by standard deviation.  For gamma you multiply by the scale parameter.  I don't know many pivotal quantities that exist outside of the normal and gamma families.
I think this is one reason why ordinary "sampling statistics" is an art more than a sciecne, because you have to use your intuition about what statistics to try.  And then you have to try and figure out if you can standardise your data.
I am almost certain your lecturer will bring up the subject of confidence intervals - be sure to ask him/her what you should do when you only have one sample, or when you have 2  or more nuisance parameters. :)
Comment: you might be cautious with your use of terminology, especially when addressing someone new to stats. For example, pivotal quantities are **not** statistics, so using the phrase "pivot statistics" can be (very) misleading. The OP uses "pivot functions", which strikes me as a little unconventional, but certainly more correct. Sorry if this seems pedantic of me. Cheers.
Comment: @cardinal - thanks for the comment.  I have updated my answer accordingly.


Drag&drop items limited

Question: is it possible to put limit to items for dragging? For example; only 3 items may dropped each boxes etc.
My Javascript codes:
<code>$("#DraggItems  .draggable-item").draggable({ helper: "clone", cursor: "drop-hover"});

$("#DraggItems").droppable({
     drop:function(event, ui) {
            hoverClass: "drop-hover",
            ui.draggable.detach().appendTo($(this));

     }
});

$("div.draggable-list").droppable({
     drop:function(event, ui) {
            hoverClass: "drop-hover",
            ui.draggable.detach().appendTo($(this));

     }
});
</code>
Answer: Yes, it is possible to put a limit to item for dragging.
<code> if($this.find('.drag').length >= 3){
    ui.draggable.draggable('option','revert',true);
    return;
 }
</code>
See this fiddle in action: http://jsfiddle.net/qqqm6/568/
Comment: Brilliant! Thanks so much @Sanyami
Comment: @Baris remember to mark the answer as correct by clicking the checkmark :-)

Drag&drop items limited


classification threshold in RandomForest-sklearn

Question: 1) How can I change classification threshold (i think it is 0.5 by default) in RandomForest in sklearn?
2) how can I under-sample in sklearn?
3) I have the following result from RandomForest classifier:
[[1635 1297]
 [ 520 3624]]
         precision    recall  f1-score   support

class 0       0.76      0.56      0.64      2932
class 1       0.74      0.87      0.80      4144

avg / total       0.75      0.74      0.73      7076
first, data is unbalanced (30% from class-0 and 70% from class-1). So, I think the classifier is more likely to be biased for class-1 meaning move some from class-0 to class-1 (there are 1297 missclassification for class-0 but 520 missclassification for class-1). How can I fix this? if downsampling can help? or changing classification threshold?
Update: class-0 has 40% of population while class-1 is 60%. However, drift from class-0 to class-1 (1297) is high while I want this becomes low.
Answer: You could indeed wrap you random forest in a class that a predict methods that calls the predict_proba method of the internal random forest and output class 1 only if it's higher than a custom threshold.
Alternatively you can bias the training algorithm by passing a higher sample_weight for samples from the minority class.
Comment: Thanks. I was thinking about increasing weight for minor class. However, I can not see anything in RandomForest classifier (there is in SGDclassifier)
Comment: The fit method accept a `sample_weight` param (one weight per parameter) that is very flexible and makes it possible to simulate `class_weight` (one weight per target class).
Comment: Thanks. when I use clf = clf.fit(X, Y, sample_weight=preprocessing.balance_weights(y) it gives me ValueError: operands could not be broadcast together with shapes. y is binary 0/1
Comment: What is the shape of `y`? Why do you have different `Y` and `y`?


marginal and conditional distributions

Question: A project has two phases. Let $X$ be the time (in months) required to do phase 1 and Y be the total time (in months) it takes to complete both phases. Suppose that the joint density of $X$ and $Y$ is $f(x,y) = c$, where $0 \leq x^2 \leq y \leq 1$, and otherwise $f(x,y) = 0$.
I found the value of the constant $c$ to be $\frac{3}{2}$, and now I'm trying to use this to find the marginal density of $X$ and the conditional probability that the total time is at least 3/4 of a month given that phase 1 takes 1/2 a month. But I'm stumped.
Comment: Are you sure about $x^2 \le y$? It looks strange
Comment: Sketch a diagram of the $x$-$y$ plane and indicate on it the region where the pdf is nonzero.  To find $f_X(\cdot)$ at a _fixed_ point $x_0$, you need to compute $\int_{-\infty}^\infty f(x_0,y) dy$ where the integral runs along the line $x=x_0$ in the plane. Draw several such lines for different values of $x_0$ and figure out the value of the corresponding integral.  See if you can figure out a pattern to fill in the blanks below: $$f_X(x)=\begin{cases} \quad & \quad \leq x \leq \quad,\\0,&\text{otherwise.}\end{cases}$$
Comment: The region does not seem consistent with the problem statement. Y (total time) can never be less than X (time of phase 1).
Comment: It is possible that the problem is as described, but the region described is not consistent with the natural condition that $Y\ge X$.
Answer: The marginal can be computed by integrating $y$ out.
$f_X(x) = \int f(x,y) dy = \int_{x^2 \le y \le 1} c dy = c(1-x^2), 0 \le x \le 1 $
The conditional probability can be found from the conditional pdf
$f(y|x) = \frac{f(x,y)}{f_X(x)}$
Answer: Your value for $c$ is correct. As for the marginal of $X$, you have everything you need, you just need to integrate the joint density over all possible values of $Y$. That is,
$$f_X(x) = \int_{-\infty}^\infty f_{X,Y}(x,y) \, dy.$$
It's harder to see why this is true for continuous random variables because the density functions don't actually give you probabilities. But, if you imagine that we're in discrete world for a second, then this is the equivalent to taking the sum of $P(X = x, Y = y)$ over all possible values of $Y$, and so what you're left with is just the distribution of $X$. In this case you're just integrating out the $y$'s instead of summing.
The trickiest part can be finding the right limits of integration. Since you're dealing with time, $x$ and $y$ both need to be positive (and satisfy the inequalities given in the definition of the joint pdf).
Finally, once you integrate and find $f_X(x)$ you can solve the next part of the problem. For that you'll want to find the conditional density of $Y$ given $X = x$, which is
$$f_{Y|X}(y|x)=\frac{f_{X,Y}(x,y)}{f_X(x)}.$$
In your case you will actually want the conditional distribution of $Y$ given $X = 1/2$, i.e.
$$f_{Y|X}(y\,|\,1/2) = \frac{f_{X,Y}(1/2\, ,\,y)}{f_X(1/2)}.$$
Then using this distribution you can find $P(Y \geq 3/4 \mid X = 1/2)$ by integrating $f_{Y|X}(y\mid \frac{1}{2})$ over the appropriate region.
I don't want to do all these computations for you (since it's a hw problem), but hopefully this gives you enough to get the ball rolling.
Comment: Thank you! Does this look good: $f_X(x) = \int_{y=x^2}^{y=1} \frac{3}{2} \, dy = \frac{3}{2}(1-x^2)$?  And then $f_{Y|X}(y\,|\,1/2) = \frac{3/2}{\frac{3}{2}(1-(1/2)^2)} = \frac{3/2}{9/8} = 4/3$. Then I can just integrate this from $y=3/4$ to $y=1$, right?
Comment: @jimmy Nice work!!

marginal and conditional distributions


dedekind cut for the square root

Question: Can somebody give me the hints to solve it ?

What is the Dedekind cut $(A, B)$ for  $\sqrt 2$ ?
What is the Dedekind cut $(C, D)$ for
$\sqrt 2 + \sqrt 3$ ?
In $\mathbb R[x]/(x^2 + 1)$, what is the value of $[x^3 + x^2 + x + 1]$?
(Just simplify $x^3 + x^2 + x + 1$)
where $[p]$ means the equivalence class that has $p$ as an element;
$[p]$ is the set of polynomials $q$ in $\mathbb R[x]$ such that
$x^2 + 1$ divides $p - q$.
$\mathbb R[x]$ is the set of polynomials whose coefficients are all real numbers.
Comment: Hint for Q1/Q2: A cut $(P, Q)$ for a real $\alpha$ is such that $P$ has all rationals less than $\alpha$ and $Q$ has all remaining rationals. For Q3, it should have been easy enough once you realize that $\mathbb{R}[x]/(x^{2} + 1)$ is isomorphic to $\mathbb{C}$ and $[x]$ plays the same role as $i$ plays in $\mathbb{C}$. This should help you out.
Answer:
Try $A=\{x\in\mathbb Q\mid x^2 <2\}\cup  \mathbb Q_{\le 0}$. Why is "$\mathbb Q_{\le 0}$" needed? And what must $B$ look like? Verify that the properties of Dedekind cut are fulfilled!
Try $C=\{z\in\mathbb Q\mid \exists x\in\mathbb Q\colon x<z, x^2<2, (z-x)^2<3\}\cup \mathbb Q_{\le 0}$, so $D=\ldots$ Verify that the properties of Dedekind cut are fulfilled! And why is this the cut for $\sqrt 2+\sqrt 3$?
There is not really a value of $[x^3+x^2+x+1]$, but it is possible to find a representative of minimal degree. Just perform polynomial division to obtani $x^3+x^2+x+1=(x^2+1)(x+1)+0$, i.e. $[x^3+x^2+x+1]=[0]$
Comment: That seems more than "hints." You've certainly left some steps out, but you've given away the hard work.

dedekind cut for the square root


How to read R F-test results?

Question:
I am trying to check that data adhere to the assumption of equal variance, so i can use a parametric test. But i do not know how to interpret the results of the Ftest.
Comment: I would recommend to read a stats 101 book and a R book, the question is too broad. which number you do not understand?
Comment: There are other ways of doing this which are often thought better like Levene's test or the Fligner-Killeen test.
Answer: Well, just like most other statistical tests, look at the p-value and as its value is 0.3569, it is much higher than your alpha of 0.05 and so you do not have enough evidence to reject your null hypothesis, so that means you can assume that the variances are equal.
I hope that was what you were in fact looking for.

How to read R F-test results?


Piano/keyboard versus synth

Question: I've been aiming to buy a keyboard for quite a while now, but just recently, I've been interested by synthesizers due to some new band I've been listening to.
I don't know much about these two instruments, but when I see people playing with synth, they do use a keyboard to play. I'm interested by the synthesizers for the large reach of sound and creativity it can get (low base or wierd sounds, etc.).
What I'm asking is :

A synthesizer seems to consist of a keyboard, with a lot more options to do a lot more than piano. If I'm interested in both instruments, should I then simply buy a synth, which will allow me to play both?

If not, what is the difference?
Answer: A synth is a machine that generates (synthesizes) sounds. There are many different types: old-school 80's style FM synthesizers, even more old-school modular synths like Moogs, software synthesizers that run on a computer, etc. Although some have a keyboard, that is not part of a synthesizer per se, that's just the controller.
A controller is a thing that can send note and other information to a synthesizer. The vast majority are keyboard controllers, but there are also drum pads, software, breath controllers, etc. All a controller does is send performance information.
Many controllers, especially keyboard ones, have built in synths, but they can usually also be used via midi cables to control other synths. For that matter, the synthesizer inside such hybrids can also be controlled by other controllers if desired.
So, if you get a keyboard controller, you will need some kind of synth to control with it if you want sounds. Although there are many awesome hardware synths out there, they are expensive, and there are fantastic software options which are relatively cheap if you already have a computer. Buying a hybrid might give you somewhat more portability, but if you have a laptop, it's not a huge difference.
EDIT TO ADD: for the most part, musicians end up collecting many different synths and controllers, especially synths. Different synths have vastly different sounds and strengths. Some have exotic sounds, some specialize in recreating traditional instruments, some favor pads while others favor leads. The majority have at least a few of all of these categories and more.

Piano/keyboard versus synth


SQL Server 2012 Management Tools - Basic not working

Question: I made a fresh install of SQL Server 2012 Express and Management Tools - Basic onto a new computer and none of the tools in the Integration Services folder will run. It gives the following message:

ERROR: The Execute Package Utility requires Management Tools - Basic or Business Intelligence Studio to be installed by the SQL Server 2012 Standard, Enterprise, Business Intelligence, or Evaluation Edition.

Its the Execute Package Utility that I'm after, but the Deployment Wizard and Project Conversion Wizard also give the error.
I've tried uninstalling and reinstalling but that doesn't seem to make a difference. This is on Windows 7 Pro 64 bit computer.
Answer: It seems obvious now, but the problem was that I was installing a SQL Server Express version. Even though it installed the needed components it would not run them. I downloaded the Eval version and it worked.
In my defense, the Microsoft SQL Server 2012 site does list SSDT as included with Express with Advanced Services:
https://msdn.microsoft.com/en-us/library/cc645993(v=SQL.110).aspx
Comment: Just a comment for anyone else really - if you get any issue with an older version of SSMS you can now download the latest bits for free without the full sql install. It is now at version 17 and changes are happening pretty quickly. The current download page is https://learn.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms "Download SQL Server Management Studio (SSMS)". The latest bits are backwards compatible so always get the latest (unless your using query analyzer and sql 7!) :)
Comment: Do the individual bits include the Integration Services and SSDT? I need to build and run 2012 SSIS packages.


AngularJS - For loop to push whole object than just title

Question: I have created a plunkr here: http://plnkr.co/edit/wPCex3lJc5E0I4fSeXI1?p=preview
The application should allow the user to perform a search, and return results.
I have got to the point where the number of correct results are shown (if just a country is selected), but i cant seem to pass the whole matching object in my for loop.
HTML:

    {{myresults}}

JS:
$scope.Results = [];

$scope.search = function(country, city, department) {
    var countryValue = country.Country;
    for (i = 0; i < $scope.Data.length; i++) {

      if ($scope.Data[i].Country == countryValue) {
        $scope.Results.push(countryValue);
      }
    }

    console.log($scope.Results);
  }

Answer: You are only pushing the country name in $scope.Results, push the whole data object instead:
$scope.Results.push($scope.Data[i]);
http://plnkr.co/edit/DH8DBUKoWKITIyaEH1yD?p=preview


Can a random forest model be extracted for practical use?

Question: I'm new to random forest models and have questions.
When I develop a linear model, I can tell people "Hey kids- here's how to predict the circumference of a circle if you know the radius.  Just plug the radius into this handy formula: A = 2 * pi * radius.  They don't have to know any stats at all to go start calculating circumferences.
But, if I have a random forest model, how can I communicate to folks how they can predict their own values with their own data?  Do I have to distribute my original dataset to them so they can run an R/Python script?  Or is there a way to extract foresty-equivalent of coefficients and slopes?  I suppose it would be the decision tree somehow. Could it ever get to something like a convoluted Excel formula I could share in a spreadsheet template?  How does the average person (not the statistician) actually use these things in the real world?
Answer: You have several options:
1) Loading trained model
Frameworks like sklearn make it easy to persist the trained estimator to disk, and then to load it up afterwards. One would distribute a Python script which uses the trained model, as well as doing any data pre and post-processing. The training data is then not needed.
From the sklearn documentation:
Training
from sklearn import ensemble, datasets
from sklearn.externals import joblib
iris = datasets.load_iris()
X, y = iris.data, iris.target
clf = ensemble.RandomForestClassifier()
clf.fit(X, y)
joblib.dump(clf, 'rf-model.pkl')

Predictions
from sklearn.externals import joblib
clf = joblib.load('rf-model.pkl')
clf.predict(Xnew)

Implementations of Random Forest will have a way to get the trained cofficients, but the details vary a lot. Fundamentally a Random Forest classifier is a number of trees. Each tree does classification by executing a bunch of if-else statements, each checking whether one of the variables is below. At the leaf is the returned class label. The trees are combined using (weighted) majority voting.
2) Generating code
Using emtrees one can export random forests trained with sklearn as C code. A bit simplified, it looks like:
int32_t digits_predict_tree_0(int32_t *features, int32_t features_length)
{
  if (features[36] < 32768) {
      if (features[57] < 65536) {
          return 9;
      } else {
          if (features[10] < 950272) {
              return 5;
  ....
}

int32_t digits_predict(int32_t *fe, int32_t features_length)
{
   _class = digits_predict_tree_1(fe, length); votes[_class] += 1;
   _class = digits_predict_tree_2(fe, length); votes[_class] += 1;
   ....
   return find_majority(votes);
}

The same could be done for other languages, theoretically even an Excel macro.
3) Providing webservice
A modern alternative is to set up a simple webservice where users can drop their input data (as .CSV/.xsl), and then download the returned predictions as a new file.
Comment: Didn't know about the compile to C thing.  Very cool.


spawn ffmpeg in nodejs and pipe to express's response

Question: i am spawning ffmpeg and pipe it's output (the video stream i want) to express's response object like this:
<code>app.get('/stream', (req, res) => {
    let _url = req.query.url;

    if(_url){

        res.writeHead(200, {
            'Access-Control-Allow-Origin': '*',
            'Connection': 'Keep-Alive',
            'Content-Type': 'video/mp4'
        });

        // transcode rtsp input from ip-cam to mp4 file format (video: h.264 | audio: aac)
        let ffmpeg = child_process.spawn("ffmpeg",[
            "-probesize","2147483647",
            "-analyzeduration","2147483647",
            "-i", _url,
            "-vcodec","copy",
            "-f", "mp4",
            "-movflags","frag_keyframe+empty_moov+faststart",
            "-frag_duration","3600",
            "pipe:1"
        ]);

        // redirect transcoded ip-cam stream to http response
        ffmpeg.stdout.pipe(res);

        // error logging
        ffmpeg.stderr.setEncoding('utf8');
        ffmpeg.stderr.on('data', (data) => {
            console.log(data);
        });
    }
    else{
        res.end();
    }
</code>
so far it's working like a charm.
however somewhere in there seems to be a cap. i can get only 3 streams running at the same time. when a 4th spawn occurs the 4th thread will block one of the cpu cores:

and naturally the 4th stream does not reach the browser..
has somebody any idea of what am i missing?
EDIT: it's not tied to the fact that i am running the nodejs project on the raspberry pi. it also behaves the same way on my windows 10 machine
Answer: Jesus. Alright here is the problem:
My IP-camera can only deliver 3-4 streams at the same time anyway...


How to encode the extra latin1 characters in smpp?

Question: How are characters in latin-1 encoded in submit_sm or deliver_sm in the case of extended characters?
Like: ¶ µ ÷
Is there a proper way to convert a string to latin-1 characters?
Answer: Ended up using an SMS splitter code & altered it to fit latin-1 characters:
https://cs.stanford.edu/people/miles/iso8859.html#ISO
And it only has 140 char compared to 160 GSM, so be mindful when using.
It'll differ for each problem, but for mine, I couldn't find an existing solution.


TSQL-failing to parse XML(namespaces)

Question: Here is a my SQL:
    create table sqm (data xml)
    insert into sqm
    select '<DataSet xmlns="http://www.bnr.ro/xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bnr.ro/xsd nbrfxrates.xsd">
            <Cube date="2017-06-30">
                <Rate currency="AED">1.0867</Rate>
                <Rate currency="AUD">3.0665</Rate>
                <Rate currency="BGN">2.3284</Rate>
            </Cube>
            </DataSet>'

    select
    m.c.value('@date', 'date') as valuta
    from sqm as s
    outer apply s.data.nodes('/DataSet/Body/Cube') as m(c)

After spending hours trying to find out why my SQL kept returning NULL in the db, I discovered that my problem was due to Hyperlink references at the very beginning of the XML(after DataSet).
I really want to know why is this happening and who can I delete everything between <DataSet end >. Is there any other option?
if you can think of a better title, please edit.
Answer: You need to declare the namespace using WITH XMLNAMESPACES
Also your example XML has no Body element so I removed that from the Xpath.
Demo
WITH XMLNAMESPACES (DEFAULT 'http://www.bnr.ro/xsd')

select
m.c.value('@date', 'date') as valuta
from sqm as s
outer apply s.data.nodes('/DataSet/Cube') as m(c)

Or alternatively you can use
select
m.c.value('@date', 'date') as valuta
from sqm as s
outer apply s.data.nodes('/*:DataSet/*:Cube') as m(c)

Comment: @s.dragos It needs to be part of the same statement that uses the Xpath
Comment: I see, but where should the namespace be put when i parse an xml from URL like [here](https://stackoverflow.com/questions/18054652/how-can-i-to-read-a-xml-from-a-url-using-t-sql) Thanks in advance.

TSQL-failing to parse XML(namespaces)


VIACEP não achou a variável CEP para retornar ao resultado em PHP

Question: Olá,
Apenas a pesquisa de CEP para achar a cidade e o estado funciona, mas a pesquisa de cidade e estado [e bairro] para achar o CEP não funciona porque no JSON, a variável CEP ($result->cep) está dentro das chaves da classe numérica e não existe classe de estado-cidade.
Eis em PHP:

if ($_SERVER['REQUEST_METHOD'] === 'POST')
{

 function webClient($url)
 {
     $ch = curl_init();

     curl_setopt($ch, CURLOPT_URL, $url);
     curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
     curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
     curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

     $data = curl_exec($ch);

     curl_close($ch);

     return $data;
 }

 $descubra = $_POST['descubra'];

 switch ($descubra)
 {
  case "lugar":
   $cidade = $_POST['cidade'];
   $estado = $_POST['estado'];
   $bairro = $_POST['bairro'];
   $url = sprintf('https://viacep.com.br/ws/%s/%s/%s/json/ ', $estado, $cidade, $bairro);
   $result = json_decode(webClient($url));
   echo $result->cep;
   break;

  case "ceplocal":
   $cep    = $_POST['cep'];
   $url = sprintf('https://viacep.com.br/ws/%s/json/ ', $cep);
   $result = json_decode(webClient($url));
   echo $result->localidade;
   echo $result->uf;
   break;

  default:
    echo "Inválido!";
 }

}

Answer: Na verdade seu código está funcionando, note o que diz a documentação sobre pesquisa por endereço:

O resultado será ordenado pela proximidade do nome do logradouro e possui limite máximo de 50 (cinquenta) CEPs.

Então sua pesquisa irá buscar todas as ruas que contém X palavra, e irá retornar um array, veja o exemplo abaixo:
$cidade = 'Americana';
$estado = 'SP';
$bairro = 'Frezzarin';

Terá o retorno:
Array
(
    [0] => stdClass Object
        (
            [cep] => 13467-019
            [logradouro] => Praça José Frezzarin
            [complemento] =>
            [bairro] => Jardim São José
            [localidade] => Americana
            [uf] => SP
            [unidade] =>
            [ibge] => 3501608
            [gia] => 1650
        )

    [1] => stdClass Object
        (
            [cep] => 13465-789
            [logradouro] => Praça Marcílio Frezzarin
            [complemento] =>
            [bairro] => Vila Frezzarin
            [localidade] => Americana
            [uf] => SP
            [unidade] =>
            [ibge] => 3501608
            [gia] => 1650
        )
    ...
)

Para usar o echo tem que informar o índice $result[0]->cep e assim por diante.

Podes ver funcionando em repl.it
Comment: eu também utilizo o CEP Aberto e observo que ele não possui `array`. Verifica: https://repl.it/@Gustavo_Benedit/NaoExisteArrayNoCEPAberto.


CodeIgniter session not writing to the DB after PHP version upgrade to 7.2.10

Question: I upgraded my PHP version from 5.6 to 7.2.10 and suddenly writting session data to MySQL database stopped working (the session table is empty). I also upgraded CodeIgniter version to 3.1.9 (the system folder only) but this didn't help. I think the problem may be in some configurations for PHP (or MySQL) because the first thing I have to do was to change autostart session in php.ini.
When I retrieve value from $this->config->item('sess_driver') I get "database". When I retrieve values from $this->session I get:
<code>object(CI_Session)[14]
  public 'userdata' => null
  protected '_driver' => string 'files' (length=5)
  protected '_config' => null
  protected '_sid_regexp' => null
</code>
That is strange because in older PHP version I get "database" as _driver value. It looks like the session data are stored in files - when I ask for $this->session->has_userdata('id') I get the correct value but it should be saved in database session table but it is not.
This is my part of config:
<code>$config['sess_driver'] = 'database';
$config['sess_cookie_name'] = 'ci_session';
$config['sess_expiration'] = 7200;
$config['sess_save_path'] = 'sessions';
$config['sess_match_ip'] = FALSE;
$config['sess_time_to_update'] = 300;
$config['sess_regenerate_destroy'] = FALSE;
</code>
Does somebody meet my problem or know the solution? Thanks
Comment: do you have the Session table in the database with the correct (required) fields?
Comment: Yes, of course I do. Otherwise I would get a DB error.
Comment: I think they changed the default database to MariaDB
Comment: mhh. I think this question should be asked or maybe even posted as an issue at the CodeIgniter GitHub repo
Comment: I'm using CI v3.1.9 and PHP v7.2.10 and it works fine. Is there a config file in an environment folder that isn't the one you show? That's one (only?) logical reason that `protected '_driver'` would say `'files'` but the config you show is set to 'database'.
Comment: @DFriend Everything happened when I upgrade my wampserver to 3.1.4. Maybe I should change something in php.ini or ...?
Comment: Upgraded from what version to 3.1.4?
Answer: I might suggest you solution.

Go to codeigniter website.
download lastest version of codeiginter.
extract it.
make rename system folder to something else in existing project.
paste new system folder and you done it.


Overhead of "boxing" primitive types via implicits in Scala

Question: Suppose I want to have a class like Java Date. Its only data member is a long which represents the milliseconds since 1970.
Would/Could it be of any performance benefit of just making a new Scala type:
type PrimitiveDate = Long

Then you can add methods by using implicit conversion, like it is done for int with RichInt. Does this "boxing" of the primitive type into a rich class involve any overhead (class creation)? Basically you could just have a static method
def addMonth(date: PrimitiveDate, months: Int): PrimitiveDate = date + 2592000000 * months

and let the type system figure out that it has to be applied when d addMonth 5
 appears inside your code.
Edit
It seems that the alias you create by writing type PrimitiveDate = Long is not enforced by the scala compiler. Is creating a proper class, enclosing the Long, the only way to create an enforced type in Scala?
How useful do you consider being able to create an enforced type alias for primitive types?
Answer: Well, escape analysis should mean that the most recent JVMs do not actually have to create your rich wrapper in order to call the addMonth method.
The extent to which this actually occurs in practice will obviously depend on how much of a runtime hotspot the JVM decides that these methods are adding in object creation. When escape analysis is not happening, obviously the JVM will have to "box" (as you say) the Long in a new instance of the wrapper class. It doesn't involve "class creation" - it would involve "creating an instance of a class". This instance, being short-lived would then be GC-d immediately, so the overhead (whilst small) is:

memory allocation for the instance
GC-ing the instance

These are obviously only going to be any kind of problem if you are definitely writing very low-latency code where you are trying to minimize garbage creation (in a tight loop). Only you know whether this is the case.
As for whether the approach will work for you (and escape analysis come to your aid), you'd have to test in the wild. Micro-benchmarks are notoriously difficult to write for this kind of thing.

The reason I don't quite like these type aliases being part of a public API is that scala does not really enforce them as strictly as I would like. For example:
type PrimitiveDate = Long
type PrimitiveSpeed = Long
type Car = String
type Meeting = String

var maxSpeeds : Map[Car, PrimitiveSpeed] = Map.empty

//OOPS - much too easy to accidentally send the wrong type
def setDate(meeting : Meeting, date : PrimitiveDate) = maxSpeeds += (meeting -> date)

Comment: @ziggystar - no there is not. I tend to use type aliases for one of two things: private internal stuff like Kevin talks about below; package object imports (i.e. why `List` still seems to be in the `scala` package)
Comment: Good point on the "enforcing aliases". That was one thing I was wondering about. So there is no way of creating a proper type alias for a primitive type in scala?
Comment: I think Miles Sabin came up with an elegant and performant [solution](https://gist.github.com/milessabin/89c9b47a91017973a35f) 11 months after this question was asked.
Answer: You haven't actually created a new type in your given example, it's simply an alias for the pre-existing Long type.
It's a technique I use quite often to deal with unwieldy nested connections.  For example, I'd alias type Grid = Seq[Seq[Int]] to avoid having to specify Seq[Seq[Int]] over and over again for various parameters.
You can quite happily pass a Long to a method taking a PrimitiveDate method, although you do have the advantage that the code is much better self-documented.
If you really do want to create a new type with enforced type-safety and convenient pattern matching, I'd use a case class:
case class PrimitiveDate(value:Long)

and, potentially, even provide an implicit Long=>PrimitiveDate conversion for convenience.
Answer: 11 months after you asked this question, Miles Sabin discovered a very simple, elegant and performant way of creating unboxed newtypes in Scala.  Unlike type aliases, type tags are enforced.  Primitive types need minimal boilerplate (one line per primitive) to provide specialization.
A year later, he added a more polished and robust version of this to Shapeless.  The concept is simple and concise enough to be duplicated in a project without adding Shapeless if you don't want the rest of that excellent library.
Of course, both you and the people who answered your question probably know this, but it's worth adding here because this is still an important question.
Comment: I suppose these unboxed newtypes are similar to the phantom types mentioned [here](http://stackoverflow.com/questions/6358651/marking-primitive-types-with-phantom-types-in-scala/6360260#6360260), right?


Reparametrization and Path integrals

Question: Let c(t)=(cost,sint) for 0≤t≤2π and let p(t)=(cost,sint) for 0≤t≤4π. Is it true that ∫c  F⋅ds=∫p  F⋅ds for every continuous vector field F on {R}^2?
*∫c  F⋅ds=∫p  F⋅ds are both path integrals.
Choices are:
a. True for all vector fields F
b. True for some vector fields F and false for others
c. False for all vector fields F
d. It's complicated
Intuitively, I thought a is correct as the endpoints of both parametrization matches, but it seems to be not the case (as indicated by my homework grading). Right answer seem to be b, but I have zero idea why.
Thanks to everyone!
Answer: Even though the endpoints hold, the path $c(t)$ corresponds to going around the unit circle one time, while $p(t)$ corresponds to going around the unit circle twice. As such, we expect the line integrals (the term path integral is usually reserved for a more complicated type of integral) along $c(t)$ and $p(t)$ to be different.
However, there are examples of vector fields where the line integrals are the same. Try to come up with one yourself if you haven't!
Comment: Oops! I meant $F(t) = (\cos(t),\sin(t))$, not $(-\sin(t),\cos(t))$.  $(-\sin(t),\cos(t))$ is the direction of $ds$, so that wouldn't work of course. My bad.
Comment: Thank you very much! Would $F(x,y)=(-2,2)$ work in this case?
Comment: Yup! Actually, any constant vector field would work, as you may check. For an example of a non-constant vector field, see $F(t) = (-\sin(t), \cos(t))$.

Reparametrization and Path integrals


No L' Hospital $ \lim_{x \rightarrow 0} \left (1+\frac {1} {\arctan x} \right)^{\sin x}$ and $ \lim_{x \rightarrow 0} \frac {\tan ^7 x} {\ln (7x+1)} $

Question: What would be the best approach to calculate the following limits
$$ \lim_{x \rightarrow 0} \left (1+\frac {1} {\arctan x} \right)^{\sin x}, \qquad \lim_{x \rightarrow 0} \frac {\tan ^7 x} {\ln (7x+1)} $$
in a basic way, using some special limits, without L'Hospital's rule?
Comment: One question per post.
Comment: For the second limit, note that $$\tan x\sim\ln(1+x)$$as $x\to0$.
Answer: Use the standard limits
$$\lim\limits_{x\to 0}\frac{\ln(1+x)}{x}=1$$
and
$$\lim\limits_{x\to 0}\frac{\sin x}{x}=1$$
In the first example take a log first.
Note that the later implies that
$$\lim\limits_{x\to 0}\frac{\tan x}{x}=1$$
and thus that
$$\lim\limits_{x\to 0}\frac{\arctan x}{x}=1$$
The first limit can be written as
$$\ln \left(1+\frac{1}{\arctan x}\right)^{\sin x}=\frac{\sin x}{x}\frac{\ln(1+\frac{1}{\arctan x})}{\frac{1}{\arctan x}}\frac{x}{\arctan x}$$
Comment: Does the first standard limit you suggested apply here? Limit of $\frac{1}{\arctan x}$ as $x$ goes to $0$ actually doesn't exist. Could you tell me  how I should evaluate the expression in the middle?
Comment: @Theta: you are correct. The middle expression tends to $0$ as $x\to 0^{+}$ via the standard limit $\lim\limits_{t\to\infty} \dfrac{\log t} {t} =0$. Thus the first limit in question is $1$ but you must consider $x\to 0^{+}$.
Answer: For the second:
$$\frac {\tan ^7 x} {\ln (7x+1)}=\frac {\tan ^7 x} {x^7}\ \frac {x^7} {7x} \ \frac {7 x} {\ln (7x+1)}=1\cdot0\cdot1=0$$
Answer: For the first limit being $x\to 0$ we have $\arctan x\sim x$, hence:
$$\lim_{x \rightarrow 0} \left (1+\frac {1} {\arctan x} \right)^{\sin x}\sim \lim_{x \rightarrow 0} \Biggl[\left (1+\frac {1} {x} \right)^x\Biggr]^{\displaystyle \frac{\sin x}{x}}=e^1=e$$
For the second limit being $x\to 0$ we have $\tan \psi(x)\sim \psi(x) \ $ and $\ln \gamma(x) \sim \gamma(x)$, hence:
$$\lim_{x \rightarrow 0} \frac {\tan ^7 x} {\ln (7x+1)}\sim \lim_{x \rightarrow 0} \frac{x^7}{7x+1}=0$$
Answer: The second is:
$$ \lim_{x\to0}\frac{\sin^7x}{\cos^7x}\frac{1}{\log(7x+1)}= \lim_{x\to0}\frac{\sin^7x}{\cos^7x}\frac{1}{\log(7x+1)}\frac{x^7}{x^7}\frac{7x}{7x}=0$$
For the first use the substitution method.
Answer: A solution for the first by Taylor series:
we can write the limit as follow:
$$\left (1+\frac {1} {\arctan x} \right)^{\sin x}=e^{sinx \ \log{\left (1+\frac {1} {\arctan x} \right)}}$$
Calculate Taylor series expansion for each term at the first order:
$$\sin x = x+o(x)$$
$$\log{\left (1+\frac {1}{\arctan x} \right)} =\log{\left (\frac {1+ \arctan x}{\arctan x} \right)} =-\log{\left (\frac {\arctan x}{1+\arctan x} \right)}\\ =-\log{\left (\frac {x+o(x)}{1+x+o(x)} \right)} =-\log{\left [(x+o(x))\cdot(1-x+o(x)) \right]} =-\log{(x+o(x))}$$
Thus:
$$\sin x \ \log{\left (1+\frac {1}{\arctan x} \right)}=(x+o(x))\cdot [-\log{(x+o(x))}]=-x \log x + o(x)\to 0$$
Finally:
$$\left (1+\frac {1} {\arctan x} \right)^{\sin x}\to e^0 =1$$