In making scatterplot for correlations between two continuous variables, can we use the choice cubic instead of linear choice

Question: In making scatterplot for correlations between two continuous variables, can we use the choice cubic instead of linear choice in "create a fit line at total", as shown in the copied Figure, please? does this sort of Figure acceptable scientifically to show negative correlation?
Comment: It seems that your SPSS plot shows a regression, not a correlation. As you know, (ordinary) regression assumes (virtually) no error in the predictor, X. In correlation, no such assumption is made, both variables (X1 and X2, if you like) are uncertain. The correlation line must hence minimise distance to observed values perpendicular to the line itself. Model 2 regressions (as a special case of "error-in-variable" models) can fit such models, but I have only ever seen linear ones. See also https://stats.stackexchange.com/questions/159261/errors-in-variables-multivariate-polynomial-regression-r
Answer: Yes and no, mostly no. Correlation asks for how strong the linear relationship is between two variables.  Although a cubic fit to data may be more explanatory  than a linear one, that does not mean that we can extract a better correlation from a cubic, not directly anyway. For example, for a cubic under the conditions of relative monotonicity the linear term (proportional to $x$) and cubic term (proportional to $x^3$) are more  related to correlation in some transformed coordinate system, and the squared term (proportional to $x^2$) often less so. However, that argument would likely be rejected if it were, for example, included in an article for publication because such methodology has not been characterized, and more direct approaches for obtaining improved correlation are available.
More typically, to extract a better correlation from data one would either 1) do significance testing of a Spearman's rank correlation or related non-parametric test (like Kendall's $\tau$). Such tests, since they are rank tests, and since the data shown are approximately monotonic, would give a test of significance under the assumption of monotonicity.  However, more appropriate would likely be to 2) transform the coordinate system so that the plot is more linear, which in the case of the data shown could be log-log transformation or reciprocal plotting. However, this requires trying a number of transforms to see which is best at rendering the plot more linear. Even after transformation of coordinate systems, Spearman's rank may  be more appropriate than Pearson's correlation, such that both could be tried.
Comment: Could you please explain what you mean by a term in a regression being "related to correlation"?
Comment: Because your replies make no sense to me, let me try again.  I understand "correlation" to refer to the Pearson correlation coefficient.  You appear to make statements about some kind of relationship between including or removing variables from a multiple regression, the regression results, and correlation.  What do they mean?
Comment: I would be happy to, however, I do not see exactly what you are asking. I will take a guess, if the assumption of relative monotonicity is exact, then odd terms of a poly have that property, and even powers do not. As I mentioned, that is not an approach that I, or in fact anyone I know, has ever suggested as being a method worth considering,  but that is what the question asked, and any other response would not actually address the question, i.e., "Why can't I do this?"
Comment: @whuber.... for example, if the linear term cannot be discarded, and is negative, but the cubic term is positive and cannot be discarded, then any statement regarding correlation is discounted. To put it another way, "We do not do that because it is at best poorly characterized if not totally incorrect."
Comment: @whuber I did not even consider adding or removing terms from a poly regression, that is yet another reason to NOT use a poly to explore correlation. I added some text to try to help. The answer is because it hasn't been characterized so no one knows how to do that at present. Moreover, we have better methods; so why bother?
Comment: @whuber It is not the answer that makes no sense, it is the question. I am trying to explain why the cubic is not used to explain correlation,  and the simple answer is that it does not, but possibly could under a set of conditions that has not yet been explored, and for which the exploration would be at least in part, unmotivated, as there are more direct ways of examining data for correlation.
Comment: @whuber If you do not like my answer, answer it yourself. So I put it to you, "Why not use a cubic to explain correlation?"
Comment: @whuber If a cubic is better at explaining data than a linear equation, then there is probably a data transformation that has a better correlation coefficient than the un-transformed data. This improvement in correlation is not something that has been related back to the cubic per se. Could it be? Maybe yes, maybe no.
Comment: Taking "correlation" in a general sense as some kind of association, I still struggle to understand your claim that a quadratic term is not "related to correlation." For instance, suppose $E[Y\mid x] = x^2$ for a set of $x$ in $[1,2].$ How is that not "correlation" in whatever meaning you are implicitly adopting?
Comment: @whuber You are correct that $x$ on [1,2] is a more linear portion of $x^2$ than [-1,1]. The point I am trying to stress is that if something non-linear (including non-trivial cubics) fits data better than a linear function would, then some transform of the data has a higher correlation in the "transformed" space. Cubics are a particularity confusing choice to show this. Your example is more simple. For that, one could take the square root of $y$ and the plot then becomes |x|, which is linear only if (0,0) in not included.
Comment: @whuber This version any better? Why did you  close the question? It is a creative question of the type that shows a bit of spunk. Hard to answer, but sometimes another set of eyes, like yours, helps,  thank you.
Comment: Because our conversation showed we could not find a common intelligible interpretation of the question, I concluded other users would have similar difficulties.
Comment: @whuber OK, thanks anyway. I may be reading too much into it, and it probably would be going too far to edit the question to make it clearer.
Comment: @whuber BTW see http://wmueller.com/precalculus/newfunc/invpoly.html re inverse cubic.
Comment: Carl, most mathematical professionals would find it offensive to be directed, without comment, to a high school textbook: that could easily be taken as a willful insult.
Comment: @whuber No offense intended, apologies. I liked it *because* it was simple. What I was trying to point out is iff monotonically for the cubic fit is present and if a cubic is a better fit than a linear fit then one could use the inverse cubic transform of the $y$ values coordinates to construct a more linear plot with higher correlation than the original.
Comment: @whuber That may be an answer to the question. So, I thought I should run it by you. Again, apologies.
Comment: Pending clarification from the OP, I cannot tell whether your idea answers the question they intended.  Ordinarily one would be reluctant to apply such a nonlinear transformation to the responses $Y$ because it changes (perhaps profoundly) the conditional distributions.  Alternative methods, such as cubic splines, might be more appropriate, depending on the model.


Logistic regression not producing an s-shaped curve

Question: I am trying to create a model that shows on the y axis a range from 0-1 and get that distinctive binary dependent variable s-shaped curve, yet I am not able to get it with the following code.
<code>logit_model <- glm(leave ~ years_education + trust_politicians +
                   years_education + eu_integration + income,
                   data = ess,
                   family = binomial(link = "logit"))
eduprofiles <- data.frame(
 years_education = seq(from = 0, to = 53, length.out = 54),
 trust_politicians = mean(ess$trust_politicians),
 income = 0,
 eu_integration = mean(ess$eu_integration),
 country_attach = mean(ess$country_attach)
eduprofiles$predicted_probs <- predict(logit_model,
    newdata = eduprofiles, type = "response")
plot(predicted_probs ~ years_education, data = eduprofiles,
    xlab = "Years of Education",
    ylab = "Probability of voting for leave",
    col = "LightSkyBlue", type = "l", frame.plot = FALSE, lwd = 3)
</code>
I feel like I am overlooking something obvious, but I can't seem to figure out what. What do you all think?
the plot shows:
Comment: My guess is years_education has a negative coefficient and you need to expand the range of the x axis below zero to see the S shape.
Comment: What does the plot show?
Comment: just added the plot!
Comment: Or expand the y-axis to 0 to 1.
Comment: That being said, the model shown by expanding the axes beyond the sampled data may not be meaningful.
Comment: why is income set to zero?  Did you mean-centre it already?
Comment: The leave rate in your data set is low, so it is impossible to get the predicted probability to close to 1. If you really want to see the S shape, extend the age to - 200.
Comment: income is a dummy variable meaning someone makes either below or above the median income of a country
Answer: As commenters have pointed out, you're not plotting enough of the range of the x-axis to see the "expected" sigmoid shape.  In your particular example you'd have to extend the education variable to take on negative values - the predicted probability at 0 years of education (already a rather unrealistic value in a modern society!) is only about 0.22. (50 years of education is also pretty unrealistic ...)
In fact, you could be even worse off (in a sense) - if your predicted probabilities were in the range from 0.3 to 0.7, the logistic curve would actually look almost linear (not just non-sigmoid).
Comment: that absolutely helped me move forward, I increased the range and saw the sigmoid shape. Thank you so much!
Comment: if this solves your problem, you are encouraged to click the check-mark to accept it ...


the procs avg and IsZugi doesn't work in my assembly code

Question: thank you very much for helping me, I have done all the changes, as you can see below. however, yet it does not work... :(
I might have done something wrong, would you mind taking another pick?
thank you for all you have done!
I forgot to mention that I created a few arrays if it makes any differens...
    ArrayA dw  9 dup(0)
    ArrayB dw  9 dup(0)
    ArrayC dw  9 dup(0)

proc    Part2
    GoAgain:    mov ah, 1   ;receives a number
            int 21h
            cmp al,65   ;checks if A
            jne TryNext
            call PrintA
            jmp GoAgain
    TryNext:cmp al,66   ;checks if B
            jne TryNext2
            call PrintB
            jmp GoAgain
    TryNext2:   cmp al,67   ;checks if C
            jne TryNext3
            call PrintC
            jmp GoAgain
    TryNext3:   cmp al,69   ;checks if E
            jne TryNext4
            push offset counter
            call IsZugi
            jmp GoAgain
    TryNext4:   cmp al,80   ;checks if P
            jne TryNext5
            push offset counter
            call IsZugi
            jmp GoAgain
    TryNext5:   cmp al,86
            jne TryNext6
            push offset counter
            call avg
            jmp GoAgain
    TryNext6:   cmp al,'#'
            jne TryAgain
            ret
    TryAgain:mov dx, offset message2
            mov ah, 9h
            int 21h
            jmp GoAgain
            ret
    endp    Part2
    proc    PrintA
            mov dl,'a'
            mov ah, 2h
            int 21h
            mov bx,0
    again:  cmp [bx],0  ;checks if index isnt empty
            jne NotEnd2
            ret
    NotEnd2:mov dl,' ' ;print space
            mov ah, 2h
            int 21h
            mov ax,[word ptr bx]
            add al,30h
            mov dl,al;print the number
            mov ah, 2h
            int 21h
            inc bx  ;goes to the next index
            jmp again
            ret
    endp    PrintA
    proc    PrintB
            mov dl,'b'
            mov ah, 2h
            int 21h
            mov bx,10
    again2: cmp [bx],0  ;checks if index isnt empty
            jne NotEnd2
            ret
    NotEnd3:mov dl,' ' ;print space
            mov ah, 2h
            int 21h
            mov ax,[word ptr bx]
            add al,30h
            mov dl,al;print the number
            mov ah, 2h
            int 21h
            inc bx  ;goes to the next index
            jmp again
            ret
    endp    PrintB
    proc    PrintC
            mov dl,'c'
            mov ah, 2h
            int 21h
            mov bx,20
    again3: cmp [bx],0  ;checks if index isnt empty
            jne NotEnd2
            ret
    NotEnd4:mov dl,' ' ;print space
            mov ah, 2h
            int 21h
            mov ax,[word ptr bx]
            add al,30h
            mov dl,al;print the number
            mov ah, 2h
            int 21h
            inc bx  ;goes to the next index
            jmp again
            ret
    endp    PrintC
    proc    IsZugi
            pop [ReturnAddress]
            pop si;counter
            mov dx,0
            mov bx,0;the beginning
    redo:   mov ax,2;to check if number%2=0
            mov dx,0
            mov cx,[bx];cx=number
            cmp cx,0;checks if index=null
            je  next;all the array was read
            mov di,ax
            mov ax,cx
            mov cx,di
            div cx;number/2
            inc bx
            cmp dx,0;to check if number%2=0
            jne redo;go to the next index
            inc [word ptr si];inc counter
            jmp redo
    next:   inc dx
            cmp dx,2
            ja  LastArray
            mov bx,10;moves to the second array
            jmp redo
    LastArray:  cmp dx,3
                je  done
                mov bx,29;moves to the last array
                jmp redo
    done:   mov dl,'P';print p
            mov ah, 2h
            int 21h
            mov dl,' ';print space
            mov ah, 2h
            int 21h
            mov al,[byte ptr si]
            add al,30h
            mov dl,al;print the counter
            mov ah, 2h
            int 21h
            mov dl,'E';print e
            mov ah, 2h
            int 21h
            mov dl,' ';print space
            mov ah, 2h
            int 21h
            mov cl,[byte ptr si]
            mov ch,9
            sub ch,cl
            add ch,30h
            mov dl,ch;print the 9-counter
            int 21h
            push [ReturnAddress]
        ret
endp    IsZugi
proc    IsLegal
        mov bx,0
        pop [ReturnAddress]
        pop ax  ;receives the number
        cmp ax, 8   ;checks if belongs to A
        jb  B
occupied?:
        cmp [word ptr bx],0;occupied or not
        je  NotOccupied
        inc bx
        jmp occupied?
NotOccupied:
        mov [bx],ax ;enters the number to it's place
        jmp finish
B:      cmp ax,6    ;checks if belongs to B
        jb CC
        add bx,10   ;moves to the next array
occupied2?: cmp [word ptr bx],0 ;checks if index occupied
        je  NotOccupied2
        inc bx  ;checks next index
        jmp occupied2?
NotOccupied2:   mov [bx],ax ;enters the value to it's array
        jmp finish
CC:     add bx,20   ;the beginning of the next array
occupied3?: cmp [word ptr bx],0 ;checks if empty
        je  NotOccupied3
        inc bx  ;checks next index
        jmp occupied3?
NotOccupied3:
        mov [bx],ax ;enters the value to it's array
finish: cmp ax,9    ;checks if legal
        ja FalseExit        ;if illegal exit
        cmp ax,0    ;checks if legal
        jl FalseExit        ;if illegal exit
        push 0      ;since we poped ax
        push [ReturnAddress]
        ret
endp    IsLegal
proc    avg
        pop [ReturnAddress]
        mov bx,0    ;first index
        pop si      ;si=sum
suming: mov ax,[word ptr si]    ;it's impossible to do- mov [si],[bx] so...
        add ax,[bx];sum+=bx
        mov [word ptr si],ax
        inc bx
        cmp bx,29
        jne suming
        mov dx, offset message  ;print avg
        mov ah, 9h
        int 21h
        mov ax,[word ptr si]
        mov [word ptr si],9
        div ax
        add al,30h
        mov dl,al
        mov ah,2
        int 21h
        push 0
        push [ReturnAddress]
        ret
endp    avg
start:
        jmp TrueStart
FalseExit:  jmp exit
TrueStart:  mov ax, @data
        mov ds, ax
        mov cx,9    ;loop*9
zloop:  mov ah, 1   ;receives a number
        int 21h
        sub al,30h  ;reveal the real number
        mov ah,0    ;we only need al, however we can only push word
        push ax     ;pushes the number
        call IsLegal
        ;loop zloop
        push offset sum
        call avg
        push offset counter
        ;call IsZugi
        ;call PrintA
        ;call PrintB
        ;call PrintC
        ;call Part2
        call Creative

<\code>
Comment: Please don't change the question the way you did with info you got from the answer!
Comment: At the very least upvote this answer - __that clearly helped you already!__ - before asking the same question again.
Answer: proc    IsZugi
    pop [ReturnAddress]
    pop si;counter

Each time you call IsZugi there no mention of an extra argument. So where do get the pop si from?
Also the comment ;counter is misleading since it seems to be the address of a counter.
TryNext3:   cmp al,69   ;checks if E
    jne TryNext4
    push offset counter      <<< add this
    call IsZugi
    jmp GoAgain
TryNext4:   cmp al,80   ;checks if P
    jne TryNext5
    push offset counter      <<< add this
    call IsZugi
    jmp GoAgain
TryNext5:   cmp al,86
    jne TryNext6
    push offset counter      <<< add this
    call avg
    jmp GoAgain

        mov dx,0
        mov bx,0;the beginning
redo:   mov ax,2;to check if number%2=0

You need to repeat the mov dx,0 with each redo to get a correct division.
        mov bx,0;the beginning
redo:   mov ax,2;to check if number%2=0
        mov dx,0

div cx;number/2
inc bx
cmp ah,0;to check if number%2=0

Since you've done a word division you have to check for any remainder in the DX register, not in the AH register!
Comment: At the very least upvote this answer - __that clearly helped you already!__ - before asking the same question again.


Solving a modified least squares problem?

Question: I am trying to solve a modified least squares problem:
$$ \underset{x\epsilon\mathbb{R} }{min} \left \| Ax - b \right \|_{2}^{2} + \left \| Lx \right \|_{2}^{2} $$
To show that the solution is:
$$(A^{T}A + L^{T}L ) x = A^{T}b$$
So far I have this:
$$(Ax-b)^{T}(Ax-b) + (Lx)^{T}(Lx) $$
$$= x^{T}A^{T}Ax - x^{T}A^{T}b - b^{T}Ax + b^{T}b + x^{T}L^{T}Lx$$
$$= x^{T}(A^{T}Ax - A^{T}b + L^{T}Lx) - b^{T}(Ax-b)$$
I seem to obtain the solution provided the term in the first bracket is 0.
However, I am not sure if this is the correct approach to solving this least squares problem. I also don't know why the remaining $ b^{T}(Ax-b)$ term appears to be neglected?
Answer: For
$$
f(x)
= \lVert A x - b \rVert^2 + \lVert Lx \rVert^2
= \sum_{i=1}^n (Ax - b)_i^2 + (Lx)_i^2
$$
we get the $x$-gradient components
\begin{align}
(\partial/\partial x_j) f(x)
&= \sum_{i=1}^n 2 (A x - b)_i a_{ij} + 2 (Lx)_i l_{ij} \\
&=  2 \left( \sum_{i=1}^n a^T_{ji}(A x - b)_i + l^T_{ji} (Lx)_i \right)
\end{align}
and for a critical point
$$
A^T A x - A^Tb + L^T L x = 0 \iff \\
(A^T A + L^T L) x = A^T b \iff \\
x = (A^T A + L^T L)^{-1} A^T b
$$


How to make tkinter's `expand` give equal gaps between widgets?

Question: If you .pack(side=LEFT, expand=True) 3 widgets, you get this:

Image's source; same is true with .grid(...).
Notice that there are 4 vertical gaps:  left, two in the middle, right.  However, the gaps on left and right are half of those in the middle.
My question is:  how to make expand work more evenly, so that all of the four vertical gaps are equal?
This is .pack's man page in case it helps.
Answer: It can be done using .grid().  You need to put those labels in column 1,3,5 and set weight=1, uniform=1 on column 0,2,4,6 using .columnconfigure().
import tkinter as tk

root = tk.Tk()
root.geometry("400x200")

frm = tk.Frame(master=root, bg="yellow")
# set weight=1 and uniform=1 on column 0,2,4,6
frm.columnconfigure(index=(0,2,4,6), weight=1, uniform=1)

# put labels in column 1, 3, 5
for i in range(1, 6, 2):
    tk.Label(master=frm, text=i, bg="red", padx=40, pady=10).grid(row=0, column=i)

frm.pack(fill="both", expand=True)

root.mainloop()

Result:


Non-monotonic interpolation methods

Question: I'm currently trying to interpolate curves through multiple dimensions (which just involves using interpolation methods on each dimension individually), and I'm a bit confused about the PCHIP (piecewise cubic hermite interpolation polynomial). Does the PCHIP only apply to monotonic data? The data I am interested in is necessarily non-monotonic, so I used a Catmull-Rom spline, but the PCHIP seems to fit my data better, but the off the shelf implementation of the PCHIP im using only works for strictly increasing/decreasing data. I would like to have written my own PCHIP interpolation algorithm (in java), but I cant seem to find the underlying algorithm anywhere on the web. I guess my questions are
1) Does the PCHIP apply to non-monotonic data?
2) If not, are there any other interpolation methods, which pass through the control points, that I could use?
3) Does anyone know of where I can find the algorithm behind the PCHIP?
Comment: *1) Does the PCHIP apply to non-monotonic data?* yes. *3) Does anyone know of where I can find the algorithm behind the PCHIP?* I googled for you: http://www.umiacs.umd.edu/~ramani/cmsc460/Lecture9_interp_2008.pdf
Answer: I don't know PCHIP as an established term, but to me the name suggests any use of a cubic hermite polynomial for interpolation, i.e. a more general term which includes Catmull-Rom among others. The main distinguishing fact from common spline interpolation seems to be the explicitely computed tangents. Both might work for your situation, as both will pass through the defining points and neither will enforce monotonicity. Wikipedia has some descriptions and some references for you if these are the concepts you have in mind. If not, you should be more specific about the context in which you read about this term, as that context might provide a more specific definition.

Non-monotonic interpolation methods


Keep Getting "fatal: This operation must be run in a work tree"

Question: I'm managing a site using the setup in this article: http://toroid.org/git-website-howto
It has been working great for several years but I recently moved the site to AWS and had to reconfigure the remote repo.  Here is my setup:

Local repo setup using git init, then added all the files
Remote repo setup using git --init bare
Created post-receive file in the remote hooks folder, chmod +x to make it executable

The hook contains the following:
<code>#!/bin/sh
GIT_WORK_TREE=/path/to/destination/folder/for/checkout git checkout -f
</code>
Whenever I try running the hook (from the root of the bare remote repo):
<code>./hooks/post-receive
</code>
I get
<code>fatal: This operation must be run in a work tree
</code>
I have tried making the remote repo NOT bare, but that did not work.  I am on Ubuntu 14.04 using git 2.5.8
Any help or pointers appreciated!
Comment: Does the directory `/path/to/destination/folder/for/checkout` exist?  Git is OK with an empty directory, but not with a non-existent one.
Comment: Yes, the directory does exist.
Answer: I solved this by treating the final destination as a repository (because it is, actually lol) So now the post-receive hook looks like this:
<code>#!/bin/sh
cd /final/destination/path
git checkout origin -f
</code>


How to turn a JSON list and convert it to table in R

Question: Hi I used the httr library to get some data from an API. Using this code
library(httr)
library(stringr)
library(tidyverse)
library(bigrquery)
#install.packages("rrapply")

library(rrapply)

    URL <- "https://api.maropost.com/accounts/2116/campaigns/4503/open_report.json?unique=true&auth_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&from=2021-12-01&to=2021-12-31"

    data <- GET(URL)

    response <- content(data)

    my_data <- rrapply(response, how = "melt")

and I am trying to convert the response, a json list into a dataframe
but with the code I have I get a dataframe with this structure
> my_data
    L1          L2                                 value
1    1  account_id                                  2116
2    1 campaign_id                                  4503
3    1  contact_id                                287529
4    1       email                     firstname.lastname@example.com
5    1     browser                               Unknown
6    1  open_count                                     1
7    1 recorded_at         2021-12-06T21:51:03.000-05:00
8    1 total_pages                                    13
9    1         uid                                  NULL
10   2  account_id                                  2116
11   2 campaign_id                                  4503
12   2  contact_id                                362856
13   2       email              firstname.lastname@example.com
14   2     browser                               Unknown
15   2  open_count                                     1
16   2 recorded_at         2021-12-04T11:40:51.000-05:00
17   2 total_pages                                    13
18   2         uid                                  NULL

this is close to what I need but that each L2 value is a column, and that L1 value of 1 will be one row and so on, and the values of each row are the values of the value column, so on that sample in reality is only 2 rows
is there a way to that from what I have?
thanks
Comment: @Bloxx  I get this error when I try your code  "Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = FALSE,  :
  invalid list argument: all variables should have the same length"
Comment: Can you upload list with `dput(my_data)`?  Uploade the output as a code to your question.
Comment: Have you tried with `do.call(rbind.data.frame, my_data)`?
Comment: Have a look at the `jsonlite` package. Details [here](https://jeroen.cran.dev/jsonlite/reference/fromJSON.html).
Answer: with your suggestion I used the jsonlite package and it was really easy
this game the results I needed
library(httr)
library(stringr)
library(tidyverse)
library(bigrquery)
#install.packages("rrapply")

library(rrapply)

URL <- "https://api.maropost.com/accounts/2116/campaigns/4503/open_report.json?unique=true&auth_token=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-ezmOaC95_Xsi_G03CGJXeg&from=2021-12-01&to=2021-12-31&per=50000"

library(jsonlite)

data3 <- fromJSON(URL, flatten = TRUE)

write.csv(as.data.frame(data3),"results.csv", col.names = TRUE)

Answer: Something like this should work.
response_1 <- fromJSON(httr::content(response , as = "text", encoding = "UTF-8"))

list_cols_you_want <- c("","")

response_with_selected_cols <- lapply(response_1 , function(x) x[names(x) %in% list_cols_you_want ])

response_with_selected_cols %>%
  map(data.table::as.data.table) %>%
  data.table::rbindlist(fill = TRUE)


Tell NetworkManager to ignore or not probe rfcomm0

Question: I'm trying to remotely connect to a system via Bluetooth but am running into problems.  Here's what I'm trying to do...
On the remote device I'd like to have the following running:  rfcomm -r watch 0 3 getty -L {} 9600
What that does is: Watch on the bluetooth device "0" for an rfcomm connection on channel 3.  When one occurs, run getty -L {} 9600 and replace the "{}" with the device created.  (usually /dev/rfcomm0 in this case)
I then should be able to connect to the remote device by running on my laptop:  rfcomm connect 0 11:22:33:44:55:66 3  (tells it to use device "0" to connect to the bluetooth address "11:22:33:44:55:66" on channel 3)   I then should get a serial console on the remote device by typing screen /dev/rfcomm0.
The problem...
Everything should be working fine except NetworkManager on my laptop somehow "sees" the connection and tries to probe it as if it was a modem.  It sends AT+GCAP strings and ~x�~ but the remote end is running getty and is expecting login credentials.
I'm able to get it to work by making the rfcomm0 connection first:
remote$  rfcomm -r watch 0 3
laptop$  rfcomm connect 0 11:22:33:44:55:66 3

wait about 60 seconds for the NetworkManger to stop probing /dev/rfcomm0
remote$  getty -L rfcomm0 9600
laptop$  screen /dev/rfcomm0 9600

However, in order to do this, I already need to have console access to the remote device in order to run the getty comamnd
Question
So, is there any way to tell NetworkManager to ignore /dev/rfcomm or to tell it to not probe the device?
Answer: This is a very old question, but in case anyone ends up here via Google, a straightforward solution that worked for me was to create a file /etc/udev/rules.d/99-rfcomm.rules with the following content:
# ignore rfcomm0 device for now
KERNEL=="rfcomm0", ENV{ID_MM_DEVICE_IGNORE}="1"

ModemManager will ignore the device in the future. Note: this is a catch-all rule that will match any rfcomm device, so if you actually want to use one as a modem in the future, you would have to disable the rule again.


Find the ratio in which the circle divides each of the sides AB and AC?

Question: A circle passes through the vertex A of an equilateral triangle ABC and is tangent to BC  at its midpoint .
Find the ratio in which the circle divides each of the sides AB and AC?
Does the line joining the vertex A and the midpoint of BC will be perpendicular to BC?
How can i achieve this?
Thanks in advance.
Comment: http://en.wikipedia.org/wiki/Secant-tangent_theorem
Answer:
Let us assume that the point where $AC$ intersects the circle is $D$ such that $AD = x, DC=y$
If we drop a line perpendicular from $A$ to $BC$, meeting $BC$ at $P$ and call that height $h$.
$$h=2r = \frac{\sqrt{3}a}{2} \Rightarrow r = \frac{\sqrt{3}a}{4}  $$
Also $CP=\frac{a}{2}$ and if we assume $AD=x, DC = y$, then
$$(x+y) = a,  \hspace{5pt} CP^2 = DC . AC =  y(x+y) = ay $$
$$ \frac{a^2}{4} = ay  \Rightarrow \left( y = \frac{a}{4}, x=\frac{3a}{4} \right) $$
Therefore the ratio $$x:y = 3:1$$
Answer: Let $|AB|=|AC|=x+y ~\text { and } ~ x > y$, then :
$x=2r \cdot \sin 60^{\circ}$
$y=a-2r \cdot \sin 60^{\circ}$
$r=\frac{a\sqrt 3}{4}$
Hence :
$x=\frac{3}{4} a \text { and } y=\frac{1}{4}a$
therefore :
$x : y =3 : 1$
Answer to your second question is affirmative .


How to view first Angular template?

Question: I'm trying to build a very basic Angular app but I'm having trouble with viewing the first template. My html looks like this:

and my app.js like this:
'use strict';
var formApp = angular.module('formApp', ['ui.router']);
formApp.config(function($stateProvider, $urlRouterProvider){
    $stateProvider
        .state('main', {
            url: '/main',
            templateUrl: 'templates/main.html',
            controller: 'MainCtrl'
        });
    $urlRouterProvider.otherwise('/main')
});

and my controllers.js like this:
var formControllers = angular.module('formControllers', []);
formControllers.controller('MainCtrl', function($scope){
    $scope.lala = 'lalallala';
});

The template under the name templates/main.html looks like this:

Yes yes people

{{lala}}

So I started node using the command http-server -a 0.0.0.0 -p 8000 and went in the browser to http://0.0.0.0:8000/. Unfortunately, the screen remains white. It does load from the server (I see it making requests to node in the terminal) but I see nothing in the browser. I tried various urls, such as:
http://0.0.0.0:8000/
http://0.0.0.0:8000/#
http://0.0.0.0:8000/#/main
http://0.0.0.0:8000/app
http://0.0.0.0:8000/app/#/

but nothing works.
Does anybody know how I can view the first template I created? All tips are welcome!
[EDIT]
The console gives no errors and when I view the source of the page, I do see my html. I just don't see anything like Yes yes people which I defined in my main.html. So My template is simply not loading. Any idea?
Comment: First Include app.js
Comment: Check console for errors
Comment: I am not sure. can you try like this:  //0.0.0.0:8000/{your folder name}/app/main
Comment: after rearranging the js files try using `//0.0.0.0:8000/{your folder name}/index.html`
Comment: @soosmca - But isn't the app.js dependent on controllers.js, because (according to the answer by dfsq) I need to provide the `formControllers` as a dependency to the formApp?
Comment: @Nilesh - But if I go to `http://0.0.0.0:8000/` I already see that my index.html is loaded. It just doesn't show the `Yes yes people` which I defined in my `main.html` template. Any idea why the template doesn't load?
Comment: I think it is a problem with `ng-view`. Try using `ui-view` instead as you are using ui-router it should do the trick
Answer: Since you defined controller in separate formControllers module, you need to provide it as a dependency to main app module:
var formApp = angular.module('formApp', ['ui.router', 'formControllers']);

Comment: Okay, that makes sense. Unfortunately I still see a white screen. Any idea what else could be wrong?
Comment: And also, which url should I use?
Comment: Template path should be relative to app base path. Also make sure you use `ui-view`, as pointer in comments by Nilesh.
Answer: For UI-Router you should use ui-view directive instead of ng-view. So your html should look like this:

Comment: Awesome, that was it! Thanks!

How to view first Angular template?


How do I include coffeescript and sass compilation steps into managing Aegir platforms?

Question: By providing a drush make file, Aegir will create a new platfrom but sites cannot be created or migrated to it until coffeescript and sass files have been compiled.
Generated javascript and css are not suited to being kept under version control as they cause unnecessary conflicts etc.
How to augment Aegir to include these extra steps?
Answer: We do our own automation of the drush provision and drush hosting commands that Aegir does, plus we add a couple of extra steps to do the sass and coffeescript compilations before migrating a site to a new platform automatically.
So we still have the benefits of the Aegir front end but also augment the power of the drush backend with our required extra steps. Plus, by tying it in with Git tags we make generating new Aegir platforms an integral part of the site-development workflow.
It was quite a big step to augment our reliance upon Aegir with automating the running of drush provision steps from our own Fabric tasks. However, at the cost of one or two limitations, our solution is giving us much more flexibility in the resulting automated workflow.
These are the components:

A Git post-receive hook that triggers a Jenkins job
A Jenkins job that runs the Fabric tasks
Fabric tasks that drive drush provision and drush hostmaster steps
A YAML configuration file.
A drush .make template file

The solution took quite a bit of effort initially but should be straightforward to copy.
It is a bit lengthy to describe in detail here but a detailed description can be seen here which should give you enough information to tailor our solution to your similar needs using the Fabric program here on GitHub


query to check detail from table if found then yes else NO

Question: Below is my query. This is working properly problem is that from like table I am not able to check whether particular user is liking the element on not. if person likes show  yes, if doesn't then show no.
SELECT
      u_m.*,concat(usr.fname,' ',usr.lname) as NAME,
      u_m_u.sent_to,u_m_u.sent_by,u_m_u.shared_of,
      u_m_u.author,u_m_u.adddate

FROM user_message_users AS u_m_u
      LEFT JOIN user_messages AS u_m
        ON u_m.messageid = u_m_u.messageid
       LEFT JOIN smsusers AS usr
        ON u_m_u.sent_by=usr.id
       LEFT JOIN likes
        ON u_m.messageid=likes.element_id
        and
        likes.liked_by='1'
        group by u_m_u.messageid,u_m_u.sent_to
       ORDER BY u_m_u.adddate desc;

Like table is
create table likes(
element_id varchar(100),
element_type varchar(100),
liked_by varchar(100),
foreign key(liked_by) references smsusers(id)
 ON DELETE CASCADE ON UPDATE CASCADE,
adddate timestamp DEFAULT CURRENT_TIMESTAMP,
primary key(element_id,liked_by));

My query link is
Comment: Please check the query.
Comment: What is problem with this query
Answer: SELECT
      u_m.*,concat(usr.fname,' ',usr.lname) as NAME,
      u_m_u.sent_to,u_m_u.sent_by,u_m_u.shared_of,
      u_m_u.author,u_m_u.adddate,
      if (likes.element_id IS NULL, "No", "Yes") Likes

Answer: Try this query :-
SELECT
      u_m.*,concat(usr.fname,' ',usr.lname) as NAME,
      u_m_u.sent_to,u_m_u.sent_by,u_m_u.shared_of,
      u_m_u.author,u_m_u.adddate,if(likes.liked_by is null,'No','Yes')
as likees

FROM user_message_users AS u_m_u
      LEFT JOIN user_messages AS u_m
        ON u_m.messageid = u_m_u.messageid
       LEFT JOIN smsusers AS usr
        ON u_m_u.sent_by=usr.id
       LEFT JOIN likes
        ON u_m.messageid=likes.element_id
and
likes.liked_by='1'
group by u_m_u.messageid,u_m_u.sent_to
ORDER BY u_m_u.adddate desc;

Try SQLFiddle


Capturar qual unique key foi violada

Question: Tenho uma função que faz insert em uma tabela. Essa tabela tem duas colunas que são únicas 'email' e 'login'. Em outras tabelas também existem unique keys compostas.
Dessa forma, uma exceção do tipo UNIQUE_VIOLATION code: 23505 pode acontecer se for tentado inserir dados repetidos em qualquer uma das colunas que estejam restritas por unique keys.
Pergunta:

Como faço para capturar qual das unique keys foram violadas, quando existir mais de uma na mesma tabela, tipo para retornar o nome dela, ou fazer algum procedimento com base nisso?
Answer: Como a versão que estou usando é a 9.5, a solução ideal é o comando GET STACKED DIAGNOSTICS.
Exemplo:
create table foo (a int not null, b int not null, unique(a), unique(b));
insert into foo (a,b) values(1,1);
insert into foo (a,b) values(2,2);
create function ffoo(x integer, y integer) returns void as $$
declare
    tabela text;
    restricao text;
begin
insert into foo (a,b) values(x,y);
exception when unique_violation then
    get stacked diagnostics tabela = TABLE_NAME,
                restricao = CONSTRAINT_NAME;
    raise notice 'tabela: % ; constraint: %', tabela, restricao;
end;
$$ language plpgsql;
select ffoo(2,3);
select ffoo(5,1);

Créditos para Euler Taveira da Comunidade PostgreSQL Brasileira (firstname.lastname@example.com)
Answer: Quando ocorre um erro, junto do código representando o tipo do mesmo, o PostgreSQL também retorna um texto contendo mensagem de erro; no caso de constraints unique, o próprio texto da mensagem de erro contém qual constraint foi violada.
Answer: create table t (
    email text unique,
    login text unique
);

with d (email, login) as ( values
    ('email@example.com','fulano')
), e as (
    select
        exists (select 1 from t inner join d using(email)) as email,
        exists (select 1 from t inner join d using(login)) as login
), i as (
    insert into t (email, login)
    select email, login
    from d
    where not(select email or login from e)
    returning 'email inexistente' as email, 'login inexistente' as login
)
select
    case when email then 'violação de email único' else 'email inexistente' end as email,
    case when login then 'violação de login único' else 'login inexistente' end as login
from e
where email or login
union all
select email::text, login::text
from i

Talvez seja mais fácil parsear a mensagem de erro se o driver usado retornar a mensagem.

Capturar qual unique key foi violada


selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally

Question: The code is :
import os
from selenium import webdriver
chromedriver = "/usr/bin/chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
driver.get("http://stackoverflow.com")
driver.quit()

run python name.py
return the message
File "test_chr.py", line 6, in <module>
    driver = webdriver.Chrome(chromedriver)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 69, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 92, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 179, in start_session
    response = self.execute(Command.NEW_SESSION, capabilities)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.27.440175 (9bc1d90b8bfa4dd181fbbf769a5eb5e575574320),platform=Linux 4.4.0-64-generic x86_64)

my selenium version
Successfully installed selenium-3.0.2
ubuntu@ip-172-31-30-227:/usr/bin$ chromedriver -version
ChromeDriver 2.27.440175 (9bc1d90b8bfa4dd181fbbf769a5eb5e575574320)
google-chrome  -version
Google Chrome 56.0.2924.87
I have no idea how to fix the problem
Comment: Are you trying to run this headless?
Comment: I give up . change the firefoxDriver .now all is ok
Comment: driver.close instead work?
Answer: You need to pass --headless as an argument to start chrome. You can do that by creating a ChromeOption object and passing it to ChromeDriver.
>>> from selenium import webdriver
>>> from selenium.webdriver.common.action_chains import ActionChains
>>> from selenium.webdriver.chrome.options import Options
>>> chrome_options = Options()
>>> chrome_options.add_argument("--headless")
>>> driver = webdriver.Chrome(chrome_options=chrome_options)
>>> driver.get("https://www.google.com")
>>> driver.find_element_by_css_selector("img").get_attribute("alt")
'Google'


How to limit the number of concurrent Http requests accross asynchronous Celery tasks

Question: I am crawling many websites asynchronously using Celery tasks ( + python requests and PhantomJs) I use crawlera as a proxy but my account has a limit of 100 concurrent requests.
I am wondering how is the best way to do this. I know I can use Redis (or something else) as a semaphore and re-trigger the task when acquire() fails after a random number of seconds but I think this approach is not good.
Answer: What makes you think this approach is not good?
It may or may not be good for your situation - it depends entirely on the details of your implementation and your environment.
An alternative approach is to constrain the number of simultaneous connections any 1 worker can make, and constrain the number of workers/tasks so that the total number of connections never exceeds 100.
Comment: I think it's bad because I don't use it for all requests just for some websites, so it would not be optimal
Comment: sorry that's kind of unclear. maybe open a new question showing some sample code - as it is this is all very vague


Mixture model fixed effects

Question: One standard model used with panel data is fixed effects: $y_{it} = \mu_i + \theta_t + \epsilon_{it}$, where $i$ is the individual and $t$ is time subscripts. This can be estimated easily with OLS and dummy variables.
The model assumes that there is a single underlying time series, $\theta_t$ for $t = 1,2,...,T$. All individuals in the data are assumed to follow this time series plus some individual effect, which is constant relative to time.
Suppose, however, that there are two or more groups of individuals, that each group has its own time series, and that, before looking at the data, we don't know who is in which group. I would like to estimate the fixed effects model in this case and to figure out which individual is in which group / cluster.
The approach that I thought of is that the model should become: $y_{it} = \mu_i + \sum_j \pi_{ij} \theta_{jt} + \epsilon_{it}$. $j$ indicates the group $1,2,...,J$. $\pi_{ij}$ is the probability that individual $i$ is in group $j$. $\theta_{jt}$ is the time effect for group $j$ at time $t$.
In my particular application, $\{\pi_{ij}\}$ needs to be a set of additional parameters. However, I can see that in other applications, it could be modeled as a function of some covariates.
Is this model a good approach? Has anyone tried it before? How do I estimate this model? The model has a lot of parameters. I've tried basic optimization, and it has not worked. Ideally, I am looking for software to do the estimation, such as a package in R; or, a solid reference that I could use to program this.
If the model that I've proposed is not a good approach, how else could I solve this problem? One possibility that I could see is first figuring out which individual belongs to which cluster, and then estimating a regular fixed effects model on each cluster. The issue then is how to perform the data classification. Regardless of the approach, I am still looking either for software or a good reference.
Comment: Jessica, of the tags you put on this, only "panel data" and "fixed effects" are relevant. "Estimation" is way too broad; "clustering" refers to uncovering unknown clusters in the data; and "time series" usually means one time series, while you are interested in many. I would suggest you retag this to show "mixture models" and "longitudinal data" (biostat synonym for economics panel data). I could do that, but since you put a bounty on this question, I don't want to screw it up for you :)
Comment: "mixed model" is a biostat term for econometrics "random effect" model. Oddly, that's still relevant here :)
Comment: Thank you everyone who has read my question and tried to answer. All of the answers gave me good ideas / starting points. Unfortunately, no answer gave a straightforward solution. I will therefore let StackExchange automatically assign the bounty based on the votes cast by the community.
Answer: This type of model has been entertained by education researchers for years under the name of growth mixture model (do different students exhibit different rate of learning?), although they work with it as a random effects model. I don't think you'd be able to come up with a proper fixed effect estimation for this model, as it may lack the sufficient statistic that you could condition on. But a full blown ML model should not be extremely difficult to fit.
In Stata, you could try fmm with a full set of panel id dummies and/or interactions with time, although this is a very wasteful approach in terms of degrees of freedom.
Answer: You are trying to fit a "mixture model", though I would suggest not having a separate set of $\pi_{ij}$ parameters for every person, but rather model these probabilities as a function (i.e. logistic) of some covariates. You could even start with a constant $\pi_j$. The typical approach for fitting mixture models is the EM (expectation-maximization) algorithm, though for the simpler versions even direct maximization would work.
Perhaps searching on these terms will help you find a solution that already exists.
Comment: @MichaelChernick, if you want to get a user's attention, consider using the '@' symbol immediately before their name so that they will be notified that they were mentioned (as I did with your name here).
Comment: Sorry, I don't know anything relevant, but that does not mean it does not exist. A quick Google search of "mixture model panel data" brought up some papers discussing this.
Comment: A question for Jessica:  Do you have enough data so that all the πijs are estimable?
Comment: @Aniko- Thank you! Any suggestion as to which package does this? (An R package, a MATLAB function, something else?)
Answer: I suspect you actually mean $y_{it}=μ_i+∑_j I_{ij}θ_{jt}+ϵ_{it}$ where $I_{ij}$ is an indicator for individual $i$ belonging to group $j$ (with probability $\pi_{ij}$. I also wonder if you really want to allow for different individuals to have different group assignment probabilities? I suspect $\pi_j$ will suffice?
I guess it could be seen as a random/mixed effect (not a mixture) with a multinomial effect. This might be an improper term since "random effects" is a typically reseved for Gaussian random effects.
Also note that you are not dealing with a Gaussian mixture, because you allow an individuals shift of the Gaussian ($\mu_i$).
Anyhow, I can think of two approaches:

Write the likelihood and maximize it using en EM algorithm which you can write yourself.
Compute $\mu_i$ by averaging within individuals (assuming you have enough observations).  The residuals will be distributed like a Gaussian location mixture at each $t$. You can use a standard EM implementation such as mixtools R package to estimate $\{\theta_{jt}\}$ and $\{\pi_j\}$.

Mixture model fixed effects


How to avoid autocomplete form with Chrome in Symfony2

Question: I have a form made with Symfony2:
<code>->add('password', 'repeated', array('data' => '', 'label' => 'Repeat New Password', 'type' => 'password', 'required' => false,
        'first_options' => array('data' => '', 'label' => 'New password', 'always_empty' => true),
        'second_options' => array('label' => 'Repeat new password'),
</code>
But in Chrome, the browser try to autocomplete the field New password.
How can I avoid this?
Thanks.
Comment: You should know better as a PHP developer using frameworks!
http://www.w3schools.com/tags/att_input_autocomplete.asp
Comment: I know this, but I want to know how can I put this in a Symfony2 form not renderized in HTML.
Comment: The browser says: "The option "autocomplete" does not exist."
Answer: Try this:
<code>->add('password', 'repeated', array('data' => '', 'label' => 'Repeat New Password',  'type' => 'password', 'required' => false,
    'first_options' => array('attr' => array("autocomplete" => "off")),
    'second_options' => array('attr' => array("autocomplete" => "off")),
</code>
Comment: @user2794692 that is because you are adding autocomplete to the 'first_option' array instead of the 'attr' array inside.
Comment: Thanks, but the browser says me: "The option "autocomplete" does not exist."
Comment: Which browser you are using?
Comment: Chrome is the browser which autocomplete the form. But the error message is a Symfony2 Error Message.


How to ignore wwwroot/lib in TFS 2013 using VS15RC and ASP.NET5?

Question: Problem
By accident, wwwroot/lib was added to our TFS13 server when we created an ASP.NET5 application.
Realizing what happened, we deleted it from the solution and committed the change. This removed the wwwroot/lib directory from TFS13 server. Yay! Right? Well, we then put .tfignore in our project root file DemoApp/src/DemoApp/.tfignore. In this file we added the line wwwroot\lib.
However, everytime we make a change to the application, it wants to add wwwroot\lib back to TFS. It's really frustrating because we have more the location of .tfignore around. We have made our workspace local and tried the pending changes trick to autogenerate a tfingore file according to msdn documents.
In short, wwwroot\lib keeps trying to come back with more tenacity than Michael Myers in Halloween. If anyone else has run into this problem, please let us know. We have tried bing, google-fu, stackoverflow, and some .NET experts. This stumped all of us.
Comment: I ran into the same exact issue, please let me know if you were able to fix this. It's incredibly frustrating!
Comment: I have noted that Microsoft has moved to github on ASP.NET5. Our team is currently building a new TFS server that will use git. We are hopeful that using git rather than 'tfs' will resolve this issue. Why? Because .gitignore handles this.
Comment: Try updating your .tfignore file to `wwwroot\lib\*` adding the  **\\***. The location of your .tfignore is important so you will need the relative path to your `wwwroot\lib` folder from the location of the .tfignore file
Answer: This is a known issue. Your tfignore should work once the aspnet team addresses it.
See here:
https://github.com/aspnet/Tooling/issues/18


Model Stability Metrics and Solutions

Question: I encountered the term of "stability" in learning theory, well described on this wiki page and its references. The idea described in this page deals with the notion of how sensitive the learning algorithm is to changes in the learning set.
I would like to find some info on another notion of stability, which is how sensitive the prediction model to changes in the prediction set. I'll explain: Once We've already trained a model using the learning algorithm, and we have a model that won't change, I might want to run it periodically.
For example, a prediction model for the risk of developing a certain disease, which I run on a monthly basis. I might want to run the model on the same patient on subsequent months, while it's the same patient, but his/her data changes just a bit (a new lab test maybe).
I want to be able to measure and account for the change of predictions that is not due to real significant change in the patient, but to noise or other insignificant factors. I want to make sure the patient doesn't get an entirely different risk score for the same disease on each subsequent month.
I know about bias-variance tradeoff, regularization etc. and I have some other ideas of how to deal with this issue, for example:

Different methods for lowering degrees of freedom of the model during training
Monitoring the changes of individuals' predictions between months, and the reasons for these changes
Feeding the model with the previous risk score of each person/sample
Smoothing features

I would like to know if there's a professional term for this field of predictions stability for the same model. Also, are there any significant works published on this issue?
Answer: I feel like you're asking the wrong question for what you want to achieve.
Bias-variance tradeoff exactly matches your description of the notion of stability. It also explains why you can't have both accuracy and consistency. The closest concept I can think of for what you want to achieve is robustness to noisy input, which can be achieved using data augmentation.
Firstly, I'll explain why asking for "notion of stability" is the wrong approach. Bias-variance tradeoff already answers that, but I'll describe it in a more generalized context in terms of implications. Suppose there is such a concept of stability, and there is some subjective procedure for measuring stability or variance of predictions to small changes in input. (Right off the bat, this procedure is unlikely to exist because what constitutes sensible changes in input varies for every problem and input/output format. It may exist for a highly specific problem type, but you haven't specified your problem type.) You go and test a whole lot of machine learning models, which are nominally ranked by accuracy or some other performance metric, and then you find that the most stable ones are the worst performing models. What use is that? Firstly you've created another problem of how to rank models by both difficulty and stability, and secondly, by default, the vast majority of known/popular models will have no support for decreasing stability (except by performing badly) so that information is barely useful. You also want to know when a change of prediction is dubious or wrong or borderline, which requires a meaningful confidence output from the model. Most models don't have a confidence output, and for those that do, confidence is generally unreliable itself; for a model to consistently and accurately gauge confidence it would have to be superior to a model that doesn't estimate confidence but is already consistent and accurate. That's begging the question and if you had good enough model in the first place then that itself solves the problem more directly.
You also have to define whether you prefer precise inaccuracy or imprecise accuracy. If the initial model prediction is wrong, and you've forced your model to prioritize consistency, then is being wrong forever really better than being giving indecisive results?
I'm going to suggest that what you're better off looking for one of:

data augmentation
a modified problem definition (like your suggestion "Feeding the model with the previous risk score of each person/sample") that encourages consistency
a model that has some of the desired characteristics (such as a confidence output or where the "reasoning" behind a prediction can be accessed and interpreted), or
a model that can be modified to have the desired characteristics, or
a post-processing algorithm that modifies the outputs from a model based on past information, or
some combination of the above

Data augmentation - This is where you generate artificially modified data from your original data, such as adding random noise to it, which will likely produce a model that is more robust to noise (and is therefore more consistent). There is a concern here in the context of diagnosis. If you add random noise, how do you know how much noise can be added without changing the true diagnosis? On one hand, it's fine if what you're saying is you want consistency even if it's factually unsound. On the other hand, if you care about factual accuracy then you need the expertise to generate sound noise-augmented data. Data augmentation aside, for a model to give stable predictions robust to noise, fundamentally there needs to be enough data.
Comment: Thank you very much for this great answer! I'll do my research on the topics you've suggested!

Model Stability Metrics and Solutions


match any column width and set

Question: I have the below HTML table row set:
 <col class="colname-c1 colwidth-7"></col>
 <col class="colname-c2 colwidth-75"></col>
 <col class="colname-c3 colwidth-16"></col>

I apply CSS to them as follows:
.colwidth-7{colwidth-7%}
.colwidth-75{colwidth-75%}
.colwidth-16{colwidth-16%}

This is working fine as I did it in an excel by using the concat function for all the widths from 1-100. But I've been getting some column widths like this:
<col class="colname-c1 colwidth-7.27%"></col>
<col class="colname-c2 colwidth-75.76%"></col>
<col class="colname-c3 colwidth-16.97%"></col>

Here I want to know if there is a dynamic way of setting the css width? This is because I am not sure how many different types of col widths are there and there are approximately 290 tables that I need to create. Please let me know if there is a way for dynamically creating css.
Answer: First up, seems to me that the col element spec says it has no content and can not have a width itself. The only purpose seems to be to "define common semantics" on cells. If you want to set a width on columns in a table you'll have to do it through the (header) cells.
To continue with the actual question:

I want to know if there is a dynamic way of setting the css width?

This can be done, but only for elements that can actually have a width.
Check out the W3 Selectors page. One way to fulfill your requirement is as follows:
[class*="colwidth-75"] { width: 75%; }

This selects all elements with a class attribute value that contains the string "colwidth-75". See this fiddle for a demo.
There's no real reliable way AFAIK to dynamically set the value of width based on the class name, or anything similar. Some similar/related techniques that may shed some light on this for you are LESS, SASS, and the new CSS-Variables module (draft phase).
You can also consider setting these CSS properties using client or server side script, but if that's possible will probably depend on your use case.
Comment: Updated the question as the `col` element doesn't quite work for setting widths. I've also updated to [a new fiddle](http://jsfiddle.net/Lf42W/3/) that uses `th` instead of `div` so you can have the effect in a table.

match any column width and set


Centering initial row of sidebar Menu

Question: I've used this tutorial http://tympanus.net/codrops/2013/07/30/google-nexus-website-menu/ to create a Sidebar in my _Layout code. I'm trying to have the sidebar the li -> a -> "Test Sidebar" element to be on centered. May I ask how do I do that responsively. I can set it with a margin value but that's not very ideal.
_Layout.cshtml code

            Menu

                            <input placeholder="Search" type="search" class="gn-search">
                            Search

                            Popular

Settings

About us

Test Sidebar

     @RenderBody()

Codepen http://codepen.io/anon/pen/LNjYJd
Comment: take a sceenshot of what you need centered and add it in your Q
Comment: Okay, it has been posted
Comment: can you give the fiddle or codepen?
Comment: @NewbieDeveloper done
Answer: some step u need to follow for your solution.
1 - Remove float:left from .gn-menu-main > li and add to .gn-menu-main li.gn-trigger.
2 - Replace display:block to display:inline-block
3 - Add text-align:center and font-size:0 (font size to set margen) to .gn-menu-main
4 - And give font:size according to your design li 13px to .gn-menu-main > li
Try by follow these steps may help you here is fiddle
Comment: i just comment out float:right in .gn-menu-main > li:last-child to be center last child if you don't want last child to be center just revert it thanks

Centering initial row of sidebar Menu


Telescoping series and finding formulas for sums

Question: This problem is from Spivak's calculus book .
So here he shows us how to derive this formula : $\;\; 1^2+...+\;n^2 $
And he begins with : $ \;\;\;\;(k+1)^3 - k^3 = 3k^2 + 3k +1  $ ,  where $\;\; k = 1,...,n$
So writing it down and counting we start with : $$ 2^3 -\;1^3=3.1^2+3.1+1$$ $$3^3-\;2^3=3.2^2+3.2+1$$ $$.$$ $$.$$$$.$$ $$(n+1)^3-n^3=\;3.n^2+3.n+1\\$$
Now here is where I am confused , he presents this :$$$$ $$(n+1)^3-1=3.[1^2+...+n^2]+3.[1+...n]+n\\$$
Where did the "1" come from on the left hand side of the equation ? And where did the "n" come from at the far right hand side of the equation ?
Comment: The $n$ comes from summing $1,$ $n$ times
Comment: We have $1=1^3$ as the motivation for the $1$
Answer: Welcome to MSE 7210!
What is done there is just adding all of the $n$ equations together. This way on the left-hand side you get:
$$[(n+1)^3-n^3]+[n^3-(n-1)^3]+\ldots+[2^3-1^3],$$
which is where the $(n+1)^3-1$ came from. The $n$ on the right-hand side is an effect of adding $n$ copies of $1$, since all of the terms are of form $3\cdot k^2+3\cdot k +1$, so end with a "$1$".
Answer: The terms $2^3,3^3,\cdots n^3$ get cancelled if we add the equations.
So in the LHS we have $$(n+1)^3-1^3 = (n+1)^3-1$$

We have n-equations.  So, $$1+1+1+\cdots+1\text{ (n -times)} = n$$


IOS How to convert float array to short array?

Question: How to convert audio float array to short array() ?
Comment: show your code and what you tried
Answer: Here is how to encode and decode a float with NSData:
encoding:
NSMutableData * data = [NSMutableData dataWithCapacity:0];
float z = ...;
[data appendBytes:&z length:sizeof(float)];

decoding:
NSData * data = ...; // loaded from bluetooth
float z;
[data getBytes:&z length:sizeof(float)];

A couple of things to note here:

You have to use NSMutableData if you are going to add things to the data object after creating it. The other option is to simply load the data all in one shot:
NSData * data = [NSData dataWithBytes:&z length:sizeof(float)];
the getBytes:length: method is for retrieving bytes from an NSData object, not for copying bytes into it.
Comment: is any way to convert audio float array to short array ?


Remove contents of  tag

Question: I have this code. The purpose of the code is to print everything before  and everything after . Everything in between  and  should not be printed. However, I am having 2 issues.

The code is compiling on windows (gcc compiler) but when I run it, it just say's "program stopped working"?
The code is printing everything. It is even printing the ,
 and     everything in between.

My #includes:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>
#include <unistd.h>

Code:
void replacer_and_print(FILE *fp) {

char* line;
size_t len;
ssize_t read;

bool found_tag = false;
int line_storer;
char* before_tag;
char* after_tag;

while ((read = getline(&line, &len, fp)) != -1) {
    if (
        ((before_tag = strstr(line, "")) != NULL) &&
        ((after_tag = strstr(line, "")) != NULL)
       ) {
        line_storer = before_tag - line;
        printf("%.*s", line_storer, line);

        printf("The Image use to be here\n");
        line_storer = after_tag - line + strlen("");
        printf("%s", line + line_storer);

    } else if ((before_tag = strstr(line, "")) != NULL) {
        line_storer = before_tag - line;
        printf("%.*s", line_storer, line);

        found_tag = true;

    } else if((after_tag = strstr(line, "")) != NULL) {
        found_tag = false;

        line_storer = after_tag - line + strlen("");
        printf("%s", line + line_storer);

    } else if(!found_tag) {
        printf("%s", line);
    }
}

fclose(fp);
}

test.html:
This is a test page

Some more text here for more testing!!!!

www.website.com/image.png
More words
TESTING 123

output:
This is a test page

Some more text here for more testing!!!!

The Image use to be here
More words
TESTING 123

Assumptions:
There will only be one
There will only be one . The  tag will always be after the

Comment: Welcome to StackOverflow!  For the most part, this is a good first post; however, you should include any errors you won't help with in your post.  You state that the code won't compile on Windows, could you include the compiler error(s)?
Comment: Then why did you include #1 in your post?
Comment: @c1moore No errors at all. It compiles fine. However, when I run it, it just stops working,
Comment: @c1moore phrased it wrong, sorry. Edited the post
Comment: Tip: when output is questionable, surround _string_ output with sentinels like `<>` and  end each output with `\n`. It better exposes excessive white-space, line feed ,carriage return issues.  E.g. `printf("%.*s", line_storer, line);` --> `printf("<%.*s>\n", line_storer, line);`
Comment: For debug, suggest a different code phrase than `"The Image use to be here\n"` as that same phrase is in the `test.html` and confuses the assessment.
Comment: OT:  this: `TESTING 123`  is NOT valid HTML.  The control blocks must be properly nested.
Comment: what happens when the `` and `` are not on the same line?
Comment: @user3629249 Its just an example.
Answer: You need to set len = 0 before your first invocation of getline().
char *line = NULL;
size_t len = 0;

From the manpage for getline ...

If *lineptr is set to NULL and *n is set 0 before the call, then getline() will allocate a buffer for storing the line.  This buffer should be freed by the user program even if getline() failed.

len is stack-allocated - so unless you initialize it it can contain any value.
Also, you should be free()ing line before finishing.
free(line);
fclose(fp);

Lastly, you should check the return value from fopen().
if (!fp) {
    perror("fopen");
    // Handle error and return if appropriate
}

getline() and Windows
I suspect the issues you are having on Windows are because Windows doesn't implement the getline() function. (It is not part of the C standard - it is a POSIX function.)
As an alternative you can try using fgets(). Example below.
N.B. Note that the memory for line is not being allocated dynamically in this case.
char line[128] = {'\0'};

while (fgets(line, sizeof(line), fp) != NULL) {
// etc..
}
// Do NOT call free(line) at end of scope in this case!

Comment: @SarahChan: In your question you state that "The code is printing _everything_". Now you state the opposite. Can you clarify?
Comment: @SarahChan: I don't have access to `mingw` or a Windows machine. However, are you sure the file is even being opened correctly? I ask because you are not checking the return value of `fopen()`.
Comment: @SarahChan: If you haven't done so already, I think this is a good time to use your debugger (I think `mingw` provides `gdb`). Place a breakpoint at the beginning of your `while()` loop, and - once it is reached - examine the contents of `line`, `len` and `*fp`. Then step through the program using `next` - periodically examining the afore-mentioned variables.
Comment: @SarahChan: I suspect `getline()` is not implemented on your system. I have updated my answer with a suggestion to use `fgets()` instead.
Comment: Added both things, still not printing anything but now it is at least not crashing. Its not meeting the while loop condition for some reason. So it doesn't go into the while loop.
Comment: It prints everything in linux and nothing in windows. My goal is to make it fully work in windows.
Comment: @SarahChan: It prints what seems to be the expected output for me (Linux 64-bit, gcc 6.3.0).
Comment: yea. It doesn't do it in windows (mingw gcc). What could the problem be?
Comment: the fopen works fine. The reason is because the while loop condition isn't met, so it never goes into the loop. I'm not getting any compiler errors but I edited my first post with all the libraries/#includes I am using. I have this as a separate function, so the file is already opened.
Comment: Updated my first post with how I have it as a function
Comment: the function: `fgets()` properly handles the input buffer, so using: `sizeof( line ) -1` is not correct.  Suggest: `sizeof( line )`  Suggest reading the MAN page for `fgets()` for a full description
Answer: There are multiple problems in your code:

line must be initialized to NULL and len to 0. Without initialization, since line and len have automatic storage (aka on the stack), your code has undefined behavior, in your case causing early termination.
HTML  usually have attributes such as src=URL

You make some assumptions, which are not necessary and actually complicate your code:
* only the first image is removed from each line.
* if the  tag and the  are on different lines, there could be a close tag appearing on the same line before an unrelated open tag. Your first test would be fooled by the situation and produce incorrect output.
Here is a corrected version with nested loops:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>
#include <unistd.h>

int main() {
    FILE *fp;
    char *line, *p, *q;
    size_t len;
    bool found_tag;

    fp = fopen("test.html", "r");
    if (fp == NULL) {
        fprintf(stderr, "cannot open %s\n", "test.html");
        return 1;
    }

    line = NULL;
    len = 0;
    found_tag = false;
    while (getline(&line, &len, fp) != -1) {
        for (p = line;;) {
            if (found_tag) {
                q = strstr(p, "");
                if (q == NULL) {
                    /* skip the rest of the line */
                    break;
                }
                found_tag = false;
                p = q + strlen("");
            } else {
                q = strstr(p, "<img");
                if (q == NULL) {
                    fputs(p, stdout);
                    break;
                }
                printf("%.*s%s", (int)(q - p), p,
                       "The Image used to be here\n");
                found_tag = true;
                p = q + strlen("<img");
            }
        }
    }
    free(line);
    fclose(fp);
    return 0;
}

Comment: Where will I replace the the `everything inside` with `The Image use to be here`. Like in my output example. Also the html thing was just an example (I know its not valid html).
Comment: @SarahChan: I updated the answer to output the image placeholder `"The Image used to be here\n"`.

Remove contents of  tag


SBS 2003 requires logon for share

Question: Win 2003 SBS has started requiring password for a domain user. It's weird because it doesn't requires it every time...sometimes it will work (automatic shares through logon script) and sometimes it won't...lets the user in after 2-3 tries (wrong password errors) and the loads the share with the same pass...
Thanks!
Comment: And the logs are empty!
Answer: Is it just for a particular user?  What if that user tries accessing the share from another computer?  To see exactly what is happening, capture a network trace with NetMon or WireShark and see what error the server is sending back to the client.

SBS 2003 requires logon for share


RdKafka setup not finding/recognizing librdkafka in Visual Studio

Question: Issue: RdKafka setup is not finding/recognizing librdkafka in Visual Studio 15
Context:
In order to get started with RdKafka in visual studio 15, I ran the generic paket+nuget setup command (via powershell): paket add nuget rdkafka
After this, I see RdKafka has been installed successfully in my solutions package location. I also see a librdkafka package present.
Adding RdKafka as a reference in my solution works as well (I can't add the librdkafka dlls due to an error message from the IDE).
However, trying to run anything using librdkafka leads to a library not found.
What I've tried:

Removing rdkafka completely and paket reinstalling/updating/restoring
Cleaning + re-building the solution in IDE
Moving the librdkafka package around/including it in path

All the above to no success
Any suggestions on how to proceed?
Comment: Thanks for the quick response. 2 questions I haven't been able to answer:
- does kafunk use rdkafka protocol or the original kafka protocol? we're strictly trying to use rdkafka.
- I'll pass kafunk by my team, but staying with rdkafka for now, any other suggestions? (sorry I edited this so much, my first time asking a question here, trying to familiarize with the ui)
Comment: Actually VS15. And I've found a solution. Attached below!
Comment: Care to try [kafunk](https://github.com/jet/kafunk)?
Comment: Sorry, I can't answer your question, I just thought I'd drop a link, in case it may be of help.
Comment: Are you actually running VS 15 (which is still in Preview) or VS 2015 (aka VS 14)?
Comment: Interesting. How do you find it?
Comment: A teammate led me to it!
Answer: So, it turns out someone else on my team figured out how to make this work, though admittedly it's a little bit of a hack. VS allows for pre-build events (Found by right clicking on your project > Properties > Build Events > Edit Pre-Build). And in there I dropped in the following:

{xcopy /y /d /f
  "$(ProjectDir)..\packages\RdKafka.Internal.librdkafka\runtimes\win7-x64\native*.*"
  "$(TargetDir)"}

And just like that, it works. The explanation for the above is that we pull the native dlls that cannot be referenced directly by copying them into our projects target directory just before the build is done.
Hopefully, this helps some future folks, who don't consider rewriting the library into an referable form, like I did.


Guaranteed payments of an annuity

Question: Suppose that $q_{61}=0.2, q_{61}=0.25, q_{62}=0.4, q_{63}=0.5$ and the interest rate is a constant $5%$ for the first $2$ years, $7%$ after that.A $5$-year life annuity on $(60)$ provides for payments of $100(1+k)$ at time $k$, where $k=0,1,2,3,4$.
(a) Find the present value.
(b) Suppose that instead of being a straight life annuity, the first three annuity payments are guaranteed regardless of whether $(60)$ is alive or not. Find the present value.

Solution for (a)
Cashflow vector $c_k:100,200,300,400,500$
Discount: $v(0)=1 v(1)=\frac{20}{21}, v(2)=\frac{400}{441}, v(3)=\frac{400}{441}(1.07)^{-1}, v(4)=\frac{400}{441}(1.07)^{-2}$
$_0P_{60}=1, _1P_{60}=0.8, _2P_{60}=0.6, _3P_{60}=0.36, _4P_{60}=0.18$
P.V=$100+(200)(\frac{20}{21})(0.8)+(300)(\frac{400}{441})(0.6)+(400)(\frac{400}{441}(1.07)^{-1})(0.36)+(500)(\frac{400}{441}(1.07)^{-2})(0.18)=609.01$
Solution for (b) is a mess.
Answer: Why would (b) be any more difficult?  You simply change ${}_1 p_{60} = {}_2 p_{60} = 1$, but leave the other survival probabilities unchanged, then recalculate.  This gives you the correct present value because you can think of ${}_k p_{60}$ as not the $k$-term survival probability of $(60)$, but rather, the probability of payment in the $k^{\rm th}$ period.  Then it becomes clear what you need to do.

Guaranteed payments of an annuity


How to keep the x axis order from the data.frame (for routine r script)?

Question: This is a sample of my datas :
structure(list(VARIETE = structure(c(3L, 5L, 4L, 1L, 2L, 6L), .Label = c("id4",
"id5", "id1", "id3", "id2", "id6"), class = "factor"), MOY_AJUST_GENE = c(115.4217669,
118.7343702, 116.8029088, 113.1666208, 114.3314785, 125.3140321
), `19` = c(115.5875947, 117.9590553, 116.8029088, 110.2799894,
115.1775659, 125.3140321), `18` = c(115.2559391, 119.5096851,
NA, 116.0532521, 113.9523044, NA), `17` = c(NA, NA, NA, NA, 115.8286885,
NA), `15` = c(NA, NA, NA, NA, 113.3820091, NA), `14` = c(NA,
NA, NA, NA, 116.8935901, NA), `13` = c(NA, NA, NA, NA, 113.1634867,
NA), `12` = c(NA, NA, NA, NA, 111.9227046, NA), c1 = c(NA, NA,
NA, NA, NA, 114.9076441), c2 = c(NA, NA, 111.9647996, NA, NA,
127.0981296), grp = structure(1:6, .Label = c("1", "2", "3",
"4", "5", "6"), class = "factor"), Z55 = c(7.5, 6.5, 7, 5, 6,
7), CLASSE = structure(c(3L, 3L, 3L, 2L, 1L, 3L), .Label = c("",
"BP", "BPS"), class = "factor"), `PROT (GPD)` = c(2L, 5L, 4L,
NA, 7L, 6L), MOSA = structure(c(2L, 2L, 2L, 2L, 1L, 2L), .Label = c("",
"S"), class = "factor"), SEPTO = c(4L, 7L, 7L, NA, 6L, 5L), RJ = c(6L,
9L, 8L, 7L, 5L, 8L)), row.names = c(NA, -6L), class = "data.frame")

This is the ggplot2 code I use to create a plot that looks like a table :
fig2 <- DTA_ecart %>%
  gather(catalogue, value, -VARIETE, -(MOY_AJUST_GENE:grp), na.rm = TRUE) %>% # put the wide table in long shape and remove a part of the columns between "MOY_AJUST_GENE" and "grp" that I use in another chart.
  ggplot(aes(x = catalogue, y = VARIETE, na.rm = TRUE)) +
  geom_text(aes(label = paste(value)), size = 2, vjust = 0.5, hjust = 0.5) +
  scale_x_discrete(position = "top")

My data frame DTA_ecart is really large and I use it for another chart before (fig1).
The question is : How to keep the x axis order from my data frame (DTA_ecart) without problems ?
At this time, the x axis is plot by alphabetical order. I know that we have to say to ggplot2 to use specific columns for the order and I use that for my previous chart : DTA_ecart[, 1] <- factor(DTA_ecart[, 1], levels = DTA_ecart[, 1][order(DTA_ecart[, 2], decreasing = FALSE)]). Now I don't know how to translate for the x axis on my second chart (fig2).
Edit : Below, there are two right answers, but for my pupose I don't want to specify "names" or "numbers" of columns (because of the routine restriction). Is there a way to get this differently ?
Thanks!
Comment: I am guessing you just need a `mutate(catalogue = factor(...))` step before passing it to *ggplot*.
Answer: In case you want to change the order in your original dataframe, you could also just get it directly from the column order:
get the column after the column named grp:
column_after_grp=grep("^grp$", names(DTA_ecart))+1

fig2 <-
  DTA_ecart %>%
  # put the wide table in long shape and remove a part of the columns
  # between "MOY_AJUST_GENE" and "grp" that I use in another chart.
  gather(catalogue, value, -VARIETE, -(MOY_AJUST_GENE:grp), na.rm = TRUE) %>%
  ggplot(aes(x = catalogue, y = VARIETE, na.rm = TRUE)) +
  geom_text(aes(label = paste(value)), size = 2, vjust = 0.5, hjust = 0.5) +
  # setting the limits here helps towards your goal
  scale_x_discrete(position = "top", limits = names(DTA_ecart)[column_after_grp:ncol(DTA_ecart)])

Comment: Figuring out what columns to use depends on how your dataframe is likely to be changing. Will the relevant columns always start after column `grp`?
Comment: They always starts after the column `grp` but the number of columns before and after could change. Is it possible to code `limits = names(DTA_ecart[, "the column after grp" : ncol(DTA_ecart)])`
Answer: Instead of converting the variable to a factor, you can also specify the order of the axis using scale_x_discrete(limits = [...]). So here you could specify the unique values of the column mapped to the x axis (as they appear).

DTA_ecart <- structure(list(VARIETE = structure(c(3L, 5L, 4L, 1L, 2L, 6L), .Label = c("id4",
  "id5", "id1", "id3", "id2", "id6"), class = "factor"), MOY_AJUST_GENE = c(115.4217669,
  118.7343702, 116.8029088, 113.1666208, 114.3314785, 125.3140321
  ), `19` = c(115.5875947, 117.9590553, 116.8029088, 110.2799894,
  115.1775659, 125.3140321), `18` = c(115.2559391, 119.5096851,
  NA, 116.0532521, 113.9523044, NA), `17` = c(NA, NA, NA, NA, 115.8286885,
  NA), `15` = c(NA, NA, NA, NA, 113.3820091, NA), `14` = c(NA,
  NA, NA, NA, 116.8935901, NA), `13` = c(NA, NA, NA, NA, 113.1634867,
  NA), `12` = c(NA, NA, NA, NA, 111.9227046, NA), c1 = c(NA, NA,
  NA, NA, NA, 114.9076441), c2 = c(NA, NA, 111.9647996, NA, NA,
  127.0981296), grp = structure(1:6, .Label = c("1", "2", "3",
  "4", "5", "6"), class = "factor"), Z55 = c(7.5, 6.5, 7, 5, 6,
  7), CLASSE = structure(c(3L, 3L, 3L, 2L, 1L, 3L), .Label = c("",
  "BP", "BPS"), class = "factor"), `PROT (GPD)` = c(2L, 5L, 4L,
  NA, 7L, 6L), MOSA = structure(c(2L, 2L, 2L, 2L, 1L, 2L), .Label = c("",
  "S"), class = "factor"), SEPTO = c(4L, 7L, 7L, NA, 6L, 5L), RJ = c(6L,
  9L, 8L, 7L, 5L, 8L)), row.names = c(NA, -6L), class = "data.frame")

library("dplyr")
library("tidyr")
library("ggplot2")

df <- DTA_ecart %>%
  gather(catalogue, value, -VARIETE, -(MOY_AJUST_GENE:grp), na.rm = TRUE)
#> Warning: attributes are not identical across measure variables;
#> they will be dropped

fig2 <- ggplot(df, aes(x = catalogue, y = VARIETE, na.rm = TRUE)) +
  geom_text(aes(label = paste(value)), size = 2, vjust = 0.5, hjust = 0.5) +
  scale_x_discrete(position = "top", limits  = unique(df[["catalogue"]])
    )

fig2

Created on 2019-11-14 by the reprex package (v0.3.0)
You could also convert this to a function to apply it to different columns, eg:
foo <- function(df, x_column, y_column)
  ggplot(df, aes_string(x = x_column y = y_column, na.rm = TRUE)) +
    scale_x_discrete(limits = unique(df[[x_column]]))`

Comment: Thanks, it works well! Perfect! Could you say more about the warning message ? It's not really an issue because it doesn't afect the result but it's always better to understand why.
Comment: Why/how would it break? Per the warning, it's because the variables are factors. I would recommend converting your variables to characters unless you have huge data or need to coorce to numeric.
Answer: As mentioned in the comments, we need to define factors, something like this:
library(ggplot2)
library(dplyr)

plotDat <- DTA_ecart %>%
  gather(catalogue, value, -VARIETE, -(MOY_AJUST_GENE:grp), na.rm = TRUE) %>%
  mutate(catalogue = factor(catalogue,
                            # set it manually
                            levels = c("Z55", "CLASSE", "PROT (GPD)", "MOSA", "SEPTO", "RJ")
                            # or get it from column names order:
                            #levels = colnames(DTA_ecart)[13:18]
  ))

ggplot(plotDat, aes(x = catalogue, y = VARIETE, na.rm = TRUE)) +
  geom_text(aes(label = paste(value)), vjust = 0.5, hjust = 0.5) +
  scale_x_discrete(position = "top")

Note: I am using intermediate data - plotDat, makes it easier to debug and check what we actually passing to ggplot, this can be skipped and passed directly to ggplot with pipes.
Comment: Oh thanks! It seems to be ok but what can I do if I don't want to write the names of the columns here `levels = c("Z55", "CLASSE", "PROT (GPD)", "MOSA", "SEPTO", "RJ")))` ? The fact is that this table could change in the furture maybe with 2 or 3 columns, or with differents names. It should be a "routine" script.


how to set images in the calender view iPhone

Question: Im developing a calender based application and in that my client need to set a image on the particular date user selects.
For example if i select 1 and 20,an image is to be set on both the dates.
For now i can set the image only for one date i.e if i select 1 the image  is set on that date and when i select 20 the image set on 1 will disappear and image will set only for 20.
Can anyone please help me how to do this.
Thanks in advance.
Comment: check this link:http://stackoverflow.com/a/8399779/845115
Comment: What are you using to display the calendar, this most be something custom since the iOS SDK does not provide a component like this.
Comment: @MuditBajpai now i can set image for one date but need to set image for more than one date.Can you please help me.
Comment: I hope you will be having, one date reference as selectedDate,No you need two dates right?
If i am correct, use one more NSDate variable as selectedDate2, and check your conditions and add image as you are doing for first one
Answer: I finally find a solution for this.I done in the method and you experts can suggest a better way for this.
<code>- (void)selectButtonForDate:(NSDate *)aDate {

    if (selectedButton >= 0) {
        NSDate *todayDate = [CalendarLogic dateForToday];
        UIButton *button = [buttonsIndex objectAtIndex:selectedButton];

        CGRect selectedFrame = button.frame;
        if ([selectedDate compare:todayDate] != NSOrderedSame) {
            selectedFrame.origin.y = selectedFrame.origin.y + 1;
            selectedFrame.size.width = kCalendarDayWidth;
            selectedFrame.size.height = kCalendarDayHeight;
        }

        button.selected = NO;
        button.frame = selectedFrame;

        self.selectedButton = -1;
        self.selectedDate = nil;
    }

    if (aDate != nil) {
        // Save
        CGRect dayFrame;
        NSDate *dayDate;
        NSInteger numberOfWeeks = 5;
   //     NSCalendar *calendar = [NSCalendar currentCalendar];
       // NSDateComponents *components = [calendar components:(NSDayCalendarUnit | NSMonthCalendarUnit | NSYearCalendarUnit) fromDate:[NSDate date]];
   //     NSDate *todayDate = [calendar dateFromComponents:components];

        for (NSInteger aWeek = 0; aWeek <= numberOfWeeks; aWeek ++) {
            CGFloat positionY = (aWeek * kCalendarDayHeight) + 60;

            for (NSInteger aWeekday = 1; aWeekday <= numberOfDaysInWeek; aWeekday ++) {
                CGFloat positionX = ((aWeekday - 1) * kCalendarDayWidth) - 1;
                dayFrame = CGRectMake(positionX, positionY, kCalendarDayWidth, kCalendarDayHeight);
                dayDate = [CalendarLogic dateForWeekday:aWeekday
                                                 onWeek:aWeek
                                          referenceDate:[name referenceDate]];
            //  NSLog(@" %@ todaydate [datesIndex objectAtIndex:[sender tag] %@",todayDate,[datesIndex objectAtIndex:[sender tag]]);
              //  NSDate *mmm = [datesIndex objectAtIndex:[sender tag]];
                if ([dayDate isEqualToDate:aDate]) {
                     NSUserDefaults *currentDefaults = [NSUserDefaults standardUserDefaults];

                    flag = 1;
                    [n addObject:[NSValue valueWithCGRect:dayFrame]];

                    [currentDefaults setObject:[NSKeyedArchiver archivedDataWithRootObject:n] forKey:SaveMonth];
                    [currentDefaults synchronize];
            }

            }
        }
        if(flag == 1){

            flag = 0;
            for (int i=0;i<[n count];i++){

                UIImageView *im = [[UIImageView alloc]initWithImage:[UIImage imageNamed:@"call.png"]] ;
                CGRect myRect = [[n objectAtIndex:i] CGRectValue];
//                NSLog(@"n is %@",n  );
//                NSLog(@"frame 111111111111111  = %@\n", NSStringFromCGRect(myRect));

                im.frame = myRect;
                [self addSubview:im];

            }
        }
</code>
Comment: Hi, I also try to do the same. Are you using the KAL library or something else?


Pass file downloaded via URL to Stream (without WebClient)

Question: I'm building an UWP app (HoloLens), and am running into a problem with streaming files downloaded from an URL. I had written a piece of code that worked perfectly, where I use WebClient.Download Data to get the file into a Stream, until I tried building and realized the WebClient class is not available in UWP apps.
In other words: how can I download a file from an URL and pass it into a Stream (to use in my BinaryReader) in an UWP app?
The code I ran:
using (Stream stream = GetStreamFromUrl(filePath))
{
    using (var binaryStream = new BinaryReader(stream))
    {
        DoStuffWithBinaryStream();
    }
}

private static Stream GetStreamFromUrl(string url)
{
    byte[] imageData = null;

    using (var wc = new System.Net.WebClient())
        imageData = wc.DownloadData(url);

    return new MemoryStream(imageData);
}

Comment: Did you look at `HttpClient`?
Answer: As Peter Torr- MSFT said you could  use HttpClient to get the stream from URL.
using (HttpClient client = new HttpClient())
{
    try
    {
        var response = await client.GetAsync(new Uri("XXXXXX"));

        if (response != null && response.StatusCode == HttpStatusCode.OK)
        {
            using (var stream = await response.Content.ReadAsStreamAsync())
            {
                //do some stuff
            }
        }
    }
    catch (Exception ex)
    {

    }
}

For more, please refer to HttpClient.


Covariance between a variable and an interaction variable

Question: If I have two independent variables $X$ and $Y$, then $Cov(X,Y)=0$. Now let $Z = X*Y$. Then I would assume $Cov(X,Z)\ne 0$, but given the expecations, variances and covariances of $X$ and $Y$ is there are formula I can use to calculate how big it should be?
Answer: I am not sure if I am clear about your question.
\begin{align}
Cov(X,Z)&=Cov(X,XY)\\
&=E(X^2Y)-E(X)E(XY)\\
\text{(because of independence)} &= E(X^2)E(Y)-E^2(X)E(Y)\\
&=E(Y)Var(X)
\end{align}


SQL Copied query from MS Access

Question: I am trying to recreate a MS Access query in SQL, the following is the Access query converted to run in SQL
    SELECT Column1
          ,Column2
          ,Column3
          ,Count('x') AS CountX
    FROM [SchmaName].[SQLView]
    WHERE ("SomeWhereStatement")
    GROUP BY Column1, Column2, Column3
    HAVING Count('x')>1;

What I am confused about is the
 ,Count('x') AS CountX

and
 HAVING Count('x')>1;

X does not come from the SQLView that has been created and when I run the above it will return a single record with Columns 1 - 3 being null and column CountX being 12896. I am expecting this return about 4285 records in total and I suspect the reason that I am not is something to do with this Count. Any suggestions as to what this count is actually doing?
Comment: Using `count('x')` is no different than using `count(1)` or `count(*)` -- it's just counting literal strings (albeit a very odd way).  Are you sure you have values for the other columns you are grouping by?  It sounds like they are all `null` thus aggregating to a single result.  Sample data and expected results would be helpful.
Comment: The query seems to look for duplicates `HAVING Count('x')>1;`
Answer: When you pass any constant argument in COUNT() (i.e. 'x' is constant argument), it will return total number of records  :
So, with your query it will return the records based on Columns which are defined with GROUP BY clause.
SELECT Column1, Column2, Column3, Count('x') AS CountX
FROM [SchmaName].[SQLView]
WHERE ("SomeWhereStatement")
GROUP BY Column1, Column2, Column3
HAVING Count('x') > 1;

Comment: Ok thanks, this explains what it is doing and points me in the right direction

SQL Copied query from MS Access


XSLT transformation-grouping and sum

Question: I have a input xml
<students>
  <student>
  <name>John</name>
  <marks>100</marks>
</student>
<student>
  <name>Arvind</name>
  <marks>90</marks>
</student>
<student>
  <name>John</name>
  <marks>100</marks>
</student>
<student>
  <name>Arvind</name>
  <marks>80</marks>
</student>
</students>

I want the above xml to be transformed into
<students>
   <student>
     <name>John</name>
    <totalMarks>200</marks>
   </student>
   <student>
     <name>Arvind</name>
    <totalMarks>170</marks>
  </student>
 </students>

so basically I want to group the input xml based on the student name and get the sum of their marks as well.
Answer: In xlst 2.0 you could use for-each-group:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fo="http://www.w3.org/1999/XSL/Format">
    <xsl:output method="xml" indent="yes" />

    <xsl:template match="/">
        <students>
            <xsl:apply-templates select="students" />
        </students>
    </xsl:template>

    <xsl:template match="students">
        <xsl:for-each-group select="student" group-by="name">
            <student>
                <name>
                    <xsl:value-of select="current-grouping-key()" />
                </name>
                <totalMarks>
                    <xsl:value-of select="sum(current-group()/marks)" />
                </totalMarks>
            </student>
        </xsl:for-each-group>
    </xsl:template>

</xsl:stylesheet>

Comment: thanks for the reply..is there any way where we can achieve the same using xslt 1.0 I don't have  available..
Comment: Yes it is, @Ken described this method
Answer: In XSLT 1, the Muenchian method of grouping is typically used when there is one key:
t:\ftemp>type students.xml
<students>
  <student>
  <name>John</name>
  <marks>100</marks>
</student>
<student>
  <name>Arvind</name>
  <marks>90</marks>
</student>
<student>
  <name>John</name>
  <marks>100</marks>
</student>
<student>
  <name>Arvind</name>
  <marks>80</marks>
</student>
</students>
t:\ftemp>xslt students.xml students.xsl
<?xml version="1.0" encoding="utf-8"?>
<students>
   <student>
      <name>John</name>
      <totalMarks>200</totalMarks>
   </student>
   <student>
      <name>Arvind</name>
      <totalMarks>170</totalMarks>
   </student>
</students>
t:\ftemp>type students.xsl
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  version="1.0">

<xsl:key name="students-by-name" match="student" use="name"/>

<xsl:output indent="yes"/>

<xsl:template match="students">
  <students>
    <xsl:for-each
        select="student[generate-id(.)=
                        generate-id(key('students-by-name',name)[1])]">
      <student>
        <xsl:copy-of select="name"/>
        <totalMarks>
          <xsl:value-of select="sum(key('students-by-name',name)/marks
                                    [number(.)=number(.)])"/>
        </totalMarks>
      </student>
    </xsl:for-each>
  </students>
</xsl:template>

</xsl:stylesheet>
t:\ftemp>

Comment: In the classroom I also teach the variable-based grouping method, useful when you are creating groups within groups.  That can be used when there is only one set of groups, but the Muenchian method is faster.
Comment: Hi... G. Ken Holman  thanks a lot for your reply..can you kindly explain the select expressions in the for-each and select in the value-of of  element.
Comment: The argument to `sum()` is `key('students-by-name',name)/marks[number(.)=number(.)]` which is to say: find all students in the key table with the same name as the group's name element, get all of the `` element children, and only keep those that are valid numbers.  The last bit can be omitted if you have faith that marks are correctly recorded, but if not, then the predicate protects the sum from being spoiled by something that is "not a number" (NaN).
Comment: Sorry, I missed that you also asked for the `` expression.  In XSLT 1.0 this is the classic "Muenchian method" for grouping, and it uses a key table.  The table is loaded with all students and is indexed by the student name.  The predicate ensures that only the first student in document order for each given name is selected.  The body of the `for-each` is then executed once for each group.

XSLT transformation-grouping and sum


the latest igb driver version

Question: I tried to make my server NIC driver update, after ethtool -i it show
Check version
but the latest version on Intel website is lower:
https://downloadcenter.intel.com/download/13663/Intel-Network-Adapter-Driver-for-82575-6-82580-I350-and-I210-211-Based-Gigabit-Network-Connections-for-Linux-?wapkw=igb%20driver
Version 220.127.116.11
do you have info about it?
Server:
Supermicro
Centos 7.8
NIC: Intel I350
Answer: You have found an interesting conundrum here. Having worked as a contractor for the networking group at Intel, I can attest that the latest version of its adapter drivers are uploaded to its website; Intel does not give special access to the latest version to any vendor first. This said, the latest version on the Intel website is not 5.4.0-k as you write (actually 5.3.6; 188.8.131.52 is the previous release) which was bundled in Centos 7.8.
My guess is that Red Hat (who created the code Centos is based on) decided to assign a new version number to the driver code for some reason. The usual reason is that they modified the code. (I did a quick look through a system with CentOS 7.8, but failed to find any documentation explaining why the version of igb was changed.)
As a final note, the -k at the end of the version number shows this is the version included in the base package, & not one later downloaded & compiled -- a useful bit of information for troubleshooting driver problems.
Comment: is it safe to download and compile the Igb listed on Intel? because i am not sure if the version included in Centos as 5.4 is up-to-date.
Comment: You don't seem to be up to date with CentOS 7.8. I also use it, I also use igb and my version is 5.6.0-k. Please make sure you have an up to date kernel.
Comment: Yes it is safe to compile the driver from the Intel website. One additional thought: check the date of your igb driver: the source on the Intel website should be more recent than what comes with CentOS.

the latest igb driver version


Instantiating a generic class where type argument comes from a string

Question: First I have to reference Jon Skeet's answer to a similar question
https://stackoverflow.com/a/266282/1184296
In his answer he wrote this
Type genericClass = typeof(Generic<>);

What I don't understand here is how to get Generic<> to show up.
I tried adding usings with all the namespaces under System but no luck. Nothing shows up in IntelliSense and Generic<> stays underlined in red stating "The type or namespace 'Generic' could not be found...".
Does anyone knows how to solve this?
Thanks.
Comment: You're missing the fact that he defined the `Generic` class himself in that same piece of code
Answer: If you look at the code in the Post, John Skeet has created a custom class..
public class Generic<T>
{
    public Generic()
    {
        Console.WriteLine("T={0}", typeof(T));
    }
}

So, you could do the same, you might need John Skeets permission to take that code though? :)
Comment: Give it a different name, and change the Constructor slightly, and nobody will notice...
Comment: Hahaha, I can't believe how obvious thing I missed. That should do it :)