Why Generalized Least Squares?

Question: So it is often advise to use Generalized Least Squares when we have a regression model with non-spherical(i.e. heteroskedastic or autocorrelated) errors. We do so by doing a weighted regression
$$
(y-x\hat\beta)^TW(y-x\hat\beta)
$$
with $W = \Sigma^{-1} =  Cov(\epsilon)^{-1}$, the inverse of the covariance matrix of errors.
The variance of the estimated $\hat\beta$ is
$$
\begin{aligned}
Var(\hat\beta_{GLS})&=(X^TWX)^{-1}X^TW\Sigma W^TX(X^TW^TX)^{-1}\\
&=(X^TWX)^{-1}=(X^T\Sigma^{-1} W)^{-1}
\end{aligned}
$$
To do GLS, we must know $\Sigma$. But if we already know $\Sigma$, why can't we just do regular OLS, and calculate $Var(\hat\beta)$ as
$$
Var(\hat\beta)=(X^TX)^{-1}X^T\Sigma X(X^TX)
$$
? Is it because $Var(\hat\beta_{GLS})$ is smaller?
Another question I've always had is that for OLS, $\beta$ is estimated as :
$$
\hat\beta_{OLS}=(X^TX)^{-1}X^Ty
$$
. For GLS or WLS, the $\hat\beta$ is estimated as
$$
\hat\beta_{GLS} = (X^TWX)^{-1}X^TWy
$$
, which is unbiased for non-spherical error. Yet, we are told that $\hat\beta_{OLS}$ is also an unbiased estimator of $\beta$, even with non-spherical errors. Does that mean
$(X^TWX)^{-1}X^TWy$ simplifies to $(X^TX)^{-1}X^Ty$?
Answer: You can indeed do regular OLS and compute the variance of the estimator (and that estimator will be unbiased and consistent).
But, GLS will be a more efficient estimator that has a lower variance of the sampling distribution (in fact out of all unbiased linear estimators, it will be the estimator with the least possible variance).

Example
Let us estimate $\mu$ with the following variables $$X_k \sim N\left(\mu, \sigma^2 \cdot k\right)$$
Then
$$\begin{array}{}
\hat\mu_\text{OLS}& =&\frac{1}{n} \sum_{k=1}^n {X_k} &\sim& N\left(\mu, \sigma^2\cdot{\frac{1+1/n}{2}}\right)\\
\hat\mu_\text{GLS}& =&\frac{1}{H_{n,0.5}}\sum_{k=1}^n \frac{1}{\sqrt{k}} {X_k} &\sim& N\left(\mu, \sigma^2\cdot{\frac{n}{(H_{n,0.5})^2}}\right)
\end{array}$$
where $H_{n,0.5} = \sum_{k=1}^n \frac{1}{k^{0.5}}$
The variance of the GLS estimator is smaller than the variance of the OLS estimator. So if you can reasonably guess the covariance matrix of the error distribution $\Sigma$ this may be beneficial.
Comment: That makes sense. What about the estimate of coefficient themselves? Is $\hat\beta_{OLS} = \hat\beta_{GLS}$? If not, how can both be unbiased estimators?
Comment: GLS estimates $\Sigma$.  You just need to tell GLS the assumed correlation pattern (e.g., AR(1)) in order for it to estimate $\Sigma$.
Comment: @wwyws - if the estimates were always the same, neither would be better than the other.  They can both be unbiased if both their *expected* values are the same (and equal to the true parameter); with each individual sample, though, their *calculated* values can still be different, as long as the differences have an *expected* value equal to zero.
Comment: @wwyws the example shows how the estimates are different. It is due to the different coefficients in the sum. But because the coefficients both sum up to 1, you get that the estimates have the expectation value $\mu$ and are unbiased.
Comment: @Sextus Empiricus - You can not do OLS for heteroscedastic dataset in order to get Best Linear Unbiased Estimation -- it is errorness way for modelling your dependency.
Comment: GLMs is OK for non-normally distributed data -- you should create hypothesis about the [distribution](https://www.hec.usace.army.mil/confluence/sspdocs/sspum/2.3/distribution-fitting-analysis/distribution-fitting-and-parameter-estimation) of your data first, as assumption,. To determine which distribution (normal, exponential, or gamma) fits the dataset most accurately, you can use a goodness-of-fit test., or [chi-square test](https://ocw.metu.edu.tr/pluginfile.php/2284/mod_resource/content/0/ocw_iam530/9.DISTRIBUTION%20FITTING.pdf)
Comment: if with GLS *you just mean WLS - then yes, I fully agree, that it's OK for heteroscedastic data... if you see no other pattern in residuals plot after modelling dependency/AR
Comment: @JeeyCi Is there something you disagree with? Where I am saying that you can do OLS (in the first sentence), I am not saying that this leads to a best linear unbiased estimator.
Comment: "Where I am saying that you can do OLS (**in the first sentence**)".. for heteroscedastic  data (wanted in OP) you can do only WLS, your first sentence is misleading for correct statistical inference, though you can assume whatever you like, just not presenting it like real opportunity for valid stat. conclusions
Comment: @JeeyCi where does the question state that it is about statistical inference? Where do I say that I speak about inference? (I am changing slightly that first sentence, since unbiased does not clearly refer to the estimator, and might be interpreted as referring to the variance)
Comment: where? - !! **stats**.stackexchange.com - no more comments ...
Comment: @JeeyCi I don't understand your point. How are your remarks related to this question and answer to it? If you don't want to explain that further then why not delete the comments? I am saying that one can do OLS in the first sentence, but I am not saying that you can do this *"in order to get Best Linear Unbiased Estimation"*. And the question is also not asking about inference but about the unbiasedness of OLS.
Comment: Still, even if you would apply inference, then this can also be done when one performs OLS, as long as one makes sure that the variance of the estimators are estimated with the correct assumptions about the error distribution. OLS is only the method to compute the estimate. It is not the method to compute the variance of the estimate.

Why Generalized Least Squares?


Area of the intersection of the region outside of a unit circle centered at origin and the region inside the unit circle centered at (1,0)?

Question: This problem was on my calculus final, and I had no idea how to set up the double integral to solve it.
Comment: This seems to be easier to calculate using good old cut-and-paste geometry than by setting up an integral.
Comment: Or, failing to follow Henning's suggestion, is using polar coordinates an option?
Answer: $r=2\cos \theta$ is the circle centered at $(1,0)$ with radius $1$. (Why?)
$r=1$ is the unit circle.
Let's find the angles of intersections:
$\displaystyle 2 \cos \theta = 1 \implies \cos \theta = \frac 12 \implies \theta = \pm \frac \pi 3$
We want the area outside the unit circle $\implies r >1$
And the area inside the other circle $\implies r<2 \cos \theta$
$$A= \int_{- \frac \pi 3}^{\frac \pi 3} \int_1^{2\cos \theta}\underbrace{r}_{\text{Jacobian}} \ dr \ d \theta$$
Comment: Just want to add that the particularly nice function you get for the circle centered at (1,0), given in ThePortakal's answer, is exactly what makes this problem tractable and appropriate for an exam question.
Answer: Solution without integral: the intersection area of the two circles is formed by two symmetric equal parts, each corresponding to the area of a circular segment whose angle is $\frac {2 \pi}{3}$. The area of each of these two circular segments is $\frac {\pi}{3}-\frac {\sqrt {3}}{4} $. So the area of the intersection of the region outside the first circle and the region  inside the second circle is
$$\pi- \left(\frac {2\pi}{3}-\frac {\sqrt {3}}{2} \right)= \frac {\pi}{3}+\frac {\sqrt {3}}{2}$$
Answer: Using good old cut-and-paste geometry:

The straight vertical sides are $2\frac{\sqrt3}2$ tall and have a mutual distance of $1$.
Total area $\displaystyle \color{#380}{\frac{\pi}3} + \color{#003ce4}{\frac{\sqrt3}{2}}$.


Quartiles of ungrouped data

Question: For a set of 48 randomly generated points using runif(), I use the command quantile(x,0.25) to calculate the first quartile in R.
I don't understand what formula is used by R to calculate quartiles of ungrouped data points. The value I am getting as a result is not the median of the data from 1 to 23rd position.
Comment: It sounds like you might have in mind the approach discussed at https://stats.stackexchange.com/questions/134229.  That concerns a textbook that makes exactly the same mistake: (a) lower quartile of 48 values can be determined as the middle value of the lowest 25 values, not the lowest 23.  (Maybe you are using the same bad text?)
Comment: Did you try ?quantile yet? There are a bunch of ways to calculate quantiles.
Comment: @Dave For ungrouped data what is the formula used in R or Excel?
Comment: @Dave Yes, I tried. Using the command quantile(x,0.25) in R.
Comment: Type ?quantile with the question mark to read the various ways that function can calculate quantiles.
Comment: @Dave Oh okay. I will try that. Thanks a lot.
Comment: @whuber Hi. Thanks for your comment. It's really helpful. And for 48 points, you mean median of first 24 data points right for first quartile? As (48+1)/2 is 24.5. So L is 24 and U is 25 in this case.
Comment: Yes, if "24" and "25" are counting from the same origin.  (Tukey counts inwards from the nearest extreme.)
Answer: In R, there are various ways to find the lower quartile of $n = 48$ observations. However, you need to know it is not always obvious how to find sample percentiles, especially for small samples or where there are lots of ties.
The default method in R of finding quantiles is one of several available methods, which you can read about in R documentation, and choose the one you want to use.
Suppose you have $n = 48$ normal observations, rounded to two-places:
set.seed(121)
x = round(rnorm(48, 100, 15),2)
sort(x)
 [1]  73.77  77.96  78.89  82.62  82.92  83.51
 [7]  86.94  87.08  87.89  88.67  89.29  90.88
[13]  91.51  91.93  92.97  93.99  95.98  96.17
[19]  96.84  98.78  99.14  99.38  99.77 101.09
[25] 101.63 101.92 103.16 104.46 104.52 105.56
[31] 106.35 107.10 107.53 108.17 109.56 109.65
[37] 109.98 110.11 110.72 110.81 111.73 113.61
[43] 113.70 115.01 118.24 120.66 121.04 124.23

Just looking at the sorted list, you might guess that
the lower quartile would be somewhere between 90.88 and 91.51. Halfway between is 91.245, but R chooses the number 91.3525, which is a little closer to 91.51.
Here are three ways in R to get the 25th percentile or lower quartile:
summary(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
  73.77   91.35  101.36  100.36  109.73  124.23

quantile(x)
      0%      25%      50%      75%     100%
 73.7700  91.3525 101.3600 109.7325 124.2300

quantile(x, .25)
    25%
91.3525

The type of method that is the default in R is
type=7, but different types give slightly different
answers, as shown below:
quantile(x, .25, type=5)
   25%
 91.195
quantile(x, .25, type=6)
    25%
91.0375
quantile(x, .25, type=7)
    25%
91.3525   # Default
quantile(x, .25, type=8)
    25%
91.1425

All of the types have advocates who have different
uses for quantiles in various fields of application.
If you are a student, use whatever method of finding
quantiles is standard in your class.
Quantiles are often used in practice with very large
datasets, for which the types tend to give very nearly
the same results. In practice, this 'quirkiness` in methods of finding quantiles
is more of a curiosity than a difficulty.
Comment: Thank you so much !

Quartiles of ungrouped data


How are referees so good at spotting penalties in real time?

Question: I'm thinking about american football in particular, but this question could be applicable to other sports.
When I'm watching a game on TV I've noticed the referees are typically very good about spotting penalties. I'm able to notice after the fact on the reply, but only in slow-motion and when the announcers tell me what I'm looking for.
Do the NFL's (or other sports') referees learn or use specific techniques that enable them to spot penalties easily?
Answer: Same way you get to Carnegie hall - practice, practice, practice.
First of all, not everyone can become an NFL official. They start at low ranks and work their way up. Only the very best make it to the NFL (sort of like the players). The second thing is that referees are looking for penalties. On a pass play, you're looking at the QB who has the ball. The referees is looking at the linemen to see if they're holding. Different referees are looking at different areas of the field to make sure they pick up on the appropriate things. And lastly, they spend a lot of time practicing, reviewing the rules, and there are constant reviews of the refs by the NFL which provide feedback.
Answer: That is their main job. They look out for these penalties all the time.
Since there are around 4 referees in a football game, they can consult each other if necessary.


How to schedule a MySQL query?

Question: Can I schedule a task to run at a specified interval in MySQL?
I have an inventory MySQL database. The structure is as follows:
table_1            fields: itmcode, avgcost
table_2            fields: itmcode(FK to table_1(itmcode)), quantity

The basis of the report is when I want the inventory valuation details item wise for a past date.
The avgcost and quantity fields is changed when a new purchase is posted into system. I can run a query to see the current stock valuation, but I want to be able to see the stock valuation at a previous date as well. How do I do that? For quantity, I can add the sales and deduct the purchases backwards from the current date until whatever date the report requires, but the avgcost is current since this gets updated each time a purchase is posted.
I was wondering if an automatic daily dump could be executed, similar to this:
SELECT itmcode, quantity, avgcost, (avgcost * quantity) as ttlval
FROM table_1
JOIN table_2 ON table_1.itmcode = table_2.itmcode

Is this task possible to schedule directly in MySQL or is there some other solution?
Answer: you have 2 basic options (at least):
1, Take a look at Event Scheduler
First create table eg. stock_dumps with fields
itemcode, quantity, avgcost, ttlval,dump_date (DATETIME)
CREATE EVENT `Dumping_event` ON SCHEDULE
        EVERY 1 DAY
    ON COMPLETION NOT PRESERVE
    ENABLE
    COMMENT ''
    DO BEGIN
INSERT INTO stock_dumps(itemcode, quantity, avgcost, ttlval,dump_date)
SELECT itmcode, quantity, avgcost, (avgcost * quantity)as ttlval, NOW()
  FROM table_1 JOIN table_2 ON table_1.itmcode = table_2.itmcode;
END

Please follow instructions how to enable scheduler on link posted above.
Note : Old versions of mysql don't have event scheduler
2, Create cron job/windows scheduled job:
create sql file:
INSERT INTO stock_dumps(itemcode, quantity, avgcost, ttlval,dump_date)
SELECT itmcode, quantity, avgcost, (avgcost * quantity)as ttlval, NOW()
FROM table_1 JOIN table_2 ON table_1.itmcode = table_2.itmcode;

schedule this command:
mysql -uusername -ppassword < /path/to/sql_file.sql

Comment: http://dev.mysql.com/doc/refman/5.1/en/events.html : "Event support was added in MySQL 5.1.6"
Comment: what is the minimum version of MySQL to support event scheduler?
Answer: There are generic tools available such as EyeShare or AutoSQL.net to do this. EyeShare is a more generic IT process automation tool and AutoSQL is a single purpose tool specifically to schedule queries and output them (to Excel or CSV).

How to schedule a MySQL query?


Create a file in Android with certain permission

Question: I'm developing an Android application I have to implement a function that create a folder with different files.
I wish the files in the folders were hidden, and this point is not a problem, but I want assign certain permission to files, for example I need that files are not readable / writable by user but only from application.
Also I wish the files in the folder were deleted if the application was uninstalled.
This is the code that i use to create hidden folder:
File JSONStorage = new File(Environment.getExternalStorageDirectory(), ".BMA");
if (!JSONStorage.exists()) {
    if (!JSONStorage.mkdirs()) {
        Log.wtf("log: ", "Failed to create directory");
    }
}

Answer: From official Android documentation:
String FILENAME = "hello_file";
String string = "hello world!";

FileOutputStream fos = openFileOutput(FILENAME, Context.MODE_PRIVATE);
fos.write(string.getBytes());
fos.close();

This will create file on device internal memory. This file can be read only by application. (Context.MODE_PRIVATE)
Comment: Well, with this setup it's actually what you get. File created with `Context.MODE_PRIVATE` can be edited/read only to app itself
Comment: I wish the files were protected, so that they can not be read / modified by editor.

I wish that when you try to open the file was denied access


Proving that for $n \equiv 0 \pmod{2}$, we get ${n \choose 0} < {n \choose 1} <\ldots< {n \choose n/2-1}<{n\choose n/2}$ etc.

Question: How can one prove that for $n \equiv 0$ mod $2$  we have
$${n \choose 0} < {n \choose 1} <\ldots< {n \choose n/2-1}<{n \choose n/2}>{n\choose n/2+1}>\ldots>{n\choose n-1}>{n\choose n}\,?$$
Can I say that for fixed $n$, the binomial coefficients $n \choose k$ increase with $k$ for $k < n/2$? If n is even (like in our case), then the central binomial coefficient $n \choose n/2$ is the largest one.
So $n \choose k+1$ is greater than, equal to, or less than $n \choose k$ according as $n-k$ is greater than, equal to, or less than $k+1$, that is according as $k$ is less than, equal to, or greater than $(n−1)/2$.
Is that correct?
Comment: Consider the ratio of successive binomial coefficients.
Comment: Hello. Please see whether this helps you: https://math.stackexchange.com/questions/2496836/combinatorial-proof-that-central-binomial-coefficients-are-the-largest-ones
Comment: It's a straightforward calculation
$$\binom{n}{m-1} < \binom{n}{m} \iff
\frac{n!}{(m-1)!(n-m+1)!}<\frac{n!}{m!(n-m)!} \iff \\
(m-1)!(n-m+1)! > m!(n-m)! \iff
(n-m+1) > m \iff n+1 > 2m \iff \\
\frac{n+1}{2} > m $$
100 bounties is a lot for this question, imho.
Comment: Or... look at the Pascal triangle. The number of $k$-element subsets of $n$-element set is equal to the number of its $n-k$-element subsets. So, the binomial coefficients grow up to more or less $k=n/2$ and then they fall.
Comment: Changed my proof. Is that O.K.?
Answer: All you need is this:
Lemma: $${n\choose k-1}<{n\choose k} \iff k\leq {n\over 2}$$
Proof: You can ask yourself:
In how many ways can we choose $k$ people, including one president, from a population of $n$ people?
First answer:
You choose $k-1$ people from $n$ people and then from the rest of them a president, so we have $${n\choose k-1}\cdot {n-k+1\choose 1}$$
Second answer:
You choose $k$ people from $n$ people and then from these selected you choose a president, so we have $${n\choose k}\cdot {k\choose 1}$$
So we have:$${n\choose k-1}\cdot (n-k+1)=k{n\choose k} $$ and from here we get $${n\choose k-1}<{n\choose k} \iff k\leq {n\over 2}$$
Answer: Let $n=2m$. You want to prove, that for

$0 \leq k<m$ we have ${{n}\choose{k}} < {{n}\choose{k+1}}$
$m\leq k< n$ we have ${{n}\choose{k}} > {{n}\choose{k+1}}$

First 1.: Let  $0 \leq k<m$. Let's study the ratio
$$\frac{{{n}\choose{k+1}}}{{{n}\choose{k}}}
= \frac{\frac{n!}{(k+1)!(n-(k+1))!)}}{\frac{n!}{k!(n-k)!}}
= \frac{k!(n-k)!}{(k+1)!(n-k-1)!}
= \frac{n-k}{k+1}
> 1$$
since $n-k > n - m = n- \frac{n}{2} = \frac{n}{2} = m \geq k+1$.
Then 2.: Assume $m\leq k< n$. It goes similarly, the ratio
$$\frac{{{n}\choose{k}}}{{{n}\choose{k+1}}}
= \frac{k+1}{n-k}
> 1
$$
because now $n-k \leq n-m = m < k+1 $.
Answer: Let's try using induction on $m$ where $n=2m$. For $m=0$, there is nothing to be proven, for $m=1$, then ${2\choose 0} = {2\choose 2} = 1 < 2 = {2\choose 1}$.
To get the induction step, observe that ${n+2\choose k} = {n+1\choose k-1} + {n+1 \choose k} = {n\choose k-2} + 2 {n\choose k-1} + {n\choose k}$. Suppose that $k<m+1 = \frac{n+2}{2}$, then
\begin{align*}
{2(m+1)\choose k} - {2(m+1)\choose k+1} &= {2m \choose k-2}+{2m\choose k-1}-{2m \choose k}-{2m\choose k+1}\\
&= \left[ {2m \choose k-2} - {2m \choose k} \right] + \left[ {2m\choose k-1} -{2m\choose k+1}\right]
\end{align*}
The first term is directly strictly negative by induction, the second term is too if $k<m$, if $k=m$ then ${2m\choose m-1}={2m\choose m+1}$ and the second term is $0$. This proves that ${2n+2\choose k}<{2n+2\choose k+1}$. The same can be done when $k>m+1$ to obtain ${2n+2\choose k}<{2n+2\choose k-1}$.


Does VirtualTreeView manage the memory for the userdata of nodes or can I?

Question: I started playing with VirtualTreeView and it seems from all the web info and documents that VirtualTreeView manages the memory of the user provided data (records if any). One does not have to New() and Dispose() any user data. One only has to tell VirtualTreeView the size of userdata and assign its field values after calling GetNodeData.
What if I wanted to manage that? Can it be done? Can I create the userdata as I want it and add it to my own lists and only provide a pointer of each record to VirtualTreeView without it interfering?
Answer:
One only has to tell VirtualTreeView the size of userdata and assign its field values after calling GetNodeData

This is not entirely correct - one also has to take care finalizing the user data record should it contain data types which require it - ie reference counted types like string and interface. This is done in OnFreeNode event. So if your data record contains string
type
  TMyNodeData = record
    Foo: string;
  end;

then you have to assign empty string to it when node's data is freed:
procedure TForm1.VT_FreeNode(Sender: TBaseVirtualTree; Node: PVirtualNode);
var ND: PMyNodeData;
begin
  ND := Sender.GetNodeData(Node);
  if(ND <> nil)then begin
     ND.Foo := '';
  end;
end;

othervise you'll leak that string.
The siplest way to manage the memory yourself is not to use the VT's user data record at all, or only have a pointer to some other data structure in it.
Answer: No you can not. VTW uses fixed size of user data thus it provides for fast calculations and access.
BUT you can request SizeOf(pointer) size of userdata and then store the pointer to your data frames in the userdata provided by VTW. Types of data at those frames and their sizes and their allocation/deallocation you would manage yourself. Still you would have to store pointers to it in VTW-provided userdata, but other than that you would not depend upon it.
In newer Delphi (2010+) you can also use TDictionary<Pointer, you-data-type> object, and retrieve the data using PVirtualNode given by VTW as a key into that container. That way you would be able to set UserData to have zero size, so VTW would only allocate nodes memory pool for its own needs.
But as far as possible you better user UserData - for it was specifically speed-optimized for you by VTW team.


GLM Poisson Regression with Overdispersion

Question: I use R for a poisson regression (GLM) to test for a difference in the number of years of schooling based on race (6 categories) and religion (3 categories), see structure of dataset and output below. Since the original poisson model suffers from rather significant overdispersion I conducted the same analysis but with a quasipoisson distribution.
Structure of dataset:

GLM Quasipoisson regression:

ANOVA - Test predictors relative to the full model:

The output of the quasipoisson regressions shows one statistically significant interaction. I read the following "use F tests with an empirical scale parameter instead of chi-squared (The R Book, Crawley M. J. 2013.  p. 570)" as yet another suggested technique to deal with overdispersion. Hence, I conduct an ANOVA with an F-test for the model which then shows that the interaction term is not statistically significant. What is the correct way to interpret this result? Should I, based on the ANOVA state that the interaction is not statistically significant and then instead go on trying a model without the interaction term?
Answer: Tests of significance are not used to determine whether or not variables are meant to be included in the model or not. If the goal is prediction or variable selection, you probably shouldn't be using quasilikelihood at all. That said, it looks like you've tested for the interaction of religion with race using the method that Crawley has advocated. The reason is that you need an estimate of the covariates and the dispersion parameter under the "reduced" model.
Comment: Ok, thanks for your answer. So the correct interpretation is that the interaction term is not statistically significant although one interaction is significant in the GLM output?
Comment: @JanC Not quite. According to Crawley, the significance test in the `Pr(>|t|)` is exactly the chi-square test that he advocates against. The "reduced" model requires a re-estimate of the dispersion parameter to give you the $F$ test. I think you can fit that full model as `~Race * Religion` and the reduced model as `~Race*Religion - RaceMulato:ReligionEvangelic`. Then the two models can be supplied to the `aov` command.


Likelihood function of truncated data

Question: I am having a little trouble understanding the concept and derivation of the likelihood of truncated data.
For example, if I want to find the likelihood function based on a sample from a distribution, but when taking a sample from the distribution, I observe the truncated values (where there is a cut-off of $M$, i.e. any $x_{i}>M$ is recorded as $M$):
$ x_{1}, x_{2}, M, x_{3}, M, x_{4}, x_{5}, ..., x_{10}$
where the number of $M$ values is $m$. Then, the likelihood is supposedly given by:
$L(x;\theta) = \prod_{i=1}^{10}f(x_{i};\theta)*[P(X>M)]^{m}$
I would very much appreciate an explanation/proof of why this is so, importantly why the second factor is as it is. Intuitively and mathematically if possible. Thanks very much in advance.
Comment: As @Alecos points out, you're using "truncated" idiosyncratically. "Censored" is the usual term.
Comment: What is lower-case "$m$"?
Comment: It is the number of occurences of $M$.. i.e. I have observed $10 + m$ data points, of which $10$ are not truncated, and $m$ of them are (I observe these $m$ picks, all with value $M$)
Comment: Some other terms you may want to search on: "ceiling/floor effects", "beta regression", and "zero-inflated models."
Answer: What you describe needs special treatment, it is not what we usually mean by "truncated random variables"-and what we usually mean is that the random variable does not range outside the truncated support, meaning that there is not a concentration of probability mass at the point of truncation. To contrast cases:
A) "Usual" meaning of a truncated rv
For any distribution that we truncate its support, we must "correct" its density so that it integrates to unity when integrated over the truncated support.
If the variable has support in $[a,b]$, $-\infty < a < b < \infty$, then (pdf $f$, cdf $F$)
$$\int_a^bf_X(x)dx = \int_a^Mf_X(x)dx+\int_M^bf_X(x)dx = \int_a^Mf_X(x)dx + \left[1-F_X(M)\right]=1 $$
$$\Rightarrow \int_a^Mf_X(x)dx = F_X(M)$$
Since the LHS is the integral over the truncated support, we see that the density of the truncated r.v., call it $\tilde X$, must be
$$f_{\tilde X}(\tilde x) = f_{X}(x\mid X\le M)=f_X(x)dx\cdot \left[F_X(M)\right]^{-1} $$
so that it integrates to unity over $[a, M]$. The middle term in the above expression makes us think of this situation (rightfully) as a form of conditioning -but not on another random variable, but on the possible values the r.v. itself can take.
Here a joint density/likelihood function of a collection of $n$ truncated i.i.d r.v.'s would be $n$ times the above density, as usual.
B) Probability mass concentration
Here, which is what you describe in the question, things are different. The point $M$ concentrates all the probability mass that corresponds to the support of the variable higher than $M$. This creates a point of discontinuity in the density and makes it having two branches
$$\begin{align}
f_{X^*}(x^*) &= f_X(x^*) \qquad x^*<M\\
f_{X^*}(x^*) &= P(X^* \ge M) \qquad x^*\ge M\\
\end{align}$$
Informally, the second is "like a  discrete r.v." where each point in the probability mass function represents actual probabilities.
Now assume that we have $n$ such i.i.d random variables, and we want to form their joint density/likelihood function. Before looking at the actual sample, what branch should we choose? We cannot make that decision so we have to somehow include both. To do this we need to use indicator functions: denote $I\{x^*\ge M\}\equiv I_{\ge M}(x^*)$ the indicator function that takes the value $1$ when $x^*\ge M$, and $0$ otherwise. The density of such a r.v. can be written
$$f_{X^*}(x^*) = f_X(x^*)\cdot \left[1-I_{\ge M}(x^*)\right]+P(X^* \ge M)\cdot I_{\ge M}(x^*) $$
and therefore the joint density function of $n$ such i.i.d. variables is
$$f_{X^*}(\mathbf X^*\mid \theta) = \prod_{i=1}^n\Big[f_X(x^*_i)\cdot \left[1-I_{\ge M}(x^*_i)\right]+P(X^*_i \ge M)\cdot I_{\ge M}(x^*_i)\Big]$$
Now, the above viewed as a likelihood function, the actual sample consisting of realizations of these $n$ random variables comes into play. And in this sample, some observed realizations will be lower than the threshold $M$, some equal. Denote $m$ the number of realizations in the sample that equals $M$, and $v$ all the rest, $m+v=n$. It is immediate that for the $m$ realizations, the corresponding part of the density that will remain in the likelihood will be the $P(X^*_i \ge M)$ part, while for the $v$ realizations, the other part. Then
$$\begin{align}  L(\theta\mid \{x_i^*;\,i=1,...n\})&= \prod_{i=1}^v\Big[f_X(x^*_i)\Big]\cdot \prod_{j=1}^m\Big[P(X^*_j \ge M)\Big] \\& = \prod_{i=1}^v\Big[f_X(x^*_i)\Big]\cdot \Big[P(X^* \ge M)\Big]^m\\
\end{align}$$
Comment: Thank you. I very much appreciate the reply. I guess my main issue is the first point under section b)... i.e., how the "second branch" of the pdf is defined. It is a discrete pmf and doesn't really define a pdf from the definition of a pdf. Could this section be explained further? Thanks very much.
Comment: These random variables are called "mixed-type", i.e. they are partly continuous and partly discrete. Intuitively it makes obvious sense, as your questions shows. For a rigorous treatment, look up "mixed type random variables" or "mixed type distributions". DON'T confuse them with "mixtures".
Answer: The likelihood theory is a fairly general framework. Most textbooks
state results for the separated cases of continuous r.vs
and for that for discrete r.vs. However mixed cases
occur in practice, as is the case here.
For a discrete r.v. $A$, the likelihood of an observation $a$ is
defined as the probability of getting the observed value $a$, say
$p_A(a)$. For a continuous r.v. the likelihood $L$ is usually defined
as the density at $x$, say $f_X(x)$.  However in practice one only
knows that $x_{\textrm{L}} < X < x_{\textrm{U}}$ - because of a
limited measurement precision, and $\Pr\left\{x_{\textrm{L}} < X <
  x_{\textrm{U}}\right\}$ should be used as likelihood. Taking
$x_{\textrm{L}}:= x - \textrm{d}x/2$, $x_{\textrm{U}}:= x +
\textrm{d}x/2$ with $\mathrm{d}x$ small, we get $f_X(x)$ up to a
multiplicative $\mathrm{d}x$ which does not matter.  So the usual
definition can be viewed as implicitly assuming an infinite precision
on the observation.
For a couple of r.vs $A$ and $X$ with mixed joint type
discrete/continuous, the likelihood will be the joint distribution,
which is usually expressed using conditional distributions, e.g.
$$
  L := \textrm{Pr}\left\{ A = a, \, x_{\textrm{L}} < X < x_{\textrm{U}} \right\}
 = \textrm{Pr}\left\{ A = a \right\} \times
\textrm{Pr} \left\{x_{\textrm{L}} < X <  x_{\textrm{U}} \, \vert\, A = a\right\}.
$$
Thus for an interval $(x_{\textrm{L}},\, x_{\textrm{U}})$ with small
length $\textrm{d}x$, $L$ is $p_A(a)$ times the density of $X$
conditional on $\{A=a\}$, say $f_{X \vert A}(x \,\vert \,a)$.  Again,
we omit the $\mathrm{d}x$ term.
Now let us come back to your example, and consider only one
observation. Then $A = 1_{\{X > M\}}$ is a Bernoulli r.v with success
probability $\Pr\{X > M\}$.  Depending on $X > M$ or not, either you
observe only $A = 1$ or you observe both $A = 0$ and the value $x$ of
$X$.  In both cases you use the formula above, but $(x_{\textrm{L}},\,
x_{\textrm{U}})$ is taken either as $(M,\,\infty)$ or as an interval
of small length $\textrm{d}x$ containing $x$. Indeed this gives
$$
L =
\begin{cases}
  \textrm{Pr} \left\{X  > M  \right\} \times 1 & \textrm{if } X > M \textrm{ i.e.  } A =1,\\
  \textrm{Pr} \left\{X \leq M\right\} \times f_{X \vert A}(x \,\vert \,a)\,\textrm{d}x &
  \textrm{if } X \leq M \textrm{ i.e.  } A = 0.
\end{cases}
$$
Since $f_{X \vert A}(x \,\vert \,0) = f_X(x) / \textrm{Pr} \left\{ X
  \leq M \right\}$, the likelihood is simply $f_X(x)\,\textrm{d}x$ in
the second case and we get the claimed likelihood, up to the
$\mathrm{d}x$ term for an observation with infinite precision. When
independent observations $A_i$ and $X_i$ are made, the likelihood is
obtained as the product of the marginal likelihoods leading to the
expression in the question.

Likelihood function of truncated data


How to inherit react SectionList prop type in typescript?

Question: I am creating a wrapper around SectionList, which should take all props that SectionList takes, plus my custom props. How can I configure typescript for that?
Here's my attempt:
import React from 'react';
import { SafeAreaView, SectionList } from 'react-native';

interface EnhancedSectionListProps {
  ...SectionListProps;   // this obviously won't compile, but shows what I'm trying to achieve
  enhancedProp: string;
}

export const EnhancedSectionList = (props: EnhancedSectionListProps) => {
  return (
    <SafeAreaView>
      <SectionList {...props} />
      // use my `enhancedProp` here
    </SafeAreaView>
  );
};

PS: We are not using the prop-types library.
Answer: To accomplish the task, you will need to extend the interface. For example:
interface EnhancedSectionListProps extends SectionListProps {
  enhancedProp: string;
}

Comment: Thanks, being a Java dev, I should have tried this. I had to use `SectionListProps`, would you know what generic type it expects?
Comment: I think SectionListProps has a mechanism of extending the default props. Sometimes, in place of ````any````, you can pass an interface to extend it by that interface. So, can you check whether ````SectionListProps<{enhancedProp: string}>```` works in place of ````EnhancedSectionListProps````? However, I can be wrong. This actually depends on how the ````SectionListProps```` is implemented.


Get a single date value from a date field on a node with entity_metadata_wrapper()

Question: I want to retrieve a date value from a date field attached to a node with entity_metadata_wrapper.
So far:
$node_wrapper->field_tournament_date->value();

Provides an array. Not many other methods work. So, I have to do:
$date = $node_wrapper->field_tournament_date->value();
$start_date = format_date($date['value'], 'short');

Is there a cleaner way of getting the value of a date field without breaking it up like that?
Answer: In these situations with meta_data_wrapper you could put the array-value before the value function, like this:
$date = $node_wrapper->field_tournament_date->value->value();

Answer: If your server's running PHP 5.4 you could use:
$start_date = format_date($node_wrapper->field_tournament_date->value()['value'], 'short');

since function array dereferencing has been added.
Other than that I don't think there's a way of reducing your code to be honest. If you're manipulating date fields a lot you might consider creating a small helper function, something like
function MYMODULE_format_date_field($value, $column = 'value', $format = 'short') {
  return format_date($value[$column], $format);
}
$start_date = MYMODULE_format_date_field($node_wrapper->field_tournament_date->value());

or similar. It's not the best but it might save a bit of time.
Comment: Yeah, dereferencing is a no-go. Not 5.4. I wish. That would be ultra convenient. I just figured there was some native method of retrieving it from a date field, without loading up a date wrapper. No manipulating values, just going for quick custom display renderings with no fuss.
Answer: I use the following approach:
foreach ($node_wrapper->field_tournament_date->getIterator() as $date_wrapper) {
  // dpm($date_wrapper->getPropertyInfo() // to see what properties you have
  $start_date = $date_wrapper->value->value();
  $end_date = $date_wrapper->value2->value();
  $duration = $date_wrapper->duration->value();
}

Answer: I used the following:
$entity->field_date->get( 0 )->value->value( );

The call to get is needed because date fields are lists.


Find my mistake: Solving quadratic programm by "brute force" triangularization

Question: Do you know the feeling when you have proven something totally incredible? In fact, it is so incredible you do not believe it yourself. So you start looking for your mistake, for the one, small logical flaw that turns all your believes upside down. And you just can't find it...
That pretty much describes my problem. Your task: Find my mistake and make me hit my head against some imaginary wall. (A real-world-wall would hurt to much, wouldn't it?)
Suppose we have some quadratic program:
$$
 \min_{x\in \mathbb{R}^d} \frac{1}{2}x^T Q x - c^Tx
$$
with positive semidefinite $Q$. It is well known that we can solve that program by solving $Qx = c$.
Now, if $x=[x_1,\ldots, x_d]$ and $Q=[q_{i,j}]_{i,j=1,\ldots,d}$, we have the following relation:
$$
x^TQx=\sum_{j=1}^d\sum_{i=1}^dq_{i,j}x_ix_j=\sum_{j=1}^{d-1}\sum_{i=j+1}^d(q_{i,j}+q_{j,i})x_ix_j+\sum_{i=1}^nq_{i,i}x_i^2=x^TQ'x
$$
with $$
Q'=[q'_{i,j}]_{i,j=1,\ldots, d}\quad q'_{i,j}=\begin{cases}q_{i,j}& \text{if }i = j\\ q_{i,j}+q_{j,i}&\text{if }i > j\\ 0&\text{if }i < j\end{cases}
$$
Thus, $Q'$ is a triangular matrix, and as $x^TQx=x^TQ'x$, solving our quadratic program is equivalent to solving the problem
$$
 \min_{x\in \mathbb{R}^d} \frac{1}{2}x^T Q' x - c^Tx,
$$
which is rather trivial, as $Q'$ is triangular and $Q'x=c$ therefore easy to solve.
So, I propose that solving $Q'x=c$ is equivalent to solving $Qx=c$ - that has to be the fastest triangularization method I have ever seen...
What am I doing wrong?
Comment: Well, finding a counterexample isn't that hard - the first example I tried was one. Seeing my problem seems to be hard, however - the example didn't help me with that.
Answer: In the equivalence of
$$ \min_{x} \frac 12 x^tQx - c^t x $$
with $Qx = c$ we use the symmetry of $Q$ (otherwise in doesn't hold). Note that for $q(x) = \frac 12 x^tQx - c^tx$ we have
\begin{align*}
  q(x+h) &= \frac 12 (x+h)^tQ(x+h) -c^t x - c^th\\
         &= q(x) + \frac 12 x^t(Q+Q^t)h - c^th + \frac 12 h^tQh
\end{align*}
so $q'(x) = \frac 12 x^t(Q+Q^t) - c^t$ and $x$ being critical is equivalent to $\frac 12(Q + Q^t)x = c$. Note that your $Q'$ fulfills $Q^t + Q = (Q')^t + Q'$.
Comment: Thanks! And yes, it was as obvious as I feared...


Problems changing the date with a trigger in oracle

Question: I want to create a trigger that can modify or insert a date in my table "contracts":
"If a client with a current contract signs a new one, the end date
Of the previous one will be a day before the start date of the new one"
So, my trigger is:
CREATE OR REPLACE TRIGGER TRIGGER_D
BEFORE
INSERT OR UPDATE ON CONTRACTS
FOR EACH ROW
DECLARE
idcliente CONTRACTS.CLIENTID%TYPE;
fecha_fin CONTRACTS.ENDDATE%TYPE;
BEGIN
  if :NEW.CLIENTID = idcliente
    then
    if TO_CHAR(fecha_fin) IS NULL OR TO_CHAR(fecha_fin) > TO_CHAR(:NEW.STARTDATE)
      then
        INSERT INTO CONTRACTS (ENDDATE) VALUES
        (:NEW.STARTDATE);
        update CONTRACTS
        set
          ENDDATE = :NEW.STARTDATE;
    end if;
  end if;
END;

I tried to insert a row like this:
INSERT INTO CONTRACTS (CONTRACTID, CLIENTID, STARTDATE, ENDDATE, CONTRACT_TYPE ,ADDRESS ,TOWN, ZIPCODE, COUNTRY)
VALUES              ('TW42260/09','99/98480296/22T',TO_DATE('18/04/17','DD/MM/YY'), NULL ,'Flat Rate Lover','69 Citadel Street, Ground floor, door 1','Summer forest', '11164' ,'Taiwan, Province of China');

This trigger compiles well, but they are not doing anything on the table.
Comment: You aren't setting `idcliente` so the first comparison will never be true (same for `fecha_fin`), so the trigger does nothing. If you insert/update inside the trigger it will fire again,so you're risking recursion, and the update will get a mutating table error. Does your assignment require you to use a trigger?
Comment: Why do you compare STRINGS by "bigger than"?
Comment: It was a work fot the university
Answer: You want something like this:
CREATE OR REPLACE TRIGGER TRIGGER_D
BEFORE
INSERT OR UPDATE ON CONTRACTS
FOR EACH ROW
DECLARE
BEGIN
  UPDATE CONTRACTS
  SET    ENDDATE   = :NEW.STARTDATE - INTERVAL '1' DAY
  WHERE  CLIENTID  = :NEW.CLIENTID
  AND    ENDDATE   > :NEW.STARTDATE
  AND    STARTDATE < :NEW.STARTDATE;
END;

However, it is updating the table that the trigger is monitoring and it seems likely that the trigger will cause updates that will invoke the trigger again and get a mutating table error.
You might be better to wrap your business logic in a stored procedure where it can perform the update on the previous rows and then do the insert/update. Then, rather than performing DML statements directly on the table instead invoke the stored procedure.
Comment: Thanks you very much, I just put "insert" on the trigger and the mutant table disapear, because I didn't really use the "update" option. Thank you so much :)


Running cgminer in the cloud

Question: I am trying to run an Amazon instance AMI: ami-b93013d0. The name would have you think that it has cgminer on it. But I can only find minerd. Does anyone know how I can find cgminer on this AMI?
Answer: Use find / -name cgminer to find the executable, assuming it's there.
Also, CGMiner only runs against CPUs (and certain other dedicated hardware, which Amazon doesn't have), AMD/ATI cores work the best by 10x. Minerd is an ancient program, which you should most definitely not be using anymore. If you need to run against a CPU, then you'll want a copy of pool-cpuminer.
I should warn you, no matter what coin you intend to mine, CPU mining costs more in electricity than you'll ever get out of it. Running a CPU miner against anything Amazon has will likely result in the same situation, even running 3-4+ year old AMD/ATI cards usually costs more in electricity than you'll get out of it. Mining is a sport for the dedicated, I'd be happy to walk you through the process, but unless you're ready for some dedication and minimal financial investment be ready to just burn money.
Finally, Server Fault is for Professional System Administration topics only. Bitcoin.StackExchange.com is a site where you can ask crypto-curreny questions (alt-coin questions are accepted there too, as they're almost all based on the Bitcoin protocol, if now the PoW Algorithm) .

Running cgminer in the cloud


Python - Transliterate German Umlauts to Diacritic

Question: I have a list of unicode file paths in which I need to replace all umlauts with an English diacritic.  For example, I would ü with ue, ä with ae and so on.  I have defined a dictionary of umlauts (keys) and their diacritics (values).  So I need to compare each key to each file path and where the key is found, replace it with the value.  This seems like it would be simple, but I can't get it to work.  Does anyone out there have any ideas?  Any feedback is greatly appreciated!
code so far:
# -*- coding: utf-8 -*-

import os

def GetFilepaths(directory):
    """
    This function will generate all file names a directory tree using os.walk.
    It returns a list of file paths.
    """
    file_paths = []
    for root, directories, files in os.walk(directory):
        for filename in files:
            filepath = os.path.join(root, filename)
            file_paths.append(filepath)
    return file_paths

# dictionary of umlaut unicode representations (keys) and their replacements (values)
umlautDictionary = {u'Ä': 'Ae',
                    u'Ö': 'Oe',
                    u'Ü': 'Ue',
                    u'ä': 'ae',
                    u'ö': 'oe',
                    u'ü': 'ue'
                    }

# get file paths in root directory and subfolders
filePathsList = GetFilepaths(u'C:\\Scripts\\Replace Characters\\Umlauts')
for file in filePathsList:
    for key, value in umlautDictionary.iteritems():
        if key in file:
            file.replace(key, value) # does not work -- umlauts still in file path!
            print file

Comment: Possible duplicate of [Why doesn't calling a Python string method do anything unless you assign its output?](http://stackoverflow.com/questions/9189172/why-doesnt-calling-a-python-string-method-do-anything-unless-you-assign-its-out)
Comment: I'm not sure what the proper term is, but "diacritic" refers to the two dots used to mark umlauts, not the two-letter orthographic alternative.
Comment: replace does not modify in place, it returns the modified string…
Answer: The replace method returns a new string, it does not modify the original string.
So you would need
file = file.replace(key, value)

instead of just  file.replace(key, value).

Note also that you could use the translate method to do all the replacements at once, instead of using a for-loop:
In [20]: umap = {ord(key):unicode(val) for key, val in umlautDictionary.items()}

In [21]: umap
Out[21]: {196: u'Ae', 214: u'Oe', 220: u'Ue', 228: u'ae', 246: u'oe', 252: u'ue'}

In [22]: print(u'ÄÖ'.translate(umap))
AeOe

So you could use
umap = {ord(key):unicode(val) for key, val in umlautDictionary.items()}
for filename in filePathsList:
    filename = filename.translate(umap)
    print(filename)

Answer: Replace the line
file.replace(key, value)

with:
file = file.replace(key, value)

This is because strings are immutable in Python.
Which means that file.replace(key, value) returns a copy of file with replacements made.
Comment: nice copy paste from the dublicate question ^^
Comment: This answer has nothing todo with the question asked.


How do you update a Cloud Property for an IoT Device in Azure IoT Hub using the SDKs?

Question: Is there a way to update a Cloud Property for an IoT Device in Azure IoT Hub?
The DTDL for the device Cloud Property is:
<code>{
    "@id": "dtmi:solutionModel:modelDefinition:rkkkbmsz:brcdv6wcme:DeviceSerialNumber",
    "@type": [
        "Property",
        "Cloud",
        "StringValue"
    ],
    "displayName": {
        "en": "Device Serial Number"
    },
    "name": "DeviceSerialNumber",
    "schema": "string",
    "minLength": 1,
    "trimWhitespace": true
},
</code>
I am using the Azure IoT SDKs and C#.
deviceClient.UpdateReportedPropertiesAsync(reportedProperties, cancellationToken);
The property is showing up as "_unmodeleddata".  The JSON is:
<code>{
    "_unmodeleddata": {
        "DeviceSerialNumber": "dsn1i90NEW"
    },
    "_eventtype": "Property",
    "_timestamp": "2022-08-14T23:13:45.966Z"
}
</code>
Answer: The Cloud Property declared in the DTDL model is visible only on the cloud facing side, in other words, there is no either any message exchange pattern between the device and IoT Hub or visibilities.
It based on the IoT Solution, where and how the Cloud Property is stored and managed. In the Azure IoT Hub the Cloud Property can be stored in the device twins such as the property tags.
In the Azure IoT Central, the Cloud Property is using own underlying storage (not the device twin tags) and the REST APIs for handling their values.
Note, that the IoT PnP Device should not send the Cloud Property to the IoT Hub, that's the reason why you are receiving _unmodeleddata for property DeviceSerialNumber
Comment: It looks like there is a bug for updating a cloud property in a respect of the device model and IoT Central extension to DTDL. The response code 202 (Accepted) should be used when the value is match with the IoT Extension to DTDL (such as Min value, Max value, etc.). In the case if the device model doesn't have this IoT Extension, the response value should be 200 (OK). Note, that the Form UI is working correct when the IoT Extension to DTDL is used it.
Comment: So are you saying that if I am using IoT Central it can only be done via the Rest API?
Comment: Currently, the Cloud Property in the Azure IoT Central App can be populated via the REST API or manually from the Views/Form, there is no SDKs. Note, that using the *Manage device/Device properties* feature in the IoT Central pane is for managing the device desired properties without the model respect, you can see it, when you send the { "DeviceSerialNumber": "1234" }
Comment: So there is no way to use the Azure IoT SDKs to update the Digital Twins for cloud properties for devices stored in IoT Central vs IoT Hub?
Comment: I just tried updating a Cloud property using the IoT Central APIs.  I am getting Http Response code of 202.  Are Cloud Properties Read Only?  Do you know why I am getting a 202?


530 Login incorrect - vsftpd w/SSL (Not using PAM?)

Question: I cannot log in with any local users on my vsftpd running on CentOS6 32-bit. The server simply reports login incorrect.
My log files look like this:
[doctorblue@guardian ~]$ sudo tail /var/log/vsftpd.log /var/log/secure
==> /var/log/vsftpd.log <==
Fri Jun  7 01:37:33 2013 [pid 2453] DEBUG: Client "188.xxx.196.134", "Connection terminated without SSL shutdown - buggy client?"
Fri Jun  7 01:41:52 2013 [pid 2489] CONNECT: Client "188.xxx.196.134"
Fri Jun  7 01:41:53 2013 [pid 2488] [doctorblue] FAIL LOGIN: Client "188.xxx.196.134"
Fri Jun  7 01:41:54 2013 [pid 2489] DEBUG: Client "188.xxx.196.134", "Connection terminated without SSL shutdown - buggy client?"
Fri Jun  7 01:44:17 2013 [pid 678] CONNECT: Client "188.xxx.196.134"
Fri Jun  7 01:44:18 2013 [pid 677] [doctorblue] FAIL LOGIN: Client "188.xxx.196.134"
Fri Jun  7 01:44:19 2013 [pid 678] DEBUG: Client "188.xxx.196.134", "Connection terminated without SSL shutdown - buggy client?"
Fri Jun  7 01:45:39 2013 [pid 704] CONNECT: Client "188.xxx.196.134"
Fri Jun  7 01:45:51 2013 [pid 703] [doctorblue] FAIL LOGIN: Client "188.xxx.196.134"
Fri Jun  7 01:45:52 2013 [pid 704] DEBUG: Client "188.xxx.196.134", "Connection terminated without SSL shutdown - buggy client?"

==> /var/log/secure <==
Jun  7 01:31:22 guardian sshd[1742]: pam_unix(sshd:session): session closed for user doctorblue
Jun  7 01:42:53 guardian sshd[1814]: Received signal 15; terminating.
Jun  7 01:43:01 guardian sshd[568]: Server listening on 0.0.0.0 port 22.
Jun  7 01:43:01 guardian sshd[568]: Server listening on :: port 22.
Jun  7 01:43:07 guardian sshd[622]: Accepted publickey for doctorblue from 188.xxx.196.134 port 60406 ssh2
Jun  7 01:43:07 guardian sshd[622]: pam_unix(sshd:session): session opened for user doctorblue by (uid=0)
Jun  7 01:44:13 guardian sudo: doctorblue : TTY=pts/0 ; PWD=/home/doctorblue ; USER=root ; COMMAND=/sbin/service vsftpd start
Jun  7 01:45:23 guardian sudo: doctorblue : TTY=pts/0 ; PWD=/home/doctorblue ; USER=root ; COMMAND=/usr/bin/vim /etc/pam.d/vsftpd
Jun  7 01:45:34 guardian sudo: doctorblue : TTY=pts/0 ; PWD=/home/doctorblue ; USER=root ; COMMAND=/sbin/service vsftpd restart
Jun  7 01:46:13 guardian sudo: doctorblue : TTY=pts/0 ; PWD=/home/doctorblue ; USER=root ; COMMAND=/usr/bin/tail /var/log/vsftpd.log /var/log/secure

And these are my configuration files.
==> /etc/vsftpd/vsftpd.conf <==
# Allow anonymous connections
anonymous_enable=NO

# Log logins
dual_log_enable=YES

# Allow login from system users
local_enable=YES

# Prevent people from going to off-limit areas
chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd.chroot_list

# Enable recursive file listing
ls_recurse_enable=YES

# Enable commands for writing to the filesystem
# These are: STOR, DELE, RNFR, RNTO, MKD, RMD, APPE and SITE
write_enable=YES

# Enable standalone mode
listen=YES

# Maximum connections per IP address, helps defeat DoS attacks
max_per_ip=3

# Expect SSL handshake without request (Implicit FTPS)
implicit_ssl=NO

# SSL support
ssl_enable=YES

# Allow SSL for anonymous connections
allow_anon_ssl=NO

# Disallow insecure connections for data transfer
force_local_data_ssl=YES

# Disallow insecure connections for login
force_local_logins_ssl=YES

# Set the cipher used for encryption
ssl_ciphers=HIGH

# Enable TLS v1
ssl_tlsv1=YES

# Enable SSL v2
ssl_sslv2=NO

# Enable SSL v3
ssl_sslv3=YES

# RSA certficiate location
rsa_cert_file=/etc/vsftpd/vsftpd.pem

==> /etc/pam.d/vsftpd <==
#%PAM-1.0
session    optional     pam_keyinit.so    force revoke
auth       required     pam_listfile.so item=user sense=deny file=/etc/vsftpd/ftpusers onerr=succeed
auth       required     pam_shells.so
auth       include      password-auth
account    include      password-auth
session    required     pam_loginuid.so
session    include      password-auth

Answers to other questions suggest that I try commenting out auth required pam_shells.so and restart the vsftp daemon but doing so makes no difference, it still reports incorrect login, and the exact same entries appear in the log files.
Comment: Very silly question, but: are you sure the user isn't trying to connect with a SFTP client? (as opposed to FTPS)
Comment: @Andrew No I have tried connecting with Filezilla using FTP requiring explicit SSL. I also tried disabling SSL to no avail. I decided to wipe my VM and install Debian instead, and now it works perfectly fine with the same configuration. Odd
Comment: When you get a chance, please self-answer the question and accept the answer since it is no longer possible to troubleshoot this.
Answer: I cannot troubleshoot the issue anymore as I fixed it by reinstalling the virtual machine.


A linux tftpd server that can log the incoming requests to stdout/stderr

Question: I find it crazy that this doesn't work out of the box, but neither atftpd nor tftpd-hpa seem to have any way to log to the console. I've run them in the foreground with verbosity right up but clearly that's not the right magic incantation.
I'm running a short-lived server for a firmware upload and I want to see the requests (and ideally when they've completed).
Answer: After some more experimentation, here's what James was aiming for back in 2010, in a form tested to work on Kubuntu Linux 20.04 LTS:
sudo atftpd --logfile - --no-fork --verbose=5 --trace --daemon /srv/tftp

The key details are as follows:

--logfile /dev/stdout apparently doesn't work in the X11 terminal emulators I was trying, but --logfile - does. What's misleading is that it appears to work, in that some output is produced.
--verbose 3 doesn't produce any request logging. You need at least --verbose 5 for that. (--verbose 7 --trace is maximum verbosity)
/srv/tftp is the default TFTP root path on *buntu that will be set up when you install atftpd.

Here's the result:
ssokolow@monolith ~ % sudo atftpd --logfile - --no-fork --verbose=5 --daemon /srv/tftp
Jan 16 18:33:28 monolith atftpd[2933437.140296925675584]: Advanced Trivial FTP server started (0.7)
Jan 16 18:33:32 monolith atftpd[2933437.140296925587200]: Serving default to 192.168.0.2:41416

Answer: Try something like this:
atftpd --logfile /dev/stdout --no-fork --verbose 3 --daemon /tftpboot


What is the quietest way to practice trumpet that is still doable?

Question: I'm in a place where even playing with a mute is too loud. I've heard some good things about silent brass devices. But I was wondering how quiet can you physically get with trumpet and still maintain a daily practice without making it too hard too play?
Comment: @AndyBonner - when you produce a note on a brass instrument you're not just blowing; you're setting up a standing wave in the air column within the instrument. The resonant peaks (aka slots) pull any produced pitch towards one of those slots. Any wind controller that let your lips buzz freely without having those slots to dictate the pitches would not feel like a trumpet nor produce control signals that sounded trumpety. As far as I know the only way to produce the effects of a resonant air column is to actually have a resonant air column. No trumpet equivalent of a woodwind controller exists.
Comment: Question for those who know: If money is no object, is there a viable digital solution? I know there are midi "wind controllers," but not sure whether they can use realistic brass mouthpieces that allow you to buzz normally. In the string world, there are electric violins that are reasonably realistic practice solutions.
Comment: Haven't played trumpet for 20 years, but apparently there is [an electronic system that seems well reviewed](https://www.wwbw.com/Yamaha-Silent-Brass-System-for-Trumpet-J46578-J46578000000000.wwbw?source=TWWR5J1BB&cntry=us&currency=usd&source=TWWRAH4BD&gclid=CjwKCAjw-sqKBhBjEiwAVaQ9a4v0NW-7XsJ9TF_RXPwHAMUGg1EExOqYIw8Jq5vxa1oFtaZi8Zy3cRoCIpAQAvD_BwE).
Comment: @AlexanderNied: Weird. Your link leads to an Ibanez electric guitar.
Answer: A regular mute changes the colour of the sound. A silent brass mute (like the Yamaha one I've used for years) won't silence you but it substantially reduces the produced volume. The instrument does "feel" different when the mute is inserted - there's more back pressure and the intonation of some of the lower notes goes off a bit. But it still feels like a trumpet.
Using the practice mute means I can play in my car or in a very small room where the volume would otherwise be absolutely crushing, so the mute makes almost any space into a possible rehearsal space.
Provided you practise at a reasonable time of day and don't go on for hours, you should be able to play every day without reasonable complaints.
If you do use a practise mute regularly, do try to make some opportunity to play unmuted too; join a band or find a space you can play in without disturbing others. Otherwise you forget how the open instrument behaves.
Best thing to do would be to try out a practice mute for yourself. Borrow one if you can or get one on approval. Don't rely on just playing it in a shop - you really need to live with it for a bit to see how you get on with it.


memcached memory usage difference on 32- and 64-bit

Question: We have a bunch of boxes running memcached. Some are 32-bit and some are 64-bit, but all run a 32-bit install of RHEL due to various Perl-related things. We'll call them type A and B respectively. Both types run memcached 1.2.8 with the same amount of memory allocated via the -m switch. Every memcached instance contains the same data -- in this installation we're not using it as a distributed hash but a local store on each machine. A separate process is responsible for seeding each instance with fresh data at a regular interval.
Today we noticed all the 64-bit machines had started evicting items:
Type A, 32-bit
STAT curr_items 115208
STAT evictions 0
STAT limit_maxbytes 67108864

Type B, 64-bit
STAT curr_items 122192
STAT evictions 1049488
STAT limit_maxbytes 67108864

Initially I thought the type B machines were using more RAM but I was misunderstanding the rusage stats (see edit history). Any ideas why the type B machines would start evicting before the others, even with the same data seeded at the same time as the type A machines? Is it down to the underlying hardware being 64-bit? I'm displaying a shameful ignorance of computer hardware here but there's only one way to fix that.
Answer: If both A- and B- types are on 32bit OS, there should be no difference in object sizes for userland apps like memcached. Differences could be seen for kernel if PAE is enabled/disabled, but even that should not affect userland memory usage.
Could you post full output of stats, stats sizes, stats items for both types? (Two last can take long for large memory, but at 67M that's not your case).
And, sure, initial output of memcached -vv.
Comment: We ended up having to restart them to bump the RAM up. I'll keep an eye on them and compare the slab allocations if their usage patterns start to differ. Thanks!


Conditionally execute update query

Question: Am preparing an update script for an old database that may potentially be run multiple times.
If it is run multiple times some db objects like columns may have disappeared so there will be some scripts that should not run depending on certain conditions.
For example the following.  I only want to run the updates if the evtype column is still available.  However, including this in a second run comes up with invalid column evtype.
How can I avoid this issue
PRINT 'Update event type IDs'
--IF COL_LENGTH('events', 'evtype') IS NOT NULL
IF EXISTS(SELECT 1 FROM sys.columns
          WHERE Name = N'evtype'
          AND Object_ID = Object_ID(N'events'))
BEGIN
    UPDATE [events]
    SET evtypeid=1 WHERE evtype='d'
    UPDATE [events]
    SET evtypeid=2 WHERE evtype='w'
    UPDATE [events]
    SET evtypeid=3 WHERE evtype='s'
END

PRINT 'Dropping events.evtype column'
IF COL_LENGTH('events', 'evtype') IS NOT NULL
BEGIN
    ALTER TABLE events
    DROP COLUMN evtype
END

Answer: You have a poor design if you are removing columns.  But to solve your problem use dynamic SQL:
IF EXISTS (SELECT 1 FROM sys.columns
           WHERE Name = N'evtype' AND Object_ID = Object_ID(N'events')
          )
BEGIN
    DECLARE @sql NVARCHAR(MAX);
    SET @sql = '
UPDATE events
    SET evtype = (CASE WHEN evtype = ''d'' THEN 1
                       WHEN evtype = ''w'' THEN 2
                       WHEN evtype = ''s'' THEN 3
                  END)
    WHERE evtypeid IN (''d'', ''w'', ''s'')
';

    exec sp_executesql @sql
END;

This will prevent the compilation error because the column does not exist.
Comment: Single quotes need to be doubled up to escape them.
Comment: It is a very poorly designed db. We are refactoring.

Conditionally execute update query


MVC HTML Compress filter exception when using html.action in view

Question: Code taken from Codeproject.
When using Html.action in main view to render a partial view it gives me value cannot be null exception on response.Filter.
Filter
public class FilterConfig
{
    public static void RegisterGlobalFilters(GlobalFilterCollection filters)
    {
        filters.Add(new CompressAttribute());
    }
}

public class CompressAttribute : ActionFilterAttribute
{
    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        var encodingsAccepted = filterContext.HttpContext.Request.Headers["Accept-Encoding"];
        if (string.IsNullOrEmpty(encodingsAccepted)) return;

        encodingsAccepted = encodingsAccepted.ToLowerInvariant();
        var response = filterContext.HttpContext.Response;

        if (encodingsAccepted.Contains("deflate"))
        {
            response.AppendHeader("Content-encoding", "deflate");
            response.Filter = new DeflateStream(response.Filter, CompressionMode.Compress);
        }
        else if (encodingsAccepted.Contains("gzip"))
        {
            response.AppendHeader("Content-encoding", "gzip");
            response.Filter = new GZipStream(response.Filter, CompressionMode.Compress);
        }
    }
}

View

   @Html.Action("GetAccountCompanies", "Account")

Action
[HttpGet]
[ChildActionOnly]
public PartialViewResult GetAccountCompanies()
{
     return PartialView("~/Views/Shared/_AccountCompanies.cshtml");
}

Exception
response.Filter = new DeflateStream(response.Filter, CompressionMode.Compress);

Response.Filter = Value cannot be null

Filter works when I comment out
@Html.Action("GetAccountCompanies", "Account")

Answer: CompressAttribute is called for each action so we need to make sure we don't compress childactions, (response.filter == null)
public class CompressAttribute : ActionFilterAttribute
{
    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        var response = filterContext.HttpContext.Response;

        if (response.Filter != null)
        {
            var encodingsAccepted = filterContext.HttpContext.Request.Headers["Accept-Encoding"];
            if (string.IsNullOrEmpty(encodingsAccepted)) return;

            encodingsAccepted = encodingsAccepted.ToLowerInvariant();

            if (encodingsAccepted.Contains("deflate"))
            {
                response.AppendHeader("Content-encoding", "deflate");
                response.Filter = new DeflateStream(response.Filter, CompressionMode.Compress);
            }
            else if (encodingsAccepted.Contains("gzip"))
            {
                response.AppendHeader("Content-encoding", "gzip");
                response.Filter = new GZipStream(response.Filter, CompressionMode.Compress);
            }
        }
    }
}


Predicting CSAT scores using GAM

Question: I am trying to fit a GAM model to predict CSAT (customer satisfaction scores on a scale of 0 to 10) which I believe are ordinal in nature. I used a categorical predictor (type) and two numeric predictors (duration, difference) which I believe have non-linear relationship to csat scores.
  gam_model = gam(
  csat_score ~ type + s(duration) + s(difference),
  family = ocat(R = 11), # since we have 11 possible values i.e. 0 to 10
  data = train_data)

However the above code throws error, which I couldn't figure out. I checked my data and it does have responses from 0 to 10 (all 11 of them are present) . No variables have missing values or outliers.
Error in eval(family$initialize) : values out of range

Structure of the dataset is as below:
csat_score : num [1:200] 8 9 9 9 10 ...
type : Factor w/ 4 levels "A","B"..
duration : num [1:200] 3.75 3.75 4.52 ...
difference : num [1:200] -3.75 -3.75 -2.98 -5.25 ...

Can you please help me with :

Is this the right approach to predict CSAT scores using GAM ?
Why am I getting this error & how can I fix it?
Should I convert the response variable (csat_score) to categorical type?

Thank you.
Answer: From ?mgcv::ocat:

The observed categories are coded 1, 2, 3, ... up to the number of categories.

I presume you used the CSAT scores as is (i.e. including a 0), which is not what is required for this model/family.
You shouldn't convert this response to be a factor; it needs to be an integer.
Comment: Thank you @Gavin. That worked once I changed the categories to 1 to 11.

Predicting CSAT scores using GAM


Regras htaccess para buscas

Question: Estou utilizando o seguinte código htaccess:
<IfModule mod_rewrite.c>
RewriteEngine on

# Include in the next line all folders to exclude
RewriteCond %{REQUEST_URI}  !(app/view/*|web/*) [NC]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?url=$1 [PT,L]
</IfModule>

Desejo que a url de uma busca seja aceito por ele:

http://localhost/previdencia/admin/pesquisa?search=ouvidoria

Como faço funcionar?
Answer: Utilize as bandeiras [QSA, L] para utilizar poder ler requisições URI via GET e também fazer reescrita. Você pode ver mais detalhes nessa pergunta: https://stackoverflow.com/questions/16468098/what-is-l-in-qsa-l-in-htaccess
Seu código ficaria assim:
# Include in the next line all folders to exclude
RewriteCond %{REQUEST_URI}  !(app/view/*|web/*) [NC]

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ index.php?url=$1 [QSA,L]
</IfModule>

Regras htaccess para buscas


How to have a function template get the parameter types of its lambda argument?

Question: Assume we have a class template like this:
template <typename baseT, typename ...argTs>
class CCallable : public baseT
{
public:
    CCallable(std::function<bool(argTs...)> lambda)
    : m_lambda(lambda) {
    }
    bool Invoke(argTs ...args) override {
        return m_lambda(args...);
    }
private:
    std::function<bool(argTs...)> m_lambda;
};

And assume we have a function template callback implemented, probably similar to this pseudo code:
template <typename baseT, typename lambdaT>
CCallable<baseT, typename lambdaT::argTs...> callback(lambdaT lambda)
{
    return CCallable<baseT, typename lambdaT::argTs...>(lambda);
}

so that we are able to do this:
auto&& functor = callback<CBase>([](int x, int y, int *sum)->bool{
    *sum = x + y;
    return true;
});
// Start passing functor around and then...
int sum;
functor.Invoke(7, 42, &sum);

Please note that the parameter types of the lambda are not passed to callback as its template type arguments.
How can we implement a function template like this, to save the users from typing more codes like:
auto&& functor = callback<CBase, int, int, int*>([](int x, int y, int *sum)->bool{
    *sum = x + y;
    return true;
});

Thanks.
By the way, why I'm asking this is because Microsoft::WRL provides a similar template named Callback which is called many times in an opensource library I want to use. However, I prefer to build the library with GNU C++ instead of Visual C++. Therefore, it seems inevitable that I have to implement a Microsoft::WRL::Callback-like template or macro myself.
Comment: You could make `Invoke()` itself a template.
Comment: @G.Sliepen Nope, a method template isn't allowed to be virtual. See the keyword `override` there after the parameter list of `Invoke`?
Comment: Remember that the lambda might be callable with multiple types...  (Auto parameter im c++14 for example)
Comment: [The doc](https://learn.microsoft.com/en-us/cpp/cppcx/wrl/callback-function-wrl?view=msvc-160) makes it look like that it is necessary to specify the template parameter, no?
Comment: @user202729 It does need to have a template type argument, but the only mandatory type parameter is the base class, not any types in the parameter list of the lambda that is passed to `Callback`. Maybe the `Callbak` gets those types from the base class, idk.
Answer: With C++17 CTAD, you could harness deduction guides for std::function:
template<class T>
struct Identity { };

template<typename baseT, typename ...argTs>
class CCallable : public baseT {
public:
    CCallable(std::function<bool(argTs...)> lambda, Identity<baseT> = {})
        : m_lambda(lambda) {
    }

    bool Invoke(argTs ...args) override {
        return m_lambda(args...);
    }

private:
    std::function<bool(argTs...)> m_lambda;
};

template<typename baseT, typename lambdaT>
auto callback(lambdaT lambda) {
    return CCallable(std::function(lambda), Identity<baseT>{});
}

CTAD is all-or-nothing, so the type baseT is wrapped into Identity to make it deducible.
Demo
Comment: In non-C++17 land, I think this basically does `template CCallback callback_impl(F &f, R (G::*)(As...)) { return CCallable(f); } template auto callback(F f) /* -> decltype(callback_impl(f, &F::operator())) */ { return callback_impl(f, &F::operator()); }` (no need for `Identity`). I actually didn't realize the standard library sanctioned this! I always thought it was a bit of a hack,
Comment: @Cody Implicit conversion is only suppressed during template argument deduction, because we might not know the type we'd need to convert to (since we haven't deduced the template parameters yet). If we've given enough template arguments to know the parameter type implicit conversion can proceed as usual.
Comment: @HTNW Thanks. Your code works perfectly, even when the argument of the `callback` is a lambda instead of a `std::function<>` object. AFAIK, compilers don't do implicit conversion for the arguments of function templates, but in this case it looks like the the lambda is implicitly converted into a `std::function<>`. Why? What happened? I'm confused.
Comment: @HTNW Allow me to make it clearer for the people who might be looking for the solution to the same problem. If use `nm` or `objdump` to check the symbols generated by the 2 function templates mentioned above, there is no evidence that implicit conversion happened when passing arguments into the function templates. The lambda is still a lambda. The `std::function<>` object is created in the `callback_impl` after `callback_impl` received the argument types of the lambda from `callback`.


Stuck on an integral $4x\sin (x^2)\cos (x^2)$

Question: I'm trying to get my head around this integral but it just doesn't click $$4x\sin (x^2)\cos (x^2) $$
I have tried substitution but I am confusing myself!
Do I substitute $\cos (x^2)$ getting $du=- 2x\sin (x^2) $ but I'm not sure where to go from there.
Comment: The integrand equals $2x\sin(2x^2)$.
Comment: Whilst Lord Sharks answer is the most simple, your substitution $u = \cos(x^{2}) \implies du = -2x\sin(x^{2}) dx$ also works. Alternatively, you can make the substitution $u = \sin(x^{2}) \implies du = 2x \cos(x^{2}) dx$, both will give the same result.
Comment: Okay so all I need to do is substitute either of the trig functions and do the simplification.
Answer: Let
$$
  u = x^{2} \qquad \Rightarrow \qquad du = 2x dx
$$
Then the primitive becomes
$$
\int 4 x \sin \left(x^2\right) \cos \left(x^2\right) \, dx \Rightarrow
\int 2 \sin u \cos u \, du =
-\frac{1}{2} \cos (2 u) \, du  \Rightarrow
-\frac{1}{2} \cos \left(2 x^2\right)
$$
Comment: I am not clear on where the $2x $ has gone from this answer!
Comment: @Ryan $du=2xdx\implies 4x=2du$
Answer: $$\int 4x\sin (x^2)\cos (x^2) \,dx $$
$$= \int 2\sin (x^2) \cos(x^2) \, dx^2$$
$$=\int 2\sin (x^2) \,  d\sin(x^2) $$
$$= \sin^2(x^2) + C $$
Comment: @JohnLou You just gradually put items into "d", e.g. first step, put $2x\, dx = d(x^2)$. Then you deal with $\cos$, since $x^2$ lives both in $\cos(x^2)$ and $dx^2$, you could view it as a whole and get $\cos(x^2)dx^2= d\sin(x^2)$, etc. Essentially, you could view the whole process as reversed chain-rule, and it is very useful when doing integral.
Comment: @JohnLou Yep, exactly, you get it! Not disappearing, but you view it as a whole, just like you substitute $u=x^2$, and then you substitute it back.
Comment: How did you progress from each step?
Comment: Where does the $x^2$ go? It seems as if some stuff is just disappearing. Is that true?

For example, if I had $\int {2x^3}dx = \int {x^2}dx^2 = \frac{1}{2}x^4 $?
Answer: The substitution $x^2=t$ gives $$-\cos^2 x^2+C$$.
Comment: @Ryan Certainly! Let $x^2=t$. Hence, $dt=2xdx$ and $\int4x\sin{x^2}\cos{x^2}dx=\int2\sin{t}\cos{t}dt=\int d(-\cos^2t)=-\cos^2t+C=-\cos^2x^2+C$
Comment: @Ryan $d(f(x))=f'(x)dx$. Thus, $d(-\cos^2t)=(-\cos^2t)'dt=-2\cos{t}(\cos{t})'dt=2\sin{t}\cos{t}dt$
Comment: Could you go into more detail for me please?
Comment: I think i understand the first step but no so much $=\int d (-\cos^2t) $
Comment: @ Michael Rozenberg thankyou for your help I appreciate it! I think I understand but have alot of work to do before my exam in 2 weeks!   This just isnt clicking for me like alot of tbings have
Answer: $4x\sin(x^{2})\cos(x^{2})=2x\sin(2x^{2})$. Now you can take the substitution $y=x^{2}$.


Get X Y position of Y-Axis element

Question: I'm looking for a way to position my custom chart legend on the chart. However, I can't find any API to get Y-Axis element so I can't calculate the x,y position of legend that I should use (knowing that size of y-axis is dynamic, depends on price value).
Any suggestion please?
Answer: The position of the various chart elements can be determined by:

Querying the PriceScale API for it's current width (https://tradingview.github.io/lightweight-charts/docs/api/interfaces/IPriceScaleApi).
Querying the TimeScale API for it's current height and width (https://tradingview.github.io/lightweight-charts/docs/api/interfaces/ITimeScaleApi).
Knowing the current layout of the chart, for example if the price scale is on the left or right side.

Lightweight charts uses a table layout to position the various main components of the chart, thus by knowing these values it is then possible to determine all the positions relative to top left corner of the container element.
So for example if you want to know the position and dimensions of the PriceScale and TimeScale (assuming that the price scale is on the right side):
<code>// Run inside a RequestAnimationFrame
// so that the chart has a chance to
// render once before querying the dimensions.
requestAnimationFrame(() => {
    const containerElement = document.getElementById("container");
    const containerDimensions = containerElement.getBoundingClientRect();

    // chart is the IChartAPI reference returned by LightweightCharts.createChart
    const timeScale = chart.timeScale();
    const timeScaleWidth = timeScale.width();
    const timeScaleHeight = timeScale.height();

    // mainSeries is the ISeriesAPI reference returned when adding data
    // for example with: chart.addLineSeries(...)
    const priceScale = mainSeries.priceScale();
    const priceScaleWidth = priceScale.width();

    const priceScalePositionAndSize = {
        x: timeScaleWidth,
        y: 0,
        height: containerDimensions.height - timeScaleHeight,
        width: priceScaleWidth,
    };

    const timeScalePositionAndSize = {
        x: 0,
        y: containerDimensions.height - timeScaleHeight,
        height: timeScaleHeight,
        width: timeScaleWidth,
    };

    console.log({ timeScalePositionAndSize, priceScalePositionAndSize });
});
</code>

Get X Y position of Y-Axis element


remove li element from list when clicking on an icon inside that li jquery

Question: I'm trying to remove a
 element from an
, using a click event on an icon which is placed inside that same
 element. The
 elements are created dinamically.
The html looks like

Enlaces añadidos

Enlace Randomdelete_forever

so when clicking on that  that has the delete_icon class, its
 parent should be removed.
this is the .on() event
$("#lista_enlaces_ul").on("click", "i.delete_icon", function (e) {

});

i've tried several ways to try to find the index of the li element inside ul to remove it, but i'm going nowhere...
Answer: You can use $(this).closest('li').remove(); or $(this).parents('li').remove();

$("#lista_enlaces_ul").on("click", "i.delete_icon", function(e) {
    $(this).closest('li').remove();
});

Enlaces añadidos

Enlace Randomdelete_forever

Answer: $("#lista_enlaces_ul").on("click", "i.delete_icon", function (e) {
$(e.target).parents('li').remove();
});

Comment: Hi there! thank you for your answer but can you provide a brief explanation behind your solution?
Comment: Yes, so from the event it will get target element and and from that it will select the parent element that is 'li', and just simply remove that.


Create a dictionary from a two dimensional list in Python

Question: i'm facing some troubles to create a dictionary from a list and a list of lists. Here's an example:
a = ['a','b','c','d']
b = [[1,2,0], [3,4,1], [5,6,0], [7,8,1]]

I want a output using the elements of list a and the third element of list b, like this output:
a = {'a': 0, 'b': 1, 'c': 0, 'd': 1]

Both lists have the same length, so i tried something like:
c = dict(zip(a,b))

But in this case, i can't select the third element of b.
In my real case, i have millions of data, so i need a fast and simple way to do it, and create a temporary list from b may not be a optimal solution.
Thanks
Answer: You could do:
dict(zip(a, (x[-1] for x in b)))  # Maybe x[2] if some of the sublists have more than 3 elements.

If you are on python2.x, you might want to use itertools.izip or future_builtins.zip
Answer: Or, using a dictionary in comprehension:
a = ['a','b','c','d']
b = [[1,2,0], [3,4,1], [5,6,0], [7,8,1]]

c = {k: v[-1] for k, v in zip(a, b)}

See PEP274 Dict Comprehensions
Answer: I've tried this one and it seemed to have worked:
>>> a = ['a','b','c','d']
>>> b = [[1,2,0], [3,4,1], [5,6,0], [7,8,1]]
>>> c = dict(zip(a, [d[2] for d in b]))
>>> c
{'a': 0, 'c': 0, 'b': 1, 'd': 1}

Answer: You can make use of itemgetter inside the operator module. Where 2 represents the index within the b sublists.
from operator import itemgetter
a = ['a','b','c','d']
b = [[1,2,0], [3,4,1], [5,6,0], [7,8,1]]

c = dict(zip(a, map(itemgetter(2), b)))

and since you have lots of elements, maybe itertools will help with memory:
from operator import itemgetter
from itertools import izip, imap

a = ['a','b','c','d']
b = [[1,2,0], [3,4,1], [5,6,0], [7,8,1]]

c = dict(izip(a, imap(itemgetter(2), b)))

These solutions will help take advantage of the underlying C modules for performance benefits.


Arquivo txt abre no navegador ao invés de ser efetuado o download

Question: Uso o seguinte código para gerar um arquivo txt e enviá-lo à minha View:
 public ActionResult geraBpa(){
    var caminho = System.Web.HttpContext.Current.Server.MapPath("~/Content");
    using(StreamWriter file = new StreamWriter($"{caminho}/BPA.txt"))
    {
        List<bpac> listaBpac = pegaBpac();

        int linhaTexto = 1;
        int linhaItem = 1;

        foreach (bpac linha in listaBpac)
        {
            file.WriteLine(
            "02" +
            linha.cnes +
            linha.cmp + //competencia
            linha.cbo +
            string.Format("{0:000}", linhaTexto) + string.Format("{0:00}", linhaItem) +
            linha.pa +
            "000" +
            string.Format("{0:000000}", linha.quant) +
            "EXT"
            );

            linhaItem++;
            if (linhaItem > 99)
            {
                linhaItem = 1;
                linhaTexto++;
            }
        }
    }

    linhaTexto++;
    linhaItem = 1;

    byte[] fileBytes = System.IO.File.ReadAllBytes($"{caminho}/BPA.txt");
    string fileName = "myfile.ext";
    return File(fileBytes, MediaTypeNames.Application.Octet, fileName);
}

O mesmo está funcionando corretamente, porém, ao invés de ser efetuado o download, ele abre na tela do navegador.
Alguma sugestão?
Comment: Italo posta o cabeçalho da sua Action por gentileza.
Comment: seria isso? `public ActionResult geraBpa()`
Answer: Você pode adicionar um header pra forçar este download
// [...] Resto do código

string fileName = "myfile.ext";
var disposition = new System.Net.Mime.ContentDisposition
{
    FileName = fileName,
    Inline = false, // <- Isso força o download
};

Response.AppendHeader("Content-Disposition", disposition.ToString());

return File(fileBytes, MediaTypeNames.Application.Octet, fileName);

Comment: alterei, mas não funcionou
Comment: @ItaloRodrigo Deve ser por causa do mime type usado, tenta usar um de texto.
Answer: Creio que assim resolveria seu problema:
try
{
    Response.Clear();
    Response.ClearHeaders();
    Response.ClearContent();
    Response.AddHeader("content-disposition", "attachment; filename=" + _Filename);
    Response.AddHeader("Content-Type", "application/Word");
    Response.ContentType = "application/octet-stream";
    Response.AddHeader("Content-Length", _FileLength_in_bytes);
    Response.BinaryWrite(_Filedata_bytes);
    Response.End();
}
catch (ThreadAbortException)
{ }
finally
{
}

Comment: onde eu incluo esse código?
Comment: @ItaloRodrigo você precisa adaptar ao seu código claro kkkk, mas mesmo assim coloquei com um _na frente de cada local aonde você ira colocar seu código


Filtrar um array de objetos dentro de outro array de objetos

Question: Eu tenho um array como o abaixo e gostaria de filtrar ele pela propriedade que esta dentro de um objeto dentro de um array que ta dentro de outro objeto.
A ideia seria retornar todos os objetos cujas movimentações sejam tipo: "saida".
Em outra pergunta no fórum encontrei este codigo e o adaptei para minha circunstância :) A solução deles foi:
var filtrado = array.filter(function(obj) { return obj.marcar == 1; });

Como eu filraria por movimentações que sejam tipo: "saida"
    var array =
    [
        {
            conta_id : "7",
            movimentações: { InternaMov : [
               {data: "20/10/1029",
                tipo: "saida",
                valor: "300,00"},
               {data: "19/10/1029",
                tipo: "entrada",
                valor: "900,00"}
            ]},
            marcar : 1,
            pag_data_emissao : "04/08/2015",
            pag_debito_credito : "D",
            pag_historico : "CHEQUE 331107  VENDA S",
            pag_id : "47782",
            pag_utilizado :"VENDA S",
            pag_valor : "7.000,00"
        },
        {
            conta_id : "7",
            movimentações: { InternaMov : [
               {data: "20/10/1029",
               tipo: "saida",
               valor: "300,00"},
               {data: "19/10/1029",
               tipo: "entrada",
               valor: "900,00"}
            ]},
            marcar : 0,
            pag_data_emissao : "07/08/2015",
            pag_debito_credito : "D",
            pag_historico : "DEPOSITO 3117  VENDA X",
            pag_id : "47783",
            pag_utilizado :"VENDA X",
            pag_valor : "640,63"
        }
    ];

Comment: Mas você quer retornar só o objeto mais interno ou quer retornar as informações do externo e o objeto interno filtrado?
Answer: Existem duas soluções para resolver o seu problema. A primeira é caso você só queira as movimentações, que daí você faria da seguinte forma:
array.map(obj1 => (obj1.movimentacoes.InternaMov.filter(obj2 => obj2.tipo === 'saida')));

var array =
    [
        {
            conta_id : "7",
            movimentacoes: { InternaMov : [
               {data: "20/10/1029",
                tipo: "saida",
                valor: "300,00"},
               {data: "19/10/1029",
                tipo: "entrada",
                valor: "900,00"}
            ]},
            marcar : 1,
            pag_data_emissao : "04/08/2015",
            pag_debito_credito : "D",
            pag_historico : "CHEQUE 331107  VENDA S",
            pag_id : "47782",
            pag_utilizado :"VENDA S",
            pag_valor : "7.000,00"
        },
        {
            conta_id : "7",
            movimentacoes: { InternaMov : [
               {data: "20/10/1029",
               tipo: "saida",
               valor: "300,00"},
               {data: "19/10/1029",
               tipo: "entrada",
               valor: "900,00"}
            ]},
            marcar : 0,
            pag_data_emissao : "07/08/2015",
            pag_debito_credito : "D",
            pag_historico : "DEPOSITO 3117  VENDA X",
            pag_id : "47783",
            pag_utilizado :"VENDA X",
            pag_valor : "640,63"
        }
    ];

console.log(array.map(obj1 => (obj1.movimentacoes.InternaMov.filter(obj2 => obj2.tipo === 'saida'))));

Caso você queira manter as informações do objeto externo, daí você deve fazer a seguinte alteração no map:
array.map(obj1 => ({...obj1, movimentacoes: obj1.movimentacoes.InternaMov.filter(obj2 => obj2.tipo === 'saida')}))

var array =
    [
        {
            conta_id : "7",
            movimentacoes: { InternaMov : [
               {data: "20/10/1029",
                tipo: "saida",
                valor: "300,00"},
               {data: "19/10/1029",
                tipo: "entrada",
                valor: "900,00"}
            ]},
            marcar : 1,
            pag_data_emissao : "04/08/2015",
            pag_debito_credito : "D",
            pag_historico : "CHEQUE 331107  VENDA S",
            pag_id : "47782",
            pag_utilizado :"VENDA S",
            pag_valor : "7.000,00"
        },
        {
            conta_id : "7",
            movimentacoes: { InternaMov : [
               {data: "20/10/1029",
               tipo: "saida",
               valor: "300,00"},
               {data: "19/10/1029",
               tipo: "entrada",
               valor: "900,00"}
            ]},
            marcar : 0,
            pag_data_emissao : "07/08/2015",
            pag_debito_credito : "D",
            pag_historico : "DEPOSITO 3117  VENDA X",
            pag_id : "47783",
            pag_utilizado :"VENDA X",
            pag_valor : "640,63"
        }
    ];

console.log(array.map(obj1 => ({...obj1, movimentacoes: obj1.movimentacoes.InternaMov.filter(obj2 => obj2.tipo === 'saida')})));

Perceba que no primeiro exemplo apenas filtramos os arrays internas, enquanto no segundo exemplo, copiam-se todas as propriedades do objeto dentro de array e filtra-se apenas os arrays internos.
Comment: Obrigado Felipe Avelar... Valeu mesmo.
Comment: @AlessandroCunhalorenz se essa solução resolveu seu problema, poderia marcar como resposta, por favor?


É errado usar o atributo position pra posicionar elementos, em vez do float?

Question: É errado usar o atributo position pra posicionar elementos na página, em vez do float ? Se sim, porque ?
Comment: Errado pra que? Se fosse errado não existira.
Comment: O float inicialmente foi projetado para alocar textos ao redor de imagens dando assim um efeito flutuante, porém ao longo do tempo foi se adaptando para o layout relativo, pode-se dizer que é um bom método, mas os outros também tem a sua importância e podem ser usados, inclusive recomendo o uso das flex-box para posicionar os elementos na página, hoje é o método mais completo, e não se usa float.
Comment: Mata a responsividade ... o efeito de Xpixel em um monitor full HD é bem diferente do efeito em um smartphone.
Comment: Li na internet que o position deve ser usado apenas pra pequenas alterações na pagina, e pra definir os elementos dentro do layout era pra usar float
Answer: Se todo o conteúdo da sua página for estático, não vejo problema em usar position para montar a estrutura. Se não for estático dificilmente você vai conseguir estruturar corretamente sem float.


Relatively Prime Relationship Equation Proof

Question: I have this math question that I am stuck on. This is the question:

Suppose that $a$ and $b$ are positive integers, with $d = \gcd(a, b)$.
   Suppose that there exists integers $r$ and $s$ so that $ar + bs= d$.
   Show that $\gcd(r, s) = 1$ using the relatively prime equation.

This is the relatively prime relationship equation: Let $a,b$ be non-zero integers. $a$ and $b$ are relatively prime iff there exists integers $x$ and $y$ such that $ax+by=1$
I know that the relatively prime equation that I have to solve is $ar + bs = 1$. However, I'm not sure how to start this. Thanks.
Comment: What is this "relatively prime equation"? As stated, you can't use [Bezout's identity](https://en.wikipedia.org/wiki/B%C3%A9zout%27s_identity) because that's what you are trying to prove.
Comment: @A.P. Let $a,b$ be non-zero integers. $a$ and $b$ are relatively prime iff there exists integers $x$ and $y$ such that $ax+by=1$
Comment: That's a special case of what you are trying to prove.
Comment: So, how do I show it using that equation? I don't start off with the equation?
Answer: Dividing $a$ and $b$ by $d$, we have two relatively prime numbers.  By the relatively prime numbers equation, it exist $r$ and $s$ such that
$r\frac{a}{d} + s\frac{b}{d} = 1.$
Those numbers are relatively prime.  If not, the left part could be factorise giving an integer factorisation of 1...
Multiplying everything by $d$ again to obtain what you need
Comment: How is that using the Relatively Prime Relationship Equation?
Comment: Since $\frac{a}{d}$ and $\frac{b}{d}$, there is $r$ and $s$ such that $r\frac{a}{d} = s\frac{b}{d}=1$.  $r$ and $s$ are relatively prime.
Comment: +1: The idea's there, but you should try to flesh-out your answers a bit more. Right now this is more a comment than an answer...
Comment: My goal was to give you a hint, I will flesh out my solution.
Answer: We know that $a = d\bar{a}$ and $b = d\bar{b}$, so dividing
$$
ar + bs = d
$$
by $d$ we get
$$
\bar{a}r + \bar{b}s = 1
$$
which is what you call the "relatively prime equation", hence $r$ and $s$ must be coprime.
To conclude without assuming this, just suppose that there is an $f > 1$ that divides both $r$ and $s$. Then the above equation implies $f \mid 1$, which is absurd.


Do Google SEO Content Keywords matter and should I remove sidebar content?-

Question: I run a Magento based store website.
At the side of every product page we have delivery information.
Because of this, Google Webmaster tools picks up words such as 'delivery' 'orders' 'returns' as significant keywords - rather than more relevant 'industry specific' keywords.
Does it matter that he gives 'delivery' a higher significant rating?
Should I remove the delivery info from the side of each page?
Or is there a way to disavow keywords to tell Google that 'delivery' isn't relevant?
Or maybe turn the text info at the side into a graphic instead?
Many thanks!
Comment: I'm voting to close this question as off-topic because it is asking for SEO advice. It might be on-topic on [webmasters.se].
Comment: I'm voting to close this question as off-topic because it's not about programming as defined in the [help].
Answer: Before SEO, you should always consider what is best for your user. If displaying shipping information in the sidebar is going to enhance the user's experience, leave it. If the information could be put on a page and a link can be added the sidebar, do that.
Having said that, I wouldn't worry about it. Unless you're trying to rank for the keywords 'return' or 'delivery', you're not likely to notice any sort of algorithm penalty that comes from having the words appear all over the website.
Furthermore, a keyword stuffing penalty is applied to each page individually. You should be careful with stuffing your keywords in tags on the side, as it will increase the keyword density.


An open database for propensity score matching

Question: I need an open biological or medical database for benchmarking methods for propensity score matching. Where can I find some?
Answer: There are several ones I guess, but the best I can recommend is the US National Inpatient Sample (NIS) or similar ones from the US Healthcare Cost and Utilization Project (HCUP).
See the web portal for additional references: https://www.hcup-us.ahrq.gov/databases.jsp.
These databases have already been analyzed and the corresponding peer reviewed publications are available to guide you. For instance, have a look here https://www.ncbi.nlm.nih.gov/pubmed/29575160 or here https://www.ncbi.nlm.nih.gov/pubmed/29608938.


Logout when java app is inactive for 30 min problem

Question: I need to implement logout when  java desktop app is ( when there is no clicking buttons for 30 min ) inactive for 30 min. How to implent this ? Does anybody have any idea ? Is there any observer already for this in Java ?
Comment: A multiuser desktop app?
Comment: Do you want to close the desktop app, end some server connection the app might have or log the user out of the OS?
Answer: There's actually two parts to this:

detect inactivity
log out the user

For the first part, this mostly depends on how you define "inactive". One option: Have some time field for "last acitivy", which gets updated by whatever code implements an acitivy. You could use a java.util.Timer that fires once a minute and checks if "last activity" is more than 30 minutes ago, then logs the user out.
Another option is to use the approach from this question to implement a resettable timer, which you could then reset whenever there is activity. Then the logout would be after exactly 30 min, not 30-31 minutes.
As to "logging out the user", this will depend on your OS. Java itself does not have an API for this, but on most platforms you can just invoke an OS binary using Runtime.exec or similar. Also note that logging out the user may require elevated privileges on some platforms, so take this into account.
Finally, maybe you could explain what the underlying problem is? Why can't you for example just set the OS screensaver to lock the screen after 30 minutes? Wouldn't that be easier?