Do components of PCA really represent percentage of variance? Can they sum to more than 100%?

Question: O'Reilly's "Machine Learning For Hackers" says that each principal component represents a percentage of the variance. I've quoted the relevant part of the page below (chapter 8, p.207). Speaking to another expert, they agreed it is the percentage.
However the 24 components sum to 133.2095%. How can that be?

Having convinced ourselves that we can use PCA, how do we do that in R? Again, this
  is a place where R shines: the entirety of PCA can be done in one line of code. We use
  the princomp function to run PCA:
pca <- princomp(date.stock.matrix[,2:ncol(date.stock.matrix)])

If we just type pca into R, we'll see a quick summary of the principal components:
Call:
princomp(x = date.stock.matrix[, 2:ncol(date.stock.matrix)])
Standard deviations:
Comp.1 Comp.2 Comp.3 Comp.4 Comp.5 Comp.6 Comp.7
29.1001249 20.4403404 12.6726924 11.4636450 8.4963820 8.1969345 5.5438308
Comp.8 Comp.9 Comp.10 Comp.11 Comp.12 Comp.13 Comp.14
5.1300931 4.7786752 4.2575099 3.3050931 2.6197715 2.4986181 2.1746125
Comp.15 Comp.16 Comp.17 Comp.18 Comp.19 Comp.20 Comp.21
1.9469475 1.8706240 1.6984043 1.6344116 1.2327471 1.1280913 0.9877634
Comp.22 Comp.23 Comp.24
0.8583681 0.7390626 0.4347983
24 variables and 2366 observations.

In this summary, the standard deviations tell us how much of the variance in the data
  set is accounted for by the different principal components. The first component, called
  Comp.1, accounts for 29% of the variance, while the next component accounts for 20%.
  By the end, the last component, Comp.24, accounts for less than 1% of the variance.
  This suggests that we can learn a lot about our data by just looking at the first principal
  component.

[Code and data can be found on github.]
Comment: I think the author's interpretation of the `Standard deviations` is slightly off. Since the standard deviations are in fact standard deviations, we must square them to see how much of the variance each component represents. The first component would represent $100\times\frac{29.1001249^2}{29.1001249^2+\cdots+0.4347983^2}$ percent of the total variance.
Comment: Yes, this is unquestionably a bug in the book. There are a few places where I misused standard deviations instead of variances. (For example, there's a point in which we use RMSE instead of MSE to calculate R-squared.) I'm hoping we have time to sit down and correct these sorts of bugs in the near future.
Comment: This question arises from two basic mistakes, I'm afraid: (1) it misses the heading announcing that the numbers are "Standard deviations" and mistakes them for variances and (2) it assumes those numbers are percents, but they are not.  (Their units are whatever the stocks are measured in: dollars or percent change per year or whatever.)  There's no bug here at all: the comment by @Max explains how to find percent of total variance.
Comment: @whuber Perhaps I should have used "typo" instead of "bug"? :-) "_Comp.1, accounts for 29% of the variance_" is wrong and should read "_Comp.1, accounts for 46% of the variance_"
Comment: Thank you, Darren: I misunderstood that the confusion was present in the book and I took "bug" to refer to the `R` software itself. Finding that error was a good catch (I hope you found it rewarding to find out what's really going on with PCA)!
Comment: A bit OT, but I think it's clever to recheck the correctness of this book's explanations once in a while...
Answer: They should sum to $100~\%.$
The total variance of a $p$-variate random variable $X$ with covariance matrix $\Sigma$ is defined as $${\rm tr}(\Sigma)=\sigma_{11}+\sigma_{22}+\cdots+\sigma_{pp}.$$
Now, the trace of a symmetric matrix is the sum of its eigenvalues $\lambda_1\geq\lambda_2\geq\ldots\geq\lambda_p.$ Thus the total variance is
$${\rm tr}(\Sigma)=\lambda_1+\cdots+\lambda_p$$
if we use $\lambda_i$ to denote the eigenvalues of $\Sigma$. Note that $\lambda_p\geq 0$ since covariance matrices are positive-semidefinite, so that the total variance is non-negative.
But the principal components are given by $e_iX$, where $e_i$ is the $i$:th eigenvector (standardized to have length $1$), corresponding to the eigenvalue $\lambda_i$. Its variance is
$${\rm Var}(e_iX)=e_i'\Sigma e_i=\lambda_ie_i'e_i=\lambda_i$$
and therefore the first $k$ principal components make up
$$\Big(\frac{\lambda_1+\cdots+\lambda_k}{\lambda_1+\cdots+\lambda_p}\cdot 100\Big)~\%$$
of the total variance. In particular, they make up $100~\%$ of the total variance when $k=p$.
Comment: Did you see the (more recent) comment by @Max to the question?  He nailed the answer.
Comment: @whuber: I hadn't seen it, so thanks. I made a similar remark in a comment to Joshua's answer.
Answer: Use summary.princomp to see the "Proportion of Variance" and "Cumulative Proportion".
pca <- princomp(date.stock.matrix[,2:ncol(date.stock.matrix)])
summary(pca)

Comment: @DarrenCook: `sdev` implies that you're looking at the _standard deviation_, i.e. the square root of the variance (or $\sqrt{\lambda_i}$, using the notation from my answer), which should explain the difference. Try `pca$sdev[1]^2/sum(pca$sdev^2)` instead.
Comment: @DarrenCook: use the source... `stats:::print.summary.princomp` shows you that it squares the `sdev` component, which `stats:::princomp.default` shows is the `sqrt` of the eigen values.
Comment: Thanks Joshua. So the first component is actually 46% of the variance. I'll send in a bug report to the book.
Comment: How is "Proportion of variance" calculated? The number shown is 0.4600083. But `sqrt(pca$sdev[1]/sum(pca$sdev))` (roughly `sqrt(29.1/133.2)`) gives 0.4673904.
Answer: Here is some R code to complement previous answers (pca[["sdev"]] is usually written pca$sdev, but it causes misformatting in the snippet below).
# Generate a dummy dataset.
set.seed(123)
x <- matrix(rnorm(400, sd=3), ncol=4)
# Note that princomp performs an unscaled PCA.
pca1 <- princomp(x)
# Show the fraction variance of each PC.
pca1[["sdev"]]^2
cumsum(pca1[["sdev"]]^2)/sum(pca1[["sdev"]]^2)
# Perform a scaled PCA.
pca2 <- princomp(x, cor=TRUE)
pca2[["sdev"]]^2
cumsum(pca2[["sdev"]]^2)/sum(pca2[["sdev"]]^2)

So, as @Max points out, working with the variance instead of the standard deviation and not forgetting to divide by the total variance solves the issue.


Modular arithmetic and using in well-ordering principle

Question: I need to prove the following, but I do not know how to go about it.
If $$   (*)\:\:\: x^{3} - y^{3}= 3^{n} $$
Then $$ x \equiv 0 (mod 3)  \:\: and \:\:\: y \equiv 0 (mod 3)$$
In addition, to prove that the equation above (*), there is no solution in $ N^{+} $ (in other words $ \quad 0\neq x,y\in \mathbb N $ ).
By using the Well-ordering principle, i.e. Any non-empty subset of N, the set of natural numbers has a minimum element.
Edit
I can use these facts: $$ \forall x\in \mathbb N^{+}  \:\:\: x^{3}  \equiv x (mod 3)$$
And also:
$$ \forall x,y\in \mathbb N^{+} , \:\:\: if \:\:\: x^{3} - y^{3}\equiv 0 (mod 3) \:\:\: \\Then \:\:\: x \equiv y (mod 3)  $$
Thanks
Answer: Let $x$ and $y$ be positive integers such that $x^3-y^3=3^n$, where $n\ge 3$. We first show  that $3$ divides $x$ and $3$ divides $y$. Then later we use this to show that in fact there are no positive solutions.
In the OP, it is observed that $x\equiv y\pmod{3}$, so $x-y$ is divisible by $3$.  Now we use the fact that $x^3-y^3=(x-y)(x^2+xy+y^2)$.
Note that if $x$ and $y$ are positive and not both equal to $1$, it follows that $x^2+xy+y^2\gt 3$, and therefore $x^2+xy+y^2=3^k$ for some $k\ge 2$. Now from the fact that $9$ divides $(x-y)^2$ and the identity
$$(x^2+xy+y^2)-(x-y)^2=3xy$$
we conclude that $9$ divides $3xy$.  Thus $3$ divides $xy$. It follows that $3$ divides one of $x$ or $y$, and therefore since $x\equiv y\pmod{3}$, it divides both.

Now that we have the divisibility result, the rest follows by the least number principle, or equivalently by infinite descent.
It is easy to verify that there are no positive solutions with $n=0$, $n=1$, or $n=2$. Suppose that for some $n\ge 3$ there are positive solutions. Then there is a smallest $n\ge 3$ with this property. Let $x^3-y^3=3^n$. From our divisibility result, we have $x=3s$, $y=3t$ for some positive $s$ and $t$. But then $3^3s^3-3^3t^3=3^n$ and therefore $s^3-t^3=3^{n-3}$, contradicting the minimality of $n$.
Remark: It is not quite true that if $x$ and $y$ are integers such that $x^3-y^3$ is a power of $3$ then $3$ divides $x$ and $y$. There is the obvious example $x=1$, $y=0$. But there is also the more interesting example $x=2$, $y=-1$, for which we have $x^3-y^3=9$.
Comment: Thank you very much I appreciate it.
Comment: You are welcome. This sort of descent proof comes up medium-often. The first example one usually meets is the irrationality of $\sqrt{2}$.


Prove $f$ is measurable iff $f_0$ is measurable.

Question: $f:\mathbb{R}^d\to\mathbb{R}$ called radial function if exists $f_0:\mathbb{R}_+\to\mathbb{R}$ such that for every $x\in\mathbb{R}^d$ $f(x)=f_0(||x||).$
Prove $f$ is measurable iff $f_0$ is measurable.
I've tried one way but I'm not sure about this.
Assuming that $f_0$ is measurable and defining the following sets
$f^{-1}(A)=\{x\in\mathbb{R}^d:||X||\in f_0^{-1}(A)\}=R_{f_0^{-1}(A)}$ and setting $B=f_0^{-1}(A)$
Assume that $B\subseteq\mathbb{R}_+$ is measurable and $g(x)=||x||$,$g$ is continuous function then for every open set $V(V\in \mathcal{B})$$\to$ $g^{-1}(V)\in \mathcal{B}$ is open, therefore $g$ is lebesgue measurable and then we get $R_B=\{x\in\mathbb{R}^d:||x||\in B\}=g^{-1}(B)$.
I have some doubts because $B$ is lebesgue and not borel and then $g^{-1}(B)$ is not borel measurable.
Answer: The composition of measurable functions is always measurable. Since you are looking at measurability with respect to the Borel $\sigma$-algebra, the continuity of the norm implies its measurability. Hence, if $f_0$ is measurable, then $f$ is measurable as composition of measurable functions.
This should answer the implication you try to show. Since these more abstract results are sometimes easier to show, I recommend proving them and applying them to this special case.
Comment: Thanks for the answer, and maybe can you give a hint for the second way in the question?


Forms API: 'file' or 'managed_file' type with multiple/unlimited values

Question: I've got a form that needs to have either a 'file' or 'managed_file' type field in it for uploading files via the form. Is there an attribute that can be set on the form item render array that will allow multiple/unlimited files to be uploaded via the one form item?
For example, when you add a new field via the admin UI to a content type, select the 'file' type widget and then in the field settings set the number of values to 'unlimited' - that's the sort of behavior I'm looking for in this file field via the forms API. Thanks in advance for any pointers!
Answer: If you are trying to build this functionality for a custom form using the Forms API, it may not yet be possible.  It's documented on Drupal.org to be an issue. See this discussion, the bottom talks about the problems with implementing AHAH with a file field.
Answer: I use the Plupload form element. Example of implementation can be found at http://drupal.org/node/1647890


How to derive this formula: $\int_a^bf(c-x)dx = \int_{c-b}^{c-a}f(x)dx$?

Question: I'm stuck in this exercise:
$$\int_a^bf(c-x)dx = \int_{c-b}^{c-a}f(x)dx$$
My attempt is this:
$$$$
\begin{align*}
\int_a^bf(c-x)dx
&= - \int_{-a}^{-b}f(x-c)dx\\
&= \int_{-b}^{-a}f(x-c)dx
\end{align*}
$$$$
But at this point I'm not sure what to do. To my understanding if one wants to integrate $f(x-c)$, which is shifted to the right, it would be the same as integrating $f(x)$ with the interval of integration shifted to the left:
$$= \int_{-b-c}^{-a-c}f(x)dx$$
But that does not seem to be the right answer.
Answer: Make the Substitution $y=c-x$. The upper bound of your integral $x=b$ is changing to $y=c-b$. Moreover the lower bound $x=a$ transforms to $y=c-b$. Also the differential $dx$ is changed; you derive $y$ by $x$ for constant $c$ and you obtain $dy=-dx$.
Answer: Let's say what you need to do, how it follows naturally and then why the claims are true. First, as the second equation contains only $x$ in the $f(x)$, and a definite integral doesn't really depend on the variable with respect to which you are integrating (the variable is really a label for the integral), you may think of an almost natural substitution of $y = c-x$. Let me just elaborate what happens with this; the crucial point is that nothing has changed with the integral, as in our old integral, $x$ traverses values from $a$ to $b$, the new substituted variable $y = c-x$ traverses values from $c-b$ to $c-a$, however, the direction of traversal of values has changed due to the negative sign in front of the $x$ in the expression $f(c-x)$. Hence, the integral on the left hand side is really the same as on the right hand side, written in a different way. It has not been simplified or operated, or anything with the series of operations of substitution. It's better to have such a picture in your mind, at least in real valued functions such as these. Although the same picture is of much help in the multivariate integration as well.
Comment: Thank you. Although making a substitution seemed the (obvious?) way to derive the formula, I didn't understand what was happening behind it. Your comment clears things up a lot.


Animation glitches while rendering on SurfaceView

Question: The FPS is around 60, but the animation still glitches sometimes. I did threads profiling and discovered that on Samsung S4 or Samsung S3 lockCanvas() could take up to 20ms sometimes.
Is there any specific reason for it? How to avoid such glitches.
Comment: Are you regulating the frame-rate yourself by sleeping the thread? I found I only got 55 fps consistently with surfaceview. Otherwise you are at the mercy of your cpu's speed and hence the glitching if it decides it is doing too much work and slows down some frames with the lock time or decides to process some other frames faster. And if you lower the framerate to 30 fps, the lockcanvas time goes down to 0-1ms and no more glitching. You could try Textureview (api 14), as the lock time is only 0-1ms running full throttle (I had to regulate the frame rate cause my animation was running to fast).
Answer: Some devices, particularly those based around qcom CPUs, will aggressively manage power by lowering clocks, so you will see occasionally glitches in 60fps animation when your finger isn't actively moving across the touchscreen.
The best way to handle this is to drop frames when necessary.  The "Record GL app" activity in Grafika uses a simple trick with Choreographer to do this.  For a detailed explanation, see this appendix in the graphics architecture doc.
I should also point out that Canvas rendering on a SurfaceView surface isn't hardware-accelerated, and as pixel counts increase the rendering time will increase as well, making it more likely that you'll miss your window.  In some ways the added CPU load can actually make things smoother -- if the power management believes the device is heavily loaded it won't lower the clocks -- but it does drain the battery faster.


DiD with time invariant and time variant treatment?

Question: I am currently working on a project that seeks to study the impact of the subprime crisis on family firms and non-family firms. In this, I have imagined to use a DiD estimator to account for differences in how family firms, versus non-family firms were affected by the crisis.
However, in this design, both groups receive the treatment, whereas one (family firms) are also subject to a time invariant treatment. Therefore, as I am currently considering it, we essentially have four states:

Pre-crisis, non-family firm (i.e., crisis = 0, family business treatment = 0)
Pre-crisis, family (i.e., crisis = 0, family business treatment=1)
Post-crisis, non-family firm (i.e., crisis=1, family business treatment = 0)
Post-crisis, family firm (i.e., crisis=1, family business treatment = 1)

Where I essentially want to compare the crisis coefficient of 3) and 4).
What I am now asking is: Does it make sense to conduct a DiD in this setting, given that there are essentially two treatments? (Family business, and the subprime crisis). Are there any competing approaches that I am currently overlooking?
I hope that I made myself clear.
Sincerely
Johan Karlsson
Comment: Just to be clear, do both *family* firms and *non-family* firms experience a crisis (i.e., treatment/shock) in the post-period?
Comment: Followup question: Could this perhaps be resolved through use of firm fixed effects, hence removing the "family business" treatment, after which we can estimate the impact of the crisis for family, versus non-family firms?
Comment: @ThomasBilach Thank you, sorry that I had not made this clear in my first post. Yes! Both groups are struck by the crisis simultaneously.This is the main source of my grievance, since there's no clear cut treated vs. untreated. I have done some further thinking and reading, and I am currently considering whether it would be more appropriate to use RDD for each group separately, and then compare the ATE?
Comment: Possibly. What threshold or cutoff point will be determining treatment assignment? Doesn't the mortgage crisis impact all firms in a particular industry? Only you can answer these questions.
Comment: Dear Thomas, my apologies for the late reply. The threshold for determining treatment assignment is based on industry sales shocks in the 2007-2009 period, where all firms within an affected industry are denoted as "treated".
Answer:
I essentially want to compare the crisis coefficient of 3) and 4).

You can do this. However, you will only be comparing family firms with non-family firms. Both types of firms 'turn on' in the post-period.

Does it make sense to conduct a DiD in this setting, given that there are essentially two treatments?

No, and not because you have more than one treatment. Let me elaborate. The "crisis" (i.e., treatment) is firm-invariant. In other words, you do not have a group of "unexposed" firms. If I understood your post properly, the subprime shock impacts family firms and non-family firms. Note: difference-in-differences requires you to observe at least some unexposed firms over time (i.e., before and after the subprime crisis).
Your second "business" treatment is a time-invariant exposure specific to family firms—only. The problem here is the absence of any pre-treatment observations for family firms. Your family firms are, in essence, always treated. A firm fixed effect would help adjust for this, but then we're back to a mortgage crisis experienced by all firms in your sample. If the subprime crisis impacts a subset of firms at different times, then maybe you could exploit variation in treatment timing. It appears, though, treatment exposure is well-defined, so I don't think you can take this route either. In sum, difference-in-differences does not seem valid in my estimation.


Regular Expression for month and year excluding some words

Question: I'm trying to make a regular expression able to handle inputs like bellow to extract month and year while handling all these different cases and extract the 2 groups (start and end) like this:
From August 2017 - September 2018   (output: {August 2017},{September 2018})
From August to September 2018       (output: {August},{September 2018})
July 2009 - August 2019             (output: {July 2009},{August 2019})
De Aout 2019 a July 2020            (output: {Aout 2019},{July 2020})
De Juillet a Aout 2020              (output: {Juillet},{Aout 2020})
Juillet - Aout 2019                 (output: {Juillet},{Aout 2019})
Juillet a Aout 2019                 (output: {Juillet},{Aout 2019})

I found this regex here which does a good job (regex101 link):
(?P<fmonth>\w+.\d*)\s+\D+\s+(?P<smonth>\D+.\d+)

But the problem with it is that it does not handle these 2 cases where there is no year in the first part:
De Juillet a Aout 2020
From August to September 2018

I think it's missing a part to exclude/ignore specific words like "From" and "De".
Any ideas or solutions ?
Comment: If you're willing to go beyond regex, might I suggest Duckling - https://github.com/facebook/duckling? You can also try it out here - https://duckling.wit.ai/. It handles From August to September 2018 nicely.
Comment: Might require some tweaking but it's a good start: `(\w+(?: \d{4})?) (?:-|to|a) (\w+ \d{4})`. See it here: https://regex101.com/r/i8BgX9/1
Comment: @borancar  I tried Duckling .. It's a great suggestion. I've never seen it before. but unfortunately it didn't handle french very well which is important in my case
Answer: Note that \D+ is a very generic pattern, it matches August to in From August to September 2018, i.e. any 1+ non-digit symbols. Also, \w matches letters, digits and _s, it may be more appropriate to only match letters when you need to match month names, and for that, all you need is to subtract \d and _ from it ([^\W\d_]).
You may use a bit more precise regex:
(?P<fmonth>[^\W\d_]+(?:\W+\d+)?)\s+(?:to|a|-)\s+(?P<smonth>[^\W\d_]+\W+\d+)

See the regex demo
Details

(?P<fmonth>[^\W\d_]+(?:\W+\d+)?) - fmonth group: 1+ letters and an optional sequence of 1+ non-word chars followed with 1+ digits
\s+ - 1+ whitespaces
(?:to|a|-) - to, a or -
\s+ - 1+ whitespaces
(?P<smonth>[^\W\d_]+\W+\d+) - smonth group: 1+ letters, 1+ non-word chars, 1+ digits


How to add data to the tableView and reload?

Question: I'm trying to add a cell to a tableView that's in viewController by sending data via a segue from another viewController.
class FavoritesViewController: UIViewController {

    var shops = [
        "hello world",
        "hello world",
        "hello world"
    ]

    @IBOutlet weak var table: UITableView!

    override func viewDidLoad() {
        super.viewDidLoad()
        table.delegate = self
        table.dataSource = self
        table.reloadData()
    }

}

//protocol FavoritesDelegate: class {
//    func add(_ shopName: String)
//}

extension FavoritesViewController: UITableViewDelegate, UITableViewDataSource{

    func add(_ shopName: String) {
        print(shopName)
        shops.append(shopName)
    }

    func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        return shops.count
    }

    func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {

        let cell = tableView.dequeueReusableCell(withIdentifier: "cell",for: indexPath)
        cell.textLabel?.text = shops[indexPath.row]
        return cell

    }

    // define the action. In this case "delete"
    func tableView(_ tableView: UITableView, editingStyleForRowAt indexPath: IndexPath) -> UITableViewCell.EditingStyle {
        return .delete
    }

    // do the actual deleting
    func tableView(_ tableView: UITableView, commit editingStyle: UITableViewCell.EditingStyle, forRowAt indexPath: IndexPath) {

        if editingStyle == .delete {
            tableView.beginUpdates()

            shops.remove(at: indexPath.row)
            tableView.deleteRows(at: [indexPath], with: .fade)

            tableView.endUpdates()
        }

    }

}

And here's the function call in the other viewController (prepare):
 override func prepare(for segue: UIStoryboardSegue, sender: Any?) {

        favoritesDestinationVC = segue.destination as! FavoritesViewController
        favoritesDestinationVC.add(shopName!)

}

I know what's causing the error (favoritesDestinationVC creates a new instance where tableView is nil), but I don't know how to solve it. Any ideas on how I could add an entry to the tableView that way (and updating the table afterwards) without my app crashing?
Comment: Use closures as callbacks to your UI instance. Do your work in another viewcontroller,  before you return to the first one, just invoke that callback that of the first VC. The code you provide is not enough to suggest you an efficient way to do it. You better share the all code related with your issue.
Comment: I updated the code.
Answer: Make your shops var public and then in the segue prepare callback use it directly. Check a variable first whether it has a valid value other than the nil, if it does then proceed to avoid crashes. You can do it using an if statement to verify as you can see in the sample code. Or you can use optionals to avoid crashes. See the code below.
override func prepare(for segue: UIStoryboardSegue, sender: Any?) {

    if segue.destination is FavoritesViewController {
        let destinationVC = segue.destination as? FavoritesViewController // Use optionals which has ? (question mark)
        // To avoid crashes check if shopName has a valid value other than nil
        if let newShopName = shopName {
            // It is possible that the "shopName" has a nil value when the program reaches here.
            // That's why we will use the verified "newShopName" instead of "shopName" itself to avoid any crash.
            destinationVC?.shops.append(newShopName) // What I recommend, if not convenient for you just delete this line,
            // destinationVC?.add(newShopName) // after deleting the recommended line, uncomment this line for your convenience.
        }
    }

}


SQL Azure transient failure

Question: I am using entity framework to connect to SQL Azure and data pushed from azure functions.
I noticed that at a particular time interval of 10 mins today there were errors like following thrown from function
An exception has been raised that is likely due to a transient failure. If you are connecting to a SQL Azure database consider using SqlAzureExecutionStrategy
When I looked at sql database statistics, it had reached 99% during that time and then it was good after that.
How can I find out how many transactions were executed during that timeframe using azure portal?
It can probably give me an idea about what caused this load on the server.
Answer: In this case what you are seeing may be throttling. When throttling occurs connections get affected but also bad programming may produce unwanted number of connections and every tier has limits of number connections. Below queries will help you monitor successful/terminated/throttled connections.
select *
from sys.database_connection_stats_ex
where start_time >= CAST(FLOOR(CAST(getdate() AS float)) AS DATETIME)
order by start_time desc

select *
from sys.event_log
where event_type <> 'connection_successful' and
start_time >= CAST(FLOOR(CAST(getdate() AS float)) AS DATETIME)
order by start_time desc

To monitor when your databases reach the DTU limit you can use below query:
SELECT
(COUNT(end_time) - SUM(CASE WHEN avg_cpu_percent > 80 THEN 1 ELSE 0 END) * 1.0) / COUNT(end_time) AS 'CPU Fit Percent',
(COUNT(end_time) - SUM(CASE WHEN avg_log_write_percent > 80 THEN 1 ELSE 0 END) * 1.0) / COUNT(end_time) AS 'Log Write Fit Percent',
(COUNT(end_time) - SUM(CASE WHEN avg_data_io_percent > 80 THEN 1 ELSE 0 END) * 1.0) / COUNT(end_time) AS 'Physical Data Read Fit Percent'
FROM sys.dm_db_resource_stats

When query above shows service level objective (SLO) of 99.9% <= go to next tier.
To identify queries creating high IO, high CPU usage, high resource usage you can use Query Store. There you can find queries creating the high DTU usage.
 -- Top 10 long running queries
SELECT TOP 10 q.query_id, p.plan_id,
rs.count_executions,
qsqt.query_sql_text,
CONVERT(NUMERIC(10,2), (rs.avg_cpu_time/1000)) as 'avg_cpu_time_seconds',
CONVERT(NUMERIC(10,2),(rs.avg_duration/1000)) as 'avg_duration_seconds',
CONVERT(NUMERIC(10,2),rs.avg_logical_io_reads ) as 'avg_logical_io_reads',
CONVERT(NUMERIC(10,2),rs.avg_logical_io_writes ) as 'avg_logical_io_writes',
CONVERT(NUMERIC(10,2),rs.avg_physical_io_reads ) as 'avg_physical_io_reads',
CONVERT(NUMERIC(10,0),rs.avg_rowcount ) as 'avg_rowcount'
from sys.query_store_query q
JOIN sys.query_store_plan p ON q.query_id = p.query_id
JOIN sys.query_store_runtime_stats rs ON p.plan_id = rs.plan_id
INNER JOIN sys.query_store_query_text qsqt
  ON q.query_text_id = qsqt.query_text_id
WHERE rs.last_execution_time > dateadd(hour, -1, getutcdate())
ORDER BY rs.avg_duration DESC

SQL Azure transient failure


I'm new to object oriented php. Trying to query database but not getting any results

Question: I am trying to query a table in my DB and am not able to get any results back.  Here is the code I am using:
function get_agent_info($agent) {
        $query = db_query("SELECT * FROM tls_connect_stage_timer WHERE '".$agent."' IN ('start_processing_agent' || 'end_processing_agent')");
        $rows = array();
        foreach ($query as $result) {
            $rows[] = array(
                $result     -> applicant_id,
                $result     -> stage,
                $result     -> event_start,
                $result -> event_end,
                $result     -> start_processing_agent,
                $result     -> end_processing_agent,
                $result     -> start_request_ip,
                $result     -> end_request_ip,
                $result     -> gwf,
            );
        }
        print_r($rows);

    }

I have already used a method like this to get info:
$results = db_query("SELECT * FROM tls_active_applicants WHERE `".$searchParams."`='".$input."'");
$rows = array();
                 foreach($results as $result){
                    $rows[] = array(
                        $result -> first_name,
                        $result -> last_name,
                        $result -> gwf,
                        $result -> visa_type,
                        $result -> current_stage,
                    );
                 }

The second one works fine, where with the first one, if I print_r($rows) it returns an empty array.  Why is the foreach loop starting in the first example but not the second?  I've tested the query, and it works fine. Also any links that could help me understand how the foreach loop is working would be greatly appreciated.  I assumed that since it's saying $query as $result, I would be able to pull a single value saying something like $query->stage but that didn't work.  I'm assuming because there are several entries I would need to do something like $query[0]->stage but I don't really know how that works when it comes to calling properties..
Comment: also if you do queries like this use *Parameterized Queries*.
Comment: What 'start_processing_agent' and 'end_processing_agent' are? Because i can't see how the first query can work
Comment: @Astrayus They are both fields in the table.
Answer: I won't address your question specifically...because you're doing it wrong, and that should stop here :)
This is very, very bad:
db_query("SELECT * FROM tls_connect_stage_timer WHERE '".$agent."' IN ('start_processing_agent' || 'end_processing_agent')");

You should never concatenate data into an SQL string, you'll leave yourself ripe for injection attacks.  This one is even worse:
db_query("SELECT * FROM tls_active_applicants WHERE `".$searchParams."`='".$input."'");

As you're liable to injection attacks from both the field name and the argument.
If you want to use dynamic field names you should use the SelectQuery class, accessible via db_select(). E.g.
$values = array('start_processing_agent', 'end_processing_agent');
$query = db_select('tls_connect_stage_timer', 't')
  ->fields('t')
  ->condition($field_name, $values);

$rows = $query->execute()->fetchAll();

You'll need to make some changes to the above for your specific query, but that's the general idea.
Comment: Thank you for the help. I don't understand the 't'?  Also, what you have as my $values are the field_names, I am trying to search for either/or, would I do a second condition or put in || and put both in the same condition?
Comment: The 't' is just the alias for your selection table. If you don't mind my saying, you're using `IN` in a peculiar way there. Usually one would express that as `WHERE (start_processing_agent = ? OR end_processing_agent = ?)`. `IN` is usually used the other way round, to see if a list of given parameters is in a particular field. If you re-think that part of the approach, it should be easy enough to construct your query using [`db_or()`](https://api.drupal.org/api/drupal/includes!database!database.inc/function/db_or/7). Maybe give it a go, and edit your question with the code if it doesn't work?
Answer: You are messing things up. Build a query with db_select. https://drupal.org/node/310075
Comment: Link-only answers are normally not considered answers, on Stack Exchange. This answer is also wrong: db_select() is used for dynamic queries, and not all the queries are dynamic.
Answer: You are using the IN statement wrong.
If $agent has to be considered as a value, you should use the '=' like for the second query.
If $agent is a field name of your table instead, the query should be like this
"SELECT * FROM table WHERE $agent IN ( SELECT field FROM table)"

On a side note, when you create a string using the double quotes, you can directly use simple variables inside, like in the example above, as opposed to:
" WHERE ".$agent." IN "

About your use of foreach instead, it seem that what you are trying to do can be achieved using
$rows = $query->fetchAll();

Comment: I would rather not concatenate strings to build SQL queries, especially when the variable contains input from users.
Comment: Yep, well, only with user input.


find out where one react component is used?

Question: is there a way to find out where a react component is used in other components?
for example:
navbar components is used in app.js
import React from 'react';

const Navbar = () => {
    return
this is Navbar
;
};

export default Navbar;

import React from 'react'; import Navbar from './Navbar';

const App = () => {return (
<Navbar />
); };

export default App;

Answer: There's two ways to find:

React Developer Tools extension (recommended) tool for debugging. It has a Search bar (text or/regex/)

Most of IDE/Code editors now days has the ability to help you find out component usage

Cheers and I hope this helps
Answer: Most IDEs should be able to find references to the component.
Otherwise, a simple search should find it.
Comment: In vscode, right click on the component name and use `find all references`, keybindings differ on OSes. :)
Comment: yeah there is a search icon in sidebar of vscode that i didn't use that before, thank you


Where does location cck module save the data?

Question: I've gone through the code in modules/all/location/contrib/location_cck but am unable to figure out where this CCK field is storing the address data.
I guest that it's install file should have table fields to store geographic data like city, lat long etc. But it isn't there.
Edit: Ultimately I want to access all those location fields separately(beyond Views) when writing modules etc. For example I want to read postal code attached to a particular node for some analysis. That's why I'd like to extract location info about nodes.
Comment: Andy, I'm sorry for this. Not only for this or the last one, you'd satisfactorily answered my questions sometime back too. I accept my mistake for not mentioning link to this question. Next time I'd take care this..
Comment: I already answered this here: http://drupal.stackexchange.com/a/16284/633.
Comment: ...but I could still not figure out answer to this.
Comment: then I think it would be better to ask on the existing question, rather than create a new one personally.
Comment: Sorry, but this question wasn't the main theme of my last question which you answered. Starting long discussion like here would have looked a misfit there.
Comment: at the very least you then respond to the first person who answered your question with a link to this question. Just ignoring someone who's taken the time to answer your question because you don't understand their answer's a little rude imho.
Answer: As mentioned, the data's stored in the location table, which is handled by the location API and indexed by location ID (lid). The CCK field stores an lid. If you have access to the node object with the field, then you can access the address fields directly from the node object. If you don't have the node object and want to avoid the cost of loading lots of nodes, you can get the CCK DB details using content_fields($field_name, $content_type_name) and content_database_info($field). Then you can use location_load_location($lid) to get the location. I've just taken a quick look at the source for location_load_location() and note that it might do JIT geocoding (so it might actually write as well as read!). I imagine you won't have any issues with doing a join directly on the location table, but I always prefer to use the API where possible.
There's also the command location_load_locations($id, $key) which looks like it can give you all locations associated with a particular nid or vid, but I don't think that's what you're after.
Comment: Looks like this is what I'm looking for. I'll first go through these functions to understand it fully.
Answer: Most CCK field modules let the CCK take care of creating tables and fields and saving data. This is done by implementing hook_field_settings().
Comment: location_cck is slightly atypical in that the field itself stores only an ID into location.module's table.
Comment: I'm guessing but not sure if I'm correct: Location is stored in location table with key lid, then this info is linked to node using nid,lid in location_instance table .
Comment: @AgA CCK stores an LID as the field's data. This is sufficient to do the look-up from the location table. I don't know what location_instance is for.
Comment: How to query location data for a particular node using drupal_query? I think I need to join location_instance and node tables on nid to get lid then use it with location table to get the location data.
Comment: @AgA can you edit your question to explain in a bit more detail what you're trying to do? (Eg. where is this code running, what's its final goal?) Direct DB queries might not be the best way to achieve what you want.
Comment: I've edited the question


Ina triangle $BFC$, show that $|KF|=b$ when $a>b$

Question: Today I've encountered a question like this;
In a triangle $BFC$ angles $<B=20^\circ,\quad <F=110^\circ,\quad <C=50^\circ$,
There are points $D$(on BC) $\text{and}$ $K$(on BF) such that $|BD|=|BK|=a+b$ and $|DC|=a$, assuming $a>b$. How do we show that $|KF|=b$?,
This question is more abstract than the questions I generally deal with , although I've given it a try and my efforts are;
Creating a point $G$ such that points $D,K,G$ are linear (1)
Drawing a line from $B$ to $G$ and notice that $<GKB=30^\circ$ (2)
Seeing that $|GD|=a+b$ too (3)
Having discovered $<BGK=50^\circ$ drawing a line from $G$to $F$ (4)
Trying to find  $<KGF$ then expect to use the cosine theorem or perhaps another relation (This is where I am stuck),
I've begun this attempt of a solution with expectations like drawing the replica's or symmetric's of several parts in the triangle or finding a trigonometric relation, but I failed.
What do you suggest?
Comment: I think the question does not have enough information. Imagine moving the line KD, in a parallel way, closer and closer to B. Then BD = a+b becomes smaller and DC = a becomes larger which implies b becomes smaller. However KF = b is also becoming larger which is impossible.
Comment: I also thought there is something fishy about this question. Thank you:))
Comment: I draw with Geogebra and the conclusion is false because $KF$ depends on $a$ . Unless the request is to find $\dfrac{a}{b}$ such that $KF=b$
Answer: The property is false.
A counterexample.-If $(a,b)=(2,1)$ then $\overline{BD}=\overline{BK}=3$ and
$\overline{BC}=3+2=5$. Therefore if $\overline{KF}=b=1$ then $\overline{BF}=4$.
But then (keeping in mind that $\sin 110^{\circ}=\sin 20^{\circ}$)
$$\frac{4}{\sin 50^{\circ}}=\frac{5}{\sin 20^{\circ}}$$
This is not true.


Decompose $\frac{x^2-2x+3}{(x-1)^2(x^2+4)}$, partial fraction braindead

Question: decompose $\frac{x^2-2x+3}{(x-1)^2(x^2+4)}$
the way my teacher wants us to solve is by substitution values for x,
I set it up like this:
(after setting the variables to the common denominator and getting rid of the denominator in the original equation)
$x^2-2x+3= \frac{A}{x-1}+\frac{B}{(x-1)^2}+\frac{Cx+D}{x^2+4}$
1)Let $x=1$, after plugging in for $x$ I got $2=5B$, or $B=\frac{2}{5}$
2)let the $x=0$, I get $3=-4A+4B+D$, and if I substitute B and simplify I get $D=4A-\frac{7}{5}$. Furthermore, no matter what value I plug in I still end up with two variables and cant seem to find a way to eliminate one. It seems like I have to set it up as the triple system of equations but I just dont know how to apply it here. Would really appreciate if you guys could give me a hint on how to go about it.
Comment: On the left side of the equation $x^2-2x+3=...$ , the denominator is missing.
Comment: [How to format your questions](http://meta.math.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference).
Answer: Hint : Start with $$\frac{x^2-2x+3}{(x-1)^2(x^2+4)}=\frac{A}{x-1}+\frac{B}{(x-1)^2}+\frac{CX+D}{x^2+4}$$
Multiply with $(x-1)^2(x^2+4)$ and then insert the special values.
Comment: I did set it up like that, but i keep getting 2 unknowns and have no clue how to go down to one.
Comment: The insert of special values does not always give ALL solutions. What are the equations you got ? Did you get $4$ equations ?
Comment: after making x=1, I've got  B=2/5, then I  did x=0, and got D=4A-(8/5)+3 and im stuck at this point since no matter what value for X i put. i still have D and C as unknowns
Comment: Try to solve the rest using the value for $B$.
Comment: I did, still end up with D and C as unknowns
Comment: Does the teachet allow ONLY the insert, or does he only suggest it as a help ?
Comment: she said, you should only do the x=.. substitution method, otherwise I would have simplified denominator and just did the long division. it seems like we're ought to use system of equations , but i have no idea how to set them up with what i have right now
Comment: $x^2-2x+3=(A+C)x^3+(-A+B+D-2C)x^2+(4A-2D+C)x+(-4A+4B+D)$. Did you get this equation ?
Comment: The solution is $A=-\frac{4}{25}$,$B=\frac{2}{5}$, $C=\frac{4}{25}$ , $D=\frac{19}{25}$.
Comment: You can insert the complex values $-2i$ and $2i$ to get the values $C$ and $D$.
Answer: If you substitute $x=2i$, this gives
$\;\;\;-1-4i=((2Ci+D)(2i-1)^2=(-3-4i)(2Ci+D)=(-3D+8C)-(6C+4D)i$
Therefore $8C-3D=-1\;\;$ and $\;\;6C+4D=4,\;$ so
$\hspace{.6 in}16C-6D=-2\;\;$ and $\;\;9C+6D=6\implies 25C=4\implies C=\frac{4}{25}$.
Then $D=1-\frac{3}{2}C=1-\frac{6}{26}=\frac{19}{25},\;\;$ and $\;\;A=-\frac{4}{25}\;$ since $0=A+C$ from the coefficient of $x^3$.
Answer: After $B=\frac25$, you may obtain $C$ and $D$ similarly with
$$\frac{x^2-2x+3}{(x-1)^2}=\frac{A(x^2+4)}{x-1}+\frac{B(x^2+4)}{(x-1)^2}+{Cx+D}$$
by setting $x^2=-4$ to get $1=-8C+3D$, $2=3C+2D$, yielding
$C=\frac4{25}$ and $D=\frac{19}{25}$. On the other hand, $A$ can be obtained by examining the limit $x\to \infty$ of above equation, which leads to $A=-C=-\frac4{25}$.


How would I make a rectangle move every second in a WPF app in c#

Question: I have a WPF app and I would like to change the position of a rectangle every second for an infinite amount of time (or until the application is closed)
Right now I have this
while(true)
{
     rect.Margin = new Thickness(rect.Margin.Left + 10, 0, 0, 0);
     System.Threading.Thread.Sleep(100);
}

but this seems to just hang the application because it just delays the application by 100ms every single update
I want a while true loop that wont hang the application and fires every 1 second
Comment: I've also tried System.Threading.Tasks.Task.Delay(1000) and System.Threading.Tasks.Task.Delay(1000).Wait() but they appear to do the same thing
Comment: And yes I know I put 100ms instead of 1000ms in my code.
Comment: Use a DispatcherTimer.
Answer: It's hard to tell from the little bit of code that you shared.  But it is most likely because your while loop is on the UI thread, so when you call System.Threading.Thread.Sleep(100), you are putting the UI thread to sleep and therefore it visually hangs.  The same will occur when you call Task.Delay with Wait() because it will synchronously run, and hold up the UI thread.
Instead, try using async/await.  This will free up the thread and allow other operations to perform on the same thread while the code is on the await line.
private async Task MyWhileLoopMethod()
{
    while(true)
    {
        rect.Margin = new Thickness(rect.Margin.Left + 10, 0, 0, 0);
        await System.Threading.Tasks.Task.Delay(1000);
    }
}

You will need to make every call up the chain async/await as well.  So for example, if you had a Button_Click event that triggered this rectangle movement, then at the top of the chain you would do this:
private async void Button_Click(object sender, RoutedEventArgs e)
{
    await MyWhileLoopMethod();
}


Rendering renderable arrays in an item_list

Question: I'm struggling to get a render-array constructed, and I think it's because of how theme_item_list works. I would be grateful if anyone could help find the problem (or tell me to implement it without item_list!)

The context: This function is called from my module's
  hook_block_view() to render the content of the block. The module is
  an example module showing how to use a new API. I hope nothing of that
  is needed to answer this point, except that these calls are not made
  from a theme.

The code I have is:
function example_render() {
  $block = array();

/* fill $providers[] ... */

  $items = array();
  foreach ($providers as $metric => $provider) {
    $value = $provider['value'];
    $class_link = $provider['name'];
    $name_link = $provider['name'];

    // Construct a renderable array wrapper for the template invocation.
    $items[$metric] = array(
      '#type' => 'container',
      '#attributes' => array('class' => array('provider')),
      'info' => array(
        'providername' => array(
          '#prefix' => '',
          '#markup' => $name_link,
          '#suffix' => '',
        ),
        'providervalue' => array(
          '#prefix' => '',
          '#markup' => $value,
          '#suffix' => '',
        ),
        'metricname' => array(
          '#prefix' => '',
          '#markup' => $class_link,
          '#suffix' => '',
        ),
      ),
    );
  }

  // Finally, bring it all together.. a container with the via text then a list
  // of the providers.
  $block['content'] = array(
    '#type' => 'container',
    '#attributes' => array('class' => array('example')),
    'ttl_text' => array(
      '#prefix' => '
',
      '#markup' => $record['title'],
      '#suffix' => '
',
    ),
    'via_text' => array(
      '#prefix' => '',
      '#markup' => $via_text,
      '#suffix' => '',
    ),
    'metrics' => array(
      '#items' => $items,
      '#theme' => 'item_list',
      '#attributes' => array(
        'class' => 'providers-list',
      ),
    ),
  );
  return $block;
}

and this is what I get when run with 9 items in the list:

A bacterial sulfonolipid triggers multicellular development in the closest living relatives of animals
via Lagotto article information courtesy of Lagotto

not very inspiring!
The three 'Array' under 'info' in the html here obviously correspond to the providername/providervalue/metricname arrays in the code i.e. the $items array isn't being interpreted as a render array at all.
How can I make this work?
Answer: According to https://api.drupal.org/api/drupal/includes!theme.inc/function/theme_item_list/7 theme_item_list is looking for data key so your list item should look like this:
// Construct a renderable array wrapper for the template invocation.
$tmp = array(
  '#type' => 'container',
  '#attributes' => array('class' => array('provider')),
  'providername' => array(
    '#prefix' => '',
    '#type' => 'markup',
    '#markup' => $name_link,
    '#suffix' => '',
  ),
  'providervalue' => array(
    '#prefix' => '',
    '#type' => 'markup',
    '#markup' => $value,
    '#suffix' => '',
  ),
  'metricname' => array(
    '#prefix' => '',
    '#type' => 'markup',
    '#markup' => $class_link,
    '#suffix' => '',
  ),
);
$items[$metric]['data'] = drupal_render($tmp);

That should give you single element:

test1test1test1


Suitable approach for serving HLS playlist from GCP using node.js server

Question: I have a dilemma in serving dynamic hls playlists to users, since all hls files will have to be served from GCP storage. Since index file is static, I can simply serve it using express static as outlined in one answer in here but it is limited to file directory (in my opinion). For streaming HLS playlist from GCP, I tried the following approach
For serving dynamic index files from GCP
router.get('/segments-list', (req, res) => {
  const playlistName = req.query;
  const remoteFile = hlsPlaylistBucket.file('${playlistName}.m3u8');
  remoteFile.download().then((data) => {
    const contents = data[0];
    res.send(contents);
  });
});

For serving individual segment files (.ts) to users
router.get('/segments/:segment', (req, res) => {
  const { segment } = req.params;
  //* Gets specific segment files from GCP cloud in request of particular segments
  const remoteFile = hlsPlaylistBucket.file(`${segment}`);

  remoteFile.download().then((data) => {
    const contents = data[0];
    res.send(contents);
  });
});

Here is the manifest file
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:8
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:8.341667,
segments/file000.ts
#EXTINF:8.341667,
segments/file001.ts
#EXTINF:8.341667,
segments/file002.ts
#EXTINF:8.341667,
segments/file003.ts
#EXTINF:5.572233,
segments/file004.ts
#EXT-X-ENDLIST

Now this approach seems to work but this is a flawed approach due to downloading small chunks for files in memory every time user requests. And serving signed urls for each segment files is also not a suitable way was well as one video might contain 200 segments files. Is my approach correct or do I have to serve HLS playlists like this in GCP. The real problem arises in serving base url segments files and I have been stuck for the past couple of days in this problem but I cannot find a suitable way to serve VOD file to users by serving HLS playlist
Any help will be appreciated
Answer: I think the best option here is to serve them from a GCP bucket using CORS (Cross-Origin Resource Sharing), here you can find a guide on how to configure CORS for a GCP bucket 1.
Even further, you could render the conversion and packaging of the original video files to the Transcoder API Google service 2, which can generate and store the HLS manifest and files into a GCP bucket, to later be used by any Video Streaming Player, take a look at this Quickstart guide 3.
Comment: Thanks for the suggestion, although atm it wont be possible for me to incorporate changes due to time constraint so I have tweaked my approach from downloading to streaming each file segment, although it works atm but in future I shall follow your suggestion for a more streamlined approach
Comment: Can I help you with other information? If you think that my answer helped you, please consider accepting it by clicking the checkmark (✔) on the left side under the vote arrows. Should change the color to green.


Drop the duplicated rows and merge the ids using groupby in pyspark

Question: I have a dataframe where some of rows having duplicated ids but different timestamp and some of rows having duplicated ids but the same timestamp but having one of following (yob and gender) columns null. Now I want to do an operation using groupby:

if the same id having difference timestamp, want to pickup the recent timestamp.
If the same ids having same timestamp but the any of column having null(yob and gender), that time, want to merge the both id as single record without null. below I have pasted the data frame and desired output.

Input data
from pyspark.sql.functions import col, max as max_
df = sc.parallelize([
    ("e5882", "null", "M", "AD", "9/14/2021 13:50"),
    ("e5882", "null", "M", "AD", "10/22/2021 13:10"),
    ("5cddf", "null", "M", "ED", "9/9/2021 12:00"),
    ("5cddf", "2010", "null", "ED", "9/9/2021 12:00"),
    ("c3882", "null", "M", "BD", "11/27/2021 5:00"),
    ("c3882", "1975", "null",  "BD", "11/27/2021 5:00"),
    ("9297d","1999", "null", "GF","10/18/2021 7:00"),
    ("9298e","1990","null","GF","10/18/2021 7:00")
]).toDF(["ID", "yob", "gender","country","timestamp"])

Desire output:

code used in this problem, but not get the accurate result, some of ids are missing,
w = Window.partitionBy('Id')
# to obtain the recent date
df1 = df.withColumn('maxB', F.max('timestamp').over(w)).where(F.col('timestamp') == F.col('maxB')).drop('maxB')
# to merge the null column based of id
(df1.groupBy('Id').agg(*[F.first(x,ignorenulls=True) for x in df1.columns if x!='Id'])).show()

Answer: Using this input dataframe:
df = spark.createDataFrame([
    ("e5882", None, "M", "AD", "9/14/2021 13:50"),
    ("e5882", None, "M", "AD", "10/22/2021 13:10"),
    ("5cddf", None, "M", "ED", "9/9/2021 12:00"),
    ("5cddf", "2010", None, "ED", "9/9/2021 12:00"),
    ("c3882", None, "M", "BD", "11/27/2021 5:00"),
    ("c3882", "1975", None, "BD", "11/27/2021 5:00"),
    ("9297d", None, "M", "GF", "10/18/2021 7:00"),
    ("9297d", "1999", None, "GF", "10/18/2021 7:00"),
    ("9298e", "1990", None, "GF", "10/18/2021 7:00"),
], ["id", "yob", "gender", "country", "timestamp"])

If the same id having difference timestamp, want to pickup the recent timestamp.

Use window ranking function to get most recent row per id. As you want to merge those with the same timestamp you can use dense_rank instead of row_number. But first you need to convert timestamp strings into TimestampType otherwise comparison won't be correct (as '9/9/2021 12:00' > '10/18/2021 7:00')
from pyspark.sql import Window
import pyspark.sql.functions as F

df_most_recent = df.withColumn(
   "timestamp",
   F.to_timestamp("timestamp", "M/d/yyyy H:mm")
).withColumn(
   "rn",
   F.dense_rank().over(Window.partitionBy("id").orderBy(F.desc("timestamp")))
).filter("rn = 1")

If the same ids having same timestamp but the any of column having null(yob and gender), that time, want to merge the both id as single
record without null. below I have pasted the data frame and desired
output.

Now the above df_most_recent contains one or more rows having the same most recent timestamp per id, you can group by id to merge the values of the other columns like this:
result = df_most_recent.groupBy("id").agg(
     *[F.collect_set(c)[0].alias(c) for c in df.columns if c!='id']
     # or *[F.first(c).alias(c) for c in df.columns if c!='id']
)

result.show()
#+-----+----+------+-------+-------------------+
#|id   |yob |gender|country|timestamp          |
#+-----+----+------+-------+-------------------+
#|5cddf|2010|M     |ED     |2021-09-09 12:00:00|
#|9297d|1999|M     |GF     |2021-10-18 07:00:00|
#|9298e|1990|null  |GF     |2021-10-18 07:00:00|
#|c3882|1975|M     |BD     |2021-11-27 05:00:00|
#|e5882|null|M     |AD     |2021-10-22 13:10:00|
#+-----+----+------+-------+-------------------+


Domain Access permissions issue

Question: I am having an issue with Domain Access in Drupal 7.
The admin user (and seemingly any user I add to the system) is unable to edit content attached to any child domains on the site.  The main domain is fine, but all others on the same site is effected.
When I attempt to edit content on the Admin (or other user) I get:

You are not authorized to access this page.

I can view all content with no problem.
How do I fix these permissions issues?
The following is the screenshot of the permissions implemented by the module.

The "Bypass content access control" permission is disabled for all roles including the "Administrator" one; no other access module is enabled. Two Domains have been set up, the primary one and a sub-domain, which shares some content and has its own. Two languages are enabled for each domain (English and French) using the i18n module.
The "Edit any content on assigned domains," and "Delete any content on assigned domains" permissions are set, but the users still cannot edit the content.
Comment: Hmm I seem to have figured out that If I log out and go to /user/login on the domain i want to edit content on, and login to the site from there, I have the ability to edit the content from that domain but nothing else. (even though the account should have access on all domains) -

I am not complaining, but it does seem rather roundabout
Comment: Did you check the issue list http://drupal.org/project/issues/domain?text=permission&status=Open&priorities=All&categories=bug&version=7.x&component=All  and documentation http://drupal.org/documentation/modules/domain especially http://drupal.org/node/1096972 ? What Domain Access version do you have installed ?
Comment: Ive been reviewing the documentation, as far as I can tell I am compliant. however it seems to not be working.

See screenshot of access perms: http://imgur.com/HUhk2

Domain Access version: 7.x-2.18 / D version: 7.12
Answer: Hi try to configure this setting
http://yoursite.tld/admin/structure/domain/roles
Answer: If you are logged in at your maindomain and want to edit content on the other domains, you have to enable the<code>$cookie_domain = '.yourmaindomain.com';</code> in your settings.php.
It is also described in this video domain acess video

Domain Access permissions issue


R to Stata: Exporting tibble with value labels

Question: My task is to write a Stata-readable dataframe with variable labels and value labels. It appears that the package haven is the most suitable one to do such an operation (better than readstata13 and foreign).
I begin with creating the tibble:
library(haven)

x1 <- labelled(
  sample(1:5),
  c(Good = 1, Bad = 5)
)
x1

x2 <- labelled(
  c("M", "F", "F", "F", "M"),
  c(Male = "M", Female = "F")
)
x2

df <- tibble::data_frame(x1, x2, z = 1:5)
df

Then, I export the tibble
write_dta(df)

When I read it in Stata, the labels replace the real values (in x1, 1 becomes equal to "Good" and 5 becomes equal to "Bad"). How can I read the tibble in Stata, in way such that the values remain from 1 to 5 and "Good" and "Bad" are stored as value labels?
Answer: It's possible but rather fiddly, you will have to attach labels manually using attr1 function as in the example below.
# Attach variable and value labels to a new R data.frame
attr(df, "var.labels") <- c("my var label 1", "my var label 2")
# To create and attach value labels
val1 <- c(apple=10, beans=20)
val2 <- c(kg=1, bag=2)
attr(df, "label.table") <- list(val1, val2)
attr(df, "val.labels") <- c("val1", "val2")
# And finally describe the dataset
attr(df, "datalabel") <- "Collected from MWI 2013"

1The original solution was provided by Melanie Bacou and is available in this gist.
Comment: That solution is based on the function `foreign:write.dta`, that has been frozen for Stata versions after 10. It is of no use to me, unfortunately.


3-part question on joint PDFs

Question: a.) Let U, V be uniformly distributed over the set $\{(u,v): $$0<u<v<1$}.
Let $X$ = $-$$log(U)$, $Y$ = $-$$log(V)$, $Z$ = $max$($X$,$Y$).
a.) Draw the support of the joint distribution ($U$, $V$) and the joint pdf $U$ and $V$.
Here, I use the identity that the joint pdf equals the conditional pdf multiplied by the marginal pdf.
$f_{u,v}$($u$,$v$) = $f_{u|v}$($u|v$)$*f_v(v)$ = $\frac{1}{1-u}$*$I(0<u<v)$, where $I$ is the identity function.
The $\frac{1}{1-u}$ part comes from the PDF of the uniform distribution of $V$ over $(u, 1)$: $f_v(v) = \frac{1}{1-u}$
b.) Find the joint PDF of (X,Y). What is its support?
Really not sure on this one. What are my first steps? I'm guessing they come from part a.) but I think my work for part a.) is wrong.
c.) Find the conditional expectation $E$($Z$|$Y$).
My work so far:
$E$($Z$|$Y$) = $E$($max(X,Y)$|$Y$) = $max(E(X|Y),Y)$, and
$E(X|Y) = \int_0^v f_{X|Y}(x|y)*xdx = \int_0^v \frac{1}{(vu)^2}*xdx
= \frac{v^2}{(vu)^2} = \frac{1}{v^2}$.
I calculated $f_{X|Y}(x|y)$ in part b.) (not shown) using the derivative formula for deriving PDFs but I'm almost entirely sure it's wrong.
Answer: In part a, it says uniformly distributed over the set $\{(u,v):0<u<v<1\}$, which means the region between lines $u=0, v=u,v=1$ ($v$ is in y-axis and $u$ is in x-axis). So, he joint PDF is 1/Area of this region.
In part b, you can apply Jacobian technique. Another method is to calculate $F_{XY}(x,y)$ and differentiate wrt $u$ and $v$. It's a good exercise, but I'd highly advise the former. Also, the region of support is very important. Hint: it'll extend to infinity but be under y=x line.
In part c, it seems $\max(X,Y)=X$ because $U<V\rightarrow Y<X$.
Comment: I'm still having issues on part a.). Drawing the support, I get that the joint PDF is 1/(2*v), because the area between u = 0, v = u and v = 1 is a triangle with height 1 and length v. But this leads to a joint PDF that does not integrate to 1, which violates the definition of the PDF. Do you have any other insights for me?
Comment: No, the area is $1/2$.
Comment: Thank you. Then I get that the PDF of (U, V) = 2 (a valid PDF) and with the Jacobian technique, I get that the PDF of (X, Y) = 2e^{-x-y} (also a valid PDF). Then, I try to get E[Z|Y] = E[X|Y] by getting the conditional PDF of (X | Y), which I calculate to be -e^{y-x} / {-e^y + 1}. This doesn't seem right to me because my expected value depends on Y even though its conditional on Y. Im not sure where I went wrong here, so if I could get one last piece of advice, that'd be great.
Comment: I'm not sure about your algebraical calculations but that is normal, E[X|Y] is a function of $Y$.
Comment: Sounds great, thank you!

3-part question on joint PDFs


Comparação de tabelas

Question: Estou criando um sistema de bolão, onde o administrador cadastra os jogos e o placar, e o usuário tem a opção de chutar um placar, caso esse placar for igual ao cadastrado pelo administrador ele informa que o usuário acertou, caso contrario que errou.
Vou colocar o código em que o administrador cadastra a partida, o do usuário que aposta e também a imagem do banco de dados. Neste caso a chave primária é o código da partida
Tabela tb_jogos onde o administrador cadastra as partidas:
CREATE TABLE tb_jogos (
  casa varchar(20) NOT NULL,
  placarcasa int(1) NOT NULL,
  fora varchar(20) NOT NULL,
  placarfora int(1) NOT NULL,
  codigo int(5) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
  PRIMARY KEY (codigo);

Código administrador para cadastrar partida:
<?php
session_start();
if(!empty($_SESSION['email'])){

}else{
    echo"";
}
$nome=$_SESSION['nome'];

$placarc = substr(mt_rand (0,5),0,1); //gera números de para time casa 0 a 5
$placarf = substr(mt_rand (0,5),0,1); //gera números de para time fora 0 a 5

$btnCadastrar = filter_input(INPUT_POST, 'btnCadastrar', FILTER_SANITIZE_STRING);
if($btnCadastrar){
    include_once ("../conn/conexao.php");
    $dados = filter_input_array(INPUT_POST, FILTER_DEFAULT);

    $jogos = "INSERT INTO tb_jogos(casa, placarcasa, fora, placarfora, codigo)VALUES(
        '".$dados['casa']."',
        '".$dados['placarcasa']."',
        '".$dados['fora']."',
        '".$dados['placarfora']."',
        '".$dados['codigo']."'
        )";
        $mensagem="";
    $resultado_jogos = mysqli_query($conexao, $jogos) or die ($mensagem);
    if($resultado_jogos):
        echo "";
    else:
        echo "";
    endif;
}

?>

<!DOCTYPE html>

    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel='stylesheet' href='//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css'/>
    <link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">

    <title>Lance Web</title>

    <link href="../bootstrap/css/bootstrap.min.css" rel="stylesheet">

      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1" aria-expanded="false">

      </button>
        Voltar

Cadastrar Partidas(current)

Editar Partidas(current)

    <form class="form-group" action="" method="post">

                Time Casa
                    <select class="form-control" name="casa" required>
                         <option selected></option>
                         <option value="Atletico-go">Atlético-GO</option>
                         <option value="Atletico-mg">Atlético-MG</option>
                         <option value="Atletico-pr">Atlético-PR</option>
                         <option value="Avai">Avaí</option>
                         <option value="Bahia">Bahia</option>
                         <option value="Botafogo">Botafogo</option>
                         <option value="Corinthians">Corinthians</option>
                         <option value="Coritiba">Coritiba</option>
                         <option value="Cruzeiro">Cruzeiro</option>
                         <option value="Chapecoense">Chapecoense</option>
                    </select>

                Placar time Casa

                            <input id="placar-c" class="form-control" type="text" name="placarcasa" maxlength="1" value="" style="text-align: center;" required />

                            <input class="btn btn-success submit-botao" type="submit" value="Gerar" onclick="myFunction()">

                Time Fora
                    <select class="form-control" name="fora" required>
                        <option selected></option>
                        <option value="Flamengo">Flamengo</option>
                        <option value="Fluminense">Fluminense</option>
                        <option value="Gremio">Grêmio</option>
                        <option value="Palmeiras">Palmeiras</option>
                        <option value="Ponte preta">Ponte Preta</option>
                        <option value="Santos">Santos</option>
                        <option value="Sao paulo">São Paulo</option>
                        <option value="Sport">Sport</option>
                        <option value="Vasco">Vasco</option>
                        <option value="Vitoria">Vitória</option>
                </select>

                Placar time fora

                            <input id="placar-f" class="form-control" type="text" name="placarfora" maxlength="1" value="" style="text-align: center;" required />

                            <input class="btn btn-success submit-botao" type="submit" value="Gerar" onclick="myFunction2()">

                    Codigo da partida

                            <input class="form-control" type="text" name="codigo" maxlength="5" style="text-align: center;" required>

                        <input class="btn btn-danger btn-lg" type="button" value="Cancelar">

                        <input class="btn btn-info btn-lg submit-botao" type="submit" value="Cadastrar" name="btnCadastrar">

    </form>

Tabela tb_aposta onde o usuário faz as apostas:
CREATE TABLE tb_aposta (
  apostacasa int(1) NOT NULL,
  apostafora int(1) NOT NULL,
  valor int(10) NOT NULL,
  data varchar(15) NOT NULL,
  usuario varchar(40) NOT NULL,
  codigopartida int(5) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
  ADD PRIMARY KEY (codigopartida);

Código do usuário onde ele realiza a aposta:
<?php
session_start();
include_once("../conn/conexao.php");
if(!empty($_SESSION['email'])){

}else{
    echo"";
}
$nome=$_SESSION['nome'];
$email=$_SESSION['email'];
$saldo="SELECT * FROM tb_usuario WHERE email= '$email'";
$exe= mysqli_query($conexao, $saldo);
$linha = mysqli_fetch_array($exe);

$btnApostar = filter_input(INPUT_POST, 'btnApostar', FILTER_SANITIZE_STRING);
if($btnApostar){
    include_once ("../conn/conexao.php");
    $dados = filter_input_array(INPUT_POST, FILTER_DEFAULT);

    $aposta = "INSERT INTO tb_aposta(apostacasa, apostafora, valor, data, usuario, codigopartida)VALUES(
        '".$dados['apostacasa']."',
        '".$dados['apostafora']."',
        '".$dados['valor']."',
        '".$dados['data']."',
        '".$_SESSION['nome']."',
        '".$dados['codigopartida']."'
        )";
    $r_aposta = mysqli_query($conexao, $aposta) or die (mysqli_error($conexao));
    if($r_aposta):
        echo "";
    else:
        echo "";
    endif;
}
?>

<!DOCTYPE html>

    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel='stylesheet' href='//maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css'/>
    <link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">

    <title>Lance Web</title>

    <link href="../bootstrap/css/bootstrap.min.css" rel="stylesheet">

      <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1" aria-expanded="false">

      </button>
        Voltar

Saldo  R$<?php echo $linha['saldo']; ?>(current)

Apostar(current)

Histórico(current)

Ranking(current)

Loja(current)

        <?php
                include("../conn/conexao.php");
                //ORDER BY serve para organizar os dados de acordo com o que voce quiser
                $buscar="SELECT * FROM tb_jogos ORDER BY codigo DESC";
                $exe= mysqli_query($conexao, $buscar) or die ("OCORREU UM ERRO AO MOSTRAR OS DADOS");
                //começo da tabela
                echo "

                              Codigo Partida
                              Time Casa
                              Placar
                              Time Fora
                              Placar
                              Lance

";

                while($linha = mysqli_fetch_array($exe)){
                    echo "<form class='form-group' action='' method='post'>

                              <input type='int' class='form-control' name='codigopartida' maxlength='1' value=".$linha['codigo']." style='text-align: center;' readonly='readonly'>
                              ".$linha['casa']."
                              <input type='int' class='form-control' name='apostacasa' maxlength='1' value='' style='text-align: center;'>
                              ".$linha['fora']."
                              <input type='int' class='form-control' name='apostafora' maxlength='1' value='' style='text-align: center;'>
                              <input type='int' class='form-control' name='valor' maxlength='5' value='' style='text-align: center;'>
                              <input class='btn btn-success submit-botao' type='submit' value='Apostar' name='btnApostar'>

                          </form>";

                }
        ?>

Comment: @DarleiFernandoZillmer Então na pagina do usuário apostar eu preciso que logo depois que ele colocar os placares da partida seja feita um comparação com os placares que o administrador cadastrou, se for igual ele acerta se for errado ele erra...
Comment: Existe duas tabelas uma (tb_jogos) para guarda o resultado que o administrador deu e outra tabela (tb_apostas) para guarda o resultado em que o usuário deu, então eu não aprendi ajax ainda, mais se quiser dar um exemplo aceito sim, mais minha grande dúvida é na comparação das tabelas
Answer: A primeira coisa que você precisa ter em mente é a ligação(referencia) entre as duas tabelas para saber qual aposta faz referencia a qual partida, então teremos:
Tabela tb_jogos onde o administrador cadastra as partidas:
    CREATE TABLE IF NOT EXISTS `tb_jogos` (
  `id_jogo` int(11) NOT NULL,
  `casa` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
  `placarcasa` int(1) NOT NULL,
  `fora` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
  `placarfora` int(1) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Tabela tb_aposta onde o usuário faz as apostas:
   CREATE TABLE IF NOT EXISTS `tb_aposta` (
  `id_aposta` int(11) NOT NULL,
  `id_jogo` int(5) NOT NULL,
  `apostacasa` int(1) NOT NULL,
  `apostafora` int(1) NOT NULL,
  `valor` float NOT NULL,
  `data` date NOT NULL,
  `usuario` varchar(40) COLLATE utf8_unicode_ci NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Note que na segunda tabela foi adicionado um campo de id do jogo, para saber sobre qual jogo foi feita a aposta.
Achei que ficaram muitos códigos para postar aqui, então subi para um repositório no Github.

Repositório
https://github.com/darleizillmer/lance

Comparação de tabelas