Stacked Bar Graph In R

Question: So I'm trying to make a stacked bar plot. This is all the data that needs to be in the plot. I am wanting a-e to be next to f-j. I am also trying to get the y-axis to be reasonable. For some reason it is putting numbers instead of a limit. I am probably doing it completely wrong. I've looke
data<- rbind(c(4655, 4212, 3838, 3583, 3124, 3078, 1411, 2589, 2524, 2429),
         c(2826, 2589, 2557, 2713, 2497, 1, 2, 66, 1757, 1822))

a <- cbind(data[, 1], 1, c("50-60", "40-49"))
b <- cbind(data[, 2], 2, c("50-60", "40-49"))
c <- cbind(data[, 3], 3, c("50-60", "40-49"))
d <- cbind(data[, 4], 4, c("50-60", "40-49"))
e <- cbind(data[, 5], 5, c("50-60", "40-49"))
f <- cbind(data[, 6], 1, c("50-60", "40-49"))
g <- cbind(data[, 7], 2, c("50-60", "40-49"))
h <- cbind(data[, 8], 3, c("50-60", "40-49"))
i <- cbind(data[, 9], 4, c("50-60", "40-49"))
j <- cbind(data[, 10], 5, c("50-60", "40-49"))

data  <- as.data.frame(rbind(a, b, c, d, e, f, g, h, i, j))
colnames(data) <-c("Number", "Year", "Age")
data$Year      <- factor(data$Year, labels = c("FY13",
                         "FY14","FY15","FY16","FY17"))

library(ggplot2)

ggplot(data = data, aes(x = data$Year, y = Number, fill = Age)) +
       geom_bar(stat = "identity")

I am probably doing it completely wrong. I've looked and looked, but cannot find something else to help me.
Comment: I was going to say that Number is a factor in your example, you would need to convert it to numeric `data$Number <- as.numeric(data$Number)`... Check the answer by @JanLauGe to see if this is what you need...
Answer: It sounds like you do not want the Year, Age groups combined... So I created another variable to keep them separate and then used facet_grid:
data <- data.frame(as.vector(data), rep(rep(1:5, each=2),2), c("50-60","40-49"), rep(c("Grp1","Grp2"), each=10))

#data  <- as.data.frame(rbind(a, b, c, d, e, f, g, h, i, j))
colnames(data) <-c("Number", "Year", "Age", "Grp")
data$Year      <- factor(data$Year, labels = c("FY13",
                                               "FY14","FY15","FY16","FY17"))

library(ggplot2)

ggplot(data = data, aes(x = Grp, y = Number, fill = Age)) + facet_grid(~Year) +
  geom_bar(stat = "identity")

Answer: You need to dodge your position in ggplot2 like below:
Data:
data  <- as.data.frame(rbind(a, b, c, d, e, f, g, h, i, j))
data <- cbind(data,Group = c(rep("a-e",5),rep("f-j",5)))
colnames(data) <-c("Number", "Year", "Age",
                   "Group")
data$Year <- factor(data$Year, labels = c("FY13","FY14","FY15","FY16","FY17"))
data$Age <- factor(data$Age)
data$Group <- factor(data$Group)

Plotting:
library(ggplot2)

ggplot(data = data, aes(x = Age , y = as.numeric(Number), fill = Year)) +
  geom_bar(stat = "identity",position="dodge") + facet_grid(~Group)

?ggplot2::position_dodge

Dodging preserves the vertical position of an geom while adjusting the
  horizontal position.

Output:
Answer: Do you want this?
# Your original data
data %>%
  # Number should be numeric
  mutate(Number = as.numeric(Number),
         # Groups ('a-e' or 'f-j') aren't in the data
         # so I am randomly generating them here
         Group = sample(c('a-e', 'f-j'), n(), replace = TRUE)) %>%
  # I assume you want one (overall) value per year, age, and group ('a-e' or 'f-j')
  group_by(Year, Age, Group) %>%
  # So we get the sum of number
  summarise(Number = sum(Number)) %>%
  # And make the plot
  ggplot(aes(x = Year, y = Number, fill = Age, group = Group)) +
  geom_bar(stat = 'identity', position = 'dodge', width = 0.9)

Comment: This is very close to what I want. But I want to have a side bar graph of  a-e next to f-j.
Comment: 'a-e' and 'f-j' aren't in your example data, so I made them up randomly
Comment: I figured it out.

Stacked Bar Graph In R


retrieve child thread output in the mid of child thread execution

Question: I am writing a perl script using threads and threads::shared modules where parent process can retrieve the result of child thread in the mid of child execution.
For eg. i need to get the output of child thread when flag=1 in the following script
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;

my $flag : shared = 0;
my $num : shared = 0;

my $thr = new threads(\&sub1);

my @res = $thr->join();

print "@res" if ($flag != 0);

sub sub1
{

  for( $num=0;$num<1000;$num++)
  {
    print "$num\t";
  }

  print "\n";
  $flag = 1;
  sleep(5);

  for( $num=50;$num<100;$num++)`enter code here`
  {
    print "$num\t";
  }

  print "\n";
}

Answer: You want to wait inside and outside the thread for the flag to be set/unset. See the comments in the code:
#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;

my $flag : shared = 0;
my $num : shared = 0;

my $thr = new threads(\&sub1);

# do some things here

#wait for thread to set the flag
while( $flag == 0) {}
print "Outside thread: num=$num\n";
#unset flag so thread can continue
$flag = 0;

# do some things here

#block until thread finishes
my @res = $thr->join();

print "Outside thread: num=$num\n";
print @res;

sub sub1
{
  #do initial calculations in thread
  for( $num=0;$num<1000;$num++)
  {
    print "$num\t";
  }

  print "\n";
  #set flag to parent can read value
  $flag = 1;
  #wait for parent to unset the flag
  while($flag==1) {}

  #continue with rest of thread
  for( $num=50;$num<100;$num++)
  {
    print "$num\t";
  }

  print "\n";
  return "I'm done\n";
}


Moonshine Apache Royal: bad interpreter: Operation not permitted

Question: Getting an error when trying to build a new project, right after it was generated.
Here is what I bee trying so far:

changing mxmlc permission: chmod +x
removing mac sandbox: cd apache-royale-0.9.6-bin-js-swf and then xattr -cr .

Error output:
: SDK path: /Users/goku/apache-royale-0.9.6-bin-js-swf/royale-asjs
: Sending to mxmlc: export ROYALE_HOME=/Users/goku/apache-royale-0.9.6-bin-js-swf/royale-asjs;export SETUP_SH_VMARGS="-Duser.language=en -Duser.region=en"; export ROYALE_SWF_COMPILER_HOME=/Users/goku/apache-royale-0.9.6-bin-js-swf/royale-asjs&& "/Users/goku/apache-royale-0.9.6-bin-js-swf/royale-asjs/js/bin/mxmlc" -load-config+=obj/CodeSearchConfig.xml -optimize=false -debug=true -o bin-debug/CodeSearch.swf -compiler.targets=SWF
: /bin/bash: /Users/goku/apache-royale-0.9.6-bin-js-swf/royale-asjs/js/bin/mxmlc: /bin/sh: bad interpreter: Operation not permitted

Answer: The answer is Downloads!
Moonshine created a cool tool to help people just for this purpose: Moonshine app store helper also called Moonshine SDK installer. It is basically puts all the needed SDKs in your /users/you/Downloads folder, where Macs quarantine restriction is not applied.

BTW, unrelated to this answer, I ended up using feathers, and Royal, as it offer a much reacher components support.


Kendoui Grid Custom Aggregate Group Header

Question: I have a purchase grid with expiration date and I need to sum only the valid values, that is, those that have not expired. The image shows what is displayed today (in red) and how it should look (in blue). Today is 2017-07-07.
Answer: As far I know there is no stadard kendo way to build conditional sum aggregate.
However - you can use footer template to obtain similiar functionality.

Example:
Column definition part:
columns: [
          "ColumnTitle",
          { field: "Field", title: "Title", footerTemplate: conditionalSum },
        ]

Function:
function conditionalSum() {
        var data = yourGrid.dataSource.data();
        var sum = 0;
        for (var idx = 0; idx < data.length; idx++) {
          //do your calculations
        }

        return kendo.toString(sum);
      }


Is there a function with the property $f(n)=f^{(n)}(0)$?

Question: Is there a not identically zero, real-analytic function $f:\mathbb{R}\rightarrow\mathbb{R}$, which satisfies
$$f(n)=f^{(n)}(0),\quad n\in\mathbb{N} \text{ or } \mathbb N^+?$$
What I got so far:
Set
$$f(x)=\sum_{n=0}^\infty\frac{a_n}{n!}x^n,$$
then for $n=0$ this works anyway and else we have
$$a_n=f^{(n)}(0)=f(n)=\sum_{k=0}^\infty\frac{a_k}{k!}n^k.$$
Now $a_1=\sum_{k=0}^\infty\frac{a_k}{k!}1^k=a_0+a_1+\sum_{k=2}^\infty\frac{a_k}{k!},$ so
$$\sum_{k=2}^\infty\frac{a_k}{k!}=-a_0.$$
For $n=2$ we find
$$a_2=\sum_{k=0}^\infty\frac{a_k}{k!}2^k=a_0+a_1+2a_2+\sum_{k=3}^\infty\frac{a_k}{k!}2^k.$$
The first case was somehow special since $a_1$ cancelled out, but now I have to juggle around with more and more stuff.
I could express $a_1$ in terms of the higher $a's$, and then for $n=3$ search for $a_2$ and so on. I didn't get far, however. Is there a closed expression? My plan was to argue somehow, that if I find such an algorythm to express $a$'s in terms of higher $a$'s, that then, in the limit, the series of remaining sum or sums would go to $0$ and I'd eventually find my function.
Or maybe there is a better approach to such a problem.
Comment: Perhaps you might want to replace the fuzzy "no tricks" condition with "real analytic", then?
Comment: @daniel: To "define" something, you normally don't have it on both sides of the equation.
Comment: What do you mean by non-trivial? Just not identically $0$?  And why don't you want to use mollifiers tricks? We can solve the problem with Borel's theorem, but it uses the previous tricks.
Comment: I am sure that this is not the answer that you are looking for, but doesn't f(x):=1 count? :p
Comment: g24l : you misunderstand the $f^{(n)}$ notation, I think - it refers to the $n$th derivative of $f$, which would be identically zero for the function you describe (and thus not equal to $f(n)$ for any $n$ other than $0$).
Comment: For what it's worth, $f(x)=\sin\left(\frac{\pi}{2}x\right)$ satisfies $f(n)=\left(\frac{2}{\pi}\right)^nf^{(n)}(0)$.
Comment: Naively, what is wrong with f(x) = f(0) + f(1)x + (1/2)f(2)x^2 + (1/6)f(3)x^3...? If the question is whether there is such a function, do we have to exhibit it explicitly?
Comment: By non-trivial I ment not equal 0, yes. And the question really became interesting for my after I translated it to this $a_n$ recursive series problem. Of course, since I only state the problem in terms of a set which is not dense in $\mathbb{R}$, I could detach the neightborhood around $x=0$ for $f(0)$ and it's derivative from the points $x=n>0$ for $f(n)$ via a mollifier, but that's somewhat beside the point.
Comment: Okay you're right, Henning & Davide, I stated that clearer in the original post now.
Comment: Could Taylor series help in this case?
Comment: Note that real entire functions need not have everywhere converging Taylor series.  E.g. $\frac{1}{1+x^2}$.
Answer: Let complex number $c$ be a solution of $e^c=c$.  For example $c = -W(-1)$, where $W$ is the Lambert W function.  Then since function $f$ defined by
$$
f(x) = \sum_{n=0}^\infty \frac{e^{cn}x^n}{n!}
$$
evaluates to $e^{e^c x} = e^{cx}$, we have $f(n) = e^{cn} = f^{(n)}(0)$.  For a real solution, let $c = a+bi$ be real and imaginary parts and let $g(x)$ be the real part of $f(x)$.   More explicitly:
$$
g(x) = \sum_{n=0}^\infty \frac{e^{an}\cos(bn)\;x^n}{n!}
$$
evaluates to $e^{ax}\cos(bx)$.  With the principal branch of Lambert W, this is approximately:
$$
g(x) = e ^{0.3181315052 x} \operatorname{cos} (1.337235701 x)
$$
Comment: @GEdgar: Thanks for the solution, looks very good. [I checked the first few cases of your approximation](http://img710.imageshack.us/img710/2629/solutionb.png) and it gives nice results. Would you want to elaborate, what made you think about this problem like this, using the $exp$ function etc.?
Comment: Note that there are infinitely many solutions of $e^c = c$ (LambertW has infinitely many branches), so the space of solutions is indeed infinite-dimensional.
Comment: would the solution be unique, up to branch? or not necessarily?
Comment: I see no reason to suppose it is unique.  If $f$ is a solution, then so is $\lambda f$ for any constant $\lambda$.  Indeed, the problem is linear, so the set of solutions is a linear space.  Probably infinite-dimensional...
Comment: Also, by "the real part of the complex function", you also mean the restriction of this real part to real arguments, right?
Comment: This is so cool!
Comment: @Robert: But, of course, we can still ask if there are other solutions.  Say we stick to series with infinite radius of convergence...
Comment: **what made you think about this problem like this** First was daniel's "Naively" comment.  Having $f$ on both sides does not make it a definition.  But sometimes when you have your unknown on both sides of an equation, you can use it to iterate, and (hopefully) converge to a solution.  So I tried the series with $a_n=1$ and got $e^x$, then plugged in $a_n = e^n$ and got $e^{bx}$ for some $b$.  So I could then try to solve for a particular $c$ such that, when I plug in $a_n = e^{cn}$ the result will be $e^{cx}$.  These were all non-real, so I made the "real part" fix you see.
Comment: I see. I was thinking about your solution and noticed that since we evaluate the derivatives at $x_0=0$ and then $e^{Ax_0}\cos{(Bx_0)}=1$ and $\cos'{(x_0)}=0$, the values $f(n)$ are actually just the constant $A=0.318...$ to the power of $n$. Strange. Maybe that can be generalized and used to find other solutions.
Answer: (not yet an answer, but too long for a comment)
upps, I see there was a better answer of G.Edgar crossing. Possibly I'll delete this comment soon
For me this looks like an eigenvalue-problem.
Let's use the following matrix and vector-notations.
The coefficients of the power series of the sought function f(x) are in a columnvector A.
We denote a "vandermonde"-rowvector V(x) which contains the consecutive powers of x, such that $\small V(x) \cdot A = f(x) $
We denote the diagonal vector of consecutive factorials as F and its reciprocal as f
Then we denote the matrix which is a collection of all V(n) of consecutive n as ZV    .
Then we have first
$\qquad \small ZV \cdot A = F \cdot A  $
and rearranging the factorials
$\qquad \small  f \cdot ZV \cdot A = A  $
which is an eigenvalue-problem to eigenvalue $\small \lambda=1 $ . Thus we have the formal problem of solving
$\qquad \small  \left(f \cdot ZV - I \right) \cdot A = 0  $
However, at the moment I do not see how to move to the next step...

[added] The solution of G. Edgar gives the needed hint
If we do not rearrange, but expand:
$\qquad \small \begin{eqnarray}
   ZV \cdot A &=& F \cdot A  \\
   ZV \cdot (f \cdot F)\cdot A &=& F \cdot A  \\
   (ZV \cdot f) \cdot (F \cdot A) &=& (F \cdot A) \end{eqnarray} $
we get a better ansatz. Let's denote simpler matrix constants $\small W=ZV \cdot f \qquad B=F \cdot A $ and rewrite this as
$\qquad \small W \cdot B = B $.
Now W is the carleman-matrix which maps $\small x \to \exp(x) $ by
$\qquad \small W \cdot V(x) = V(\exp(x)) $
Thus if $\small x = \exp(x) $ for some x , so x is some (complex) fixpoint $\small t_k$  of $\small f(x)=\exp(x)$ and we have one possible sought identity:
$\qquad \small W \cdot V(t_k) = V(t_k) \to B = V(t_k) \to A = f \cdot B = f \cdot V(t_k) $
Then the coefficients of the power series are
$\qquad \small f(x) = 1 + t_k x + t_k^2/2! x^2 + t_k^3/3! x^3 + \ldots $
and $\small f(x) = \exp(t_k \ x) $
Because there are infinitely many such fixpoints (all are complex) we have infinitely many solutions of this type (there might be other types, the vectors $\small V(t_k) $ need not be the only type of possible eigenvectors of W )
Answer: I thought the problem was interesting and there seem to be a lot of guessing to come up with a solution. So I would just like to try to supplement with my own dubious and handwavy approach separate from GEdgar to motivate the solution, that might be more direct.
$$f(n) = f^{(n)}(0)$$
I then do a bit of arrangement by rewriting in terms of a translation operator,
$$\left.T^n f(x)\right|_{x=0} = \left.\frac{d^n}{dx^n} f(x)\right|_{x=0}$$
It is sometimes valid to write a translation operator as $T = e^{\frac{d}{dx}}$ and so,
$$\left.(e^{\frac{d}{dx}})^n f(x)\right|_{x=0} = \left.\left(\frac{d}{dx}\right)^n f(x)\right|_{x=0}$$
From here I consider the operator equation:
$$e^D = D$$
Which has the "solution" $D = -W(-1)$
Now I can use this to create an ansatz of the form $f(x) = Ke^{-W(-1)x}$ which we can then check in a fairly straightforward manner by simply plugging into the original
$f(n) = f^{(n)}(0)$ and seeing that it does work out.
Comment: I really like the translation operator approach. I'd cut it off at the last foruma line which amounts to $e^D-D=0$ and here study this in some function space context and see on which functions it could hold. Maybe something of spectral theory sorts.


jQuery background-attachment in Chrome

Question: Helping a friend working on the early beta of his site, I wrote a short jQuery snippet to keep a fixed background image from scrolling past the bottom edge of a div. Everything works perfectly in Firefox and Safari and even IE8, but Chrome (found on both Win7 and OSX 10.6) seems to have a problem once the background-attachment is changed from scroll back to fixed, where the image doesn't scroll with the page back up, and chunks of it get placed repeatedly on scrolling back down (hard to explain, check the demo link below). Is this something anyone else has encountered at all?
The javascript:
<code>$(document).ready(function(){
var imageHeight = 585;
$(window).scroll(function(){
    if($(window).scrollTop() + $(window).height() > ($("div.main").height() + $("div.main").offset().top)) {
        $("div.main").css("background-position", "0px " + ($("div.main").height() - imageHeight) + "px");
    $("div.main").css("background-attachment", "scroll");
    }else{
        $("div.main").css("background-position", "right bottom");
        $("div.main").css("background-attachment", "fixed");
    }
});
</code>
});
Live demo:
http://mydigitalworld.dyndns.info/dns.html
Comment: Clarification, this is what happens in Chrome/Win7: http://grab.by/aop1
Comment: @David Yep, same happens on latest nightly Chromium.
Comment: @David, exactly what I'm seeing. Same in Mac and Ubuntu.
Answer: You might want to use background-repeat:no-repeat.
Having said that, I'm not clear what you're trying to achieve and why that couldn't be done with CSS alone.
Comment: We want to have the red gradient background image scroll with the page, unless it's past the end of its div, in which case the image will not pass the bottom edge of the div. I can't think of any way to do exactly that in purely CSS.
Comment: Unfortunately background-repeat makes no difference.

jQuery background-attachment in Chrome


Saving user specific customization settings

Question: My module prints entries from the database,
I use Pager and ->limit to choose the number of records to show per each page,
instead of hardcoding limit, I'd like user to decide, between e.g. 10,20,50,100 entries per page.
I've got form with select, and the question is - how I can save and pass user defined value to ->limit.

use database - it works, but I'd like to avoid creating extra entry to database concerning such a silly setting like number of records per page
variable_set - if I'll set variable e.g. number_of_rows, change in the limit made by one user will affect rest of the users - not acceptable

maybe there's some solution concerning cookies/sessions?
Answer: The best place maybe the drupal user object's data array for tiny amounts of information, like preferences on a per user basis.
<code>$user->data
</code>
I'm trying to solve this myself but these links seems useful:

drupal 6
drupal 7
http://www.braahm.be/node/7
http://randyfay.com/node/85
Answer: The <code>$_SESSION</code> variable is definitely the right place to store this data, e.g. in your form's submit handler you would do something as simple as this:
<code>$_SESSION['MYMODULE_num_per_page'] = $form_state['values']['num_per_page'];
</code>
And when you're actually defining the form element:
<code>$default = 10;
$current = isset($_SESSION['MYMODULE_num_per_page']) ? $_SESSION['MYMODULE_num_per_page'] : $default;

$form['num_per_page'] = array(
  '#type' => 'select',
  '#title' => 'Number per page',
  '#options' => drupal_map_assoc(array(10, 20, 50, 100)),
  '#default_value' => $current
);

// Build up the query and add $current to the ->limit() call on the query object
</code>


Get all URLs for resources in sub-directory in Swift

Question: I am attempting to create an array of URLs for all of the resources in a sub-directory in my iOS app. I can not seem to get to the correct path, I want to be able to retrieve the URLs even if I do not know the names (i.e. I don't want to hard code the file names into the code).
Below is a screen shot of the hierarchy, I am attempting to get all the files in the 'test' folder:

Any help is greatly appreciated, I have attempted to use file manager and bundle main path but to no joy so far.
This is the only code I have currently:
let fileManager = FileManager.default
let path = Bundle.main.urls(forResourcesWithExtension: "pdf", subdirectory: "Files/test")

print(path)

I have also tried this code but this prints all resources, I can't seem to specify a sub-directory:
let fm = FileManager.default
let path = Bundle.main.resourcePath!

do {
    let items = try fm.contentsOfDirectory(atPath: path)

    for item in items {
        print("Found \(item)")
    }
} catch {
    // failed to read directory – bad permissions, perhaps?
}

Based on an answer from @vadian , The folders were changed from virtual groups to real folders. Using the following code I was able to get a list of resources:
let fileManager = FileManager.default
    let path = Bundle.main.resourcePath

    let enumerator:FileManager.DirectoryEnumerator = fileManager.enumerator(atPath: "\(path!)/Files/test")!
    while let element = enumerator.nextObject() as? String {
        if element.hasSuffix("pdf") || element.hasSuffix("jpg") { // checks the extension
            print(element)
        }
    }

Comment: Please show the code you've written so far
Comment: @AshleyMills Updated question, don't have any code really as I have tried things and deleted if I had no luck
Answer: Consider that the yellow folders  are virtual groups, not real folders (although Xcode creates real folders in the project directory). All files in the yellow folders are moved into the Resources directory in the bundle when the app is built.
Real folders in the bundle are  in the project navigator.
Comment: Thanks for that, I actually did not know that. It has helped me build a possible answer. Will edit my question to let others know.
Answer: You can follow the following steps to get them:

Create a new folder inside your project folder with the extension is .bundle (for example: Images.bundle).
Copy resource files into that new folder.
Drag that new folder into the project that opening in Xcode.
Retrieve the URLs by using the following code snippet:
let urls = Bundle.main.urls(forResourcesWithExtension: nil, subdirectory: "Images.bundle")

You can also view the guide video here: https://youtu.be/SpMaZp0ReEo
Comment: Damn, this is elegant solution.
Answer: I came across a similar issue today. I needed to retrieve the URL of a resource file in a bundle ignoring its path.
I wrote the following:
public extension Bundle {
/// Returns the file URL for the resource identified by the specified name, searching all bundle resources.
/// - Parameter resource: The name of the resource file, including the extension.
/// - Returns: The file URL for the resource file or nil if the file could not be located.
func recurseUrl(forResource resource: String) -> URL? {
    let contents = FileManager.default.allContentsOfDirectory(atPath: self.bundlePath)
    for content in contents {
        let fileNameWithPath = NSString(string: content)
        if let fileName = fileNameWithPath.pathComponents.last {
            if resource == fileName {
                return URL(fileURLWithPath: content)
            }
        }
    }
    return nil
}

Based on this:
public extension FileManager {
/// Performs a deep search of the specified directory and returns the paths of any contained items.
/// - Parameter path: The path to the directory whose contents you want to enumerate.
/// - Returns: An array of String objects, each of which identifies a file or symbolic link contained in path. Returns an empty array if the directory does not exists or has no contents.
func allContentsOfDirectory(atPath path: String) -> [String] {
    var paths = [String]()
    do {
        let url = URL(fileURLWithPath: path)
        let contents = try FileManager.default.contentsOfDirectory(atPath: path)

        for content in contents {
            let contentUrl = url.appendingPathComponent(content)
            if contentUrl.hasDirectoryPath {
                paths.append(contentsOf: allContentsOfDirectory(atPath: contentUrl.path))
            }
            else {
                paths.append(contentUrl.path)
            }
        }
    }
    catch {}
    return paths
}

}
Which achieves the goal of retrieving the URL for the first match of a given resource filename in a bundle's resources, all directories wide.
I tend to think that Swift's func url(forResource name: String?, withExtension ext: String?) -> URL? should behave this way in the first place.


REST user authentication

Question: OK... the basic idea is to have SERVER and CLIENT physically separated (two systems).
My idea is to build a stand-alone web service (REST, XML, API-KEY) that will provide

Authentication: User login, logout
Data: Get list of products

Then I will create clients in different languages (Flash, PHP, JavaScript). Data will be served only to authenticated users.
Tipical communication for user to get list of products will be:

(1 request) Login / start session
(1 request) Get list of products
(1 request) Get list of products
...

OK... Now the problem I have is the user session. Say we want to build Javascript client, we actually have to create PHP client that will communicate with REST (PHP knows about REST API-KEY) and will forward info to Javascript (CLIENT) right? User will login through PHP to REST server right and then request data through PHP to REST server?
Questions:

Now how does PHP store info about opened user session on REST server?
If my idea is bad, what is the right way of implementation?
Alternatives?
Comment: I'm not sure about one thing. Once user is authenticated on the client side and you make calls to other services like get the list of products. How that service will know that the user was authenticated?
Comment: (1) Send the user credentials with *every* request, via HTTPS. (2) Use an existing, secure and convenient authentication library. For example, take a look at https://github.com/delight-im/PHP-Auth which is both framework-agnostic and database-agnostic. (3) Log in. (4) Do the actual work on your server. (5) Log out again.
Answer: A RESTful interface does not store any information about a particular user's session.  It is the client's job to maintain the information about what it is doing.
Authenticate the user on every request by providing information in the Authorization HTTP header. IF this becomes a performance problem, then look at alternative solutions to optimize perf.
Answer: To your first question: XmlHttpRequest requests to a service will still pass along cookies, which can be used to propagate a session ID. You can even (assuming the enduser's browser supports it) mark cookies as 'HttpOnly' to reduce your XSS footprint. See Jeff Atwood's article for some detail on that.
Comment: Cookies kind of break the "STATELESS" part of REST.
Comment: You can still store state on the client and be RESTful. The restriction is that application state shouldn't be stored on the server.
Comment: In these discussions, it is important to distinguish __application state__ (session state) from __resource state__. In a RESTful architecture, resource state is stored server-side, whereas application state is stored client-side. [http://www.infoq.com/articles/mark-baker-hypermedia]
Answer: You should probably use HTTP authentication for the user auth, and so not need to do any sort of session management.
Answer: You don't need PHP to store an API-KEY if you make your client classes in javascript smart enough to append the API-KEY (loaded when logging in) into the headers of each XmlHttpRequest your class will spawn.
Also it might be good to note that API-KEY does not necessary mean authentication key.
Answer: I'm facing the same problem. I used restserver in php.
Ofcourse the traffic goes over SSL connection.
When I would like to get information about certain user I must authenticate first to the rest server before I can get his info. I would like to know any better approaches?
Similar post: RESTful Authentication
Good resource is also OAuth2.
Also Google uses oauth:

OAuth 2.0 is a new, simplified authorization protocol for all Google APIs. OAuth 2.0 relies on SSL for security instead of requiring your application to do cryptographic signing directly. This protocol allows your application to request access to data associated with a user's Google Account.

When application uses this: http://restserver/user/login and let's say that the authentication went ok, the application itself creates session like this:
Rest client/Application
<code>    public function login() {

    .... form_validation

    // get restserver salt so we can send hashed password
    $message = $this->rest->get('https://restserver/user/salt');

    if($message['status_code'] !== '0')
      exit;

    $data = array(
      'username'  => $this->input->post('username'),
      'password'   => prepare_password_salt($this->input->post('password'), $message['salt'])
    );

    // authenticate with restserver, check if the user and password exist
    $msg = $this->rest->post('https://restserver/user/login', $data);

    if($msg['status_code'] === '0')
    {
      // make session
      $session_data = array(
        'logged_in'     => true,
        'username'    =>  $data['username']
      );

      $this->session->set_userdata($session_data);
      redirect(base_url() . 'some_page');
</code>
Rest Server
<code>/**
 * http://restserver/user POST
 * - insert new user
 *
 * http://restserver/user/id PUT
 * - update existing user
 *
 * http://restserver/user/login POST
 * - check if username exists and the password match
 *
 * - return true on success login
 * - return false on failure
 *
 * http://restserver/user/id/hashed_pass GET
 * again client gets salt and then send hashed_pass
 * - return array(
 *   'username' ..
 *   'email' ....
 *   .. other information
 * );
 * or we could use some access token but that means that the restserver
 * must save token for each user for some time and then delete
 *
 * HTTP server Allowed methods: GET, POST, PUT
 * HTTP server disallowed methods: DELETE
**/

class User extends Rest_Controller
{
  public function __construct()
  {
    parent::__construct();
    $this->load->library('form_validation');
  }

  /**
   * Check username and password with database.
   *
   * @link /
   * @param $_POST['username'] (string)
   * @param $_POST['password'] (string - sha1(password . salt))
   * @return array which contains
   *                array['status_code']
   *                array['status']
   *                status_code 0 means that login was successful
   *                status_code 1 means that username or password is incorrect
   */
   public function login_post()
   {
     $this->load->model('User_Model', 'user_model');

     $this->form_validation->set_rules('username', 'Username', 'trim|required|min_length[3]|max_length[64]');
     $this->form_validation->set_rules('password', 'Password', 'trim|required|exact_length[40]');

     if($this->form_validation->run() === true)
     {
       // check with the database
       if($this->user_model->authenticate($this->input->post('username'), $this->input->post('password')))
       {
         // update user last_login field
         $this->user_model->updateLogin($this->input->post('username'));

         $message = array(
           'status_code'    => '0',
           'status'     => 'Login ok.'
         );
       }
       else
       {
         $message = array(
           'status_code'    => '1',
           'status'     => 'Username or password is incorrect.'
         );
       }
     }
     $this->response($message, 200);
   }
</code>

REST user authentication


php Rotating image and resaving

Question: So I have an image uploaded what I'm trying to do is have the user rotate it, receive that value and rotate the image and replace the original with the new image.
So I have a button:
Rotate

In my js file:
$('.rotate').on('click', function(e){
   e.preventDefault();
   if(rotate < 360)
    rotate = rotate +90;
   else
    rotate = 0;

   var id = $(this).data('id');

   $('.img-container img').attr('style', '-webkit-transform: rotate(' + rotate + 'deg)');

   $.ajax({
       type: "POST",
       url: "functions/post",
       data: { rotate: rotate, id: id }
   });

});
and inside "functions/post":
$degrees = $_POST['rotate'];
$postid = $_POST['postid'];

//user the postid to get the imagefilepath from db
$filename = '../img/uploads/' . $post->getimagepath();

// Content type
header('Content-type: image/jpeg');

// Load
$source = imagecreatefromjpeg($filename);

// Rotate
$rotate = imagerotate($source, $degrees, 0);

// Output
imagejpeg($rotate);

list($width, $height) = getimagesize($rotate);
$newImage = imagecreatetruecolor($width, $height);
imagecopyresampled($newImage, $source, 0, 0, 0, 0, $width, $height, imagesx($source), imagesy($source));
imagejpeg($newImage, realpath('../img/uploads/' . $post->getimagepath()));
// Free the memory
imagedestroy($source);
imagedestroy($rotate);
imagedestroy($newImage);

but it doesn't save the rotated file.. :(
this code i got it from searching through so and google but can't seem to make it work for me yet.  None of this code is set in stone so anything that needs changing aslong as it makes it work then its all good and greatly appreciated.
Comment: Have you checked `$post->getimagepath()`?
Comment: Also, why are you setting the header to `image/jpeg`?
Comment: yes the $post->getimagepath returns the filename and the header idk I thought I had to..
Answer: You're not saving the rotated image, you're saving the original image over again.
Try this:
$degrees = $_POST['rotate'];
$postid = $_POST['postid'];

//user the postid to get the imagefilepath from db
$filename = '../img/uploads/' . $post->getimagepath();

// Content type
header('Content-type: image/jpeg');

// Load
$source = imagecreatefromjpeg($filename);

// Rotate
$rotate = imagerotate($source, $degrees, 0);

imagejpeg($rotate, realpath('../img/uploads/' . $post->getimagepath()));
// Free the memory
imagedestroy($source);
imagedestroy($rotate);

(not tested) but you get the idea.
Comment: yea I just saw that too, but for some reason it still didn't save it.
Comment: Like I said, it's untested. Play around with it ;) You're probably close.
Comment: it worked but i had to use $_SERVER['DOCUMENT_ROOT'] otherwise it wasn't finding it.  Thanks!

php Rotating image and resaving


PyQt5 - Definir um InputMask em um QLineEdit para digitar apenas a tecla "F10"

Question: Tenho uma aplicação que contém um QLineEdit que ao estar ativo o usuário tem a opção de pressionar a tecla "F10" e assim abrir outra janela.
Porém gostaria de bloquear esse QLineEdit para edição, deixando apenas disponível a tecla "F10"
Com a propriedade inputMask consigo fazer isso?
from sys import argv, exit
from PyQt5.QtCore import Qt
from PyQt5.QtWidgets import QApplication, QLineEdit, QVBoxLayout, QWidget, QShortcut, QLabel

class Tela_1(QWidget):
    def __init__(self):
        super(Tela_1, self).__init__()
        self.setWindowTitle("Tela 1 - TESTE")
        self.setGeometry(50, 50, 400, 400)

        # ADD WIDGETS
        self.lineEdit = QLineEdit()

        # ADD LAYOUT
        mainLayout = QVBoxLayout()
        mainLayout.addWidget(self.lineEdit)
        self.setLayout(mainLayout)

        # AO PRESSIONAR A TECLA "F10" CONECTAR COM A FUNÇÃO "abrir_janela2"
        QShortcut(Qt.Key_F10, self.lineEdit,  self.abrir_janela2, context=Qt.WidgetShortcut)

    def abrir_janela2(self):
        self.janela2 = Tela_2()
        self.janela2.show()

class Tela_2(QWidget):
    def __init__(self):
        super(Tela_2, self).__init__()
        self.setWindowTitle("Tela 2 - TESTE")
        self.setGeometry(50, 50, 300, 300)
        self.label = QLabel()
        mainLayout = QVBoxLayout()
        mainLayout.addWidget(self.label)
        self.setLayout(mainLayout)
        self.label.setText("TELA 2")

app = QApplication(argv)
w = Tela_1()
w.show()
exit(app.exec_())

Answer: Basta aplicar setReadyOnly(true)
Assim em PyQt5:
self.lineEdit.setReadOnly(True)

Comment: Deu certo, muito obrigado, achei que o setReadOnly não entendia que o QLineEdit estava ativo quando clicado.


Beta function integral appearing in the $O(n)$-model

Question: While studying the $O(n)$-model I found myself in the need of integrating
$$\frac{\int_0^{\pi}\text{d}\theta\sin(\theta)^{n-2}\cos(\theta)^{2r}}{\int_0^\pi\text{d}\theta\sin(\theta)^{n-2}},$$
where $n\in\mathbb{N}^+$ and $r\in\mathbb{N}$. I imagine however that the author (Brézin, Introduction to Statistical Field Theory) is only considering $n>1$. He claims that this can be done through Beta function integrals. Following his suggestion, I noticed that the integrands are even around $\pi/2$. Then the integral can be reduced to
$$\frac{\int_0^{\pi/2}\text{d}\theta\sin(\theta)^{n-2}\cos(\theta)^{2r}}{\int_0^{\pi/2}\text{d}\theta\sin(\theta)^{n-2}}.$$
We have the beta functions
$$\frac{(x-1)!(y-1)!}{(x+y-1)!}=B(x,y)=2\int_0^{\pi/2}\text{d}\theta\sin(\theta)^{2x-1}\cos(\theta)^{2y-1},$$
for $x,y\in\mathbb{N}^+$. Then the integral I'm looking for is
$$\frac{\left(\frac{n-3}{2}\right)!\left(\frac{2r-1}{2}\right)!\left(\frac{n-2}{2}\right)!}{\left(\frac{n-3}{2}\right)!\left(-\frac{1}{2}\right)!\left(\frac{2r+n}{2}\right)!}$$
This doesn't seem correct though. All I know is that
$$\sum_{r=0}^\infty\frac{x^r}{r!}\frac{\int_0^{\pi}\text{d}\theta\sin(\theta)^{n-2}\cos(\theta)^{r}}{\int_0^\pi\text{d}\theta\sin(\theta)^{n-2}}=1+\sum_{r=1}^\infty\frac{n^r}{2^rr!n(n+2)\cdots(n+2r-2)}x^{2r}.$$
I suspect there is however an error in the book and the correct statement is
$$\sum_{r=0}^\infty\frac{(\sqrt{n}x)^r}{r!}\frac{\int_0^{\pi}\text{d}\theta\sin(\theta)^{n-2}\cos(\theta)^{r}}{\int_0^\pi\text{d}\theta\sin(\theta)^{n-2}}=1+\sum_{r=1}^\infty\frac{n^r}{2^rr!n(n+2)\cdots(n+2r-2)}x^{2r}.$$
This means of course that, the integral I'm looking for equals
$$\frac{\int_0^{\pi}\text{d}\theta\sin(\theta)^{n-2}\cos(\theta)^{2r}}{\int_0^\pi\text{d}\theta\sin(\theta)^{n-2}}=\frac{(2r)!}{2^rr!n(n+2)\cdots(n+2r-2)}$$
Can somebody help me understand this integral?
Answer: $$\begin{align}
\frac{\left(\frac{n-3}{2}\right)!\left(\frac{2r-1}{2}\right)!\left(\frac{n-2}{2}\right)!}{\left(\frac{n-3}{2}\right)!\left(-\frac{1}{2}\right)!\left(\frac{2r+n}{2}\right)!}
&=\frac{\left(\frac{2r-1}{2}\right)!\left(\frac{n-2}{2}\right)!}{\left(-\frac{1}{2}\right)!\left(\frac{2r+n}{2}\right)!}\\
&=\frac{\left(r-\frac{1}{2}\right)!\left(\frac{n}{2}-1\right)!}{\left(-\frac{1}{2}\right)!\left(\frac{n}{2}+r\right)!}\\
&=\frac{\overbrace{(r-\frac12)(r-\frac32)\dots(\frac32)(\frac12)\left(-\frac{1}{2}\right)!}^{(r-\frac12)!}\left(\frac{n}{2}-1\right)!}{\left(-\frac{1}{2}\right)!\underbrace{(\frac{n}2+r)(\frac{n}2+r-1)\dots(\frac{n}2+1)(\frac{n}2)\left(\frac{n}{2}-1\right)!}}_{(\frac{n}2+r)!}\\
&=\frac{(r-\frac12)(r-\frac32)\dots(\frac32)(\frac12)}{(\frac{n}2+r)(\frac{n}2+r-1)\dots(\frac{n}2+1)(\frac{n}2)}\\
&=\frac{(2r-1)(2r-3)\dots(3)(1)}{(n+2r)(n+2r-2)\dots(n+2)(n)}\\
&=\frac{(2r)!}{(2r)(2r-2)\dots(4)(2)(n+2r)(n+2r-2)\dots(n+2)(n)}\\
&=\frac{(2r)!}{2^rr!(n+2r)(n+2r-2)\dots(n+2)(n)}\\
\end{align}$$
Which is the answer required, so you were not wrong.
Comment: Thanks! I guess I just got scared by the non integer factorials


GAM - parametric coefficients - what is mgcViz::pterm() actually plotting?

Question: I built a GAM with two categorical variables and two smooth terms, following this structure:
model <- ik ~ population_id_cat + s_status + s(n_locs, bs = "re") + s(animal_id, bs = "re")

The model summary for the parametric coefficients is:
Parametric coefficients:
                                Estimate Std. Error t value Pr(>|t|)
(Intercept)                      1.90509    0.07798  24.432  < 2e-16 ***
population_id_catB              -1.08790    0.13199  -8.242 2.27e-16 ***
population_id_catC               0.06718    0.08880   0.757 0.449368
population_id_catD              -0.27599    0.07689  -3.589 0.000336 ***
population_id_catE              -0.33914    0.13555  -2.502 0.012391 *
population_id_catF              -0.71104    0.07586  -9.373  < 2e-16 ***
population_id_catG              -0.39243    0.07276  -5.393 7.32e-08 ***
population_id_catH               0.31115    0.18431   1.688 0.091449 .
population_id_catI               0.06530    0.10203   0.640 0.522174
s_status_2a_m                   -0.16331    0.05263  -3.103 0.001930 **
s_status_2fam                   -0.36656    0.05200  -7.050 2.10e-12 ***

When I plot these coefficients with the mgcViz::pterm() function, the values do not seem to match the std. error (but their significance does match with the p-value provided).
For example, the plot for the parametric coefficients mentioned above:
gviz_model <- getViz(model)
gviz_model_1 <- pterm(gviz_model, 1)
plot(gviz_model_1) + l_ciBar(colour = "blue") + l_fitPoints(colour = "red") + l_rug(alpha = 0.3)

Question 1:
What values are actually plotted in this plot? They seem to match the "Estimate" value from the summary, but they do not match the "Std. Error" (e.g., look at population H, according to the std error, the o value would be significant and therefore would be above 0).
If I plot the Estimate+-Std. Error in a ggplot (which is my goal, because I want to plot these parametric values for two models in the same plot), I get this instead:
gviz_model_pop <- termplot(gviz_model, se = TRUE, plot = FALSE)$population_id_cat
ggplot(gviz_model_pop , aes(x=x, y=y)) +
     geom_errorbar(aes(ymin=y-se, ymax=y+se), width=.15, size=1, position=position_dodge(0.2))  +
     geom_point(position=position_dodge(0.2), size=3.5, shape=16)

Similar trends, but the interval is definitely not the same (and influences the significance of each category).
Question 2:
If I want to plot these parametric coefficients for two models in the same plot, and I would need to use the output from pterm(), instead of extracting the estimates/std error as I did previously, how can I do this? Preferably with a ggplot output.
Question 3:
If I want to plot these parametric coefficients including the intercept (to make the plot more interpretable), as mentioned here in "Transformed standard errors (2)", how do I do it? Do I need separate calculations or is there a function for the parametric coefficients too? Preferably with a ggplot output.
Thank you very much in advance for any help!
Answer: Q1
The blue bars are confidence intervals formed (likely) as +/- 2 * Std. err
Q2
I think this would be easier with parametric_effects() from my {gratia} package; what pterm() seems to be doing is wrapping up the model object with some extra information, which doesn't seem to be the information computed for the plot.
Using the example from ?pterm, we have
library("gratia")

set.seed(3)
dat <- gamSim(1,n=1500,dist="normal",scale=20)
dat$fac <- as.factor( sample(c("A1", "A2", "A3"), nrow(dat), replace = TRUE) )
dat$logi <- as.logical( sample(c(TRUE, FALSE), nrow(dat), replace = TRUE) )
bs <- "cr"; k <- 12
b <- gam(y ~ x0 + x1 + I(x1^2) + s(x2,bs=bs,k=k) + fac + x3:fac + I(x1*x2) + logi,
         data=dat)

p <- parametric_effects(b)

Which gives
> p <- parametric_effects(b)
Interaction terms are not currently supported.
> p
# A tibble: 6,005 × 6
   term  type       value level partial    se
   <chr> <chr>   <I<dbl>> <fct>   <dbl> <dbl>
 1 x0    numeric    0.168 NA     -0.310 0.309
 2 x0    numeric    0.808 NA     -1.49  1.48
 3 x0    numeric    0.385 NA     -0.711 0.708
 4 x0    numeric    0.328 NA     -0.605 0.602
 5 x0    numeric    0.602 NA     -1.11  1.11
 6 x0    numeric    0.604 NA     -1.12  1.11
 7 x0    numeric    0.125 NA     -0.230 0.229
 8 x0    numeric    0.295 NA     -0.544 0.541
 9 x0    numeric    0.578 NA     -1.07  1.06
10 x0    numeric    0.631 NA     -1.17  1.16
# ℹ 5,995 more rows
# ℹ Use `print(n = ...)` to see more rows

The confidence is easy to add:
p <- p |>
  dplyr::mutate(lower_ci = partial - (1.96 * se), upper_ci = partial + (1.96 * se))

I would also append a model column to indicate which model this is:
p <- p |>
  dplyr::mutate(model = rep("Model 1", nrow(p))

If you repeat that for the other model, to create p2 say, you should be able to bind the two objects together with
p_both <- dplyr::bind_rows(p, p2)

You can also select which terms you want to extract the info for using the terms argument to parametric_effects().
Q3
The best way to do this is to just predict from the model including only the constant term and parametric term of you choice. In the example from ?pterm that I used above, there is a factor variable fac which we'll use to illustrate the point.
# get a data slice where only `fac` varies
# hold all other variables are representative values
ds <- data_slice(b, fac = evenly(fac)) |>
  dplyr::mutate(logi = as.logical(logi)) # <-- work around bug

# use fitted_values() to predict from the model using only the effects
# of `fac` and the constant term
fv <- fitted_values(b, data = ds, terms = c("(Intercept)", "fac"))
fv

There's a bug in data_slice() currently; data_slice() seems to be converting logical variables to numeric. We work around that by coaxing them back to logical. And we only need this to keep predict.gam() happy; none of the other variables but fac will be used in the predictions we computed in fv.
Having done the above fv is:
> fv
# A tibble: 3 × 10
  fac      x0    x1    x2 logi     x3 fitted    se  lower upper
  <fct> <dbl> <dbl> <dbl> <lgl> <dbl>  <dbl> <dbl>  <dbl> <dbl>
1 A1    0.511 0.521 0.493 FALSE 0.476   5.75  2.55  0.753 10.8
2 A2    0.511 0.521 0.493 FALSE 0.476   4.60  2.54 -0.367  9.57
3 A3    0.511 0.521 0.493 FALSE 0.476   2.94  2.63 -2.21   8.09

Here, as fac only had three levels, we get the three values needed for the plot. The fitted column is now has the intercept added to each of the parametric effects, hence A1, being the reference level has the same value as the intercept:
> coef(b)[1]
(Intercept)
   5.753834

while the other values in fitted are (Intercept) plus the respective coefficient for the other levels. The lower and upper columns contain the confidence interval as it would be plotted in the output from mgcViz. Note that fitted_values() returns, by default, predicted values on the response scale. If you have a non-Gaussian family, you can choose to keep the values on the link scale using scale = "link" in the call to fitted_values()
Comment: Yes, `fv` contains fitted values from the model on the response scale (given the code shown). The interpretation isn't that these estimated effects are *independent* of the other terms. Technically they are for the reference level of `logi` (FALSE) because that is what the constant term represents `A1` *and* `FALSE`. You would get slightly different answers if you did this for `logi == TRUE`, but importantly (apart from differences in the Std err of the estimated effects) each of the estimates for `A1` is being shifted by the same value (intercept) so relative to one another the effects 1/2
Comment: ...are comparable. The isn't much you can do to avoid this; you either set `logi == FALSE` (which is implied when you use only the intercept) or you set `logi = TRUE` and you include the effect of `logi` in the `terms` argument. There are other things you could do, such as generate estimates given both values of `logi` and average those estimates; you difference those estimates, etc. Doing all that is not something that is easy to do with {gratia} (there are no canned functions to do those steps) but the {emmeans} or {marginaleffects} packages can do this and would be my goto for this 2/2
Comment: Also, the estimates will never be independent of the `logi` effect (or any other estimated effect); the coefficients associated with `fac` (in the example) are those effects *given that `logi` and the other terms are included in the model*. If you dropped `logi` from the model, the estimated effects of `fac` would change.
Comment: With random effects, it depends on what you are trying to do/show. Typically, when you are showing the effects of the, or selected, "fixed" terms (the scare quotes because smooths are random effects too but you wouldn't treat them as being a random drawn), you would exclude the random effects or average over them. The former is easy (just add the random effect to the `exclude` argument, or as I did here just don't include that term in the `terms` argument), the latter not so much when a non-identity link function is being used.
Comment: Regardless, the code shown would provide a value for the random effect(s) when creating `ds` and then ignore the contribution from the random effect term when creating `fv`. I'm not sure what you mean by "change the `logi` parameter", do you mean specify that you want both value `TRUE` and `FALSE` in the data slice/predicted values? If that, you should be able to do `data_slice(b, fac = evenly(fac), logi = c(FALSE, TRUE))` and you can skip the `mutate()` step as the bug doesn't affect that particular call. Now `ds` and hence `fv` will have 6 rows for the combinations of `fac` and `logi`.
Comment: Thank you very much for the explanation, it makes sense! And the code works as well, thanks a lot for your help!
Comment: Thank you very much! This is very helpful. Just a couple more details, regarding question 3, "fv" then is now interpretable directly at the scale of the response variable, right? Including the reference category. And this interpretation is independent of the other categorical variable (in your example,  "logi")?
Comment: this is a great explanation, thank you! Actually, in my case, it would not be a bad idea to show the two effects of "logi" (In my example, social_status_2, which has 3 categories), as this would be another variable to show in a plot similar to the one for the populations. Two more (hopefully last) questions: 1) should I worry about the random effects, or somehow consider them in the "ds"/"fv" objects? 2) in the code you provided, where should I change the "logi" param? I changed it manually before creating the "fv" object but the values for "fitted" etc stayed the same


How can I scroll listitems up and down inside a div using click?

Question: I'm trying to build a simple list of itens and make them scroll inside by clicking on next or prev arrows. I didn't want to use any plugins cause i think that is very simple but i'm not abble to make them scroll 3 itens at time or scroll just the height pixels.
Somebody can help me?
Fiddle Here
Answer: add
.movies-list{display:block;height:620px;overflow:hidden}
and remove
div.section-movies ul {
    overflow: hidden;
    height: 2000px;
}
then add some jQuery
//get li height
var gh = $('.movies-list li').eq(0).height();
//prev
$('.movie-prev').on('click', function() {
    var cs = $('.movies-list').scrollTop();
    if (cs>gh) {
        cs = cs-gh;
    } else {
        cs = 0;
    }
    $('.movies-list').animate({scrollTop: cs});
});
//next
$('.movie-next').on('click', function() {
    var cs = $('.movies-list').scrollTop();
    cs = cs+gh;
    $('.movies-list').animate({scrollTop: cs});
});

something similar to this should work, you may have to tweak the value of gh depending on how everything else is set.
fiddle here: http://jsfiddle.net/filever10/4Gb4x/
Comment: Cool, I've updated the answer and fiddle to work that way.
Comment: hi, i want to scroll but using click on the itens prev and next :)
Answer: Inside
.window-movies {

add:
max-height:400px;
overflow-y:scroll;

but this simple solution will break your footer. I guess you will need js for that.
EDIT:
$('.movie-next').css({
            position: 'fixed',
            bottom: 0
        });

Add this to your js to keep your footer at the bottom.
Comment: max-height:400px; is just an example


Does the kind of wood used in making a violin affect its sound?

Question: Whenever I check on violin-learning-related sites, they usually begin talking about the quality of a violin. The strings, the species of woods...
... the species of wood. Ebony? Maple? Rosewood? I think I understand how important the string quality might be, but I don't really see why should I even care for the type of woods used.
Since everyone keeps talking about it, I wonder: does the kind of wood used in a violin matter in any way?
Answer: The wood type in any stringed instrument matters a great deal, especially on acoustic instruments.  Some parts of the violin contribute more to the overall tone quality than others, but all the parts make a difference.
A stringed instrument is a case study in engineering trade-offs.  After all, how does a violin produce its sound?  To begin with, note that the body of a violin is basically an empty box full of air.  The sound you hear a violin producing is the result of that air vibrating and escaping out the F-holes of the instrument.  How does this happen?  Drawing the bow across a string causes that string to vibrate.  The vibration of the string causes the bridge to vibrate, which in turn causes the top of the violin to vibrate.  The vibration of the top of the violin is passed through the sound post to the back of the violin, and the two parts---the top and the back---now both vibrate.  This causes the air trapped between them to vibrate and get pumped out of the F-holes into the surrounding air, and that vibration is what you hear.
So, you want the wood of the violin to transfer the vibrations of the strings to the box of air as efficiently as possible in order to produce the most sound, and so you want the wood of the instrument to be light and resonant.  Maybe a nice light balsa would do the trick?
But no luck, because of the inherent engineering challenge: While you want the violin to be as light and resonant as possible, it also has to withstand the significant string tension of four strings pulling on it and trying to fold it up.  If the wood is too light and weak, the violin will collapse like a cardboard box, which obviously is no good.  If you build the violin to be too heavy and sturdy, though, it will absorb and reflect the strings' vibrations rather than passing them along to the air.  It's a tough problem to solve.
So to build a violin, you want wood that has all these qualities: it has to be light, resonant, strong, stiff, and sturdy.  These are difficult qualities to find all in one species of wood, but a few kinds have proven to be effective: cedar, spruce, mahogany, walnut, maple, etc.  All of these woods work well for acoustic stringed instruments, but they each behave slightly differently when transferring the strings' vibrations into the body of the instrument.
The point is: The entire acoustic mechanics of a violin (or any stringed instrument) completely depend on the ways in which the various pieces of wood transfer the vibrational energy of the strings into the box of air.  Different kinds of wood resonate differently and thus transfer that energy differently, and as such, the kind and quality of the wood of a violin makes all the difference in its sound.
Comment: @horatio It's true that the soundboard (aka the top of the soundbox) is more important than the back and sides (though they all contribute to the overall sound), but the material--and not just the shape---of the soundboard makes a huge difference in tone quality. Have you ever had the chance to play two guitars of the same make and model but with different soundboard materials? Seagull, for example, offers a cedar vs. spruce option on several of their models, and it makes a big difference.
Comment: I have seen unsourced statements about guitars that 80-90% (however *that* was derived!) of the "sound" of a guitar comes from the soundboard and the materials used for the sound box were less important than the shape. (More simply: the soundboard = "sound," the sound box = volume). *If* this is true, is this commonly thought of as true for violins as well?
Comment: I have, I was more thinking, in terms of the OP question: the materials of the soundboard are very important, but how important are the materials for the soundbox? If you are going to skimp on something for economy...
Comment: @horatio The soundboard is part of the soundbox, right? But I take your point, and as you probably know, many guitar manufacturers offer a model with a solid wood top and laminate back and sides as a cost savings. But these models, while they can sound really good, still can't match the quality of tone of an all-solid wood instrument.
Comment: @horatio The back and sides are definitely less important than the soundboard, at least for the guitar. [Antonio Torres](http://en.wikipedia.org/wiki/Antonio_Torres_Jurado) once built a playable guitar with papier-mâché back and sides, just to prove this point. But that's doesn't mean back and sides can be completely ignored!
Comment: @AlexBasson: My guitar has laminated back and sides, and sounds better than some more expensive all-solid models I have tried in the shop. It all depends on a lot of factors.
Answer: I would also add that the species of woods used is only part of the equation. The quality of each piece of wood makes a huge difference. A builder will go through many pieces of maple or spruce from a lumberyard to select only the logs or planks of wood with the highest quality for making instruments. They will reject the vast majority of the wood as only suitable for furniture or chopsticks (seriously).
I know little about what criteria are used, but I understand that a musical instrument maker will only select and purchase specific pieces of lumber that come from trees where the growth of the wood was extremely straight (not twisted or gnarled) and of a certain density and even pattern of grain, because only these are likely to result in an instrument that resonates well. We call these "tonewoods".
Wood selected for instruments must come from extremely old trees, because only in the oldest trees do the cells in the wood reach the required density to make a resonant instrument. (The age of the tree needed for good tonewood varies with the species.) The best cut wood that is selected is then usually "seasoned", or left to dry in controlled conditions, for many months, sometimes for decades, before it is used to build an instrument. This is so the wood can dry out and achieve a stable consistency suitable for carving into an instrument.
Consequently, good tonewood is rare and expensive. Throughout the past century, the supply of quality tonewoods around the world has become extremely constrained due to over-harvesting, and certain species of wood used for instruments have even been put on endangered species lists and have become practically unavailable for building new musical instruments.
The result of this is that the price of a violin or guitar reflects whether or not it is made from expensive and rare old-growth tonewood-quality spruce and maple, or whether it is made from inferior and cheap spruce and maple. Of course the skill of the people who make the instrument is also extremely important.
Answer: In his answer Alex mentions that "you want the violin to be as light and resonant as possible, it also has to withstand the significant string tension". In the book Materials Selection in Mechanical Design by Michael F. Ashby there is a table showing the relative strength and density of different materials:

This is then tailored to violin use by Dr Claire Barlow in her paper "Materials Selection for Musical Instruments" in the Proceedings of the Institute of Acoustics 19(5) pages 69−78:

(N.B. This is not the exact diagram from the paper, but one supplied by Barlow with the 'solid wood' point correctly plotted at 35GPa, as stated in the text of the paper.)
I encountered this figure in a fascinating introduction to the science and engineering of violin design given locally by Professor Jim Woodhouse from the University of Cambridge's Dynamics & Vibration Research Group. After his fabulous talk he took questions. I asked if carbon fibre will make a better material for violins. He dug out Barlow's diagram and pointed out that the island of materials sitting in the top left quadrant are the best; and they are woods! If the axes measured were the only ones that mattered then the best violins would be made from balsa wood (!) though clearly that material has other problems that prevent its use in violin making. (Jim mentioned that there is someone experimenting with balsa wood violins, Douglas Martin perhaps.)
Looking on Google Scholar at the papers which cite Barlow there are others we may want to read. These two look particularly interesting:

"Wood for sound" by Ulrike G. K. Wegst with this tantalising quote from the abstract: "Using material property charts on which acoustic properties such as the speed of sound, the characteristic impedance, the sound radiation coefficient, and the loss coefficient are plotted against one another for woods we analyze and explain why spruce is the preferred choice for soundboards, why tropical species are favored for xylophone bars and woodwind instruments, why violinists still prefer pernambuco over other species as a bow material, and why hornbeam and birch are used in piano actions"
"Acoustical properties of wood in string instruments soundboards and tuned idiophones: Biological and cultural diversity" by Iris Brémaud whose abstract starts: "The acoustical properties of wood for instruments have mostly been studied on a few archetypal woods in Western musical instruments. The objective of this paper is to extend knowledge on the diversity in wood properties and uses in instruments from different geo-cultural areas. A wide set of data has been collected on vibrational properties of 452 species, through experiments and literature survey."
Comment: Alternative link to  *Acoustical properties of wood in string instruments
soundboards and tuned idiophones: Biological and
cultural diversity* https://hal.science/file/index/docid/808347/filename/Acoustical_properties_wood_Bremaud.pdf
Answer: It's also worth noting that a lot of workmanship goes into producing the balance between strength and pliability of the material.  If you use CNC techniques for producing important string instrument parts, particularly the top, then you are not working with the grain and the strengths of the material.  In addition, cheap wood is less stable.  The result is that such a junk violin has to have its sounding parts milled considerably thicker than a hand-made violin.
Adding to that the basically inferior quality of the used woods, you can expect quite bad resonance, and you also can expect parts to warp over time.  A roommate once had a $30 guitar bought few years ago, and the fretboard was already so warped that there was a range of about a fifth that was not playable at all because it would register with a higher fret than fingered.
Really, you cannot fathom just how bad instruments can be until having tried some of those gimmicks few years after they left factory.  It's possible that they work for a limited time right after production if you don't expect useful sound quality.
The problem of getting proper wood, and the problem of machines not being to able to work with the properties of the wood mean that mass-manufactured "Antonius Stradivarius" knockoff violins from the beginning of the 20th century near the Bohemian woods, if in reasonable shape, are quite competitive.
"Manufactured" meant hand-made, and even when using a ridiculously small amount of time for improving the instrument, the factory workers had a job to lose and improved over time.  It's similar with the mass-manufactured bellows instruments of that time: "Alfred Arnold" Bandoneons were cheaper than regular accordions, and are nowadays desperately sought after and painstakingly restored for Argentinian Tango.


Why does MINRES converge in 3 iterations on matrices of specific form?

Question: The MINRES algorithm for solving $Ax = b$ for symmetric $A$ can be described as follows: The $k$-th iterate of the algorithm is
$$x_k = \arg\min_{K_k(A)} \lVert Ax-b \rVert_2$$
where $K_k(A)=\text{span}\{A^ib\mid i < k\}$ is the $k$-th Krylov subspace of $A$
By this definition, it is clear that it converges to the exact solution in $n$ iterations, if $A$ is an $n\times n$ matrix. By using this solver on matrices with a very specific structure, I noticed that in that case, the solver converges in just 3 iterations. The matrices are of the form
$$
A = \begin{bmatrix}
I_{n-1} & v\\
v^T & 0
\end{bmatrix}
$$
where $v$ is a column vector and and $I_{n-1}$ is the $(n-1) \times (n-1)$ identity matrix. How can this early convergence be explained?
My thoughts
The matrix $A$ can in this case be seen as an Identity matrix plus 2 rank-one corrections:
$$A = I_n +
\begin{bmatrix}
0 & v\\
0 & -\frac{1}{2}
\end{bmatrix} +
\begin{bmatrix}
0 & 0\\
v^T & -\frac{1}{2}
\end{bmatrix}$$
This means that we can exactly invert the matrix using the Sherman-Morrison formula twice. I currently avoid doing this explicitly as it leads to instabilities, while 3 MINRES iterations do not. Maybe MINRES algorithm implicitly exploits the fact that we are just a rank-two matrix away from the identity?
You can verify this behaviour with this example python snipet.
Comment: That's a great description of MINRES. Do you know an equally simple description for GMRES? What about the conjugate gradient method?
Comment: @littleO Yes, GMRES can be defined in exactly the same way, but each iteration is more difficult to compute because $A$ is no longer symmetric (if $A$ is symmetric, then you should use MINRES). The conjugate gradient algorithm can also be expressed in this from: the $k$-th iteration minimizes $\lVert x-A^{-1}b \rVert_A$ over the $k$-th Krylov subspace. This is a bit less natural though, but good for comparison. Not that this minimizer is even easier to compute then the one from MINRES, but we need that $A$ is symmetric positive definite such that the $A$-norm exists.
Comment: Thanks. Do you know a good reference that takes these descriptions as a starting point and then derives / discovers the iterations for each of these methods? The books I've seen tend to state the iterations without motivation, and then prove that certain properties are satisfied. I want to know a thought process by which you could discover the iterations, given the simple descriptions above as a starting point.
Answer: The reason is that there is a large Eigenspace corresponding to the eigenvalue $1$. If you solve the eigenvalue problem
$$
\begin{bmatrix} I_{n-1} & v \\ v^T & 0\end{bmatrix}
\cdot
\begin{bmatrix} x \\ \alpha \end{bmatrix}
= \lambda \begin{bmatrix} x \\ \alpha \end{bmatrix}
$$
you'll find that there are $n-2$ eigenvectors with $\alpha=0$ and $x\perp v$ corresponding to the eigenvalue $1$ and 2 eigenvectors with $\alpha\neq 0$ and $x\| v$. More specifically in the latter case we have the eigenvectors $x_\pm=v$ and $\alpha_\pm = \frac{1}{2}(-1\pm\sqrt{1+4 v^T v})$ corresponding to the eigenvalue $\lambda_\pm = \frac{-1\pm\sqrt{1+4v^T v}}{2v^T v}$. Note that since $A$ is symmetric, those $EV%$ are orthogonal.
Consequently, given a target vector $y=\begin{bmatrix}b\mid\beta\end{bmatrix}$ we can split it up as a linear combination
$$
y =
\underbrace{\mu_1\begin{bmatrix}v\mid\alpha_+\end{bmatrix}}_{=: z_1}
+ \underbrace{\mu_2\begin{bmatrix}v\mid \alpha_-\end{bmatrix}}_{=: z_2}
+ \underbrace{\mu_3\begin{bmatrix}v^\perp\mid 0\end{bmatrix}}_{=: r}
$$
Consequently, $ A y = \lambda_1 z_1 + \lambda_2 z_2 + r $, $A^2y =\lambda_1^2 z_1 + \lambda_2^2 z_2 + r$, etc. Now it should become clear why we only need 3 iterations: We only need to figure out 3 coefficients: the coefficient to $z_1$, the coefficient to $z_2$ and the coefficient to $r$.
Lemma: assume $A$ is orthogonally similar to the $n\times n$ block matrix $A' = \begin{bmatrix}I_r & 0 \\0 & D\end{bmatrix}$, where $D$ is a diagonal matrix with $D_{ii}\neq D_{jj}\neq 1$ for all $i\neq j$.  Then minres terminates in  exactly $n-r+1$ iterations.
Proof: Given  orthogonal $U$, which diagonalized $A$, we have
$$
\|Ax - y\|_2
=\| A U^T Ux - y\|_2
= \|UA U^T U x - U y\|_2
= \|A' x' - y'\|_2
$$
So both optimization problems are equivalent,in the sense that if $x_k = \arg\min_{\xi\in K_k(A)}\|A\xi-y\|$ then $x_k =U^Tx_k'$ where $x_k' = \arg\min_{\xi\in K_k(A')}\|A'\xi-y'\|$. And the early termination in the diagonal case follows by induction over the size of $D$.
Corollary: If $A$ is orthogonally diagonalizable and has exactly $r$ disjoint eigenvalues then minres terminates in exactly $r$ iterations. (in practice this is not guaranteed due to numerical error)
import numpy as np
from scipy.sparse.linalg import minres
from scipy.stats import ortho_group

# create a random matrix of the specific form
N = 100
v = np.random.randn(N-1,1)
b = np.random.random(N)
A = np.block([[np.eye(N-1), v], [v.T, 0]])

# run MINRES for 3 iterations
callback = lambda x: print(np.linalg.norm(A.dot(x)-b))
x, info = minres(A, b, tol=1e-15, callback=callback)
print("MINRES returned:", info)
print("The returnvalue 0 means that it has converged.")

# orthogonal similarity transform
U = ortho_group.rvs(N)
A_dash  = U.T @ A @ U
b_dash  = U @ b
callback = lambda x: print(np.linalg.norm(A_dash.dot(x)-b_dash))
x, info = minres(A_dash, b_dash, tol=1e-15, callback=callback)
print("MINRES returned:", info)
print("The returnvalue 0 means that it has converged.")

# 4 disjoint EVs
U = ortho_group.rvs(N)
A  = np.diag(np.concatenate([2*np.ones(N//4), 3*np.ones(N//4), -1*np.ones(N//4), 10*np.ones(N//4)]))
A_dash = U.T @ A @ U
b_dash  = U @ b
callback = lambda x: print(np.linalg.norm(A_dash.dot(x)-b_dash))
x, info = minres(A_dash, b_dash, tol=1e-15, callback=callback)
print("MINRES returned:", info)
print("The returnvalue 0 means that it has converged.")


Is there a name for matrices that are symmetric along the cross diagonal?

Question: Something like
$$
A=
\begin{bmatrix}
    a & b & c\\
    b & d & e\\
    c & e & f
\end{bmatrix}
$$
would be a symmetric matrix because the values are reflected along the diagonal, and $A=A^\intercal$
Is there a name for a matrix that's symmetric along the cross diagonal? Something like
$$
B=
\begin{bmatrix}
    c & b & a\\
    e & d & b\\
    f & e & c
\end{bmatrix}
$$
Comment: You're right. I tried searching, but it's difficult to find without knowing the terminology in the first place.
Answer: If Wikipedia is to be believed, then this is a persymmetric matrix.
From the linked article:

In mathematics, persymmetric matrix may refer to:
  1. a square matrix which is symmetric in the northeast-to-southwest diagonal; or
  2. a square matrix such that the values on each line perpendicular to the main diagonal are the same for a given line.
Comment: The text cited actually gives two possible definitions, with only the first one being relevant in this case.


t-Test for dependent samples for z-standardized values?

Question: I've got children's questionnaire scores (mean of 6 items) regarding a particular variable (here: how well the child likes to interact with technology), and a score of the same child by a parent (they used a different questionnaire, also mean of a couple of items regarding how well their -- this -- child likes to interact with technology).
I want to find out whether their judgments differ, i.e., whether (H0) children's and parent's judgments are the same vs (H1) whether they do differ (and if so, whether the parents over- or underestimate their children's desire to interact with technology).
Normally (if the assumptions hold) I would use a t-Test for dependent samples, giving that parent and children are linked and judge the same thing (how well these particular child interacts with technology, self-rating vs. parent-rating). But the children's questionnaire has an answer scale of 1 to 4 (more or less: strongly disagree, disagree, agree, strongly agree), while the parent's questionnaire has an answer scale of 1 to 6 (strongly disagree, disagree, slightly disagree, slightly agree, agree, strongly agree).
So just calculating a t-Test for dependent samples would be a bad idea.
My more or less intuitive first idea would be to calculate a t-Test for dependent samples but with z-standardized scores. Just use a z-Transformation on the child scale and parent scale and then calculate the t-Test. Alternatively perhaps to stretch or shorten the answer scale.
But I'm confused here, so that's why I ask the following questions:

Can I calculate a t-Test for dependent samples with z-standardized scores this way or am I missing something?

Is there a better way to see whether the judgments differ?

Anything else I have missed (like I said, currently a bit confused)?
Answer: You ask students if they like apples and grapes in school lunches.
You ask parents if they think students like bananas and oranges in
school lunches. Can the answers help you decide whether parents
and the student children are on the same page about the desirability
of fruit for lunch?
Substantive (non-numerical) difficulties abound for this project.

First, you will have to look
at the questions globally to make sure the same issues are addressed
with about equal emphasis on the student and parent questionnaires.
(Are the kinds of technology the same? Are envisioned degrees/kinds of
interaction the same?)

Second, you will have to look at the questions individually to make
sure that "Agreement" always indicates enthusiasm for interaction
with technology. (Sometimes questionnaires try to test whether
respondents are equally likely to say they Agree and to say they
Disagree by reversing the sense of a few questions.)

Third, if the first two issues can be favorably resolved, you will
have to find a way to make the 4-point and 6-point Likert scales
agree.
Issues about the comparability of 6-point and 4-point Likert scales are discussed below.

You ask only about the third part. I don't know exactly what you mean
by "z-standardized scores," but if you mean to subtract the sample mean
and divide the difference by the sample standard deviation, then that
would clearly be counterproductive because both sets of scores will
be centered at $0,$ with no hope that a test will find a difference.

The simplest and perhaps least controversial method of making responses
similar for the two groups would be to decide if each answer is
favorable 1 (or not 0) as to the child's interest (or guessed interest) in interaction
with technology.

In my experience it is difficult to convert from one Likert
scale to another, while keeping essential information intact. Here are some things you might try:

If the 6-point scale has lots of 1s and 6s and
the 4-point scale has lots of 1s and 4s, then it may be OK to reduce
the 6-point scale to a 4-point scale by mapping 6s into 4s and
collapsing 2-3 to 2 and 4-5 to 3.

If there are relatively few extreme scores on both questionnaires,
then it may be best to map 1-2 onto 1 and 5-6 onto 4.

You might try make the 6-point scale "match" a 4-point scale by
using values something like 1 1.5 2 3 3.5 4 instead of 1 2 3 4 5 6. But once
differences are found for a paired test, this scheme still might give the
(modified) 6-point scale an 'edge' in determining ranks in a Wilcoxon (signed-rank)
test. (There would be no corresponding difficulty for t tests, if used.)
Also, there is controversy whether Likert scores should be
taken as interval numerical data instead of the ordinal categories that they actually are.
This scheme goes far in the direction of deciding in favor of numerical
data.

Experiments with simulated data. You don't say how many parent-student
pairs of subjects you have. I will assume 100 pairs here. Simulated scores
might be as summarized and plotted below.  [Summaries, plots, and tests were done in R.]
st = rbinom(100, 3, .6)+1
table(st)
st
 1  2  3  4
 6 30 48 16
table(pr)
pr
 1  2  3  4  5  6
 3  4 21 18 23 31

cor(st,pr)
[1] 0.7339378

set.seed(617)
stj = st+runif(100, -.2,.2)
prj = pr+runif(100, -.2,.2)
plot(prj, stj, ylab="Student", xlab="Parent", pch=20)
 abline(v=1:6, col="green2");  abline(h=1:4, col="green2")

In the scatterplot, integer values have been slightly 'jittered' (randomly
displaced) to avoid overplotting at a few grid points.
Students and Parents both have 'Successes' and 'Failures'. Reducing the data to 1 = Favorable, 0 = Not, and then looking at
a test of binomial proportions:
st.01 = st
st.01[st <= 2] = 0;  st.01[st >= 3] = 1
table(st.01)
st.01
  0  1
 36 64

pr.01 = pr
pr.01[pr <= 3] = 0;  pr.01[pr >= 4] = 1
table(pr.01)
pr.01
 0  1
28 72

sum(st.01==0 & pr.01==0)
[1] 24

A contingency table with rows counting Students Less (0) and More (1)
inclined toward interacting with technology and columns counting
Parents' assessments of their children as being Less (0) and More (1)
inclined is as follows.
TAB = rbind(c(24,12), c(4,60));  TAB
     [,1] [,2]
[1,]   24   12
[2,]    4   60
rowSums(TAB);  colSums(TAB)
[1] 36 64
[1] 28 72

A chi-squared test for independence of responses between students and parents
strongly rejects the null hypothesis of independence.
chisq.test(TAB)

        Pearson's Chi-squared test
     with Yates' continuity correction

data:  TAB
X-squared = 38.773, df = 1, p-value = 4.76e-10

For counts this large, a continuity correction may not be appropriate, but the null hypothesis is rejected either way.
chisq.test(TAB, cor=F)$p.val  # without continuity correction
[1] 1.055272e-10

Fisher's exact test for independence gives a similar result.
fisher.test(TAB)$p.val
[1] 1.642062e-10

Parents do suspect slightly more enthusiasm than students report, but of the $4 + 12 = 16$ instances of 'disagreement', it would not be significantly rare (5% level) to have a 4:12 or a 12:4 split or a stronger disagreement by chance alone.
2*pbinom(4, 16, .5)
[1] 0.07681274

Parent Scores between 1 and 4. It is possible that reducing the data to 0s and 1s has lost
too much information about differences between parents and their children. Let's try the second scheme for changing
the 6-point scale to a scale that runs from 1 to 4:
pr.4 = pr
pr.4[pr==2]=1.5;  pr.4[pr==3]=2;  pr.4[pr==4]=3
pr.4[pr==5]=3.5;  pr.4[pr==6]=4
table(pr.4)
pr.4
  1 1.5   2   3 3.5   4
  3   4  21  18  23  31

Here is a summary of the differences when student 4-point scores are
subtracted for parent scores converted to values between  1 and 4.
d = pr.4 - st
summary(d)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
 -1.000   0.000   0.000   0.355   1.000   2.000
t.test(d)

For my simulated data, a paired t test overwhelmingly rejects the null hypothesis that
Parents have a realistic view of the their children's enthusiasm
for interacting with technology. They tend to overestimate it. (Maybe the
enthusiasm for interacting with technology they perceive is
sometimes just an excuse to get out of helping with the dishes.)
        One Sample t-test

data:  d
t = 5.7189, df = 99, p-value = 1.14e-07
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 0.2318301 0.4781699
sample estimates:
mean of x
    0.355

Note: In case it is of interest, the method of simulating my artificial
data is shown below.
set.seed(616)
st = rbinom(100, 3, .6)+1
table(st)
st
 1  2  3  4
 6 30 48 16
pr = rbinom(100, 5, st/4) + 1
table(pr)
pr
 1  2  3  4  5  6
 3  4 21 18 23 31
cor(st,pr)
[1] 0.7339378

Comment: Note: The original version of this answer posted yesterday had a mistake in a chi-squared test for data reduced to 0's and 1's; I have just now revised my answer to correct this mistake.
Comment: Kudos. And thank you :-)


Integrability of the Hilbert transform of a Schwartz function

Question: Given a Schwartz function $f\colon\mathbb{R}\to\mathbb{R}$, define its Hilbert transform by
$$(Hf)(x)=\frac{1}{\pi}\left(\int_{|t|\leq 1}\frac{f(x-t)-f(x)}{t}\,dt +
\int_{|t|\geq 1}\frac{f(x-t)}{t}\,dt\right)$$
(the first integral is interpreted as an appropriate limit/principal value).
It can be shown that $Hf$ is continuous and that
$\lim_{|x|\to\infty}x(Hf)(x)=\frac{1}{\pi}\int_{\mathbb{R}}f(t)\,dt=\frac{1}{\pi}\hat{f}(0)$. Using this, it is easy to prove that, if $Hf$ is absolutely integrable, $\hat{f}(0)$ must be $0$. I want to prove the converse.
So assume $\lim_{|x|\to\infty}x(Hf)(x)=0$. This by itself doesn't guarantee integrability of $Hf$, since $Hf$ might behave like $\frac{1}{x\log x}$ at infinity. A stronger decay condition for $Hf$ is needed, but I'm not sure where to get it from. The fact that $f$ is Schwartz should be important; does this imply $Hf$ decays faster than any polynomial?
Note, since this is a homework question, please don't be overly explicit in your answers.
Comment: I don't know if it is supposed to use the formula $F[Hf]=-i\,\mathrm{sign\,} \xi \hat{f}(\xi)\,$. Since $\hat{f}$ is smooth (from the Schwartz class), the possible slow decrease of $Hf$ is given by (and equal to $C_1/x$, where $C_1$ is proportional to) the jump of $F[Hf]$ at the origin. The next member of asymptotic should be like $C_2/x^2$, there $C_2$ depends on the jump of the first derivative of $F[Hf]$ at the origin etc.
Comment: @Andrew I found this multiplier formula in literature as well, but I don't think we're supposed to use it since nothing similar was ever mentioned.
Comment: Note, you don't need to interpret the first integral as a principal value, it simply converges absolutely because $f$ is differentiable at $x$.
Answer: $Hf$ does not have to decay faster than any polynomial; as @Andrew suggested, $1/x^2$ is the best we can expect.
The assumption $\int_{\mathbb R}f(t)\,dt=0$ implies that the antiderivative $F(t):=\int_{-\infty}^t f(s)\,ds$ is also integrable (in fact, it's in the Schwartz class too). This should allow you to integrate the second integral by parts (and the first one can be estimated with the MVT.)


sh: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 24

Question: I have a C application running on Linux and it processes thousands of messages...but not whe I run the application after a couple minutes it brings up the following error:
sh: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 24
The application is doing about 30 messages per seconds, could it be some file open handle issue?
STDOUT receives: "Process 19374 received signal 11 "
Any ideas?
Thanks for the help ;-)
Lynton
Comment: Make sure you aren't leaking file descriptors. `lsof -p ` can tell you what you've got open.
Comment: signal 11 means segmentation fault. Try using a debugger, e.g. gdb ("man gdb" for help). Not sure about the libc error. How many processes does the application start? Is process 19374 a sub process? Can you list the processes running ("ps aux | grep ") before and after the application crashes? What is the order of the error messages (comes the "cannot open shared object file" before "process 19374 received signal 11"?)
Answer: Errno 24 means: "Too many open files".
After some thoughts I think the application opens a file, which is not allowed by the system anymore. The application can not handle this situation correctly, and crashes with a segmentation fault. Are the return values from the "open" system call checked correctly everywhere?
Answer: To handle this situation, I found 2 ways to resolve it. Either to work on same alloted file descriptor number or close FD just after their work have been done.


How can I find out why a task is running?

Question: I'm working with a complicated Ansible system with a lot of roles in various repositories. When I run a certain playbook, a task is running that I think shouldn't be. Is there a way to find out what triggered it?
Comment: This is a good question. There are multiple ways to specify task and role dependencies in ansible, i.e. `import_` and `include_` directives, role `dependencies` and more. Ansible does not provide a simple way to trace it.
Answer: Ansible does not provide at the moment any tools to visualize the task and roles dependency tree.
What you could do though is

Get a list of tasks with the --list-tasks option from ansible-playbook.
Consider the list of tasks from the top level playbook in the order of appearance and mark these top-level tasks in the list.
Determine where the task that you are interested in appears between the marked tasks.
Make a playbook that includes only the marked task that appears directly below the task that you are interested.
Repeat 1 through 5 until the path through role and task dependencies is recovered.

Example
This description is too general so here is an example taken from a real site.
ansible-playbook --list-tasks site.yml

outputs some 900 lines. The playbook contains multiple plays and we are interested in a play that applies to a specific group of hosts called computeservers.
The play may look like this:
- hosts:
    - computeservers
  roles:
    - basic-pkg
    - ntp
    - zabbix
    - slurm
    - jupyter
    - gitlab-ci

Suppose we would like to know why the task Install compilers is running on this group of nodes.
Suppose we know (using grep) that this task belongs to the role called compilers. We see that compilers is not in the list of top-level roles, so it is somewhere down the dependency tree.
In the task list we look for a play that starts with
  play #2 (computeservers): nodes   TAGS: []
    tasks:
       basic-pkg : task1
       basic-pkg : task2
       ...
       slurm : task1
       slurm : task2
       ...
       compilers : Install compilers
       ...
       jupyter : task1
       ...

So we see that our task of interest Install compilers is somewhere between top-level roles slurm and jupyter. Therefore we need to further investigate the jupyter role.
This jupyter role may have a dependency
dependencies:
  - scipy

Eventually you would recover the path in the role dependency tree that leads to the task in question. In our example that would be
jupyter -> scipy -> anaconda -> compilers -> Install compilers

Answer: Ansible runs all tasks in the order they are listed by default. A task executing that you did not expect implies some conditional. Typically when: expressions,  handlers, or includes.
You need to debug this like you would software. Find the simplest test case. Print out the state, such as using the debug module.  For more specific advice, post your (sanitized of sensitive information) playbook and roles.


How does the match time work in soccer?

Question: How does match time works in soccer?
I think I know a little bit about how it works, like how they add injury time to the end of the regular time, but I do not know it for sure.
Answer: Well the regular amount of time for each half normally lasts 45 minutes:

The match lasts two equal periods of 45 minutes, unless otherwise mutually agreed between the referee and the two participating teams. Any agreement to alter the periods of play (for example to reduce each half to 40 minutes because of insufficient light) must be made before the start of play and must comply with competition rules.

Injury Time is added at the end of each half:

Allowance is made in either period for all time lost through:

substitution(s)
assessment of injury to players
removal of injured players from the field of play for treatment
wasting time
any other cause

The allowance for time lost is at the discretion of the referee.

And in certain competitions, Extra Time may be required at the end of a game, for example in the World Cup, where two halves of 15 minutes in length are played, with the above rules of added time applying to those two halves.
Answer: The duration depends on the referee.  Injury time is established by the referee and he gives how many minutes of stoppage time he wants.
From the "Laws of the Game":

The duration of stoppage time is at the sole discretion of the referee.

As you probably knew already, any kind of match interruptions will extend the stoppage time. However, only the referee can be the timekeeper.
Facts: The ball is in game for about 22 minutes each half in the most important leagues.
The longest injury time given was 23 minutes.
Germany referees usually give no more than 3 minutes of additional time.
Comment: just a quick addition, even if there are no remarkable injuries, the first half is usually extended at least 1 mins. And regardless of the 4th officials sign regarding the injury time, the referee can decide to call an end to the game as he sees it fit. One such example was during a match I saw, it was 4-0 and the losing team was one man down and had already relegated to the lower division, the ref blew it off immediately after 90 mins.
Comment: you're perfectly right. in Bundesliga is quite usual to see only one minute or even none for the first half.
Comment: commonly in the premier league injury time is around 4 minutes. most adhere to the 30 seconds per substitution rule, which could mean 3 minutes. but yes it's the at the referees discretion.
Comment: What about delays that occur (injuries, goal celebrations, etc.) that take place *during*  the previously specified injury time?

How does the match time work in soccer?


Convergence of random variables (Durrett: Probability Theory and Examples)

Question: I was working out some problems from Rick Durrett's Probability theory and Examples (2010 edition), when I came across a very unusual question(reproduced here ad-verbatim):
If $X_n$ is ANY sequence of random variables, there are constants $c_n \to \infty$ so that
$$\frac{X_n}{c_n} \to 0 \quad \mbox{a.s}$$
You will find it in Chapter 2: Laws of Large Numbers, under the section on Borel Cantelli Lemma. What makes this question unusual is that there are NO assumptions made on the random variables.
My attempt:
I rephrased the question equivalently as:
If $X_n$ is ANY sequence of random variables, there are constants $a_n \to 0$ so that we need to show
$$a_nX_n \to 0 \quad \mbox{a.s}$$
Then I showed that it was sufficient to assume $a_n > 0$ and $X_n \geq 0$. To see this, since the limit is going to , the sign of the constants do not matter, hence positivity of $a_n$ can be assumed w.l.o.g. As for an arbitrary r.v, any r.v can be written as:
$$X_n = X_n^+ - X_n^-$$
where $X_n^+ = \max(X_n,0)$ and $X_n^- = \max(-X_n,0)$
Suppose we prove the result for non-negative random variables, then say we have
$$b_n X_n^+ \to 0 \quad c_n X_n^- \to 0 \quad \mbox{a.s}$$
Then pick $a_n = \min(b_n,c_n)$. Then this will ensure
$$a_nX_n \to 0 \quad \mbox{a.s}$$
For the non-negative case, I was able to prove the result for simple functions: If $X_n$ is simple, and $X_n = \sum_{k=1}^n s_k 1_{A_k}$, take $a_n = \frac{1}{2^{n} \sum_{k=1}^n s_k}$ would work, but only if all functions were simple.
Now the tough part. Handling the non -ve measurable function case. I had difficulty here.
Hence my question is:
I'd like a hint/answer (preferably a hint) on how to solve this particular case. Right now I am looking at manipulating the lemma that every non-negative measurable function can be approximated by a monotone sequence of simple functions.
Thank you.
Note: I use measurable functions and random variables interchangeably. But note that the space is a probability space. Additionally, I didn't find any similar question (I typed convergence random variables). If it has been answered, kindly provide the link.
Answer: Its very interesting. Based on Thomas' reply, let me respond:
Let us consider the case where $X_n \geq 0$. Now, we know this property of CDF
$$\lim_{u \to \infty} P(X_n > u) = 0$$
So given $n \geq 1, m \geq 1 \quad\exists \delta_{n,m} > 0$ s.t
$$ u \geq \delta_{n,m} \Rightarrow P(X_n > u) \leq \frac{1}{2^m}$$
Now pick $c_{m,n}$ such that $c_{m,n} > \delta_{n,m}2^m$ and substitute above to get
$$P\left(\frac{X_n}{c_{m,n}} > \frac{1}{2^m}\right) \leq \frac{1}{2^m}$$
Now arrange the $c_{m,n}$ in a grid and pick the diagonal sequence i.e. let $c_n = c_{n,n}$. Now apply the Borel Cantelli Lemma to conclude:
$\forall \epsilon > 0$, pick m s.t $2^{-m} < \epsilon$ and
$$\sum_{n=1}^{\infty}P\left(\frac{X_n}{c_n} > \epsilon\right) \leq \sum_{n=1}^{\infty}P\left(\frac{X_n}{c_n} > \frac{1}{2^m}\right) $$
$$ = \sum_{n=1}^{m}P\left(\frac{X_n}{c_n} > \frac{1}{2^m}\right) + \sum_{n=m+1}^{\infty}P\left(\frac{X_n}{c_n} > \frac{1}{2^m}\right)$$
$$ \leq  \sum_{n=1}^{m}P\left(\frac{X_n}{c_n} > \frac{1}{2^m}\right) + \sum_{n=m+1}^{\infty}P\left(\frac{X_n}{c_n} > \frac{1}{2^n}\right)$$
$$ \leq \sum_{n=1}^{m}P\left(\frac{X_n}{c_n} > \frac{1}{2^m}\right) +  \sum_{n=m+1}^{\infty}\frac{1}{2^n} < \infty$$ and thus
$$\Rightarrow P\left(\frac{X_n}{c_n}\to 0 \right) =1$$
Answer: Hint: Consider $\xi_n(c) := P(|X_n| \geq c)$ and choose $c_n$ such that $\xi_n(2^{-n}c_n)$ is very small.
Comment: Can this be done even if $X_n$ is not integrable?


Example of discontinuity of position operator $X\colon f(x) \mapsto xf(x)$ for $f(x) \in L^2(\mathbb{R})$

Question: It is well known that continuous linear operators are bounded and vice versa. It is also well known that the position operator (which I shall call $X$) causes many headaches in quantum mechanics due to its unboundedness, which results in the construction of rigged Hilbert spaces and so on.
I know that
$$f(x) = \begin{cases} \frac{1}{x}, & x \geq 1\\ 0, & x < 1\end{cases}$$
can be used as an example for a function $ f(x) \in L^2(\mathbb{R})$ that demonstrates the unboundedness of $X$. However, I would be interested in an example that explicitly shows the/a discontinuity of $X$.
By linearity, it should even be possible (for example) to find a sequence that converges to the everywhere zero function where the $X$ operator is discontinuous, however, after playing around with a few Gaussian function sequences and trying to look up some canonical example, I couldn't come up with such a case. What would an example be?
Answer: If $X$ is unbounded, then there exists some sequence $(f_n)$ such that $\|X(f_n)\| \to \infty$ but $\|f_n\| \le 1$. Let $g_n = f_n / \|X(f_n)\|$. Then
$$0 \le \|g_n\| = \frac{\|f_n\|}{\|X(f_n)\|} \le \frac{1}{\|X(f_n)\|} \to 0.$$
Thus, $g_n \to 0$, but
$$\|X(g_n)\| = \frac{\|X(f_n)\|}{\|X(f_n)\|} = 1,$$
and hence $X(g_n)$ does not converge to $0$. This provides your counterexample.
In this situation, $X$ is not just discontinuous at $0$, but discontinuous everywhere. Simply shift this sequence by any $f_0$, and you'll get a sequence $g_n \to f_0$, but $X(g_n) \not\to X(f_0)$.
Comment: Very nice. A concrete example following your schema could be $g_n = \sqrt{\frac{2}{n}} x^{-\left(\frac{3}{2} + \frac{1}{n}\right)}$.
Comment: ... for $x \geq 1$, $0$ else; as above. I forgot that part.


Prove that if deg(vertex) > number of vertices/2, then G contains a triangle.

Question: A triangle would mean a cycle of length 3. To show a cycle, I would have to prove that each vertex has degree 2.
However, I am unsure of how to show the above using the assumption that deg(vertex) > number of vertices/2. While this intuitively makes sense, how can I go about proving it?
Can the Degree-Sum formula be used?
Comment: Is your hypothesis that **every** vertex has degree $> n/2$, where the number of vertices is $n$?  Show that if $a$ and $b$ are two neighbours, the sets of neighbours of $a$ and $b$ have nonempty intersection.
Comment: Do you mean the deg($v$) for all $v\in V$ is greater than $\frac{|V|}{2}$?
Comment: False. The star $K_{1,n}$ has a vertex of degree $n$ which is greater than number of vertices$/2$ when $n\ge2.$
Answer: So, given a graph $(V,E)$, for each $v \in V$, let us define $N(v)$ via:
$$
N(v) = \{q \in V : (q,v) \in E\}
$$
Take a vertex $v_1 \in V$, and identify another vertex $v_2 \in N(v)$. We will show that,
$$
N(v_1) \cap N(v_2) \neq \emptyset,
$$
which will prove the claim. Assume for the sake of contradiction that this intersection is empty. Then, $|N(v_1) \cup N(v_2)|=|N(v_1)|+|N(v_2)|>|V|$, a contradiction! Hence, $\exists v_3 \in N(v_1)\cap N(v_2)$, such that $v_1-v_2-v_3$ forms a triangle.
Comment: Can you explain what N(v) is? Is that the set of all vertices that v shares an edge with? Also, I do not understand why |N(v1)∩N(v2)|=|N(v1)|+|N(v2)|.
Comment: For a given vertex $v$, $N(v)$ is the set of all vertices that share an edge with $v$. On the second item above, it was a typo; which is fixed now.
Answer: Assuming $deg(v)=\frac{|V|}{2}$ for $\forall v \in V$ we can form a complete bipartite graph $K_{\frac{|V|}{2},\frac{|V|}{2}}$, i.e., a graph with maximal number of edges without a $3$-cycle. Since $deg(v)>\frac{|V|}{2}$ there is at least one $3$-cycle in $G$.
Comment: To be more precise, we need to know that $K_{n/2,n/2}$ is the graph with the *maximum* number of edges that does not have a $3$-cycle: $\frac{n^2}{4}$ edges. This is [Mantel's theorem](https://en.wikipedia.org/wiki/Mantel%27s_theorem). Just saying that $K_{n/2,n/2}$ is maximal without a $3$-cycle is not enough, since not all graphs with minimum degree $>n/2$ contain $K_{n/2,n/2}$ as a subgraph.