How to create a view with inner join with variables in PostgreSQL?

Question: I am using PostgreSQL DB. I need to create a view which consists of a query with joins on multiple tables, but I am stuck at a point. Please read the below table definition and the Query which I have created.
Tables:

Students
---------
id -> UUID
full_name -> VARCHAR
email_id -> VARCHAR
location -> VARCHAR
created_at -> timestamp
modified_at -> timestamp

Subjects
--------
id -> UUID
title -> VARCHAR
classroom -> VARCHAR
created_at -> TIMESTAMP

Applications
------------
id -> UUID
student_id -> FOREIGN KEY (Students.id)
subject_id -> FOREIGN KEY (Subjects.id)
verified -> BOOLEAN
application_response -> JSON
status_id -> FOREIGN KEY (ApplicationStatus.id)
created_at -> TIMESTAMP

ApplicationStatus
-----------------
id -> UUID
application_status -> VARCHAR
category -> VARCHAR

Scores
-------
student_id -> FOREIGN KEY (Students.id)
subject_id -> FOREIGN KEY (Subjects.id)
score -> NUMERIC

Following is the SQL query which I have created:
create or replace view testing_list_view as
select c.id as student_id,
a.subject_id as subject_uid,
c.full_name,
c.email_id,
c.created_at,
p.score,
s.application_status,
a.verified,
a.application_response
from students c
inner join scores p
on p.student_id=c.id and p.subject_id = 'ff342ada-f5gb-44fb-bdth-44e3n59f5448'
inner join applications a
on a.student_id = c.id and a.subject_id= 'ff342ada-f5gb-44fb-bdth-44e3n59f5448'
inner join applicationstatus s
on s.id = a.status_id
where c.id in
(select student_id from applications where subject_id='ff342ada-f5gb-44fb-bdth-44e3n59f5448')

Here, I am getting the list of students for the given subject_id along with the scores, application_status and some other fields for which I need to do joins.
My requirement is to make a view for this query so that I pass only the subject_id to the view (select * from testing_list_view where subject_uid='ff342ada-f5gb-44fb-bdth-44e3n59f5448') and I will get a list of the students who have applied to the given subject. But, I am stuck here as in the above join query, subject_id is required in multiple times in the join clauses and is hard-coded. So, I am unable to make it a view.
I was thinking if some kind of variable exists using which, in the join clauses, I will use the variable in the required clauses and which querying the view I will pass it the value (subject_id).
Please let me know if any more clarification is needed.
Comment: Views can't have parameters, but you can create a SQL function (`returns table (...)` ) that takes a parameter as an input and simply returns the result of the query
Comment: look at answers here - https://stackoverflow.com/questions/11401749/pass-in-where-parameters-to-postgresql-view
Comment: @JosMac Looks relevant to my problem. Let me try the solution of that question.
Answer: You should define the view without the WHERE clause and all the other restrictions on subject_id:
create or replace view testing_list_view as
select c.id as student_id,
       a.subject_id as subject_uid,
       c.full_name,
       c.email_id,
       c.created_at,
       p.score,
       s.application_status,
       a.verified,
       a.application_response
from students c
   inner join scores p
      on p.student_id=c.id
   inner join applications a
      on a.student_id = c.id and a.subject_id = p.subject_id
   inner join applicationstatus s
      on s.id = a.status_id;

You specify the condition when you use the view, e.g.
SELECT * FROM testing_list_view
WHERE subject_uid = 'ff342ada-f5gb-44fb-bdth-44e3n59f5448';

Comment: Thanks for your answer. I could not understand the answer. Can you please elaborate?
Comment: I have elaborated by adding the view definition I envision.
Comment: When I ran the query you have given, I got an error - SQL Error [42702]: ERROR: column reference "subject_id" is ambiguous. It is because subject_id column is a foreign key in both applications and scores tables. That's why it is giving that error.
Comment: Which query? The column `subject_id` in the view definition is qualified and cannot be ambiguous.
Comment: There was a mistake in my query that's why error is coming. Well, the query ran successfully. But, the results are not correct. Because, When I ran my original query (in the question) it is giving a count of 215 (which is correct). But, when I created a view without subject_id and which querying, I used subject_id, it gave me a result count of 2150.
Comment: Probably, It's doing some sort of cross join somewhere I guess. because in the application table, that much of students are not present for the given subject_id.
Comment: Sorry. I forgot a join condition. Try again.


Factor the square of a complex number

Question: While calculating some eigenvalues I stumbled upon this root $3+i \pm \sqrt{6i}$
Try as I might I was unable to simplify $6i$, I looked at the given answer and saw that $\sqrt{6i} = \sqrt{3} + \sqrt{3}i$
I set on this for a bit, searched google, but I can't seem to understand who this was done, can anyone explain?
Comment: Do you know [Euler's formula](https://en.wikipedia.org/wiki/Euler%27s_formula) and how to find the [polar form of a complex number](https://en.wikipedia.org/wiki/Polar_coordinate_system#Complex_numbers)? The principal square root $\sqrt{z}$ of an element $z=re^{i\theta}\in\Bbb C$ is defined by $\sqrt{z}:=\sqrt{r}e^{i\theta/2}$
Comment: Square $\sqrt3+\sqrt3i$, what do you get?
Comment: P.S. Further to the answers, do note that $(6i)^{\displaystyle\frac12}=\pm\sqrt3(1+i),$ and that when dealing with complex numbers, the notation $\sqrt{\;}$ isn't a good idea; for example, is the principal root of $-1$ (A) $e^{i \frac\pi3}$ or (B) $-1 ?$
Answer: Simply use the definition of complex
$\sqrt{i}
=\left[ e^{i\frac{\pi}{2}} \right]^{1/2}
=e^{i\pi/4}
=\frac{\sqrt{2}}{2}(1+i)$.
The rest follows.
Answer: The simplest approach in my opinion is to consider the exponential form of the complex number i. We know that $i=e^{i\frac{\pi}{2}}$, so we get that:
$$\sqrt{6i}=\sqrt{6e^{i\frac{\pi}{2}}}=(6e^{i\frac{\pi}{2}})^\frac{1}{2}=6^{\frac{1}{2}}e^{i\frac{\pi}{4}}$$
But then using Euler's formula you get:
$$\sqrt{6i}=\sqrt{2}\sqrt{3}(\cos(\pi/4)+i\sin(\pi/4))=\sqrt(3)(1+i)$$
Where the $\sqrt{2}$ was cancelled.
Answer: $ \sqrt {6i} = \sqrt {6i+3-3} = \sqrt {(\sqrt3 + \sqrt3 i)^2} = \sqrt3 + \sqrt3 i $

Factor the square of a complex number


Intuitions about KNN and linear regression in a regression setting

Question: I'd like to develop some intuitions about linear regression and KNN in a regression setting.
Let's say I have 6 different datasets below. They all have 1 independent variable X and corresponding real value Y, and let's assume that X values do not overlap in each dataset.  Imagine that I separate them into a training set and test set, apply KNN(K with 3 or 5) and linear regression(without nonlinear kernels) and validate with the test sets using MSE. Red lines are regression lines that I will get from linear regression. Here what I like to think about is the performances of KNN and linear regression with different shapes and variabilities of datasets.

Here are my conjectures.
With the dataset 1-1, KNN will result better performance because, although there is some linearity, the dataset disperses a lot making it more non-linear. With the dataset 1-2, I think KNN and linear regression will perform somewhat the same because the linearity and variability of the dataset compensates each other.
With 2-1, KNN will outperform a linear regression because the datasets look very non-linear. KNN also will outperforms with 2-2 because the dataset is also very non-linear.
The dataset 3-1 seems to have a strong linearity with gaussian noise. Thus, linear regression will bring about the better performance, because when KNN tests the points close to the linear line, it will predict values far from the linear line. The dataset 3-2 seems to have the same linearity, but with noise uniformly distributed. In this case, I think KNN will perform better because the points are not centered around the linear line, making linear regression hard to predict.
I think my reasoning is very weak and somewhat non-systematic. Can you please let me know I should interpret these datasets in relation to KNN and linear regression?
Answer: Well, at first you need to define order of your KNN model and order of your Linear regression
2-2 has some curvature, so obviously plain linear first regression would perform bad (MSE perspective).
With K=[6-9] and good training set I would say that KNN will beat Liner regression. Because in nature it tries to approximate based on some local vicinity. So if in this area you have 6-9 points your approximation will be good enough.
MSE KNN in most of the cases is smaller than Linear Regression, on rest of the cases it's slightly worse (so it seems that KNN is almost always better). But KNN has bad performance in high dimension spaces, MSE degrades much faster than LinearRegression solution (Intuition - if in this space you don't have any example, so point will be estimated based on long distant examples).
ISLR has good explanation of both cases in Chapter 3.
http://www-bcf.usc.edu/~gareth/ISL/
Comment: I don't see the connecting between linear regression and KNN clustering, The pictures are confusing too.
Comment: Don't want to type chapter 3 from the book, but look [picture](https://image.ibb.co/cGKFx5/Screen_Shot_2017_07_24_at_8_41_40_PM.png)
Answer: Overall:
Why did I put this paragraph before the one where I try to address your question? Because the assumptions and the basic idea behind the models dictate how you interpret the output and thus what you use the model for.
Linear regression is 'simply' a projection of a random variable $Y$ unto a subspace spanned by your independent variables. You can do this no matter how Y is actually distributed (https://en.wikipedia.org/wiki/Linear_least_squares_(mathematics)#Properties_of_the_least-squares_estimators).
Intuitively: Adding more dimensions to the problem (provided you can't describe the newly added dimension with those you've already included in the model) will always minimize the error between your model and your data. How much you minimize the error should be related to the actual descriptive power of the added variable. Adding someone's astrological sign to a regression against life-time earnings might not explain a lot of the variation is people's life-time accumulation of money, whereas the level of education might be a better variable. This to say that linear regression relies on a bit of expert knowledge or assumptions about the data, and that knowledge then leads to a model with a lot of nice properties like the existence of Confidence Intervals (disputed as they may be) and p-values etc...
K-nearest neighbor regression is a local method, where the size of the local neighborhood is inversely proportional to $K$, that uses the (potentially weighted) average of the supplied data points to describe the behavior in that local neighborhood. In KNN no assumptions are made about anything. We're free to describe anything as a collection of local behaviors. The flipside is that we're missing out on CI, p-values, possibility of extrapolating from our model[1], physical interpretation of our model and so on.
Have you seen: Why would anyone use KNN for regression?
How to interpret these datasets in relation to KNN and linear regression:
Be mindful of the assumptions underlying the two different approaches.
If you have a meaningful description of independent components, ie if you have 'expert knowledge' about how the input and the outputs are related then linear regression makes sense and has a physical interpretation. You can then talk about how much a certain amount of input changes the output.
If you cannot make any justifiable, and/or simplifying, assumptions about your data then KNN is a reasonable choice. Just remember that KNN tells you something about how a small, local ensemble of data behaves, not why the data behaves the way it does.
Your outset seems to be which is more accurate in terms of MSE, but that then relates back to which assumptions can be made about the data and also why you'd only care about the MSE and not about the physical interpretation of your model. If the first thing you did was to plot your data and found, say plot 1-1, would you honestly sit down and try to choose between KNN regression and a straight line slope?
That is why you always plot your data and do residual analysis: data does not exist in a vacuum where math comes to the rescue and provides a summary statistic that says: That one is best!
The data and how the data was collected determines the best approach, not the model that provides the smallest MSE.
To relate to your own analysis: In plot 2-1 I believe that a weighted regression would be much more appropriate than a KNN. If you have reason to believe that there is a straight line behaviour through that cloud of data, then taking heteroscedacity into consideration is sensible. From an 'armchair' point of view KNN's tendency to low bias would cause the KNN regression to wiggle around the middle of the graph and stabilize out at the ends. Is that a meaningful representation of the data? Honest question, not having a stab at you.
Plot 1-1 and plot 1-2 are obvious cluster analysis data sets, given no other information than the data I'm looking at.
Plot 2-2 is an tempting linear regression with a spline/polynomial/trig function depending on what kind of system the data were measured on.
3-1 and 3-2 both fit right into the linear model framework.
2-2, 3-1, and 3-2 are the only data sets where KNN would capture the qualitative behavior in any meaningful sense.[2]
TOO LONG; DIDN'T READ
Either I think you're asking the wrong kind of question by insisting on comparing two different and possibly ill-fitting models to your toy data or I completely misunderstood what you wanted to know.
[1] Before you downvote: sometimes, just sometimes we need to extrapolate a bit beyond the domain where our model was initially trained...
[2] I am now sorely tempted to patch together some data that looks like yours and actually try it out.


C# - TreeView - Data Algorithm

Question: I work with C# WPF TreeView.
I have folowing input data: (like 1000+ lines that I need to process)
\Folder\Base\Data\dat.cfg
\Folder\Base\Data1\dat.cfg
\Folder\dat.cfg

Collection class - TreeItem. Used for TreView binding.
    public class TreeItem : MyViewModel
    {
        public TreeItem()
        {
            this.Children = new ObservableCollection<TreeItem>();
        }

        public string Name { get; set; }

        public ObservableCollection<TreeItem> Children { get; set; }
    }

I'm spending on this fews days and I'm not able to figure out algorithm how to obtain data to requested point.
Please I would really appreciate if somebody could give me help or just way how to achieve it.
Thank you in advance.
This is my class for collection
I need to acomplish input data to be parsed as following in my defined collection class - TreeItem.
Basicaly just to be working in TreeView as nodes with levels.
Requested data structure

CollectionTreeItem
[0] Name - "Folder"
    Children
    [0] Name - "Base"
        Children
            [0] Name - "Data"
            Children
                [0] Name - "dat.cfg"
                Children

Only example with procesed line - \Folder\Base\Data\dat.cfg. Of course there will be more child with another lines.
Answer: You could use a recursive algorithm to build the tree. treeObjects should hold the structure you're trying to achieve.
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Linq;

class Program
{
    static void Main()
    {
        string[] paths = new string[]
        {
            @"\Folder\Base\Data\dat.cfg",
            @"\Folder\Base\Data1\dat.cfg",
            @"\Folder\dat.cfg"
            // Rest of your paths
        };

        var root = new TreeItem { Name = "Root" };

        BuildTree(root, paths);

        var treeObjects = GetTreeObjects(root);
    }

    static void BuildTree(TreeItem root, string[] paths)
    {
        foreach (var path in paths)
        {
            AddPathToTree(root, path.Split('\\').Skip(1).ToArray());
        }
    }

    static void AddPathToTree(TreeItem root, string[] path)
    {
        TreeItem currentNode = root;

        foreach (var folder in path)
        {
            var existingNode = currentNode.Children.FirstOrDefault(c => c.Name == folder);

            if (existingNode == null)
            {
                var newNode = new TreeItem { Name = folder };
                currentNode.Children.Add(newNode);
                currentNode = newNode;
            }
            else
            {
                currentNode = existingNode;
            }
        }
    }

    static List<CollectionTreeItem> GetTreeObjects(TreeItem node)
    {
        var result = new List<CollectionTreeItem>
        {
            new CollectionTreeItem
            {
                Name = node.Name,
                Children = GetChildrenObjects(node.Children)
            }
        };

        return result;
    }

    static List<CollectionTreeItem> GetChildrenObjects(ObservableCollection<TreeItem> children)
    {
        var result = new List<CollectionTreeItem>();

        foreach (var child in children)
        {
            var childObject = new CollectionTreeItem
            {
                Name = child.Name,
                Children = GetTreeObjects(child)
            };

            result.Add(childObject);
        }

        return result;
    }
}

public class CollectionTreeItem
{
    public string Name { get; set; }

    public List<CollectionTreeItem> Children { get; set; }
}

Comment: Thank you so much. I appreciate your effort on this topic. I had to do little adjustment but works well for me.

C# - TreeView - Data Algorithm


Can I include a list of sub-menu pages in the content of the parent page?

Question: I have a page that is the parent of a series of menu items, e.g. Projects, with some project pages as child items in that menu. I would like this parent page to automatically have the list of children in the content. Is this possible?
I have a block that I created that lists this menu via the Views module...if there is a way to use that, it might be an alternative.
Thanks
Answer: By some coincidence, there is an article on how to embed a block in the middle of your node body content on Drupal Planet today.
It seems to me, though, that if you put the block at the top of your content and floated it with css, maybe with a border attribute, the result would perhaps be better than adding content to an arbitrary location inside the content.


Number of inversions and length of a word in a symmetric group

Question: Let $w$ be a reduced word in the symmetric group $S_n$. Is the number of inversions of $w$ equals to the length of $w$? I checked some small examples and this is true. Is it correct in general? Thank you very much.
Edit: the symmetric group $S_n$ consists of permutations on $\{1,2,\ldots,n\}$. The   group $S_n$ are generated by simple reflections: $s_i=(i,i+1)$, $i=1,2,\ldots,n-1$. A word in $S_n$ is a word in $s_i=(i,i+1)$, $i=1,2,\ldots,n-1$.
Comment: What do you mean by a word? A word in what?
Comment: A "word" is not defined until you define the set of letters. Presumably you have some generating set in mind?
Comment: @arctictern It has to be of shortest length possible. This is equivalent to there *never* being the same simple reflection twice in a row even when one can rearrange the letters using the braid relations.
Comment: There are plenty of possible symmetric groups on words. Which one is it (elements of your group) and what is the operation of your group ?
Comment: In the edited version, the answer is yes. The number of inversions is precisely the length of an element when seen as an element in the Coxeter group with the usual set of generators. I don't have a reference handy, but I recall it being a not too hard induction proof on the length.
Comment: @TobiasKildetoft Out of curiosity, what other property does a word have to have to be reduced besides not having inverses next to each other? (Or is it possible "all reduced word for the same element have the same length" is true for the free group, not necessarily for quotients thereof like the symmetric group?)
Comment: (note that these terms need some extra structure to be well-defined, in this case the fact that this is a Coxeter group).
Answer: Yes the number of inversions of $w$ equals the length of $w$ when $S_n$ is viewed as a Coxeter Group with the simple reflections $s_i = (i,i+1)$ as a generating set. This is Proposition 1.5.2 in Combinatorics of Coxeter Groups by Bjorner, Brenti.
Answer: Taking the definition of word as a product of elements of the symmetric group (the set of all permutations on a set of n elements) and "number of inversion" as the number of inversions this said product would lead to if $S_n$ operates on a non-cyclically ordered set, the answer is no.
Take the word $w_1 = a_1 a_2 a_3$. It has the length 3 and some number of inversions. Now define $b = a_2 a_3$. $w_2 = a_1 b$ has the length 2 but still the same number of inversions.
Comment: The question clearly stated what generating set is used (the standard one).
Comment: Maybe you happened to try only with different transpositions as they generate $S_n$.


Trying to find a 3-variable generating function

Question: For months, I've been trying to find a closed form for a generating function for this three-variable sequence
$$a(G,M,T)=(-1)^{G+T+M} 2^{M-1} \frac{(M-1)!}{(2M-2)!} \binom{M}{T} {}_3F_2(M+1, -\frac{G}{2},\frac{1-G}{2};M-\frac{1}{2},M-T+1;1)$$
where $G\geq0, T\geq0, M\geq2$ and $G\geq 2T-2M$.
If the generating function is
$$f(x,y,z)=\sum_{G,M,T} a(G,M,T) x^G y^T z^M$$
what I have so far is this
$$f(x,y,z)= -\frac{2 \sqrt{\pi}}{1+x} \sum_{M=2}^\infty  \frac{\left(\frac{z}{2}(y-1)\right)^M}{\Gamma(M-\frac{1}{2})} {}_2F_1\left( \frac{1}{2},1;M-\frac{1}{2},\frac{x^2(1-y)}{(1+x)^2} \right)$$
and this
$$ f(x,y,z) = \sum_{k=0}^{\infty} 2^{k-\frac{1}{2}} (1+x)^{-1-2k} (x-xy)^k (z(y-1))^{\frac{3}{2}-k} \left( - \Gamma\left(k+\frac{1}{2}\right) + \Gamma\left(k+\frac{1}{2}, \frac{z}{2}(y-1)\right)\right) $$
without any luck summing the remaining series in both cases.
In the first case, I can use the form given to me in this answer but only the first part sums out to a closed form.
Similarly, in the second case, the part with the gamma function can be summed out relatively easily, but the part with the incomplete gamma function becomes problematic.
I've tried everything I know using Mathematica, Maple and I digged through endless papers with various hypergeometric identities, but to no avail, I could really use some help.
Why do I even believe this can be written in closed form? I can't give you a satisfying answer, I guess I believe it enough to spend months of work on it, but I might be wrong.
Comment: @metamorphy I got to something similar when I tried to use the lower incomplete gamma function and somehow ended up on integrating something with an error function, don't remember exactly. Could you perhaps expand it into an answer?
Answer: I only see an integral representation (like the first one here but incomplete).
The first expression for $f(x,y,z)$ contains a sum of the form (after $M=n+2$) $$F(u,v)=\sum_{n=0}^\infty{}_2F_1\left(\frac12,1;n+\frac32;u\right)\frac{v^n\sqrt\pi}{\Gamma(n+3/2)}.$$ Substituting the integral representation $${}_2F_1\left(\frac12,1;n+\frac32;u\right)=\frac{\Gamma(n+3/2)}{n!\sqrt\pi}\int_0^1 t^{-1/2}(1-t)^n(1-ut)^{-1}\,dt$$ here, we get $$F(u,v)=\int_0^1 t^{-1/2}(1-ut)^{-1} e^{v(1-t)}\,dt.$$
A number of expansions can be obtained from here. Say, the geometric series $(1-ut)^{-1}$ results in $e^v$ times a series with incomplete gamma functions (not the one you have though). Another one comes from $(1-ut)^{-1}e^{-vt}=\sum_{n=0}^\infty(ut)^n\sum_{k=0}^n(-v/u)^k/k!$.


Linear algebra with a linear model (Matlab)

Question: Given the equation
$$r = B + e(r\cos(\theta))$$
and the corresponding data:
$\theta: 0.88; 1.1; 1.42; 1.77; 2.14$ and $r:     3; 2.4; 1.65; 1.25; 1.01$
How do you input these data for matlab to solve for $B$ and $e$?
Comment: You know how to do a linear regression in MATLAB?
Comment: No, I do not. I don't understand the answer given at all.
Comment: You know how to solve a least-squares problem, don't you?
Comment: By hand. I don't see how that applies to this
Comment: Okay, I added a hint to my answer.
Answer: Cleve Moler's Numerical Computing with MATLAB has this excellent chapter on how to do least squares; you can adapt any of the methods discussed there. To give you a few nudges on how to do your code: you have r as a dependent variable, and you can construct a new independent variable rc=r.*cos(theta);, where r and theta are appropriately constructed arrays. You can use [] to form the columns of the matrix required for the linear regression, and then use \ to get the least squares solution.
Or, there's polyfit()...

OP seems to have some difficulty seeing how least squares applies here, so here's a hint on how to assemble the equations:
$$\begin{pmatrix}1&r_1\cos\,\theta_1\\1&r_2\cos\,\theta_2\\1&r_3\cos\,\theta_3\end{pmatrix}\begin{pmatrix}B\\e\end{pmatrix}=\begin{pmatrix}r_1\\r_2\\r_3\end{pmatrix}$$


After writing the highest voice, does it matter what order I write the rest of the voices?

Question: Suppose I am writing for a string quartet. Don't make any assumptions about what I'm writing, just how many instruments (violin, violin, viola, cello). Suppose I write the violin part for a small stretch of melody. I now want to write the other voices.
I read somewhere recently that it is best to write the outer voices before writing the inner voices. Seemingly in support of this idea, Stephen Stone in Music Theory and Composition
A Practical Approach states

"When writing a contrapuntal piece, all three voices should be equally independent and melodically interesting. The goal of this text, however, is to work toward understanding traditional harmonic progressions and part writing. In this light, it is appropriate to write "boring" inner voices; in fact, it is encouraged. While using many repeated notes and common tones in the outer voices leads to a dull exercise, using them in the inner voice is wise. If the inner voice does not move, it cannot create voice-leading problems." (page 128)

In Joseph Swain's Harmonic Rhythm: Analysis and Interpretation, he states:

"Acoustical science tells us that it is the lowest-sounding pitch, the fundamental, that determines the pitches of the other, higher partial frequencies, which some writers have cited to explain why the most stable chords are in root position. Indeed, the perceived strength of a chord's identity and its function derive partly from its inversion. The pitch of the bass has always remained conceptually distinct from the identity of the triad, and that allows the bass voice its melodic freedom; it is not entirely a slave to the chord progression. Changing the inversion of a triad is a harmonic change of some independence."

To me, these suggest that, after writing my first violin part, then writing the bass would come next, followed by the "inner voices." I'm confused why that is a good ideas.
I can see if I were writing a piece (suppose for vocals, so SATB) like the Hallelujah Chorus from Handel's Messiah, like this

Then it would make sense to write the outer voices first. They aren't that different rhythmically in this case.
But consider Haydn's String Quartet No. 62 in C Major Op 76 No. 3 Movement 2

The second violin here has substantially more notes and rhythmically mimics the first violin much more than the viola or cello.
So in this case, it would seem to me to make more sense to write the alto part first.
If a composer has written the highest voice already, under what circumstances would someone write the inner voices first before writing the bass?
Comment: The Haydn isn't strictly homophonic.  That's one clue to how you should apply the recommendation.  For the first two half-measures, the second violin _is_ the lowest voice.
Answer: I think you should consider the instruction to write the voices in a certain order is more of an educational tactic.
When taught that homophonic music is primarily melody with bass and inner voices as filler, then writing exercises which start with treble and bass then last add inner voice filler is just a way to make you learn the importance of melody and bass over inner voices via putting your attention to the more important stuff first.
In reality there could be all sort of levels of sketching and revising of parts. When compositing a homophonic work a composer isn't going to literally write all the melody first, then all the bass, then fill in inner voices.
Also, melody and soprano are often treated synonymously, but they aren't the same. Melody happens in all the voices. Even in homophonic texture, with one main melody, the melody can go in any voice. In a string quartet it's common for the melodic focus to shift around the four parts. If, for example, the melody is placed into the viola, it's likely a composers would work out the detail of that melody first and perhaps sketch out the other voices.
Comment: Also, the example given here is not homophonic, which is all the more reason not to take the recommendation too strictly.
Comment: Do you mean the two examples in the OP?
Comment: I meant the Haydn example in the question, which prompts the asker's comment that "it would ... make more sense to write the alto part first."
Answer: Your Haydn example uses the two violins to work in harmony, whereas the viola and cello are providing the 'backing'. So there, it's pretty clear that the two violin parts would need to be written simultaneously, or perhaps the 2nd violin straight after the 1st.
In your first example, the parts play differently. It's ensemble mainly, all the way. So, we have a melody line, next the bass part would underpin it, followed by some 'filling in' of harmony for the middle two. Rather different concept.
So, you'll have to decide which path your composition is taking. Is the 1st violin the only melody line? Is it working with the 2nd violin? Does the cello need a line of its own, going its own separate way? Is it working with the viola? Will 2nd violin and viola work as a team providing the harmonic part?
There is not, and can not be one formula for this sort of writing.
Answer: Write the viola part next, and make it interesting!   Viola spends too much time 'filling in', and the player gets tired of it.
OK, I'm kidding (mostly).  You seem to have decided the main melody is in Violin I for this section.  That's fine, but do let the others take a turn at playing the tune later on.   Next decision is what sort of texture you want accompanying it. Do you want all four instruments to play - duet and trio texture is good and leaves you 'somewhere to go' as the piece develops.  Or perhaps sustained harmonies would be good.   Or a rhythmic pattern?   Once you have a clear concept, you can start writing the other parts.  Probably bar-by-bar rather than trying to complete one instrument before adding the next (but is there a chance for a decent amount of melodic imitation by one or more of them - think on a broader scale too).
What I'm saying is that it isn't about adding parts in any particular order, more about planning what texture you want.
Answer: Pietro Aron wrote, in the 16th century (as quoted in Grout and Palisca, A History of Western Music, 5th ed. p. 162):

Many composers contended that first the cantus should be devised, then the tenor, and after the tenor the contrabass. ... Such compositions were bereft of sweetness and harmony, because when you write the cantus or soprano first and then the tenor, once this tenor is done, there is no place for the contrabass, and once the contrabass is done, there is often no note for the contralto.  If you consider only one part at a time, that is, when you write the tenor and take care only to make this tenor consonant, and similarly the contrabass, the consonance of every other part will suffer.

Therefore the moderns have considered this matter better... Every one of their parts occupies a comfortable, easy, and acceptable place, because composers consider them all together and not according to what is described above.

Learn to write all the parts at once.  Yes it's a bit of a challenge to keep three or four different lines in your head at once, but most people can learn to do that with practice.
Of course, this isn't meant literally simultaneously all the time (though I can improvise in 3 part counterpoint at the piano) - but you should not let any part get too far ahead of the others.


How to use ExtFilterDefine for png files in perl as one liner?

Question: Because of md5 hash scanning tools like wpscan, I want to prevent the script kiddies to detect as much information as possible about my wordpress site. With to following perl snippet, I m trying to add some extra characters to all requested png files. But it does not work and I don't know why. Does somebody can help me out?
My goal is not to change it right inside the files - just for requested output on screen.
ExtFilterDefine pngfilter mode=output intype=image/png cmd="/usr/bin/perl -pe 'END { unless (-f q{/tmp/md5_filter.tmp}) { print qq(\/*) . time() . qq(\*/) } }'"
I use the same snippet logic for css and js files. Here it works as expected.
Answer: It does work.
$ perl -pe 'END { print qq(/*) . time() . qq(*/) }' derpkin.png >derpkin_.png

$ diff <( hexdump -C derpkin.png ) <( hexdump -C derpkin_.png )
3023,3024c3023,3025
< 0000bce0  00 00 00 00 49 45 4e 44  ae 42 60 82              |....IEND.B`.|
< 0000bcec
---
> 0000bce0  00 00 00 00 49 45 4e 44  ae 42 60 82 2f 2a 31 36  |....IEND.B`./*16|
> 0000bcf0  35 36 33 35 30 37 37 36  2a 2f                    |56350776*/|
> 0000bcfa

At least, it works in the sense that it does exactly what you wanted it to do. But does it makes sense to add arbitrary text to the end of a PNG? I'm not familiar enough withe PNG file format to answer that.
Caveat: It will not work on Windows because of CRLF ⇔ LF translation.
Comment: Thanks ikegami. I guess it will work, because wpscan is a linux toll :-).
Everybody feel free to comment.
Comment: I want to use it for all requested png files like a filter. Not only one certain file like in your snippet above `derpkin.png >derpkin_.png`
Comment: So? the point was to show the program you posted works.
Comment: Could someone give me more tipps? My goal is, that if a png file will be requested, some extra string or timestamp should be added to change the md5 checksum of this file. But the changes should not apply right on the file. That's why I want to use something like ``ExtFilterDefine pngfilter mode=output intype=image/png cmd="/usr/bin/perl -pe 'END { unless (-f q{/tmp/md5_filter.tmp}) { print qq(\/*) . time() . qq(\*/) } }'"``
Comment: Yeah, never said not to
Comment: Please only correct answers to the question. Thanks
Comment: Please only questions that actually identify a problem. Thanks.
Comment: Question was posted in my original post. But thanks for help.
Comment: No, the Question did not identify a problem.


Using AdaBoost on multi-class in R on unbalanced data

Question: I have a data set which is highly imbalanced and I have used the SMOTE algorithm (using the R package DMwR) to balance the binary class in the data set. I have been using the R Ada package to then train an Ada Boost model on this data set to predict the binary class, with very good results.
In the same data set, I have another class variable which has multiple values (6 in total). In this case I realise that I can't use the AdaBoost algorithm as implemented in the ada package as it only deals with the binary case.
I therefore have 2 problems:

I'd like to use the SMOTE algorithm on second class variable but this also only works with binary classes. Is there an algorithm or package I can use in R to "rebalance" a data set based on a class with multiple values in a similar way to SMOTE?
I'd like to use a classifier to predict the multiple class variable. I have tried using the one-vs-all approach with AdaBoost but I cannot get this to work well (my approach is below). Boosting seems to work well with this data set. Are there any other boosting algorithms or other approaches I could use in R that handle classes with multiple values. I have tried using Random Forest but one of my nominal inputs has too many discrete values to use it.

Approach for AdaBoost one-vs-all

Build a vector with a binary variable for each discrete class value
Train one AdaBoost model against each binary class vector
Generate probability prediction for each AdaBoost model
Select the class with the highest probability

Many thanks
Comment: From my neural network experience, could you make the 6 values into binary values such as 1000000, 0100000, 0010000, 0001000, etc and then use adaboost. just a thought.
Comment: Has any solution to this been found?
Answer: You can use maboost package in R. It implements mutliclass boosting. Its multiclass boosting is in a sense the generalization of adaboost.MM. It directly solves the multiclass boosting without reduce it to binary classification problems and perhaps appropriate for your application.


How to get Datadog Alert data through API call

Question: I have set the threshold value to get the alert in data dog for infrastructure. Alert is coming on  data dog UI but how to get this all alert data through API call either using JAVA or python. I need only alert data.
Answer: I suspect you're probably looking to query the event stream which is where all alerts from monitors can be found.  The docs at https://docs.datadoghq.com/api/#events-get-all are a pretty good starting place.
You'll want to query this endpoint with the proper source and tags, but this should be a starting point.  If this doesn't quite work, I'd recommend looking at pulling the details from the monitor as shown here:  https://docs.datadoghq.com/api/#monitor-get-details. This may be a second option if you're unable to get the information you're looking for from the event stream.
Comment: Yes, the event stream should be where you want to go then. You'll want to likely query every minute or so and shift your timestamps to match this window. You can then parse out any events with the alert_type ("error", "warning", "info" or "success") you're looking for.

In the case of wanting all alerts, you'll want your sources to be filtered as `*`.  This should give you any and all events and you can programmatically filter out the ones with the appropriate alert_type.
Comment: agreed, although if all you want is the monitor alert events, you probably want `source_type_name` to be set to `alert` in addition to `alert_type` being set as `error` or whatever.
Comment: Thanks for your reply. I want  all the Alert not any Extra data
Comment: for both application and Infrastructures
Comment: Thank you very much
Comment: api.Monitor.get(2081, group_states='all')  .  from where i will get this Id 2081, beacuse we need to pass this id to get data
Comment: You can grab the monitor ID from the URL itself for example: https://app.datadoghq.com/monitors#1234567?group=all&live=1h the ID here is 1234567
Comment: Alert is showing warn or critical  even after agent is stopped . i mean when agent is stopped then alert should not come but even after agent is stopped  previous data is coming by calling    api.Monitor.get_all()
Comment: Can u please tell me that how can I get only Violated Alert through API call .


Interpreting Labour Economics regression model

Question: Instrumental variables (2SLS) regression
Number of obs =     603
F(  5,   597) =   41.96
Prob > F      =  0.0000
R-squared     =  0.1386
Root MSE      =  .26523
         |
 logwage |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]

  higher |   .2566441   .1030163     2.49   0.013     .0543257    .4589625
     age |  -.0054228   .0055142    -0.98   0.326    -.0162524    .0054068
    age2 |   .0000223   .0000675     0.33   0.741    -.0001103    .0001549
urban    |   .2587201   .0304102     8.51   0.000     .1989962    .3184441
   rural |  -.0207374   .0296745    -0.70   0.485    -.0790164    .0375417
   _cons |   2.370127   .1065349    22.25   0.000     2.160899    2.579356

Instrumented:  higher
Instruments:   age age2 metropolitan rural father_educated
I am doing an assignment titled does expansion of higher education improve the earnings? The case of Russia
Regression has been run as shown above, the regression model is based on the widely used Mincer Model with minor modification and proxy to it.
Can anyone of you who are kind enough to assist in term of what do you think of the model and its output? And how would you interpret the output?
Comment: You might want to use the robust option to calculate heteroscedasticity robust standard errors. People with little schooling tend to have less variance in their earnings than those with lots of education, so the error variance is not going to be constant. Also the rural dummy can be problematic because the choice of living in cities is potentially endogenous.
Answer: I suppose that higher stands for years of schooling. If this is the case, what you are doing is correcting for the endogeneity problem likely to affect the variable higher. That is, a variable is said to be endogenous when it is likely to be correlated with unobserved factors in the error term. Therefore you are instrumenting it by doing a 2 stages estimation. That is, your Stata command implicitly did the following:
1) It regressed higher on all the exogenous regressors plus the instrument father_education. That is, you assume that having more years of schooling depends on the education of father's. By doing this, you clean your endogenous variable of the endogeneity problem.
2) It runs your original regression using the instrumented (fitted values of the) variable higher as regressor, in place of the original one.
By doing this, the coefficient on higher is now a consistent estimator of the true parameter. The interpretation of the results must account for the fact your dependent variable is in logs, while higher is not. Therefore,  a year more of schooling increases (on average) the wage by exp(.2566441)= 1.292585 = 29% and this effect is statistically significant, since p-value<0.05.
Note that this is true as long (1) there is a statistically significant relationship between higher and father_educ (you can test this by verifying that the coefficient on father_educ in the first stage is statistically significant (and positive)), (2) the instrument father_educ is not correlated with the error term.
Comment: Whops. I made a mistake in the writing, still for the percentage change I have to take the exp(), since the dependent variable is in logs, the independent is not. `exp(.2566441)= 1.292585 = 29% increase`
Comment: I don't think your interpretation of the coefficients is correct. The Mincer equation is a log-linear model, so the coefficients are semi-elasticities but you are calculating elasticities by exponentiating them. So in this case a schooling coefficient of .2566 translates into approximately 25.66% higher earnings for an additional year of education.
Comment: Exactly. Also my quick approximation by multiplying the coefficient with 100 was poor because this only holds roughly for $\beta < |0.1|$
Comment: However, I think that your reasoning was not completely wrong. Indeed, a log variation, i.e. a log difference, is an approximation of a percentage change. Therefore, 0.2566 translates in a % change of the dependent variable due to a UNITARY change in the independent variable X. The exp() transformation should change the interpretation in a percentage change in y due to a PERCENTAGE change in X. Am I right ?
Comment: Hi all, thanks for your guidance and advice. Yes, I did run robustness test for that. Thanks for your reminder.

May I know why would it be the case where age which is supposedly to be positive is now negative while age2 is now positive and is totally the opposite of what it normally was.

What can we infer from there then? Any thought?
Comment: A significant value of `age2` implies a non linear effect of the variable. However, your result sounds strange. Typically it should be age = positive and age2= negative. If you get the opposite it means that Age negatively affects the wage, until some Point from which the effect is reversed. Think at it as a `U` shaped effect. If age was positive and age2 negative, then the effect would have been `reversed U` shaped


Indexing image url in search api drupal 8

Question: How to index the absolute image url instead of image uri in search api drupal 8.
Image field formatter renders the image url but there is no option to index the url part.
Answer: It looks like that should be available as a field.

If your field name is "field_image", you would select "field_image:entity:uri".  I think that would give you "public://path-to-file-in-public-directory.jpg"
You could add a custom field to your index in a custom module at src/Plugin/search_api/processor:
In this way, you can add any sort of custom value that's not available by default in the fields provided.


What are equivalents to literary theories for music?

Question: Are there equivalents to literary theories for music ? I'm trying to find extensive music review "theories" which would be arguably philosophical ( similar to for example structuralism/psychoanalysis/deconstruction etc..) and involves analyzing the music it self rather than the lyrics. Who are some canonical thinkers/philosopers like Roland Barthes/Freud/Jacques Derrida etc.. for music?
I'm finding it difficult to get a starting point for googling.
Comment: I think Schenkerian analysis might be an example of what you're looking for, but there's a problem which is that Schenker developed his analytic method in order to advance his white supremacist views and "prove" that white European music was the best music made in the world.
Comment: "Philosophy of Music"
Comment: @ToddWilcox thank you for introducing Schenkerian analysis! excited to explore more..
Comment: @ToddWilcox - I'm as anti-white supremacist as the next person, but we still listen to, and play (or sing) Wagner.  Do you have a link that will convince me that the Schenker analytical approach should be thrown out like the baby with the bathwater?  At some point, I started a master's at Ann Arbor (a hotbed of radical leftism) and when I took a class that introduced me to the Schenker approach, there was no political propaganda in the class, whatsoever.  cf. Freud was sexist, but his contribution was still important, no?
Comment: @aparente001 I have no desire to convince you of anything (you could have easily googled and found several sources on your own by now) and I didn't say anything about throwing anything out. Just putting an asterisk next to Schenker. And I feel like Freud and Wagner might not be totally analogous because their work was not solely intended to further white (or male, or whatever) supremacy. Schenker developed his theories to be deliberately biased. The fact that US universities heavily rely on Schenker is not validation, it's a big systemic problem.
Comment: @ToddWilcox - A couple years ago I came across something about the guy and his extremist political views, and read the quasi-whistleblower paper.  I thought from the way you were talking, there was maybe something new. / I went to Indiana and they did not teach the Schenker approach.  I think U of Mich is an outlier.  Even if it were as you say, I don't understand why the guy's music analysis and his politics can't be separate.  As they were in the course I took.
Comment: @aparente001 Not sure how to make it more clear that his theories of music analysis are inherently political. His goal was to advance a political agenda through music analysis.
Comment: @ToddWilcox - Okay, those were his goals.  My question is, do we throw out the baby with the bath water (his contribution to music analysis due to his immoral motivation)?  I wasn't sure if that's what you were suggesting in your first comment.  I honestly wasn't sure and I'm still not.  You can do that if you want to, of course.  (I wouldn't.)
Comment: @ToddWilcox I agree it may be something to keep in mind, but his conclusions should be treated on their own merits. Otherwise it is an example of the Ad Hominem fallacy in action - discrediting an argument source to avoid dealing with the argument's content. If a tool is created for some racist motivation, but the tool still functions when lifted out of that context and put in the hands of a non-racist, the creator's racist views are irrelevant. Despicable as they may be.
Comment: @magic_diagram My point is that the argument's content is racist. And that's not my personal interpretation. That's the intention of the argument according to its author. The analysis method inherently derates non-European music. It was designed to do that. So it doesn't matter who uses it. It was designed to be racially biased by its creator. His conclusions are that music made by white people is inherently better than music made by non-whites, so you can treat that conclusion based on its.. umm… "merits". Or lack thereof in this case.
Comment: @aparente001 In your view, what's the baby and what's the bath water in this case? To me there isn't a bad thing that is linked to a good thing. There's one thing that many people think is good but it has a built in problem. Some people value Schenkerian analysis despite that fact that the very analytical method itself was designed to confirm its creator's biases. I'm suggesting people be wary of using Schenkerian analysis because either they only analyze white European music (which would be bad) or their analysis would have problems with non-white music (also bad).
Comment: @ToddWilcox - Well, is there some single analytical approach which yields helpful insight when applied to *all* types of music? / The baby thrown out with the bathwater = his contribution to music analysis thrown out with our rejection of his political motivation when he was being creative.  (Please see what I put in the parentheses, above.)
Comment: @aparente001 I'm suggesting his "contribution to music analysis" is the bath water. Not the baby. It's not the good thing. It's the bad thing. I don't see a good thing related to Schenker. No baby, only bath water. He wasn't being "creative". He was being white supremacist.
Comment: @ToddWilcox - Do you reject his theory because (a) you've given it a fair chance but you see no intrinsic value, or (b) his racist ideology taints his theoretical work too much for you to give it the time of day, or to allow it to stand even though you would find it useful if not for the author's tainting ideology?  If (b), you're of course entitled to your opinion; I'm seeking to understand your original strong comment, and your subsequent comments have left me with this uncertainty.  (Thanks for trying, though.)
Comment: @aparente001 I will not comment further on this
Comment: @ToddWilcox - Okay!  Total confusion - suit yourself!
Comment: I want to thank you both for being so civil in your discourse! My last piano professor (RIP) was a great believer and practitioner of Schenckerian analysis, and I'm certain he was in no way a white supremacist. He concertized around the world for decades, and was a full professor at Cal State Northridge, and simply found the analysis method highly useful for understanding and approaching WESTERN music from the macro to the micro perspective. I personally believe it has inherent value in that respect, and simply isn't relevant to most non-western music.
Answer: German philosopher Theodor W. Adorno (1903-1969) wrote extensively about music. He was a member of the Frankfurt School, and heavily influenced by Sigmund Freud, Marxism, and Hegel's dialectic. He was also a pianist, and he studied composition with Alban Berg.
His most important writing on music has been compiled in the book "Essays on Music". As the publisher puts it:

Theodor W. Adorno (1903-1969), one of the principal figures associated
with the Frankfurt School, wrote extensively on culture, modernity,
aesthetics, literature, and—more than any other subject—music. To this
day, Adorno remains the single most influential contributor to the
development of qualitative musical sociology which, together with his
nuanced intertextual readings of musical works, gives him broad claim
as a continuing force in the study of music.
Comment: thank you! I'm so excited to explore his works..
Answer: Yes.
Over the centuries of European music theory there were many theorists and treatises that could be named.
Probably one of the biggest names to look into, especially in terms of linking ancient music theory with modern theory, is Jean Philippe Rameau and his Treatise on Harmony, where he introduced the theory of root progression.
Questions asking for sources are supposed to be off topic, so your question may get closed soon.
Comment: Yes I meant it, but the grammar was pretty mangled. I edited it.
Comment: Would you mind proofreading?  I could dive in but I might misinterpret some part of what you wrote.
Comment: @aparente001, proofread what? Dive into what? Do you see a mistake in my answer?
Comment: Do you mean "over the centuries of theory there are many people and treatises that could be named" for example?  I'm just not sure enough about what you are trying to express to try to fix your first sentence myself.


How important is the material a guitar slide is made from?

Question: Most slides seem to be glass. What difference does the material itself make to anything, if any? Copper, stainless steel, brass - or are any other better materials used? For what reasons?
Comment: I believe the most important characteristic of the material is its mass. You want as little of the energy of the string to dissipate into the slide. Other than that, I've never heard a difference between glass or steel. But I have not enough experience for a complete answer.
Comment: I wonder whether different materials are partly about friction as you slide? I haven't really tried, but I imagine steel to be less and glass to be more?
Comment: https://happybluesman.com/how-choose-right-guitar-slide/ seems to have a lot to say about tonal characteristics.
Comment: Probably stiffness (Young's modulus) will play a role: even a heavy rubber will dampen the sound a lot more than a light and stiff (say glass) material.
Comment: Agree that stiffness & hardness matter, since you only want the slide to force a node on the string at the contact point.  The slide will (or should) never be a resonating body.   <--,  well, unless you want it to act as a sort of megaphone, in which case how you hold it will matter a LOT.  I'll add that you could deliberately choose a soft material if you wanted a quieter sound with less of the higher harmonics.
Comment: Not strictly "slide playing," but there are some sound effects using slide that *might* benefit from the little bit of texture that steel slides have. Thinking of e.g. Roger Waters use of a slide on his bass. Though I cannot see it clearly to be sure what material it is made from (approx 51:40 https://www.youtube.com/watch?v=TfBNJLdVvoc ) Never tried doing that with glass.
Answer: In my own experience, the material matters less so than how comfortable the slide feels on your finger.
If it's hard and very much smooth, then glass is as good as copper is as good as steel or aluminum.
Don't concern yourself too much about what somebody else says about the tonal qualities of this material versus that.
If it feels really comfortable and intuitive on your finger, you will use it, and you will get really good at it.


Which electric piano tone on keyboard can have good accent and can be played loud at the same time?

Question: I have to play a song which has electric piano tone which has a good distinct accent sound and can also be played at different dynamics.
The problem is that you've to hit the key hard to make it accented (considering touch response here), which makes that note loud and vice-versa you have to hit the key hard to play loud, so you cannot play a note which is loud but not accented or vice-versa accented but not loud. I want to solve this problem.
In my keyboard, I can change the voice/tone dynamics, by changing its reverb, vibrato, touch response (so much that I can even play an organ soft or loud, without the swell pedal), etc. So, if I know that there is some setting which controls the accent of a tone, that'll help me.
EDIT: My Keyboard is YAMAHA PSR-S-670.
Please help me with it.
Comment: It sounds like you are trying to play a note louder but not accented? My understanding of an accented note is that it is a bit louder than the notes around it so louder and accented are the same thing? Perhaps what you want is an EP sound that doesn't have the "bark" when you play it hard enough? If that's the case then this site isn't right place to ask to find a specific kind of sound, and also the "bark" on an EP is a widely desired quality so it will be hard to find. You could try to build one from samples or something.
Comment: Would you consider getting sound from a computer (e.g. plugin instrument with effects), or does it have to be pure hardware solution? There's a lot you can do with virtual instruments. Also I imagine you need some additional controller except from the keyboard, e.g. a pedal or wheel, as otherwise the only articulation you can get from keyboard is velocity (and maybe aftertouch, which rather won't help you).
Comment: They solve this in the 'real' world with a compressor. Some instruments have them built into the effects chain, but not all. You'd have to tell us which keyboard you have. Without it, then directly from the instrument's own speakers, this cannot be 'fixed'.
Comment: On some of my kbds, there are various Rhodes sounds which have good attack when the keys are hit harder - can't say how you'd do it with whatever keyboard you'll use, but it's worth checking out the (usually) several Fender Rhodes sounds, and use different attack as you play. No electronic effect, it's just a built in sound - on mine.
Comment: If you're asking for help on configuring your keyboard differently, please add in the exact make and model of the keyboard.
Comment: My keyboard is YAMAHA PSR-S-670.
Comment: How is this different from "loud vs. accent" on an acoustic piano?
Comment: Not much. But, the accent on electric piano is more distinctive than acoustic piano, imo.
Answer: An option would be to play the accentuated note more loud than the other loud notes, but from your post I get the idea that you have already tried out something like that. I don' t know how nuanced you can play on your keyboard model. It might be hard to make a clear difference. So here is another idea:
A church organ player once told me a little trick you can do on an organ. On the organ the keys are not touch sensitive, and yet you can accentuate a note. The trick is to play the note staccato. Try it out, it might work for you.
Comment: Yes, actually, the notes I'm supposed to play accented in the song are staccato!! So, not a problem there. And I can show the difference between accented (moderately loud) and loud notes, but it's not very distinctive. The problem is that the accented notes aren't the loudest there, so somehow I have to play them softer and still manage to play them accented.
Comment: @Ishan.J  you are giving conflicting information. A note with an accent is a note that stands out compared with the other notes. But now you are saying that a note with an accent should be softer than than the other notes. That doesn't make sense.
Comment: Yes, it does. In the song, I'm playing, the accented notes are supposed to be mezo forte while other parts are supposed to be forte. It is conflicting right?? And that's the problem. The accented notes have to stand out from others but at the same time, they should also be moderately loud - mezo forte, not loud I.e. not the loudest in the piece.
Comment: So are you saying that is not a matter of a note  that has an accent, but a whole section of notes where all the notes are accentuated.?
Comment: Yup. And another part which is supposed to be played louder than the accentuated notes.


scipy.stats.t.logsf infinite values

Question: I have a t score of 69.667, when my degrees of freedom are 726 logsf returns the following:
>>> scipy.stats.t.logsf(69.667, 726)
-744.4400719213812

However at df = 727 I get the following
>>> scipy.stats.t.logsf(69.667, 727)
-inf

Can someone explain how I am hitting the number limit here? The reason I used the log survival function is to avoid this issue. Is there any other way to approach this problem?
Answer: The implementation of the t distribution in SciPy doesn't override the _logsf method, so it defaults to computing the log of the sf function:
In [24]: t.sf(69.667, [726, 727])
Out[24]: array([5.e-324, 0.e+000])

In [25]: np.log(t.sf(69.667, [726, 727]))
/.../ipython:1: RuntimeWarning: divide by zero encountered in log

Out[25]: array([-744.44007192,          -inf])

By the way, 5e-324 is the smallest magnitude nonzero 64 bit floating point number (it is a denormal number), so that result has just 1 bit of precision.  Don't put too much faith in all those digits in -744.44007192 returned by t.logsf(69.667, 726).

scipy.stats.t.logsf infinite values


How should virtualenv be set up in a production Web server (user, location, etc.)

Question: I'm switching my production sever to use virtualenv, and I want to understand the best practices for setting up virtualenv in a production environment. The following questions are ones I've run into today while trying to figure this out:

what user should I use to create the virtualenv?
where should that user put the virtualenv?
how do I activate the virtualenv with the permissions of that user?

Virtualenv is going to be used almost exclusively by Apache, so I'm thinking the correct user is www-data, and the location of the virtualenv could be /var/www/.virtualenvs.
That seems OK, but the next problem is that www-data doesn't actually have a login shell and can't write to its own home directory (/var/www/), so when I try to activate the virtualenv, it doesn't work, and when I try to install things using pip, I get errors like:

The directory '/var/www/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

(even if I use -H I get this error.)
So, I could create a login shell for www-data, but that feels like a security anti-pattern.
And I could give www-data access to its home directory (/var/www/), but that's obviously a security anti-pattern.
Is there a norm around this? I've burned WAY too many cycles trying to figure this out, and it feels like it should be much simpler.
Comment: Did you ever figure this out?
Comment: Nope, not really. Maybe we need a bounty.
Answer: Running in a virtualenv is no different that any other application. You use a user that has the least amount of privileges necessary to do what the application needs to do.
It would not be advisable to run it as yourself.
Comment: And I suppose the next question is **where** should the virtualenv be placed?
Comment: In practice, then I suppose that becomes `www-data`?
Comment: I updated this with more details, in case you know where to go from here. I'm pulling my hair out trying to do something that should be quite simple.


Atualizar Div HTML PHP Javascript

Question: Estou com alguns problemas ao atualizar uma div sem atualizar a página inteira.
Segue o código:

          <?php echo file_get_contents('arquivo.txt'); ?>

...

Já pensei em criar um "frame" - não sei bem se esse seria o nome - e usar "meta refresh" para atualizar dentro dele.
Minha duvida é: Como atualizar esse Div usando o PHP ou Javascript? E, se possível, teria como atualizar esse Div sempre que houver uma mudança no "arquivo.txt"?
Atualização
O "arquivo.txt" é atualizado normalmente:
$texto = htmlspecialchars($_POST['text']);
$string = $texto . "\n";
$fp = fopen('arquivo.txt', 'a');
$fw = fwrite($fp, $string);
fclose($fp);

Comment: Como/porque é que esse "arquivo.txt" é alterado?
Comment: Você pode usar `.ajax` ou `.get` do jQuery. Veja [aqui](http://pt.stackoverflow.com/questions/54525/atualiza%C3%A7%C3%A3o-din%C3%A2mica-sem-atualizar-a-p%C3%A1gina) uma pergunta relacionada com `getJSON`.
Comment: O arquivo.txt seria como um "banco de dados". Os usuários enviam o conteúdo para lá, em vez de usar um banco de dados normal. Lá esta tudo configurado e organizado de acordo com a saída que eu quero.
Comment: Isto é algum sistema para teste ?
Comment: Sim, seria só uma "experimento".
Comment: Também recomendo o uso do Ajax
Answer: Tente isso:
index.php
<!DOCTYPE HTML>

    <meta charset="UTF-8">
    <title>Index</title>

    Dados:
    <input name="text" id="text" value="#Teste">

    <button type="submit" id="btn_submit">Confirma</button>

script.js
$(function () {
    $('#btn_submit').click(function () {
        grava();
    });
    $.post("grava.php", {},
    function (resp) {
        $('#resultado').html(resp);
    });
});

function grava() {
    $.post("grava.php", {
        text: $("#text").val()
    },
    function (resp) {
        $('#resultado').html(resp);
    });
}

grava.php
<?php

if (!empty($_POST['text'])) {
    $texto = htmlspecialchars($_POST['text']);
    $string = $texto . ";\n";
    $fp = fopen('arquivo.txt', 'a');
    $fw = fwrite($fp, $string);
    fclose($fp);
}
echo file_get_contents('arquivo.txt');

OBS: dê permissão de gravação no diretório.
Answer: Utiliza o setInterval() para carregar o arquivo com intervalo de tempo.
Html:

    Texto desatualizado!

JQuery:

O .html(answer); vai colocar a informação recebida entre as tags de abertura e fechamento respectivas  e

Atualizar Div HTML PHP Javascript


Problema com includes usando urls amigáveis

Question: Estou com um problema que já está me deixando doido, estou usando urls amigáveis em um sisteminha que estou fazendo e os arquivos css,js etc não estão carregando, já vi vários casos aqui que isso acontece por causa de não colocar o caminho de forma correta, mas isso eu já concertei e todo página eu estou usando incluo um arquivo chamado config.php que tem o diretório base e a url do site armazenados em variaveis:
<link rel="stylesheet" href="<?php echo $url_base; ?>/painel/css/font-awesome.min.css">

Arquivo config.php:
<?php
$path_base = dirname(__FILE__);
$url_base = "http://localhost/iesav2";

Arquivo .htaccess está assim:
RewriteEngine on
RewriteCond %{SCRIPT_FILENAME} !-f
RewriteCond %{SCRIPT_FILENAME} !-d

RewriteRule ^error404/?$ 404.php [NC,L]
RewriteRule ^adm_usuarios/?$ adm_usuarios.php [NC,L]
RewriteRule ^login/?$ login.php [NC,L]
RewriteRule ^adm_usuarios/([0-9]+)/([a-z0-9-]+)/?$ adm_usuarios.php?nivel=$1&pesquisa=$2 [NC]
RewriteRule ^adm_usuarios/([0-9]+)/?$ adm_usuarios.php?nivel=$1 [NC]
RewriteRule ^adm_usuarios/([a-z0-9-]+)/?$ adm_usuarios.php?pesquisa=$1 [NC]
RewriteRule ^(.*)$ index_friend.php

O arquivo index_friend.php onde as paginas são direcionadas está assim:
<?php
ini_set('display_errors', 1);
ini_set('display_startup_erros', 1);
error_reporting(E_ALL);
require_once '../config.php';
//Pega toda url apos o dominio e transforma em array
$url = explode("/", $_SERVER['REQUEST_URI']);
//Pega diretório base e transforma em array
$path = explode("/", dirname(__FILE__));
//Compara a url com diretório base e remove o que não precisa
$url = array_diff($url, $path);
//Reseta indices do array
$url = array_values($url);
//Remove extensoes do nome do arquivo
$url[0] = str_replace(array('.php','.html'), '', $url[0]);
//Nome do arquivo requisitado
$file = $url[0].'.php';
//Se o arquivo existir então faz include para dentro
if(file_exists($file)){
    include ($path_base.'/painel/'.$file);
}else{ // Se não exibe error
    include ($path_base.'/painel/404.php');
}

Sem usar a pagina index_friend.php e acessando diretamente por exemplo http://localhost/iesav2/painel/login os arquivos carregam normalmente.
Answer: Não é necessário colocar url no link para incluir.
Basta deixar no modelo abaixo:
  <link rel="stylesheet" href="/painel/css/font-awesome.min.css">

Daí você olha no código fonte do navegador para ver se realmente está redirecionando para o arquivo css.


Mapping ID's to Names and Removing Duplicates Algorithmic Development

Question: I have a column in a CSV file that has names such that each cell in that column could be the same as a slightly misspelled cell.  For example, "Nike" could be the same as "Nike inc." could be the same as "Nike Inc".
My Current Script
I've already written a program in Python that removes prefixes and suffixes from
each cell if that value occurs more than 2 times in the column as prefixes or
suffixes. I then compared one row to the next after sorting alphabetically in
this column.
My Current Problem
There are still many cells that are in reality duplicates of other cells, but they
are not indicated as such.  These examples are:
    a) Not exact matches (and not off just by capitalization)
    b) Not caught by comparing its stem (without prefix and without suffix) to
    its alphabetical neighbor
My current Questions
1) Does anyone have experience mapping IDs to names from all over the world
(so accents, unicode and all that stuff is an issue here, too, although I managed
to solve most of these unicode issues)
and has good ideas for algorithm development that are not listed here?
2) In some of the cases where duplicates are not picked up, I know why I
know they are duplicates.  In one instance there is a period in the middle of a
line that is not present in its non-period containing brother cell. Is one good
strategy to simply to create an extra column and output cell values that I suspect
of being duplicates based on the few instances where I know why I know it?
3) How do I check myself? One way is to flag the maximum number of potential
duplicates and look over all of these manually. Unfortunately, the size of our
dataset doesn't make that very pretty, nor very feasible...
Thanks for any help you can provide!
Answer: Try to transliterate the names, to remove all the international symbols, then consider using a function like soundex or http://en.wikipedia.org/wiki/Levenshtein_distance (e.g. http://pypi.python.org/pypi/Fuzzy) to calculate text similarity.
Comment: Thanks for pointing me in the right direction. I just read a bit more and it looks like double metaphone will be the best place to begin. Is there a good function you know of to do the transliteration or is that just me doing a bunch of "if" statements since there aren't that many characters involved?
Comment: nvm, i think i'm going to use unicodedata or unidecode, just need to figure out which one I want...


How to fight underfitting in deep neural net

Question: When I started with ANN I thought I'd have to fight overfitting as the main problem. But in practice I can't even get my NN to pass the 20% error rate barrier. I can't even nearly beat my score on random forest!
I'm seeking some very general or not so general advice on what should one do to make his NN start capturing trends in data.
For implementing NN I use Theano Stacked Auto Encoder, the code from tutorial that works great (less than 5% error rate) on classifying MNIST dataset.
It is a multilayer perceptron, with softmax layer on top with each hidden later being pre-trained as autoencoder (fully described at tutorial, chapter 8).
The number of input features are ~50 and output classes ~10. NN has sigmoid neurons and all data normalized to [0,1]. I tried lots of different configurations: number of hidden layers and neurons in them (100->100->100, 60->60->60, 60->30->15, etc,etc), different learning and pre-train rates, etc.
And the best thing I can get is %20 error rate on validation set and %40 error rate on test set.
On the contrary, when I try to use Random Forest (from scikit-learn) I easily get %12 error rate on validation set and %25(!) on test set.
How can that be that my deep NN with pre-training behaves so badly? What should I try?
Comment: You do not say how many data-points you have but I suspect you need more data. Deep neural networks excel in huge datasets.
Comment: Which dataset are you training on? Not every neural net works for every problem
Answer: The most obvious issue that I see is the size of your hidden layers. They're generally trained on a GPU and the training can take hours, days, and sometimes weeks.
The "capacity"of your network is not large enough to fully represent the underlying variability in the data (MNIST). Try increasing the size of your hidden layers.


Creating Gantt Chart in SSRS 2015 and Data Will Not Display

Question: I have the following result set giving me a specific status of an item .  I need to build a SSRS 2015 Gantt chart to represent this data.  I am having difficulty getting the data to display.
Result Set: StatusDate,Status,BegDate,EndDate, StatusDays
In the chart, I want the category to be on monthly intervals of the range provided (BegDate – EndDate) and the series to be each DAY in the range provided (BegDate – EndDate).
The bars should represent the item being Up or Down with "Up" being Green and "Down" being black for each day (x-axis) of each month (y-axis).
Hoped for Results
The current preview shows the X & Y axis' correctly, but does not show any "bars" to represent the data.
This is my first chart using SSRS and I am quite lost.  Does anyone know where I can get a true tutorial on Range Bar Charts for SSRS 2015 or how I might overcome this "data Display" challenge I am stuck on?  I appreciate any direction / guidance offered.
Comment: You've posted 2 identical images, none of which offers much insight into your data unfortunately. Can you edit your question and add a sample of the data as it comes from your dataset.
Comment: @Alan Schofield - Thank you.  I have added the result set as an imbedded pic.  Am a bit of a Stack novice when posting so please be patient :)
Comment: Just a bit more... Based on the sample data you posted, can you post your expected output. There doesn't seem to be a lot to build a gantt chart from to be honest, but I noticed in your design you are referencing fields that I don;t see in your sample data (statEnd, statStart). Anyway, make sure your example output ties up with your sample data so it's easier to understand. Do the expected output in anything, Excel ,Paint or whatever just so I can see the image.
Comment: @Alan Schofield - I updated the post but due to my Newbie limits, I had to remove one imbedded pic for another.  Basically, I "mapped" out a Gantt chart.  I hope this helps.
Comment: Sorry @JustPeachy Had no time today, work busy. I'll follow up tomorrow hopefully. P.S. You in B.Hill?
Comment: @AlanSchofield No worries.. I was able to play with the chart and get it to where I need it.  I followed this video: https://www.youtube.com/watch?v=4dhgOLKuJPA and it helped get me where I needed to go.  P.S. B. Hill?  Must not be :)
Answer: This issue was ultimately resolved by @AlanSchofield through another (and probably more clear) question, found here.
After choosing the correct chart, setting the correct properties, and determining the correct values I was able to get the chart I needed, but the behavior was still off.  It would not show the a status more than once.
To fix this, and simply put, I needed to sort my data using Dense_Rank and then add the new "sort" column to my Series Group in the Chart properties.  Worked like a charm!
Also got some useful guidelines on this blog post.
Comment: @DarkCygnus  To get the first answer I followed most of this blog post: http://pnarayanaswamy.blogspot.com/2012/12/ssrs-2008-r2-range-bar-chart-gantt.html.  This got me part of the way there, but to show the data (repeating status) I used Alanschofield 's recommendation of using Dense_Rank as shown here in this question: https://stackoverflow.com/questions/48283688/ssrs-range-bar-chart-not-displaying-repeated-status-based-on-date-range
Comment: Mind including that question that solved your doubt? For purposes of documentation and reference
Comment: Thanks, much more useful. If you don't mind, I'll edit those links to your post, as comments may be deleted without notice and this useful clarification lost.
Comment: Any help I can get is much appreciated.  I am still learning my way around this community as a participant instead of just a spectator.
Comment: Done :) it's ok, good thing you decided to contribute for a change. For future reference, your answer was ok, however, more detail (code snippets, links to other questions, links to outside resources *with* description of the solution proposed there, etc.) is always highly valuable. This will make your posts more useful to the community and as well give you better chances of them to be taken positively. Cheers and good luck


Is the Schröder Equation valid for higher dimensional iterated maps?

Question: The Schröder's functional equation is the eigenfunction equation for the composition operator given as:
$$
\psi \circ y (x) = s \cdot \psi(x) ~~~~~~~~~~~ (1)
$$
The interesting bit about it (at least for me) is related to the iteration of a function over a point. Suppose that $y,x \in \mathbb{R}$. If we denote $y(y(x)) = y \circ y(x) = y_2(x)$ we can extend the notation like so $y_n(x) = y \circ y_{n-1}(x)$ and, more generally $y_{n+m}(x) = y_n \circ y_m(x)$. It turns out that once a Schröder's conjugate $\psi(x)$ is found for $y(x)$, it is possible to compute the $n$-th composition of $y(x)$ straightforwardly like so:
$$
y_n(x) = \psi^{-1}(s^n \cdot \psi(x))
$$
By computing $y_n$ like this, not only the integer values of $n$ are well-defined, but also all the intermediate values, turning $n$ into a real parameter and the iterating function $y_n(x)$ into the Flow $y(n,x)$. Meaning that the entire orbit containing all iterates of $y$, including the fractional ones, is obtained. This equation is often solved for an arbitrary function $y(x)$ using the Carleman matrices to turn composition into multiplication.
Handy as it is, I could only find discussions about the Schröder equation, and its solution strategy, in the one-dimensional domain. So my question is whether there is a more general version of the Schröder's Equation for iterated maps of greater dimensionality, which generalizes equation (1) like so:
$$
\vec \varphi \circ \vec u(\vec x) = s \cdot \vec \varphi(\vec x) ~~~ \forall \vec \varphi, \vec x, \vec u \in \mathbb{R}^n, s \in \mathbb{R}.
$$
And if there is, whether or not we have a quasi-Carleman method to solve it as well.
Answer: Frankly, I have not seen many multivariate techniques for functional iteration. You might consider experimenting with powers, emulating Schroeder's technique. Take two variables, x and y, for simplicity.
Again for simplicity, consider diagonal vector functions, e.g.,
$$
\vec u (x,y)= (x^2,y^3) ,
$$
so
$$
\vec u_n(x,y)=(x^{2^n},y^{3^n}).
$$
I guess
$$
\vec \varphi (\vec u (x,y)) = \operatorname{diag}(2,3) ~~\vec \varphi (x,y)
$$
would do the trick for $\vec \varphi (x,y)=(\log x , \log y)$, so, by functional conjugacy,
$$
\vec u _n (x,y)= \vec \varphi^{-1}(\operatorname{diag}(2^n,3^n)~\vec \varphi(x,y) ).
$$
Of course, for a scalar Schroeder eigenvalue as you have, you need the diagonal eigenvalue matrix to be proportional to the identity, so both entries 2 or 3 in our case. I took the slight generalization here to relax this restriction, even though there is no essential coupling between the two modes, so their orbits simply do not influence each other.
As for perturbative techniques around fixed points, given the suitable equation arrays, such are always available. But make no mistake, the above diagonal generalizations are trivial, i.e. they are simply addressing a multiplexing of the scalar equation of one variable, with innocuous parallel moves and no essential coupling. In point of fact, the renormalization group equation for multicoupling QFT systems is of this form, but people bypass this entire structure in favor of coupled ODE systems—not functional equations anymore. I can see you asking for more elaborate constructs next...
Comment: One thing that caught my eye was the use of the "diag" symbol. The way I am understanding this is that you are working in the speific case where $s$ is a diagonal matrix. So for a multivariate Schroeder Equation I would have, in general, instead of a scalar eigenvalue $s$, an "eigenmatrix" $\bf S$ for which $ {\vec u}_n = \vec \varphi^{-1} ( {\bf S}^n \cdot {\vec \varphi}(x,y) ) $ ? In that case I could only find a continuous orbit for $\vec u$ if $\bf S$ is diagonalizable, right? So that I can apply fractional coefficients to it.
Comment: Yes. A diagonal "eigenmatrix". This is a simple tractable paradigm, and your appreciation of it is right, but it is ambitious to talk about what one "could only" do.... Imagination rules supreme...
Comment: So is it right for me to conclude that the reason the proof for the one-dimensional Schröder's Equation validity does not extend to higher dimensions is because it stops being an eigenvalue equation and becomes this weird "eigenmatrix" equation? I can see how the method of playing around with examples can be illustrative o specific cases where it does apply, but I believe you're saying that no general method for solving $\vec \varphi \circ \vec u = {\bf S} \cdot \vec \varphi$ for $\vec \varphi$ with a known $\vec u$ exists. And apparently the very validity of the equation is not proven. Right?
Comment: "In point of fact, the renormalization group equation for multicoupling QFT systems is of this form, but people bypass this entire structure in favor of coupled ODE systems. I can see you asking for more elaborate constructs next..." Okay... It is probably a long shot but, is there a general way to go from this functional equation to a system of coupled ODEs? Or is this a particular feature of the QFT theory itself that allows for the students of renormalization group to make this switch?
Comment: You want a manual for a field that does not exist, properly. Diagonal eiegnmatrix problems are basically decoupled arrays of scalar problems, so all trivially tractable. The QFT RG techniques are very-very different than here, and basically coupled ODEs, not functional equations at the end of the day. This is an essentially *different* story, meriting different questions...


Can ADFS connect to other SSO services?

Question: I have a .net application that's wired up to my local ADFS server (connected to our corporate AD server) and everything is working fine.  My question is, can my ADFS establish a trusted connection to additional SSO services out on the internet like Azure AD, AWS, Google login, Facebook, Twitter, OpenID, etc. so that my application could use claims from multiple trusted sources other than my Activate Directory?
Comment: I've never used it but isn't ADFS a full SAML2 implementation?
Comment: I believe so: https://support.techsmith.com/hc/en-us/articles/204837367-Adding-Single-Sign-on-Authentication-with-AD-FS-and-SAML
Answer: I've done this.  In our model, we have three sets of users:

Internal users authenticating via Active Directory (Kerberos or NTLM) against the local AD FS instance.  In this set, the AD FS server runs as the IdP.
External users authenticating via Active Directory (Kerberos or NTLM) against their own AD FS instance and domain.  In this set, the AD FS Server has a Claims Provider Trust configured to the issuing AD FS server.  Our AD FS server operates as an SP-STS by transforming and validating claims issued elsewhere, and reissuing a trusted token to our application.  The IdP is the other domain's AD FS server, where our AD FS server is configured as a relying party.
External users without corporate login.  These users authenticate to small .Net STS which is configured as a Claims Provider Trust on our AD FS server.  (We did also consider - and it was technically straightforward - to use Azure Active Directory B2C for this role, but other concerns prevented it.)

Tokens are issued as:

To make it transparent for our customers, we use IP ranges to detect (via nginx) and redirect to the appropriate corporate AD FS – and otherwise rely upon the standard AD FS HRD page.
Comment: Thats probably meant to say NTLM
Comment: Authentication to AD via NTFS, as in NT File System?
Answer: It seems that chaining relying parties is indeed possible. This guy has written a series of posts about it, here's one. You can use ADFS as the "hub" for your app to auth against, and it will chain the request back to whichever service the user's identity actually lives in.
https://cloudidentityblog.com/2013/06/17/why-use-aad-as-idp-via-ad-fs-rp/
I haven't done this myself, so I'm not sure how much work this and what the pitfalls might be. I'm sure you could run into problems if a user lives in more than one IdP.
Don't forget, there's nothing preventing you from writing your .NET application to be able to take advantage of multiple SSO providers natively.
Answer: Yes, it can.
Each of these external IDP is added as a claims provider in ADFS and on the IDP side, ADFS is added as a Relying Party.
When you authenticate, ADFS will show a Home Realm Discovery screen that lists all the IDP.
You then choose which one you want to use.

Can ADFS connect to other SSO services?


Correct way to apply model uncertainty in bootstrapping Monte Carlo

Question: I have a simple linear model that predicts sales as a function of a temperature-related variable.
For the sake of simplicity, let's assume that the model is
$$\hat{Y} = a+b \cdot T$$
I train this on a set of historical values $\{T_i, Y_i\}$. The estimated coefficients $a$ and $b$ have their own statistical uncertainty
and the residuals $e_i = (\hat{Y}_i - Y_{i})$ are gaussianly distributed with $N(0, \sigma^2_e)$ with
$$\sigma^2_e = \sum{(\hat{Y}-Y_i})^2/(N-2)$$   (the standard error of the estimate)
Notice that this variance is intrinsic in the data, in the sense that for the same temperature I can have a range of $Y$ values.
This can be imputed to missing explanatory variables in my model or genuinely random behavior.
Then I want to forecast the number of sales, taking into account the natural variability of weather conditions.
Because I don't have a model that accurately describes the p.d.f. of the temperature, I bootstrap this using a long series of temperature data.
I simulate thousands of synthetic weather days in this way and for each one of them I calculate an estimated number of sales with my formula.
On top of the weather variability, there is my model uncertainty. I want that my simulations correctly capture the model uncertainty mentioned above: for the same temperature, I should get different calculations for $Y$. I smear gaussianly the predictions.
Question: The question is what is the correct sigma to use when I apply the Gaussian smearing.
I'm considering three options but I don't know which one is correct.
1. Use the standard error of the estimate, $\sigma_e$
2. Compute the uncertainty of the prediction, $\sigma_{pred}$ applying the error propagation rules
to the stat uncertainties of the coefficients and the input temperatures used in the simulation
3. Sum in quadrature $\sigma_e$ and $\sigma_{pred}$
What is the correct thing to do in this case?
Answer: When you say bootstrap Monte Carlo I assume you really meant bootstrap.  Monte Carlo simulation is for when you know the true underlying (population) model.
You did not state what types of model uncertainties pertain.  You wrote more about parameter estimate uncertainty and didn't mention entertaining any other distributions, transformations, or predictors.  Assuming you only need to deal with parameter (regression coefficient) uncertainty, all the mechanisms of standard regression modeling apply and your solution is easy with a little study of statistics.  Get the prediction interval and a confidence interval (e.g. 0.95 confidence interval) for the prediction.  Note this is a much wider interval for the usual one which is for estimating long-term averages ($E(Y|X)$).  Confidence limits for individual $Y$ are dominated by the estimated residual variance of $Y|X$ but also contain a variance component describing the difficulty of estimating the regression parameters.  It is only this component to converges to zero as $n \rightarrow \infty$.
Your comment about imputing missings is very hard to understand.
Comment: Thanks for the reply, Frank. Yes, I am interested only to the uncertainty on the parameters. The comment about "This can be imputed to missing explanatory variables in my model or genuinely random behavior" simply refers to the fact that there are more drivers of sales than just the temperature. Ideally one would improve the model adding more independent variables, but not having the necessary data this cannot be achieved.
Comment: Summarising, the correct answer is neither of those I listed. I will use the prediction interval formula for the linear regression as you pointed out.


Is it possible to do appsrc--ximagesink setup in Gstreamer

Question: I want to stream some random bytes to Gstreamer and display it as follows:
[Rand Bytes]--[Video source=appsrc]--[Video sink=ximagesink]

The following Python code I found in this SO post works
source = gst.element_factory_make("appsrc", "source")
caps = gst.Caps("video/x-raw-gray,bpp=16,endianness=1234,width=320,height=240,framerate=(fraction)10/1")
source.set_property('caps',caps)
source.set_property('blocksize',320*240*2)
source.connect('need-data', self.genRandBytes)

colorspace = gst.element_factory_make('ffmpegcolorspace') #To remove

videosink = gst.element_factory_make('ximagesink')
caps = gst.Caps("video/x-raw-yuv,width=320,height=240,framerate=(fraction)10/1,format=(fourcc)I420")
videosink.caps = caps

gst.element_link_many(source, colorspace, videosink)

However if I remove the colorspace part and set videosink.caps as the same as source's, it stops working(nothing happens after clicking start).
My question is why is colorspace needed here? Is it possible to do just appsrc--ximagesink setup?
Answer: ximagesink doesn't support x-raw-gray, hence the need for colorspace conversion.
Having said that, the documentation says that ximagesink only supports video/x-raw-rgb, so I'm thinking that setting that caps property on the sink isn't doing much. In fact, looking at the output of the sink pad on ximagesource's get_caps() method, it's still video/x-raw-rgb even after that caps property is set and playback has started. Taking out the line setting the caps property has no effect on the output.
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-base-plugins/html/gst-plugins-base-plugins-ximagesink.html


$scope condicional no angular

Question: Olá!
Tenho estes radio buttons:

                  <input type="radio" name="optradio" ng-bind="pizza">Pizza

                  <input type="radio" name="optradio" ng-bind="barras">Barras

essa é a ideia..
edit:
O que preciso é que quando escolher um radio button o angular execute uma ação, como exibir um gráfico através do $scope
edit:
Tentei isto, deu certo:

                  <input type="radio" name="optradio" ng-click="pizza()" >Pizza

                  <input type="radio" name="optradio" value="barra()" >Barras

app.controller('GetDados', function($scope, $http){

    $http.get('http://localhost/api/v1/index.php/dados').
        success(function(data){
        $scope.dados = data;
    });

    $scope.pizza = function() {
        console.log("oi");
    }
});

Comment: Qual erro mostra no console? Formate sua pergunta, tem uma parte do código que saiu errado
Comment: Pergunta editada
Comment: Por que você não usa o ng-Click e coloca sua função para ser executada?
Comment: foi o que tentei...
Comment: "Tentei isto, deu certo:". Então ta bom. Porque está perguntando?
Answer: Você pode usar uma só função no ng-click e verificar qual radio foi clicado através do $scope desta forma:

              <input type="radio" name="optradio" ng-model="escolha" value="pizza" ng-click="getInfo()">Pizza

              <input type="radio" name="optradio" ng-model="escolha" value="barras" ng-click="getInfo()">Barras

no controller ficaria mais ou menos assim:
app.controller('GetDados', function($scope, $http){
    $scope.getInfo = function(){
        $http.get('http://localhost/api/v1/index.php/dado').then(function(data){
            console.log(data);
            if($scope.escolha == "pizza"){
                console.log($scope.escolha);
            }
            else{
                console.log($scope.escolha);
            }
        }, function(err){
            console.log(err);
        });
    }
} );

$scope condicional no angular


Proof of Ito quotient rule.

Question: I've tried my best to search this problem but failed to find any on this site.
Please let me know if this problem is duplicated.
I would like to show the Ito's quotient rule as follows:
$$d(\frac{X}{Y})=\frac{X}{Y}(\frac{dX}{X} - \frac{dY}{Y} - \frac{dX}{X}\frac{dY}{Y}+(\frac{dY}{Y})^2)$$
My approach is to apply Ito's formula to
$$d(X\frac{1}{Y})=\frac{1}{Y}dX+Xd\frac{1}{Y}+dXd\frac{1}{Y}=\frac{X}{Y}(\frac{dX}{X}-\frac{dY}{Y}-\frac{dXdY}{XY})$$
Where I think $d(\frac{1}{Y})=-Y^{-2}dY$. However, I don't know where this term, $(\frac{dY}{Y})^2$, comes from? Any help will be appreciated! Thanks in advance!
Answer: When you apply Ito's lemma,
$$
df(Y_t) = f'(Y_t) dY_t + \frac{1}{2}f''(Y_t) (dY_t)^2
$$
so
$$
d \left(Y^{-1}\right) = -Y^{-2} dY + 2 \times \frac{1}{2} \times Y^{-3}(dY)^2
$$
Comment: thanks a lot! i think there isnt a 2 in front of dY^2 term.
Comment: :I think you forgot $\frac12$ before $2Y^{-3}(dY)^2$ ?! am I right ?
Comment: @Pandaaaaaaa fixed thank you

Proof of Ito quotient rule.


Refresh View in block using js

Question: I have two different blocks.
One has a form and the other one is a view with some files uploaded from the form above.
On submit of the form I need to update the view results.
I thought I could do it using js but I can't make it work.
Please check my code. Any advice would be helpful!
Drupal.behaviors.myViewsRefresh = {
    attach: function( context , settings) {
        var viewName = 'application_documents';

        // Your views should be Ajax enabled
        var instances = Drupal.views.instances;
        var myViews;

        // then simply need to iterate through the various
        // views instances name to find the view
        $.each( instances , function getInstance( index, element){
            if(element.settings.view_name == viewName ){
                myViews = '.view-dom-id-' + element.settings.view_dom_id;
            }
        });

        $(document).once('mysubmitbutton').ajaxComplete(function(event, xhr, settings){
            if(typeof myViews !== 'undefined'){
                $(myViews).trigger('RefreshView');
            }
        });

    }
}

I can get into the ajaxComplete but the view doesn't reload. The view id is correct.
Thank you in advance.
Answer: The way you construct your myViews variable seems to be the reason this doesn't work for you.
What worked for me was changing this:
            myViews = '.view-dom-id-' + element.settings.view_dom_id;

to
            myViews = '.js-view-dom-id-' + element.settings.view_dom_id;

I just tried a simple version of that without any form submiting, simply calling this from my browser's console:
jQuery('.js-view-dom-id-c58e9660da223aaf7c1645b2746e0b2e16f6a2f3e1de74c8533e6ad6117613db').trigger('RefreshView')

and it worked as expected. As a final remark and as is commented in the code you provided, your views should be Ajax enabled, otherwise, this will not work.
Good luck!
P.S.: The most relevant piece of information from the codebase (this comment says it all):
// Add a trigger to update this view specifically. In order to trigger a
// refresh use the following code.
//
// @code
// $('.view-name').trigger('RefreshView');
// @endcode

(http://cgit.drupalcode.org/drupal/tree/core/modules/views/js/ajax_view.es6.js#n94)
Comment: Thanks!! It worked. But now when refreshes the view triggers an infinite loop. Also the view I get is empty but maybe has to do with the infinite loop.
Comment: Grand! Did you try simply triggering the RefreshView manually outside of the form submit? In order to narrow down what works and what doesn't?
Comment: Thank you @Stefanos. I changed a bit my code in order to call my custom function only after ajax submit and now the infinite loop is gone! :) but still I don't get the view results. I tried to trigger the RefreshView directly in my browser's console but still no luck. This view is a combination of 4 different views displayed in the header of my current view. I have also a hook_views_alter which is not triggered when I trigger the refresh(I guess it should). Is there some way to test it? Thanks!
Comment: This is now a different question (or even more than one). I would start by checking if there is an actual ajax request sent; you can see that in the "Network" tab of your browser. After that, check if the manual triggering of RefreshView works with the View running on it's own page, and not inside a block. Just some ideas. :-)
Comment: Thanks! Yes I'm checking also the network. I managed to make it work partly. As there are different views I had to refresh also them. So I enabled the Ajax on those and I also refresh them. Two of those doesn't work but I guess it has to do with some configuration so I'm on it at the moment. Thank you so much for your help!

Refresh View in block using js