Calculate stock returns from long format data frame

Question: What would be best practice for taking a long format data frame of stock prices (containing multiple stocks) and producing a new data frame with summarized return statistics for each stock? I'm thinking of some solution with dplyr, group_by, and summarize. My input data frame has columns: Date, Ticker, and Price. I'd like the output to have a column for Ticker then multiple columns for each return statistic. I want return statistics to be customizable, so I can have "1 month return", "year to date", and even things like "return between March 3, 2019 and Feb 14, 2020".
Here's an example of what my input looks like:
library(lubridate)
set.seed(100)
d_start <- as.Date("2019-01-01")
input <- data.frame(
  Date = rep(seq(d_start, today(), by = "1 day"), 3),
  Ticker = c(rep("AAPL", today() - d_start + 1),
             rep("MSFT", today() - d_start + 1),
             rep("NFLX", today() - d_start + 1)),
  Price = c(runif(today() - d_start + 1, min = 20, max = 40),
            runif(today() - d_start + 1, min = 50, max = 90),
            runif(today() - d_start + 1, min = 150, max = 400))
)

And here's what I'd expect the output to look like.  With:
r1 as the return over the past 1 month,
r2 as the return from year to date, and
r3 as the return from Mar 3, 2019 to Feb 14, 2020
Ticker         r1         r2          r3
1   AAPL  0.1788484  0.1370241  0.24244384
2   MSFT -0.1466788 -0.2850910 -0.06895183
3   NFLX -0.4755626 -0.4964146 -0.06005490

Comment: please include a reproducible example including a sample of your dataset and an expected outcome. Also include any code you have already tried
Answer: You're specifically looking for tidyquant package. More specifically tq_mutate().
A sample code would be as follows:
Stocks <- c("AAPL","MSFT","NFLX") %>%
              tq_get() %>%
              group_by(symbol)%>%
              tq_mutate(
                                  select = 'adjusted',
                                  mutate_fun = periodReturn,
                                  period = 'monthly',
                                  col_rename = 'return')

String vector should contain yahoo finance symbols. When importing multiple symbols tq_get() returns by default a long df and has additional arguments 'from' and 'to' to select dates of interest. If you already have your own information tq_mutate() works like an usual dplyr function.
Any doubt, this package is really well documented.
Comment: Thanks for this, but 2 things: 1) I cant seem to get that to work and 2) I still don't see how I could get the "custom date return" (r3 in my example) with this.  When I run the code I get "Error in merge_two_tibbles(tib1 = data, tib2 = ret, mutate_fun) :
  Could not join. Incompatible structures."
Comment: The previous edit makes the code work, sorry. For each summary in you question you need to run a different set of tidyquant commands. For r3 in your question I would run a tq_mutate() with period.apply() after calculating the returns.


why acf for stationary process is positive semi definite

Question: I am studying the time series, and the lecture note just state that
For the autocovariance function of a stationary time series one of the property is positive semi definite .
I want to see the proof, but I cant google it
Comment: If you model the acv function by a function which is **not** positive semidefinite, then you will be able to calculate, from that model, some negative variances. You don't want that.
Answer: kjetil b halvorsen's comment is witty and accurately pointed out the necessity. In any case, this is a routine procedure covered by any good time series textbook. Nevertheless, for the sake of completeness, I am jotting down (paraphrasing) the proof, which basically is the characterization of acv functions.

Kolmogorov Consistency Theorem: Let $\{\mathbf F_{\mathbf t}, ~\mathbf t := \left(t_1,t_2,\ldots,t_n)^\mathsf T\in \mathbb R^n, ~t_1<t_2<\cdots<t_n,~~n\in \mathbb N\right\}$ be the set of finite dimensional distribution functions. $\bf F_t$s can be the distribution functions of a stochastic process if and only if $$\lim_{x_i \to \infty} \mathbf F_{\mathbf t}(\mathbf x) = \mathbf F_{\mathbf t(i)} (\mathbf x(i)),$$ $\mathbf t(i)$ is $\mathbf t$ sans the $i$ th component.
It would take more of a space to derive this and it would, frankly, derail from the matter in hand. Nonetheless, here is discussed elegantly the proofs both for countable as well as uncountable products.

Theorem: A function $\gamma: \mathbb Z \to \mathbb R$ is acv function of a stationary time series if and only if it is even and non-negative definite.
I would adumbrate the proof here.
Let $\mathbf a\in \mathbb R^n,~ \mathbf t \in \mathbb Z^n$ and let $\mathbf Z_{\mathbf t} := (X_{t_1}- \mathbb EX_{t_1},\ldots, X_{t_n}- \mathbb EX_{t_n})^\mathsf T. $ Since $\mathbb V(\mathbf a^\mathsf T \mathbf Z_{\mathbf t}) \geqslant 0$ and $\mathbb V(\mathbf a^\mathsf T \mathbf Z_{\mathbf t}) = \mathbf a^\mathsf T \mathbb E \mathbf Z_{\mathbf t}\mathbf Z_{\mathbf t}^\mathsf T\mathbf a, $ one can conclude $[\gamma(t_i-t_j) ]_{i, ~j~=~1}^n$ is non-negative definite.
For the converse, the approach would be to show the existence of a stationary process that is associated with an even non-negative definite function $\gamma(\cdot),$ the latter being the acv of the former. Let $\mathbf K:= [\gamma(t_i-t_j) ]_{i, ~j~=~1}^n.$ For each $n\in \mathbb N, ~\mathbf t\in \mathbb R^n, :t_1<t_2<\cdots<t_n,$ let $\mathbf F_{\mathbf t}$ be the distribution function  in $\mathbb R^n$ with characteristic function
$$ \varphi_{\mathbf t}(\mathbf u) := \exp\left(-\mathbf u^\mathsf T\mathbf K\mathbf u/2\right),~\mathbf u\in \mathbb R^n.$$ By assumption, $\mathbf K$ is non-negative and this means $\varphi_{\mathbf t}(\cdot) $ is the characteristic function of an $n$-variate normal distribution with mean $\bf 0$ and covariance matrix $\bf K. $ Now, the critical thing is to note that
$$\varphi_{\mathbf t(i)}(\mathbf u(i)) = \lim_{\mathbf u(i) \to 0}\varphi_{\mathbf t}(\mathbf u) , \forall\mathbf t. $$ This implies $\mathbf F_{\mathbf t}$ is Kolmogorov consistent and hence there exists a time series $\langle X_t\rangle$ with the distribution function $\mathbf F_{\mathbf t}.$

Reference:
Time Series: Theory and Methods, Peter J. Brockwell, Richard A. Davis, Springer Science & Business Media, 1991.


How to store a global array reference but initialise values later in some method of implementation file?

Question: I want to keep a reference to array in interface like:
@interface
{
   float myArray[];
}
@end

and then use it in my implementation file like:
@implementation

-(void)someMethodToFillValuesInArray
{
  myArray = { 10.0f, 20.0f, 30.0f };
}

-(void)someMethodToUseArray
{
   float sum = myArray[0] + myArray[1];
}

@end

However, XCode throws syntax errors and I can't compile.
I am failing to understand the syntax and I tried a lot of ways but was unable to get it going. Can anyone help me ?
Comment: You need to use pointer to `float` instead of the array. Initialize it when you need with `malloc`. When you are done with the array (in `dealloc` for example) call `free`.
Comment: It's not clear what you want. Title says "...store global array reference...", but in the code example you are trying to declare instance varible myArray.
Comment: Talking about instance variable as global reference, means the global reference with regards to any method in its implementation file.
Comment: What you want to do and why you cannot use singleton if you must insist upon global access?
Answer: You can use static initialiser at the time when array is declared not after it is declared.
It is permitted as you are initialising at the time of its declaration
float myArray = { 10.0f, 20.0f, 30.0f };

Else you should declared array with length as follows in your interface
float    *myArray;

and use it in you implementation file as follows
To dynamically allocate space, use calls to malloc passing in the total number of bytes to allocate
int size = 3;    // This size can be whatever you set during runtime or compile time
myArray = (float *)malloc(sizeof(float)*size);
myArray[0] = 1.0;
myArray[1] = 2.0;
myArray[2] = 3.0;

but since manually allocated a certain amount of memory for array, you are responsible for freeing it once you are done with it. Use free(myArray) to free the allocated memory.
Comment: I want to initialise the array somewhere later in code. How can I do that and still maintain a global reference in the interface file ?
Comment: You can with above mentioned approach provided that an array's length is given.
Comment: Did you specify the length?
Comment: I missed adding a semicolon to the above myArray declaration.
Comment: Please refer the answer again.
Comment: ok, its working for known length ahead of time. But my actual question is to keep a reference to an array of unknown length. How can I do this ?
Comment: You cannot use primitive type array for unknown length unless it is statically initialised, instead you should use NSArray or NSMutableArray of Objective C and convert those primitive float values to respective wrapper object and store it in NSArray.
Comment: NSMutableArray is a good way. However, there is a lot of overhead in creating mutableArrays and then the instances of NSNumber(s) within the arrays to hold float values. Basically I want to transfer large amount of float data per frame (i.e. 60 times a second ) to opengl and converting local float arrays from NSMutableArrays will slow down the render loop too much.
Comment: refer the answer again.
Comment: There is a problem with your answer it should be float * myArray = (float*)malloc(sizeof(float)*3); You have used int at certain places which is not proper.


Orientation of a bipartite graph

Question: I need to determine whether the complete bipartite graphs $K_{n,m}$ with $n,m \ge 2$. Are orientable.
Am I correct in going about doing this by induction and using the fact that all of the complete bipartite graphs can be created using a number of circuits.
I am just unsure about how I go about proving this, I'm pretty sure that all are orientable by drawing a number of the graphs.
Answer: If $n,m\geq 2$ then $K_{n,m}$ stays connected after the removal of one edge, hence $K_{n,m}$ has a strong orientation by Robbins' theorem.

Orientation of a bipartite graph


How can we access the first row's value in the Window in Spark

Question: I want to access the value of the very first row in the window of DataFrame in Spark.

Account
Transaction
Created Date

Acc1
T1
2021-08-01

Acc1
T2
2021-08-02

Acc1
T3
2021-08-03

Acc1
T4
2021-08-04

Acc2
T1
2021-08-03

Acc2
T2
2021-08-03

Acc2
T3
2021-08-07

If above one is the dataframe, I want to find out the first row's Created Date value when window partitionBy("Account").

Account
Transaction
Created Date
First Txn Date

Acc1
T1
2021-08-01
2021-08-01

Acc1
T2
2021-08-02
2021-08-01

Acc1
T3
2021-08-03
2021-08-01

Acc1
T4
2021-08-04
2021-08-01

Acc2
T1
2021-08-03
2021-08-03

Acc2
T2
2021-08-03
2021-08-03

Acc2
T3
2021-08-07
2021-08-03
Answer: Using spark sql.
If the first transaction for an account is always the earliest transaction then the following would provide the desired results.
SELECT
    Account,
    Transaction,
    CreatedDate,
    MIN(CreatedDate) OVER (PARTITION BY Account) as FirstTxnDate
FROM
   my_temp_view_or_table

Here I've used the MIN as a window function to achieve the earliest transaction for each partition/group/account.
Comment: The `MIN(CreatedDate)` doesn't require your data to be sorted. It will always find the earliest date in that group. If you would like to retrieve the first record in a group when having your data ordered by a nother column you could always use the [`FIRST_VALUE`](https://spark.apache.org/docs/2.4.1/api/sql/#first_value) function eg `FIRST_VALUE(ColumnYouWantFirstValueFor) OVER (PARTITION BY Account Order By ColumnYouWantYourDataOrderedBy)`
Comment: is there any other way to access the column values of first row?
It may not be always sorted based on that column.


How to make a subsample from a dataset with equal labels, like the sklearn.digit dataset

Question: For our demonstration, we'll just use the ten digits dataset from sklearn. Pendigits dataset consists of 10 classes from digit 0 to digit 9.
from sklearn.datasets import load_digits
digits = load_digits()
print(digits.data.shape)
print(digits.target.shape)

Output looks like -
(1797, 64)
(1797,)

So each digit consists of some sample dataset. I would like to have a subsample of each class from the dataset. For example from digit 0 to digit 9, I need 50 subsamples of each class present in the dataset.
print(digits.data.shape)
print(digits.target.shape)

Result should be(50 subsample * 10 class = 500 subsample) -
(500,64)
(500)

Result should consist of subsample of each class available in the dataset.
Answer: One option can be to use sklearn.model_selection.train_test_split and use stratify on labels (here : digits.target) to split data in a stratified fashion.
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits
import numpy as np

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target,
                                                    stratify=digits.target,
                                                    train_size=500)

# we can check labels that split equality like below
print(np.unique(y_train, return_counts=True))
# (array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),
# array([50, 51, 49, 51, 50, 51, 50, 50, 48, 50]))

print(X_train.shape)
# (500, 64)

print(y_train.shape)
# (500,)

Answer: from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X, _, y, _ = train_test_split(
    digits.data, digits.target,
    stratify=digits.target, train_size=500
)
X.shape, y.shape #((500, 64), (500,))

Comment: @TrentonMcKinney yes it is. For this reason I up-voted that answer, although it was identical to my solution.
Comment: This answer was reviewed in the [Low Quality](https://stackoverflow.com/help/review-low-quality) queue because it lacks an explanation. Here are some guidelines for [How do I write a good answer?](https://stackoverflow.com/help/how-to-answer). Code only answers are **not considered good answers**, and are likely to be downvoted and/or deleted because they are **less useful** to a community of learners. It's only obvious to you. Explain what it does, and how it's different / **better** than existing answers. [From Review](https://stackoverflow.com/review/low-quality-posts/32302722)
Comment: The other answer is a good example of how to write answers on SO.


Nature of the function

Question: I want to check the nature of the function $f:\mathbb{R}\to \mathbb{R}$ defined by $f(x)=x+\sin x$ for all $x\in \mathbb{R}$, where cofinite topology is considered in $\mathbb{R}$.
Is this function strictly increasing? If so, then it is one-one and so continuous. But I am not sure? Any help would be appreciated.
Comment: Remember your calculus!! How do you check that a function is increasing?
Comment: If $f^{'}(x)>0$, then $f$ is strictly increasing...but in this case $f^{'}(x)=0$ for all multiples of $\pi$. Thus $f$ is not strictly increasing. So, is it one-one?
Comment: This is a pretty easy one to figure out... maybe just a tad more than cookbook calculus may be required. If you are learning topology, then surely you can do this.
Answer: I suppose I should go ahead and answer this.
Yes, the function is strictly increasing, i.e., $x < y$ implies $x + \sin(x) < y + \sin(y)$. In the case where the interval $(x, y)$ contains no odd multiple of $\pi$, this is clear from the mean value theorem, since $f'(t) = 1 + \cos(t) > 0$ for $x < t < y$. Otherwise, we can let $x_1, \ldots x_n$ enumerate the odd multiples of $\pi$ in $(x, y)$, and use the exact same argument to conclude
$$f(x) < f(x_1) < f(x_2) < \ldots < f(x_n) < f(y)$$
by restricting attention to each of the intervals $(x, x_1), (x_1, x_2), \ldots, (x_{n-1}, x_n), (x_n, y)$.

Nature of the function


Alternatives to scale-centric beginning music theory?

Question: I'm a noob music student and I recently asked a question here about the widespread use of a bVII chord in a scale where the seventh scale degree is vii°, and the general consensus in the many helpful answers was that scales are overemphasized in beginning music classes and books and cause more confusion than enlightenment, and that the differences between the minor scales (natural, melodic, harmonic) is largely artificial and doesn't represent music in the real world and I should escape the tyranny of scales and of being diatonic.
So what is a good way to do that?
As a beginning student I have the luxury of being able to self-direct my studies.  I often doodle on my keyboard or DAW and make chords that sound good with a melody or in a progression. But I want a theoretical framework to know WHY or HOW these chords "work" and others don't, so I can build on that experience.  I don't want to invent my own personal music theory - I want to draw on the theory and terminology that exists.  So if I toss out or de-emphasise scales and diatonicity/chromaticity as organising principles, what's an alternative foundation I should try to focus on?
Should I de-emphasise different kinds of scales (major, minor, etc) but keep the concept of a scale so I have scale degrees, and use those in functional terms (e.g., Dominant, Predominant, Tonic, etc)?  How should a beginning music student who wants to have a strong theoretical foundation start off?
Comment: "...and that the differences between the minor scales (natural, melodic, harmonic) is largely artificial and doesn't represent music in the real world and I should escape the tyranny of scales and of being diatonic"  This is very confusing to me.  Are you saying that this is true, or that you perceive that others have promoted this idea to you in answers to your questions?  I would disagree entirely.  There is nothing artificial about the minor scales.
Comment: @ggcg - I think OP means that minor scales don't reflect reality in music. Therefore they're artificial, as in man-made - as actually, all scales are.
Comment: It sounds to me like you are (1) creating your own theory based on what you think sounds good or not and (2) asking us to provide classic justification for it so you don't have to reinvent a wheel that you have already reinvented.  It's not a fair question in this light.  Scales, chords and harmony are interrelated.  Throwing one out doesn't make the others more clear.  And it seems what you are delving into is "Western Music Theory" so the diatonic scales will take a front seat.
Comment: Scales themselves are simply different sets of notes set out in ascending and descending order. They can and do simply put together several notes which have certain uses together. There are many different 'scales', put together for all sorts of different purposes. It's what humans do - compartmentalise and pigeon-hole as much as they can. (Often in the hope that it will be helpful in some way to those who come behind). Does it work???
Comment: @Tim, but what is the criterion for "reality"?  That the notes came from harmonics like parts of the Just diatonic?  Or that we really like to use these scales?  One could argue that some of our diatonic patterns are less artificial than others.
Answer: Only after we understand theory fully are we able to toss it aside in favor of a more fluid, holistic, flexible approach. The problem is that the primary way of achieving that later stage is by studying and learning theory, which means diving into all the rules and minutia. When experienced musicians encourage you to focus less on scales or theory, they may be taking for granted the extensive background knowledge and rich experience they already possess.
Here's my take: the mere existence of exceptions doesn't mean we should stop studying theory. Begin cataloging the exceptions, and you'll start to notice that the same ones recur. (For example, in the key of CMaj, Fmin-Bb7-CMaj is a turnaround.) Study the theory, but take note of the way it is and isn't applied faithfully according to the "rules." Think about the music in terms of scales, chords, etc. when you analyze. But when you encounter an exception that goes against theory, think about why it might work, whether you like how it sounds, and the ways in which it deviates from standard theory. That's a more productive way forward than trying to "fit" every exception into theory.
Over time, this type of analysis will enhance your understand of the theory itself (and its rules, etc.) as well as how it can be applied, bent, and abandoned. There are patterns in those exceptions, and perhaps those patterns could qualify as a 'deeper theory' of some sort. But at some point, too many descriptions can get in the way, and it's more helpful to just use the existing theory concepts and reflect on how they appear in practice.
In short, study scales, study songs, and study how they work together.
Answer:
I want a theoretical framework to know WHY or HOW these chords "work"

This question is often asked, and the answers vary. They might be something like, "because they have 2 notes in common" or "it borrows from such and such key or scale" or "it creates dissonance or consonance" or "it is expected/unexpected" or "there's a leading tone" or "it's a tritone substitution" or "it comes from the diminished scale" or "because blah blah on the circle of fifths" or "look at the steps in the voice leading", "the fifth is omitted", Etc. Check out Barry Harris, he talks about scales and/or chords having a love child, producing offspring... it gets a bit over my head. There are many possible perspectives and observations you can make, but usually none of them is the ultimate "why" answer for all intents and purposes. Several perspectives can be equally valid at the same time, depending on how you're used to looking at things.
If you can locate and reproduce a chord in relation to the then-prevailing tonic, then you know a lot already. If you can reproduce the same chord in other keys, you know even more. Is that not enough? Maybe not.
Why a chord "works" is not the right thing to ask. You should ask, what does it do and what other chords could you use to accomplish essentially the same things in that context. The answer to the "what does a chord do" question comes in the form of multiple perspective-dependent parameters or dimensions.

What does it do to my perception of other plausible intervals around the tonic at the moment?
Does it perhaps seem to even move the tonic?
To which role in terms of V (left) - I (center) - IV (right) would you reduce the chord's effect, if you had to substitute it with the most basic triads? Would that even be possible, or would something essential be lost in such a reduction?
If there's a V - I sort of motion going on, where would the target "I" be?
Where is the bass now?
What is the highest note?
What's the essence of the chord, does it feel like some inversion of a basic triad, and if so, is it a major, minor or diminished triad, and which inversion is it?
How's the voicing, the spacing of notes - is it open or closed, and what could you turn it to by moving one voice by one step in some direction?
Do I know this same pattern from somewhere else?
Could I turn this into a well-known progression with a few chord exchange operations, for example Am - Em - Dm - E --> Am - G - F - E (Hit the Road Jack).

Demanding all of that from a beginner feels very hard or impossible. Don't demand too much from yourself immediately, you have to be able to get on, get more experience. Learn more tunes, play them in different keys, even though you don't know all possible aspects and perspectives of analysis. Nobody knows all aspects. Somehow this reminds me of the question of an adult violin student complaining about the teacher not demanding all possible aspects of playing to be perfect before moving to the next piece. You just have to move on and accumulate more music. You'll remember something of each new chord and note. Maybe that there was a major chord on the major tonic's leading tone and it felt nice going to the tonic from there. Hey, this sounds like some Elvis Presley tune! Easy to recognize, easy to remember, easy to reproduce on the guitar and on the piano. I don't feel any need for anything more analytical to say about that chord.
The thing with harmony is, it's like the game of Go, which is often compared to Chess. In chess, each piece has its own particular abilities and ways of moving, and you use these differences for classifying each situation. But in Go - so I've heard - the role of a stone i.e. piece on the table can be fairly ambiguous. Is it an offensive or a defensive stone? Can it "do" anything at all? It depends on its position, and where other stones are relative to it. Maybe you recognize the pattern in advance, and then you can classify what's going on. In harmony it is the same. Sounding notes are like stones in Go: the plot of the game is built from the combination of notes and their relations with each other. There can be many simultaneously valid "explanations". For example, a B half-diminished chord can be seen as Bm7-5 or Dm6/B, and it is both at the same time. A G7 chord can be seen as almost a Bdim, and C9 can be seen as having a Gm triad, and an E half-diminished. You don't have to see all possible re-interpretations, or any of them. You know what notes there are in each chord, what notes you expected in your "current scale" and if the chord was against the expectations, maybe making a chromatic alteration, modal mixture or even a modulation. You don't have to get it "right". It's enough to be able to reason about the notes even on some very simple level. Your understanding of the game gets stronger by playing.
Instead of a theoretical framework, I offer you a practical one. Learn to play tunes as melody and backing chords, in different keys, and make improvised changes to both, to get a hands-on feel for what's happening. You need to play the game, not get stuck on some fancy theory about strategies.
For example, let's take this chord progression.
Am - Dm - G - Am - F - Dm - E - Am.
Play an improvised melody over that - even simply running scales or arpeggios up and down. Grope around, trying to feel where the walls are. Which scale notes seem to fit over each chord? On the first Am, can you play the so-called natural, harmonic and/or melodic minors? All of them or just one? How about on the Dm? On the G? Etc. How about on the E chord, does the E chord demand either an F or F# note in the melody, or could you freely choose either?
Let's take another one:
Am - Dm - G - Am - F - Dm - B7 - E7.
Can you feel what notes would fit over the B7 chord? How about if we make some changes:
Am - Bb - E7 - Am - F - Dm - B7 - E7 - Am6.
What happens to the scale possibilities over the Bb chord? How about the Am6, how does it change the scale?
Can you play the same chord sequences in other keys, relative to other tonics?
In this question's title you're asking for something less scale-centric, partly because of my complaining about harmonic minor. My complaint wasn't about thinking about scales as such, the problem with the harmonic minor thing was, you tried to fit it as a rigid unchangeable structure over entire chord progressions or songs, as if it was a mode. Thinking about a scale is OK, and it can be even essential, but the question should be, what could the scale be now. A single E major chord doesn't say anything about the F question: is it F or F#... or F##.
Answer: First, let's have a look at what kind of knowledge make up music theory:

Science - basic results that relate to music that can be measured or observed in physics, maths or psychology.
Terminology and notation - ways of naming and notating things.
Stylistic norms and preferences - observations about patterns in music that have been observed over the ages in various cultures.

Then we have to consider that all these areas of musical knowledge are patchy. There's a limited amount of objective scientific knowledge that can help us understand 'how music works'. Our notation and terminology is something of an accident of history, and perhaps represents some musical ideas better than others. Stylistic norms and preferences have been observed and catalogued for some musical styles more extensively than they have for others.
Then consider that beginners' music theory is a simplification of this set of knowledge that is already patchy and incomplete. You will see that trying to use a basic or intermediate theory book as a foundation for your musical activities isn't going to work well - too much soft ground, too many sinkholes.
However, that doesn't mean that the ideas are invalid. For example, the fact that any given musical piece (or section thereof) probably has a home note, and a certain feel that comes from the recurring use of certain notes in relation to that note, usually is valid. So you can build your own observations around that idea. Just don't get too hung up on the idea that every piece you encounter is going to fit in one of the mental 'boxes' that you've already got.
Unfortunately, sometimes you do have to make your own theory up a little bit until such time as you happen across something in your theory studies that ties in with your own observations.
How do people put up with all this uncertainty and nebulousness? I think the answer is that most people approach music as a practical rather than theoretical pursuit. This works for most people in many other activities in life - if you want to play soccer, you're not unlikely to read three books about it first. Likewise for dance, or refreshing your summer wardrobe, or any number of other things. All of these do have an amount of formalized knowledge surrounding them, but... it's patchy. And so it is with music. More than anything, people tend to gravitate to what they like, and learn its nature through dint of repeatedly enjoying it!
I would say that if you really want a foundation: be a music historian and a musicologist, as well as just a musician. Look at how our ideas about music have changed over the past few hundred years, and how they vary across cultures. And why not be a scientist while you're at it - one of my favourite 'theory' books is The Science of Musical Sound (which I should probably re-read...)
But that sounds time consuming. if you want a more limited foundation, you could simply focus on an area of music that is more strongly 'theorized-about'. That could be 18th century classical music, or Jazz, for example. Just be prepared to find out that the not all the knowledge you learn in relation to a certain style of music will be generally applicable.
Or - you could just learn to play what you enjoy, take some time to read as much theory as you can, and take comfort in the fact that the human brain is optimized for 'messy' learning! You don't really need a foundation, or a unified approach.
Answer: I have learnt first the modes (in secondary school)  and   the 3 minor scales, before knowing anything about functions.
I think this is a good plan. You can harmonize without problems different songs based on the different scales. So where is the problem?
Another question arises: how far the harmonic minor scale is diatonic. But I think questions like these are purely academic.
Interesting for discussion is the historical evolution of such phenomena like the leading tone that has been explained by Aaron in the other question.
So my answer here is: learning first songs based on the different scales and then harmonizing songs (piano or guitar) are going hand in hand. The synthesis of both
approaches will be the most preferable.
How the functions are spelled and why has to be secondary.
Answer: Let 'theory' grow from playing a wide range of music.  So your first job is to become fluent with notation. Play music.  Play gigs (if only! :=()
Play scales to develop fluency on your instrument.  That's all.
Not everyone's a composer.  That's OK.  And improvisation is a very over-rated pastime.
Don't look for a Theory Of Everything.
Answer: I'll try a different answer, but I don't want to change my first one to something completely different, because people already voted for it.
There's nothing wrong with thinking about scales. For a musician, scales are one of the most important tools. In the previous question you mention, you seemed to see the harmonic minor scale as some sort of a modal scale where you stay "diatonically" without having to make chromatic alterations - which is not how songs in minor keys are usually composed and played. Chromatic alterations are a very basic elementary tool, don't be afraid of them.
You have to realize that there are scales or "scales" in two different places:
(1) The composer or player is (usually) thinking about one or more scales, which he/she uses as toolsets or palettes to draw notes and chords from. Each scale offers a certain set of interval combinations.
(2)  The listener hears the notes that the player is creating, and the listener's mind forms a perceived harmonic context, which consists of things like the tonic note, tonic chord, and other probable intervals around the tonic. It's more like a probability field, and it might have an "opinion" about all scale degrees, or only a part of a scale. And it's time-variant, it changes flexibly all the time according to what is heard.
For example, let's assume the following sequence of events

Player plays a strong C note. --> The listener's mind probably sets C as a tonic pitch, but the interval probabilities around the tonic are unclear, missing, ambiguous, in an "unknown" state. Or floating like in electronics. The listener's mind might do anything, for example imagine melodies, scales, chords, around and over the C note. Who knows.
Player plays an E note. --> The listener's mind considers that relative to the previous context, i.e. C being the tonic. Now the listener has a vague perception of perhaps something C majorish. The rest of the picture is left floating.
Player plays a C major chord. --> Now the listener's C major idea gets stronger.
Player plays a D major chord. --> The listener's mind feels that the fourth scale degree is an F sharp now. Some sense of expectation and tension starts to build.
Player plays a G7 chord. --> The listener's mind releases the fourth scale degree back to F natural, and now has a rather clear picture of a C major context, and feels that if the next chord is a C major, then all expectations would be fulfilled and we would be at rest.
Player plays a C minor chord. --> Surprise! The listener gets mixed feelings. The harmony arrived back to a center-of-balance point, tonic, but very unexpectedly, now the third scale degree forms a minor third with the tonic. Ok. The third scale degree is an E flat now. It could end here. What happened to the sixth and seventh scale degrees? Who knows. The listener doesn't hear those played now, so they're basically left floating. Maybe the listener's previous musical experiences put Ab and Bb there as slightly more probable choices than A and B.

Do you see what I'm trying to say here? You can select the harmonic minor scale as your note/interval palette and decide to create whatever harmony that scale has to offer. That's your arbitrary artistic choice. But if, as a listener, you hear a C minor chord, it is not "in harmonic minor".
When talking about jazz improvisation, an often heard question is, what scales could I use over this chord or this chord sequence. In other words they're asking for toolsets or palettes for tweaking the harmonic context. If you use a scale in such a way that it creates a harmonic context where the tonic stays bolted down in one place and the set of intervals around it stays in one static shape, with all scale degrees defined all the time and without any chromatic alterations happening at any point, then your playing is in a mode. A mode is a static harmonic shape.
When learning how music works and studying harmonic patterns, you should know how to test your own feelings and probe the harmonic context in your mind by playing notes and chords. Ask yourself, "how do I feel about the scale here". Thinking about scales is essential. But if a theory lesson talks about a scale like harmonic minor, it doesn't mean that that is the complete contents of your perceived harmonic context. It is a toolset that a composer or improviser can use in certain situations, when applicable. But even if the composer chooses to select notes from it, it doesn't mean that the scale gets transferred into the listener's mind as a complete monolithic entity.
Answer: It's great that you are not limiting yourself to "being diatonic". But you are selling both music theory and scales short here.
Yes, scales are part of music theory, but they are not the only part. There's chromatic alteration, too --- of notes in a tune, of notes in a chord (major in stead of minor, or vice versa, and witness how many different sorts of seventh-chords there are), and even of root notes (a piece in a major key may borrow ♭III and ♭VI from the tonic minor, for example). These are all ways to escape the tyranny of being diatonic.
"Should I de-emphasise different kinds of scales (major, minor, etc) but keep the concept of a scale so I have scale degrees [...]?" Yes, that seems a good idea. The other ancient modes offer some variety: Lydian, Mixolydian, Dorian, Phrygian, for example. You ask why ♭VII is widespread in major keys. The Mixolydian mode offers one explanation; what is ♭VII in the conventional major is the ordinary VII in Mixolydian. (Harmony offers an additional explanation: ♭VII in major is IV of IV.)
"The differences between the minor scales (natural, melodic, harmonic) is largely artificial and doesn't represent music in the real world" I disagree. The harmonic minor scale, and the upward and downward melodic minor scales, were codified in order to describe music in the real world. Musicians felt that when a tune goes up "la ti do", the major 6th and 7th sound better than the minor 6th and 7th, even if the 3rd is minor, but the same does not apply when a tune goes down "ti la so". Hence the two melodic minor scales.
For even more variety, choose whether to make the 2nd degree major or minor, and similarly the 3rd, 6th and 7th, and whether to make the 4th degree perfect or (as in Lydian) augmented. Some combos of these choices give you the ancient modes again, but other combos give you a scale with augmented seconds and more minor seconds --- and that's OK!
Comment: You need to separate "music" as an observable phenomenon from commonly applied practice of making music among some group of people in some culture during some period of time. For some people somewhere, in some point in history, the different note sets for upward/downward movement were appropriate for the harmonic effects they wanted to achieve. But different styles and genres have emerged, where e.g. the melodic minor scale is used like a modal scale, in order to achieve different harmonic effects. Both styles are cultural conventions, not physical or biological laws of nature.
Answer: Scales and chords are overrated.
I taught myself how to write music by listening to a lot of "tracker module" and MIDI music from the 1990s, then imitating the style in a modern digital audio workbench - without going through the standard "music theory" training.
I hardly use scales in my compositions. Chords are largely reserved for basslines and chiptune-style arpeggiation.


Error using form API on AJAX submit

Question: I am writing a form using the Drupal 7 form API. On submitting the form (via AJAX), the following error is displayed:

An AJAX HTTP error occurred.
  HTTP Result Code: 500
  Debugging information follows.
  Path: /en/system/ajax
  StatusText: Internal Server Error
  ResponseText: Recoverable fatal error: Argument 1 passed to drupal_array_nested_key_exists() must be an array, null given, called in /drupal7/includes/form.inc on line 1986 and defined in drupal_array_nested_key_exists() (line 6296 of /drupal7/includes/common.inc).

I have two buttons on the form (copy & submit); one is an AJAX call which does some stuff, and the other is used to save the entered data.
This is the code I'm using:
function foo_form ($form, &$form_state, $foo) {
  $form['#foo'] = $foo;
  $form_state['values'] = $foo;

  $form['#prefix'] = '
';
  $form['#suffix'] = '
';

  $form['copy'] = array(
    '#type' => 'submit',
    '#value' => t('Copy foo'),
    '#limit_validation_errors' => array(),
    '#submit' => array('foo_form_submit'),
    '#ajax' => array(
      'wrapper' => 'foo-form-wrapper',
      'callback' => 'foo_form_callback',
    ),
  );

  $form['name'] = array(
    '#type' => 'textfield',
    '#title' => t('Name'),
    '#required' => TRUE,
    '#maxlength' => 255,
    '#default_value' => (!isset($form_state['values']['name']) ? $foo->name : $form_state['values']['name']),
  );

  $form['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Save'),
  );

  return $form;
}

function foo_form_callback($form, $form_state) {
  return $form;
}

function foo_form_submit ($form, &$form_state) {
  // Did we use the AJAX Call?
  // @todo: can we check this some other way?
  if (!isset($form_state['triggering_element']['#ajax'])) {

    // We didn't, save the entered values

    $form_state['rebuild'] = false;
    return;
  } else {

    // We did, do some behind the scenes stuff...

    $form_state['rebuild'] = true;
  }
}

Does anyone have an idea of what could be wrong?
Answer: This line is wrong:
$form_state['values'] = $foo;

No idea what you are trying to do here, but you must not mess with $form_state['values'] directly.
And what you are trying to do with the #default_value shouldn't be necessary either. Form API should automatically re-use an already entered value, without you having to set it as the default_value.
Comment: Yes, and exactly that shouldn't be necessary. Simply use `#default_value' => $foo->name`
Comment: Concerning the default value: should be (!isset($form_state['values']['name']) ? $foo->name : $form_state['values']['name']), I'll edit the post.
Comment: I have removed the '$form_state['values'] = $foo;' and edited the default values. The error now is gone but when I press the "copy" button the page reloads. Also, in what method am I supposed to change what's in the "name" field?
Comment: form_set_value() should help you. Also, the limit validation errors thing looks wrong...
Comment: The limit validation errors is used because when I push the "copy" button, the validation triggers and errors are displayed. I got this from the examples module (examples/ajax_example/wizard). What I am trying to do is change the "name" field when I press "Copy".
Comment: Found it, thanks. I still have a problem though, I'll make a new question for that. The problem is that fields with type "text_format" (wysiwyg module enabled, ckEditor) seem to force a reload on the page.

Error using form API on AJAX submit


Freeing up memory in php script

Question: I have finished writing my php scripts for a project I am doing. My next step is I would like to see if I can improve my code from a memory stand point as some of my scripts eat a lot of memory. I have been doing research on this and one suggestion is to NULL and unset variables, but I never see an example of doing this. So I wanted to give an example of a common action done in my scripts and wanted to know if this is the proper way of doing this:
<code>    $query = $dbconn->get_results("SELECT id,name FROM account WHERE active = 1");

    if(isset($query))
    {

    foreach($query AS $currq)
    {

    $account_id = intval($currq->id);
    $account_name = trim($currq->name);

    //Code to stuff with this data

    //NULL the variables before looping again
    $account_id = NULL;
    $account_name = NULL;

    //Unset the variables before looping again
    unset($account_id);
    unset($account_name);

    }

$query = NULL;
unset($query);

$currq = NULL;
unset($currq);
</code>
Would that be the correct way to free up memory? I read the garbage collection in PHP can be lazy, so that is why they recommend to NULL the value as it will shrink it right away.
I know this might be too vague for this site, but if anyone can just let me know if this is the proper way of freeing up memory? Or if there is a different way, can you please provide an example just so I can visually see how it work. Thanks in advance!
Comment: what is $dbconn ? custom db class ? doesn't look like mysqli or PDO ?
Comment: this should be posted on http://codereview.stackexchange.com/
Comment: you dont see many examples, its in almost all case its pointless
Comment: @SirNarsh Its a dbclass called EzSQL
Answer: Please read up on PHP generators, that is what they are exactly for.
You don't want to fetch all records at once, this would blow holes into your memory like a shotgun.
Instead you want to fetch your records one at the time, process it then fetch the next one.
Here is an example:
<code>function getAccountData(\PDO $pdo)
{
    $stmt = $pdo->prepare("SELECT id,name FROM account WHERE active = 1");
    $stmt->execute();

    while ($row = $stmt->fetch()) {
        yield $row;
    }
}

foreach (getAccountData($pdo) as $account){
    //process the record for each iteration
    //no need to unset anything
}
</code>
Comment: Speaking strictly - the `$stmt->execute();` is likely to fetch the whole dataset into the php process memory anyway, regardless of how you `fetch` it. If every row was fetched separately it would be extremely slow.
Comment: `execute()` would just send the prepared query to the server, it does not fetch the whole result set into memory
Comment: Just check it :-) Generate 100 million rows in your database, prepare, execute and fetch a single row.
Comment: I have used generator in this manner before with millions of records, never ran out of memory
Comment: Weird, it's not what I see in postgresql pdo driver: http://stackoverflow.com/q/38192529/251311
Comment: It must be specific to postgres dbms because mysql does not even support scrollable cursor.
Comment: But fetching one row at a time would be extremely slow, due to network latency... I simply cannot believe that's what pdo mysql does
Comment: It really depends, if the script runs close to server or perhaps same network that would be faster, but at least it wont exhaust memory.
Answer: Well, if the function $dbconn->get_results returns an array with all the data, then there is no point in using generators since the memory has already being allocated for the data.
You can also use the mysqli_fetch_assoc function to get one row at a time. It should be more memory efficient then fetching all rows at once. http://php.net/manual/en/mysqli-result.fetch-assoc.php

Freeing up memory in php script


Extremely fast regularized linear model libraries

Question: I have a huge dataset (millions of rows, thousands of columns) and glmnet Lasso regularised regression is too slow.
I am wondering what other libraries there are that try to implement regularised linear model estimation extremely efficiently? I don't necessarily need a distributed solution (it can estimate on one CPU), and it is fine for me to load all data to RAM (so an R solution may be fine, if one exists that's faster somehow than glmnet). It also does not necessarily need to be Lasso or Elastic Net.
I can use java, python or R.
My data is pretty messy, some columns are sparse, some have heavy autocorrelation, etc.
Answer: Some form gradient descent algorithm will be fastest on your problem.
There are a number of variants of gradient descent algorithms and and a number of different algorithms.
The most basic variant is Stochastic Gradient Descent (SGD).
Most of the major machine learning libraries/systems will have a version of this.
For massive datasets that do not fit into memory Vovpal Wabbit is probably the go,
but for your data sets most of the major implementations should be fine.
Enhancements to SGD include Pegasos, PassiveAggressive algorithms, Confidence Wighted Learning, Adagrad, AROW, and Natural Gradient Descent.
If you use Python then the scikit library has SGD and Passive Agrresive Preceptrons. I have used them very succesfully for datasets like you describe.
Training should take only seconds or possibly minutes depending on the particulars of your data and the settings you use.
If your inputs are sparse, make sure the algorithm your algorithm is using sparse data structures to represent them. This could be what made glmnet slow.
Comment: Do you have thoughts on `liblinear` for this kind of problem?
Comment: I think liblinear would also be a good choice. It has been a while since I used it & I have not benchmarked it against other libraries, but it served me well when I did use it. If I remeber correctly there were considerable differences in speed between the algorithm it provides so you want to play around a bit. I suspect the "-s 2" option to work in the primal space may help in your case.
Comment: I think with liblinear you will probably get best results on your data with L1-loss L2-penalty and solving the primal problem as I mentioned in the previous comment.
Comment: I can confirm good/fast results with primal & L2 linear SVR. Few minutes to solve. Haven't tested the quality of the model yet but it estimates quickly.
Comment: Is Stochastic Gradient Descent is just optimisation algorithm that can be applied to different formulations of optimization function, can you elaborate on this?
Comment: @mrgloom Yes, SGD is a generic optimization algorithm. It is applicable whenever the loss function is the sum or average of a large number of terms (eg items in a training set). It differs from ordinary gradient descent in that updates the solution estimate after each item rather than in batch. The data should be well shuffled before using SGD.


How to access static assets in CodeIgniter?

Question: I am using CodeIgniter for a Project .
Folder Structure - htdocs/NewProject/application
I created a controller called Home -  applications/controllers/home.php
I created a view - applications/view/index.php
I can access the index.php without issue .
My route is
$route['default_controller'] = "home";

I can access localhost:8888/NewProject  - which takes me to home#index
I have put some css and images folders inside views folder .
How do i access the images and css ?
Trying to access NewProject/css/xxx.css leads to not found error .
Is there any alternate folder where I should add these static files ? Do I need to add anything to the routes file ?
Answer: Make the structure something like this
root (is your root directory)
    /application
    /system
    /static
        /images
        /css
        /js

Change your base URL in config.php
/*
  |--------------------------------------------------------------------------
  | Base Site URL
  |--------------------------------------------------------------------------
  |
  | URL to your CodeIgniter root. Typically this will be your base URL,
  | WITH a trailing slash:
  |
  | http://example.com/
  |
  | If this is not set then CodeIgniter will guess the protocol, domain and
  | path to your installation.
  |
 */
 // use this if your project is in root
$config['base_url'] = 'http://example.com/';

// use this if your project is in folder/subfolders
//$config['base_url'] = 'http://example.com/newproject/liveproject/';

Autoload URL helper in autoload.php
/*
  | -------------------------------------------------------------------
  |  Auto-load Helper Files
  | -------------------------------------------------------------------
  | Prototype:
  |
  | $autoload['helper'] = array('url', 'file');
 */

$autoload['helper'] = array('url');

In your view file
<link rel="stylesheet" type="text/css" href="<?php echo base_url(); ?>static/style.css" />

<img src="<?php echo base_url(); ?>static/images/myimage.jpg" alt="" />

Hope this helps. Thanks!!
Comment: This works for me when i `leave $config['base_url']` as an empty string Thx for the solution


SQL Server to check all columns in a databases views for a specific string value

Question: My employer has a third party application that has a very complex set of obliquely named views. I am trying to locate the views that contain specific data that is entered in the applications UI.
I have SQL to build a CTE with all view names and column names...
What I can't figure out is how to test every column (that could hold a string value) in every view for a specific string value.
Here is the start I made which doesn't work, but will illustrate what I am trying to do. Please note that it is also missing a constraint on the columns to check based on what can hold a string value.
The other obvious problem with the below is that I want to evaluate the value of the column and NOT the column name for value.
WITH ViewColumn_CTE (ViewName, ColumnName) AS
(
    SELECT TOP 100
        V.Name as ViewName,
        C.Name as ColumnName
    FROM
        sys.views V
    JOIN
        SysColumns C ON V.Object_ID = C.ID
)
SELECT TOP 1
    ViewName,
    ColumnName
FROM
    ViewColumn_CTE
WHERE
    ColumnName = 'Cash Equivalents'

Comment: This is a bad idea since N number of views can reference a single table where the data is *actually* stored. Views are just *virtual* tables that are defined by a query.  What you should care about is what *tables* the data is stored in. From that, you can find what *views* reference these *tables* if you really care. Regardless, searching every row for every table (or view) that has a `varchar` value sounds horrible.  What would be better is to use SQL Profiler or another tool to trace what query is being called when the user enters the data. Then, you'd know the tables.
Comment: Where do you have the view definition in your CTE? Most (if not all) database system offer you the definition in their system tables somewhere, but you will have to read the documentation or consult with your DBA what the name of these is. Please also [edit] your post and add the SQL server product that you are using as a tag.
Comment: I edited the tags to include SQL server 2014.  Sys.views is only returning the view names, I am not seeing any useful definition information there.  SysColumns would seem to have datatype information but it isn't jumping out at me.  I can see field length, nullability, etc... when I select * from it.
Answer: I have such a script flying around here. You can use it for a start.
It gets the list of interesting columns from the catalog iterates over them and queries them using dynamic SQL.
DECLARE @searched_value nvarchar(MAX) = 'a'; -- set to the value you search for

SET NOCOUNT ON;

DECLARE @schema_name sysname;
DECLARE @table_name sysname;
DECLARE @column_name sysname;
DECLARE @sql nvarchar(MAX);
DECLARE @result TABLE ([schema_name] sysname,
                       [table_name] sysname,
                       [column_name] sysname,
                       [value] nvarchar(MAX));

DECLARE cursor_all_columns CURSOR
                           LOCAL
                           FAST_FORWARD
FOR
SELECT s.name,
       o.name,
       c.name
       FROM sys.schemas s
            INNER JOIN sys.all_objects o
                       ON o.schema_id = s.schema_id
            INNER JOIN sys.all_columns c
                       ON c.object_id = o.object_id
            INNER JOIN sys.types y
                       ON y.user_type_id = c.user_type_id
       WHERE o.type = 'U' -- set to 'V' for views
             AND lower(y.name) IN ('char',
                                   'nchar',
                                   'varchar',
                                   'nvarchar'); -- include more types if needed

OPEN cursor_all_columns;
FETCH NEXT FROM cursor_all_columns
           INTO @schema_name,
                @table_name,
                @column_name;
WHILE @@fetch_status = 0
BEGIN
  SET @sql =   N'SELECT ''' + quotename(@schema_name) + N''',' + nchar(13) + nchar(10)
             + N'       ''' + quotename(@table_name) + N''',' + nchar(13) + nchar(10)
             + N'       ''' + quotename(@column_name) + N''',' + nchar(13) + nchar(10)
             + N'       ' + quotename(@column_name) + N'' + nchar(13) + nchar(10)
             + N'       FROM ' + quotename(@schema_name) + N'.' + quotename(@table_name) + N'' + nchar(13) + nchar(10)
             + N'       WHERE lower(' + quotename(@column_name) + N') LIKE N''%' + lower(replace(replace(replace(@searched_value, '%', '!%'), '[', '!['), ']', '!]')) + N'%'' ESCAPE ''!'';' + nchar(13) + nchar(10);

  INSERT INTO @result
              EXEC sp_executesql @sql;

  FETCH NEXT FROM cursor_all_columns
             INTO @schema_name,
                  @table_name,
                  @column_name;
END;
CLOSE cursor_all_columns;
DEALLOCATE cursor_all_columns;

SELECT [schema_name],
       [table_name],
       [column_name],
       [value]
       FROM @result;

Is is meant for tables, but it should also work for views, if you change the object type from 'U' to 'V'. (Though the tables might be more interesting thing anyway.) No guarantees though.
Set the value you search for at the beginning.
It searches for any column of the (n)(var)char type if it contains the searched value, case insensitive. If you want any custom types derived from string types included you have to adapt it accordingly.
The output is the schema, the table (or view) name, the column name and the value, that matched (a column might be listed more than once if multiple rows contain the searched value in that column).
(Disclaimer: Might have room for improvement or bugs.)


Chord change question (C#m, F# and G#)

Question: So this song is a no capo song. It's an all barre chord song, it's simple but a pain. The song is 8 minutes long of all barre chords. It's a C#m, F#m, A (barred version), and G# in that order. My hand cramps and hurts too much from doing all barre. How can I play the chords easier by putting a capo somewhere? Like A (which is easy) chord, if I put a capo on second fret and play G, it makes the A chord. So is there an easier way at playing these chords if I had a capo on somewhere to where it changed the shape?
Comment: Could you post the sheet music? It would make it easier to see what the problem is.
Comment: With an A and a G#, there's nowhere you can put a capo and play all 'open' chords.
Comment: The song is GUNSLINGERS GLORY BY the dead south
Answer: You could put the capo on the 4th fret and play open chords Am, Dm, F (the F would have to be barred), E. 1 out of 4 barred chords isn't bad. Like @Tim says they can't all be open chords though.
Answer: With capo on the 4th fret:

Am (C#m) x02210
Dm (F#m) xx0231
F  (A)   x3321x : fingers 3421, T mutes string 6, finger 1 mutes string 1
E  (G#)  022100

...this is essentially @JohnBelzaguy's answer except you don't need to play the F (A) as a barre chord, not at all! It's not open, but it's also not using a barre. I don't know if you balk at the bass note being the fifth instead of the root, but I often play regular, no capo F that way, because a barre on fret 1 is hard for me. I don't mind the fifth in the bass when strumming chords if the chord isn't too low.
Comment: Good solution for banning the barre altogether, +1. Personally I like to avoid the 5th in the bass unless it's specifically an inversion, how about a compromise, a "baby barre" (just made that up) xx3211, with fingers 3211? It makes the voice leading a little smoother too.
Comment: I like that "baby" barre chord!

Chord change question (C#m, F# and G#)


Breaking down statistics at API level or Flow level in K6?

Question: I was taking a quick look to K6 from loadimpact.
The graphs that I got so far show TPS, Response Time, Error rates at the global level and that is not too useful.
When I load test, I rather have those stats at the global level, but also at the flow level or at the APi level. This way if for example if I see some high latency I can tell right away if is caused by a single API or if all APIs are slow.
Or I can tell of a given API is giving say HTTP/500 or several different APIs are.
Can K6 show stats like TPS, Response Time, HTTP status at the API level, the flow level and global level?
Thanks
Answer: Yes, it can, and you have 3 options here in terms of result presentation (all involve using custom metrics to some degree):

End of test summary printed to stdout.
You output the result data to InfluxDB+Grafana.
You output the result data to Load Impact Insights.

Global stats you get with all three above, and per API endpoint stats you get out of the box with 2) and 3), but to get stats at the flow level you'd need to create custom metrics which works with all three options above. So something like this:
<code>import http from "k6/http";
import { Trend, Rate } from "k6/metrics";
import { group, sleep } from "k6";

export let options = {
    stages: [
        { target: 10, duration: "30s" }
    ]
};

var flow1RespTime = new Trend("flow_1_resp_time");
var flow1TPS = new Rate("flow_1_tps");
var flow1FailureRate = new Rate("flow_1_failure_rate");

export default function() {
    group("Flow 1", function() {
        let res = http.get("https://test.loadimpact.com/");
        flow1RespTime.add(res.timings.duration);
        flow1TPS.add(1);
        flow1FailureRate.add(res.status == 0 || res.status > 399);
    });
    sleep(3);
};
</code>
This would expand the end of test summary stats printed to stdout to include the custom metrics:
Comment: Funny that I didn't see this in the docs, at least not at first glance. Even I installed grafana and nothing. Many thanks for the feedback!


Is there anyway to make git-push interactive?

Question: On my remote git server, I have a pre-receive hook that will check-in (commit) into a clearcase repository using rcleartool's non-interactive command. What I'm working on is essentially a git to clearcase bridge. Unfortunately, this commit command requires a username and password as arguments.  Ideally, I would like the client to be prompted for the username and password when the git-push command is invoked, which will then be passed into the clearcase commit command.  However, the docs for git-push state that I can only echo messages back to the client, thus it looks like I cannot prompt the user for input.
Does anyone know of a way around this?
Has anyone tried something to make git-push seem interactive and succeed?
Any help would be grateful.
Answer: Without having an interactive git-push, there is no easy solution for this.
One possible way would be to have an intermediate between the user push and the rcleartool checkout.
If you have gitolite for instance, based on the forced command ssh principle, you can then used that intermediate script to:

identify who does the push (more conclusively than by just looking at a user.name git property)
look for an associated password stored (crypted) somewhere on the Git server (which can be problematic if said password change on a regular basis)

With those two additional informations, the Git server can then proceed with the rcleartool checkout part.
Comment: Hi VonC, thanks for the quick feedback! I've thought about storing the encrypted password somewhere on the git server, but ultimately, I'll need to send the password to rcleartool checkout in an decrypted form...which is a security issue.  And I don't think rcleartool accepts encrypted passwords =(.  You wouldn't know of a way around this too would you? =)
Comment: @Ken: no I don't know a workaround, since `rcleartool` don't support `SSL` yet (https://www-304.ibm.com/support/docview.wss?uid=swg21424834)


notion of the minimum of a function over a polytope

Question: Let $P$ be a polytope with $M$ vertices. (The polytope $P$ is the intersection of the hypercube $0≤x _j ≤1$  with the hyperplane $\sum_{j=1}^nx_j=t$, $0\leq t\leq n$). Suppose that the volume of $P$ is $Vol(P)=A$. We subdivide $P$ into $M$ pieces each of the volume $A/M$.
Let $f$ be a function, such that the integral below exist. Then,
$$
\int_{P}f(x)ds\geq \sum_{i=1}^{M}\min_{P_i} f(x)|P_i|=\frac{A}{M}\sum_{i=1}^{M}\min_{P_i} f(x).
$$
Could you please help me to understand the notion of the $\min_{P_i} f(x)$, i.e. what is this and how one can find this minimum?
Thank you for your help.
Answer: It's no big deal: The $M$ pieces $P_i$ are essentially disjoint and make up $P$. Therefore
$$\int_P f(x)\ {\rm d}(x)=\sum_{i=1}^n \int_{P_i} f(x)\ {\rm d}(x)\ .$$
If $f$ is, e.g., continuous on $P$ then for each $i$ there is a point $\xi_i\in P_i$ such that $\mu_i:=f(\xi_i)\leq f(x)$ for all $x\in P_i$. The minimal value $\mu_i=\min\{f(x)\ |\ x\in P_i\}$ is well defined, but there might be several points $\xi\in P_i$, where the minimum is attained.
From basic properties of the integral it follows that
$$\int_{P_i} f(x)\ {\rm d}(x)\geq\int_{P_i} \mu_i\ {\rm d}(x)=\mu_i\ |P_i|\ ,$$
and putting it all together we arrive at
$$\int_P f(x)\ {\rm d}(x)\geq \sum_{i=1}^n \mu_i\ |P_i|={A\over M}\sum_{i=1}^n \mu_i\ .$$
Remains the problem of determining the $\mu_i$, resp., finding the $\xi_i$. Since the combinatorial description of $P_i$ can be quite complicated the use of calculus methods is very cumbersome (one would have to look at all lower-dimensional faces of $P_i$). When $f$ is in fact linear you can be sure that the $\min$ is taken at a vertex of $P_i$; in this case it suffices  comparing the vertex values and selecting the lowest one.


How can I add values from a dictionary into a new column in a CSV?

Question: If I have data that I'd like to add as new columns in an already built CSV.
I have a dictionary with the new data to add for each dictionary key (corresponds to the person's ID in the CSV).
import pandas as pd

#New values to add for each ID
#'ID': Area, Age, Temperature
myData = {
    'A00147': (189, 40, 39),
    'A00198': (142, 28, 28),
    'A00360': (202, 40, 39),
    'A00367': (168, 33, 33),
    'BA9330': (142, 29, 29),
}

I have this CSV:

ID,Sex,Height,Weight,Hair,Eyes
A00147,Male,5 ft. 07 in.,185 lbs.,Brown,Blue
A00198,Male,6 ft. 02 in.,190 lbs.,Red or Auburn,Brown
A00360,Male,5 ft. 09 in.,167 lbs.,Gray or Partially Gray,Green
A00367,Male,6 ft. 00 in.,245 lbs.,Black,Brown

I would like it to become:

ID,Sex,Height,Weight,Hair,Eyes,Area,Age,Temperature
A00147,Male,5 ft. 07 in.,185 lbs.,Brown,Blue,189,40,39
A00198,Male,6 ft. 02 in.,190 lbs.,Red or Auburn,Brown,142,28,28
A00360,Male,5 ft. 09 in.,167 lbs.,Gray or Partially Gray,Green,202,40,39
A00367,Male,6 ft. 00 in.,245 lbs.,Black,Brown,142,29,29

So I would be adding the new columns with their values to the corresponding ID.
So the first iteration of the loop would look into the dictionary myData, look at the first key ('A00147'), check if it exists in the CSV's ID column. If it does, it'll create the 3 new columns and add their respective values. Then it would look for 'A00198', if it's there, add the 3 new values in those previously created new columns. If it finds a key that doesn't exist in the CSV ID (ex: 'BA9330'), it would just skip over it.
I'm thinking of using Pandas for this task, but not entirely sure how to approach it.
Answer: Create a dataframe from myData then merge it with your actual dataset:
import pandas as pd

df = pd.read_csv('data.csv')

cols = ['Area', 'Age', 'Temperature']
df1 = (pd.DataFrame.from_dict(myData, orient='index', columns=cols)
         .rename_axis('ID').reset_index())

out = df.merge(df1, on='ID', how='left')
out.to_csv('data.csv', index=False)

Output:

ID
Sex
Height
Weight
Hair
Eyes
Area
Age
Temperature

A00147
Male
5 ft. 07 in.
185 lbs.
Brown
Blue
189
40
39

A00198
Male
6 ft. 02 in.
190 lbs.
Red or Auburn
Brown
142
28
28

A00360
Male
5 ft. 09 in.
167 lbs.
Gray or Partially Gray
Green
202
40
39

A00367
Male
6 ft. 00 in.
245 lbs.
Black
Brown
168
33
33


how to use Flask to save both uploaded file and user text input in json

Question: in html, I have
<form method="POST" enctype="multipart/form-data" action="/">
  <input type="number" name="speed " id="speed">
  <input type="file" id="stl_file">
  <input type="submit" value="submit">
</form>

I would like to save the value for speed in json file and save the uploaded stl file as well.
What I did in my app.py is
from flask import Flask, render_template, request,flash, redirect, url_for
from werkzeug.utils import secure_filename
import os
import json
app = Flask(__name__)

ALLOWED_EXTENSIONS = {'stl','STL'}

def allowed_file(filename):
    return '.' in filename and \
           filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS

@app.route("/")
def index():
    return render_template('main.html')

@app.route('/' , methods=['POST'])
def get_data():
    if request.method == 'POST':
        results =request.form
        file = request.files['stl_file']
        with open('file.json','w') as f:
            json.dump(results, f)

        if 'file' not in request.files:
            flash('No file part')
            return redirect(request.url)
        # if user does not select file, browser also
        # submit an empty part without filename
        if file.filename == '':
            flash('No selected file')
            return redirect(request.url)
        if file and allowed_file(file.filename):
            filename = secure_filename(file.filename)
            file.save(filename)
            return redirect(url_for('index'))

        return render_template('main.html')

if __name__ == '__main__':

   app.run()

I can run it and I see my main page. However, when I input a number into speed, upload a file, and click submit, I get
Bad Request
The browser (or proxy) sent a request that this server could not understand.

Any suggestion on that? Thanks in advance.
Answer: You haven't defined the name attribute of file.
<input type="file" name="stl_file" id="stl_file">


Beginner books for music appreciation for music-theory illiterate adult

Question: I'm a married adult with a full-time occupation. A long time ago, I took some lessons for the keyboard but have since forgotten almost everything. Could you please recommend me a few books given the following 2 goals:

A basic understanding of music theory and an ability to play a few basic songs on the keyboard (piano). I still have a basic keyboard at home.
A basic ability to appreciate (classical) music on an intellectual level. This includes a basic ability to verbalize certain musical patterns and effects.

The things I have learned properly in life almost always begin with some reading materials and then practice. Please also advise if this is a proper way to go as far as some basic music understanding is aimed for. Thank you very much.
Comment: Please give ideas for editing/improving the question instead of downvoting. I've been wanting to ask this question for years. Please be lenient.
Comment: AB Guide Music Theory (its pink)
Answer: ABRSM Music Theory books are very well put together. I prefer the Josephine Koh books above all the others. When you do the music theory you will learn all the ways and manners in which musicians use to portray feelings and emotions.
You can find out more at ABRSM USA, then scroll down to the page for 'Publications'. Alternatively, see this list for all Music Theory publications by ARBSM.
Answer: If you are truly a beginner and a serious student who really enjoys music, I recommend the three-book series of Alfred's Basic Adult Piano Course: All-In-One Piano Course Lesson Theory Solo starting with Level 1.  I purchased Level 3 on Amazon.com. who has both new and used books available. Mine is spiral bound. I believe that there is a piano instructor on youtube who teaches levels 1&2, so before you buy go to youtube search for this book level 1 and listen to the videos. They are so helpful, taking a slow tempo approach and give some technique tips, according to your level of pianism that are very well done! When your book arrives you will be excited to see a complete Table of contents, broken down by sections, fun songs just for fun, comprehensive theory workbook pages, Lesson music scores, Scale and Chord Sheets, An Ambitious Section, Dictionary  of Musical Terms and Certificate of Award.
The songs are varied in genre: Some Folk, Jazz, Rag, Pop, Spiritual, Classical, National, Holiday, Opera and others by over 20 composers. You will discover the joy of music in a stepwise fashion, learn to understand the rules of music theory and music definitions step by step, have solos to perform and enjoy the accomplishment of completing the lessons in Level 1 to receive your Certificate included, so you then will progress to level 2 then 3.  I wish they included Level 4 Advanced- well maybe if we email them they will see the need to write levels 4&5- I sure hope so.
Also there is an interesting piano instructor on youtube withe a great sense  of humor-PianoTV that I believe you will really enjoy after you complete level 1 or mat be evendors before.  She provides interesting and entertaining videos on practically all aspects of learning music History, Appreciation, Theory (her videos show the specifics in everything from beginner tips, tutorials on how and what to practice to reading intervals and specific chord and music patterns and she provides free scores on her website. She did a beautiful "Oh Holy Night" tutorial for Christmas for Level 3 students.
Hope this sounds good to you-Best!


Boolean algebra - dual function and self dual function

Question: I'm having a problem understanding what influences the dual function has on the original function output.
For example,  if I have a self dual function -  for the input 11 the function out put will be 1,  and the output of the dual function will be the same.
Now for dual function -  I saw on the Internet that  the dual function always  give the opposite output   compare to the original function.
Im pretty confused  because if we look on the function  A*B
Then the dual function is A+B.
now for the input A=1,B=1 they giving the same output.
Some one can explain me what dual function  is beside changing the operators?
Thanks!
Answer: Dual function simply implies to change an expression to its opposite logic I.e changing negative logic to positive logic and vice versa.
By negative logic, I mean logic where high means 0 and low means 1.
By positive logic, I mean logic where high means 1 and low means 0.
Dual function of an expression is simply obtained by exchanging OR operator to AND and AND to OR.
Sometime dual function of an expression results in same expression.
For example F=AB+BC+AC
Talking about A*B, A+B is surely dual function of it. Output should be opposite, as I said dual function mean changing logic(positive to negative or negative to positive).
During changing operators, change your input also (1 to 0 or 0 to 1)


How to update temporal document in MarkLogic

Question: I have requirement to change the position of the few fields in my historical harmonized data(temporal).Which function I can use for this
Comment: you might want to elaborate your question with details and also what have you tried
Answer: You can use temporal functions, but generally you cannot do any non-temporal operations on a temporal document, if that is what you are asking.  You can use
https://docs.marklogic.com/temporal:collection-set-options
to allow an override.  With the override you can do any non-temporal operation.
The issue is that temporal queries are not guaranteed consistent in the presence of non-temporal updates.  The results of a temporal query can be different after an override update is made.


intellisense python

Question: Estou aprendendo a programar em Python e está ocorrendo, de forma recorrente, a seguinte situação (isso ocorre na IDE pycharm, atom e spyder):
Importei a matplotlib da seguinte forma:
import matplotlib.pyplot as plt

Quando digito: plt. abre uma janela contendo todos os métodos que eu posso invocar. Então seleciono plt.axes(). Se digitar plt.axes() nenhuma janela é exibida com os métodos que eu posso chamar.
No livro que estou usando ele executa o comando:
plt.axes().get_xaxis().set_visible(False)

Não entendo o porque não aparece nenhuma janela exibindo os métodos conforme apresentado acima.
Como resolver tal situação?
Comment: Isso não é só no Python, as IDEs em geral não dão sugestões ao retorno de uma função, basicamente porque pode acontecer muita coisa numa função e retornar inúmeros valores ou nenhum. O DevTools do chrome não dava sugestões ao retorno de funções, mas recentemente isso mudou, agora quando você escreve no console JS, por exemplo, `document.body.getElementById('el')` e existir esse elemento já tem sugestões como `innerHTML`, `value`, etc
Comment: Você sabe qual a versão do python o livro esta utilizando ?
Comment: Removi a minha resposta, já que interpretei a pergunta erroneamente.
Comment: Ele usa a versão 3 do python. Então pelo que eu entendi é normal esse comportamento da IDE?
Answer: Python é uma linguagem dinâmica, portanto, não é possível saber de antemão quais métodos podem ser invocados em um objeto, sem executar o código. O que a maioria das IDEs fazem é uma análise estática dos objetos, retornando métodos associados ao seu "tipo inferido", porém, no caso de respostas dinâmicas de funções, atributos dinâmicos e classes com __getattr__ definido isso é muito difícil ou impossível de fazer sem executar o código. Pode até retornar resultados inválidos.
A solução é ignorar o que o editor está completando e escrever o código na raça - Use o recurso de completar código da IDE para te ajudar a escrever nomes longos, mas não dependa dele para listar os métodos disponíveis; Para isso use a documentação.
Pra ilustrar segue um exemplo de código dinâmico: Uma classe é definida com um método chamado metodo_nao_existe porém logo depois esse método é removido da classe. A sua IDE provavelmente vai se confundir:
class Teste:
    def metodo_nao_existe(self):
        print("Eu nao existo")

def _metodo(self):
    print("Eu existo")

del Teste.metodo_nao_existe
Teste.metodo_existe = _metodo

t = Teste()
t.metodo_existe()

intellisense python


Let Gmap center on the users geolocation, but still load all the node markers onto the map

Question: For a website I am building I have a used a map to display all the nodes inside my country, by using a view. When the map is loaded, it shows the whole country (by standard settings in the Gmap module).
What I would like to do having this map centered on my location, but still show/load all these node markers that are spread across the map.
I manage to have the map centered on my location by using Filters and SmartIP in the view, but offcourse this limits the loaded nodes to only that location. I do not want to filter anything, just center the map. I am new to Drupal, and can't figure out how to make this work.
Comment: How are you centering the map with views? Are you using the a contextual filter from the location module?
Answer: I can clarify after my comment on your question is addressed but if I had to guess I would say you are using an nid contextual filter to center the map?
If so, that is why you aren't getting other nodes.
Alternatives are:

Add the Global: Null contextual filter and pass the nid through that. The null contextual filter will not affect results at all.
Keep using the nid contextual filter, but check the "Exclude" checkbox in the "More" fieldset of the filter. This will mean all nodes except the given nid will be displayed, but the given nid can still be used to center the map.

Use whichever method gives you the functionality you want.


for a survey conducted in 3 countries, how to examine the equivalence of factor structure by country?

Question: We conducted a survey in 3 countries, and we want to run factor analysis (EFA) to identify latent factors. With the combined data sets, we can run EFA to develop factors.
But to examine the equivalence of factor structure by country, what shall I do? shall I do 3 different EFA using 3 subsets, each from one single country?
Answer: You can do a Procrustes rotation to a target matrix to get descriptor of the similarity of the three solutions.
A better solution is to move to a multiple group confirmatory factor analysis approach, which lets you directly test the equivalence of the three factor solutions.
Comment: Look at Vandenberg & Lance (2000) for a nice conceptual review of the process of evaluating measurement invariance (what Jeremy is encouraging you to do in CFA). In terms of implementing it yourself for your research, I would look at the lavaan and semTools packages for R, which can be used with relative ease to fit measurement invariance models (you can easily google their documentation, and the lavaan package has a nice website with easy-to-follow tutorials)
Comment: Thank you, Jeremy. I will google how to do it. If you can explain a little bit here how to do it, it will be greatly appreciated.


Views 3: display "block" doesn't show full node

Question: I would like to show full node including comments in views. Everything is ok if I choose display "page" but in case of "block" all comments disappear from the view only content and link "add comment". What is wrong in my case and how to correct it? I need a block with full node content including comments.
My parameters in views:
Format:
Format: Slideshow
Show: Content; All (I also cheked "show comments" and "show links")
Thanks in advance
Comment: Which module are you using for slideshow format ?
Comment: views slideshow
Answer: When you select teaser view instead of full content you can see comments in your block.
If you see implementation of comment_node_view() comments are displayed only in node lading pages.
if ($node->comment && $view_mode == 'full' && node_is_page($node) && empty($node->in_preview)) {
  $node->content['comments'] = comment_node_page_additions($node);
}

If you enable the same block in node landing pages you can see the comments in your view.
If you enable the block other than node landing pages, then you cannot see the comments because of node_is_page($node) condition i think.


Proving that $R$ is a field $\iff (x)$ is a maximal ideal in $R[x]$

Question: EDIT: I know that there is a commonly known way of proving this, namely using the isomorphism theorem and the quotient $R[x]/(x)$ but I didn't get any feedback whatsoever on my proof, why is it valid/invalid? Can somebody please comment on that?
Let $R$ be a commutative unital ring.
The question is to prove that for an $x\in R$ we have "(x) is a maximal ideal" $\iff$ "$R$ is a field".
$(x)$ is the ideal generated by $x$ in $R[x]$, so $(x)=\{p_1x+...+p_nx^n\ :\ n\in\mathbb{N},\ p_i\in R\}$
My attempt:

If $R$ is not a field $\exists \tilde p_0\in R$ s.t. $\tilde p_0$ is not a unit.
Now if we look at (x), we have that since there is no constant element in $(x)\ \ $ $(x)\subsetneq\{k\tilde p_0+p_1x+...+p_nx^n\ :\ k,p_1,...,p_n\in R\}:=I\subsetneq R$ which is an ideal that is different from $R[x]$ because $k\tilde p_0$ can never be equal to $1$. So $(x)$ is not a maximal ideal.

for the other way:

If $(x)$ is not a maximal ideal, $\exists I\ne R$ s.t. $(x)\subsetneq I\triangleleft R[x]$ and because $I$ is a maximal ideal it cannot contain $1$ (otherwise we get $I$=$R[x]$) so the constant element of every element of $I$ has to be $\ne 1$ so there exists some element that stays $\ne 1$
no matter by what you multiply it $\implies R$ is not a field

I am not really sure about those reasonings so any comment is welcome.
Comment: Of course you mean $(x)$ is a maximal ideal *in $R[x]$*.
Comment: @Wojowu yes, sorry
Comment: @StanislasHildebrandt If it is not very obvious why polynomials with constant terms are not in the ideal generated by $x$, you should consider what causes you to doubt that and revisit the definitions of ideal, ideal generated by a set, and/or polynomial that you're are using if necessary. Then "rigorously" prove the statement. You really want to be at a point where that statement is intuitively obvious.
Comment: @DerekElkins yes, I re-examined the definition of an ideal generated by a set and I have no doubt about that anymore
Answer: There is a map $R[X]\to R$ that sends a polynomial $p$ to $p(0)$. The kernel of this map is $(X)$, and this map is onto, so there is an isomophism $R[X]/(X) \simeq R$. Thus $R$ is a field iff $R[X]/(X)$ is a field iff $(X)$ is maximal.
Comment: Could you explain the last equivalence please?
Comment: @StanislasHildebrandt Incidentally, that is explained in the other answer!
Comment: https://math.stackexchange.com/questions/147047/shorter-proof-of-r-i-is-a-field-if-and-only-if-i-is-maximal?rq=1 If someone is reading this


Abstract physical drives to create a resizable raid array: is it possible?

Question: I want to set up two hard drives with a portion of them set in raid1. I want that portion to be scalable too, as logical LVM volumes are. Is it possible to abstract physical volumes or to create an array based on logical volumes?
I'm thinking about something like nested LVMs.
Comment: Next to that, you can create raid arrays from lvm logical volumes, too. Nothing prohibits that, although it is a little bit uncommon (and raid autodetection isn't the best on it too).
Comment: Thank you for your answer, aren't LVs treated like partitions and not physical drives? I thought that raid would only work with physical disks
Comment: Linux software raid works between block devices, if they are partitions, physical disks, logical disks, or even ramdisks, it doesn't matter. This is the main ground behind my heretic opinion, that the so-named hw raids never could reach the flexibility of the linux software raid.
Answer: I know you can do the following in Linux LVM and can only assume that it may be possible on other OS's:
Use the Volume Manager to set up your redundancy and data striping for reliability and performance.
Simply use both disks as physical volumes for a LVM volume group and create a LVM logical volume with the correct redundancy and striping  when setting up logical volumes.

-m, --mirrors Mirrors
                Creates  a  mirrored  logical  volume  with Mirrors copies.  For
                example, specifying -m1 would result in a mirror with two-sides;
                that is, a linear volume plus one copy.

So for example the commandline lvcreate -m1 -L 10G -n <name> <volume_group> would create a mirrored logical volume or the equivalent of a RAID1 array.

-i, --stripes Stripes
                Gives the number of stripes.  This is equal  to  the  number  of
                physical volumes to scatter the logical volume.  When creating a
                RAID 4/5/6 logical volume, the extra devices which are necessary
                for  parity  are internally accounted for.  Specifying -i3 would
                use 3 devices for striped logical volumes, 4  devices  for  RAID
                4/5, and 5 devices for RAID 6.

If you have three disks 2 would be the maximum number of stripes (the third is for parity) and lvcreate --type raid5 -i2 -L 20G -n <name> <volume_group> would set up the equivalent of three disk RAID5 array.
Comment: Thank you! That's what I need. I didn't know about the mirroring option in LVM. Unfortunately, I can't even upvote you since I don't have enough points :(
Comment: @InfiniteSnow: marking the answer as accepted if nothing better comes along is enough. And please don't cross-post on multiple Stack Exchange sites please.
Comment: Was waiting to be enabled to do it, had to wait 5 minutes. Is cross posting a frowned-upon practice? I have deleted the other question.
Comment: @InfiniteSnow The idea is that if your question is off-topic on one site it can be moved to  a more relevant one. Thanks for accepting the answer and hope to see more of you!


Multiple categorical IVs in meta-regression

Question: I'm running a meta-regression and am inputting at different study characteristics as IV's. The problem is some are continuous and some are categorical.
The IV's are:
Age
Gender (2 values, dummy coded)
Study design (2 values, dummy coded)
Population type (4 values, dummy coded).
I did univariate analyses first, and age and study design were significant. I also did a univariate analysis with population type, and one of them was significant (although I understand this means it is only significant from the reference category).
When I put all the significant ones into a multiple meta-regression (i.e. age, study design, populationtype1, populationtype2, populationtype3), only one of the population type dummy codes is significant.
My questions are:
Can you put two (or more) categorical iv's in one analysis (does the constant apply to both categorical variables e.g. the reference study design and population type)?
How do I interpret this if this is the case?
What further analysis should I do to unpick the effect of population type?
I hope this makes sense
Answer: Since meta-regression is very similar to standard regression you can do anything you could do in the standard form in meta-regression. So it is fine to include as many moderators as you wish of whatever type, categorical or continuous. Just as in standard regression though the results you get will depend on the correlation between the moderators and the number of primary studies you have.In your case you have six moderators (counting population type as three) so I would have thought the minimum number of primary studies to get a sensible result was between 50 and 100.


Keyboard library python3

Question: I am trying to do a python script and while the script is running in the background I want it to listen at some buttons, the problem is that I can't even this simple program to get to work, I get segmentation fault 11. When I run it I use sudo python3 prog.py.
import keyboard
keyboard.wait('esc')
print("test")

Answer: Couple of things:

You don't compile a Python program
The problem is that you don't have permission to access that memory, hence the segfault. The fix depends on your system, on my Mac I just changed the permissions of the terminal (so that it could control my computer).
Also, I just want to mention that a process in the background does not take in data from STDIN, so you may have a problem there depending on what you mean.
Comment: Thx! It solved the problem. Mb I meant run the script!

Keyboard library python3


Is the front page considered in_active_trail?

Question: I'm learning Drupal 8, so still very new to it. I'm working on customizing the main menu. I've been able to add Twitter Bootstrap to my custom theme.  I've made a custom menu--main.html.twig file.
I'm trying to change the "is-active" class to "active" using what Drupal has available.  I see when I click to my About page that they About link is highlighted with the active link class.  Great!
What I can't seem to figure out is how do I get the Home page to get the active class too?  It looks like it is being treated differently.
I tried adding a new page called Homepage, but that didn't work either.
Here is my twig file, which i took pieces from another theme that use twitter bootstrap to try to compile it together.
{% import _self as menus %}

{#
  We call a macro which calls itself to render the full tree.
  @see http://twig.sensiolabs.org/doc/tags/macro.html
#}
{{ menus.menu_links(items, attributes, 0) }}

{% macro menu_links(items, attributes, menu_level) %}
  {% import _self as menus %}
  {% if items %}

    {% if menu_level == 0 %}

    {% else %}

    {% endif %}
    {% for item in items %}
    {%
      set classes = [
        menu_level ? 'dropdown-item' : 'nav-item',
        item.is_expanded ? 'menu-item--expanded',
        item.is_collapsed ? 'menu-item--collapsed',
        item.in_active_trail ? 'active',
        item.below ? 'dropdown',
      ]
    %}

      {%
          set link_classes = [
            not menu_level ? 'nav-link',
            item.in_active_trail ? 'active',
            item.below ? 'dropdown-toggle',
            item.url.getOption('attributes').class ? item.url.getOption('attributes').class | join(' '),
          ]
        %}
        {% if item.below %}
          {{ link(item.title, item.url, {'class': link_classes, 'data-toggle': 'dropdown', 'aria-expanded': 'false', 'aria-haspopup': 'true' }) }}
          {{ menus.menu_links(item.below, attributes, menu_level + 1) }}
        {% else %}
          {{ link(item.title, item.url, {'class': link_classes}) }}
        {% endif %}

    {% endfor %}

  {% endif %}
{% endmacro %}

Update:  I went into Views in my admin panel and edited the Frontpage.  I noticed that it was not assigned to any menus.  I changed that by adding it to the main menu.  Looked at my homepage and the Home link is now highlighted as the active page.
So, what worked is adding it through the view page.  What did not work was adding the homepage from the "menu" screen where you build the menu. Clearly the homepage is treated differently from other pages in the menu and if you add it via the menu panel, it is not considered as part of in_active_trail.  I don't totally understand so I am hoping that someone can explain this to me.
Now, my question is did i do this right?
Update #3
I check my "Home" link in my main navigation.  The original "Home" link says "This link is provided by the Standard module. The title and path cannot be edited."  This link will not add the active class when I use "in_active_trail".
I had manually added a link via the Menu -> Main Navigation -> Add New Link.  I gave it a title called "Homepage" and the link "".  This link did not accept the "active" class either.
From what I can gather, the only way to get it to repsect "in_active_trail" is y using the Views -> Front page and then configure the menu setting.  That did the trick.
Comment: I have a similar problem, in fact the opposite.  in_active_trail was false when rendering a view attached to the main menu.Removing from the view and enabling from the menu seemed to add this.
Answer: As mentioned on Active on home menu item not working, it could happen because you are using "" as menu link instead of <front>.
The Menu Trail By Path module could resolve your issue. That module sets the menu item to active if it's for the page you are currently on.

Menu Trail By Path sets the active trail on menu items according to the current URL.
For example, if you are at https://example.com/blog/category1/article1, menu items with the following paths will get the active-trail class on them and expand accordingly.

blog
blog/category1
blog/category1/article1
Comment: When I added Home to the menu via the menu panel, i used  as the page link. I had put "/" in there, but it changed it for me.


Reorder list rows in flex mobile app

Question: How can I let the user reorder the rows of a list in a flex mobile app?
You know, you see it a lot in native iOS apps. Usually you hit an Edit button at the top of a list view. Each row then gets a thumb icon that lets you move the rows around in the list.
This might be totally obvious, but I'm writing my first mobile app in flex and can't seem to find an example anywhere.
Cheers.
Answer: Spark List control supports Drag and Drop, besides dragging and dropping with the list itself, you can also do so between two different list controls.  To enable the feature, you could set    dragEnabled,    dragMoveEnabled,    dropEnabled properties to true, more details may be found in this help document.
To enable the feature as you described, you could set those drag and drop properties to true when the edit mode is enabled.  Optionally, you could also display a drag handler (as decorator).
Update: For mobile application, there is a write-up on how to get around with mobile limitation of drag-and-drop.
Comment: The documentation says drag & drop is not supported in mobile apps. Is this wrong or did I misinterpret the documentation?
Comment: Forgot to include the work-around of mobile list drag and drop.
Comment: Just keep in mind that the work-around really is nothing more than a work-around. Not that it doesn't work, but it's not very nice as the performance is very, very bad. Works perfectly fine in the device simulator, but on an actual device it is sluggish as hell. So not production ready.

Reorder list rows in flex mobile app


Windows Server 2008 R2 "Missing Operating System" on boot

Question: I have a demo server that I brought over with me to Berlin for TechEd 2009, I booted the server up after I arrived and I got a message saying "Missing Operating System".
I carried on the server and it didn't get jarred or bumped. It's running Windows 2008 Server R2 Enterprise with two Intel X25-M 160GB SSD drives.
I checked the connections to the drives and they're tight.  Anything I should troubleshoot?  I'm trying to get my hands on an OS disk so I can boot to the command prompt to see if I can see anything there.
Thank you
Answer: There are some possible reasons for this:

The machine's BIOS doesn't detect the hard disk
The boot hard drive is damaged
Sector 0 of the hard disk has an damaged master boot record ( third-party programs or disk corruption can damage an MBR)
An incompatible partition is marked as "active"
A partition that contains the MBR is no longer active

If the MBR is corrupted, you could try to boot from the Windows installation DVD-ROM and use the
<code>bootrec.exe
</code>
command. Here is a desciption of the utility: How to use the Bootrec.exe tool in the Windows Recovery Environment to troubleshoot and repair startup issues in Windows.
Comment: Thanks, when I try to access the primary drive from a command prompt, it says "the volume does not contain a recognized file system".  Can SSD's get fried like this??
Answer: SSD's can fail just like anything else and NAND Flash based drives have some more interesting failure modes than other electronics as they are basically one huge exercise in managing statistical failure in the first place. Ars Technica published a nice overview of the history and state of the art of SSD storage this week that explains why.
Have you got any more details on the specific Intel SSD model (although if it's 128GB then I think it must be an X-25E) and whether you made any specific tweaks to the system. While the drives are excellent Intel have had issues with firmware releases, the most recent one for the X-25M G2 resulted in some drives getting bricked under Windows 7 (before it was pulled) and its likely that the same could happen with W2K8R2.
Are these set up as two standalone drives\RAID-0 or as a RAID-1 pack? If it's a RAID config then it's also possible that something went wrong at the controller level.
Comment: Helvick, they're Intel X25-M 160GB (I corrected my original post) set up as standalone drives.  I'm flying back from Berlin in a couple of days and am carrying on the server again.  Will be interesting to see if this happens again.