$R^2$ (coefficient of determination) and linearity in multiple linear regression

Question: For simple linear regression (SLR), in order for $R^2$ (the coefficient of determination) to be a meaningful measure, it must be true that $X$ and $Y$ are linearly correlated. Specifically, $R^2=r^2$, where $r$ is Pearson's correlation coefficient.
When we move into the multiple linear regression (MLR) framework, I'm curious how this linearity requirement transfers.
Take, for example, a polynomial model where $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. In this model, assume the regression assumptions are met (i.e., the data are truly related according to a parabolic curve, so fitting $x^2$ as a predictor allows us to meet the linearity assumption).
Now, $X$ and $Y$ are not linearly related, but $X$ and $X^2$ jointly allow us to correctly model the association with $Y$. So, since we've met the linearity assumption of MLR, $R^2$ is meaningful, correct?
So would the conclusion be that $R^2$ is meaningful if the modeled relationship between $Y$ and the predictors (whatever they may be (i.e., even if they're polynomials)) satisfies the regression assumptions?
If so, we would say: In the case of SLR, this forces the requirement of $X$ and $Y$ being linearly related, but for MLR, the relationship between $\textbf X$ and $Y$ may be curved, as long as the linearity assumption is met.
Comment: I can't create new tags due to my reputation being < 300, so if anyone would like to create R2, linearity and/or linearity-assumption, they seem reasonable.
Comment: I added the tag for `[r-squared]`. There is a tag for `[linear-model]`, but note that this means *linear in the parameters* (ie, the parameters of the model are coefficients (see [here](http://stats.stackexchange.com/a/111304/7290)); "linearity" in the sense of *rectilinear* isn't really an assumption of regression.
Comment: Thanks for adding r-squared. I was hoping to add a tag about linearity, indicating this problem deals with the linearity assumption (i.e., linear in the parameters) of regression. It also happens to deal with the "other" definition of linear, in that we need $X$ and $Y$ to be linear in order to meet the linearity assumption in SLR. I think adding the linear-model tag seems appropriate.
Comment: One resolution of this issue is to note that $R^2$ is the square of the correlation coefficient between the fitted and actual values.  This reduces the question to whether the predicted and fitted values appear to follow a linear relationship -- regardless of how nonlinear the relationship among the explanatory variables might be.  Some people might find the illustrations I posted at https://stats.stackexchange.com/a/354256/919 to be helpful, too.
Answer: Coefficient of determination in multiple linear regression: Firstly, it is not correct to say $R^2$ is "meaningless" if it is zero.  In a simple linear regression this just means that the response and explanatory vectors are uncorrelated, so it is very meaningful indeed.  More generally, in multiple linear regression the coefficient-of-determination can be written in terms of the correlations for the variables using the quadratic form:
$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}},$$
where $\boldsymbol{r}_{\mathbf{y},\mathbf{x}}$ is the vector of correlations between the response vector and each of the explanatory vectors, and $\boldsymbol{r}_{\mathbf{x},\mathbf{x}}$ is the matrix of correlations between the explanatory vectors (for more on this, see this related question).  If the regression model form is correct then this statistic is meaningful, insofar as it gives a measure of goodness-of-fit which derives from the underlying correlations between the variables.  If it is equal to zero then that gives a particular meaning, but it does not render the statistic meaningless.

Quadratic regression with no linear relationship: In the special case of the quadratic model you have specified, if there is no linear relationship, but there is a quadratic relationship then you have:
$$\boldsymbol{r}_{\mathbf{y},\mathbf{x}} = \begin{bmatrix} 0 \\ r_2 \end{bmatrix} \quad \quad \quad \boldsymbol{r}_{\mathbf{x},\mathbf{x}} = \begin{bmatrix} 1 & r_{1,2} \\ r_{1,2} & 1 \end{bmatrix}.$$
This gives you:
$$R^2 = \boldsymbol{r}_{\mathbf{y},\mathbf{x}}^\text{T} \boldsymbol{r}_{\mathbf{x},\mathbf{x}}^{-1} \boldsymbol{r}_{\mathbf{y},\mathbf{x}} = r_2^2 = \frac{(\sum (y_i - \bar{y}) (x_i^2 - \overline{x^2}))^2}{(\sum (y_i - \bar{y})^2) (\sum (x_i^2 - \overline{x^2})^2)}.$$
In this case the coefficient-of-determination is equal to the square of the correlation between the response vector $\mathbf{y}$ and the quadratic explanatory vector $\mathbf{x}^2$.
Comment: I'm not sure what the first half of your answer is regarding, as my question doesn't purport that $R^2$ is meaningless if it is 0. This is a question about when the assumptions of $R^2$ are met, so that its value - whatever it may be (0 or otherwise) - may be trusted ("is meaningful"). As with any statistic, its estimated value is meaningless if its theoretical assumptions are not sufficiently met in practice.
Comment: Perhaps I've misunderstood what you intended, but your first sentence claims that for $R^2$ to be a meaningful measure there must be correlation between $X$ and $Y$.  That is what I was responding to.  The issue here is that there are no assumptions required for $R^2$ to be a well-defined and meaningful measure.  If the regression assumptions are met then this quantity will have a particular stochastic *behaviour*, but regardless of this, it is a measure of the squared multiple correlation in the problem.
Comment: If you can be more specific about what behavioural properties you would like $R^2$ to have for this statistic to be "trusted" then perhaps we can assist in saying what assumptions are required for that behaviour.
Comment: Thanks for clarifying. My first sentence says, "For simple linear regression (SLR), in order for $R^2$...to be a meaningful measure, it must be true that $X$ and $Y$ are **linearly** correlated." The keyword is **linearly** - if they are not linearly related, this measure is not meaningful, because a value of 0 doesn't tell us there's no correlation, just no **linear** correlation ($X$ and $Y$ could be strongly related via a parabolic relationship, e.g., which this measure would miss). This question is about how that assumption translates into higher-level models.
Comment: Very simplistically: If the linearity assumption of a regression model (whether it be simple or multiple (including polynomial)) is met, does $R^2$ always have the same meaning?
Comment: Okay, so I think I see the problem here.  You are using the term "linear correlation", which is a strange term to use because ---in standard parlance--- the correlation coefficient is *always* a measure of the linear component of a relationship.  The meaning of $R^2$ relates to the measure of the relationship specified in the model, so if you have a parabolic relationship, but you fit it to a model with only a linear component, then the $R^2$ is going to be a measure that relates only to the linear approximation to that parabolic relationship.
Comment: *Pearson's* is a measure of linear correlation, but Spearman's, e.g., is not. Also, my question is not about a truly parabolic relationship incorrectly represented by a model with only a linear component. It's about a model that *correctly* has a quadratic term to account for the quadratic nature between the variables. Since the assumptions of the polynomial model are met, does $R^2$ have its standard interpretation as "the percentage of variation in $y$ accounted for by the predictors in the model"? That's my question: Does it matter what the model is as long as it's the correct model?
Answer: The nonlinear terms are new variables.
You write your model as $\hat y = \hat \beta_0 + \hat \beta_1x+ \hat \beta_2x^2$. Instead, define $x_1 := x$ and $x_2 := x^2$. Now $\hat y = \hat \beta_0 + \hat \beta_1x_1+ \hat \beta_2x_2$, and that looks linear, doesn't it?
The math does not have to know how you got $x_2$. All the model cares about are the values of your features. One you measured; one you calculated. The model just sees numbers.


How do you clear a field value with entity_metadata_wrapper()?

Question: Suppose I have an entity with a field field_foo on it and I want to clear that field's value.
What should I pass to $wrapper->set()?
I have tried NULL and array() and both produce an error message.
Comment: Are you looking for the [`EntityStructureWrapper::clear()`](http://drupalcontrib.org/api/drupal/contributions%21entity%21includes%21entity.wrapper.inc/function/EntityStructureWrapper%3A%3Aclear/7) or [`EntityValueWrapper::clear()`](http://drupalcontrib.org/api/drupal/contributions%21entity%21includes%21entity.wrapper.inc/function/EntityMetadataWrapper%3A%3Aclear/7) methods perhaps?
Comment: Calling `::clear` directly is not equivalent to setting the field to an empty value, as it does not call `updateParent` in the way calling `set` with an empty value does. Amongst other things, `updateParent` ensures that the `setter callback` defined in the property info (see http://drupalcontrib.org/api/drupal/contributions!entity!entity.api.php/function/hook_entity_property_info/7) is called.
Answer: The problem is that you must set an empty value that is compatible with the data type of your field. If you do not, you will get an exception raised. Passing NULL or array() when a string is expected will therefore raise an error.
Another thing to keep in mind is that the data you pass will also depend on whether your field is a single value, a multi-value field, or a field with multiple properties.
If your field is a single value (and thus the wrapper for the field is an instance of EntityValueWrapper) then you should assign it an empty value compatible with the data type in question. For instance the two following methods are equivalent:
$wrapper->title = '';
$wrapper->title->set('');

However the following three examples will raise an exception, because the data types are not compatible with the title field:
$wrapper->title->set();
$wrapper->title->set(NULL);
$wrapper->title->set(array());

If your field is a field with multiple properties (for instance a formatted text field, which defined both a value and format property) and thus an instance of EntityStructureWrapper, then array() or NULL will be the correct empty value. So you can do the following:
$wrapper->field_formatted_text = array();
$wrapper->field_formatted_text = NULL;

But in that case, passing an empty string would have raised an error. Note that you could have chosen to make the value property empty instead, in which case a string is the correct data type:
$wrapper->field_formatted_text->value = '';

Finally, if your field is a multi-value field (and thus your wrapper is an instance of EntityListWrapper) then array or NULL are the correct empty values, and the following three lines are equivalent:
$wrapper->field_example_multiple->set();
$wrapper->field_example_multiple = array();
$wrapper->field_example_multiple = NULL;

Note:
Calling the clear method on the wrappers is not equivalent to setting the field to an empty value. When the field is set to an empty value, it then calls EntityMetadataWrapper::updateParent on the field's parent wrapper. This ensures amongst other things that the setter callback defined by hook_entity_property_info is called. Calling clear does not do that.
Comment: Note that if the field is multiple *and* required, setting as `array()` or `NULL` can fail, because the field can't be empty. This is different from normal `$node` field assignment, where you can programmatically save an empty required field (it just won't save via Drupal's own UI). In this case, a workaround is `array(N)`, where N is the ID of a non-existent yet referenced entity. Note that it will save with that ID, so your data is arguably "broken" in a relational sense; but it shouldn't affect the theme layer if you're doing all the right things there (e.g. using Display Suite or Panels).
Comment: `$w->field_allowed_regions->set(array(null));` is the only option that worked for my multi-value taxonomy reference field.
Comment: In my case I have an entity reference field with a single value. The following worked for me: $wrapper->field_entity_reference->set(NULL);
Answer: Further to other answers and comments, if the field is multiple and required, as previously noted you cannot use
$wrapper->field_example_multiple->set()
$wrapper->field_example_multiple->set(NULL)
nor even $wrapper->field_example_multiple->set(array()),
but instead you can use the following if you want to clear the field of all its values:
$wrapper->field_example_multiple->set(array(NULL));

In fact, this works whether or not the multiple-value field is set to 'required', and so I'd recommend always using this, to ensure that your code is robust.
(Of course, if the field is 'required' then you perhaps shouldn't be completely clearing it anyway, but your code might be doing this as a preliminary step to deleting the whole entity or something similar, so there are times when it might just be legitimate.)
Comment: Be aware that using ` $wrapper->field_example_multiple->set(array(NULL))` will lead to having a NULL element in the data array. This method does not clear values but rather sets values array to a single `NULL` value.
Comment: Good point. I guess that returns us to my remark about not clearing a required value.  It is probably made impossible intentionally.
Comment: In point of fact, a required field must have at least one non-null value. If you want to reset a required multivalued field, just overwrite it with the new value. I.e.: `$product_display->field_product = array($product_id);`
Answer: It appears that the complexities listed in the other comments are only relevant to a required field.  If the field is not required, then this should be pretty simple:
$wrapper->field_foo = NULL;

You can use the wrapper to check the properties of the field:
$properties = $wrapper->getPropertyInfo();
$field_required = !empty($properties['field_foo']['required']);

Depending on the context, you can also just get the properties of the one field using:
$wrapper->getPropertyInfo('field_foo');

Answer: Another solution for this problem could be EntityMetadataWrapper::clear
$entity_wrapper->field->clear()
Comment: EntityMetadataWrapper::clear method is declared as "protected", so it can not be called from your code: only "public" methods can be accessed directly from outside of the object.


Hosting service for audio/music social website

Question: Im building a music social network on zend framework 2.You can just imagine it as it was the same as SoundCloud, users upload songs, other users play them, those are the basic actions on the site.
I know that the server that will host the page will need a lot of bandwidth and disk space to host my site ( i think its a good idea to store the mp3 files on the server that hosts the page. am i wrong? ).
My two options are:
Setting my own server at home (core i7) and hiring an internet service with 10Mbps upload, 30Mbps download. (Maximum upload rate where i live)
Hosting the page in a "conventional" server, and pay the month fee.
I came across with Hostgator, i think is a good option (and cheap).
http://www.hostgator.com/promo/special-offer
Is that service the service i need? I mean, i think is too good to be true, 4 dollars per month, all the bandwidth and all the disk space i want. What about the computing power for processing all the php that my site requires?
Can anyone give me some advice? (Im a newbie web developer)
Comment: Neither is a good idea. Host your MP3s on a file storage system like Amazon S3, for starters.
Answer: Hostgator would definitely be able to run your website (providing it has the support for your chosen language and database e.g. PHP and MySQL). Beware though that some shared hosts such as this may have performance issues or some bandwidth restrictions (I don't have experience with Hostgator specifically though).
For the audio content, you will want to use some sort of CDN (content delivery network) to achieve the speed you will need to stream the music. An example is Amazon Cloudfront. Using a CDN for content will also take some load off your web server.
It is hard to recommend specifics for your particular case without knowing how much traffic you will be expecting, size and quality of audio files etc.
EDIT: Regarding the comments below, I should have been clearer in saying that although these hosting solutions can have performance issues, the price is usually very low. You could always start out with a service like Hostgator and then move to something better like Amazon AWS or Microsoft Azure in the future when the site starts to take off a bit.
Comment: I really dont know how much traffic to expect. I guess at the beggining the traffic will be very very low, and later (if i have some luck and the website becomes successful) it will grow. my ambition is to have a really big site in the future, so the solution must have to be able to scale correctly for a large amount of traffic/users uploads and downloads.

I didnt decide the quality of the mp3 files, i think converting them to 128kbps is a good idea. So, that the average mp3 file size will be 4 or 5 MB.
Comment: This answer says "would definitely be able to run your website" and then mentions performance issues / bandwidth restrictions that make it clear that you can't say that for certain at all. Odd.
Comment: I forgot to mention that these hosting solutions can be good to start off with as they are low cost, but the performance issues will definitely not work for a large site so when the site takes off, you would probably want to look for more costly but better performance solutions. Updated the answer to reflect that.


Python datetime.strptime month specifier doesn't seem to work

Question: The month format specifier doesn't seem to work.
from datetime import datetime
endDate = datetime.strptime('10 3 2011', '%j %m %Y')
print endDate
2011-01-10 00:00:00
endDate = datetime.strptime('21 5 1987', '%j %m %Y')
print endDate
1987-01-21 00:00:00

Now, according to the manual the manual:

%m =  Month as a decimal number [01,12].

So, what am I missing, other than the hair I've pulled out trying to understand why my django __filter queries return nothing (the dates going in aren't valid!)? I've tried 03 and 05 to no avail.
Versions of things, platform, architecture et al:
$ python --version
Python 2.7
$ python3 --version
Python 3.1.2
$ uname -r
126.96.36.199-83.fc14.x86_64 (that's Linux/Fedora 14/64-bit).

Answer: You can't mix the %j with others format code like %m because if you look in the table that you linked %j is the Day of the year as a decimal number [001,366] so 10 correspondent to the 10 day of the year so it's 01 of January ...
So you have just to write :
>>> datetime.strptime('10 2011', '%j %Y')
datetime.datetime(2011, 1, 10, 0, 0)

Else if you you wanted to use 10 as the day of the mount you should do :
>>> datetime.strptime('10 3 2011', '%d %m %Y')
datetime.datetime(2011, 3, 10, 0, 0)

Comment: Thank you. I've been sat in front of my screen for faaaar to long I think!
Comment: @Ninefingers: glad i was able to help :)
Comment: So how would I extract the date from a string that uses this format: `4/14/11 14:23`?
Answer: Isn't %j the "day of year" parser, which may be forcing strptime to choose January 21, overriding the %m rule?
Answer: %j specifies a day of the year.  It's impossible for the 10th day of the year, January 10, to occur in March, so your month specification is being ignored.  Garbage In, Garbage Out.


pandas >= 0.18 - setting multi-index name on df from read_csv resulting in TypeError

Question: I've come across an error that has only started occurring after updating pandas to >= 0.18
I have a piece of code that opens a number of csv files and concatenates them into one multi-indexed dataframe. As part of the pd.concat step I set the index names using names=['Sweep', 'Index']. Prior to updating from 0.17 to 0.18 this worked fine, but since updating I'm getting the traceback shown below.
I've reduced the issue down to being able to reproduced by this (the example csv can be found here):
filename = './example.csv'
df = pd.read_csv(filename)
pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', 'Index'])

This results in the following traceback:
Traceback (most recent call last):
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\range.py", line 124, in _simple_new
    return RangeIndex(start, stop, step, name=name, **kwargs)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\range.py", line 73, in __new__
    start = _ensure_int(start, 'start')
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\range.py", line 59, in _ensure_int
    new_value = int(value)
TypeError: only length-1 arrays can be converted to Python scalars

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\core\base.py", line 63, in __repr__
    return str(self)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\core\base.py", line 42, in __str__
    return self.__unicode__()
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\core\frame.py", line 534, in __unicode__
    line_width=width, show_dimensions=show_dimensions)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\core\frame.py", line 1486, in to_string
    show_dimensions=show_dimensions)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\core\format.py", line 378, in __init__
    self._chk_truncate()
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\core\format.py", line 450, in _chk_truncate
    frame.iloc[-row_num:, :]))
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 834, in concat
    copy=copy)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 972, in __init__
    self.new_axes = self._get_new_axes()
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 1059, in _get_new_axes
    new_axes[self.axis] = self._get_concat_axis()
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 1111, in _get_concat_axis
    concat_axis = _concat_indexes(indexes)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\tools\merge.py", line 1129, in _concat_indexes
    return indexes[0].append(indexes[1:])
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\multi.py", line 1029, in append
    label = self.get_level_values(i)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\multi.py", line 666, in get_level_values
    tz=getattr(unique, 'tz', None))
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\range.py", line 126, in _simple_new
    return Index(start, stop, step, name=name, **kwargs)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\base.py", line 214, in __new__
    return Int64Index(data, copy=copy, dtype=dtype, name=name)
  File "C:\Users\User\Anaconda3\lib\site-packages\pandas\indexes\numeric.py", line 107, in __new__
    not issubclass(np.dtype(dtype).type, np.integer)):
TypeError: data type "Index" not understood

I've tried this on a couple different machines (thinking maybe it was something anaconda), but I get the issue everywhere.
If I just generate a dummy dataframe and do the same thing, this does not occur. I.e.:
df = pd.DataFrame({"Vals1": range(8), "Vals2": range(8)})
pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', 'Index'])

works fine.
I can set the name of the index of the dataframe from the imported csv without issue. I.e.:
filename = './example.csv'
df = pd.read_csv(filename)
df.index.names = ['Index']

works fine.
Oddly, if I just run the concat step without the names argument, and then set names in a separate line, this completes fine. However, as soon as I go to use that dataframe, I get the traceback shown above. I.e.:
filename = './example.csv'
df = pd.read_csv(filename)
new_df = pd.concat([df, ], keys=['Sweep1', ])
new_df.index.names = ['Sweep', 'Index']

Will run, until I go to view new_df. Even more oddly, is I can do things like new_df.head(), and that runs fine. But if I just type new_df in my terminal to just have it print out, I get the above traceback.
I'm at a loss as to what's going on here, so any help would be appreciated.
** Edit 1**
These are the first 5 rows from the file provided in the link above.
data_dict = {' LED': {0: -0.00030517578125,
  1: 0.0,
  2: -0.00030517578125,
  3: -0.00030517578125,
  4: -0.00030517578125,
  5: 0.0},
 ' Primary': {0: -2.74688720703125,
  1: -2.74566650390625,
  2: -2.74627685546875,
  3: -2.74810791015625,
  4: -2.74749755859375,
  5: -2.745361328125},
 ' Secondary': {0: -0.00152587890625,
  1: -0.00457763671875,
  2: -0.0048828125,
  3: -0.00335693359375,
  4: -0.0048828125,
  5: -0.00518798828125},
 'Time(ms)': {0: 0.0,
  1: 0.10000000000000001,
  2: 0.20000000000000001,
  3: 0.29999999999999999,
  4: 0.40000000000000002,
  5: 0.5}}

df = pd.DataFrame(data_dict)
df.to_csv('temp.csv', index=False)
df = pd.read_csv('temp.csv')
pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', 'Index'])

This does not produce an error, however.
Comment: what is example.csv?
Comment: It's a csv file generated from my data acquisition software. Basically it's four columns of numeric data with the first row being column headers. I've provided a link to a file that can be used to reproduce this issue (the issue generalizes across different files though)
Comment: right, but without a representative example of that data, how are we supposed to help you?
Comment: I don't understand - I'm providing a file that can be used.
Comment: embed a small subset of that data in your code to generate the dataframe. most people aren't too keen on downloading random files from anonymous people on the internet
Comment: It seems a small subset would not trigger the issue actually. The example `df = pd.DataFrame({"Vals1": range(8), "Vals2": range(8)})` works fine as poster noted, but `df = pd.DataFrame({"Vals1": range(61715), "Vals2": range(61715)})` will produce the error.
Comment: I was just going to comment that the length seems to be the issue. See Edit 1
Comment: Try this: `pd.options.display.max_rows = 10`, then `df = pd.DataFrame({"Vals1": range(11), "Vals2": range(11)})` and the concat thing :P. *60* is magic since it's the default value for `max_rows`. The error is in the display code which is trying to split larger dataframes for display, I think.
Comment: ha, interesting. It specifically doesn't like renaming the default index. If you reorder the levels, it still just doesn't like renaming what is now level 0 (was level 1), but renaming the Sweep index is fine
Answer: This is a very interesting problem.  I'm certain this points to a bug.
Evidence
Without downloading your data, just using your simple example that works:
df = pd.DataFrame({"Vals1": range(8), "Vals2": range(8)})
pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', 'Index'])

Yields something like what you want.  However, if you change the range(8) to range(61).  It blows up in a similar fashion.  Yes!, there seems to be something special about the number 61.  What's more is that if you instead did this:
df = pd.DataFrame({"Vals1": range(61), "Vals2": range(61)})
df1 = pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', 'Index'])

That works fine... until you try to print df1.  The error is coming when it is attempting to display the dataframe.
Work around.
I could not get a name in the second spot of the MultiIndex.names attribute and also be able to print/display it.
df = pd.DataFrame({"Vals1": range(61), "Vals2": range(61)})
df1= pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', ])

Demonstration
To prove it works on your data
filename = './example.csv'
df = pd.read_csv(filename)
pd.concat([df, ], keys=['Sweep1', ], names=['Sweep', ])

              Time(ms)   Primary   Secondary       LED
Sweep
Sweep1 0           0.0 -2.746887   -0.001526 -0.000305
       1           0.1 -2.745667   -0.004578  0.000000
       2           0.2 -2.746277   -0.004883 -0.000305
       3           0.3 -2.748108   -0.003357 -0.000305
       4           0.4 -2.747498   -0.004883 -0.000305
       5           0.5 -2.745361   -0.005188  0.000000
       6           0.6 -2.749634   -0.004578  0.000305
       7           0.7 -2.748413   -0.002441 -0.000305
       8           0.8 -2.746277   -0.004883 -0.000305
       9           0.9 -2.743530   -0.005188  0.000305
       10          1.0 -2.743835   -0.006104  0.000610
       11          1.1 -2.747192   -0.003052 -0.000610
       12          1.2 -2.746277   -0.003052 -0.000305
       13          1.3 -2.742310   -0.009460  0.000000
       14          1.4 -2.747192   -0.007324  0.000305
       15          1.5 -2.746887   -0.004272  0.000305
       16          1.6 -2.740479   -0.001526 -0.000305
       17          1.7 -2.745972   -0.004883 -0.000305
       18          1.8 -2.743530   -0.007629  0.000305
       19          1.9 -2.742310   -0.005188  0.000000
       20          2.0 -2.739258   -0.001526  0.000305
       21          2.1 -2.743835   -0.000610  0.000000
       22          2.2 -2.745056   -0.004883 -0.000305
       23          2.3 -2.742615   -0.006714 -0.000305
       24          2.4 -2.739868   -0.004272  0.000610
       25          2.5 -2.738342   -0.006104  0.000000
       26          2.6 -2.740784   -0.003662 -0.000916
       27          2.7 -2.738647   -0.000610  0.000305
       28          2.8 -2.737732    0.000610  0.000000
       29          2.9 -2.739563   -0.004578  0.000000
...                ...       ...         ...       ...
       99970    9997.0 -2.576599   -0.000305 -0.000610
       99971    9997.1 -2.577515   -0.004578 -0.000305
       99972    9997.2 -2.577209   -0.003967  0.000000
       99973    9997.3 -2.578430   -0.003052  0.000000
       99974    9997.4 -2.580566   -0.004272  0.000305
       99975    9997.5 -2.575684   -0.003357 -0.000305
       99976    9997.6 -2.574463   -0.000916 -0.000305
       99977    9997.7 -2.579651   -0.002747  0.000000
       99978    9997.8 -2.576294   -0.010376 -0.000305
       99979    9997.9 -2.578125   -0.005798 -0.000916
       99980    9998.0 -2.576294   -0.001831  0.000000
       99981    9998.1 -2.577820    0.000000  0.000916
       99982    9998.2 -2.581482   -0.007019  0.000000
       99983    9998.3 -2.575073   -0.005493  0.000000
       99984    9998.4 -2.578125   -0.002136 -0.000305
       99985    9998.5 -2.577515    0.003967 -0.000305
       99986    9998.6 -2.575378   -0.005188  0.000000
       99987    9998.7 -2.577515   -0.007935  0.000610
       99988    9998.8 -2.578430   -0.003967 -0.000305
       99989    9998.9 -2.579346   -0.001221  0.000000
       99990    9999.0 -2.577209    0.001221  0.000000
       99991    9999.1 -2.578125   -0.005798  0.000000
       99992    9999.2 -2.577515   -0.005493  0.000000
       99993    9999.3 -2.573853   -0.002747 -0.000305
       99994    9999.4 -2.575378    0.002441 -0.000305
       99995    9999.5 -2.577820   -0.000305  0.000305
       99996    9999.6 -2.575989   -0.006104  0.000000
       99997    9999.7 -2.576294   -0.007019  0.000305
       99998    9999.8 -2.576599   -0.003662  0.000305
       99999    9999.9 -2.573853   -0.000916 -0.000305

[100000 rows x 4 columns]

Finally
If you really need that 'Index' name in there, you can leave it, just don't try to display it.
Also Important
Paul H is correct in that people are not keen on downloading random files from the internet.  I took a risk.  However, this bug only manifests if the sample data exceeded 61 records/rows.
Comment: Yes, I had come to find that I could rename the level 0 index, but not the level 1 index and still be able to display it. I assume this is a pandas bug then, and should be opened over on their issue tracker?


Results of autocomplete field in form depending on another field - Drupal 7

Question: I have an autocomplete field in a Drupal form made with Form API. It works great, suggesting the data from a db_query.
However, I would like to alter this db_query depending on the selection made in another field of the form (a select field).
But I don't know how to do it, as I would need to change the db_query parameters on-the-fly (Javascript) but they are hard-coded in PHP!!
Is what I want possible at all? (it can be done in pure HTML + Javascript + AJAX PHP but I cannot imagine how to do it in Drupal).
Answer: Keep a wild card in your autcomplete menu path something like
function mymodulename_name_menu() {
  $items['example/autocomplete/%/%'] = array(
    'page callback' => '_mymodulename_autocomplete_callback',
    'page arguments' => array(2, 3),
    'access callback' => TRUE,
    'type' => MENU_CALLBACK,
  );
}

In your form
function mymodulename_my_form($form, &$form_state) {
// Your select field.
  $form['select_field'] = array(
    '#type' => 'select',
    '#options' => array(
      'option1' => 'Option1',
      'option2' => 'Option2',
    ),
    '#ajax' => array(
      'callback' => '_mymodulename_ajaxcallback',
      'wrapper' => 'replace_autocomplete_field',
    ),
  );
  // A container field
  $form['autocomplete_field_container'] = array(
    '#type' => 'container',
    '#tree' => TRUE,
    '#prefix' => '
',
    '#suffix' => '
',
  );
  $query_parameter = isset($form_state['values']['select_field']) ? $form_state['values']['select_field'] : 'default_value_to_start_with';
  // Your autocomplete field.
  $form['autocomplete_field_container']['autocomplete_field'] = array(
    '#title' => t('My autocomplete field.'),
    '#type' => 'textfield',
    '#autocomplete_path' => "example/autocomplete/$query_parameter",
  );
}

You Automcomplete menu callback
function _mymodulename_autocomplete_callback($query_parameter, $string) {
  // Use $query parameter to decide autocomplete field behavior.
}

Basically I am trying to update the autocomplete field by attaching '#ajax' behaviour to select field, and while updating the autocomplete field we will change the query parameter which db_query will use.
Comment: Thank you very much. As I see, your response is similar to that in this tutorial: http://complexdan.com/passing-custom-arguments-drupal-7-autocomplete/ . I will try your specific example in any case, I found the tutorial by looking for strings in your example, but I will try if there is any conceptual difference. Thank you again.
Comment: I am glad it helped you :)
Answer: I have found this reference: How to dynamically reconfigure Drupal's jQuery-based autocomplete at runtime?
However, I see it somehow complicated. I prefer this approach of mine that I am using right now (and working for an exposed filter):
I create a hook to add a Javascript Ajax call to the select list. In that Ajax call, I use a cookie associated to the user, and the value of the selected value in the select list, and insert both in a database table.
Now when I make the Autocomplete call, in the db_query I use the user cookie and read the value inserted, so I can restrict the suggestions based on that value.
The only problem is that the call is asynchronous, but if the autocomplete function cannot find the cookie value in the table, will yield a warning in the suggestion list, indicating to type more letters or type again, giving time to the first Ajax call to complete.


Singular values and the spectrum of an operator

Question: Are the singular values of a bounded linear operator on an Hilbert space in the spectrum of a that operator in general?
If not, is there a subclass of operators for which this hold?
Comment: Consider the map $\Bbb {C}\to\Bbb {C},x \mapsto -x $. What is the spectrum, what are the singular values?
Answer: Singular values are the same for an operator $A$ and any unimodular multiple $\alpha A$, with $\alpha\in\mathbb C$. On the other hand, multiplication by $\alpha$ rotates the spectrum of $A$. This leads to an exaemple, given by PhoemueX, of $x\mapsto -x$, with singular value $1$ and spectrum $\{-1\}$.

is there a subclass of operators for which this hold?

For positive self-adjoint operators the singular values are in the spectrum, since they are the eigenvalues of $\sqrt{A^*A}=A$.


Relate Content to a Group feature when creating a node outside a group

Question: In my drupal site, I'm using the Group module.
My problem:

I created a regular Drupal node type, let's call it userpost
I created a group type called "usergroup"
The content type "userpost" is also installed as a group node of group type "usergroup"

Is there a way to put a reference field or something similar into my "regular" node form for my content type called userpost? So that it's possible to (optional) relate a node during creation, to one (or more) group(s) at the same time?
In the admin interface of a group it's possible to relate existing content to a group. This kind of reference field I would like to place into my node creation form :)
Thanks for any hint in advance.
Best regards
Phil
P.S. I opened for this problem an issue at Drupal.org too.
Comment: This post is quite similar to my problem: https://drupal.stackexchange.com/questions/265785/select-groups-to-relate-the-node-to-directly-in-the-node-creation-form
Comment: I haven't used group module but what you just described is easily achievable by organic groups module https://www.drupal.org/project/og. You can set the content type to be either group (parent) or group content (child) or both, then when you create child nodes you can link them to as many groups as necessary with a supplied entity reference field. NB I have done this in Drupal 7 but I would imagine similar settings/behavior on D8. If you're not tied to using group module I'd definitely recommend og.
Answer: Well, if I understand your problem correctly, you could just add an entity_reference field to your node type 'userpost' via admin toolbar. Go to Structure->Content types->userpost->Manage fields->Add field->Choose Reference (other)->Select Group and set cardinality to whatever you want.
This would allow you to add group references when you create a new node of content type userpost
Update
What I described adds a reference from a node to one or multiple group(s) via a new field, it does NOT add this node as a group-content to your group(s). It depends what you are trying to achieve? If you want to add your node as a group-content and you want to do this via the node creation form, you need one more extra step. You can implement a hook_ENTITY_TYPE_insert(). From there get the group(s) that you entered to your node field (from $node), $groups = $node->get('field_groups') and add the $node as group_content to each group, similar as done in the answer you linked.


Daterangepicker select date by weekly

Question: Is there a way, to select weekly on daterangepicker?

E.g if I select December 5 it will select the whole week.

I tried to search for the answer and i found nothing

if anyone can help much appreciated
Answer: You may create additional field for selection day of week and then you can use value of the field in the isInvalidDate function. You also can point common parent element across parentEl property and from this get access for the field

$(document).ready(function () {
  $('#daterangepicker').daterangepicker({
    parentEl: '.row',
    isInvalidDate: function(date) {
      return parseInt($(this.container).parent().find('select').val()) !== new Date(date).getDay();
    }
  });
});

<link rel="stylesheet" type="text/css" href="https://cdn.jsdelivr.net/npm/daterangepicker/daterangepicker.css" />

    <input id="daterangepicker" type="text" />
    <select>
          <option value="1">Mo</option>
          <option value="2">Tu</option>
          <option value="3">We</option>
          <option value="4">Th</option>
          <option value="5">Fr</option>
          <option value="6">Sa</option>
          <option value="0">Su</option>
      </select>

Daterangepicker select date by weekly


Find matrix that satisfies matrix equation

Question: I am trying to find a matrix $A$ that satisfies the following type of equation:
$$A^\dagger A = f(B,C,...)$$
where the right-hand side is a function of known matrices $B,C,...$
Is there a general approach for this?
EDIT: The matrices $B,C,\dots$ are fixed and independent, and the right-hand side does not depend on $A$.
Comment: This is at best vague. Are $B, C, ...$ fixed or do they depend on $A$? Ditto for $f$.
Comment: $\dagger$ is Hermitian transpose
Comment: Sorry but it is still very unclear what you mean. Are you fixing a function $f$ of matrix arguments and want to know what matrices can appear in the LHS for arbitrary $B$, $C,$ ... of some sort, or what else? Maybe you can try to spell out an example.
Comment: For example, find $A$ that satisfies $A^\dagger A = B+CD^\dagger+C^\dagger D$
Comment: Matrices are square, so say $n\times n$
Answer: We have the following matrix equation in $\mathrm X \in \mathbb C^{n \times n}$
$$\mathrm X^* \mathrm X = \mathrm A$$
where $\mathrm A \in \mathbb C^{n \times n}$ is given. Since $\mathrm X^* \mathrm X$ is Hermitian and positive semidefinite, $\mathrm A$ must also be Hermitian and positive semidefinite. Hence, $\mathrm A$ is diagonalizable. Let its eigendecomposition be
$$\mathrm A = \mathrm V \Lambda \mathrm V^*$$
Thus, one solution to the matrix equation is
$$\bar{\mathrm X} := \Lambda^{\frac 12} \mathrm V^*$$


Core plot osx not plotting

Question: I'm writing a simple cocoa app for a school project which requires to graph some data from a serial port and plot it into a simple graph. I have chosen for Core- Plot as framework to generate and display the dataplots, however I can't get it to work.
I managed to get the axis drawn and the program runs fine, but there are no data lines appearing. I use the following code:
<code>#import "Controller.h"
#import <CorePlot/CorePlot.h>

@implementation Controller

-(void)dealloc
{
    [plotData release];
    [graph release];
    [super dealloc];
}

-(void)awakeFromNib
{
    [super awakeFromNib];

// Create graph from theme
graph = [(CPTXYGraph *)[CPTXYGraph alloc] initWithFrame:CGRectZero];
CPTTheme *theme = [CPTTheme themeNamed:kCPTDarkGradientTheme];
[graph applyTheme:theme];
hostView.hostedGraph = graph;

// Setup scatter plot space
CPTXYPlotSpace *plotSpace = (CPTXYPlotSpace *)graph.defaultPlotSpace;
plotSpace.xRange = [CPTPlotRange plotRangeWithLocation:CPTDecimalFromDouble(-1) length:CPTDecimalFromDouble(10)];
plotSpace.yRange = [CPTPlotRange plotRangeWithLocation:CPTDecimalFromDouble(-1) length:CPTDecimalFromDouble(10.0)];

// Axes
// Label x axis with a fixed interval policy
CPTXYAxisSet *axisSet = (CPTXYAxisSet *)graph.axisSet;
CPTXYAxis *x          = axisSet.xAxis;
x.majorIntervalLength         = CPTDecimalFromString(@"1");
x.orthogonalCoordinateDecimal = CPTDecimalFromString(@"0.0");
x.minorTicksPerInterval       = 2;

x.title         = @"Tijd (s)";
x.titleOffset   = 30.0;
x.titleLocation = CPTDecimalFromString(@"8.5");

// Label y with an automatic label policy.
CPTXYAxis *y = axisSet.yAxis;
y.labelingPolicy              = CPTAxisLabelingPolicyAutomatic;
y.orthogonalCoordinateDecimal = CPTDecimalFromString(@"0.0");
y.minorTicksPerInterval       = 2;
y.preferredNumberOfMajorTicks = 8;
y.labelOffset                 = 10.0;

y.title         = @"Kracht (N)";
y.titleOffset   = 30.0;
y.titleLocation = CPTDecimalFromString(@"8");

// Create a plot that uses the data source method
CPTScatterPlot *dataSourceLinePlot = [[[CPTScatterPlot alloc] init] autorelease];
dataSourceLinePlot.identifier = @"Date Plot";

CPTMutableLineStyle *lineStyle = [[dataSourceLinePlot.dataLineStyle mutableCopy] autorelease];
lineStyle.lineWidth              = 3.f;
lineStyle.lineColor              = [CPTColor greenColor];
dataSourceLinePlot.dataLineStyle = lineStyle;

dataSourceLinePlot.dataSource = self;
[graph addPlot:dataSourceLinePlot];

// Add some data
NSMutableArray *contentArray = [NSMutableArray array];
for ( NSUInteger i = 0; i < 10; i++ ) {
    id x = [NSDecimalNumber numberWithDouble:i];
    id y = [NSDecimalNumber numberWithDouble:1.2];
    [contentArray addObject:[NSMutableDictionary dictionaryWithObjectsAndKeys:x, @"x", y, @"y", nil]];
    NSLog(@"waarde %@", contentArray);
}
NSLog(@"plotting");
plotData = [contentArray retain];
}

-(NSUInteger)numberOfRecordsForPlot:(CPTPlot *)plot
{
return plotData.count;
}

@end
</code>
The code is based of one of the examples.
The NSLog(@"waarde %@", contentArray); returns a good looking array, containing correct datapoints.
Comment: have you implemented the numberForPlot:field:recordIndex method?
Answer: Try
<code>hostView.frame = CGRectMake(0,0,self.view.frame.size.width,self.view.frame.size.height);
[self.view addSubview:hostView];
</code>
Is that fact that it is not appearing due to the fact you haven't added it as a view?
Comment: Hello Mark, Thanks for the response, however i get a error message when i try to compile the code. "Property 'view' not found on object of type 'Controller *' ". Any suggestions?
Comment: Ahh just seen Andrew's comments, check that you have implemented all the datasource methods such as numberForPlot:field:recordIndex and numberOfRecordsForPlot:plot
Comment: If you are able to draw the axis, the problem isn't getting it onto the screen it is probably the datasource methods that you have missed
Comment: Thanks, i actually implemented those but i mistyped "(fieldEnum == CPTScatterPlotFieldX ? @"x" : @"y")"

Many thanks for all the help.

Core plot osx not plotting


Obj-c, What's the quickest way to execute many SQLite insert / update queries, without core data?

Question: I'm committed along the route of using SQLite without core data.
I need to speed up a function which performs some database transactions after querying the database. I've created a dictionary for the rows with all the values I'll need.
I need to do this to avoid the database locking.
At the moment I'm calling my add record to database function, which opens and closes the database each time.
Obviously this is where the process is slow.
I was thinking that it's common for apps to be embedded with a database setup script, so it must be possible to run a batch of queries.
So I'm thinking if I can build up a string with all my queries I could just execute that.
But I'm not 100% this is the best approach or how to execute batch queries.
Can anyone advise me how to proceed?
Comment: Either by not using a database at all and trying a different approach (like putting your data (compressed, fe with snappy) in memory and writing to the disk it when your app is closed) or by reducing the number of calls/cells (important step: always rethink your infrastructure and be sure only that only the ones that are absolutely necessary will be saved). It will be a hard challenge to optimize the speed "behind" the SQLite API.
Answer: For starters .. check out these links:
how-do-i-improve-the-performance-of-sqlite
ios-coredata-batch-insert  (Yes I know that you said no core data - but it is worth a read)
fast-bulk-inserts-into-sqlite (Looks similar in content to the first link)
Comment: That 2nd was the most helpful :)
Answer: I was about to do the same - using plain SQLite instead of CoreData - but changed my mind later. In that process if found this link useful: Improve INSERT-per-second performance of SQLite? . Beyond the obvious (transaction,prepared statement,..) it uses some SQLite specific performance tweaks.


How to explain integrals and derivatives to a 10 years old kid?

Question: I have a sister that is interested in learning "what I do". I'm a 17 years old math loving person, but I don't know how to explain integrals and derivatives with some type of analogies.
I just want to explain it well so that in the future she could remember what I say to her.
Comment: I have a 7 year old brother myself, and he always asks me this question ; but I'm still at the stage of explaining to him how to compute exponents... (which is quite brilliant for his age, though!) Keep up the good work!
Comment: Even a bright ten year old may not have encountered the word slope, or function, or much Cartesian graphing!
Comment: What have you tried? (Just kidding!) Didn't the explanations: *"The derivative realtes to the slope of a function."* and "*The integral is related to the area under the function."* help?
Comment: @countinghaus Right. What about explaining slope with the effort it takes to climb up a hill?
Answer: This is a question for a long car journey - both speed and distance travelled are available, and the relationship between the two can be explored and sampled, and later plotted and examined. And the questions as to why both speed and distance are important can be asked etc
Answer: Explain derivatives using the speedometer ! Ask her how one could find the speed of something , and if she goes saying the average speed , ask her how one could calculate the instantaneous speed. And boom , there comes your derivatives.
Answer: Isn't there a way to describe with drawings that the derivative is the limit of the slopes of the secants, that the integral is the area under a curve, and that the derivative of the area is the height of the function at the point at which you compute derivative? Sure, she might not know what "slope" means, but certainly you can explain that to her if she has the patience ; she's not too far away from it mathematically.
If you make "clear enough" drawings, I think geometry is a good enough analogy to explain what you do. If you put in contexts where derivatives apply (physics/chemistry/economics), I think your little sister will just look at too many pictures at the same time and get lost in what you're saying. Try to be patient and stick to the "closest-to-the-real-picture portrait" of derivatives and integrals.
Oh, and one more thing ; if she isn't interested in what you do, don't lose your time. Only do it if she has the courage to listen to you, remember that mathematics are pure pain to the closed ears.
Hope that helps,
Answer: I quote parts of this Reddit answer.

The easiest way to understand a little calculus is to sit in the middle seat and look at the speedometer in the car.  What speed does it say?  Maybe it says 31 miles per hour.  This means that, if you keep traveling at this speed, you'll go 31 miles in an hour.  Any kid can understand that (even if the kid doesn't really know how far a mile is).  But then your dad slows down and stops at a red light.  The speed is 0 miles per hour now.  Did you actually go 31 miles in an hour?  No; 31 miles per hour was your speed only at that instant in time.  Now the speed is different.  The idea that it even makes sense to have a speed at an instant in time is... calculus!  You calculate speed by seeing how far you go and dividing by how long it took you to get there, but that only gives you average speed.  For the speed right now, you have to see how far you go in a very, very, very tiny amount of time.  You only go a very, very, very tiny distance.  And you divide by that very, very, very tiny amount of time to get a speed in numbers that you understand.  Calculus is when you make that amount of time tinier and tinier and tinier, and that makes the distance tinier and tinier and tinier too, so that, at that moment, the tiny distance divided by the tiny time is 31 miles per hour, but a second later it might be 30 mph or 32 mph or something else.
You generally use calculus to talk about how fast things change -- in the case of the car, it's how fast your position changes, but lots of things can change.  How fast something is changing right now is called the derivative.  Sometimes you know how the rate of change for something is related to other things.  For example, if you have a weight on a spring, you can write how fast the speed of the weight is changing based on its position on the spring, and you can write an equation called a differential equation.
[...]
You can also use calculus to talk about how lots of little things can add up to a big thing.  For example, let's say you have an object, and you want to know how much it weighs.  You can break it up into tiny little pieces, figure out the density for each piece, figure out how much each little piece weighs, and add them all together.  That's calculus!  (Or you can just put it on a scale -- that's physics.)
The calculus of how fast things change is called differential calculus, and the calculus of adding up lots of little things is called integral calculus.  In differential calculus, you take a tiny little number and divide by another tiny little number to get a regular-sized number.  In integral calculus, you add together a very, very, very large number of tiny little numbers to get a regular-sized number.
Answer: I remember the explanation I was always given..
"Do you know how to find the area of a circle"
"Yes"
"Well, do you know how to find an area of a random curved figure?"
"No"
"That is what we use calculus for".


R, plyr, with a complex function

Question: I have the following dataset (CEU):
group  x      y
1     -23     100
1     -0.90   69.62
1     -0.90   72.03
2     -23     100
2      0.69   48.01
2      0.69   45.63

For each value of group, I want to apply functions noted below to each subset of x and y values. I would then like to combine all of the results and write them in a table to export.
I am unsure how exactly to apply the plyr function to do this...if that is indeed the right course of action.
x<-c(-23.0000,-0.9031,-0.9031)
y<-c(100,85.72,86.65)

par<-c(16.88,100.28,-.75,4.129)

dcrit<-function(d) {
    sumsq<-0
    for (i in 1:length(x)){
      sumsq<-sumsq+ (y[i]-(par[1]+(par[2]-par[1])/(1+10^((x[i]-par[3])*d))))^2
    }
    sumsq
}

S<-function(par) {
    a<-par[1]
    b<-par[2]
    c<-par[3]
    d<-par[4]
    sumsq<-0
    for (i in 1:length(x)){
      sumsq<-sumsq+ (y[i]-(a+(b-a)/(1+10^((x[i]-c)*d))))^2
    }
    sumsq
}
optim(par,S)

CEU <- read.csv(file="C:/files/CEU.csv",head=TRUE,sep=",")
CEU

data <- ddply(CEU,.(group),function(xy)
{
par[1]<-min(y)
par[2]<-100
par[3]<-x[[which.min(abs(y-50))]]
par[4]<-optimize(dcrit,interval=c(-100,100))$minimum

o<-optim(par,S)
par<-o$par

a<-par[1];
b<-par[2];
c<-par[3];
d<-par[4];

k<-(b-a)/(20-a)-1
if (k>0) ec20<-c+1/d*log10(k) else ec20<-NA
ec20

z<-(b-a)/(50-a)-1
 if (z>0) ec50<-c+1/d*log10(z) else ec50<-NA
ec50

j<-(b-a)/(80-a)-1
if (j>0) ec80<-c+1/d*log10(j) else ec80<-NA
ec80

data.frame(ec20, ec50, ec80)

})

data

The code runs without errors but only on the original x and y values set by:
 x<-c(-23.0000,-0.9031,-0.9031)
 y<-c(100,85.72,86.65)

The x and y values in dataset CEU are not used by ddply. They do not replace the original x and y in an iterative fashion as they do with the group values.  data has the appropriate number of groups and the ec20/ec50/ec80 values are accurate but only for the original x and y.
> data
   group       ec20       ec50       ec80
1      1 -0.3652977 -0.6843279 -0.8530892
2      2 -0.3652977 -0.6843279 -0.8530892
3      3 -0.3652977 -0.6843279 -0.8530892
4      4 -0.3652977 -0.6843279 -0.8530892
5      5 -0.3652977 -0.6843279 -0.8530892

Comment: Optimize takes a first argument of "f" (a function) and a second argument of "interval" (a range). But you seem to be sending it an undefined function, `dcrit`, and then doing something with the result, but  since "S" only appears once in your code.
Comment: Edited the original post with the complete code. Thanks!
Answer: It looks like you have it right, you just need to produce the output.
I'm guessing this is where your outputs are?
k<-(b-a)/(20-a)-1
if (k>0) ec20<-c+1/d*log10(k) else ec20<-NA
ec20

z<-(b-a)/(50-a)-1
 if (z>0) ec50<-c+1/d*log10(z) else ec50<-NA
ec50

j<-(b-a)/(80-a)-1
if (j>0) ec80<-c+1/d*log10(j) else ec80<-NA
ec80

Put them into a data.frame at the end of the function:
    ...
    data.frame(ec20, ec50, ec80)
}

Now you'll get a data.frame with all of them, with three columns for ec20, ec50 and ec80

To your problem with optim: I think the problem lies in
par[3]<-x[which.min(abs(y-50))]

Single [ in R is not regular subscript -- it gets a slice -- in this case of the data.frame columns. That line is turning par from a numeric vector into a list. Add more brackets:
par[3]<-x[[which.min(abs(y-50))]]

Comment: He's being brave. Ignoring the inconsequential details like defining functions and testing. Cutting to the chase. Let's see... if I give his answer a plus and your comment a plus and a -1 to the (incomplete) questioner... how much am I out?
Comment: If this is correct, you are truly a better mind reader than me! ;)
Comment: Thanks for the comments, I realize it was unclear as to what I was trying to do with the optim function initially. My concern isn't whether I am using the optim/dcrit functions properly, I know those work. I would just like to do it an iterative fashion using ddply. Alas, I am getting the following error Error in optim(par, S) : (list) object cannot be coerced to type 'double'
Comment: Owen, adding the brackets, creates the following warnings:In par[3] <- x[[which.min(abs(y - 50))]] :
  number of items to replace is not a multiple of replacement length
Comment: @Sash OK hm. What are you trying to do with that line? `x` is a  chunk of `data.frame` and you are selecting one column -- is this what you want to do? Or are you trying to select one row?
Comment: @Owen thanks for your help and patience! I have changed the code and added comments to reflect where I am at now. Thanks!
Comment: @Sash If you want to get the actual columns of your data from `ddply` you should do something like `ddply(CEU, .(group), function(df) with(df, { ... body of code ... }). Then you'll be able to refer to the columns as `x` and `y`.

R, plyr, with a complex function


Making a basis out of the eigenvalues of a normal endomorphism without using the Spectral Theorem

Question: Consider the following question:

Define the adjoint $\alpha^*$ of an endomorphism $\alpha$ of a complex
inner-product space $V$. Show that if $W$ is a subspace of $V$, then
$\alpha(W) \subseteq W$ if and only if $\alpha^*\left(W^{\perp}\right)
> \subseteq W^{\perp}$.
An endomorphism of a complex inner-product space is said to be normal
if it commutes with its adjoint. Prove the following facts about a
normal endomorphism $\alpha$ of a finite-dimensional space $V$.
(i) $\alpha$ and $\alpha^*$ have the same kernel.
(ii) $\alpha$ and $\alpha^*$ have the same eigenvectors, with complex
conjugate eigenvalues.
(iii) If $E_\lambda=\{x \in V: \alpha(x)=\lambda x\}$, then
$\alpha\left(E_\lambda^{\perp}\right) \subseteq E_\lambda^{\perp}$.
(iv) There is an orthonormal basis of $V$ consisting of eigenvectors
of $\alpha$.
Deduce that an endomorphism $\alpha$ is normal if and only if it can
be written as a product $\beta \gamma$, where $\beta$ is Hermitian,
$\gamma$ is unitary and $\beta$ and $\gamma$ commute with each other.
[Hint: Given $\alpha$, define $\beta$ and $\gamma$ in terms of their
effect on the basis constructed in (iv).]

I have already seen a proof of (iv) with the spectral theorem, however, in this course the theorem was not lectured. (I checked similar questions here but they all seem to be using it).
Moreover, I can't seem to think of another way to quickly get an answer as this question is not meant to be too long since the whole thing should take about 25 minutes. I also get the sense that this construction must be explicit because of the very last part.
Now I should say that this is from an exam that was 23 years ago so there is a chance the syllabus has changed and that I am asking for something that is unreasonable.
If anybody sees a way to do (iv) without the spectral theorem I would greatly appreciate it if they could share.
Note: The only version of the Spectral Theorem that is allowed in the current version of the course is that if an ednomorphism is self adjoint then there is an orthonormal basis of eigenvectors of that endomorphism. I am sure that is not the full version of the theorem which seems to be used by most answers associated with this question on the website.
Remark: I would only like to be able to answer (iv) of the question.
Comment: I have now typed the question for the convenience of the forum. How can I address the focus issue? I have done research as to how to approach it, and have found no similar questions that have been asked here. What would be the appropriate next step?
Comment: No worries. I am glad I was able to clarify my intentions with the question.
Comment: Please do not rely on pictures of text.
Comment: This is from an exam. What would you like me to do?
Comment: May I ask why is this being down voted?
Comment: It's probably because you didn't type up the question. There's also the fact that it needs focus.
Comment: The focus issue is that there are multiple parts to the question. Ask one at a time, please.
Comment: I did ask for only one. I asked only for (iv). I will add a remark in the question clarifying this.
Comment: Oh, I see. I'm sorry.
Answer: Part (iii) tells us that $\alpha(E_\lambda^\perp) \subseteq E_\lambda^\perp$, but it is also true that $\alpha^*(E_\lambda^\perp) \subseteq E_\lambda^\perp$. One way to see this is that by (ii), a different definition of $E_\lambda$ is $\{x \in V : \alpha^*(x) = \overline \lambda x\}$, so we can apply (iii) from $\alpha^*$'s point of view instead of $\alpha$'s. Therefore  $\alpha$ and $\alpha^*$, restricted to $E_\lambda^\perp$, are also endomorphisms of $E_\lambda^\perp$.
This lets us inductively solve the problem:

Pick an eigenvalue $\lambda$ of $\alpha$ (and $\alpha^*$);
Find an arbitrary orthonormal basis of $E_\lambda$.
If $E_\lambda = V$ we are done. (In particular, this must always happen if $V$ is $1$-dimensional, proving our base case.)
Otherwise, by strong induction on the dimension of $V$, we can find an orthonormal basis of eigenvectors of $\alpha$ restricted to $E_\lambda^\perp$; together with the orthonormal basis found in step 2, this gives us an orthonormal basis of all of $V$ consisting of eigenvectors of $\alpha$.
Comment: Hi, thank you for the answer. Just to make sure I am understanding the inductive step correctly. In 4 we say that we can split $V$ into $E_\lambda$ and its orthogonal complement. If the complement is empty we are done, else if it is not, then its dimension must be between $1$ and $\dim V -1$ which are all covered by the strong induction. For the rest, that is $E_\lambda$ we know that orthogonal basis exists (of eigen values). So combining the two gives an orthogonal eigen basis for the whole of $V$. Am I interpreting this correctly?
Comment: That's correct. (For $E_\lambda$, we know that an orthonormal basis exists because any subspace of $V$ has an orthonormal basis - we don't have any further constraints because $E_\lambda$ already consists only of eigenvectors.)
Comment: Also, for the last part, does the inductive construction allow us to answer it "deductively"? Part (iv) clearly tells us that it is diagonalizable. So we can think of how the matrix acts on each eigen space. I would think that the unitary requirement is some sort of rotation for the eigen basis where as the other matrix is to introduce scaling in each direction, however, I don't know why it would be hermitian. For the backwards argument, I can verify it algebraically
Comment: Think about the $1$-dimensional case first. Why can every complex number be written as the product of a "unitary complex number" and a "Hermitian complex number"? Then generalize by dealing with each eigenvector in the basis separately.


Get the value from Excel Sheet as Date in PHP

Question: I use PhpExcelReader--
    include 'excel_reader.php';     // include the class
// creates an object instance of the class, and read the excel file data
$excel = new PhpExcelReader;

Data Read and Function Call--
$excel->read('test.xls');
sheetData($excel->sheets[0]);

Function Code--
function sheetData($sheet)
{
    while($x <= $sheet['numRows'])
    {
        if(@$sheet['cells'][$x][1])
        {
            while($y <= $sheet['numCols'])
            {
                $cell = isset($sheet['cells'][$x][$y]) ? $sheet['cells'][$x][$y] : '';
                echo $cell = @date($cell)."
";
            }
         }
     }
}

It shows only numbers like 36400
And after then i tried
echo $cell = @date("Y-m-d",$cell)."
";

But it shows the default value like 1970-01-01
But my data 2004-05-12
Answer: The Date field in Excel is "Number of days since Jan 0 1900", where as the time in PHP is "Number of Seconds since Jan 01 1970 00:00". Conversion should be pretty easy from there.
Answer: Use this function detailed here https://phpexcel.codeplex.com/discussions/219301
$PHPDate = PHPExcel_Shared_Date::ExcelToPHP($cell);
echo date("Y-m-d", $PHPDate);


access the view but not the node

Question: I have a view that displays a list of items. How to make anonymous user see this list of items but when accessing the node, it has to be logged? Any idea?
I use the latest version of drupal
Answer: You can disable node_access check for the view (Edit the View → Advanced → Query settings → Disable SQL Rewriting).
Answer: you can also create a page and embed your view inside it. in this way users can see your view. also create a node page and check this condition
if($logged_in)
{
    //your code
 }

Comment: Hi, Mohammad I'm finally using what you say. But how to make if the user is authenticated not show me the rest of the node and show me another page?
Comment: Hi, I could not understand you, if the user is authenticated then your page in the bracket will be displayed. now this is my question: what is `the rest of the node` and what is `another page`? please clarify your question then I`ll be able to help you.
Answer: in order to create a page for your view, I guess your view has a page display, set a path for it, say /test . now create a .php file and name it page--test.tpl.php and embed your view using this instruction :
views_embed_view('NAME_OF_YOUR_VIEW','page');

clear the cache and go to /test you will see your view.
and about your node page, there are couple of things you need to do, first in your template file add this function
function YourThemeName_preprocess_page(&$variables)
{

    if (isset($variables['node']))
    {
         if($variables['node']->type == "YourContentTypeName")
         {
                $suggest = "page__node__test";
                $variables['theme_hook_suggestions'][] = $suggest;
         }

    }
}

now create a .php file and rename it to page--node--test.tpl.php, this will be your node page and you can set authentication condition for it. you can even go further and check this authentication condition in your preprocess function, I mean create two different $suggests in preprocess function :
function YourThemeName_preprocess_page(&$variables)
{

    if (isset($variables['node']))
    {
         if($variables['node']->type == "YourContentTypeName")
         {
                if($logged_in)
                {
                    $suggest = "page__node__test_authenticated";
                    $variables['theme_hook_suggestions'][] = $suggest;
                }
                else
                {
                        $suggest ="page__node__test_notAuthenticated";
                        $variables['theme_hook_suggestions'][] = $suggest;
                }

         }

    }
}

and now create two node pages in the same way I said, one for authenticated and one for non-Authenticated users. you can also get and set fields of your content type fields in preprocess function and display them in node page. don't forget to clear the chache

access the view but not the node


In a normed space, the sum of a Closed Operator and a Bounded Operator is a Closed Operator.

Question: The book "Introductory Functional Analysis with Applications" (Kreyszig) presents the following lemma

Let $T:\mathcal{D}(T)\to Y$ be a bounded linear operator with domain $\mathcal{D}(T)\subset X$, where $X$ and $Y$ are normed spaces. Then:
(a) If $\mathcal{D}(T)$ is a closed subset of $X$, then $T$ is closed.
(b) If $T$ is closed and $Y$ is complete, then $\mathcal{D}(T)$ is a closed subset of
$X$. (page 295)

and the following problem

Let $X$ and $Y$ be normed spaces. If $T_1:X\to Y$ is a closed linear operator and $T_2\in\mathcal{B}(X,Y)$, show that $T_1 + T_2$ is a closed linear operator. (page 296)

If $X$ and $Y$ are Banach spaces, then we can conclude (by Closed Graph Theorem) that $T_1$ is bounded. So, $(T_1+T_2):X\to Y$ is bounded and thus it's closed (by lemma above).
My question is: how to solve it when $X$ and $Y$ are not necessarily complete spaces?
Thanks.
Comment: Did you tried to prove it without this lemma, only by using the definition?
Answer: Possible Method: Let $x_n  \to x \in X$ and $(T_1+T_2)x_n \to y \in Y$. What you need to show is that $(T_1+T_2)x = y$. Now, $T_2 x_n \to T_2 x$ since $T_2 : X \to Y$ is continuous. From here we can prove that $T_1 x_n \to y-T_2 x$. Indeed $|T_1 x_n + T_2 x - y| \leq |T_1 x_n + T_2 x_n - y| + |T_2 x_n - T_2 x| \to 0$. Therefore $T_2 x_n \to y - T_1 x$. Since $T_1$ is closed this means that $y-T_2 x = T_1 x$. Equivalently $y = (T_1 + T_2)x$.


multiplication or composition?

Question: I am studying the ring homomorphism of the following function:
If $f: M \to N$ is a smooth function, then  $$f^*: C^{\infty}(N) \to C^{\infty}(M), \textbf{ defined by } \phi \mapsto \phi \circ f$$ is a ring homomorphism.
I know that the given function $f^*$ should satisfy that for every $h,g \in C^{\infty}(M),$ we have that $f^*(hg) = f^*(g) f^*(h).$ my question is should the operation between $g,h$ be  multiplication or composition and why?
Could someone explain this to me please?
Comment: There is a reason to use a different alphabet for the elements of $C^{\infty}$ and $f:M\to N.$ $f^{*}(hg)$ makes $f,g,h$ look similar. They are not.
Answer: It is multiplication. If $h$ and $g$ are maps from $N$ into $\Bbb R$ how could you possibly compose them? Only if $N=\Bbb R$.
When I write that it is multiplication, what I mean is that $hg$ is the map $n\mapsto h(n)g(n)$
Comment: but then it will not be a ring homomorphism which is what is required to be proved ... right?
Comment: After this $f^*(g h) = (gh) \circ f$ what should I do?
Comment: Is $(gh \circ f) = (g \circ f)(h \circ f)?$
Comment: Sure it is! You have\begin{align}(gh\circ f)(x)&=(gh)\bigl(f(x)\bigr)\\&=g\bigl(f(x)\bigr)h\bigl(f(x)\bigr)\\&=(g\circ f)(x)\cdot(h\circ f)(x).\end{align}
Comment: I have never seen this before .... multiplication in case of composition is like coset multiplication?
Comment: I fail to see a connection between those two concepts here. All I use is composition of functions and multiplication of real numbers.
Comment: oh, so you are looking to $f(x)$ like a new variable say $y$ and you are evaluating $g$ and $h$ on it?
Comment: Sure. That's how function composition works, right?!
Comment: yeah, you are correct but in the end it looks like coset multiplication

multiplication or composition?


Possible typo in Brown and Churchill's "Complex Variables and Applications"

Question: The following theorem is presented on pages 56 and 57 in the 6th edition book.
Theorem: If $f'(z)=0$ everywhere in a domain $D$, then $f(z)$ must be constant throughout $D.$
The proof sensibly derived that $u_x=u_y=v_x=v_y=0$ using the fact that $f'(z)=u_x+iv_x=0$ and the Cauchy-Riemann equations. Then, it was reasoned that $\vec{\nabla}u=\vec{0}\Rightarrow D_{\hat{u}}u(x,y)=0$ for any unit vector $\hat{u}$. I understood all of that.
The problem is, they then said exactly: "Consequently, $u$ is a constant along any line segment lying entirely in $D$; and, since there is always a finite number of such line segments, joined end to end, connecting any two points in $D$, the values of $u$ at those points must be the same."
Question 1: Is the idea that you can keep connecting line segments inside of $D$ forever until you fill up all of $D$ and conclude that every point has the same value?
Question 2: If so, shouldn't it say infinite instead of finite?
Any input is appreciated.
Comment: no, the idea is that you can connect any two points $w,z$ by a finite number of such segments, so by previous observations $f(z)=f(w)$; the statement is good as is
Comment: yes because it shows that $f$ takes the same value at every pair of points so now you can reason for example by fixing a base point $z_0$ and then $f(z)=f(z_0)$ for any other $z$ by considering the pair $(z_0,z)$
Comment: I must have been confusing the amount of *ways* to connect them instead of the actual number of line segments connecting points in general. And since this applies for every pair of points in $D$ it proves the theorem?
Answer: The text is correct as written. You can indeed connect any two points in a domain (in fact, in any connected open subset of Euclidean space) using finitely many straight lines. The rough idea as to why this is true is that it is "clearly" true on open balls. I won't bother showing that rigorously, but I hope it's intuitively clear from geometric reasoning. In any case, the proof isn't too bad.
Anyway, let $U \subseteq \mathbb R^n$ be open, connected, and nonempty. Pick some $u \in U$ and let $W \subseteq U$ be the set of all points in $U$ that can be connected to $u$ via finitely many straight lines. From now on, I will call such a path a polygonal path. We will show that $W$ is both open and closed in $U$, so as $W$ is nonempty and $U$ is connected, $W = U$ proving that any two points in $U$ can be connected via a polygonal path.
Let's start with showing that $W$ is open. Let $w \in W$. Then there is some open ball $w \in B \subseteq U$, as $U$ is an open subset of Euclidean space. Now, as I remarked above, any two points in an open ball can be connected via a polygonal path. Hence, every single point in $B$ can be connected to $w$ via a polygonal path. As $w$ is connected to $u$ via a polygonal path by assumption (it is in $W$), then by concatenation, every point in $B$ can be connected to $u$ via a polygonal path. Now by definition of $W$, $B \subseteq W$ so $W$ is an open subset of $U$.
Now we show that $W$ is closed, i.e. that $U - W$ is open. The argument is essentially the same. If there was an $x \in U - W$ then take a ball with $x \in B \subseteq U$. If there was a $y \in B \cap W$ then $x$ would be connected to $y$ and $y$ would be connected to $u$ via polygonal paths. Hence, $x$ would be connected to $u$, but we assumed $x \notin W$. Contradiction. Hence, $U - W$ is open so $W$ is open, closed, and nonempty. Thus, as $U$ is connected, $W = U$ and we are done.
Note that the above argument could be slightly refined using equivalence classes, but I felt that this was good enough and a bit more elementary, in case you are unfamiliar with these. Regardless, here's the idea. Let $a \sim b$ in $U$ if $a$ and $b$ can be connected via a polygonal path. Then the above argument shows that the equivalence classes of $U$ under $\sim$ are open. Since equivalence classes are disjoint, they form a partition of $U$ into open subsets. But as $U$ is connected, there can be only one equivalence class, so every pair of points in polygonally path connected. This is essentially the exact same proof I gave above, but with the language of equivalence classes.
Comment: Glad I could help!
Comment: Thanks! I am not particularly familiar with equivalence classes, but I appreciate the proof you offered.

Possible typo in Brown and Churchill’s “Complex Variables and Applications”


Equation $\sin(\pi x)=|\sin(\pi ax)|$?

Question: When $a$ is an integer and $x\in[0, 0.5]$,
is there any closed-form solution to the equation
$\sin(\pi x)=|\sin(\pi ax)|$?
I just want to find the largest solution $x\in[0, 0.5]$ (there are more than one solutions if $a>2$.)
When $a=2$, the solution is easily found by using high-school math, but generally....
Thanks in advance.
This picture is when $a=4$ for example.
Answer: Absolute values make things more complicated than necessary.
$sin(a)+sin(b)=2sin(\frac{a+b}{2})cos(\frac{a-b}{2}).$
$sin(a)-sin(b)=2sin(\frac{a+b}{2})cos(\frac{a-b}{2}).$
The first equation's solutions contain the solutions to $sin(a)-|sin(b)|$ when $sin(b) \leq 0,$ the second when $sin(b) \geq 0.$

Equation $\sin(\pi x)=|\sin(\pi ax)|$?


Why is sample variance squared? (Basic Question)

Question: I have two questions, each of which I think might be related to each other but I'm not sure. Both concern the definition of variance as:

$var(x) = s_x^2 = \dfrac{1}{n-1}  \sum_{i=1}^{n}(x_i - \bar{x})^2$

(1) How do we prove that $\sum_{i=1}^{n}(x_i - \bar{x}) = 0$? I can see that this is true using a few examples but I'm unsure how to do a general proof.
(2) In this definition, what is the conceptual motivation for having the squared difference squared, as opposed to cubed or some other exponent?
Comment: Ah okay, this answers question (2) that I had. Thank you so much!

Question (1) nonetheless remains for me.
Answer: Well, the first one is really easy:
Since
$\bar{x} = \frac{1}{N}\sum_i x_i$
it follows that
$\sum_i x_i = \bar{x}\space N $
so you have:
$\sum_i x_i - \sum_i \bar{x} = \bar{x}\space N  - \bar{x}\space N = 0$
Regarding your second question, I think it is mostly related to analytical tractability, but I'll let others make a more informed contribution.
Comment: In English, that says: "sum N times the average value $\bar{x}$. Imagine the average is 5 and there's N=100 observations. You are summing 5, 100 times right? that's obviously 100*5.
Comment: This is somewhat helpful however:

(a) How does $\Sigma_i \bar{x} = \bar{x}N$?

(b) How is it that $\Sigma_i x_i - \Sigma_i \bar{x} = \Sigma_{i=0}^{n} (x_i - \bar{x})$?
Comment: @letsmakemuffinstogether $\bar{x} = \frac{\sum_{i}{x_{i}}}{N} $ , multiply both sides by $N$ gives $N\bar{x} = \sum_{i}{x_{i}}$. $\sum_{i}{\bar{x}} = \bar{x} + \bar{x} + \dots + \bar{x}$ $N$ times.
Comment: Hi Alexis. I understand the claim you just made however if you look very closely at my question (a), you'll see that I am asking instead how it is that $\Sigma_{i}\bar{x} = \bar{x}N$, and not the more obvious question of how $N\bar{x} = \Sigma_i x_i$... Notice the difference in that there is a bar over the $x$ on both sides in my question.


Why do I get a castError when I try to push an object to an array in MongoDB and Nodejs?

Question: I have a mongoDB and I am trying to make a Nodejs server to manipulate the data in the database. I get a castError when I try to push a Comment to the Comments array in a BlogPost object.
Source code below, please tell me if you're missing important information.
Thanks in advance!
The route:
routes.post('/comments/push/:id', function(req, res) {
const blogPostId = req.param('id');
const commentProps = req.body;

BlogPost.findById(blogPostId)
  .then((blogPost) => {
    blogPost.comments.push(commentProps);
    return blogPost.save();
  })
    .then((blogPost) => res.status(200).json({
    'status': 'Comment is deleted.',
    'comment': blogPost
}))
.catch((error) => res.status(400).json(error)) });

The BlogPost schema:
const BlogPostSchema = new Schema({
content: {
type: String,
validate: {
  validator: (content) => content.length > 5,
  message: 'Content must contain at least 6 characters.'
},
required: [true, 'Content must be filled in.']
},
rating: Number,
user: { type: Schema.Types.ObjectId, ref: 'user' },
board: {type: Schema.Types.ObjectId, ref: 'board'},
comments: [{
  type: Schema.Types.ObjectId,
  ref: 'comment'
  }]
});

The Comment schema:
const CommentSchema = new Schema({
content: {
type: String,
validate: {
  validator: (content) => content.length > 5,
  message: 'Content must contain at least 6 characters.'
},
required: [true, 'Content must be filled in.']
},
user: { type: Schema.Types.ObjectId, ref: 'user' },
rating: Number

// board: Board
});

Here is the error in postman:
postman screen
Help would be greatly appreciated!
Answer:
first make sure what's you are receiving in req.body, it's not good
to store directly from req.body.
second ref. comments Schema aspect a objectId while req.body is
Object itself. i'm not sure what are you going to do but it
something like blogPost.comments.push(req.body.someValidId);

third why 2 queries for simple update. you can use $push, $addToSet to push directly to comments or $pull to remove from comments.
BlogPost.findOneAndUpdate({
    _id:blogPostId
}, {
    $addToSet:{
        comments : someValidId
    }
}, {
    new :true,
    upsert:false
})
.then((blogPost) => {
    res.status(200).json({
        'status': 'Comment is deleted.',
        'comment': blogPost
    })
})
.catch((error) =>
    res.status(400).json(error))
});


jvm options not passed on to forked process

Question: According to the documentation, an sbt forked process should receive the jvm settings of the current process:
By default, a forked process uses the same Java and Scala versions being used for the build and the working directory and JVM options of the current process. See: http://www.scala-sbt.org/0.13/docs/Forking.html
However that does not seem to be the case for me. Take the following test:
object Test {

        def main(args: Array[String]): Unit = {

                println("Conf: " + System.getProperty("config.resource"))
        }
}

If I run this with sbt -Dconfig.resource=test.conf then "Conf: test.conf" gets printed. But once I add fork in run := true in my build.scala "Conf: null" is printed out. Which implies to me that the jvm options are not in fact getting passed to the forked process. Can someone tell me what am I missing here?
import sbt._
import Keys._

object Build extends Build {
        lazy val root = (project in file(".")).
        settings(
        fork in run := true
        )
}

Answer: The SBT documentation is correct, the JVM properties do get passed to the forked process.  You're concerned about the System properties, however, which need to be passed manually.  Try this to pass all system properties:
import scala.collection.JavaConversions._

javaOptions in run += {
  val props = System.getProperties
  props.stringPropertyNames().toList.map { configKey =>
    s"-D$configKey=${props.getProperty(configKey)}"
  }.mkString(" ")
}

Answer: This is what I used. It's an updated version of josephpconley's answer.
  javaOptions ++= {
    val props = sys.props.toList
    props.map {
      case (key, value) => s"-D$key=$value"
    }
  },

Answer: if you ask sbt to fork the VM in which it runs your code, then it doesn't inherit the system properties of the parent VM
fork in run := true

fork in console := true

javaOptions in run += s"-Dconfig.resource=${Option(System.getProperty("config.resource")).getOrElse("default")}"

javaOptions in console += s"-Dconfig.resource=${Option(System.getProperty("config.resource")).getOrElse("default")}"

This Works for me...
Comment: This is not my question. I want the jvm options from the current process as specified in the documentation.
Comment: Sorry, I misunderstood the question at first.... I've Edited the answer to help you answer
Answer: When you fork JVM, you actually create new process. Sbt will not copy JVM arguments to the new process. You have to specify those explicitly, for example:
javaOptions in test += "-Dconfig.file=app.test.conf"

When you forbid forking in test, for example:
fork in test := false

your tests are being run in the same JVM(with it's arguments). Hope this helps.


Homomorphisms and exact sequences

Question: For given $A$-modules and homomorphisms $M'\stackrel{u}\to M\stackrel{v}\to M''\to 0$ this is an exact sequence iff for all $A$-modules $N$, the sequence
$$0\to\operatorname{Hom}(M'',N)\stackrel{\overline{v}}\to \operatorname{Hom}(M,N)\stackrel{\overline{u}}\to \operatorname{Hom}(M',N)$$ is exact.
How to prove exactness at $\operatorname{Hom}(M,N)$ ?
Any help would be appreciated.
Answer: Suppose $M' \rightarrow M \rightarrow M'' \rightarrow 0$ is exact.
Since $v$ is surjective, $Hom(M'', N) \rightarrow Hom(M, N)$ is injective.
Let $p\colon M \rightarrow N$ be a homomorphism such that $pu = 0$.
Let $y \in M''$.
There exists $x \in M$ such that $y = v(x)$.
Since $Im(u) = Ker(v) \subset Ker(p)$, $p(x)$ is independent of a choice of $x$.
Hence we get a map $q\colon M'' \rightarrow N$ such that $q(y) = p(x)$.
It is easy to see that $q$ is a homomorphism.
Since $qv = p$, $Ker(Hom(M, N) \rightarrow Hom(M', N)) \subset Im(Hom(M'', N) \rightarrow Hom(M, N))$.
The other inclusion is clear.
Conversely suppose $0 \rightarrow Hom(M'', N) \rightarrow Hom(M, N) \rightarrow Hom(M', N)$ is exact.
I will prove that $M' \rightarrow M \rightarrow M'' \rightarrow 0$ is exact.
Since $Hom(M'', N) \rightarrow Hom(M, N)$ is injective, $v$ is surjective.
So it suffices to prove that $Im(u) = Ker(v)$.
Since $0 \rightarrow Hom(M'', M'') \rightarrow Hom(M, M'') \rightarrow Hom(M', M'')$ is exact.
$vu = 0$. Hence $Im(u) \subset Ker(v)$.
Let $N = M/Im(u)$.
Let $p\colon M \rightarrow N$ be the canonical homomorphism.
Since $pu = 0$, there exists $q\colon M'' \rightarrow N$ such that $qv = p$.
Let $x \in Ker(v)$.
Since $0 = qv(x) = p(x), x \in Im(u)$.
Hence $Ker(v) \subset Im(u)$.

Homomorphisms and exact sequences


C++ Linked List

Question: I am trying to create a Linked List to download a txt file and use the linked list to handle the file line by line. When handling the downloaded Linked List operations will be performed on it such as a text editor would.
I am encountering some problems however. It seems the "Node(string value)" section of the code has something wrong with it even though the original Node() declaration with no arguments passes. I am unable to figure out quite what it is.
Node.h
class Node
{
public:
    Node();
    Node(string value);
    void setNext(Node *nextNode); // Allows the user to set where the "next" pointer of a node points

    friend class LinkedList;
private:
    string data; // Data box
    Node* next; // Pointer box
};

Node.cpp
# include <string>
# include "Node.h"

using namespace std;

Node::Node()
{
    data = "";
    next = NULL;
}

Node::Node(string value)
{
    data = value;
    next = NULL;
}

void Node::setNext(Node *nextNode) // Allows the user to set where the "next" pointer of a node points
{
    this->next = nextNode;
}

Comment: Please be more specific than "some problems" and "something wrong". Does it not compile? Does it crash? Does it print "0x3434" all over your terminal? Does it call the police and report that you've gone missing?
Comment: The problem is there is no `string`. There is `std::string`. Related, never, **ever**, build a header-dependency chain in a source file's include list. If header `X.h` doesn't work until the source file including it includes `Y.h` before it, you did it wrong. As written even if you fixed what I mentioned `Node.h` is useless unless `` is included by the source cpp file first. Properly include-guard your headers and have them pull in what you need (and *nothing more*) for that header.
Comment: @BrunoFerreira and then [forget you ever thought about `using namespace std;`](https://stackoverflow.com/questions/1452721/why-is-using-namespace-std-considered-bad-practice).
Comment: @molbdnilo, may be it just posts random questions on SO? ;)
Comment: The following errors are given:

missing type specifier - int assumed.  Line: 17
'data': unknown override specifier Line: 17
Comment: Maybe you want to change `Node(string value);` to `Node(std::string value);` in the header. And also add `#include ` to the header before your class declaration.
Comment: see `using namespace std`
Comment: `using namespace std` is in the cpp file not the header. Well at least the code we see.
Comment: @WhozCraig indeed `using namespace std` is terrible. I myself rather continue `using std::whatever` :)
Comment: Can anyone also tell my why "Data is not a member of node"?
Answer: Your #include <string> should be in your header file since it is you are using the std::string type with your methods.
Since it is not recommended to add using namespace in header files (see this answer for more info), declare your string types with the namespace : change your string value to std::string value
Your files will look like this (compile test was done with GCC)
Also you should put an include guard in your header file
Example:
// some_header_file.h
#ifndef SOME_HEADER_FILE_H
#define SOME_HEADER_FILE_H
// your code
#endif

Node.h
#include <string>

class Node
{
public:
    Node();
    Node(std::string value);
    void setNext(Node *nextNode); // Allows the user to set where the "next" pointer of a node points

    friend class LinkedList;
private:
    std::string data; // Data box
    Node* next; // Pointer box
};

Node.cpp
#include "Node.h"
#include <cstddef>  // For NULL

Node::Node()
{
    data = "";
    next = NULL;
}

Node::Node(std::string value)
{
    data = value;
    next = NULL;
}

void Node::setNext(Node *nextNode) // Allows the user to set where the "next" pointer of a node points
{
    this->next = nextNode;
}

Comment: [Or maybe `#pragma once` instead of include guards](http://stackoverflow.com/questions/1143936/pragma-once-vs-include-guards)
Comment: It suggests string is not a component of the std library.
Also I already have
#ifndef NODE_H
#define NODE_H
and #endif.. they are just not what I'm asking for.
Comment: ***It suggests string is not a component of the std library.*** Did you replace `string` with `std::string` in the header and cpp file?
Comment: Did you replace string with std::string in the header and cpp file?
Yes I did and it works now.

C++ Linked List


How to use Wildcards in Obfuscator XML

Question: I have a Obfuscator XML that obfuscates the hardcoded file names
<?xml version="1.0" encoding="utf-8" ?>
<Obfuscator>
  <Var name="InPath" value="." />
  <Var name="OutPath" value="Obfuscator_Output" />
  <Module file="$(InPath)\Pranay.Hi.abc.GeneratorToken.dll" />
  <Module file="$(InPath)\Pranay.Hi.abc.HandlerException.dll" />
  <Module file="$(InPath)\Pranay.Hi.abc.CertHandling.dll" />

</Obfuscator>

Instead of hardcoding the files name, is there any way to include all the files by giving something like  so that all files starting from Pranay. are included and we don't need to hardcode them at all.
Comment: You might refer to https://github.com/obfuscar/obfuscar/pull/254
Answer: You can include files using wildcards with the following configuration:
  <Modules>
    <IncludeFiles>$(InPath)\Pranay.Hi.abc.*.dll</IncludeFiles>
  </Modules>

The author of this feature provided an example that can be viewed here. I have also provided their example below:
<Module file="$(InPath)\SomeFile.dll">
    <Include path="obfuscar.skip.xml" />
</Module>
<Modules>
   <IncludeFiles>$(InPath)\Acme.*.dll</IncludeFiles>
   <ExcludeFiles>$(InPath)\Acme*Tests.dll</ExcludeFiles>
   <Include path="obfuscar.skip.xml" />
</Modules>

The documentation for this feature hasn't been written at the time of writing this answer. It should be linked here once available.

How to use Wildcards in Obfuscator XML


Deriving Eulers method

Question: I am wondering if it is possible to derive Eulers method without doing it graphically.
I am trying to give a detailed derivation of Eulers method, and how the formula is found
$$y_i=y_{i-1}+hf(x_{i-1},y_{i-1})$$
If anyone could derive this formula and show how its found would be great, or even a link to a website/textbook. Thanks!
Answer: Here's one simple way to discover Euler's method for solving the differential equation $y'(x) = f(x,y(x))$. You know that
$$y(x_i) \approx y(x_{i-1}) + y'(x_{i-1})\Delta x = y(x_{i-1}) + f(x_{i-1}, y(x_{i-1})) \Delta x.$$
That comes directly from the definition of the derivative. Unfortunately, $y(x_{i-1})$ is unknown, but at least we have the approximation $y_{i-1}$ that we already computed at the last iteration. This suggests approximating $y(x_i)$ by the number $y_i$ given by
$$
y_i = y_{i-1} +
f(x_{i-1}, y_{i-1}) \Delta x.
$$
Answer: Euler's Step Method attempts to find the solution of an ODE numerically.
You are given an initial value, of $y$, and you want to use the gradient of the function to calculate subsequent values.
I will assume that $f(x_{i-1},y_{i-1})$ is the gradient function.  Thus, plugging $x$ and $y$ values into this equation will provide the gradient at a particular point.
Recall that the gradient is the change in the $y$-value divided by a change in $x$.  Thus, multiplying your step-size of $h$ by the gradient will estimate the change in the $y$-value.  You then add this to the $y$-value at the point to get the new $y$-value of $y_{i}$.
As the step-size gets smaller, one obtains a better approximation.

Deriving Eulers method


O código não respeita os ifs

Question: Olá, neste código quando executa ele não respeita os ifs, ele passa reto executando tudo. Por exemplo, caso apareça a mensagem "Esse usuário não existe" ele deveria simplesmente encerrar. mas também aparece a mensagem "Senha incorreta".
<?
include "conection.php";

$login = $_POST['login'];
$senha = $_POST['senha'];

$sql = mysqli_query($con, "SELECT * FROM usuarios WHERE login = '$login'");

while($linha = mysqli_fetch_array($sql))
{
    $senha_db = $linha['senha'];
}

$cont = mysqli_num_rows($sql);

if($cont == 0)
{
    echo "<meta http-equiv='refresh' content='0; url=index.php'>
    ";
}
else
{
    if($senha_db != $senha)
    {
        echo "<meta http-equiv='refresh' content='0; url=index.php'>
        ";
    }
    else
    {
        session_start();

        $_SESSION['login_usuario'] = $login;
        $_SESSION['senha_usuario'] = $senha;

        header("Location: perfil.php");
    }
}

mysqli_close($con);
?>

Answer: Na verdade você está atualizando a pagina quando você usa o META REFRESH.
Então quando o script para no segundo IF, ele recarrega a pagina d exibe o alerta. No entanto, a pagina volta a ser executada, mas desta vez não existe mais o $_POST, então o script cai no primeiro IF porque não havia usuário. E vai ficar assim.
Sugiro que deixe de usar o META REFRESH e passe a usar o próprio php:
Ou redirecionando pelo header:
header("location: sua_pagina.php");

Ou usando ifs para exibir o alerta JavaScript:

O código não respeita os ifs


Getting the results of an Array.toString() without having the items separated by a comma

Question: I have an array with messages on them, some of which contain commas. I would like to make this whole array into a string by using .toString(), but instead of separating them by commas, I'd like to separate them by line breakers (\n).
Unfortunately, since some of these messages are long and may contain commas of their own, I don't think simply replacing them all using myArray.replace(/,/g, '\n') will do the trick.
I know I could also simply do a .forEach() and copy the strings into a new big string. But I was wondering if there is a better way to do this? Perhaps somehow overload the toString() method so that the items are separated by \n instead of the comma?
Answer: Use join('\n') for that case. Array.join() allows you to specify the combiner between the array elements. \n in this case. Similarly, you can use any character or text for the combiner between the array elements like join(','), join('-') or any other. join(',') is same as toString()

var arr = ['some', 'word', 'here'];
var res = arr.join('\n');
console.log(res);
Comment: In fact, [`toString` *calls* `join`](https://tc39.github.io/ecma262/#sec-array.prototype.tostring). :-) It doesn't provide any argument, so `join` uses its default `","`.
Comment: :-) It's not really a matter of opinion.
Comment: @T.J.Crowder agree with you.
Comment: @T.J.Crowder that's good to know! I'll be using mainly `join` now


neural networks - What is meant by "linear combination of inputs"

Question: Just starting out with MLPs. I am reading a tutorial that I found here.
It says that the disadvantages of using a linear function is that the neural net will only be restricted to learning "linear combination of inputs"
Can someone please explain via an example what a linear combination of an input is ?
I know what linearly separable functions are, but what is a linear combination of inputs ?
Answer: Suppose we have inputs $x, y, z$, a linear combination of inputs simply means:
$ax+by+cz$.
Ref: http://mathworld.wolfram.com/LinearCombination.html
That would be undesirable for neural networks since usually we want the network to approximate some non-linear functions.


Computing regression coefficient values in OLS

Question: I have a question on testing the hypothesis that a particular regression coefficient in a simple OLS scheme with all the good assumptions is zero or not.
In particular eq 3.12, in the book by Tibshirani and coll., Elements of Statistical Learning defines the z-score as
$$z = \beta/\hat\sigma\sqrt{v_j}$$
My question is that given the regression coefficients are all jointly normal, how can we separate one coefficient out like that? Is that a conditional z-score assuming all others betas have a particular value?
Answer:
If regression coefficients are jointly normal, then single coefficient is normal. Since we want to test hypothesis only about single coefficient we use its own distribution. Why should we care what happens to other coefficients?
If however we want to take into account other coefficients, then we should modify the hypothesis. So for example we can test that two coefficients are zero. So the answer is no, it is not conditional z score, but it is possible to have a statistic similar to z-score involving more that one coefficient.
Comment: Thanks a lot for the response mpiktas, here are some follow up thoughts. Forgive my ignorance! How about independence, I guess under classical OLS it is fair to say that the betas are independent, is that the case?. If yes, then the multivariate normal degenerates to individual normal distributions and then I can build any test on any single coefficient. If no, then how does one take into account the values other coefficients are taking. We cannot test a condition on a dependent variable (beta) without thinking what the other variable (beta) is doing. (...)
Comment: (...) What happens to a more complex case of lasso, where we know that betas are dependent? (copy/paste since the last sentence was lost when converting to comment)


Hibernate Interceptor don't receive changed element collection

Question: I have an entity with @ElementCollection inside and I also have Hibernate Interceptor to intercept changes.
Once I change fields of entity, interceptor can intercept properly but once I only changed fields of component inside @ElementCollection, interceptor cannot intercept it (both entity itself and embedded component).
I also tried debugging at onFlushDirty, onCollectionUpdate, onCollectionRecreate but has no luck.
Answer: The problem is about default dirty checking of hibernate don't do it properly on PersistentSet, I have to manually check for dirty by overriding findDirty method and check it via PersistentSet#isDirty manually.