IID Assumption in Sequential Supervised Learning

Question: I have seen two posts about this before (Are RNNs inherently flawed? Supervised Learning assumes IID data but sequential data is not IID, Realistically, does the i.i.d. assumption hold for the vast majority of supervised learning tasks?), however, the answers did not really satisfy me - thus I would like to open this topic again.
I am wondering how the iid assumption about our data distribution works for recurrent problems / models.
Say we apply an LSTM to a sequential problem.
We usually put up an iid assumption, because we use MLE to find best parameters, i.e. maximize P(X|θ), with X being the data and θ the parameters, and for calculating this break this down to P(x_1|θ) * ... * P(x_n|θ) (more specifically, assume we are doine classification, thus "data" and the x_i's are actually our predictions).
Now assume our first task is classifying full sequences, i.e. having multiple points per sequence and only outputting one prediction  per sequence. Then, I believe the iid assumption still correctly holds.
However, when we now predict one label in each step, these depend on previous predictions, and our above made assumption and reformulation of the likelihood should be wrong - is that correct?
Answer: Time series don't have i.i.d. components, otherwise there's no need to model them sequentially taking time into account but we could treat each sample individually (shuffle all of them and learn distribution). In general we assume that the distribution of a given sample depends on the past samples as you said. But this doesn't contradicts the assumption that each of the sequences (not samples of each sequence) in the dataset are i.i.d.
Say you have a speech dataset. You can consider each chunk or each audio as a vector and each of such vectors to be i.i.d (not amplitude values but a representation of the speech sequence).
To illustrate this better, you can think of the problem of modelling images. You can vectorize the image or treat it as a matrix. You usually have a lot of statistical dependence between neighboring pixels (that's what image coders exploit) but still you can consider each image to be i.i.d (each vector or each matrix come from the same distribution).
Hope this helps you clarify.
PS: For the formulation of the joint probability of a sequence and how to predict sequences with CNNs you can check the famous WaveNet paper.
Comment: Thanks for your answer! I understand your points, still, unfortunately am not sure about one thing: Lets say we input speech and output words (roughly assuming there is a one to one mapping between "one" speech input and the written word, so in every step there is one input and one output). I understand, the different sequences are iid, so that does not matter here. However, when trying such a model I would maximize the likelihood of the given data, namely sth like P(Y|θ), where Y are my predictions for that sequence given my parameters
Comment: (could e.g. modelled by cross-entropy to the ground truth or whatever). Then, what applying the cross-entropy function actually does imo is writing this as P(y_1|θ) * ... * P(y_n|θ), e.g. calculating cross-entropy between every predicted work and every target word, and multiplying, and then averaging for example. Here, as far as I see it, we still make use of the iid assumption between the data points (otherwise we could not decompose our likelihood as such) - isn't that, theoretically a problem? I hope, I explained my problem somewhat clearly ...
Comment: If I get what you say, even if you are evaluating elementwise differences with your target, it doesn't mean you need this i.i.d. assumption. For example, when fitting a line in a scatter plot it makes perfect sense to use MSE to assess your model even if you are modelling some correlation, right?. Using cross-entropy (as WaveNet does) is more of the same, but instead you do not assume any distribution as MSE does (if you penalize near errors less as in MSE you implicitely assume some shape of error distribution). You can assume the error is independent but not the data itself.
Comment: Thats very helpful, thanks - just to clarify one last time: youre saying, it is important that our predictions roughly are iid, which basically is the case for sequential models, not that the data is? further, for RL this yields problems because there we are interested in sth like that because there our "predictions" are actions or value estimations, and these are not iid because the agent actually is taking actions, reaching different states?
Comment: What I am saying is that if the model is "good enough", then the error is independent. If you see classical models for time series such as ARIMA, one way to know if the model fits the data correctly is to do statistical tests on the error signal to see if it corresponds to noise (usually white Gaussian). So note that the predictions and the data are not independent. And if noise has some time dependence, you can augment the capacity of the model, stack another LSTM... then it makes sense to assume independence.
Comment: Note that even if independence doesn't hold, the model could give good enough results (e.g. the case of naive bayes). In the case of RL this assumption may be too strong as you said. But once you condition the model on the current state this assumption becomes more realistic (in Wavenet they do something similar conditioning on speaker, speech features, ...). Note that more conditioning means less variability of the output and if we pick this information correctly it can yield less correlation between states. A classical method using this idea is the Hidden Markov Model.


Is the adjusted R square good for the interpretation prediction accuracy of machine learning regression model and how to interpret it?

Question:
Is the adjusted R square good for the interpretation of machine learning regression (SVM, Random Forest, Elastic Net, etc) model prediction accuracy?
How to interpret machine learning regression prediction accuracy model with adjusted R square?
Comment: "Accuracy" tends to be used in classification problems rather than regression problems. (That it happens to be problematic, even for balanced classes, is a separate issue.) Do you mean a "regression" problem or a "classification" problem?
Comment: @Dave regression problem.
Answer: An adjusted $R^2$ is problematic in practice for many machine learning models, since the degrees of freedom range from complicated to calculate to totally unclear. Combine that with the fact that many models are fit after performing preliminary steps like hyperparameter tuning, and calculating the degrees of freedom turns into a real mess.
The degrees of freedom matter to the adjusted $R^2$ because the standard adjusted $R^2$ from OLS linear regression uses a comparison of the residual variance of your model and the residual variance of a model that predicts the mean of $y$ every time, and adjusted $R^2$ considers unbiased estimators of those respective variances whose calculations require use of the degrees of freedom (that's the $n-p-1$ in the link).
If you want an $R^2$-style measure of model performance that penalizes the model for having many parameters that put it at risk of overfitting, you might want to calculate an out-of-sample $R^2$-style metric. As I discuss here, several calculations can be defended as out-of-sample $R^2$-style calculations, though the one that makes the most sense to me that I discuss here uses the following formula.
$$
1-\left(\dfrac{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\hat y_i
\right)^2
}{
\overset{N}{\underset{i=1}{\sum}}\left(
y_i-\bar y_{\text{training}}
\right)^2
}\right)
$$
If you can get a handle on the degrees of freedom, however, something like adjusted $R^2$ makes perfect sense as a comparison of variances that are estimated with unbiased estimators.
Being explicit:

If you get a handle on the degrees of freedom, I say this makes perfect sense. (However, do not underestimate the difficulty of getting a handle on the degrees of freedom.)

I would interpret such a statistic as a comparison of unbiased estimates of error variances.


Validating the percent variance explained of principal components on out-of-sample data

Question: I'm trying to ascertain whether the variance explained by a certain PCA on an out-of-sample dataset is not due to random chance.
Suppose I have a dataset X with size n-x-p, and I run a PCA and obtain the p-x-p matrix W of coefficients. I obtain another dataset Y with size m-x-p, and I project the values of Y onto the principal axes of the 1st dataset X by W*X. I am able to obtain the variance explained by calculating the ratio of the variance of the residuals and actual variance of Y. We are assuming that X and Y are drawn from a population following a normal distribution.
Now, I want to know whether the variance explained by these PCs on Y is not just due to random chance.
My attempt:
I've tried bootstrapping the original dataset X and running PCA on the bootstrapped dataset and projected Y onto those new PCs. I calculate the percent variance explained by from each bootstrapped PCA, and they are fairly close to the initial calculated % variance explained.
Now to know what the percent variance explained due to random chance, I shuffle the original dataset X several times, and run the PCA on these. I expect that the value of the percent variance explained is 1/p. However, when I do this calculation with my out-of-sample dataset, I am now getting negative values for the percent variance explained.
So there is clearly an error in my approach. Can anyone point me in the right direction, please?
Answer: Please clarify your bootstrap approach.  The Efron-Gong optimism bootstrap (i.e., using the bootstrap to estimate bias in a quantity such as $R^2$ or proportion of variance explained by $PC_1$) involves estimating parameters in one of many bootstrap samples and applying these parameters to the original whole dataset.   In your setting the parameters involve the loadings of $PC_1$.  These loadings are also a function of the pre-analysis standard deviations that you have to estimate to use $PC$s in the usual way.  So you'd also be validating the normalization stage simultaneously with validating the loadings on the standardized quantities.
I'm glad you are considering the volatility of $PC$ because most analysts act as if the loadings are estimated without error.  By following the full bias-estimating bootstrap procedure (and I hope you provide code that all of us can benefit from) you average over a few hundred bootstrap resamples this quantity: proportion of variance explained in a bootstrap sample minus what that same set of loadings explains in the original sample.  Then you subject that average optimism from the original apparent proportion explained to get a bias-corrected estimate of the proportion.  This estimates the likely future performance of your original $PC_1$.


How can I connect my condenser microphone to my PC using a Saffire PRO 24?

Question: I'm pretty new to all this and I have a RODE NT 1A condenser microphone and an old Saffire PRO 24 DSP audio interface. I've connected the interface to my computer using firewire and a Pcie express card. I tried connecting the microphone to the interface using a XLR to TRS cable because it doesn't have any XLR inputs. However, that doesn't work.
Is it true that a 1/4 connector can't provide phantom power?
I really hope someone can show me a way to make it work.
Equipment:
https://www.thomann.de/gb/rode_nt1a_complete_vocal_recording.htm?shp=eyJjb3VudHJ5IjoiZ2IiLCJjdXJyZW5jeSI6NCwibGFuZ3VhZ2UiOjJ9&reload=1
https://www.pdf-manuals.com/pdf/focusrite-saffire-pro-24-16-x-8-audio-saffire-pro-24-b-h-98025-user-manual.pdf
https://www.amazon.de/gp/product/B09PVCFJ99/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&th=1
Comment: I'm confused: The Saffire PRO seems to have two big ol' XLR inputs in the front? p8 of the pdf. Anyway, I can answer one part: no, 1/4 inch can't provide phantom power. You can get a little in-line adapter with a battery to provide it, but that's probably not the best solution here.
Comment: @AndyBonner - I guess they're immediately recognisable as dual purpose sockets if you've seen a lot of them in your time. I remember thinking they were pretty snazzy first time I ever saw one… 20? years ago. I've still to this day never owned anything with them.
Comment: (Dude the XLR is on the front!  it's a "combo" one). Hold it - don't you need a lift with that Rode ?
Answer: The Rode needs phantom.
The Saffire can supply phantom, on XLR.
Nothing is built to run phantom along a TS/TRS jack, because the chances of shorting the socket or accidentally sending it to something that will be destroyed by 48v are far too high to risk it.
Therefore, phantom runs on XLR > XLR only.
All you need is the right cable [already supplied in the kit you linked]. The XLR>TRS in your separate link is of no use whatsoever.
The input sockets [1] on the Saffire will accept either jack [probably mono TS, not TRS] or XLR. Because of the voltage concerns mentioned, it will not send phantom up the jack line, only the XLR.
Comment: Thank you! I didn't know it was both a TRS AND a XLR input haha
Comment: @FeaturingTimm At the risk of sounding like a smartass, I've owned a Saffire interface and you *really* need to read the manual to find some of the features. Not just the dual jack/XLR connectors.
Comment: @Tetsujin - don't you need a lift with that Rode??  (Or does that interface have a cheap preamp built in or such ??)
Comment: @Fattie - these USB interfaces are all 'everything in one box'. The only mic I know that people seem to always put a lifter on is the SM7.
Comment: @Tetsujin - gotchya. modern times :)
Answer: Yes it's true that a 1/4" input will not provide phantom power. We want it this way because phantom power on 1/4" cables is dangerous for the power supply and devices. A 1/4" connector can be momentarily shorted when plugging or unplugging it.
An XLR maintains each contact at a fixed distance and cannot be plugged in incorrectly, so there's no danger of a short on XLR cables.
As mentioned elsewhere, you can actually use XLR cables with your interface. That's because your interface has combo connectors that can accept either 1/4" or XLR cables on the same connector. They don't exactly look like either kind of connector, so it's reasonable to have missed the XLR capability. Simply get and XLR cable and you should be fine.
Comment: +1 for giving a reason *why* XLR plug is that shape. I never thought about that - don't delete!
Comment: Thank you so much! That was exactly my issue! I didn't recognize the inputs as XLR because i was able to fit my 1/4 into it haha
Comment: Incidentally, my RØDE NT5s even have the ground pin a little longer so it makes contact first.  That helps if you plug into a channel that already has P48 applied.


Views Data Export print feed icon in template file

Question: I created a template for my view (views-view-fields-xxx.tpl.php).  On my view page I use the 'Views Data Export Module' to export filtered views in XLS format.
I cannot figure out how to display the button on the top of the page.  I tried printing it out in my template file using :
<?php print $view->feed_icon ?>

but then it prints out the button on every field view.  Is there a way to print this field only once? Or another way to move the "feed-icon" to the top of my page?
Answer: Solution from: https://www.drupal.org/node/2117433#comment-9422881

Copy the views-view.tpl.php file from views into your theme (or its
templates directory if it has one). Note that you can rename the
file in certain ways to make it specific to an individual view or
display. The "Theme: Information" section of the views UI will give
you more information. Also see the blog post for more information on
that.
Clear the cache so the site starts using your new template file
instead of the default views one. The "Theme: Information" part of
the views UI should now indicate that your file is being used.
In your theme's version of the template file move the feed icon
section wherever you want it. Potentially you would put it under the
exposed filters or something. Currently the the feed icon part of
the template looks like this (move this whole section):

    <?php if ($feed_icon): ?>

        <?php print $feed_icon; ?>

    <?php endif; ?>

Comment: Thanks @Jhilke. I've got one question.  Why does it work in my 'views-view.tpl.php' template, but not in my 'views-view-fields--xxx.tpl.php' template?
Comment: template files are bit tricky sometimes. you just need to try with different combinations... there is a documentation about how to use it, in drupal.org I guess...
Answer: Another alternative (if you can't override the views-view.html.twig) would be to use a preprocess function:
// mysite.module

/**
 * Implements hook_preprocess_views_view().
 */
function mysite_preprocess_views_view(array &$variables) {
  // Move feed icon to the top of the result, instead of buttom.
  if (isset($variables['feed_icons'])) {
    $variables['attachment_before'] = array_merge($variables['attachment_before'], $variables['feed_icons']);
    unset($variables['feed_icons']);
  }
}


Negative factor-level correlations between factors, which don't show at the manifest level

Question: I am developing a scale which consists of six scale dimensions, six items per dimension, with hypothesized positive relationships between all factors. All scale dimensions have positive correlations between each other. However, in an EFA of the scale which gives a working six-factor solution (based on Kaiser's K1 rule and scree plot; principal axis factoring, direct oblimin rotation with deltas =0), even though almost all items load well on their "expected" factors, one of the factors correlates negatively to other factors at the factor-level correlations. Let's call this scale dimension "dimension A" below for clarity. So, at the manifest level, "dimension A" correlates positively with all other dimensions, but at latent level it correlates negatively to other factors.
I further tested "dimension A" in separate EFAs with all other scale dimensions, looking for whether the two-factor solutions all worked. Turns out that in one of these EFAs, there are two factors where on the first factor all items from both scale dimensions load positively in the .7 to .8 range, and on the second factor the items from "factor A" load positively and items from the other factor load negatively in the .3 to .5/ -.3 to -.5 range. There is no clear theoretical explanation for this, so I was wondering if someone had an idea what might be happening here. Low sample size (n=100) could affect the results.
Comment: @MiguelIC Welcome to the site! At the manifest level, I am guessing you scale scored each factor, e.g. (item1 + item2 + ... item6)/6. How large exactly are correlations between A and other factors at the scale-scored manifest level? Also, how large are the actual respective negative correlations?
Comment: @PsychometStats Thank you! Yes, you guessed right. At the manifest level, correlations between A and other factors are .5 to .7 so quite high (rather similar as other correlations between factors). Whereas the factor-level negative correlations between A and other factors are -.3 to -.5.
Comment: thank you for providing more information. Couple more questions. What software are you using? Also, do you observe negative variance for any of the items or factors? Bonus question: are you confident with Confirmatory Factor Analysis?
Comment: I am using SPSS (version 25). Unfortunately I am not quite sure how to compute the factor-level variances with SPSS, are these commonly used also in EFA like in CFA? The item communalities are all >.5. I have a little bit of experience with CFA, and I will try to implement that at least later on. Starting with an exploratory approach seemed reasonable to me, since the studied phenomenon is new (although I have an expected structure for the scale in mind, so in that sense a CFA could also work).
Comment: that's great! So the suggestion for trying CFA would be to see if CFA-based latent correlations also give you a trouble, i.e. A is still negatively correlated with other factors. Admittedly, it is perplexing why this is happening
Comment: @PsychometStats I ran a CFA of the model in AMOS. The fit indices are not very good, but strangely there is no trace of this problem in the CFA (all correlations between factors, including A, are clearly positive, and there are no negative item variances either).
Comment: that's exactly what I suspected! This is a good indicator that at least all correlations are in the same direction. The lack of fit is not good for obvious reasons. I think your safe bet would be to report mean-based scale correlations since you are sure about the maths behind it. So report EFA results as usual and for correlations,  report observed scale-scored ones
Comment: Yes, that sounds very reasonable. Thanks again for the help!
Answer: Factor analysis solutions can be indeterminate with respect to sign, because covariances between items are built out of products of loadings and factor covariances. With a product of multiple parameter estimates, you can flip the signs on many parameter estimates but end up at exactly the same end point.
So a solution with one factor having negative covariances and negative loadings will show the same item covariances as a solution where the factor covariance is positive and all loadings are positive.
It is a matter of starting values--if you start parameter estimates on one side of 0, they will tend to stay on that side, unless the solution really calls for something different. If the software allows, specify a positive starting value for one of the loadings for that factor. That may resolve the situation.


Loading and reading JSON file is very slow after writing in python

Question: I'm trying to read in a JSON file, determine how many words are in the "text" field, add that information as a new field, "length", and write the new JSON object to a file.  I've done that with the following code:
import json

with open("file_read.json", "r") as review_file, open(
    "file_write.json", "w") as review_write:
    for line in review_file:
        review_object = json.loads(line)
        review_object["length"] = len(review_object["text"].split())
        json.dump(review_object, review_write)

The original file is over 200mb, but I can view it alright with Vim; however, the file I write which is only 3mb larger takes a very long time to load if it loads at all.  Furthermore, even if I read only the first JSON object, there are issues.  I tried the following after writing the file:
with open("file_write", "r") as review_file:
    print review_file.readline()
    print("abcd123")

I'm using Vim with python-mode, and when I traverse the first printed statement with the JSON info it is very choppy, but the second statement is not.
Comment: @user62117 So what is your question? You want to speedup your code? Why reading the 2nd file is slower?
Comment: Uh, is every single line of your input file an entire JSON object?
Answer: The way you are writing your file, you will have only one HUGE line.
# example
json.dump([1,2,3], fp)
json.dump({"name": "abc"}, fp)
json.dump(33, fp)
# content of file
# [1, 2, 3]{"name": "abc"}33

This may explain why it is so slow to read: it will have to load ~200mb of text in one time. Also loading it as json will probably fail.
To solve it you can use instead:
fp.write(json.dumps(review_object) + "\n")

Comment: This was exactly the problem.  Now it makes sense why print was affected as well.
Comment: Is there any equivalent way to read fast?


How to tell if a remote file path corresponds to one filter

Question: Currently, I have obtained these paths in a remote folder.
/folder 1/subfolder 2/filea.jpg

/folder 1/subfolder 2/fileb.pdf

/folder 1/subfolder 3/filea.jpg

And i have settled filters to decide whether I should download a file in the path list, based on the windows file explorer, e.g.
/folder ?/subfolder ?/*.jpg

/*/*/abc.*

/*/*/*.*

What is the best way to achieve a decision whether download it or not?
Thanks
Comment: What are you trying to do exactly? Are you trying to select all .jpg files in a directory?
Comment: No it is just an example, it could be any kind of file.
Answer: One way to do this would be to use a combination of the Directory.EnumerateDirectories() and Directory.GetFiles() methods, both of which take in a string searchPattern argument.
Note: To use the Directory class, you need to use the System.IO namespace at the top of your file:
using System.IO;

I believe you will have to enumerate each directory separately, since a backslash character is illegal in a file or directory name.
For example, you could have a method that takes a rootPath (the topmost directory you're searching under) and returns all "*.jpg" files that reside in a directory named "subfolder ?" (where the ? is a placeholder for a single character) only if the subfolder is directly under a directory named "folder ?":
///
/// Searches for files using the pattern "/folder ?/subfolder ?/*.jpg"
///
/// <param name="rootPath">The directory in which to begin the search</param>
/// <returns>A list of file paths that meet the criteria</returns>
public static List<string> GetDownloadableFiles(string rootPath)
{
    var files = new List<string>();

    // First find all the directories that match 'folder ?', anywhere under 'rootPath'
    foreach (var directory in Directory.EnumerateDirectories(rootPath, "folder ?",
        SearchOption.AllDirectories))
    {
        // Now find all directories directly under 'folder ?' named 'subfolder ?'
        foreach (var subDir in Directory.EnumerateDirectories(directory, "subfolder ?"))
        {
            // And add the file path for all '*.jpg' files to our list
            files.AddRange(Directory.GetFiles(subDir, "*.jpg"));
        }
    }

    return files;
}

In use, you would do something like:
List<string> downloadableFiles = GetDownloadableFiles(@"\\server\share");

With that method completed that addresses your specific example, we can see now how we could make this more generic, so the client can pass in any sort of search string.
If we say that the delimiter character in our search string is a forward slash (/), then we can let the user pass in the string and we can split it on that character. This will give us an array of directory search patterns and a file search pattern (the last item).
Once we do that, we can loop over the search patterns, getting directories that match the current pattern and storing them in a temporary list on each iteration, until we get to the last part (which is the file search pattern), where we get the files that match that pattern.
For example:
///
/// Searches for files using the pattern defined in 'searchPattern'
/// Example search patterns: "/project ?/photoshoot ?/flowers ?/*.jpg"
///                          "/project ?/plans ?/*.pdf"
///                          "/project ?/*.txt"
///
/// <param name="rootPath">The directory in which to begin the search</param>
/// <param name="searchPattern">The directory and file search pattern</param>
/// <returns>A list of file paths that meet the criteria</returns>
public static List<string> GetDownloadableFiles(string rootPath, string searchPattern)
{
    if (!Directory.Exists(rootPath))
        throw new DirectoryNotFoundException(nameof(rootPath));
    if (searchPattern == null)
        throw new ArgumentNullException(nameof(searchPattern));

    var files = new List<string>();
    var searchParts = searchPattern.Split(new[] {'/'},
        StringSplitOptions.RemoveEmptyEntries);

    // This will hold the list of directories to search, and
    // will be updated on each iteration of our loop below.
    // We start with just one item: the 'rootPath'
    var searchFolders = new List<string> {rootPath};

    for (int i = 0; i < searchParts.Length; i++)
    {
        var subFolders = new List<string>();

        foreach (var searchFolder in searchFolders)
        {
            // If we're at the last item, it's the file pattern, so add files
            if (i == searchParts.Length - 1)
            {
                files.AddRange(Directory.GetFiles(searchFolder, searchParts[i]));
            }
            // Otherwise, add the sub directories for this pattern
            else
            {
                subFolders.AddRange(Directory.GetDirectories(searchFolder,
                    searchParts[i]));
            }
        }

        // Reset our search folders to use the list from the latest pattern
        searchFolders = subFolders;
    }

    return files;
}

Example usage:
var files = GetDownloadableFiles(@"\\server\share", "/folder ?/subfolder ?/*.jpg");

Comment: The thing is that the paths are remote paths, so the system can't check locally for files.

Besides it could be any other type of filter, given by the user, so I need to look for .pdf or .pnd, It depends all on the filter.
Comment: The example I showed uses a remote path. Did this not work for your remote path? Also, you could parameterize the filters on the method to make it more generic. I'll post an example of that.
Comment: Added an example that takes the root path to search and a string that represents the search pattern.


Sort Elastic search result with best _score in Aggregation

Question: I am trying to do an aggregation and then sort the best scored result in top. Even though aggregation is successful, yet its not score sorted. How to make it possible?
<code>   "aggs": {
            "group": {
                "terms": {
                    "field": "description.keyword",
                    "order": {
                        "description_score": "desc"
                    }
                },
                "aggs": {
                    "group_docs": {
                        "top_hits": {
                            "size": 1
                        }
                    }
                }
            },

        }
</code>
Comment: @Lupanoide Can you share your views?
Comment: As just said, this is not possible because terms aggregation doesn't return any score. It's a kind of groupBy in SQL. It group all values of a certain field, there is no score. You need to change strategy. Please explain better what is your goal. What do you need to be scored? According to which criteria?
Answer: As @Lupanoide mentioned scores are not calculated in aggregation so you cannot sort your bucket on document score. Instead field collapsing can be used

Allows to collapse search results based on field values. The
  collapsing is done by selecting only the top sorted document per
  collapse key

<code>{
  "query": {
    "match_all": {}
  },
  "collapse": { --> group by on given field
    "field": "sources.keyword"
  }
}
</code>
By default search results are sorted on "score" so you don't need to add "sort"


Application of Fubini theorem to a proof of the coarea formula, or why the product of $\mathcal H^{n-m}$ with $\mathcal L^m$ equals $\mathcal L^n$

Question: While reading the proof of the coarea formula in Evans and Gariepy's book, Measure Theory and Fine Properties of Functions, I stumbled upon the following affirmation

Let $A\subseteq\mathbb R^n$ be $\mathcal L^n$-measurable, and let $P:\mathbb R^n\rightarrow\mathbb R^m$ be the orthogonal projection upon the first $m$ variables (we are supposing $n\geq m$). Then, for each $y\in\mathbb R^m$, $P^{-1}(y)$ is an $(n-m)$-dimensional affine space, a translate of $P^{-1}(0)$. Thus by Fubini's theorem,
$$y\mapsto\mathcal H^{n-m}(A\cap P^{-1}(y))\text{ is }\mathcal{L}^m\text{-measurable}$$
and
$$\int_{\mathbb R^m}\mathcal H^{n-m}(y\cap P^{-1}(y))dy=\mathcal L^n(A)$$

I understand the first application, for each $P^{-1}(y)$ divides the whole $\mathbb R^n$ into parallel $(n-m)$-dimensional slices, but I don't know how to derive the second equation. The righthand side should be equal to
$$(\mathcal H^{n-m}\times\mathcal L^m)(A)$$
rather than $\mathcal L^n(A)$.
Are the authors using that in $\mathbb R^{n-m}$ both $\mathcal H^{n-m}$ and $\mathcal L^m$ are equal? If so, why do we integrate the $(n-m)$-dimensional Hausdorff measure instead of the Lebesgue measure over $\mathbb R^m$? It doesn't seem clear to me why this choices are made at all. It seems somewhat arbitraty. I guess that the formula could be equally true if we change the roles of $\mathcal L$ with $\mathcal H$ and so forth, but that wouldn't led to a coarea-formula type of result... is that the case?
Thanks in advance for your answers.
Answer: You are right. They invoke $\mathcal{H}^s =\mathcal{L}^s$ on $\mathbb{R}^s$ for $s \in \mathbb{N}$. (And then the fact that Lebesgue measure splits as product of lower dimensional Lebesgue measures.)
Use of $\mathcal{H}^s$ is motivated (actually necessitated) by the fact that in the general coarea formula your preimages will not be identifiable with Euclidean spaces in any canonical way, so this notation is adopted in prediction of that and so that at the end one can compare and contrast.
Finally, you need Lebesgue measure in applying Fubini's theorem and also we like to integrate things against Lebesgue measure rather than Hausdorff measure, although ultimately they coincide.
Comment: Ok, your answer is quite satisfactory. Thanks.
Comment: I have extensively studied the coarea formula, so I can refer to sources, or answer any questions you wish!


Bootstrap THEME_menu_tree() on a new menu

Question: I'm using Drupal 7 and the Bootstrap theme. In the template.php file, I am successfully theming the primary menu (main menu) to enable extra list styles with the following code.
function MYTHEME_menu_tree__primary($variables) {
  return '
' . $variables['tree'] . '
';
}

I would also like to theme a new menu (menu-banner), but I can't access it as expected, with the following code.
function MYTHEME_menu_tree__menu_banner($variables) {
  return '
' . $variables['tree'] . '
';
}

The new menu is successfully defined in MYTHEME_preprocess_page() in the template.php file.
$menu = menu_navigation_links('menu-banner');
$vars['banner_nav'] = theme('links__menu_banner', array('links' => $menu));

I'm able to render it as banner_nav in the page templates.
Is there something I'm missing, or is there something specific to Bootstrap that is different from what reported from theme_menu_tree() documentation?
Answer: This isn't related to Bootstrap. In your preprocessor you're using theme_links, which is unrelated to menus. Your "menu" isn't being output as a menu at all.
If page preprocessing is necessary, you can try:
function THEMENAME_preprocess_page(&$vars) {
  $menu = menu_tree_all_data('menu-banner');
  $vars['banner_nav'] = menu_tree_output($menu);
}

Or you can render the menu block (these are automatically generated if you have the Blocks module enabled):
function THEMENAME_preprocess_page(&$vars) {
  $block = block_load('menu', 'menu-banner');
  $vars['banner_nav'] = _block_get_renderable_array(_block_render_blocks(array($block)));
}

In page.tpl.php:
<?php print render($banner_nav); ?>

Either will work.
If you don't need to preprocess the menu, you can simply position it via the Blocks admin UI.
I hope that helps!
Comment: Thanks!, I'll give it a go.

In the meantime, I was able to get it working using the secondary navigation instead of a custom menu, which is good sort of.

My main motivation with this is to get to a place where I can re-use classes instead of having a ziggurat of rules for similarly styled things like menus that are mostly the same design.


How to export an R dataframe to a Power Query table

Question: I'm using an R script within Power Query to do some data transformations and return a scaled table.
My R code is like this:
# 'dataset'

Answer: It does seem like odd that this fails to return. A quick glance online gave this 3 minute youtube video, which uses the same method, which you are using. Further searching down a source, one may come across the Microsoft Documentation, which gives a possible reason for why there might be an issue.

When preparing and running an R script in Power BI Desktop, there are a few limitations:

Only data frames are imported, so make sure the data you want to import to Power BI is represented in a data frame

Columns that are typed as Complex and Vector are not imported, and are replaced with error values in the created table

These seem like the most obvious reasons. Betting that there is no complex columns in your dataset, I'd believe the prior is likely the reason. A quick recreation of your dataset shows that the scale functions changes your dataset into a matrix class object. This is kept by cbind, and as such output is of class matrix and not data.frame.
>dataset <- as.data.frame(abs(matrix(rnorm(1000),ncol=4)))
>class(dataset)
[1]"data.frame"
>library(dplyr)
>df_normal <- log(dataset + 1) %>%
>    select(c(2:4)) %>%
>    scale
>class(df_normal)
[1] "matrix"
>df_normal <- cbind(dataset[,1], df_normal)
>output <- df_normal
>class(output)
[1] "matrix"

A simple fix would then seem to be adding output <- as.data.frame(output), as this is in line with the documentation of powerBI. Maybe it would need a return like statement at the end. Adding a line at the end of the script simply stating output should fix this.
Edit
For clarification, I believe the following edited script (of yours) should return the data expected
# 'dataset' contém os dados de entrada neste script

library(dplyr)

df_normal <- log(dataset+1) %>%
select(c(2:4)) %>%
scale
df_normal <-cbind(dataset[,c(1)], df_normal)
output <- as.data.frame(df_normal)
#output  ##This line might be needed without the first comment

Comment: I am glad i could help. The documentation seems very sparse. I'd suggest looking at other's examples, some which are availible [here](https://community.powerbi.com/t5/R-Script-Showcase/bd-p/RVisuals). I sadly do not have enough experience with powerBI to give better pointers. That said, from the two points above, you could do any kind of calculations, and then transform the data into a `data.frame`. If one used a linear model and wanted the coefficients, you could then have a column that specifies the coefficient, such that it afterwards could be identified in powerBI directly.
Comment: Power BI is a great tool for BI (as it is supposed to be). Combining its capabilities in terms of data cleaning and presentation with R algorithms, it can be incredibly powerful, I believe (since R's interface is for the data scientist). How to get data out of R into Power BI is something I'm exploring just now, so thanks a lot again for your help!
Comment: thanks for your help! I did search the documentation, but couldn't find where "dataset" and "output" are defined.Anyway, it seems obvious now that they mean the table we just imported and the table to we want to export to Power Query. Are you aware of where I can find other relevant syntax for integrating R scripts with Power Query, by the way? Now, to answer your comments above, you're absolutely right! I didn't realyze the normalization would change the df to a matrix (nor why). But converting it back to a df, made it, thanks!


Oracle extract variable number from string

Question: I'm looking to extract a specific number out of a string of mixed alphanumeric characters that are variable in length in a query. I will need this in order to calculate a range based off that number. I'm using Oracle.
Example:
D-3-J32P232
-I need to grab the J32 at least, and most likely even the 32 out of that string. This range of numbers can change at any given time.
It could range from:
D-3-J1P232
to
D-3-J322P2342
The numbers after the second and third letters can be any number of length. Is there any way to do this?
Answer: This is simpler and gets both the numbers for the range
select  substr( REGEXP_SUBSTR('D-3-J322P2342','[A-Z][0-9]+',1,1),2),
             substr( REGEXP_SUBSTR('D-3-J322P2342','[A-Z][0-9]+',1,2),2)
from dual

Answer: REGEXP_SUBSTR could work (11g version):
SELECT REGEXP_SUBSTR('D-3-J322P2342','([A-Z]+-\d+-[A-Z]+)(\d+)',1,1,'i',2) num
  FROM dual;

A test of your sample data:
SQL> SELECT REGEXP_SUBSTR('D-3-J322P2342',''([A-Z]+-\d+-[A-Z]+)(\d+)',1,1,'i',2) num
  2    FROM dual;

NUM
---
322

SQL>

This will accept any case alpha string, followed by a dash, followed by one or more digits, followed by a dash, followed by another any case alpha string, then your number of interest.
In 10g REGEXP_REPLACE, it's bit less straightforward, as they did not add the ability to reference subexpressions until 11g:
SELECT REGEXP_SUBSTR(str,'\d+',1,1) NUM
  FROM (SELECT REGEXP_REPLACE('D-3-J322P2342','([A-Z]+-\d+-[A-Z]+)','',1,1,'i') str
          FROM dual);

Your sample data:
SQL> SELECT REGEXP_SUBSTR(str,'\d+',1,1) NUM
  2    FROM
       (SELECT REGEXP_REPLACE('D-3-J322P2342','([A-Z]+-\d+-[A-Z]+)','',1,1,'i') str
  3       FROM dual);

NUM
---
322

Answer: REGEXP_SUBSTR would do the job


Machine learning tutorials / examples on data sets larger than a terabyte

Question: I am trying to gather a list of practical ML examples / tutorials on more than a terabyte of data. I'm particularly interested in feature extraction from large data sets that involves aggregation (the actual ML algorithm could ultimately be run on a much smaller set of data). Does anyone know of any blog posts / ebooks / etc. that walk through an example on a public data set greater than a terabyte?
Comment: Look for the large hadron collider dataset plus the toolset they use. I believe they have just released another huge batch of data to the public.
Comment: have you checked this course in coursera? https://www.coursera.org/course/mmds
Comment: Check out the databricks product: https://databricks.com/

In the free cluster, you get around 6GB of memory and they have a huge Wikipedia dataset.
Answer: I do not know any such tutorial, but I ultimately would question the interest of it. I can see no case where there is a need for a terabyte of data (within the context of a tutorial, of course). It will only cause longer running times, and I believe significant "big data" results can easily be gathered from a few GBs of data.
Though, the MAWILab experiment (http://www.fukuda-lab.org/mawilab/) might provide what you need. Its captures of IP traffic taken for the last 15 years on a transpacific backbone router. There an anomaly classification run over it, with different methods, and as such can serve as a ground truth example to your needs.


TinyTds::Error: Procedure expects parameter

Question: Given the following ruby:
    sql2 = 'SELECT (CASE WHEN freq = 0 THEN \'pass\' ELSE \'fail\' END) FROM (
SELECT COUNT(*) freq FROM Grant_Award WHERE Grant_Award.org_ticker IS NULL )x;'
    pp ActiveRecord::Base.connection.select_all(sql2)

    sql1 = "SELECT (CASE WHEN freq = 0 THEN 'pass' ELSE 'fail' END) FROM ( SELECT COUNT(*) freq FROM Grant_Award WHERE Grant_Award.org_ticker IS NULL )x;"
    pp ActiveRecord::Base.connection.select_all(sql1)

    sql0 = File.readlines(File.join(BASE,'db','scripts','test','pass_or_fail.sql'))
    sql = (sql0.map {|l| l.strip}).join(' ')
    pp sql
    pp ActiveRecord::Base.connection.select_all(sql)

The above produces the following results:
#<ActiveRecord::Result:0x38b0ce0
 @column_types={},
 @columns=[""],
 @hash_rows=nil,
 @rows=[["pass"]]>

#<ActiveRecord::Result:0x38b5ae0
 @column_types={},
 @columns=[""],
 @hash_rows=nil,
 @rows=[["pass"]]>

"SELECT (CASE WHEN freq = 0 THEN 'pass' ELSE 'fail' END) FROM ( SELECT COUNT(*) freq FROM Grant_Award WHERE Grant_Award.org_ticker IS NULL )x;"

C:/Ruby193/lib/ruby/gems/1.9.1/gems/activerecord-sqlserver-adapter-4.2.2/lib/active_record/connection_adapters/sqlserver/database_statements.rb:336:in `each': TinyTds::Error: Procedure expects parameter '@statement' of type 'ntext/nchar/nvarchar'.: EXEC sp_executesql 'SELECT (CASE WHEN freq = 0 THEN ''pass'' ELSE ''fail'' END) FROM ( SELECT COUNT(*) freq FROM Grant_Award WHERE Grant_Award.org_ticker IS NULL )x;' (ActiveRecord::StatementInvalid)

The first two attempts at executing the sql return just fine.  The third one, which is read from a file into an array and then joined, bombs.  I know this must have something to do with the treatment of strings.  But I cannot figure this out.
What am I doing wrong?  Why does it (tinytds or activerecord?) think this is a stored procedure?
What I want to do is have a whole bunch of sql scripts in a folder and then execute them one by one.
Details: ruby 1.9.3, activerecord (4.2.0), and tiny_tds (0.6.2-x86-mingw32)
Comment: What's the x at the end of the statement doing? That looks wrong to me.
Answer: I woke up this morning and realized it must have to do with the character set of the file on disk.  The following is a solution works [without changing the query]:
ActiveRecord::Base.connection.select_all(sql0.join(" ").encode("UTF-8"))

I forced the character-set to utf-8 when submitting it, and now everything executes fine.
Proof:
#<ActiveRecord::Result:0x3bede68
 @column_types={},
 @columns=[""],
 @hash_rows=nil,
 @rows=[["pass"]]>

#<ActiveRecord::Result:0x3bf2c68
 @column_types={},
 @columns=[""],
 @hash_rows=nil,
 @rows=[["pass"]]>

"SELECT (CASE WHEN freq = 0 THEN 'pass' ELSE 'fail' END) FROM ( SELECT COUNT(*) freq FROM Grant_Award WHERE Grant_Award.org_ticker IS NULL )x;"

#<ActiveRecord::Result:0x3bf5410
 @column_types={},
 @columns=[""],
 @hash_rows=nil,
 @rows=[["pass"]]>

Answer: I think the statement generated by the file import should be:
SELECT (CASE WHEN freq = 0 THEN 'pass' ELSE 'fail' END) FROM ( SELECT COUNT(*) freq FROM Grant_Award WHERE Grant_Award.org_ticker IS NULL );"

I think the x at the end of the statement is causing the error and wonder if this has slipped into your other lines by mistake.
Comment: TSql requires a label for the subquery.  Sql Server will throw an error without the label.  I use a lower case "x" to give it a visually low profile and to not distract from the rest of the code, given the label is functionally useless in this query.
Comment: Ah! I'd normally put an AS between the subquery and the x.
Comment: I'm learning little bits here and there, as I have not had to deal with code pages before.  The string in the code, sql2 or sql1, is encoded US-ASCII.  The string read from the file, sql0, is encoded IBM437.  I'm not sure if IBM437 is typical, but that is the way my laptop is configured.  When I force the code page to UTF-8, all is good with the world.
Comment: I tried adding "encoding: utf-8" to my database.yml, hoping activerecord/tinytds would pick that up and do the conversion internally, but no dice.  I have to explicitly force the code change.
Comment: One nit-picky comment to add.  Can the -1 be removed from my question?  I really think the problem and solution is noteworthy, and I don't want someone to ignore this post when they might receive some benefit from it.
Comment: I agree. The need to force encoding isn't obvious IMO.


If $x>0$ is a real number, show that $\lim_{n\rightarrow{\infty}}x^{1/n}=1$.

Question: Suppose $x>0$ is a real number. I have to prove that $\lim_{n\rightarrow{\infty}}x^{1/n}=1$, and I'm working in the context of sequences, not continuous functions. Here's what I've done so far: there are two cases, $x\geq{1}$ and $0<x<1$. In the first case, it is possible to show that $(x^{1/n})_{n=1}^{\infty}$ is a decreasing sequence, and moreover is bounded from below by $1$, so we know it converges to a real number and furthermore, that: $$\lim_{n\rightarrow{\infty}}x^{1/n}=\inf(x^{1/n})_{n=1}^{\infty}$$
The problem then becomes to verify that the infimum of the sequence is indeed $1$. We already know that $1$ is a lower bound, so the task becomes to show that $1$ is the greatest lower bound. I have tried to employ a proof by contradiction of assuming we could find some lower bound $L$ such that $L>1$, but I'm not allowed to use logarithms so I'm stuck at this point (a similar argument to the above would establish for the case $0<x<1$ that $\lim_{n\rightarrow{\infty}}x^{1/n}=\sup(x^{1/n})_{n=1}^{\infty}$, etc, but I'm stuck for the same reason there). Any suggestions on how to move forward would be appreciated (in case you're curious, this is Exercise 6.5.3 in Tao's Analysis 1).
Note there are some duplicate questions, but all of the methods which were used to prove them are "too advanced" for my purposes. It needs be as elementary as possible, appealing to the most basic properties of suprema, infima, and possibly limit superior and limit inferior, etc.
Comment: It is a similar question, for the case $0 1$ we have $x = [1 + (x^{1/n} - 1)]^n > 1 + n(x^{1/n}-1)$ by Bernoulli's inequality which implies that $0 < x^{1/n}-1 < (x-1)/n \to 0$.  Does Bernoulli's inequality qualify as elementary? The case $0 < x < 1$ can be handled similarly with $x^{1/n} = 1/(1 +a)$ where $a > 0$.
Comment: No, not elementary enough.
Comment: $0\leq L=\sup_n x^{1/n}$, so $x^{2/(2n)}=x^{1/(2n)} x^{1/(2n)}$ and both factors are $m$ be integers and let $x>1$ be a real number. Then $0<{1/n}<{1/m}$, and I had elsewhere as a lemma that if $x>1$ and $q,r$ are rationals, then $x^{q}>x^{r}$ if and only if $q>r$.
Comment: @RRL The reason I'm saying Bernoulli's inequality is "not elementary enough" is that I just don't think Tao would expect a reader who's never seen it before to have it as some sort of sudden insight, and nor has it been introduced in the book thus far.
Comment: @user10354138: just to be clear: your comment assumes $x<1$, right?
Answer: Suppose $x^{1/n}$ tends to some $L>1$. Then $L^2>L$, so there exists $N$ such that $x^{1/n}<L^2$ whenever $n\ge N$. But $x^\frac{1}{2n}=\sqrt{x^{1/n}}$, which is less than $L$ if $n\ge N$. This contradicts the assumption that $x^{1/n}$ tends monotonically to $L$ from above.
And for the case $0<x<1$, just apply the above to $1/x$.
Comment: Your solution is pretty good, but I have to confess I liked user10354138's solution a little bit better! Nonetheless, I'll accept it as the best answer.


How do I detect state change in multivariate time series?

Question: I have a multivariate time series . For each row in the data we have the values of inputs and a label for stability (0 or 1 ) . What are the algorithms that can detect the stability for an unlabelled time series, using this historical data. The sample data looks like this

Is this a machine learning problem or a time series classification problem.What are the methods that can be used and how?
Answer: Your use case is similar to a hidden-markov-model (HMM) but not quite since your state names are known and your training data is already labelled.
Based on your labelled dataset, you can learn not only the transition matrix between states (stable/unstable) but also the observation probabilities per state. In other words, given [speed1=3, speed2=2, Pressure=1, acceleration=0.7], are you more likely to be in a stable or unstable state? You learn this based on MLE on your labelled set.
Then, when it comes to labelling the unlabelled series, you know your initial state probabilities (stable / unstable), your [2*2] transition matrix and the observations (and therefore likely state) at each time point.
Since you have multiple observed features (emissions), you can implement something similar to a Dynamic Naive Bayes solution, which is a generalisation over HMM's.  Or you can use an HMM and model your emission as a multivariate Gaussian.
You can then run a Viterbi algorithm to find the best sequence of states. You can start with looking at Rubiner, HMM paper.
There are implementations of HMM in R, Matlab, Python, etc. I am sure you will find these easily upon Googling.
Comment: Thanks. Any example about how this goes , I have used markov chain in isolation , but never used them with multivariate time series for classification
Comment: Based on your labelled dataset, you learn not only the transition matrix between states (stable/unstable) but also the observation probabilities per state. In other words, given [speed1=3, speed2=2, Pressure=1, acceleration=0.7], are you more likely to be in a stable or unstable state? You learn this based on MLE on your labelled set. Then, when it comes to labelling the unlabelled series, you know what your initial states is, and the transition matrices and the observations (and therefore likely) state at each time point.
Comment: You can then run a Viterbi algorithm to find the best sequence of states. Start with looking at `Rubiner, HMM` paper. Your use case is not strictly HMM but similar.


Modify an array passed as a method-parameter

Question: Suppose I have an int-array and I want to modify it. I know that I cannot assign a new array to array passed as parameter:
public static void main(String[] args)
{
    int[] temp_array = {1};
    method(temp_array);
    System.out.println(temp_array[0]); // prints 1
}
public static void method(int[] n)
{
    n = new int[]{2};
}

while I can modify it:
public static void main(String[] args)
{
    int[] temp_array = {1};
    method(temp_array);
    System.out.println(temp_array[0]); // prints 2
}
public static void method(int[] n)
{
    n[0] = 2;
}

Then, I tried to assign an arbitrary array to the array passed as parameter using clone():
public static void main(String[] args)
{
    int[] temp_array = {1};
    method(temp_array);
    System.out.println(temp_array[0]); // prints 1 ?!
}
public static void method(int[] n)
{
    int[] temp = new int[]{2};
    n = temp.clone();
}

Now, I wonder why it prints 1 in last example while I'm just copying the array with clone() which it's just copying the value not the reference. Could you please explain that for me?

EDIT: Is there a way to copy an array to object without changing the reference? I mean to make last example printing 2.
Answer: In your method
public static void method(int[] n)

n is another name for the array that way passed in. It points to the same place in memory as the original, which is an array of ints. If you change one of the values stored in that array, all names that point to it will see the change.
However, in the actual method
public static void method(int[] n) {
    int[] temp = new int[]{2};
    n = temp.clone();
}

You're creating a new array and then saying "the name 'n' now points at this, other array, not the one that was passed in". In effect, the name 'n' is no longer a name for the array that was passed in.
Answer: Your examples 1 and 3 are virtually the same in context of the question - you are trying to assign a new value to n (which is a reference to an array passed by value).
The fact that you cloned temp array doesn't matter - all it did was create a copy of temp and then assign it to n.
In order to copy values into array passed into your method method you might want to look at:System.arraycopy
It all, of course, depends on the sizes of your n array and the one you create inside method method.
Assuming they both have the same length, for example, you would do it like that:
public static void main(String[] args)
{
    int[] temp_array = {1};
    method(temp_array);
    System.out.println(temp_array[0]);
}
public static void method(int[] n)
{
    int[] temp = new int[]{2};
    System.arraycopy(temp, 0, n, 0, n.length);
    // or System.arraycopy(temp, 0, n, 0, temp.length) -
    // since we assumed that n and temp are of the same length
}

Answer: As you correctly note, you cannot assign to the array reference passed as a parameter.  (Or more precisely, the assignment won't have any effect in the caller.)
This is about the best that you can do:
public static void method(int[] n) {
    int[] temp = new int[]{2};
    for (int i = 0; i < temp.length; i++) {
        n[i] = temp[i];
    }
    // ... or the equivalent using System.arraycopy(...) or some such
}

Of course, this only works properly if the size of the input array is the same as the size of the array you are copying to it.  (How you should deal with this will be application specific ...)

For the record Java passes the array reference by value.  It doesn't pass the array contents by value.  And clone won't help to solve this problem.  (At least, not with the method signature as declared.)
Comment: then what does `clone()` do exactly?
Comment: It just creates a copy of the object.  But the point is that you don't need to create a copy of the array.  Rather you need to copy its contents.
Answer: In your method method, nothing that you assign to n will ever change the value of the object passed in and assigned to n.  At the beginning of method, n points to an array.  When you assign n to equal another array, you've simply re-pointed which array n points to, and haven't changed anything about temp_array from the main method.


Statistical test for continuous predictor variable, multiple binomial response

Question: I have a dataset with hundreds of individual trees. These individuals were from seven sites that demonstrated different rates of land cover change over time (% change in area over 30 years). I'm looking to test how the rate of land cover change influences the growth trend of trees (outcomes for each individual could be positive, negative, or neutral - these are each columns in my dataset).
A reviewer suggested that logistic regression would be most appropriate for this data, but I have not been able to find a method of performing a multiple logistic regression of some sort that works for multiple binomial predictors. I could run individual logistic regressions for each outcome, however this strikes me as potentially inflating error. Does anyone have suggestions as to a test that could be used for this sort of data, or would it be best to use something like a bonferroni correction?
I'm working in R, if you have any specific suggestions for resources in the R programming environment!
Thanks in advance.
Answer: I assume that a single tree could have either a positive, neutral, or negative growth trend. So the categories are mutually exclusive. This sounds like either a multinomial logit or ordered logit. The ordered logit is a more parsimoneous model, but that parsimony comes with some strong assumptions that may not be met. You could also consider a generalized ordered logit, as a sometimes feasible middle ground between the ordered logit and multinomial logit.


How can I view my log files via ssh?

Question: I currently cannot access http://example.com/admin/reports/dblog as I cannot authenticate as an admin as my page is getting a 500 error.
I wonder if there is an actual location on my production server where Drupal errors are logged.
Answer: If you have drush set up, you can use drush watchdog-list to look at the items logged to watchdog.
If you don't have access to drush, but your DB is up and running fine, you can always query the watchdog table with Select * from watchdog;
From personal experience, you might want to check your apache error logs as well to see if those hold anything that can help.

How can I view my log files via ssh?


API Instagram obter informações sobre uma determinada publicação

Question: É possível ter acesso às mídias (claro que públicas) sem ter a autorização do usuário responsável por ela? A ideia é saber o número de curtidas e comentários de uma determinada publicação via API.
Uma das alternativas que encontrei em alguns posts é carregar o html da publicação exemplo: https://www.instagram.com/p/tsxp1hhQTG/ E manualmente buscar o número de curtidas e comentários. Essa alternativa até resolveria o meu problema, porém estou implantando essa solução em um sistema onde vários usuários fariam essa requisição, que talvez seja um tanto quanto "estressante" para o servidor e correria o risco de desconfiarem de DDoS.
Já criei uma "Aplicação Cliente" no Instagram, possuo o Client ID, Client Secret e o ACCESS TOKEN.
Comment: Você quer obter as mídias através de quê? Hashtag? Por usuário?
Comment: @Randrade por media-id ou shortcode.
Answer: O Instagram agora está com o modo seguro (SandBox Mode).
Esse modo começou a ser implantados para todos os clientes no dia 01 de junho de 2016.
Antes dele, você conseguia obter imagens públicas através de consultas com seu access_token, agora isso não é mais possível.
Para obter esses dados utlizando o access_token, sua aplicação deve ser submetida ao Instagram para sair do modo SandBox e ir par ao modo online.
Mas então, o que é SandBox Mode?
Cada novo aplicativo criado na plataforma Instagram inicia no modo Sandbox. Este é um ambiente totalmente funcional que permite que você teste terminais de API publicamente disponíveis antes de enviar seu aplicativo para revisão. O modo seguro é ideal para desenvolvedores que são novos para a plataforma Instagram e querem explorar a plataforma API, bem como para as equipes que precisam de vários clientes para o desenvolvimento, teste e outros ambientes não-vivos.
Para ajudá-lo a desenvolver e testar seu aplicativo, os usuários e os meios disponíveis no modo Sandbox são dados reais Instagram (ou seja, o que é normalmente visível no aplicativo Instagram), mas com as seguintes condições:

Apps na caixa de areia são restritas a 10 usuários
Os dados são restritas para os utilizadores 10 e 20 os meios mais
recentes a partir de cada um desses utilizadores
API reduzidos limites de taxa

O que isso interfere para mim?
Isso é simples meu caro amigo.
Enquanto seu aplicativo estiver no SandBox Mode, você só conseguirá interagir com quem estiver configurado junto com você. Ou seja, não terá acesso as fotos e demais informações dos restantes dos usuários.
Existe outra forma de obter os dados?
Você pode baixar o HTML, como dito da pergunta. Mas o Instagram pode te bloquear com isso. Mas a melhor forma é realmente submeter seu aplicativo para a aprovação.
Explicação no SOen.
Comment: Mas você tendo seu App publicado, você possui diversas informações, como mostra a documentação
Comment: Então se o Instagram aprovar minha aplicação, isso significa que poderei ter acesso as midias (likes, comentarios, etc) sem necessitar da autorização do usuário?
Comment: @LeonardoRocha Você conseguirá ter acesso as coisas públicas. Para ter acesso as coisas do usuário, ele ainda precisa ser autenticado
Comment: Todas as postagens são públicas?
Se um usuário for público, para que eu possa obter o número de seguidores, ainda sim preciso da autenticação dele?
Comment: @LeonardoRocha Infelizmente sim. Eu odiei essa modificação, mas traz mais segurança para os usuários, e com isso, muito menos requisições para a API do Instagram.


Python regex HTML

Question: I am going crazy over this, i hope someone can help me.
I am trying to regex this url: https://www.reddit.com/r/spacex/?count=50&after=t3_xxxxxxx where the x are numbers and letters.
The url is from an HTML file:
https://www.reddit.com/r/spacex/?count=25&after=t3_319905
I tried this:
re.search(r'(<a href=")(https://www.reddit.com/r/spacex/?count=25.+?)(")', subreddit).group(2)

but i keep getting NoneType' object has no attribute 'group'.
Comment: I would recommend looking into a scraper like [Beautiful Soup](http://www.crummy.com/software/BeautifulSoup/).
Comment: yes yes i know, but for this i want to use regex. I am trying to learn why my regex is not working.
Comment: you have plenty of characters in there that have special meanings in regular expressions ... they need escaping
Comment: First extract (with beautiful soup as recommended) urls you are interested by using an XPath query to filter urls that begin with `https://www.reddit.com/r/spacex/?count=25` and after extract with a regex (or an url parser) the part of the url you want.
Comment: @user2369869 Hi there, I'm /u/EchoLogic, one of the mods of /r/SpaceX. What are you trying to accomplish? I may already have done whatever you're trying to do.
Comment: @EchoLogic It has nothing specifically do to with spacex, just that i frequent the subreddit and so i used it as an example :p Basically i am trying to get a url of each page of a subreddit.
Comment: @user2369869 No worries! Take a look at [PRAW](https://praw.readthedocs.org/en/v2.1.21/), you'll find it a lot easier to grab the page urls of a subreddit by using Reddit's API directly than scraping it with regex like this. You'll find the task much nicer to complete that way :)
Answer: Use an HTML Parser, like BeautifulSoup. It provides you a way to specify a regular expression to match an attribute value:
soup.find_all('a', href=re.compile("after=t3_\w+"))

Working example:
import re
from bs4 import BeautifulSoup
import requests

url = "https://www.reddit.com/r/spacex/?count=25&after=t3_319905"
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36'}
response = requests.get(url, headers=headers)

soup = BeautifulSoup(response.content)

print soup.find_all('a', href=re.compile("after=t3_\w+"))

Also see the must-provide link for regex+HTML questions:

RegEx match open tags except XHTML self-contained tags
Answer: ? is  a special character in regex which makes the previous token as optional. You need to escape ? in the regex in-order to match a literal ? character. You need to escape the dots also but not the one in .+?.
re.search(r'(<a href=")(https://www\.reddit\.com/r/spacex/\?count=25.+?)(")', subreddit).group(2)
                                                          ^
                                                          |

Extra capturing groups are unnecessary here. Just a single capturing group would be enough.
re.search(r'<a href="(https://www\.reddit\.com/r/spacex/\?count=25.+?)"', subreddit).group(1)

Comment: what is the difference between .+? and .+ or similarly .*? and .*
Comment: `.*` matches any character (_except line breaks_) zero or more times. `.+` matches any character one or more times. Also see [this](http://stackoverflow.com/questions/2301285/what-do-lazy-and-greedy-mean-in-the-context-of-regular-expressions) question to learn about lazy and greedy.
Comment: yeah but what makes it different when you add the ? after * or +

Python regex HTML


Is the function 1 - x a bijection from [0,1] to [0,1)

Question: I want to prove the set [0,1] is equinumerous to [0,1) I came up with 1 - x
I can prove it's bijective, but I was curious if this might be too simple of a function.
Comment: Why do you think it's a bijection? Let's start with that and work our way through...
Comment: Ok here is what I came up with.         it gets mapped to itself if x doesn't equal 1/2^n and if x does equal 1/2^n then the function becomes 1/2^(n+1) that way it works for 1, and it works for 1/2 etc and keeps going.
Comment: $x\mapsto1-x$ is not a function from $[0,1]$ to $[0,1)$.
Comment: What value does it take for $x=0$?
Comment: You don't need to give an explicit bijection to show that they are equinumerous.
Answer: The function is not well defined at $0$, so it cannot be a bijection. At $0$, the value of the function should be $f(0)=1$, however this is absurd as $1\notin [0,1)$.
To prove find a real bijection, I would advise you to simply start with $f(x)=x$, and then "fix" the problem of $f(1)$ so the function maps $1$ to some other number $x_1$ in $[0,1)$. Then, fix the problem of $f(x_1)$, and repeat.
Comment: In particular, you could simply use the sequence of powers of 2; $f(2^{-n}) = 2^{-n-1}$, and otherwise $f(x) = x$.
Comment: Ok here is what I came up with. it gets mapped to itself if x doesn't equal 1/2^n and if x does equal 1/2^n then the function becomes 1/2^(n+1) that way it works for 1, and it works for 1/2 etc and keeps going. –


How to access values from textfield in javascript

Question: I have a text_field inside a nested loop.I want to access the value inside all the text fields by using javascript. Here is my html view:
<% @empsInProject.each do |empInProject| %>
<%= empInProject.name %>
<% (1..@numberOfMonths).each do |one| %>
<input type="text" id="text1" class="text2">
<% end %>
<% end %>
<button value="check" id="button">Calculate</button>

I used the code given below to access the value but it returns the first text field value only.
Here is my javascript code:

Please help me..
Thanks in advance
Answer: $('.text2').each(function(){
     $(this).val();
})

User this jQuery function to get each value of the input in your form. This will only work if all your inputs have the class "text2".
Answer: HTML id should be unique, and $('#text1') will only give you first element with id text1.
class is for multiple selection.
$("button").click(function() {
   var val = $('.text2').map(function() {
     return $(this).val();
   }).join(',');
   alert(val);
});

Answer: get the text fields by calling document.getElementsbyClassName("text2")
it will return a list of input fields then get the values


Ridge regression parameter estimation in R

Question: I have seen about 16 parameter estimation methods in different research papers. How can these methods be implemented in R language using Monte Carlo simulation technique.
Comment: Just some clarifying questions which may hopefully make your question more specific (and hence get a better answer).  1) Are you talking about estimating the ridging parameter? 2) why do you want Monte Carlo simulations as opposed to other techniques(e.g. optimisation,MCMC)? 3) are you looking to account for the noise/error variance in your estimation method (i.e. the fact that the "ridged" betas won't exactly predict the response $y$)?
Answer: The current go to tool for performing ridge regression in R (in my estimation) is the glmnet package. Default behavior is lasso regression: for ridge regression, set alpha=0 when you call the glmnet or cv.glmnet functions for your regression. Setting alpha to any other value between 0 and 1 gives an elastic net penalty.
Additionally, base R has the lm.ridge function.
UPDATE:
If you want to homebrew your own ridge regression, you should probably start by picking which of the 16 parameter estimation methods you've come across you want to work with. Beyond that advice, your question is extremely broad and I don't really know what you want to hear.
A very simple way you could implement ridge regression would be to do a bayesian linear regression with normal priors centered at 0. Otherwise you will probably need to apply an optimization technique to the RSS loss function augmented with regularization penalty. I'm guessing the parameter estimation techniques you have read about largely pertain to this optimization problem.
Comment: Could you suggest why you prefer glmnet for ridge regression (out of the several packages that provide it)? (Bizarrely, ??"ridge" does not pull up glmnet.) It's common with R to have several packages offering more or less the same thing, and there's no systematic source of comparison info. (The r-sig-mixed-models FAQ does this for nlme, lme4, etc, but it seems to be a one-off.)
Comment: They're all just tools. I probably prefer glmnet because I learned it early (so I'm familiar with it), see people using it a lot (so there's some social confirmation that it's a good tool), and I'm also probably more likely to do a lasso regression anyway, so if I decide to change my model from lasso to ridge (or elastic net) instead of rebuilding my code to accommodate a different package, I literally just change a single parameter in my model.
Comment: initially I want to make function myself for ridge regression instead of using R packages.
I also need some learning material about ridge, although I have but not enough I feel.
Comment: @itfeature.com, if you want to "build you own", you just have to create a piece of software that minimizes $$ \sum_{i=1}^{n} (y_i - x_i {\boldsymbol \beta})^2 + \lambda \sum_{j=1}^{p} \beta_{j}^2 $$ for a given data set and input value, $\lambda$. So you need to a) calculate this criterion function - for this look into the syntax for user defined functions - and b) optimize it as a function of ${\boldsymbol \beta} = \{\beta_1, ..., \beta_p \}$. For this you can either "build you own" again or use a built-in `R` routine for optimization, such as the `optim` function. Hope this helps.
Comment: +1 for glmnet.  It's a generalization of both lasso and ridge regression, that can do either one or a combination of both.


Transform a probability distribution to a special "reversed form"

Question: Suppose we have a distribution $p(x_i),i\in[1,N]$. I want to transform it into a "reversed" distribution $q(x_i)$. By reverse, I mean if $p(x_i)$ is high, then $q(x_i)$ is low. I think it can be formally expressed as:
$$\begin{aligned}
\sum_{i=1}^N p(x_i)&=1\\
q(x_i)&=f(p(x_i))\\
\sum_{i=1}^N q(x_i)&=1\\
p(x_a)\ge p(x_b) &\Rightarrow q(x_a)\le q(x_b)
\end{aligned}$$
Is there any existing transform of $f$. If not, how can I find the $f$?
Comment: Interesting question. Besides normalizing afterwards, this makes me wonder if there's such thing as the discrete analogue of volume-preserving map. The "volume" here as per the question statement is one-dimensional, but of course generalization to 2-dim or 3-dim would be also of great practical use.
Answer: I am not aware if there are "usual" such transformation but you can build one quite easily as follow: reverse the probability (without caring about the sum equal 1). and then normalize to get a sum equal 1.
For example you can define $q'(x_i)=(1-p(x_i))$.
Thus $S = \sum_i (1-p(x_i))$ will be used for normalization.
And you obtain $q(x_i)=q'(x_i)/S=(1-p(x_i))/S$ that fulfil your conditions.


Transpose of two vectors

Question: Given vectors $a,b\in\mathbb{R}^n$, is it true that $ab^\intercal=ba^\intercal$?  I have tried to decompose this matrix, but am having trouble with keeping track of the columns and rows.
Comment: Did you try any numerical examples?
Comment: Compare the column spaces of the two matrices. Those should be easy to figure out.
Answer: Assuming they are column vectors, they are transposes of each other.
$(ab^T)^T = (b^T)^T(a)^T = ba^T$
or, looking at the product element-wise,
$[ab^T]_{ij} = a_ib_j$ and $[ba^T]_{ij} = b_ia_j$
then in general $a_ib_j \neq b_ia_j$ for $i \neq j$.
Comment: @M.Winter I did the same thing!
Comment: True, but how does that show that they are or are not equal?
Answer: Assuming $a,b\;$are interpreted as column vectors, the answer is "no", not necessarily.

Try almost any numerical example.

For example, let
$$
a=\begin{bmatrix}1\\1\end{bmatrix}
\;\;\text{and}\;\;
b=\begin{bmatrix}1\\2\end{bmatrix}
$$
On the other hand, if $a,b\;$are interpreted as row vectors, then the given expressions are both just dot products, so
$$ab^T = a\cdot b = b \cdot a = ba^T$$

Transpose of two vectors


For which values of x does the power series converge or diverge?

Question:
First, I know that the series converges when |x+2| < R and diverges when |x+2| > R. So now I have that the radius of convergence is somewhere between 2 and 3. However, this doesn't really give me too good of an indication of the interval of convergence. I would say that since 2 < R < 3 and a=2, then the interval of convergence would be 0 < x < 5, but please correct me on that if I am wrong. I'm kind of just taking a shot in the dark on that one.
So, I'm lead to believe that a and c would be divergent and b and d would be convergent. I just am in need of more insight on this because I'm trying to explain it to someone else but it has been years since I learned the topic so I'm very unsure of my knowledge.
Answer: The power series must be convergent when $x$ satisfies that $-R <x+2< R$, and be divergent
when $x <-R$ and $x>R$ and be unkown when $x+2=-R$ or $x+2=R$, where $2\le R \le 3$.
So (a) is convergent; $b$ is divergent; $c$ and $d$ are unknown.
Comment: Just a question to make sure I've got this correct. So when x=-3, x+2=-1 which falls between -2 and 2 which is the smallest possible interval of convergence so it converges. Then when x=2, x+2=4 which falls outside of -3 to 3 which is the largest possible interval of convergence so diverges. Then when x=-4, we have x+2=-2 which is a problem because R is possibly equal to -2. And lastly, if x=1, x+2=3 and since R is possibly equal to 3 we also have a problem.


Angular Service - return private value versus assignment by reference to private value

Question: Discussing with a coworker the difference between getting a return value of a private array versus an assignment to a variable that is a reference to the private array , but couldn't think of a good reason why the following would yield different results:
Controller
app.controller("MyController", ["MyService", function(MyService) {
    var ctrl = this;
    ctrl.arrayAssign = [];

    ctrl.doSomething = function() {
        // Empty out array, usually assigned at one point, code is omitted for example
        ctrl.arrayAssign = []

        // This does not work, ctrl.arrayAssign = []
        ctrl.arrayAssign = MyService.serviceArray;

        // This works, ctrl.arrayAssign correctly assigned to serviceArray in MyService
        ctrl.arrayAssign = MyService.getServiceArray();
    }
}]);

Service
app.service("MyService", [function(){
    var service = this;
    var serviceArray = [];

    service.serviceArray = serviceArray;

    // This is called, either from controller above, or another controller
    service.assignToServiceArray = function(arr) {
        serviceArray = angular.copy(arr);
    };

    service.getServiceArray = function() {
        return serviceArray;
    };

    return service;
}]);

Note: ctrl.doSomething() is called after service.assignToServiceArray()
What is a good explanation to why the two controller assignments above have different results as described?
I thought service.serviceArray would expose the private array serviceArray and just become a reference to the private array (service.serviceArray = serviceArray), so a new assignment to serviceArray (i.e. from service.assignToServiceArray), should allow service.serviceArray to reference the new array assignment which can be accessed directly by a controller.
service.getServiceArray() would also expose the private variable serviceArray, but directly, as in I'm not trying to access a variable that is just a reference to the array (i.e. service.serviceArray above).
Would trying to assign a reference to a private variable not be ideal in this case and just returning a private value be favored? I'm probably missing something really obvious...
Answer: When you call assignToServiceArray you assign serviceArray the local variable to point to a new array you don't update the reference or update the original array so service.serviceArray is still pointing at the empty array you started it off with... if you instead did angular.copy(arr,serviceArray) without the assignment you'd get the result where the actual array remains the same (same array object) but it's contents get emptied and repopulated in which case both service.serviceArray and serviceArray would have the updated array.  If I missed the question let me know.
Comment: gotcha! Figured that service.serviceArray was still referencing the old array from the initialization, wasn't too sure about how  the copy assigns a completely different array to serviceArray
Comment: Cool yeah slightly better formatted and clearer explanation from estus as well.
Answer: service.serviceArray = serviceArray assigns a reference to [] array to serviceArray property, not a reference to serviceArray variable.
When a new value is assigned to serviceArray, it is assigned to this variable and nothing else.
A good thing about angular.copy is that it accepts two arguments and allows to keep the reference to the object:

If a destination is provided, all of its elements (for arrays) or properties (for objects) are deleted and then all elements/properties from the source are copied to it.

So it would work with:
angular.copy(arr, serviceArray);

The service in fact misuses OOP features that are provided by JS. service service accepts a constructor function. There's no problems like this if it behaves like normal object constructor:
app.service("MyService", [function(){
    this.serviceArray = [];

    service.assignToServiceArray = function(arr) {
        this.serviceArray = arr;
    };

    service.getServiceArray = function() {
        return this.serviceArray;
    };

    // not needed
    // return service;
}]);

If MyService.serviceArray is supposed to be used after its reassignment, it should be referred everywhere only by its getter and setter methods. The actual reason for getters and setters in OOP is that the value is being kept private and never accessed from the outside directly. E.g. in view:
{{ vm.MyService.getServiceArray()[0] }}

Comment: quick question, in service.assignToServiceArray, does this.serviceArray become a reference to arr? in general javascript, using a constructor like this would make a reference to the passed in array.  in your experience, would you want a reference (if possible) or a copy of that array in your service? just seemed to me you'd want a copy in your service because passing references from different controllers could get messy, er unless you're careful and know what you're doing :P
Comment: Yes, it becomes a reference. It always depends if it should be copied or assigned. But I cannot inagine a scenario where several controllers should assign an array like that. Any way, there's nothing really bad with it, it is tidy and clean if it is covered with tests. If this array is used for ng-repeat, this won't cause a problem (maybe less performant than angular.copy but not sure about this).
Comment: makes sense, thanks for your input!  i was just thinking of the classic angular question "how do i share data between controllers" but i figure you would want each controller to reference (or even copy) the service's array and configure the setter to not reference the controller array
Comment: There's more than one right answer to the question, in the end it always depends on the case. I often end up with solutions that wouldn't be my first choice in vitro. Feel free to post a more specific question if you will have problems with this approach.