Which data preprocessing steps do I need to perform on which data subset?

Question: I want to make predictions using several supervised Machine Learning algorithms and apply 10-fold-cross validation. For doing so, I randomly divided my dataset into in-sample and out-of-sample sets. Moreover, I randomly subdivided the in-sample dataset into training and test set for being able to perform cross-validation. I want to perform a regression task and the predictors include both, numerical and categorical predictors.
As I am quite new to the field of Machine Learning, I am not sure on which subsets I need to perform the following data preprocessing steps.

K-nearest-neighbor imputation of missing numerical data
Outlier elimination
Log transforming variables with a right skew (on which subset do I need to determine the skew?)
Feature selection using filter methods (This includes determining the Pearson correlation or Chi-square of predictors with other predictors and with the target variable as well as determining the variance of predictors)
Further dataset descriptions such as histograms, mean, median, variance, ...

I am aware that I am trying to prevent a look-ahead bias. However, I am not sure whether I need to perform these steps only on the training subset? Or on the entire in-sample set? Or some of them even on the entire dataset?
Comment: Consider that the test set should be used in testing only in order to give you an idea as close as possible of the reality. Cleaning fit, feature selection or other transformation using historical data need to use the training set only.
Comment: whenever you perform any analysis the first step is to divide the data into in-sample and out-sample, so that you does not add any bias in the test dataset from the training
Comment: @Mayeulsgc But how can I do these things with 10 different training sets? See my question in the answer by frank.
Answer: You have to make sure that your preprocessing of the training set doesn't use information from the validation or test set ("leakage"). E.g., in your k-nearest neighbor imputation, you must not include data from the validation or test set in this k-NN method.
Similarly for the outlier elimination, transformation, feature selection, ...
All of that could potentially leak information from the validation or test set into the model.
Note that leakage can sometimes be very subtle. So, to be on the save side, repeat all those preprocessing anew with each training set.
Comment: @moby1209 The ten-fold cross-validation is only for finding the best model. Once you have it, you throw away all the inferior models. Then you take the complete in-sample dataset and apply again preprocessing and training (and finally test it on the single test set).
Comment: I don't understand: If you cannot train without leakage, then either don't do it or live with the knowledge that your results might not generalize well.
Comment: Yes. Unless you are sure that it does not leak information into the training set, which is often hard to tell.
Comment: That sounds reasonable. However, in my paper, I also need to report statistics such as histograms, etc. But this is  a problem when I have 10 different test sets. Is there a common solution to this?
Comment: Moreover, how can I then make decisions regarding log transformations, feature selection, etc. when I have 10 different training sets?
Comment: @moby1209 But in general, if you need to compare histograms of ten different datasets, I would suggest side-by-side [violin plots](https://seaborn.pydata.org/generated/seaborn.violinplot.html).
Comment: Alright, I get that point. However, I still do not understand how I decide which variables I want to log transform, because I have 10 different training sets that I could use to make this decision. The same question remains for all decisions I have to make based on the training set, such as fetaure selection. How do I do this if I have multiple training sets?
Comment: The problem is that I have to make these decisions up front. Another related problem is that making the decision up front also implies that I will use part of the data that will later be used as a test set, leading to data leakage.
Comment: OK, so you mean that I need to make the decision for feature selection, log transformation individually for each iteration?
Comment: Thanks for your help. Now I only need to find out which statistics of the dataset to report in my paper and think about whether ten different statistics are really interesting in that case. I would tend to show the statistics of the entire in-sample set and mention that these are not the values on which the decisions for feature selection, etc. were made. Do you have any ideas on that?


Create Issue Jira REST API (400 Bad Request)

Question: The Problem
I am trying to create a REST API call using a HttpWebRequest to our in-house Jira server. But somehow I keep getting back a (400) Bad Request error. I have also tried with WebClient and other ways but I just don't seem to find the correct approach. Any suggestions?
URL is correct
User is correct
Password is correct
JSON Data also correct
There must be another way of accessing the remote server right? I have been searching but not seem to find a solution.
My Code
public static void CreateJiraRequest(JiraApiObject.RootObject jiraApiObject)
{
    string url = "https://jira-test.ch.*********.net/rest/api/latest/issue/";
    string user = "peno.ch";
    string password = "**********";

    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Method = "POST";
    request.ContentType = "application/json";
    request.Credentials = new System.Net.NetworkCredential(user, password);

    string data = JsonConvert.SerializeObject(jiraApiObject);

    using (var webStream = request.GetRequestStream())
    using (var requestWriter = new StreamWriter(webStream, System.Text.Encoding.ASCII))
    {
        requestWriter.Write(data);
    }

    try
    {
        var webResponse = request.GetResponse();
        using (var responseReader = new StreamReader(webResponse.GetResponseStream()))
        {
            string response = responseReader.ReadToEnd();
        }
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

JSON
{
    "fields": {
       "project":
       {
          "key": "FOO"
       },
       "summary": "Test the REST API",
       "issuetype": {
          "name": "Task"
       }
   }
}

Exception
The exception occurs when entering the try block on request.GetResponse();
Additional information: The remote server returned an error: (400) Bad Request.
Visit the Jira Wiki here
Comment: Read the error response, it's bound to contain more details. Your assumption of the JSON being correct is probably incorrect. For example your user doesn't have permissions, your data is missing a required field, or something like that. Without any relevant data or error we can't say anything about this.
Comment: There's no use guessing based on the input; the output contains the details necessary to solve this. Your input may be syntactically correct, but the server may require more fields to be set or disallows your user to create issues of the given type in the given project. Obtain the response body and read it to discover why you get a 400 response.
Comment: Try examining the innerException, I've found that a useful message is usually buried somewhere deep in the stack
Comment: listen keyboard warrior the in the Try block the function throws an exception the only error thrown there is: The remote server returned an error: (400) Bad Request. without any further Information. I have also compared my generated JSON with the one listed on the wiki page they match 1:1
Comment: Look buddy you're using the ancient HttpWebRequest which is a horrific HTTP API to use. See for example [.Net HttpWebRequest.GetResponse() raises exception when http status code 400 (bad request) is returned](https://stackoverflow.com/questions/692342/net-httpwebrequest-getresponse-raises-exception-when-http-status-code-400-ba) to obtain the response body which should contain more information (as explained [in the docs](https://docs.atlassian.com/software/jira/docs/api/REST/7.12.0/#api/2/issue-createIssue)). Or switch to HttpClient altogether.
Comment: And sure, your JSON may match an example JSON, but your particular JIRA installation may be configured to require more fields than are present in that example, or other rules that that JSON doesn't adhere to.
Comment: I have checked with our Jira Administrator. According to him, my JSON is correct. I could add more fields but they would not be required. I'll edit the post and add my current JSON.
Comment: @socalcheesehead Yes i have tried that but the inner exception was null
Answer: #Solution#
The problem in the code above is that Jira requires encoded credentials. Without encoding the credentials the Jira server will return a 400 Bad Request error with no specific information.
I have written two new functions one for the API request and one for the Encoding  of the credentials.
#API Call#
public static string PostJsonRequest(string endpoint, string userid, string password, string json)
    {
        // Create string to hold JSON response
        string jsonResponse = string.Empty;

        using (var client = new WebClient())
        {
            try
            {
                client.Encoding = System.Text.Encoding.UTF8;
                client.Headers.Set("Authorization", "Basic " + GetEncodedCredentials(userid, password));
                client.Headers.Add("Content-Type: application/json");
                client.Headers.Add("Accept", "application/json");
                var uri = new Uri(endpoint);
                var response = client.UploadString(uri, "POST", json);
                jsonResponse = response;
            }
            catch (WebException ex)
            {
                // Http Error
                if (ex.Status == WebExceptionStatus.ProtocolError)
                {
                    HttpWebResponse wrsp = (HttpWebResponse)ex.Response;
                    var statusCode = (int)wrsp.StatusCode;
                    var msg = wrsp.StatusDescription;
                    throw new HttpException(statusCode, msg);
                }
                else
                {
                    throw new HttpException(500, ex.Message);
                }
            }
        }

        return jsonResponse;
    }

#Encoding Function#
private static string GetEncodedCredentials(string userid, string password)
{
    string mergedCredentials = string.Format("{0}:{1}", userid, password);
    byte[] byteCredentials = UTF8Encoding.UTF8.GetBytes(mergedCredentials);
    return Convert.ToBase64String(byteCredentials);
}

Additional Notes:
Jira API is case sensitive so for "POST" If you do Fields, Summary, Project it won't work
it has to be fields, summary, project
Answer: I had a very similar problem by creating an issue with a webhook in Jira. The problem was that I was not able to tell why I am getting a "Bad request" from Jira while the request looked OK. The solution was to enbale the logging on the right level.
In order to debug the request it was necessary to go

System -> Logging and Profiling -> HTTP Access logging [Enable] and
HTTP dump log [Enable].

In addition

Configure logging level for another packages -> and insert
org.apache.http with DEBUG level and if you are using a webhook also
add com.atlassian.webhooks with DEBUG level.

Then it is very handy to install the "LASTLOG-ADD-ON" which is then available in the

Manage apps

and you can view and search the logs. The error message, which showed me why my issue creating request failed, was in "atlassian-jira-http-dump.log".
Hope that this setup will help in debugging.


SSH connection falling down

Question: I've set up a connection with autossh that creates some tunnels at system startup,
but if I try to connect, after successful login (with RSA key)  connection fall down,
here a trace:
debug1: Authentication succeeded (publickey).

debug1: Remote connections from LOCALHOST:5006 forwarded to local address localhost:22
debug1: Remote connections from LOCALHOST:6006 forwarded to local address localhost:80
debug1: channel 0: new [client-session]
debug1: Requesting firstname.lastname@example.com
debug1: Entering interactive session.

debug1: remote forward success for: listen 5006, connect localhost:22
debug1: remote forward success for: listen 6006, connect localhost:80

debug1: All remote forwarding requests processed
debug1: Sending environment.
debug1: Sending env LANG = it_IT.UTF-8
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype email@example.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 2400, received 2312 bytes, in 1.3 seconds
Bytes per second: sent 1904.2, received 1834.4
debug1: Exit status 1

What can be the problem?
All this stuff is managed by a script already running on another machine (creating reverse tunnels on the same machine but with different ports)
Answer: Since the client is sending an environment, but no command, it means the client is trying to start a shell as that user. But that shell exits immediately causing the ssh connection to terminate.
Try to ssh in as that user from the command line without going through autossh. It may provide some additional hints, and is interesting to know, if you can get to a shell that way.
The user may have been configured with the wrong shell, or access to the shell may have been overridden in ~/.ssh/authorized_keys by using command="…"
Comment: ok, it was my fault, I wrongly used -n instead of -N so it was actually trying create a shell session (while shell of user was /bin/false). Since I only wanted to create a tunnel, -N was the right flag
Answer: Try add the KeepAlive options in /etc/ssh/ssh_config file (or ~/.ssh/config if you wish apply to single user)
For example:
     ClientAliveInterval 300
     ClientAliveCountMax 2

If a client, and
Host *
     ServerAliveInterval 300
     ServerAliveCountMax 2

if a server
Comment: The `ssh` command exits after just 1.3 seconds. It couldn't have been idle for 300 seconds already.

SSH connection falling down


Adding items to sub-array of an array doesn't work

Question: I have this class:
ActiveCampaign
{
     public Campaign campaign;
     public IEnumerable<CampaignView> Views;
}

ac is type of ActiveCampaign[]
The following code executes without any errors but nothing is added to its views. What amd I missing here?
ac.FirstOrDefault(a => a.Campaign.Id == c.Id).Views.ToList().Add(cv);

Answer: When you call ToList() you're creating a separate list; you're not adding to ac.  You need to assign the return value:
List<ActiveCampaign> newList = ac.FirstOrDefault(a => a.Campaign.Id == c.Id).Views.ToList();

and then you can use it...
newList.Add(cv);

Comment: I get an error `Cannot implicitly convert type 'void' to 'System.Collections.Generic.List'`
Answer: You're creating a new list that is a copy of the Views property, then adding to it, then throwing that list away.
If you want to permanently affect Views then make it a List or IList type and add to it directly.
If you want to see the views plus that new item then get the list into a variable before the add, or use Concat (or even better Append though it's currently only available in dotnet core) to create an enumeration of the views items along with the addition.


Error C2403 using inline x86 C++ assembly moving register contents to variable

Question: I've worked with the inline assembler before, but I'm getting an error I haven't seen before and can't seem to find around the interwebs. The closest I've gotten is this or this but neither have helped. The error is when moving the contents of the EAX register to a variable dx. It seems to me it should just be the below, and in fact disassembly generates the same code as I wrote, but I get two errors:
    int main(int argc, char * argv[]){
         auto x1 = 10, x2 = 2, dx = 0;
         __asm{
              MOV EAX, DWORD PTR [x1]
              SUB EAX, DWORD PTR [x2]
              MOV DWORD PTR [dx], EAX
              ;error C2403: 'dx': register must be base/index in 'first operand'
              ;error C2415: improper operand type

              ;Disassembly generated the following
              mov eax,dword ptr[x1]
              sub eax,dword ptr[x2]
              mov dword ptr[dx],eax
        }
    }

I don't think I need to use a different register and I thought that MOV could move contents of a register into a variable like this. Oddly, the following line is exactly the same but using y1, y2, and dy, and generates no error. I've tried just using [dx] to the same result, and by now I can only think of something like MMX instructions, which seems unnecessary. Thanks for your help in advance!
Side-note: Is there a preference to all-caps versus lowercase when writing in x86 (or any) assembly language?
[Edit] Now I'm curious, how can it generate assembly code using dword ptr [dx] if my using of it confused the compiler?
Answer: DX is a 16-bit x86 general purpose register in its own right and the assembler is getting confused by your syntax (hence 'register must be base/index in first operand'--what would, for example, dword ptr [ax] mean?). Choose a different name for your variable.
Comment: Oh, you're right, I forgot about that. Thank you for saving me from another hour of head-scratching!


PCA for Feature Selection and Calculating the Total Contributions of Each Features

Question: First of all, I know that using PCA for feature selection is not a true approach however, I have found some articles which uses PCA for feature selection and I want to imitate them. I am having some troubles to get the real logic behind these articles. Here you can find the links of those aforementioned articles below;
https://personal.utdallas.edu/~jiezhang/Journals/JIE_2017_AE_short_term_wind_forecasting.pdf
https://doi.org/10.1016/j.neucom.2014.09.090
Let's assume that I have five different variables (features) for forecasting the outcome. These are wind speed, temperature, humidity, pressure and wind directions.
dat.sample = data.frame(windspeed = rnorm(100, mean = 10, sd = 2),
                        temp = rnorm(100, mean = 20, sd = 2),
                        humid = rnorm(100, mean = 80, sd = 5),
                        press = rnorm(100, mean = 950, sd = 10),
                        winddir = rnorm(100,mean = 180, sd = 5))

Now lets scale and center the data to ensure that having standart deviation of 1 and mean of 0 for each variable.
library(caret)
preproc = preProcess(dat.sample, method = c("center","scale"))
dat.sample.cs = predict(preproc, dat.sample)

#Ensuring the standart deviation is 1 and mean 0 before proceeding with PCA.
apply(dat.sample.cs, 2, function(x) {c(sd(x),round(mean(x),3))})

PCA is applied to the scaled and centered data with base R function prcomp. After applying PCA, in order to get the eigenvalues of each principal component (PC) and the contribution of each variable for each PC, factoextra library is used.
library(factoextra)

get_eigenvalue(pca)

      eigenvalue variance.percent cumulative.variance.percent
Dim.1  1.2263264         24.52653                    24.52653
Dim.2  1.1581302         23.16260                    47.68913
Dim.3  0.9905302         19.81060                    67.49974
Dim.4  0.8372833         16.74567                    84.24540
Dim.5  0.7877299         15.75460                   100.00000

It is found that while the PC1 represents the ~24.5 percent of the total variance of whole PCs, other variance percents can also be seen. Now, I would like to see the contribution of each variable to the each PCs.
pca.var = get_pca_var(pca)
(contrib = pca.var$contrib)

               Dim.1       Dim.2      Dim.3     Dim.4      Dim.5
windspeed  5.2483398  0.71103782 91.6450906  1.535375  0.8601568
temp      39.4126852  8.99578568  0.9641489  8.337931 42.2894495
humid     43.0894490  0.03033891  1.1556426 42.025891 13.6986782
press      0.1220664 55.14999755  0.2033069 14.220377 30.3042517
winddir   12.1274594 35.11284004  6.0318111 33.880426 12.8474639

Now, it is clear to say that while the maximum contribution to the PC1 comes from the humidity; pressure, wind speed, humidity and temperature are the variables which contributes most from PC2 to PC5 respectively (contributions have directly been associated with the importance of the features). Here comes my questions;

Assuming that the first PCs are enough for representing the data; how can a feature selection be made by using this information? Is it okay to use the first variables which contributes most for each PCs within selected first n (n is chosen 4 here) PCs? For instance, in the example above, should the humidity, pressure, wind speed and humidity features be chosen? It also means that having only 3 variables since humidity is selected two times.

How can I obtain the total contributions of each variable like the article which is cited above. Since we obtained the importance of each feature for each individual PCs, how can I get the total contributions for each feature? Is it okay to take weighted average for each feature across the whole PCs or the selected PCs (which are 1:4 in this example)? The weights will be the variance percent of each PC. In summary; I would like to get a table like in this article and it can be seen below.

The thing that confuses me in this table is the name of the table. It states that; Contribution rates of principal components. Here, each feature is mentioned as principal component. I do not know if they have used the contributions from only PC1 or whole PCs and it is not completely clear how they calculated this table in the article. Here you can find the related section of the PCA method for feature selection in the article.

Am I missing a thing? How can I obtain a table like that?
Answer: Since this was too long for a comment, I will post it as an answer, though perhaps not a proper answer. As far as I understand, by using PCA you will lose the connection to the original variables. What you end up with are new features that best explain the total variance.
Each of these new features is a linear combination of the original variables. CR_i and CC_i are measuring how much each of these new features are contributing to the variance. Maybe they are doing something like looking at the weights of each of the original variables in the linear combinations that give PC1, PC2 and so on. There are not enough details to know I guess.
By the way, your toy example is not a good use case for PCA. After rescaling and centering, the data columns are nearly i.i.d. draws from a standard normal. All these variables will be nearly orthogonal, hence PCA will just give you the coordinate axes more or less (all the components will contribute equally to the variance as well and they are aligned with the original variables roughly). In fact, in the large-sample limit, the PCs would not be individually identifiable.


Wrong User Image Display - Weird Twig Cache?

Question: I'm having a weird issue with incorrect user images being displayed and it seems to be a caching issue or perhaps I am passing the user image incorrectly to my template.
Problem: If I login to a different user account I will still see the image from my account. If I rebuild the cache and then refresh the browser, I will receive the correct user image.
Am I building the user image incorrectly?
mytheme.theme
 <?php
  use Drupal\user\Entity\User;

 function gary_preprocess_block(&$variables) {
  $user = User::load(\Drupal::currentUser()->id());
  $file_uri = $user->get('user_picture')->entity->getFileUri();
  dpm($file_uri); //correct uri
  $styled_image_url = ImageStyle::load("user_pic")->buildUrl($file_uri);
  dpm($styled_image_url); //correct url
  $picture = $styled_image_url;
  $variables['user_pic'] = $picture;
 }

block--gary-account-menu.html.twig

The variables in the theme file are correct, but the output from twig when inspected, continues to give the wrong built url to the previous user image (until the cache is flushed).
I doubt I need to delete twig cache or flush caches anytime a user pic is changed or account is switched. Anyone know what I am doing wrong here?
Comment: Which hook/template? From where do you get $user?
Comment: @4k4 updated the question
Answer: When you add data from the current user you also have to add this cache context:
$variables['#cache']['contexts'][] = 'user';

You can also add the cache dependencies of the user, file and image style. Then modifying one of these entities will be effective immediately, see https://www.drupal.org/docs/8/api/render-api/cacheability-of-render-arrays
Comment: Great thanks. Its worth noting that I ran into weird script errors initially like "cannot read property theme of undefined in drupal.js" when I added this line to my hook_preprocess_block. This was because I wasnt targeting my block specifically, and adding this user cache context accidentally to other blocks led to errors.


Javascript mostrar varias div''s em diferentes intervalos de tempo

Question: minha dúvida é sobre como fazer várias elementos aparecerem com classes e tempos diferentes. A idéia é fazer como a interação de Live do Facebook. O Código é o seguinte:

:)

:o

:(

:/

:-

:*

Pela lógica, o Javascript (ou Jquery) teria que printar esses 6 elementos de maneira randômica sorteando tamanho e "li", não faço idéia de como fazer isso. Eu já tenho eles em html, nao será necessário puxar o banco. Podem me ajudar?
Comment: Não sei exatamente que interação te referes, podes dar um exemplo com uma imagem?
Comment: Algo como isso: https://goo.gl/gPAxrX   (me refiro a reactionns superiores) a animação eu já tenho, os  já estão programados para quando retirados o diplay:none deles (pois as classes size1, size2 e size3 possuem display:block) eles já animam e vão em direção a esquerda. Só preciso q o javascript faça essa injeção de classes "size" em tempos diferentes.
Comment: É algo tipo ver os emoticons a passar flutuando no ecrã, na secção superior como no [exemplo deste video](https://www.cnet.com/videos/facebook-perks-up-live-video-by-copying-ideas-from-other-apps/)?
Comment: Exatamente isso. Meu Css jà faz isso quando eu mudo o diplay:none  para block, ele já pega o li e aplica animação, meu problema está no javascript intercalar isso e aplicar em tempos diferentes
Answer: Não sei se é a melhor forma de fazer, mas funcionou:

$(document).ready(function() {
  var ul = $("ul").find("li");
  var rand = function() {
    var randNumber = Math.floor((Math.random() * ul.length) + 0);
    ul.each(function() {
      $(this).hide()
    });
    $(ul[randNumber]).show();
    setTimeout(rand, randNumber * 300);
  }
  rand();

})

:)

:o

:(

:/

:-

:*

Eu criei um loop com SetTimeout() com tempo aleatório, e uma array com todos os  li que fica variando, aproveitando o mesmo número randômico e jogando no índice do array. Este numero vai de 0 até o comprimento do array, e dentro do tempo  do loop ele é multiplicado por 300, mas você pode criar outro numero aleatório para especificar o tempo máximo como desejar.
Exemplo no jsfiddle
Comment: Consertado o html, os li devem estar 'hidden' logo no início, fazendo isto com seu código inicial já é o suficiente, funcionou perfeitamente.
Comment: Mais ou menos por aí, peguei o trecho do seu código e coloquei no que eu tenho aqui. Veja que a "bolinha" precisa percorrer toda a pagina e não sumir antes da outra aparecer, elas tem q aparecer e percorrer todo a pagina até o css sumir com ela de um lado e reaparecer do outro. http://codepen.io/duarteleme/pen/MpQjZN   para ver a animação toda mude o intervalo de tempo para 1200
Comment: Ficou como você desejava?
Comment: Ficou Excelente edson mas ainda não é o que preciso como formato final, como pode ver codepen.io/duarteleme/pen/MpQjZN preciso que as "bolinhas" permaneçam na tela até sumirem, mas não uma de cada vez, preciso que ele apareça e não faça "sumir" via .hide(). Apenas faça ir aparecendo, assim consigo mostrar varias bolinhas aparecendo ao mesmo tempo, como no facebook live mesmo goo.gl/gPAxrX
Comment: fiz uma uma mudancinha no JS que funcionou:   `var ul = $(".user");
 var rand = function(){
    var randNumber  = Math.floor((Math.random() * ul.length) + 0);
    ul.each(function(){
      if($(this).offset().left < 50)
      $(this).hide()
    });
   $(ul[randNumber]).show();
    setTimeout(rand, randNumber * 900);
  }
  rand();`
Comment: aqui o link: http://codepen.io/anon/pen/aJqwNK, na primeira "remessa" vai tudo junto, depois vai do jeito que vc queria. Vou analisar o porque disso.
Comment: Edson, excelente !!!! MUITO obrigado. Sua contribuição valeu muito. Deixarei no codepen para quem precisar. http://codepen.io/duarteleme/pen/MpQjZN
Comment: Opa, Fico feliz em ajudar, Abraço


Regression in $p\gg N$ setting (predicting drug efficiency from gene expression with 30k predictors and ~30 samples)

Question: I have a dataset of 29 cell lines and the IC50 values of a test drug. I want to find a relation between the gene expression profiles of each cell line (nearly 31000 genes) and the IC50 values.
My problem is the huge number of independent variables (the genes) and the low number of samples (cell lines). I'm trying to perform a linear regression using Lasso to reduce the number of genes, dividing the samples in a train set of 14 cell lines and a test set of 15 cell lines. The division is performed by randomly sampling among the 29 samples. The problem is that Lasso is not stable and every time I train the model I get different results.
So I tried to reduce the dimensionality using PCA, but as far as I have read, PCA doesn't work well when the number of covariates is greater then the number of samples. Is this true?
Can you suggest me some kind of regression which is robust when the number of samples is low?
Comment: PCA is probably not the best way to approach your problem, but just for the record: PCA does work fine in $p>n$ situation, or at least not less "fine" than anything else.
Comment: Related question: [Regression in $p>n$ setting: how to choose regularization method (Lasso, PLS, PCR, ridge)?](http://stats.stackexchange.com/questions/108614)
Answer: Most of the 31000 genes are unlikely to differ much in expression among the cell lines (at least when appropriately normalized), so they add no information to the problem.
For a practical biological problem like this, it may help to concentrate on genes whose expression levels are relatively high on an absolute basis. That way it will be easier to validate your results on these 29 lines and then apply and test your predictions on cell lines beyond those you are now examining, for example with standard PCR instead of the expensive microarray or RNAseq methods used to examine 31000 genes at once.
Start by (a) limiting your analysis to highly expressed genes whose normalized expression levels have the greatest variance among cell lines (typically on a log scale in gene expression work) and closest relation to IC50 values, so that your intractable $p \gg n$ problem becomes a less difficult $p > n$ problem. Then (b) combine information from different genes whose expression levels co-vary among cell lines.
The "supervised principal components" method described in section 18.6 of The Elements of Statistical Learning, second edition, provides a documented way to accomplish this. Genes are rank-ordered with respect to univariate relations to IC50 values (accomplishing goal a, if you limit to highly expressed genes) and PCA is performed on a subset of genes with the highest relations to IC50s (accomplishing goal b). The number of genes included in the PCA and the number of principal components retained are chosen by cross-validation.
Comment: +1 Doesn't what you are describing (find genes with most variation; find groups of genes that are highly correlated) sound a lot like PCA? Would you still advice against PCA but in favour of such "manual" feature selection? Why?
Comment: Each of the principal components found by PCA could in principle contain contributions from all 31000 genes, which could be a computational nightmare and would be difficult (at best) to interpret functionally. The biological questions underlying this type of analysis are best analyzed by focusing on a few genes or networks of genes.
Comment: The approach that _The Elements of Statistical Learning_ calls "supervised principal components" is also possible here, and close to what I suggested: find the subset of genes whose expression levels have highest univariate correlations to IC50 levels, and restrict PCA to those genes. The number of genes included in the PCA and the number of principal components retained are chosen by cross-validation.
Comment: Interesting. So maybe sparse PCA (with very strong sparsity enforced) could be an interesting approach.
Comment: Yes, section 18.6 of *The Elements* is indeed very relevant, thanks for the pointer. Maybe you want to update your answer to add what you said in the comments.
Answer: Can I suggest you the paper robustness of lasso solutions under cross-validation variability and Stability selection (Meinshausen and Bruhlman, 2009)?
They propose stable version of the lasso estimator.
Comment: @Donbea: can you give a bibliographically  correct reference? (scientific articles often have authors, date of publications and sometimes even journals)
Comment: http://www.stats.ox.ac.uk/~meinshau/stability.pdf
Comment: Google Scholar search for "robustness of lasso solutions" returns zero hits. What paper were you referring to?
Comment: I can not find the paper anymore. Maybe I have reported a wrong name. Anyway I will say that Stability Selection should definitive be the important reference
Answer: I believe the reason you will be getting varying answers is because you have $p>>n$, i.e more variables than samples. In this situation the LASSO can only selected $n$ variable and I assume will have problems with convergence. While having no experience with dealing with this, the Elastic Net supposedly overcomes some of these issues
Comment: this link maybe helpful as well http://stats.stackexchange.com/questions/29580/model-stability-when-dealing-with-large-p-small-n-problem?rq=1
Comment: if you use cd algorithm with lasso you can select more than n variables


Tensorflow error "No module named 'tensorflow_core.estimator'"

Question: I built a new environment in conda using the following .yml file:
name: py37 channels:

conda-forge
defaults dependencies:
python=3.7.7
cartopy=0.18.0
earthengine-api=0.1.221
flake8=3.8.1
jupyterlab=2.1.2
jupyter_contrib_nbextensions=0.5.1
matplotlib=3.2.1
mypy=0.770
nb_conda=2.2.1  # optional, useful for running R in Jupyter notebooks
numpy=1.18.4
pandas=1.0.3
pillow=7.1.2
scikit-learn=0.23.0
scipy=1.4.1
seaborn=0.10.1
tqdm=4.46.0
tensorflow=1.15.0  # only supports up to Python 3.7

I am now running the following section in a .ipynb file
from pprint import pprint import os import sys

import matplotlib.pyplot as plt import numpy as np import pandas as pd import tensorflow as tf

sys.path.append('../') from preprocessing.helper import get_first_feature_map, get_feature_types from utils.geo_plot import setup_ax

I keep getting the following error:
Error in callback <bound method AutoreloadMagics.post_execute_hook of <autoreload.AutoreloadMagics object at 0x7fba286b3a50>> (for post_execute): Traceback (most recent call last):   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/core/events.py", line 89, in trigger
    func(*args, **kwargs)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 538, in post_execute_hook
    _, pymtime = self._reloader.filename_and_mtime(sys.modules[modname])   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/extensions/autoreload.py", line 184, in filename_and_mtime
    if not hasattr(module, '__file__') or module.__file__ is None:   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import   File "<frozen importlib._bootstrap>", line 983, in _find_and_load   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked   File "<frozen importlib._bootstrap_external>", line 728, in exec_module   File "<frozen importlib._bootstrap>", line 219, in
_call_with_frames_removed   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/contrib/__init__.py", line 39, in <module>
    from tensorflow.contrib import compiler   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/contrib/compiler/__init__.py", line 21, in <module>
    from tensorflow.contrib.compiler import jit   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/contrib/compiler/__init__.py", line 22, in <module>
    from tensorflow.contrib.compiler import xla   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/contrib/compiler/xla.py", line 22, in <module>
    from tensorflow.python.estimator import model_fn as model_fn_lib   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/estimator/model_fn.py", line 26, in <module>
    from tensorflow_estimator.python.estimator import model_fn   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/model_fn.py", line 29, in <module>
    from tensorflow.python.types import core ModuleNotFoundError: No module named 'tensorflow.python.types'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 2077, in showtraceback
    stb = value._render_traceback_() AttributeError: 'ModuleNotFoundError' object has no attribute '_render_traceback_'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/core/ultratb.py", line 1101, in get_records
    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/core/ultratb.py", line 248, in wrapped
    return f(*args, **kwargs)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/IPython/core/ultratb.py", line 281, in _fixed_getinnerframes
    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/inspect.py", line 1502, in getinnerframes
    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/inspect.py", line 1460, in getframeinfo
    filename = getsourcefile(frame) or getfile(frame)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/inspect.py", line 696, in getsourcefile
    if getattr(getmodule(object, filename), '__loader__', None) is not None:   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/inspect.py", line 733, in getmodule
    if ismodule(module) and hasattr(module, '__file__'):   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)   File "/Users/prernamishra/opt/miniconda3/envs/py37/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import   File "<frozen importlib._bootstrap>", line 983, in _find_and_load   File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked ModuleNotFoundError: No module named 'tensorflow_core.estimator'

Do I need to install anything else? The original yml file had - tensorflow==1.15.0 but I changed it because I am running it on a mac and that wasn't working.
Any help is appreciated!
Answer: Can you please upgrade to tensorflow 2.c [2.7] and ensure python and OS is 64 bits.
Can you also try:
pip install --upgrade pip
pip uninstall tensorflow
pip install tensorflow

If the error persist:
Create an virtual environment
cd <your project path>

Install virtualenv

pip install virtualenv

Create the virtual environment
virtualenv <envname>
Activate the env
Windows Powershell: .<envname>\Scripts\activate
Unix with Bash or zsh: source /bin/activate
Then now you install tensorflow
(<envname>) $ pip install tensorflow

And then run Hello World successfully.
*Don't forget that you need to activate or configure everytime the virtual environment jupyter, command-line, etc.


PySpark Numeric Window Group By

Question: I'd like to be able to have Spark group by a step size, as opposed to just single values. Is there anything in spark similar to PySpark 2.x's window function for numeric (non-date) values?
Something along the lines of:
sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame([10, 11, 12, 13], "integer").toDF("foo")
res = df.groupBy(window("foo", step=2, start=10)).count()

Answer: You can reuse timestamp one and express parameters in seconds. Tumbling:
from pyspark.sql.functions import col, window

df.withColumn(
    "window",
    window(
         col("foo").cast("timestamp"),
         windowDuration="2 seconds"
    ).cast("struct<start:bigint,end:bigint>")
).show()

# +---+-------+
# |foo| window|
# +---+-------+
# | 10|[10,12]|
# | 11|[10,12]|
# | 12|[12,14]|
# | 13|[12,14]|
# +---+-------+

Rolling one:
df.withColumn(
    "window",
    window(
        col("foo").cast("timestamp"),
        windowDuration="2 seconds", slideDuration="1 seconds"
     ).cast("struct<start:bigint,end:bigint>")
).show()

# +---+-------+
# |foo| window|
# +---+-------+
# | 10| [9,11]|
# | 10|[10,12]|
# | 11|[10,12]|
# | 11|[11,13]|
# | 12|[11,13]|
# | 12|[12,14]|
# | 13|[12,14]|
# | 13|[13,15]|
# +---+-------+

Using groupBy and start:
w = window(col("foo").cast("timestamp"), "2 seconds").cast("struct<start:bigint,end:bigint>")
start = w.start.alias("start")
df.groupBy(start).count().show()

+-----+-----+
|start|count|
+-----+-----+
|   10|    2|
|   12|    2|
+-----+-----+

Comment: Understood, that worked perfectly. Added the `start` call as well as using `groupBy` to the answer.
Comment: Is there a way to avoid the above casting to timestamp and back to integer? Also, is there a way to retrieve just the first value in the window range? As in, instead of seeing [9, 11], we would just see '9'.
Comment: _Is there a way to avoid the above casting to timestamp and back_ - not built in. You can check the expression and implement the same logic yourself. _is there a way to retrieve just the first value in the window range_ - `$"window.start"`.
Comment: Is there a way to do this time of grouping multiple times? E.g. group by a time window, then this numeric window? If I do that right now, I'm seeing `Multiple time window expressions would result in a cartesian product of rows, therefore they are currently not supported.`

PySpark Numeric Window Group By


Shadowing with x0vncserver - No mouse pointer

Question: I am working on a solution to shadow a user's X session in a multiuser desktop environment.  If one wants to shadow a session they use my script.  My script sets the proper XAUTHORITY environment variable for the display one wants to shadow then executes the following:
<code>x0vncserver display=:XX MaxDisconnectionTime=30 SecurityTypes=VncAuth PasswordFile=/root/.vnc/passwd
</code>
I am not able to view the shadow user's mouse cursor.  I can see them clicking around but can not see the arrow.  I am not sure if it's my environment or  and issue with x0vnc.  I am running Oracle Linux (RedHat) with Sunray software.
I tried several VNC clients and can not see the mouse on any of them.  Has anyone else using x0vnc ran into this?
Answer: I experienced the same problem just now, the VNC server is a Fedora 20 host, client is OS X Mavericks.
While this is not a way to get x0vncserver to show you a mouse cursor with any client, you can show a local cursor when using vncviewer as the client. It's also included in TigerVNC.

vncviewer DotWhenNoCursor=1 localhost

Versions
Server: tigervnc-server-1.3.0-7.fc20.x86_64 (installed from base repo)
Client: 1.2.0 (installed from Homebrew)


Adding Vue.js to my theme with Webpack not working

Question: I am trying adding Vue.js into my Drupal 8 theme with my Webpack setup (which I had already working).
It's all compiling fine with Webpack, without errors after my npm script npm run build.

In my index.js I have:
import './styles.css';
import Vue from 'vue'
import App from './App.vue'

Vue.config.productionTip = false

new Vue({
  render: h => h(App)
}).$mount('#app')

In my App.vue I have:
<template>

    <HelloWorld msg="Coming Soon: Near-Earth Objects"/>

</template>

In my ./components folder I have HelloWorld.vue
First I am trying now to render my Vue component inside header.html.twig like
<hello-world></hello-world>

My thoughts are, by the element  in page.html.twig I can show, arrange and position all my Vue.js components where I want. Is that correct?
But why is my component not showing up?
Update:
I figured out. My webpack.config.js is splitting Vue into /themes/custom/my_theme/dist/vendors~main.bundle.js. By this setting:
config.optimization = {
      splitChunks: {
        chunks: "all"
      },

What is the purpose of this? Isn't it better to compile it all into the same file main.bundle.js?
Comment: I mean, menu.html.twig will output whatever you tell it to output. If you wanted to drop a script tag there and feed it an object or array, you can. I'm not sure what any module is going to provide you here.
Comment: Is the Javascript even loaded? Have you verified all of that?
Comment: @Kevin I found something. I updated my question.
Answer: I recommend looking into the Decoupled Pages module which can make integrating Vue.js simpler.


How to save configs in BrowserTestBase?

Question: In a functional test I try to test a functionality after changing the module configuration:
public function setUp() {
  parent::setUp();
  ...
  $config = \Drupal::configFactory()->getEditable('move_file.settings');
  $config->set('vocabulary', 'tags');
  $config->save();
  ...
}

The tested functionality is located primarily in MoveFileService.php. But the service does not see the configs. When I debug the configs directly in the service, then I get:
public function __construct(ConfigFactory $config_factory, LoggerChannelFactoryInterface $logger_factory) {
    $this->config = $config_factory->get('move_file.settings');
    $this->logger = $logger_factory->get('move_file');
}

public static function create(ContainerInterface $container) {
    return new static(
    $container->get('config.factory'),
    $container->get('logger.factory')
    );
}

public function configsAreSet() {
    ...
    fwrite(STDOUT, "Settings: {$this->config->get('vocabulary')}\n");
    // Returns: Settings:
    // The configuration is null.

    $config = \Drupal::service('config.factory')->getEditable('move_file.settings');
    fwrite(STDOUT, "Settings: {$config->get('vocabulary')}\n");
    // Returns: Settings: tags
    // This returns the expected value. But the good practice is to use the dependency injection.
    ...
}

How can I save the config in the setUp() method of the functional test, so that the service can get the saved configuration, when the test is running?
Answer: I have found, that the problem was caused, that in the setUp() method of the test class, was at first used the $this->drupalCreateNode([...]) method and then were created the settings. After moving the $config->set(...) method at the beginning of the setUp() method, works everything perfect.
See:

The test class
Test results

How to save configs in BrowserTestBase?


Given three postive numbers $a,b,c$ so that $a+b+c=3$. Prove that $\sum\frac{1}{a}\geqq (\frac{1}{2}+\frac{5\sqrt{3}}{18})(\!a^{2}+b^{2}+c^{2}\!)$ .

Question: Given three postive numbers $a, b, c$ so that $a+ b+ c= 3$. Prove that
$$\sum\limits_{sym}\frac{1}{a}\geqq \left (\!\frac{1}{2}+ \frac{5\sqrt{3}}{18}\!\right )(\!a^{2}+ b^{2}+ c^{2}\!)$$
I use discriminant and uvw and find it with the best constant $5\sqrt{3}\div 18$ .
Titu lemma and Holder inequality both are unsuccessful here, so I need to the help. Thanks a real lot
Answer: Let $a+b=c=3u$, $ab+ac+bc=3v^2$ and $abc=w^3$.
Thus, we need to prove that $f(w^3)\geq0,$ where
$$f(w^3)=\frac{u^3v^2}{w^3}-\left(\frac{1}{2}+\frac{5\sqrt3}{18}\right)(3u^2-2v^2).$$
But we see that $f$ decreases, which says that it's enough to prove our inequality for a maximal value of $w^3$, which happens for equality case of two variables.
Let $b=a$ and $c=3-2a$, where $0<a<\frac{3}{2}.$
Id est, it's enough to prove that
$$\frac{2}{a}+\frac{1}{3-2a}\geq\left(\frac{1}{2}+\frac{5\sqrt3}{18}\right)(2a^2+(3-2a)^2),$$ which is smooth.
I got that the last inequality is equivalent to
$$((10+6\sqrt3)a^2-(23+13\sqrt3)a+12+8\sqrt3)(2a-3+\sqrt3)^2\geq0,$$ which is obvious.


Evaluating the sums $\sum\limits_{n=1}^\infty\frac{1}{n \binom{kn}{n}}$ with $k$ a positive integer

Question:
How to evaluate the sums $\sum\limits_{n=1}^\infty\frac{1}{n \binom{kn}{n}}$ with $k$ a positive integer?

For $k=1$, the series does not converge.
When $k=2$, I can prove that:
$$\sum_{n=1}^\infty\frac{1}{n \binom{2n}{n}}=\frac{\pi}{3\sqrt{3}}$$
Usually, this can be proven by differentiating $\sum_{n=1}^\infty \frac{x^{2n}}{n^2 \binom{2n}{n}}=2(\arcsin{\frac{x}{2}})^2$, but I have an alternative proof.

Using the result of:
$$\int_0^\infty \frac{x^ndx}{(x+1)^{y+n+1}}=\frac{1}{y \binom{y+n}{n}} \tag1$$
, which can be easily proved.
I can substitute $y=n$ to obtain
$$\int_0^\infty \frac{x^ndx}{(x+1)^{2n+1}}=\frac{1}{n \binom{2n}{n}}$$
$$\sum_{n=1}^\infty\int_0^\infty \frac{x^ndx}{(x+1)^{2n+1}}=\sum_{n=1}^\infty\frac{1}{n \binom{2n}{n}}$$
Therefore,
\begin{align}
\sum_{n=1}^\infty\frac{1}{n \binom{2n}{n}} & = \int_0^\infty \frac{xdx}{(x+1)(x^2+x+1)} \\
 & = \lim_{L\to \infty}  \frac{1}{2}\ln(x^2+x+1)-\ln(x+1)+\frac{\tan^{-1}(\frac{2x+1}{\sqrt{3}})}{\sqrt{3}}\large{|_0^L} \\
&= \frac{\pi}{3\sqrt{3}}
\end{align}

Now for $k=3$, I tried to substitute $y=2n$ into $(1)$:
$$\int_0^\infty \frac{x^ndx}{(x+1)^{3n+1}}=\frac{1}{2n \binom{3n}{n}}$$
$$\sum_{n=1}^\infty\int_0^\infty \frac{x^ndx}{(x+1)^{3n+1}}=\sum_{n=1}^\infty\frac{1}{2n \binom{3n}{n}}$$
So we can have
$$\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}}=\int_0^\infty\frac{2xdx}{(x+1)(x^3+3x^2+2x+1)}$$
However, by partial fraction $$\frac{2xdx}{(x+1)(x^3+3x^2+2x+1)}=-\frac{2}{1+x}+\frac{2x^2+4x+2}{x^3+3x^2+2x+1}$$
The left part does not seem to converge.

Feeling frustrated, Wolfram Alpha plays its part. It spits out these results:
$$\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}}=\frac{1}{3}{}_3F_2\left(\left.\begin{array}{c} 1,1,\frac{3}{2}\\ \frac{4}{3}, \frac{5}{3}  \end{array}\right| \frac{4}{27}\right)$$
$$\sum_{n=1}^\infty\frac{1}{n \binom{4n}{n}}=\frac{1}{4}{}_4F_3\left(\left.\begin{array}{c} 1,1,\frac{4}{3},\frac{5}{3}\\ \frac{5}{4}, \frac{6}{4}, \frac{7}{4}  \end{array}\right| \frac{27}{256}\right)$$
However, I am not very familiar with hypergeometric function.
The pattern suggests that $$\sum_{n=1}^\infty\frac{1}{n \binom{2n}{n}}=\frac{1}{2}{}_2F_1\left(\left.\begin{array}{c} 1,1\\ \frac{3}{2}  \end{array}\right| \frac{1}{4}\right)=\frac{\pi}{3\sqrt{3}}$$
Thus, $${}_2F_1\left(\left.\begin{array}{c} 1,1\\ \frac{3}{2}  \end{array}\right| \frac{1}{4}\right)=\frac{2\pi}{3\sqrt{3}}$$

Arriving these results, I have the following questions:

How can ${}_2F_1\left(\left.\begin{array}{c} 1,1\\ \frac{3}{2}  \end{array}\right| \frac{1}{4}\right)$ be expressed into this simple elementary form?
  How can we arrive to the result for $\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}}$ given by Wolfram Alpha?
  Ultimately, can we evaluate $\sum_{n=1}^\infty\frac{1}{n \binom{kn}{n}}$ for all integers k $\ge$ $2$?
Comment: If you have pushed your computations that far, then you are mature enough to understand that there will be no satisfactory answer to your problem. Expressing a result in terms of hypergeometric functions is just a way of hiding the dirt under the carpet, i.e. you give a name to something that you will still not know. A few years of mathematical practice should be enough to convince any intelligent person that the subset of analytically computable expressions is negligible. Most mathematical expressions (even innocent-looking ones) are, sadly, not computable.
Comment: @Did: It seems that the OP himself is the first not to understand what I've written in that comment: he has placed a bounty on this question, now... Plus 19 upvotes for the question! I believe that Martians are wondering whether there exist intelligent life on Earth...
Comment: @AlexM. In your comment, I believe you are saying that most hypergeometric functions cannot be expressed into simplier forms. Although the sum cannot be simplified into exact values, there should be a way for the computer to convert the sum into hypergeometric form. So I am looking for the process of conversation.
Comment: @McCheng This part is standard, computing the ratio of consecutive terms as $$\frac1{(n+1){3n+3\choose n+1}}=\frac1{n{3n\choose n}}\cdot\frac4{27}\frac{n(n+\tfrac12)}{(n+\tfrac23)(n+\tfrac13)}$$ hence, using Pochhammer's symbols, $$\frac1{(n+1){3n+3\choose n+1}}=3\left(\frac4{27}\right)^n\frac1{n!}\frac{(1)_n(1)_n(\tfrac12)_n}{{}{}{}{}{}{}{}(\tfrac53)_n(\tfrac43)_n},$$ from which the ${}_3F_2$ formula should be clear. Likewise, each sum $$\sum_{n=1}^\infty\frac1{n{kn\choose n}}$$ can be rewritten as a value of some ${}_kF_{k-1}$ function. **But this is a mere rewriting of the series...**
Comment: hint: you can find the zeros of the denominator by cardanos method, which enables you to perform a partial fraction decompostion and the result will be logs with ugly roots as arguments (the divergencies will cancel in the end)
Comment: Furthermore: http://functions.wolfram.com/HypergeometricFunctions/Hypergeometric3F2/
Comment: @tired Two of the roots are in complex form, which I am not good at handling.
Comment: @AlexM.'s comment hits the nail on the head and could be meditated upon with profit by numerous users of the site (unfortunately, the chances it will be are close to zero).
Comment: @Did Okay. I have deleted part of the question.
Comment: @AlexM. Yes, but how can we express the sum into that hypergeometric form?
Comment: Similar techniques are discussed by Borwein and Girgensohn in http://dx.doi.org/10.1007/s00010-005-2774-x , entitled "Evaluations of binomial series" of 2005.
Answer: The exact value of $\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}}$
We have already found that $$\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}}=\int_0^\infty\frac{2xdx}{(x+1)(x^3+3x^2+2x+1)}$$ if we can evaluate the integral, we are done.
First of all, we need to find the roots $\{l,m,n\}$ of $x^3+3x^2+2x+1$ by Cardano's method. We can obtain: $$l=-\frac{\sqrt[3]{\frac{1}{2} \left(9-\sqrt{69}\right)}}{3^{2/3}}-\sqrt[3]{\frac{2}{3 \left(9-\sqrt{69}\right)}}-1$$ $$m=\frac{\left(1+i \sqrt{3}\right) \sqrt[3]{\frac{1}{2} \left(9-\sqrt{69}\right)}}{2\ 3^{2/3}}+\frac{1-i \sqrt{3}}{2^{2/3} \sqrt[3]{3 \left(9-\sqrt{69}\right)}}-1$$ $$n=\frac{\left(1-i \sqrt{3}\right) \sqrt[3]{\frac{1}{2} \left(9-\sqrt{69}\right)}}{2\ 3^{2/3}}+\frac{1+i \sqrt{3}}{2^{2/3} \sqrt[3]{3 \left(9-\sqrt{69}\right)}}-1$$ Now we need to break down the integrand. Let $a=-l$, $b=-(m+n)$ and $c=mn$ . Then, $x^3+3x^2+2x+1=(x+a)(x^2+bx+c)$ .
By partial fraction,
\begin{align}
\frac{2x}{(x+1)(x^3+3x^2+2x+1)} & = \frac{\alpha}{1+x}+\frac{\beta}{a+x}+\frac{\gamma+\delta x}{x^2+bx+c} \\
 & = \frac{\alpha}{1+x}+\frac{\beta}{a+x}+\frac{2x\delta+b\delta}{2x^2+2bx+2c}+\frac{-b\delta+2\gamma}{2x^2+2bx+2c}
\end{align}
where $\alpha=\frac{2}{(1-b+c)(1-a)}$, $\beta=\frac{2a}{(a^2-ab+c)(1-a)}$, $\gamma=\frac{2(1+a-b)c}{(a^2-ab+c)(1-b+c)}$ and $\delta=\frac{2a-2c}{(a^2-ab+c)(1-b+c)}$.
Now we can integrate $\frac{2x}{(x+1)(x^3+3x^2+2x+1)}$ . Since $\int_0^\infty \frac{dx}{x^2+2ax+b}
=\frac{1}{\sqrt{b-a^2}}\left(\frac{\pi}{2}-\arctan\left(\frac{a}{\sqrt{b-a^2}}\right)\!\right)$, we can have
\begin{align}
\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}} & = \int_0^\infty\frac{2xdx}{(x+1)(x^3+3x^2+2x+1)} \\
& = \int_0^\infty (\frac{\alpha}{1+x}+\frac{\beta}{a+x}+\frac{2x\delta+b\delta}{2x^2+2bx+2c}+\frac{-b\delta+2\gamma}{2x^2+2bx+2c})dx
\end{align}

$$\sum_{n=1}^\infty\frac{1}{n \binom{3n}{n}}= -\beta\ln{a}-\frac{1}{2}\delta\ln{c}+(-\arctan(\frac{b}{\sqrt{-b^2+4c}})+\frac{\pi}{2})(\frac{2\gamma-b\delta}{\sqrt{-b^2+4c}})$$

The exact value of ${}_2F_1\left(\left.\begin{array}{c} 1,1\\ \frac{3}{2}  \end{array}\right| \frac{1}{4}\right)$
Since $$\frac{(\beta)_k}{(\gamma)_k}=\frac{\Gamma(\gamma)}{\Gamma(\beta)\Gamma(\gamma-\beta)}\int_0^1 t^{\beta-1+k} (1-t)^{\gamma-\beta-1}dt$$ for non-negative integer k, and by the binomial theorem, $$\sum_{k=0}^\infty \frac{(\alpha)_k}{k!}(zt)^k=(1-zt)^{-\alpha}$$ where $0 \le t \le 1$, $-1 \lt z \lt 1$, we have: $${}_2F_1\left(\left.\begin{array}{c} \alpha,\beta\\ \gamma  \end{array}\right| z\right)=\frac{\Gamma(\gamma)}{\Gamma(\beta)\Gamma(\gamma-\beta)}\int_0^1 t^{\beta-1}(1-t)^{\gamma-\beta-1}(1-zt)^{-\alpha}dt$$ So, $${}_2F_1\left(\left.\begin{array}{c} 1,1\\ \frac{3}{2}  \end{array}\right| \frac{1}{4}\right)=\frac{\Gamma(\frac{3}{2})}{\Gamma(\frac{1}{2})}\int_0^1 \frac{dt}{\sqrt{1-t}(1-t/4)}$$ Since $\Gamma(z+1)=z\Gamma(z)$ and the integral can be easily calculated, we finally obtain

$${}_2F_1\left(\left.\begin{array}{c} 1,1\\ \frac{3}{2}  \end{array}\right| \frac{1}{4}\right)=\frac{2\pi}{3\sqrt{3}}$$
Comment: The expression after the substitution is too messy. I am not able to upload it onto this site.
Comment: To get less daunting formulas, one could introduce the notations $$\omega=\sqrt[3]{\frac{3}{2} \left(9-\sqrt{69}\right)}\qquad j=\frac12(1+i\sqrt3)$$ hence $1+j+j^2=0$ and $j^3=1$, and use them to simplify the formulas, for example, unless I am mistaken, $(l,m,n)$ are $$l=-\frac13\omega-\omega^{-1}-1$$ $$m=\frac13j \omega+j^2\omega^{-1}-1$$ $$n=\frac1{3} j^2\omega+j\omega^{-1}-1$$ and so on... (In particular one sees there might be some sign mistakes in your $l$, or in your $m$ and $n$ since one wants $l+m+n=-3$...)
Comment: @Did I have checked the roots with Mathematica. Maybe I will follow your suggestion and edit my answer later.
Comment: My bad, $\frac12(1\pm i\sqrt3)$ are not third roots of unity, $\frac12(-1\pm i\sqrt3)$ are (a fact which supports your formulas). Coherent notations would be $$j=\frac12(-1+i\sqrt3)$$ hence $$\frac12(1+i\sqrt3)=-j^2\qquad\frac12(1-i\sqrt3)=-j$$
Answer: I'm late for this party but the appearance of the plastic constant's minpoly, or $x^3-x-1=0$, got me interested.
The binomial sum can be expressed as a concise finite sum of logarithms. For $k>1$,

$$\sum_{n=1}^\infty\frac1{n\binom{kn}n} =\sum_{n=1}^k \frac{\ln(1-x_n)}{k-(k-1)x_n}=\int_1^\infty\frac1{x(x^k-x+1)}$$

and where the $x_n$, naturally enough, are the roots of $x^k-x+1=0$. Example, for $k=3$,
$$A=\sum_{n=1}^\infty\frac1{n\binom{3n}n} = \frac{\ln(1-x_1)}{3-2x_1}+ \frac{\ln(1-x_2)}{3-2x_2}+ \frac{\ln(1-x_3)}{3-2x_3}=0.371216\dots$$
and the $x_n$ are the three roots of $x^3-x+1=0$, one of which is the negated plastic constant $x\approx-1.32472$. It was pointed out that equivalently,
$$3A ={_3F_2}\left(\frac32,1,1;\ \frac43,\frac53;\ \frac4{27}\right)$$
Interestingly, the plastic constant also appears in a similar generalized hypergeometric function,
$$2B ={_3F_2}\left(\frac12,1,1;\ \frac43,\frac53;\ \frac4{27}\right)$$
discussed by Reshetnikov  in this post.
Comment: very interesting ! thanks for showing
Comment: Hey, I'm late to comment, but I have to say: being late to the party is better than not making it to the party at all!
Comment: @Tito Piezas III: Hi, thanks for this answer. I was wondering whether you could help me  transform my answer (see below) to the integral representation that you gave.Thanks in advance.
Answer: For the most general case,
$\sum\limits_{n=1}^\infty\dfrac{1}{n\binom{kn}{n}}$
$=\sum\limits_{n=1}^\infty\dfrac{\Gamma(n+1)\Gamma((k-1)n+1)}{n\Gamma(kn+1)}$
$=\sum\limits_{n=1}^\infty\dfrac{\Gamma(n)\Gamma((k-1)n+1)}{\Gamma(kn+1)}$
$=\sum\limits_{n=0}^\infty\dfrac{\Gamma(n+1)\Gamma((k-1)n+k)}{\Gamma(kn+k+1)}$
$=~_3\Psi_1\left[\begin{matrix}(1,1)~~(1,1)~~(k,k-1)\\(k+1,k)\end{matrix};1\right]$ (according to http://en.wikipedia.org/wiki/Fox%E2%80%93Wright_function)
Comment: Although it is only a rewriting of the sum, I will still give you the reputation for doing so.
Answer: Again, this is not going to be a full answer. However since my approach is slightly different from what was presented here so far and since my result is similar to the result given by Tito Piezas III I will post my answer.
\begin{eqnarray}
&&S_k:=\sum\limits_{n=1}^\infty \frac{1}{n \binom{ k n}{n}}=\\
&&\sum\limits_{n=1}^\infty \frac{k n+1}{n} \cdot \int\limits_0^1 t^n \cdot (1-t)^{(k-1) n} dt=\\
&&\int\limits_0^1 \left[ k \frac{t\cdot (1-t)^{k-1}}{1-t \cdot (1-t)^{k-1}} - \log(1-t \cdot (1-t)^{k-1})\right] dt=\\
&&-\int\limits_0^1 t \cdot \frac{ (1-t)^{k-3} (k t-1) \left(t (1-t)^k+k (t-1)+t-1\right)}{\left(t (1-t)^{k-1}-1\right)^2} dt=\\
&&\int\limits_0^1 \frac{(u-1) (k (u-1)+1) u^{k-2} \left(-u^{k-1}+u^k+k+1\right)}{\left(-u^{k-1}+u^k+1\right)^2}du=\\
&&\int\limits_1^\infty \left(\frac{\left(k^2-1\right) (v-1)}{v \left(v^k-v+1\right)}-\frac{k (k+1) (v-1)}{v^2 \left(v^k-v+1\right)}-\frac{k^2 (v-1)^2}{v^2 \left(v^k-v+1\right)^2}+\frac{(k-1) k (v-1)^2}{v \left(v^k-v+1\right)^2}\right) dv=\\
&&\int\limits_1^\infty \frac{(1-v) (k-(k-1) v) \left((k+1) v^k-v+1\right)}{v^2 \left(v^k-v+1\right)^2} dv
\end{eqnarray}
Let us discuss the steps from the very top to the bottom.
In the second line  we used the integral representation of the beta function and in the third line we did the sum in question. In the fourth line we integrated by parts to get rid of the logarithm.In the fifth line we substituted for $u:=1-t$ and finally in the sixth and seventh lines we substituted for $v:=1/u$ and simplified the integrand.
Now, oddly enough the equation in the last line is similar to that given by Tito Piezas III. This would suggest that maybe the integrands of both results are the same, which they are not, as I have checked. It would be interesting to simplify my result further and bring it to the form given by Tito Piezas III.
Update: Note that:
\begin{equation}
\frac{(1+k) v^k-v+1}{v^2(1-v+v^k)^2}= \frac{d}{d v} \left[\frac{1}{v \cdot(1-v+v^k)} \right] + \frac{1}{v (1-v+v^k)^2}
\end{equation}
Inserting this into the last line above and integrating by parts we get:
\begin{eqnarray}
S_k&=&\int\limits_1^{\infty} \frac{1}{v \cdot (1-v+v^k)} dv + \int\limits_1^\infty \frac{(k-(k-1)v)}{v \cdot (1-v+v^k)^2} \cdot (-2 v^k-1+v) dv\\
&=&\int\limits_1^{\infty} \frac{1}{v \cdot (1-v+v^k)} dv + \left.\left( \frac{1-v}{1-v+v^k} + \log[\frac{1}{v^k}-\frac{1}{v^{k-1}}+1]\right)\right|_{1}^\infty\\
&=&\int\limits_1^{\infty} \frac{1}{v \cdot (1-v+v^k)} dv+0\\
&=&\int\limits_1^{\infty} \sum\limits_{p=1}^k \frac{1}{v\cdot(v-\xi_p)} \cdot \prod\limits_{q=1,q\neq p}^k\frac{1}{\xi_p-\xi_q)}dv=\\
&=&-\sum\limits_{p=1}^k \frac{\log(1-\xi_p)}{\xi_p} \cdot \prod\limits_{q=1,q\neq p}^k\frac{1}{(\xi_p-\xi_q)}
\end{eqnarray}
Here $\left\{ \xi_p \right\}_{p=1}^k$ are roots of the polynomial $v^k-v+1$.
As a final comment we note that in general the following holds:
\begin{equation}
\sum\limits_{n=1}^\infty \frac{x^n}{n} \cdot \frac{1}{\binom{k\cdot n}{n}} =x \cdot \int\limits_1^\infty \frac{1}{v\cdot(v^k+x-x v)} dv
\end{equation}
where again the integral on the right hand side can be expressed through roots of the polynomial in the denominator.


How to combine rows and put into single row in dataframe by sql or python

Question: I'd like to aggregate rows in certain column base on the relationship with other column and create certain column which contain aggregated data in json format.
This is the example.
Original data table
Child Name     Child Age    Father Name    Father Age
     Peter             5        Richard            40
     James            15           Doug            45
       Liz             2           Doug            45
      Paul             6        Richard            40
    Shirly            11        Charles            33
       Eva             9          Chris            29

Converted Data table will be either
Father Name    Father Age     Children
    Richard            40     {"Peter":"5", "Paul":"6"}
       Doug            45     {"James":"15","Liz":"2"}
    Charles            33     {"Shirly" : "11"}
      Chris            29     {"Eva" : "9"}

Or
Father Name    Father Age     Children Name       Children Age
    Richard            40     {"Peter", "Paul"}      {"5","6"}
       Doug            45     {"James", "Liz"}      {"15","2"}
    Charles            33     {"Shirly"}                {"11"}
      Chris            29     {"Eva"}                    {"9"}

My code is
import pandas as pd
df = pd.DataFrame({
    "Child Name" : ["Peter","James","Liz","Paul","Shirly","Eva"],
    "Child Age" : ["5","15","2","6","11","9"],
    "Father Name" : ["Richard","Doug","Doug","Richard","Charles","Chris"],
    "Father Age" : ["40","45","45","40","33","29"] })

 print df

g1 = df.groupby(["Father Name"])["Child Name"].apply(", ".join).reset_index()
g1.columns = ['Father Name','Children Name']
print g1

and the output will be
  Father Name   Children Name
0     Charles          Shirly
1       Chris             Eva
2        Doug      James, Liz
3     Richard     Peter, Paul

I can't figure out how to add "Father Age" and "Children Age" in the columns.
how can I convert this in dataframe in most efficient way?
I'd like to avoid loop via python as it will take long to process.
thanks,
Comment: Your original table is not normalized. You should have a separate father table and child table.
Comment: I don't get it about normalization on original table. can you elaborate it more detail with simple example?
Answer: Quick dirty inefficient hack, but it avoids for loops. Would love to have a better solution; I assume the multiple df copies and multiple merges could be simplified.
import pandas as pd
df = pd.DataFrame({
    "Child Name" : ["Peter","James","Liz","Paul","Shirly","Eva"],
    "Child Age" : ["5","15","2","6","11","9"],
    "Father Name" : ["Richard","Doug","Doug","Richard","Charles","Chris"],
    "Father Age" : ["40","45","45","40","33","29"] })

g2 = df.groupby(['Father Name'])["Child Name"].apply(list).reset_index()
g3 = df.groupby(['Father Name'])["Child Age"].apply(list).reset_index()
g4 = df[["Father Name", "Father Age"]].drop_duplicates()

df2 = g2.merge(g4)
df2 = df2.merge(g3)
print(df2)

Output:
  Father Name     Child Name Father Age Child Age
0     Charles       [Shirly]         33      [11]
1       Chris          [Eva]         29       [9]
2        Doug   [James, Liz]         45   [15, 2]
3     Richard  [Peter, Paul]         40    [5, 6]

Comment: Thanks Evan for your time, this is close to the output I want although it doesn't create data either json or list format.
Comment: Ah, got it. There's an easy enough change to get a list - see revised code :)


Heading in plots by mvpart()?

Question: How can I put in a heading in the tree output when using mvpart()? I am using xv="1se".
e.g. using the spider data in mvpart
fit<-mvpart(zora.spin~water+sand+moss+reft+twigs+herbs,data=spider)
from this I get a tree in R. And I am wondering how to "put in" a heading in this figure.
Comment: You need to explain more, with a full example; `mvpart()` is a complex function doing all sorts of things. What do you mean by "tree output"? Use the example in `?mvpart` and edit your Q to include a worked example of what you are trying to do.
Answer: Assuming you don't want the PCA plot, here's how you can add some text to the top of the plot. You can use margin argument in mvpart to add some extra space around the edges. Feel free to play around to massage the title to something of your liking.
fit <- mvpart(data.matrix(spider[,1:12]) ~ herbs + reft +
           moss + sand + twigs + water, spider, xv = "1se", margin = 0.1)
mtext("Mission accomplished", side = 3, cex = 2)

Heading in plots by mvpart()?


Regex for egrep to find lines with at least one 0 and at least two 3's

Question: I'm  having trouble implementing this pattern (I can't use awk,sed or any other command and have to use regex)
So far I have tried it with
<code>egrep '0+\|(3|33)+' file
</code>
but it produces no output. Any help is appreciated
Answer: Something like this?
<code>$ cat user122
asdfsdf
sadf 0 sdfgfasdf 33
sadf 1 sdfgfasdf 3
sadf 33 sdfgfasdf 0
sadf 3 sfdsdf 3 sdfgfasdf 0
sadf 0 sdfgfasdf 3

$ grep -E '0.*3.*3|3.*3.*0' user122
sadf 0 sdfgfasdf 33
sadf 33 sdfgfasdf 0
sadf 3 sfdsdf 3 sdfgfasdf 0
</code>
Comment: oh, ok, that wasn't clear to me. Updating response.


DMA7th 1.1 Propositional Logic - Exercise 2

Question: Which of these are propositions? What are the truth values of those that are propositions?
a) Do not pass go.
b) What time is it?
c) There are no black flies in Maine.
d) 4 + x = 5.
e) The moon is made of green cheese.
f ) 2 n ≥ 100.
Answer: In logic, a proposition is a contextual declarative statement that is either true or false, but never both.

a) Do not pass go.

Is NOT a proposition;

b) What time is it?

Is NOT a proposition;

c) There are no black flies in Maine.

Is a proposition; with the truth value False.

d) 4 + x = 5.

Is NOT a proposition; as its truth value Unknown.

e) The moon is made of green cheese.

Is a proposition; with the truth value False.

f ) 2 n ≥ 100.

Is NOT a proposition; as its truth value Unknown.


xslt create ordered list from string

Question: Have a string in xml
<anons>
      1. first list item. 2. second list item. 3. third list item.
</anons>

Is it possible to create an ordered list like this:

first list item.

second list item.

third list item.

Comment: Please do not tag questions with both `xslt-1.0` and `xslt-2.0` - they exclude each other.
Answer: It is possible but it would have to be a string operation, meaning it might not be possible in some cases if the text is more complex than this.
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:xd="http://www.oxygenxml.com/ns/doc/xsl"
exclude-result-prefixes="xs xd"
version="2.0">

<xsl:template match="/">
    <xsl:apply-templates/>
</xsl:template>

<xsl:template match="anons">

        <xsl:for-each select="tokenize (./string(), '\d+\.')[normalize-space()]">

<xsl:value-of select="normalize-space(.)"/>

        </xsl:for-each>

</xsl:template>

</xsl:stylesheet>

Answer: If you use e.g.
<xsl:template match="anons">

    <xsl:for-each select="tokenize(., '[0-9]+\.')[normalize-space()]">

        <xsl:value-of select="normalize-space()"/>

    </xsl:for-each>

</xsl:template>

you should get what you want. Another option would be to use analyze-string.

xslt create ordered list from string


Share content with All Groups

Question: Is there a way around to get Share with All Groups option in the group audience field for a specfic group content type.With share with All Group I mean to say share content to all groups we are not even if are not member of.
I am using og module on drupal 7 installation.
Answer: I managed to it with some custom coding basically i alter the form add a one more option to group audience field and then on form validation I am checking if user has input that field then I am making a db query to grab all og groups on site and then assign the group ids to the group audience field,looks kinda hackish but did'n get any better solutions.

Share content with All Groups


Move uploaded file in network drive

Question: I don't know what's wrong with my code. I have Z: network drive named ntserver and i want to move file in it. Only working in a local drive such as C and D.
The error displays :

Warning:
  move_uploaded_file(\ntserver\test\oggy_and_the_cockroaches-wide.jpg):
  failed to open stream: No such file or directory in
  C:\wamp\www\michael\upload1.php on line 30
Warning: move_uploaded_file(): Unable to move
  'C:\wamp\tmp\php3FDE.tmp' to
  '\ntserver\test\oggy_and_the_cockroaches-wide.jpg' in
  C:\wamp\www\michael\upload1.php on line 30

<?
       if(isset($_FILES['image']))

       {
          $errors= array();
          $file_name = $_FILES['image']['name'];
          $file_size = $_FILES['image']['size'];
          $file_tmp = $_FILES['image']['tmp_name'];
          $file_type = $_FILES['image']['type'];

          $expensions= array("jpeg","jpg","png");

          if($file_size > 2097152) {
             $errors[]='File size must be exactly 2 MB';
          }

          if(empty($errors)==true) {
             move_uploaded_file($file_tmp,"\\\\ntserver\test\\".$file_name);
             echo "Success";
          }else{
             print_r($errors);
          }
       }
?>

Comment: try to remove last backslash and change path to move_uploaded_file($file_tmp,"\\\\ntserver\test\".$file_name);  or replace the "ntserver" with IP address of machine. Next one tip: try to write to Z:\test as you have mounted that drive as letter Z. You also need to check if permissions are valid and have you mounted drive permanently in windows
Comment: @Fiil , yes., i have already tried with Z: and error still the same. i have permission to write a file in Z:
Comment: @Michel make sure the `test` directory exist on remote path and php process has enough rights to write to the path.
Comment: @jagad89 , yes i created a test folder in Z: , what do you mean by enough right to write with php ? how am i supposed to do that. I can write in the Z: test folder
Comment: @MichaelAngeloJopia just right click on the `test` folder-->Properties-->Security tab and give write permission to every user, and try again.
Comment: @jagad89 i have already check it. i have permission to right the file.
Comment: @MichaelAngeloJopia What about the system user rights ?
Comment: Possible duplicate of [Failed to open stream : No such file or directory](http://stackoverflow.com/questions/36577020/failed-to-open-stream-no-such-file-or-directory)
Answer: Try removing a back slash. Before trying this make sure that the network location is accessible through cli or run.
move_uploaded_file($file_tmp,"\\ntserver\test\".$file_name);

Move uploaded file in network drive


customizing login credentials for a single guard in laravel multi-authentication

Question: I have multi-authentication set up in my laravel app. I set up different guards and providers for admin, doctor and patient. I've also set up the redirects for the various guards in my config\auth.php.
The only issue i'm having right now is, I want doctors to login with a unique number called MDCN which is saved to the database table instead of email. Please how do i customize the login for the doctors?
Ps: I've also set up the models for each, and i use laravel default auth.
my guard:
'guards' => [
        'web' => [
            'driver' => 'session',
            'provider' => 'users',
        ],

        'api' => [
            'driver' => 'token',
            'provider' => 'users',
        ],

        'admin' => [
            'driver' => 'session',
            'provider' => 'admins',
        ],
        'admin-api' => [
            'driver' => 'token',
            'provider' => 'admins',
        ],

        'doctor' => [
            'driver' => 'session',
            'provider' => 'doctors',
        ],
        'doctor-api' => [
            'driver' => 'token',
            'provider' => 'doctors',
        ],
    ],

and provider:
'providers' => [
        'users' => [
            'driver' => 'eloquent',
            'model' => App\Models\User::class,
        ],

        'admins' => [
            'driver' => 'eloquent',
            'model' => App\Models\Admin::class,
        ],

        'doctors' => [
            'driver' => 'eloquent',
            'model' => App\Models\Doctor::class,
        ],
],

Answer: In the App\Http\Controllers\Auth\LoginController you can override a method which returns by what fields the authentication should be checked and there you can pass the MDCN if you're logging a doctor:
protected function credentials(Request $request)
{
    return $request->only('MDCN', 'password');
}

Comment: but isnt that going to affect other guards? like the user logging in with email and password?
Comment: @OkoroaforChukwuemeka yes, this will affect all logins, so you'll have to check some condition when only a doctor is logging to get the MDCN instead of the email.
Answer: i finally solved this, thank to Alex Curtis.
In order to do this, you need to customize each model to tell it what you want to use to login.
So on any model that you want to log in with a username instead of an email you need to add this block of code to the model:
public function username()
{
    return 'username';
}

And just make sure that the value you return (username in this example) matches the column name in that model's table that has the identifier you are looking to use.
Obviously you would also want to make sure you have a column in the DB to hold this data, you want it to be mass-assignable (by customizing the $fillable array in the model) and you also want the column to be indexed and unique.
To get this you want to create the column in a migration that looks like this:
$table->string('username')->unique();

This will make a unique index for that column.
The default is email, so if you are using email then you don't need to worry about this, but any model using something other than email will need this change made to that user type's model.


Drupal theme error. functions from add on libraries are not included when running my custom script

Question: I'm trying to load a javascript function in my custom script with jquery and materialize. When the script load it says
<code> Uncaught TypeError: $ is not a function and sidenav() is not defined.
</code>
My theme library is as follows
<code>    global-styling:
  version: VERSION
  css:
    base:
      materialize/css/materialize.min.css: {}
      css/style.css: {}
  js:
    materialize/js/materialize.min.js: { minified:true}
    js/layout.js: {}
  dependencies:
    - core/jquery
</code>
My layout.js is simple
<code>  $('.sidenav').sidenav();
</code>
If I look at dev tools it shows materialize and jquery loading before my script. I followed the answer to get jquery working but sidenav() comes back undefined. sidenav() is a materialize function and material.js is loading before this script.
How do I properly attach a 3rd party js framework and call a function from it?
Answer: $ is a shortcode for jQuery, does your layout.js contain just
<code> $('.sidenav').sidenav();
</code>
Maybe try
<code>(function ($, Drupal) {
    $('.sidenav').sidenav();
 })(jQuery, Drupal);
</code>
You can read more here: https://www.drupal.org/docs/8/api/javascript-api/javascript-api-overview
Comment: That seemed to work to fix jquery. It won't, however, read .sidenav() which is a materialize function. I'm guessing I'm going need to somehow include materialize.js in the ($, Drupal) part?


Given the function $f(x)=\left(3x^{\frac75}\!-\!\frac{\pi}x\right)\!\cdot\!\cos\left(\frac{\pi}2\!-\!x\right)$. Calculate $\,\lim\limits_{x\to0}f(x)$

Question:
Given the function $f(x)=\left(3x^{\frac75}\!-\!\dfrac{\pi}x\right)\!\cdot\!\cos\left(\dfrac{\pi}2\!-\!x\right)$. Calculate the limit $\,\lim\limits_{x\to0}f(x)\;.$

So I started by using the definition of the derivative that
$$f'(x_{0})=\lim_{x \to x_{0}}\frac{f(x)-f(x_{0})}{x-x_0}$$
after doing that I tried to set $x_{0} = 0$ since then u get the limit to $x \to 0$, but the problem is that $f(0)$ is not defined, because if I substitute $x=0$ in the function I get $f(0)=(0-\infty)\cdot{}0$ which is undefined, same goes for $f'(0)$
Now I can't seem to find the solution, so I'm asking the question here
Comment: Remember the limit of $\frac{\sin(x)}{x}$ when $x \to 0$.
Comment: If $f$ is undefined at some point then $f'$ is also undefined at that point.
Comment: Split it into two terms.
Comment: Don't use the definition of the derivative on $f$ but on $\sin$.
Comment: Lynx_s, what do you want to calculate? $\lim\limits_{x\to0}f(x)$ or $f'(0)$?
Comment: @Angelo I need to calculate the limit but I thought using the definition of derivative to then manipulate it to then get something of the limit of f(x) with x going to 0, I tried this since this question is asked in the chapter of differentiation so I assumed we had to use the definition or derivative in some way
Comment: @geetha290krm do you mean the limit I proposed or?
Comment: @Cactus Sorry I don't quite get what you mean
Comment: @lynx_s, if you want to know what Cactus means, read my answer and tell me what you think about it.
Answer: If you want to use the definition of derivative in order to calculate your limit, you can proceed in the following way:
$\begin{align}\lim\limits_{x\to0}f(x)&= \lim\limits_{x\to0}\left(3x^{\frac75}-\frac{\pi}x\right)\!\cdot\!\cos\left(\frac{\pi}2-x\right)=\\[3pt]&=\lim\limits_{x\to0}\left(3 x^{\frac75}-\frac{\pi}x\right)\!\cdot\!\sin(x)=\\[3pt]&=3\lim\limits_{x\to0}\left(x^{\frac75}\!\sin(x)\right)-\pi\lim\limits_{x\to0}\dfrac{\sin(x)}x=\\&=3\!\cdot\!0-\pi\lim\limits_{x\to0}\dfrac{\sin(x)-\sin(0)}{x-0}=\\[3pt]&=-\pi\!\cdot\!\big[D\sin(x)\big]\bigg|_{x=0}=-\pi\!\cdot\!\big[\cos(x)\big]\bigg|_{x=0}=\\[3pt]&=-\pi\!\cdot\!\cos(0)=-\pi\!\cdot\!1=-\pi\;.\end{align}$
Comment: Ahh I see, thanks a lot this just gave some new insight on how to solve something like this, tricky tho, I needed the answer because it was an exam question in 2015-2016, there is also a extra question, where they ask how the function is continuous on R if we extend the function, I'm supposing we could do that by defining f(x) as a Piecewise function where f(x) = (3x/..)sin(x) if x < 0 and x > 0 and f(x) = $-\pi$ where x = 0?
Comment: @lynx_s, yes, you can extend the function $f(x)$ by defining it as a piecewise function in the way you did. So you will get a function continuous on all $\Bbb R\,.$ It is correct what you said.
Answer: Note that $\cos\left(\frac{\pi}{2}-x\right)\,\,\sim\,\,x$ when $x\to 0$. So:
$$\lim_{x\to 0}\left(3x^{\frac{7}{5}}-\frac{\pi}{x}\right)\cdot x=\lim_{x\to 0}3x^{\frac{12}{5}}-\pi=-\pi$$


Failed to add custom DNS Server

Question: My company wants me to setup our own DNS Servers, but i'm running into problem with it. So here's the story:
We bought a domain example.com from marcaria and bought a new VPS located in Russia. Our goal is to setup example.com to use ns1.example.com and ns2.example.com. But we cant do that because our domain still pointing to marcaria NS.
So first i create DNS Server using BIND within our VPS with this tutorial http://linuxgravity.com/configuring-bind9-domain-name-server-on-centos-or-red-hat , and when i try to change my domain DNS server to point ns1.example.com, it says

Address lookup for ns1.example.com  failed: Host not found

I know its like chicken and egg problem, but i have no clue for the solution. I saw another thread solution but its godaddy as the domain registrar, it doesnt work for me.
Thanks
Answer: Delegating name service
You must ask your domain registrar, Marcaria, to arrange that the nameservice for example.com be delegated to your nameservers. You'll have to give them the Internet-facing publicly-accessible IP-addresses of those nameservers.
In the case of example.com the delegation is carried out by configuring the nameservers for the .com domain. In your case it may be different but your domain registrar should arrange this for you.
Configuring other computers
Most businesses operate split-DNS where the DNS namespace visible to internal users is separate from that available to outside users. This often means that internally you use a separate nameserver for the example.com domain. Either way, for obvious reasons, you specify the nameserver the computer is to use by the nameserver's IP-address, not it's name.
Comment: I give the ns1.example.com accessible IP address, its just they lookup for the hostname. I guess i have to contact their support.
Comment: I dont really understand your last paragraph, does it mean i used one IP address as a web server and dns server?
Comment: If you configured BIND on the same VPS you are using for Apache httpd server then yes.
Answer: I think that on the current nameservers handling example.com, you must first set up A records ns1.example.com and ns2.example.com in order to delegate NS control of example.com to them.
Don't forget to add the same A records to your own bind so that when the change take place, the records still exist.
note: I might be wrong here, I'm not very good with DNS :-)
Comment: So, what your saying is i need to add A record in marcaria NS and in my bind? hmm..worth to try
Comment: If you intend to run your own nameservers for example.com then Marcaria's nameservers are not involved. How familiar are you with DNS? Is the Wikipedia article of any interest? http://en.wikipedia.org/wiki/Domain_Name_System
Comment: "change my domain DNS server to point ns1.example.com, it says
Address lookup for ns1.example.com failed: Host not found". Before you can delegate control of example.com to ns1.example.com, ns1.example.com record must exist, therefor start with creating it in the nameservers that are currently controlling example.com (I suppose Marcaria's).
Answer: You want to use a name for your DNS server that is part of your domain. Normally to resolve names from your domain, it is necessary to query your DNS server, but the address or your DNS server is not known to the resolver.
To solve this, you need to provide not only the name but also the IP address of your DNS server. This is called glue records and is exactly to solve this problem. How you can provide the IP addresses depends on the registrar of your domain.
The glue records are stored in the com. domain (in this example). So a DNS query to the com. DNS servers for name.example.com would not only return an NS record pointing to ns1.example.com, it would also contain an authoritative A record for ns1.example.com so that a resolver knows where to continue. Of course, the A record for ns1.example.com in your zone example.com and the glue A record for ns1.example.com should match.
Answer: When you have an existing domain example.com (the setup will be slightly different for a domain you are about  to create at the same time) and you want to use in-bailiwick nameservers for it, that is nameservers whose name are under the domain name itself, like your ns1.example.com and ns2.example.com, then it means the registry will need to create glue records in its zone, that is A and AAAA records for the nameservers' names because otherwise no resolution could happen at all.
To have that to happen it needs to be done by your registrar. It will also depend on the TLD, because some are using hosts as "objects" and others as "attributes". So in some cases the registrar will mandate you to use a specific procedure, before changing the domain name itself, to register the nameservers at the registry, where you will need to enter their name and IP addresses.
After that the change of nameservers for the domain name should succeed.
Of course, before all of that, ns1.example.com and ns2.example.com should be properly set up with the appropriate zone (containing their own IP addresses because they are in bailiwick), answering from public queries, etc.
You can use a tool such as Zonemaster to try that, it has a mode to test "undelegated domains", that is to test your domain name but not with its current set of nameservers, with the new one you are attempting to use.
As a final advice: glues can create all sorts of problem, I do not think it is a good idea to use them when you start being a DNS administrator.

Failed to add custom DNS Server


Selenium Python can I sort a list of web elements alphabetically

Question: I have a list of web elements.  I would like to sort it alphabetically.
There is no sort function for the data type of web elements.
I got the following error when i tried sort:
Traceback (most recent call last):
  File "C:\Webdriver\ClearCore Regression Test\ClearCore - Regression Test\TestCases\DataPreviewsPage_TestCase.py", line 375, in test_a2_sort_data_preview
    print data_previews_page.compare_lists_2()
  File "C:\Webdriver\ClearCore 501 Regression Test\ClearCore - Regression Test\Pages\data_previews.py", line 610, in compare_lists_2
    sortedTable = DataPreviewsPage.column1_list_before_sort.sort() # This table holds the data previews name in sorted order of what it should be
AttributeError: 'NoneType' object has no attribute 'sort'

My code snippet is:
class DataPreviewsPage(BasePage):
            column1_list_before_sort = None # This list will hold the data from col1 before ClearCore has done the sort.  This list will then be sorted into ABC order.  We can then use it to compare with the list after ClearCore has done the sort

def compare_lists_2(self):
        sortedTable = DataPreviewsPage.column1_list_before_sort.sort() # This table holds the data previews name in sorted order of what it should be
        sortedList = DataPreviewsPage.list_after_ClearCore_has_sorted # This compares the sort with the expected sort (expected sort = sortedTable)
        for item in sortedTable:
            print sortedTable[item]
            print sortedList[item]

the method which returns the list of web elements is:
def get_table_column_1_values_into_a_list_before_ClearCore_has_done_the_sort(self):
        try:
            WebDriverWait(self.driver, 20).until(EC.presence_of_element_located((By.ID, 'data_configuration_data_previews_ct_fields_body')))
            table_id = WebDriverWait(self.driver, 20).until(EC.presence_of_element_located((By.ID, 'data_configuration_data_previews_ct_fields_body')))
            rows = table_id.find_elements(By.TAG_NAME, "tr")
            return [row.find_elements(By.TAG_NAME, "td")[1].text for row in rows] # returns the data preview name from col1 for each row
        except NoSuchElementException, e:
            print e
            return False

How can I sort the list of web elements alphabetically?
The web elements variable is "column1_list_before_sort", I would like to sort this one.
Thanks,
Riaz
Answer: You can create your cutom comparator to sort a list of object in your custom way, example
def comparator(e1, e2):
    return e1.text > e2.text

array.sort(comparator)

However, I think the problem is the value of column1_list_before_sort .Since the error is AttributeError: 'NoneType' object has no attribute 'sort', it mean nothing inside column1_list_before_sort.


When should I use Compiled LINQ vs Normal LINQ

Question: I just read up on a performance of LINQ, and there is a HUGE amount to be gained by using Compiled LINQ.
Now, why won't I always use compiled LINQ?
Comment: Does this apply to all LINQ expressions or just LINQ to SQL ?
Answer: Short answer: when it's only going to happen once in a long time.
Long Answer.
Answer: You should use it when some linq query is executing most of the time. Those can be converted to compiled lynq. Performance will be improved as execution path of query will be set at compile time. I used it in my project and performance went up a notched.


keepalived nopreempt option not working

Question: I want to use nopreempt option with keepalived vrrp setting to run backup node as master when master goes down and again reback into network.
I setup nopreempt option in both servers and set state as backup in both servers but due to high priority nopreempt is not working.
Please guide to resolve it?
  Master Machine:
  ! Configuration File for keepalived
   vrrp_instance VI_1 {
   state BACKUP
   nopreempt
   interface eth0
   virtual_router_id 1
   priority 250
   advert_int 1
   virtual_ipaddress {
    192.168.1.2/24
    }
  }

  Backup Machine :
  ! Configuration File for keepalived
   vrrp_instance VI_1 {
   state BACKUP
   nopreempt
   interface eth0
   virtual_router_id 1
   priority 200
   advert_int 1
   virtual_ipaddress {
     192.168.1.2/24
   }

}
Regards,
Ben
Answer: Changed both server state as BACKUP. Primary having higher priority and with nopreempt, both having the same router ID. This works for me.
Answer: I have tried many configurations to achieve this and the only one that works was setting both servers to state BACKUP, one server with priority 51, the othe one with 50 and setting nopreempt. Here is the example configuration:
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 51
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass XXXXXXXX
    }
    virtual_ipaddress {
        192.168.69.100/28
    }
}

Just set priority to 50 on the second server and all should work.
Answer: First of all: I'm using CentOS 6 and keepalived 1.2.7 (02/21,2013)
I tried nopreempt as well and after some testing it works as expected.
Running master and backup with the same vrrp priority is a bad idea because there is a bug (AFAICT) which leads to a situation were both keepalived server start the VIP.
according to https://www.rfc-editor.org/rfc/rfc5798#page-26 the keepalived with the highest IP should stay/become master.
I could not observe this behavior!
(AFAICT again) You have to remove the statement "state" from the config file if you intend to see nopreempt working.
If the keepalived with "state MASTER" in config file starts up it starts as master and advertises it self as such = bad.
If there is no "state" statement in the config file keepalived starts in state BACKUP and listens to vrrp. because of "nopreempt" it will not become MASTER even if it runs with highest priority. this is contrary to the man 5 keepalived.conf page, whereupon state does not matter much.
The keepalived with prio 255 always becomes master - regardless of nopreempt option on or off.
If you test keepalived have a close look at firewall/nat rules. it's always a good idea to run something like tcpdump to check if the vrrp advertisements get through and have the correct sender IP (the one from the other keepalived(s), not the standard gw!)
This is especially true if you use kvm/qemu guests as there are default nat rules which change vrrp advertisements.
nopreempt only works for keepalived in state BACKUP. (RTFM)
if keepalived (1.2.7) is running and the network link goes down keepalived changes to state FAULT. when network link comes back keepalived becomes MASTER if prio is high enough. I don't know if this is a bug but it makes nopreempt somehow useless. I'll send a question to the mailing list.
Answer: This may or may not be the correct solution for you, but I only have two keepalived servers.
If you don't want one server to pre-empt the other, one server having a higher priority than the other doesn't really matter in a two server scenario like mine. It works for me if I enable nopreempt and set both servers to have the same priority.
UPDATE
Config example as requested:
vrrp_sync_group VRRP_SYNCS {
    group {
        public_http_ips
    }
}

vrrp_instance public_http_ips {
    state MASTER
    interface eth0
    virtual_router_id 18
    priority 100
    advert_int 1
    nopreempt
    virtual_ipaddress {
        192.168.0.254/24 dev eth0
    }
}

The backup is exactly the same but it says "state BACKUP".
Comment: Hi Javano, Can you share your config to resolve nopreempt problem?
Comment: @Ben Added as requested :)
Answer: You must set "fall" and "rise" options in vrrp script. Example running configuration:
vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 1
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    interface eth0

    track_interface {
        eth0
        eth1
    }

    state BACKUP # same as other node
    priority 101 # your choice
    virtual_router_id 53
    advert_int 1
    nopreempt

    authentication {
        auth_type PASS
        auth_pass 8CHARPASS
    }

    unicast_src_ip 172.31.10.11 # other node: 172.31.10.12
    unicast_peer {
        172.31.10.12 # other node: 172.31.10.11
    }

    virtual_ipaddress {
        172.31.20.20 dev eth1
    }

    track_script {
        chk_haproxy
    }
}

Answer: I think the same param "virtual_router_id" in two configs is the key here, with different "virtual_router_id" values the nopreempt doesn't work.
Try to set the same "virtual_router_id".

keepalived nopreempt option not working