How to get predicted values from cross validation?

Question: This question is regarding cross validation and prediction with regularized logistic regression, so by parameters here I mean the beta-coefficients for each predictor variable, for output I get predicted probabilities of belonging to a group, and for performance measure I use AUC. I am using glmnet on matlab.
For my training set I have group A (normal group) and group B (patient group), and for my validation set I have group C (high-risk group that later converts to patient group) and group D (high-risk group that later does not convert).  I want to take parameters that best differentiate group A and B to predict conversion to patient group among people in group C and D.
To this end, I have done 10-fold cross validation on group A and B to find the best lambda (lambda_1se to be exact), and I am taking the parameters at that lambda value and applying them to Group C and D to get predicted probability values (via cvglmnet and cvglmnetPredict).
Here, it seems that:

The best lambda is the only thing that will be searched for from the CV, much like hyperparameter optimization that would happen in an inner loop of a nested cross validation.
After finding the best lambda value from CV, my whole training dataset (Group A,B) is retrained to get parameters for that specific lambda value, and those parameters are used to predict outcome in my validation dataset.
Is my understanding correct?

What I would like to do is make a boxplot of predicted probabilities of groups A~D so that I can see the trend of predicted values across those groups (ideally the values would be gradiently descending from patient-> highrisk-later convert -> high risk-not convert -> normal).
Here is my main question:
I have already gotten the predicted probability values from groups C and D, but how do I get the predicted probability values from group A and B (the training set on which I did CV on)?
I first thought of taking the predicted probabilities calculated from the left-out test set in each of the 10 folds.  However, my current understanding is that if I were to do that, each of the test sets' predicted probabilities would be based on different lambda values, since each training set in each fold would have had different optimal lambda values.
Answer: I do not know about MATLAB, but in scikit-learn you can get the probabilities by doing a manual nested cross-validation.
The code would be roughly,
<code># Dividing data set for outer cv
cv_outer = StratifiedKFold(n_splits = 10, shuffle = True, random_state = 43)

# Outer cv beginning
for (train, test) in cv_outer.split(X, y):
    # parameters for grid search
    pipe_logis = Pipeline([('logis', LogisticRegression(random_state = 1))])
    param_range = [0.0001, 0.001, 0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]
    param_grid = [{'logis__C': param_range, 'logis__penalty': ['l1']},
             {'logis__C': param_range, 'logis__penalty': ['l2']}]

    #Dividing data set for inner cv
    cv_inner = StratifiedKFold(n_splits = 5, shuffle = True, random_state = 42)

    # Inner cv done by these 2 lines
    gs = GridSearchCV(estimator=pipe_logis, param_grid=param_grid, scoring='f1',
         cv= cv_inner)
    gs.fit(X[train], y[train]) #we are using only the training fold for
                                hyper-parameter tuning

    # get best parameters
    print gs.best_params_

    # predict with best parameters
    scores = gs.best_estimator_.predict(X[test])

    # These are your probabilities. Here I am getting it for the test fold,
    but you can get probabilities for your training fold.
    probas_ = gs.best_estimator_.predict_proba(X[test])
</code>
Also, a nested cross-validation is only for an unbiased performance measurement. After getting a sense of performance measurement for various algorithms, you choose the algorithm with the best performance. Then you do a k-fold cross-validation with the whole dataset to finally choose your best set of hyper-parameters for that algorithm. You do not want to use the performance measurement from this later k-fold cross-validation.
Of course, all of this is for small datasets. If you have a large dataset, you can just divide it between training set, validation set and test set. Then you use the validation set for hyper-parameter tuning.


Does CacheStorage (Web Cache) API care about headers like Content-Encoding?

Question: I am using the newly added CacheStorage API in Deno 1.26. My understanding is that this basically functions as a Map<Request, Response>. I question what aspects of the Request object factor into determining whether or not it's a match. For instance, is only the URL taken into account? I began to think about what would happen if a request came in for a particular URL with Accept-Encoding: gzip, deflate, br and then the response contained a gzip compressed body with header Content-Encoding: gzip. What then would happen if the same URL was requested by a client whose request contains e.g. Accept-Encoding: deflate. Would the CacheStorage API match this based off the URL alone and return the response with the gziped body?
Well I tested this scenario in Deno and it appears to be the case. Deno appears to only match based off the URL and doesn't care about the Content-Encoding header. So my question is whether this is a bug in Deno's implementation or whether or not its simply an oversite in the web API. Perhaps I'm overthinking this and in reality its moot because virtualy all HTTP clients in use today support gzip compressed responses.
Answer: The only header considered in the Request matches cached item algorithm is the Vary header.
External clients don't call the Cache API itself. Your application code is able to manage those calls completely.
Here are some ideas on making an application that uses the Cache API and supports both compressed and uncompressed responses to its clients:

internally use a specific compression with the Cache API
require clients to support the encoding or add support for on-demand decompression (see next item)
when serving a Response from the Cache to a client…

if the client specifies that it accepts the same encoding then return the compressed response directly from the Cache
if the client specifies that it accepts other encodings but not the one your application is using internally then decompress and possibly re-compress using a different algorithm to return the desired Response
if the client specifies no encodings that it accepts then decompress the Response from Cache to return to the client


Comparison of explaned variances in PCA

Question: I computed a PCA and am interestet in the explained variance of the first unrotated component. The same procedure was used in a previous study.
Question: How do I test whether the two explaned variance ("mine" and the one from the previous study) differ significantly?
Note: Procedure and input variables are the same, samples are independent and differ in size.
Comment: Suppose yours explains more variance than the previous one, then what does it mean? I do not think it mean that your PCA is better than previous PCA.
Comment: - Some samples could don't come from the same distribution
- In some sample could exist lacks of data
- Did you remember about scale variables before start of PCA?
Answer: Since the "explained variance" is a proportion, you could do a test of the differences in the proportions. For instance, R has the prop.test function.
Why you are interested in this is a good question; what do you hope to do if they are significantly different? What about if they are not?
Comment: This is being automatically flagged as low quality, probably because it is so short.
At present it is more of a comment than an answer by our standards. Can you expand
on it? We can also turn it into a comment.
Comment: I added some more.

Comparison of explaned variances in PCA


What jitter value should I use for this scatter plot?

Question: I plotted a scatterplot between humidity and temperature (air) in centigrade. I got the following graph;

It is evident that the points fall in discrete columns. This might be because of the rounding of the humidity estimates to one decimal point. To break up these columns, I think some random noise (jitter) could be added to the humidity estimates. I want to know what value of jitter should I use? The new variable (jittered humidity) shall be of the form; humidity + jitter.
Comment: @Sycorax I would maintain it *is* inherently a problem because the discretization potentially (and almost certainly in this case) creates a great deal of overplotting and thereby can cause the visual impression of regions of greater or less density to be deceptive or even horribly incorrect.  See https://stats.stackexchange.com/a/506611/919 for an illustration.
Comment: What is the problem that you're trying to solve with jitter? The data being arranged in those vertical lines is not *inherently* a problem, because it reflects exactly how the data are recorded.
Comment: If distance between vertical 'lines' is $w$ then maybe jitter with $\mathsf{Unif}(-w/3,w/3)$ to (mostly) avoid overplotting while still making it clear you're using rounded data. // Similar to suggesting in @whuber's Comment. I have used this and was happy with results.
Comment: In this case, $0.01.$ So, add *iid* uniform noise supported on $[-0.005,0.005]$ (to the horizontal coordinate) as one of your initial attempts at visualization and go from there.
Comment: @Sycorax ; Though it is not inherently a problem, it is probably not how the original data had been collected. The variable has been rounded to 2 decimal points so perhaps adding some random noise will make it look more empirical.
Comment: If your goal is to make some account of the fact that the values were rounded, consider using something like [interval arithmetic](https://en.wikipedia.org/wiki/Interval_arithmetic) in your subsequent analysis.
Comment: BTW, the amount of jitter to use depends on your objective.  It's almost always a good idea to experiment.  Start with the full width of the discretization, then look at substantially less and substantially more jitter.  All three conditions can be useful, even for the same dataset.
Comment: @whuber Your point about the visual impression of density is well-taken. As an alternative, one could leave the data in vertical stacks and use transparency to convey density -- showing that it's not stacks that harm density visualization. Whether or not transparency solves the problem depends on the problem to solve. OP said they're concerned that the data don't "look empirical," so leaving the vertical lines in place wouldn't solve that. (Although I'm not sure that "looking empirical" is a problem that needs solving via plotting -- it sounds like premature rounding is the real culprit.)
Comment: @whuber ; I second to your point; "I would maintain it is inherently a problem..." for the graph that is depicted in the question showcases small scatters. It was very difficult to distinguish between these distinct bars with the normal size of the dots.
Comment: @whuber ; What is the "full length of the discretization"?
Comment: Thank you @whuber
Answer: There is not a unique way to do this, nor is it easy to formalize the space of possibilities here, so here is one way to set the jitter.
In your case, there appears to be even spacing between columns of points. For a point from a column of points at $x=a$ you could uniformly sample a new position of that point from $[a - \frac{w}{2}, a + \frac{w}{2}]$ where $w$ is the width between columns of points. This should almost-surely protect the order of points along the horizontal axis between columns while spreading out the values.


Compare rows from 2 worksheets and update a specific column

Question: Please see attached picture.
Workbook 1 is the end product of merging, comparing, formatting, and calculating datasets from a couple of sources. At the beginning of each month, one of the sources gets a staffing update. While this update is running, the data represented in Column O will contain 0's. The process takes several hours, but data is updated in this workbook each hour.
What I need to achieve is running a report after the staffing update is completed (represented by Workbook 2) and comparing the data in Workbook 2 to Workbook 1. If ALL data in each row matches between the two workbooks, except for Column O, then Column O needs to get updated from Workbook 2. I should note that the data in the two workbooks will be identical except in column O.
I have looked at other questions/answers on other forums and here in SO, but cannot find anything that matches what I need. If there is an answer out there that matches my requirements, please comment me the link.
Comment: I foresee lots of loops in your future.  That is as far as my crystal ball will take me today.
Comment: @findwindow Range.Find and Range.FindNext will loop through only those rows that the SKUs match, thus cutting out many unneeded loops.
Comment: Hmm this has been asked numerous times on SO. Not sure if it was ever answered fully...
Comment: You will need to use the Range.Find method to find the correct row then iterate through the columns testing the values.  If all are equal then use the value.
Comment: @ScottCraner is that really best architecture? I am with Karl below. Wouldn't it be easier to concat everything then compare? Edit: hmm I guess to concat, you still have to loop columns XD Otherwise, it's an array then you'd have to loop array which I guess is technically faster but not easier XD
Comment: @findwindow either way if the rows do not line up then loops are involved, either find the correct row first or loop twice trough an array, which now that I think of it may be faster.
Comment: Not sure what you mean by finding correct row. We need to test all rows? But yea, loops are unavoidable.
Comment: I must've misunderstood the question.
Comment: I don't think it is a one for one placement, as in I do not think the rows are in order.  If so then yes loading the whole into an array and looping through each row and each column would be the quickest. @findwindow
Comment: @findwindow I have updated the question with information that I hope clarifies what I need. All I really need to do is update the data in column O where all the data in the rest of the row matches between each workbook. If column ) in workbook 2 is >0 and column O in workbook 1 <> column O of workbook 2, then the value in workbook 2 column O needs to be copied to Column O in workbook 1.
Comment: Ok, that's my understanding. So yea, I would array each row and loop that for the test.
Comment: @ScottCraner You are correct, the rows will not always match up in their order between the 2 workbooks, that is what makes this tricky. Otherwise, I would just manually copy and paste. The number of rows is always growing.
Comment: Ooooohhh. Now I get it. Edit: yea, so do what Scott says.
Comment: What about my future?? @ScottCraner
Comment: @ScottCraner Great! More loops. Great! More loops. Great! More loops. Great! More loops. Great! More loops. Great! More loops. Great! More loops. Great! More loops. Great! More loops.
Answer: Take all the rows except for Row O and put them in a long string, that is (concatenate them in a line) and use an IF THEN statement comparing the differences.  If a difference is found between the rows, you can then update them with the value of Column O from Workbook 2.
Comment: As pointed out in the question, I know where the difference is and need to copy that difference from workbook 2 to workbook B, but only if all other columns in that row match.
Comment: Concatenating the rows should still work, just leave out the one row (Column O) you know is wrong, then make the compare.  Your question sounded like it wanted a broad solution.  Let me know if I'm not completely understanding your question, if there is some other way I can help.  Thanks!
Comment: Please edit your answer so I can change the down-vote. I think I jumped the gun because I misunderstood what you meant. Please explain further what you mean by concatenating the rows.


QUnit - test in loop, index always the same

Question: I am trying to test some code and for that I need to make some tests in loop, like this:
for (var i = 1; i <= 5; i++) {
    QUnit.test('Hello ' + i, (assert) => {
        console.log(i);
        assert.ok( 1 == '1', 'Result: ' + i);
    });
}

Click here for working example at jsFiddle
But for some reason i in the loop (and result) is always 6, so this code gives me output like this:
6
6
6
6
6

What am I doing wrong?
Answer: Given that QUnit defines all tests prior to running them, you're a victim of the classic problem of var scope - vars are bound to the function, not to the for loop.
What this means is:
You define your test with a given value for i, but this value will have changed when the test is actually running.
You have a few ways around this:
Create an IIFE and define your test inside it
for (var i = 1; i <= 5; i++) {
    (function (j) {
        QUnit.test('Hello ' + j, (assert) => {
            console.log(j);
            assert.ok( 1 == '1', 'Result: ' + j);
        });
    })(i);
}

Why this works: the j variable above will be bound to the scope of that IIFE. Its value will not change when the test is run.
Use let keyword in an ES6 environment
for (let i = 1; i <= 5; i++) {
    QUnit.test('Hello ' + i, (assert) => {
        console.log(i);
        assert.ok( 1 == '1', 'Result: ' + i);
    });
}

Why this works: ES6 finally introduces block scoping, but this is done via the keywords let or const.
some parts of this answer corroborated from here


Como calcular a média para grupos e identificar o valor máximo?

Question: Tenho uma planilha onde preciso achar o mês com maior média de vendas.
Com o código abaixo eu consegui, mas quando imprimo o objeto aparece todas as informações desordenadas como pode ser visto na imagem.
https://prnt.sc/xka518
Eu preciso Imprimir o mês com o maior valor faturado para todos os funcionários (exemplo: "Agosto de 2020 - Total Faturado: 15.000"
O código que estou usando.
dfseason = df.groupby(by='Month')['Billed'].sum().nlargest(1)
print(dfseason)

Quando imprimo o objeto tenho as informações como na imagem
https://prnt.sc/xka7uw
Answer: Se eu entendi o que você precisa, é possível resolver usando o método agg:
tab =df.groupby(by='Month').agg({'Billed':'sum'})
#ordenando a tabela
tab=tab.sort_values('Billed',ascending=False)
tab

Retorna:
       Billed
Month
May       918
Nov       885
Apr       735
Dec       694
Jul       634
Jun       542
Sep       539
Aug       438

O valor máimo, portanto, é obtido por tab.iloc[0,:].
Imprimindo os resultados formatados:
for k, v in tab.to_dict()['Billed'].items():
    print("{} 2020 - Total Billed {}".format(k,v))

Retorna:
May 2020 - Total Billed 918
Nov 2020 - Total Billed 885
Apr 2020 - Total Billed 735
Dec 2020 - Total Billed 694
Jul 2020 - Total Billed 634
Jun 2020 - Total Billed 542
Sep 2020 - Total Billed 539
Aug 2020 - Total Billed 438

Para imprimir apenas o maior:
print("{} 2020 - Total Billed {}".format(tab.index[0],tab.Billed[0]))

May 2020 - Total Billed 918

Comment: Usando seu método a saída sai exatamente igual a  minha.
Comment: Bom, então não entendi a pergunta. Qual o problema dessa saída? Como vc gostaria que ficasse?
Comment: Eu preciso manipular esse objeto e dar um print exemplo.
May 2020 - Total Billed : 918
Comment: Entendi agora. Vou editar a resposta
Comment: Está retornando oito valores iguais (já que são oito meses com valores). Seria possivel retornar apenas um valor?
Answer: Uma forma de retornar os valores de forma "ordenada" é utilizar o reset_index
dfseason = df.groupby(by='Month', sort=True)['Billed'].sum().nlargest(1).reset_index()
dfseason

Saída
    Month   Billed
0   May     918

Para customizar uma frase
print(f"Mês {dfseason.loc[0,'Month']} - Total faturado {dfseason.loc[0,'Billed']}")

Saída
Mês May - Total faturado 918


Linux Central Authentication/Authorization Methods

Question: I have a small but growing network of Linux servers. Ideally I'd like a central place to control User Access, change passwords, etc... I've read a lot about LDAP servers, but I'm still confused about choosing the best authentication method. Is TLS/SSL good enough? Whats the benefits of Kerberos? Whats GSSAPI? Etc... I haven't found a clear-cut guide that explains the pros/cons of these different methods. Thanks for any help.
Comment: Do't forget NIS.  It may be old, but it's still widely used for a reason.
Comment: I think that reason may be appliances and other operating systems that don't fully support LDAP. If you have a 100% Linux shop, then LDAP is the way to go. Solaris has trouble talking to OpenLDAP using OpenSSL for encryption. FreeBSD can't use netgroups over LDAP. You can always set up an NIS gateway for non-authentication data needed by incompatible systems.
Comment: @MadHatter I'm pretty sure that reason NIS is still used is called "Inertia".
Answer: For this problem, FreeIPA is the "best" FOSS solution out there.
Since you are just starting to learn about the scope of your problem, you should do your research before attempting to play with FreeIPA.
Answer: TLS encryption is good enough to secure the transmission of passwords from the clients to the server given the following:

Your LDAP server's ACLs properly restrict access to password hashes.
Your server's private key is never compromised.

TLS encrypted plain authentication is the most simple method of secure authentication to set up. Most systems support this. The only prerequisite your client systems have is getting a copy of your SSL certificate authority's certificate.
Kerberos is mainly useful if you want a single sign on system for your workstations. It would be nice to be able to log in once and have access to web services, IMAP email, and remote shells without entering your password again. Unfortunately, there is a limited selection of clients for kerberized services. Internet Explorer is the only browser. ktelnet is your remote shell.
You may still want to encrypt traffic to your kerberized LDAP server and other services with TLS/SSL to prevent traffic sniffing.
GSSAPI is a standardized protocol for authentication using back ends such as Kerberos.
Answer: LDAP works well for multiple servers and scales well.  startTLS can be used to secure LDAP communications.  OpenLDAP is increasing well supported and more mature.  Master-master replication is available for redunancy.  I have used Gosa as an administrative interface.
I still haven't bothered limiting access per server, but the facility is there.
You may also want to look at shared home directories using autofs, or some other network mount mechanism.  It not you will likely want to add the pam module that creates missing home directories on first login.
While NIS (aka yellowpages) is mature, it also has some reported security issues.
Answer: If you're looking for a straightforward solution for your local network, Sun'S Network Information Service is convenient and has been around for a long time.  This link and this one describe how to set up both the server and client instances.  LDAP services, such as described here, can provide the centralized administration you want as well.
That said, if you need higher levels of security, you may want to go with other packages.  TLS/SSL won't work for initial login unless you have separate dongles/smartcards or something similar.  Kerberos can help, but requires a secured, trusted server.  What are your needs?
Comment: Well right now my needs are strictly for a central authentication server, so I only have to change a password in one place instead of every server. But I'd like a solution that scales well so when I need more granular access controls, I can easily add it. Thats why I was looking at LDAP as opposed to NIS.
Comment: I think Chris was referring to TLS/SSL only to encrypt the network traffic between the LDAP client and server. In that case, you don't need any extra hardware.


About the theorem for an open subset $O$ of $R^{d}$.

Question: Theorem: Every open subset $O$ of $R^{d}$, $d\geq1$ can be written as a countable union of almost disjoint closed cubes.
Let's take an example for $d=1$. We take the interval $(0,1)$. If I understand correctly, we say that the union of two closed intervals $[0,0.5]$ and $[0.5,1]$ represents our interval.
These two intervals are almost disjoint because there interior points are disjoint. Theire union is $[0,1]$.
But our interval $(0,1)$ and the union $[0,1]$ are different intervals. I know that the length is the same. But theorem say nothing about the length.
Where am I wrong?
Answer: To write $(0, 1)$ as a (countable) union of almost disjoint closed intervals, you need infinitely many closed intervals, since a finite union of closed intervals is a closed set. However, the almost disjoint intervals $[1/(n+1), 1/n]$, $[1 - 1/n, 1 - 1/(n+1)] $, for $n = 2, 3, \ldots$ do cover $(0, 1)$. Read the proof of the theorem to see how this works out in general for an arbitrary open subset of $\Bbb{R}^d$.


Pick Makes Clicking Sound

Question: When I play my guitar, my pick makes a clicking sound every time it hits the pickguard. I was trying to record a song the other day and the clicking sound is annoying. Am I holding my pick wrong? Is there a technique on how to play to avoid making this clicking sound?
Comment: Welcome to the site! [This related question](https://music.stackexchange.com/questions/624/how-can-i-prevent-the-noise-my-guitar-pick-makes#:~:text=6%20Answers&text=First%2C%20sound%20is%20perceived%20when,Thus%2C%20pick%20sounds%20are%20unavoidable.) may help you, I think...
Answer: Not a clue what guitar you use, but pickguards are at least 1/4" lower than the strings, so you shouldn't be anywhere near with your pick, strumming.
Also, once the strings have been strummed, there's no point in continuing the downstroke - there's nothing else to do except change direction of your hand/arm.
It could be that you have too much pick protruding so it catches on the guard. Or you dig in after strumming the strings. A picture or soundbite might help. A teacher would help even more!

Pick Makes Clicking Sound


Styling button click text

Question: When you click an button HTML button, button goes inwards hence making the text inside the button move downwards slightly. My question is, how can I stop the button changing (going inwards) when it is clicked?
I hope you can understand my question.
Comment: What's your html and css for the button you're asking about? It's usually to do with borders, padding, margins; but without seeing what's happening with your CSS, I've no way to know what to suggest.
Answer: After you have done whatever you like with the border etc., just put a span inside the button around the text like so:
<button class="button" type="submit">Blah</button>
Then the CSS becomes:
button {position:relative; width:40px; height:20px /* set whatever width and height */}

buttonspan {
height: 30px;
left: 0;
position: absolute;
top: 0;
width: 100%;
}

Making sure to set the span absolute and the button relative.. works like a charm.
Answer: Most cross browser friendly and simple answer is probably to replace it with an image - this will give you the most control as all browsers treat buttons differently.. even if you were to try a border: 0 style, the text in the button may still move down in some browsers.

Styling button click text


How to invisible line in list view for android Honey comb

Question: I m creating an app in that app i want to invisible this Mr. bla bla two line when i clicked on chat image.
any suggestion will be appriciate.
thanks in advance.
this is my xml file.
<RelativeLayout
  android:layout_width="wrap_content"
  android:layout_height="wrap_content"
  android:id="@+id/task_list_frag_list_message_layout"
  android:background="#EBEBEB"
  android:layout_marginBottom="10dp"
  android:descendantFocusability ="blocksDescendants"
  android:layout_marginTop="10dp"
  android:layout_below="@+id/task_list_text"
  android:layout_marginLeft="80dp"
  >

   <TextView
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:id="@+id/task_list_chat_persion1"
    android:text="Mr. Test : "
    android:textSize="18dp"
    android:layout_marginTop="13dp"
    android:textColor="@color/sky_blue_color"
    />
     <TextView android:textColor="@color/black_color"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:id="@+id/task_list_chat_text1"
    android:text="A send you a data file kindly get it."
    android:textSize="18dp"
    android:layout_marginTop="13dp"
    android:layout_toRightOf="@+id/task_list_chat_persion1"
    />

     <TextView
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:id="@+id/task_list_chat_persion2"
    android:text="Mr. me : "
    android:textSize="18dp"
    android:layout_marginTop="13dp"
    android:layout_below="@+id/task_list_chat_persion1"
    android:textColor="@color/sky_blue_color"
    />
     <TextView android:textColor="@color/black_color"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:id="@+id/task_list_chat_text2"
    android:text="I got the data file and i have some doubt in them"
    android:textSize="18dp"
    android:layout_marginTop="13dp"
    android:layout_below="@+id/task_list_chat_persion1"
    android:layout_toRightOf="@+id/task_list_chat_persion2"
    />

  </RelativeLayout>

this two lines are ij relative latout.
Comment: can i GONE or INVISIBLE whole relative layout?
Answer: I think you want this ...
just need to add onClick() event on chat image and then on clicking set visibility of TextView to invisible by code.
TextView _text= (TextView)findViewById(R.id.task_list_chat_text);
        _text.setVisibility(TextView.INVISIBLE);

Answer: When you click on chat button in the list view you should setTest="" blank for both remove it.
It may be solution of this problem.


JavaFX ListView

Question: I have a ListView<String> list. I can easily get the next item in the ListView with a button using this code:
 public void toNext(){
    list.getSelectionModel().selectNext();
 }

But if I want to go through the ListView automatically with a button, it won't work. I tried the following code:
 public void play(){
    try {
        for (int i = 0; i < list.getItems().size() ; i++) {
            list.getSelectionModel().select(i);
            Thread.sleep(500);
        }
    } catch (InterruptedException ex) {
        ex.printStackTrace();
    }
 }

This code doesn't select the different strings in the list but just skips to the last string after the given waittime. How can I get it to select each string in the list with a delay of 500 ms?
Comment: This is probably because you do not call `lijst.getSelectionModel().select(i);` in the UI thread, meaning play is not called on the UI thread, as it should not be.
Comment: Sorry that was just a typo but that's not the problem ;) @hotzst
Answer: The problem is, your loop completely runs in FxApplicationThread.
So, the UI won't update during the loop.
Better would be something like:
public void play(){
    Thread t = new Thread(new Runnable() {
        @Override
        public void run() {
            for (int i = 0; i < lijst.getItems().size(); i++) {
                final int idx = i;
                Platform.runLater(new Runnable() {
                    @Override
                    public void run() {
                        lijst.getSelectionModel().select(idx);
                    }
                });
                try {
                    Thread.sleep(1000);
                } catch (InterruptedException ex) {
                    ex.printStackTrace();
                }

            }
        }
    });
    t.start();
}

Answer: Consider using an animation:
Timeline animation = new Timeline(
    new KeyFrame(Duration.millis(500), e -> list.getSelectionModel().selectNext()));
list.getSelectionModel().select(0);
animation.setCycleCount(list.getItems().size()-1);
animation.play();

Advantages to this approach include

It is lightweight, and doesn't involve creating a new thread of execution. It simply hooks into the existing FX Application Thread and pulse mechanism.
The code is cleaner.
It has additional built-in API, so it is much easier to stop or pause the animation than it would be if you created a new thread to manage this.
Answer: You have to wrap the call that makes the UI update in the UI thread otherwise all the updates are done in one bunch at the end, hence it jumps to the end of the list.
 public void play(){
    try {
        for (int i = 0; i < list.getItems().size() ; i++) {
            final int index = i;
            Platform.runLater(() -> {
                list.getSelectionModel().select(index)
            });
            Thread.sleep(500);
        }
    } catch (InterruptedException ex) {
        ex.printStackTrace();
    }
 }

Comment: It will wait 500ms and then select the first one and the last one, but the ones in between won't get selected...

JavaFX ListView


How to handle CSV file where address column data's are seperated by comma as a single column instead of multiple column in Spark

Question: If I have incoming data for address field in CSV file is seperated by comma(','), how can I handle this in Spark?
If I want that data as record to my address column.
Example-Suppose I have input data in form of CSV file as below,
Bob,Delhi,NCR,8984124789
Scott,Bangalore,Karnataka,9040788301
Robert,Andheri,Mumbai,Maharastra,9338075922

I want final dataframe as,
Name   Address                    MobileNo
Bob    Delhi,NCR                  8984124789
Scott  Bangalore,Karnataka        9040788301
Robert Andheri,Mumbai,Maharastra  9338075922

How can we can handle this in Spark?
Comment: Please take a look at this and see if you can apply it using python: https://stackoverflow.com/a/60965746/5761558
Comment: Change your delimiter something else other than the delimiter inside your address column.
Answer: You can use an RDD to clean up the data before making a data frame from it:
rdd = sc.textFile('path-to-csv.csv')

Then clean it up by shifting the last column in order to isolate the Address data to the end of the line:
df = rdd.map(lambda l: l.split(','))\
        .map(lambda l: Row(Name=l[0],Mobile=l[-1],Address=', '.join(l[1:-1])))\
        .toDF()

And to explicitly set the schema:
df = rdd.map(lambda l: l.split(','))\
    .map(lambda l: Row(Name=l[0],MobileNo=l[-1],Address=', '.join(l[1:-1])))\
    .toDF(schema=StructType(fields=[StructField('Name',StringType()),
                                    StructField('Address', StringType()),
                                    StructField('MobileNo', StringType())]))

The result:
+------+---------------------------+----------+
|Name  |Address                    |MobileNo  |
+------+---------------------------+----------+
|Bob   |Delhi, NCR                 |8984124789|
|Scott |Bangalore, Karnataka       |9040788301|
|Robert|Andheri, Mumbai, Maharastra|9338075922|
+------+---------------------------+----------+

Comment: compared to the other wrong answer above, this is a nice answer. Clearly, the obvious issue is to generate such a weird and imo ill-formed CSV instead of either wrapping column values inside quotes " " or using a different separator like TSV
Comment: @UninformedUser Good point. This should only be done if changing the delimiter is impossible. I also thought one could just move bits of substrings before using `read.csv` on a dataset, but pyspark doesn't seem to have such an option (unlike scala or java). Would be nice to see if someone knows of such an alternative (to splitting into unnecessary lists when one can move substrings followed by a powerful spark.read.csv...)


Remove or hide the "View" tab on a node

Question: I know how to rename "edit" or "track" tab on user page and node page. Here's the code, it might help someone.
<?php

function MYMODULE_menu_alter(&$items) {
//    drupal_set_message('Menu Items:
'.print_r($items, TRUE).'
');
    $items['user/%user/view']['title'] = "Profile";
    $items['user/%user/edit']['title'] = "Profile edit";
    $items['node/%node/edit']['title'] = "Node edit";

}

...but I don't know how to remove or hide the "View" tab on created node.
my url page of the node is content/(node title).
What snippet should I use to remove the "View" tab on node page?
Comment: This question has been already answered on stackoverflow http://stackoverflow.com/questions/3530756/how-to-hide-edit-view-tabs
Comment: I think the view tabs don't appear if there is only one tab. So if you have devel enabled and have view and devel as tabs, the view will disappear when you disable devel.
Comment: I have two tabs. "View" and "Edit". I've renamed the second one, but I want to hide only "View" tab, so only "Edit" tab will be visible on node page.
Answer: You should be able to do this using css, or try using the code below or the tab tamer module
function MYMODULE_menu_alter(&$items) {
  $items['node/%node/view']['access callback'] = 'MYMODULE_disable_node_view';
  $items['node/%node/view']['access arguments'] = array(1);
}

function MYMODULE_disable_node_view($node){
        return false;
}

Comment: I've tried the code and there's is one notice "Notice: Undefined offset: 0" and warning "Warning: Invalid argument supplied for foreach()". It's for this line: **foreach ($data['tabs'][0]['output'] as $key => $value) {**
Comment: try the new update
Comment: I've tried the new code. It doesn't have any error and the "View" tab is hidden now. First I've checked with admin user, but with authenticated user there are no tabs on page node, so the edit tab is hidden too.
Comment: With your new code the view tab is hidden on both roles, but with authenticated user now the "Edit" tab is hidden too and that's why there are no tabs.
Comment: Does the authenticated user you used have permission to edit nodes?
Comment: Yes. Without your code the authenticated user has tabs "view" and "edit". I just only want to hide "View" tab and that's only on node pages, so only "Edit" tab on node page will appear.
Answer: If you only want to hide the tab but maintain the access you could alter the menu callback type this way:
/**
 * Implements hook_menu_alter().
 */
function yourmodule_menu_alter(&$items) {
  // Example 1: hide tabs on user/* for anonymous:
  $items['user/login']['type']    = MENU_CALLBACK;
  $items['user/register']['type'] = MENU_CALLBACK;
  $items['user/password']['type'] = MENU_CALLBACK;
  // Example 2:
  $items['user/%user/view']['type'] = MENU_CALLBACK;
  $items['user/%user/edit']['type'] = MENU_CALLBACK;
  $items['node/%node/edit']['type'] = MENU_CALLBACK;
}

Answer: Since your requirement is more of a presentation issue you could either use the hidden option in the Tab Tamer Module or a simple css hide hack.
If your requirement is related to user access then you could use the code hack for access callback or disabled option in Tab Tamer Module.
Answer: For Drupal 8: you can use hook_menu_local_tasks_alter

/**
 * Hide view tab from node edit.
 */
function MYMODULE_menu_local_tasks_alter(&$data, $route_name, \Drupal\Core\Cache\RefinableCacheableDependencyInterface &$cacheability) {
  if($route_name == 'entity.node.edit_form'){
    unset($data['tabs'][0]['entity.node.canonical']);
  }
}

Remove or hide the "View" tab on a node


Whitespace in webpage even after using display:hidden to hide objects

Question: Here is my code in JsFiddle (so you can also see the code in action) - https://jsfiddle.net/2yvjm4h1/
Despite using display:none; to hide my objects instead of visibility:hidden to hide my form objects (before conditionally displaying them with javascript), there is still a ton of whitespace between the textarea element and the first file upload element. Once the user clicks the buttons to add all 10 file upload elements on the screen, the white space will be gone. But I want the white space to always be gone, and the page size to dynamically increase.
How do I eliminate white space from the start?
Comment: You have a `` after each file input row. Get rid of those and you should be good
Answer: It looks like you have a br after every div. Try either removing them or putting them inside the divs that have display:hidden. See my jsfiddle fork.
EDIT: if you are planning to unhide the divs, they won't have any breaks. To fix that, put the line breaks inside the divs.


Show that $\sum r^n \cos(nx)=r\cos(x)-r^2/(1-2r\cos(x)+r^2)$

Question: I'm a little unsure about how to approach this.  I've been told that we have to use the relationship that $\sum r^n=1/1-r$. However, I'm not too sure what to do with the $\cos(nx)$.  Can someone give me some help?
Comment: Hint: $$\sum_n r^n\cos(nx)=\Re\sum_n r^ne^{inx}=\Re\sum_n(re^{ix})^n=\ldots$$
Answer: Let me continue @Did answer.
$$\Sigma=\mathcal{R}\frac{1}{1-r\cdot e^{ix}}$$
Let us multiply numerator and denominator by $1-r\cdot e^{-ix}$. So
$$\Sigma=\mathcal{R}\frac{1-r\cdot e^{-ix}}{1-2r\cdot\mathcal{R}e^{ix}+r^2}=\frac{1-r\cos{x}}{1-2r\cos{x}+r^2}$$
Comment: Why do you only take the real component though?  What happens to the imaginary part?
Comment: Because the sum is the sum of $r^n\cos{x}=\mathcal{R}(r\cdot e^{ix})^n$


Organizar pacotes num projeto java

Question: Quando se cria um projeto no Android, a própria IDE sugere que o pacote principal tenha um tipo de nomeação própria (com.example.nomedopacote), e também cria toda uma hierarquia de diretórios já definidos.
Assim como acontece no Android, há alguma convenção a se seguir ao programar em java (seja desktop ou web), sobre organização e nomenclatura dos pacotes da aplicação, ou somente o MVC é aplicável para este tipo de organização?
Answer: A própria Oracle definiu certas convenções para evitar conflitos entre classes Java de diferentes programadores/empresas/projetos.
Ela sugere que uma empresa (ou qualquer programador) utilize seu próprio domínio web invertido como nome dos seus pacotes (visto que o domínio é único). Exemplo: minha empresa possui o domínio www.empresaficticia.com.br. O pacote principal dos meus projetos seria definido como segue:
package br.com.empresaficticia;

Para os casos em que o domínio web não é um nome de pacote válido (possui hífen, pontos, começa com dígitos, etc), a Oracle sugere a utilização do underscore _. Exemplo: meu domínio é www.empresa-ficticia.com.br, então meu pacote seria:
package br.com.empresa_ficticia;

Além dessa organização top level sugerida pela Oracle, o resto se torna basicamente opinião e preferência de cada programador/empresa. Uma dica para quando seu projeto ficar muito grande é separá-lo em módulos que tratem de apenas uma categoria de funcionalidades. Exemplo: você possui X classes em seu projeto que lidam apenas com acesso à database, então coloque-as em um sub-pacote db:
package br.com.empresa_ficticia.db;

e um outro sub-pacote poderia se chamar package br.com.empresa_ficticia.fs; e conteria suas classes que lidam com arquivos do sistema operacional.
E uma dica final, dê uma olhada em projetos grandes de código aberto. Analise como eles são organizados. Alguns projetos java com os quais já tive contato e recomendo para ter uma maior ideia do assunto: Apache Hadoop, Apache Giraph e Apache Hama.
Mas antes de sair criando pacotes com nomes que possuam 40 caracteres, uma magia de invocação e o sangue de uma virgem, analise se isso é realmente necessário. Se você estiver apenas fazendo um pet project (projeto para aprender algo novo, que não vai realmente ser utilizado por ninguém), dificilmente vai ser necessário encapsular seu projeto em um pacote.
Referência: https://docs.oracle.com/javase/tutorial/java/package/namingpkgs.html

Organizar pacotes num projeto java


tableView.insertRows throws NSException, but not when `insertRows` at `[[0,0]]`

Question: I am following a modified (simplified) version of the tutorial very much like what is found here:
https://developer.apple.com/library/content/referencelibrary/GettingStarted/DevelopiOSAppsSwift/ImplementNavigation.html#//apple_ref/doc/uid/TP40015214-CH16-SW1
and here is my UITableViewController:
import UIKit

class NewsTableViewController: UITableViewController {

    @IBOutlet var newsTableView: UITableView!

    /*
        MARK: properties
    */
    var news = NewsData()

    override func viewDidLoad() {
        super.viewDidLoad()
        dummyNewData()
    }

    // MARK: - Table view data source
    override func numberOfSections(in tableView: UITableView) -> Int {
        return 1
    }

    override func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
        return news.length()
    }

    // here we communicate with parts of the app that owns the data
    override func tableView(_ tableView: UITableView
                           , cellForRowAt indexPath: IndexPath
                           ) -> UITableViewCell {

        // note here we're using the native cell class
        let cell = tableView.dequeueReusableCell(withIdentifier: "newsCell", for: indexPath)
        // Configure the cell...
        let row : Int = indexPath.row
        cell.textLabel?.text = news.read(idx: row)
        return cell
    }

    // MARK: Navigation ****************************************************************

    // accept message from CreateNewViewController
    @IBAction func unwindToCreateNewView(sender: UIStoryboardSegue){

        if let srcViewController = sender.source as? CreateNewsViewController
        , let msg = srcViewController.message {

            // push into news instance and display on table
            news.write(msg: msg)
            let idxPath = IndexPath(row: news.length(), section: 1)

            // tableView.insertRows(at: [idxPath], with: .automatic)
            tableView.insertRows(at: [[0,0]], with: .automatic)

            print("unwound with message: ", msg, idxPath)
            print("news now has n pieces of news: ", news.length())
            print("the last news is: ", news.peek())

        }

    }

    /*
        @DEBUG: debugging functions that display things on screen **************************
    */
    // push some values into new data
    private func dummyNewData(){

        print("dummyNewData")
        news.write(msg: "hello world first message")
        news.write(msg: "hello world second message")
        news.write(msg: "hello world third message")
    }

}

The problem is in the function unwindToCreateNewView:
let idxPath = IndexPath(row: news.length(), section: 1)
tableView.insertRows(at: [idxPath], with: .automatic)

where news.length() gives me an Int that is basically someArray.count.
When I insertRows(at: [idxPath] ...), I get error:
libc++abi.dylib: terminating with uncaught exception of type NSException

But when I just hard code it to do:
tableView.insertRows(at: [[0,0]], with: .automatic)

It works just fine. And on the simulator I see new messages are inserted below the previous ones. What gives?
Comment: You need to edit your question to include the exception text; it will tell you what went wrong, but in your first code your are inserting into section 1, but in your second code you are inserting into section 0. your table only has 1 section, so inserting into the second section (section 1) won't work
Comment: You've hardcoded the number of sections as 1.  Since the section numbers begin at 0, any attempt to insert a row in section 1 will trigger an exception.
Answer: You have an "off by one" problem with the following code:
news.write(msg: msg)
let idxPath = IndexPath(row: news.length(), section: 1)

Let's say that just before this code is called, you have no items in news. This means there are 0 rows. When you want to add a new row, you need to insert at row 0 since row numbers start at 0.
Calling news.write(msg: msg) add the new item and its length is now 1.
Calling IndexPath(row: news.length(), section: 1) sets the row to a value of 1 but it needs to be 0.
One simple solution is to swap those two lines:
let idxPath = IndexPath(row: news.length(), section: 1)
news.write(msg: msg)

That will create the index path with the proper row number.
And since this is the first (and only) section, the section number needs to be changed to 0 in addition to the above change.
let idxPath = IndexPath(row: news.length(), section: 0)
news.write(msg: msg)


BinaryMessageEncoding with HTTPSTransport

Question: Actually I'm having problems implementing both HTTPS and compression on the same WCF Web Service, but the problem is easier to explain using BinaryMessageEncoding, and the cause is propably the same.
I have configured both the service and the client like this:
<customBinding>
  <binding name="OwnBinding" closeTimeout="00:10:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00">
    <binaryMessageEncoding>
      <readerQuotas maxDepth="999999999" maxStringContentLength="999999999" maxArrayLength="999999999" maxBytesPerRead="999999999" maxNameTableCharCount="999999999" />
    </binaryMessageEncoding>
    <httpsTransport maxBufferSize="999999999" maxReceivedMessageSize="999999999" authenticationScheme="Anonymous" proxyAuthenticationScheme="Anonymous" useDefaultWebProxy="true"/>
  </binding>
</customBinding>

But it keeps giving me following CommunicationException:
An error occurred while making the HTTP request to https://localhost:8036/Services/FileService. This could be due to the fact that the server certificate is not configured properly with HTTP.SYS in the HTTPS case. This could also be caused by a mismatch of the security binding between the client and the server.
InnerException: {"The underlying connection was closed: An unexpected error occurred on a send."}
I know it suggest this is due to a invalid sertificate, and I have tried this also with this trick, but it still doesn't work. (I don't have a sertificate, and really don't need one. I just need SSL encryption and compression. No need to authenticate either server or client.
ServiceTraceViewer doesn't show anything useful either.
Answer: You need a server certificate in order to enable HTTPS. This is a requirement since the Transport Level Security (TLS) protocol uses asymmetric cryptography in order to exchange the session key with the client through a secure channel.
You can create a temporary self-signed certificate to use during development and install it in the CurrentUser\My store using the makecert utility:
makecert -n "CN=MyTestCertificate" -r -ss my

Then, you can easily configure your WCF service to use that certificate using the <serviceCertificate> element in a custom service behavior:
<behaviors>
  <serviceBehaviors>
    <behavior name="SecureService">
      <serviceCredentials>
        <serviceCertificate findValue="CN=MyTestCertificate" />
      </serviceCredentials>
    </behavior>
  </serviceBehaviors>
</behaviors>

If you're hosting your service in IIS, you also have to enable SSL on the web site where the service is running before you'll be able to communicate with it through HTTPS.
Answer: Actually, in order to do HTTPS comminucation, you need a certificate. You dont have to buy one for testing purposes, you can make your own using makecert.exe
MakeCert (Makecert.exe) is a command-line tool that creates an X.509 certificate that is signed by a system test root key or by another specified key. The certificate binds a certificate name to the public part of the key pair. The certificate is saved to a file, a system certificate store, or both.
Please see link to information on parameters.
Inorder to create the certificate try this on the cmd line.
makecert -r -pe -n "CN= compaq-jzp37md0 " -b 01/01/2000 -e 01/01/2050 -eku 220.127.116.11.188.8.131.52.1 -ss my -sr localMachine -sky exchange -sp "Microsoft RSA SChannel Cryptographic Provider" -sy 12

See this link to find out how to assign SSL cert to your website.
P.S. The trick your mentioning is for accepting an invalid or expired certificate. It wont work, if you don't have a certificate.


Orthogonal contrasts and the poly function in R: Interpreting the coefficients as "absolute" or "incremental" effects

Question: in a growth curve like model, I try to test whether a trajectory (over time) is better described by a linear or quadratic trend. I fitted a linear mixed model (time nested in participants) and called orthogonal polynomial contrasts using the poly() function in R:
mod <- lmerTest::lmer(outcome ~ poly(time, 2) + (poly(time, 2) | id), data=dat.long)
parameters::parameters(mod) returns the following output:

Parameter
Coefficient
SE
95% CI
t(1215)
p

(Intercept)
6.75
0.15
[  6.46,   7.04]
45.35
< .001

Time [1st degree]
-58.01
4.08
[-66.01, -50.02]
-14.22
< .001

Time [2nd degree]
-9.00
2.56
[-14.02,  -3.98]
-3.51
< .001

I now wonder which of the following interpretations is correct:
(1) The test statistics refer to the "absolute" fit of the corresponding trends.
Thus, the linear model fits the data better (|t| lin > |t| qua) than the quadratic model.
(2) The quadratic term describes the increment of the quadratic model (vs. linear term only), thus suggesting that the quadratic model fits the data better (t(1215) = -3.51, p < .001).
(I suspect it is (2), but I would very much appreciate any comments/advice.)
Thank you so much for your help!
Marcel
Answer: There seems to be some confusion.
You have fitted one model that has a fixed intercept, a linear term and a quadratic term, plus some random effects.

(1) The test statistics refer to the "absolute" fit of the corresponding trends. Thus, the linear model fits the data better (|t| lin > |t| qua) than the quadratic model.

This does not make sense because you have not fitted two models.

(2) The quadratic term describes the increment of the quadratic model (vs. linear term only), thus suggesting that the quadratic model fits the data better (t(1215) = -3.51, p < .001).

Again this does not make sense for the same reason.
Comment: Does this answer your question ? If so please consider marking it as the accepted answer. If not, please let us know why. Also, if you haven't already, please consider upvoting it.
Comment: Thanks a lot for your helpful answer! To make sure I interpret the results correctly:
(1) Can the second row (i.e., 1st degree) be interpreted as a negative linear trend?
(2) Does the significance of the quadratic term mean that we should _not_ drop the quadratic component of the model?

I remember being tought that the first polynomial does not reflect a linear trend but the "velocity" at x = 0. Here, however, where orthogonal contrasts are used, the linear component remains the same, no matter whether this is the only component in the model or whether I add a quadratic term (see above).


Modifying O365 service plan with PowerShell

Question: For the roll out of Office 365, we did not want to give all our users access to all the available applications/plans. So we only enabled Exchange, and Skype.
Now we're at the point where we want to enabled Yammer, and Office online, and we're running into issues.
$LicSKU = "<Hidden>:STANDARDPACK"
$ServicePlans = ((Get-MsolAccountSku | Where-Object {$_.AccountSkuId -eq $LicSKU}).ServiceStatus | Select-Object ServicePlan -ExpandProperty ServicePlan).ServiceName
$EnabledPlans = 'EXCHANGE_S_STANDARD','YAMMER_ENTERPRISE', 'SHAREPOINTWAC', 'MCOSTANDARD'
$DisabledPlans = @()
foreach($Plan in $ServicePlans) {
    if($EnabledPlans -notcontains $Plan) {
        $DisabledPlans += $Plan
    }
}
$LicOption = New-MsolLicenseOptions -AccountSkuId $LicSKU -DisabledPlans $DisabledPlans
try {
    $UserLicense = @{
        UserPrincipalName = $UserPrincipalName
        AddLicenses = $LicSKU
        LicenseOptions = $LicOption
    }
    Set-MsolUserLicense @UserLicense -ErrorAction Stop
}
catch [Microsoft.Online.Administration.Automation.MicrosoftOnlineException] {
    $UserLicense.Remove('AddLicenses')
    Set-MsolUserLicense @UserLicense -ErrorAction Stop
}

The error I get when I run this is the following :
Set-MsolUserLicense : Unable to assign this license.
+         Set-MsolUserLicense @UserLicense -ErrorAction Stop
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [Set-MsolUserLicense], MicrosoftOnlineException
    + FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.InvalidLicenseConfigurationException,Microsof
   t.Online.Administration.Automation.SetUserLicense

If I set the $DisablePlans = $null, then the same command runs fine without issues.
My current work around is to remove the license from the user, then add it back on with the updated plans enabled.
I'm hoping somebody has run into this problem before and found a proper solution for this.
Answer: Well, I'm an idiot. I went through the GUI and tried to add Office Online and it complained that it requires Sharepoint as well.
So when I updated the code to
$ServicePlans = ((Get-MsolAccountSku | Where-Object {$_.AccountSkuId -eq $LicSKU}).ServiceStatus | Select-Object ServicePlan -ExpandProperty ServicePlan).ServiceName
$EnabledPlans = 'EXCHANGE_S_STANDARD','YAMMER_ENTERPRISE', 'MCOSTANDARD', 'SHAREPOINTSTANDARD', 'SHAREPOINTWAC'
$DisabledPlans = @()
foreach($Plan in $ServicePlans) {
    if($EnabledPlans -notcontains $Plan) {
        $DisabledPlans += $Plan
    }
}

I was able to modify the license without issues.
Answer: You could also try using Azure Portal for license management (Azure Active Directory - Licenses).
It supports assigning licenses to user groups.
So you would only need to do assignment once for some group which includes all users. And you would never need to do any assignment when new users are included.
Comment: Thanks you for mentioning this option, I was looking into this recently and you need paid Azure AD to use this option, while the assign to user is free.


Send email notification from a web app

Question: I'm building a web app from which I need to send out email notifications to the users who subscribe for them. Say my domain name is mywebapp.com. I'm using Google Apps to host my email service currently. Ideally I would like to send all my notifications using Google Apps itself and that's what I've been doing all this while using msmtp. But then Google Apps has a limit to the number of emails that can be sent per day which is around 4000 emails.
So is it possible to use Google Apps to host my primary email service for mywebapp.com and then run my own email server to just send email from say firstname.lastname@example.com.
This way, I can still use google apps for checking my regular email and also eliminate the upper limit on the number of emails I can send out per day, since I'll be running my own server.
Also, what packages do I need to install on Ubuntu?
I've looked at postfix, but as far as my understanding goes it is a package that can fetch email. Can I use it to send email using a cron job for example?
Links to useful resources would be helpful.
Answer: Postfix is not really a tool to "fetch mail". That is "fetchmail" :-)
Depending on where is hosted your web app, you may already have a local SMTP server available to send your mails (supplied by most providers).
If not, Postfix (or any other MTA, or in other words, "SMTP server") is what you need. You should configure it just to send mail, without being responsible for any domain. This configuration is mostly "out-of-the-box" - but a minimum knowledge is suggested.
In that case, start with "www.postfix.org".
Comment: I realised that about postfix a little while later, after reading up more! :)
Comment: http://www.linode.com/forums/viewtopic.php?t=3946
http://www.informit.com/articles/article.aspx?p=24672

I found these articles helpful and almost got what I wanted! Thanks :)

Send email notification from a web app


How to properly unresolve melodies to achieve dissonance and darkness in music

Question: I'm looking to understand music theory behind dissonance, unresolved dark music. What makes a song sound serious and not "rainbows". When writing a bassline or melodies, is it possible to achieve darkness in all 12 major scales? Or is it only necessary to use minors and some other modes? I know about tritones and that no matter what scale, you can make tritones and achieve dissonance. But what other ways I can achieve this? I know that close intervals sounds more dissonant like minor 2nd, but then that doesn't explain why tritone sounds dissonant when it's sits next to perfect 5th.
I want to know what intervals should  I play and how to to end my melody progression. I know that I should not resolve to tonal center at the end of progression but that doesn't always work either. Like I-VI-II-VI would be a good example of unresolved progression? It doesn't end on tonal frst note but on 6th degree which makes tension I suppose?
Looking for all the logic behind dark music from theory side. I just learned all the 12 major scales and their fingerings and now I when I jam on my synthesizer, my basslines and melodies sounds cheesy.
Here is the example of sound I want to achieve:
Comment: You have a lot of questions in one. It's a bit of a big topic, maybe too broad for this site. Two things I suggest: 1) learn to play the music that sounds dark to you so you see how other musicians do it. Experience is much more useful than theory. 2) keep writing bass lines and melodies but when you come up with an idea that sounds cheesy, change it. You just found a way to **not** write dark sounding melodies. Remember it and don't do it again. If you have a cheesy four note melody change two of the notes, maybe by just a half step. Experiment and keep doing things you never did before.
Comment: @ToddWilcox I think there is a valid question in there, with already a nice answer. OP could just simplify it... no?
Comment: @ToddWilcox Fair enough. I was really rushing when creating this post, so it's all over the place. Tomorrow when I have time, I will restructure it properly.
Answer:
I know that close intervals sounds more dissonant...

It's not the closeness, but rather about the ratio between the two pitches where the idea is 'simple' ratios sound consonant. Look at the 'pitch ratios' on this wiki chart.
But, that is just a technicality. It seems to me you have a good idea how musical elements work. Dark music can be achieved with: minor tonality, dissonant intervals, unresolved dissonances.
From the two clips I notice: the tempos are fairly slow for EDM, and the harmonic rhythm is slow meaning the chords are held a long time before a change is made. Holding something for a long time is a way to build up tension.
I think your observation about not resolving is a good insight. You could say that leads to a sort of static quality. If normally tension and resolution lead to a dynamic feel - a feel of movement and progression, then a constant tension doesn't really move. You stay stuck in a state of anxiety.
Avoiding strong chord progression could heighten that static quality. You could try avoiding V to I or other root progression by descending fifth. I don't mean avoid entirely, but this is something to pay attention to.
Consider the modal scale degrees and the color shading you can get with altered scale degrees. The modal degrees are ^2, ^3, ^6, and ^7. You can think of the modes as getting darker from Lydian to Locrian by the gradual lowering of scale degrees. I made a chart arranging modes by lowered degree for another post.
Major chords are often thought of as happy and so it is natural to think dark music = no major chords. But, the context in which major chords are used is important. Try taking major chords out of a diatonic context.
Major chords which are chromatic mediants have a different feel - dramatic, spooky, dark, etc.
Also, try exotic scales that contain major chords. One example is the double harmonic scale. It has a major ^3 degree, but also lowered scale degrees which create augmented seconds that are similar to the harmonic minor scale. In C double harmonic the tonic chord is C major and the lowered supertonic chord is Db major. Two major chords, but they are a half step apart. That never happens in diatonic scales. It creates a sort of uncanny use of major chords. Try exploiting those possibilities.
Comment: +1 for mentioning chromatic mediants and double harmonic scale. also look at phrygian dominant scale. @Limpuls, take a look at 'dark metal' music they use alot of chromatic mediants. I'm also looking for ideas how to create scary/dark sounds.
Comment: +1 for mentioning modes. It was my first thought after listening to the songs! Simplest solution is to write in slow tempo using darker modes, such as minor, frygian and locrian. It can't get cheesy this way!
Comment: So many great insights. You said that slow harmonic rhythm creates tension, I have never thought about it this way. That really explains why so many songs similar to the ones I posted, often has this pad/drone/ambient sound sitting back in the background playing 1-2 notes throughout 4 bars not changing much. I would also assume you would want to play darker notes for slow harmonic rhythm and some brighter notes for melody, arpeggios. As for not resolving, can you notice unresolved progressions in any of my examples? If so, can you explain it to us the progressions used, like degrees, notes.
Comment: @Limpuls the first one uses phrygian mode. The bass line plays the degrees 1(x6) 3 2 and repeats. Second one seems locrian, and the main synthesizer phrase is degrees 1 5.
Comment: @coconochao Thanks,  tried to reproduce bassline for first track in phrygian mode on my piano app on phone, but sounded almost nothing like it. Probably, timbre has a lot to do, too.
Answer: Context can create a lot of dark atmosphere look at type o negative

Alot of this album is in major scales but it's considered one of the greatest goth metal records of all time.
Similarly Mike Patton of faith no more Mr bungle etc would create incredibly dark songs when in "theory" they shouldn't be. Consider music theory as a basis but think outside the box and stand out.


Durable tennis players with strings of monofilament, why?

Question: The Babolat advertising here mentions the durability with players such as Rafael Nadal. According to google, durability means "the ability to withstand wear, pressure, or damage". I am not sure what it means here: it does not certainly mean too muscular because muscle is very expensive cell in terms of natural metabolism rate aka energy needed per time unit. Too muscular person cannot continue playing because of getting tired during a long run so not durable.
Now I am trying to make the connection to durability and monofilament strings. What do the two things have to do with each other? I cannot see the connection. Please, explain:

Why should durable tennis players use strings of monofilament?
Answer: Monofilament (mono = one) strings are one single strand of string, extruded during the string manufacturing process as a single solid strand.
Multifilament (multi = many) strings are several (usually hundreds) of tiny filaments inside a coating or sheath - no single solid strand.
What the guy in that video is saying is that monofilament string made from polyester material is made for a single purpose: to be as durable of a string as possible. Players such as (like the guy says) Rafael Nadal use monofilament polyester strings because they last the longest when you hit with an extreme amount of spin and strength, as Nadal does. So the monofilament poly string lasts him longer than any other string would.
You can read up more about different string types (materials and construction) here.
Comment: monofilament strings are BOTH - more durable, which makes them more suitable for powerful players because otherwise those players would break strings a lot more often.
Comment: I would assume that monofilament strings are even more suitable for powerful players, not durable. Can u clarify this?


Why is the Long Line not a covering space for the Circle

Question: I know of several reasons why the long line can't be a covering space for the circle, but I'm more curious in what exactly goes wrong with the following covering map.
Let $L$ be the long line and define $p: L \rightarrow \mathbb S^1$ by wrapping each segment of the line around the unit circle once, essentially in the same manner as with $\mathbb R$. Clearly we have that for $x \in \mathbb S^1$ the cardinality of the fiber $p^{-1}(0)$ is uncountable, which isn't possible since the fundamental group of the circle is countable. But it's not clear to me why $p$ is not a covering map. It's certainly surjective and seems to be continuous and a local homeomorphism. Yet one of these conditions must fail. Is the map not as well-defined as I originally thought.
I assume I'm misunderstanding the long line in a fundamental way.
Comment: The problem seems to lie in the phrase "essentially in the same manner as with $\mathbb{R}$." As kahen suggests you haven't proven that the naive interpretation of what this means is continuous. (The long line does not have the colimit topology with respect to the intervals that make it up; if it did, it would be disconnected.)
Comment: @Qiaochu, That would be a fundamental thing I'm missing.
Answer: The "obvious" function from the long line to the circle is not continuous at the limit ordinals.  As you approach a limit ordinal from the left, the map revolves around the circle infinitely many times, and therefore does not converge to a limiting value on the circle.
Comment: Okay, that makes sense.
Answer: Recall that every continuous map from the long line to the real numbers is eventually constant. This gives a contradiction with continuity of $p$ (as far as I can tell from a cursory glance at least).


Python: variable changes without being called

Question: I have a doubt about storing a class variables in a second variable, in order to be called later.
This is my code (simplified to be readable):
class Agent(object):
    def __init__(self):
        self.actual = []

class Play(object):

    def __init__(self):
        self.x = 0.45 * 400
        self.y = 0.5 * 400
        self.position = []
        self.position.append([self.x, self.y])
        self.x_change = 20
        self.y_change = 0

    def do_move(self, move, x, y):
        move_array = [self.x_change, self.y_change]

        if move == 1 and self.x_change == 0:  # right
            move_array = [20, 0]
        self.x_change, self.y_change = move_array
        self.x = x + self.x_change
        self.y = y + self.y_change

        self.update_position(self.x, self.y)

    def update_position(self, x, y):
        self.position[-1][0] = x
        self.position[-1][1] = y

def run():
    agent = Agent()
    player1 = Play()
    agent.actual = [player1.position]
    print(agent.actual[0])
    i = 1
    player1.do_move(i, player1.x, player1.y)
    print(agent.actual[0])

run()

>> [[180.0, 200.0]]
>> [[200.0, 200.0]]

This is what I cannot understand. Why, if agent.actual stores the player.position and it is not modified after agent.actual = [player1.position], its value actually changes between the two print()?
I modify player.position, but not agent.actual, which means it should stay the same. I cannot figure this out!
EDIT:
I tried, as suggested, with the following methods:
agent.actual = player1.position.copy()

agent.actual = player1.position[:]

agent.actual= list(player1.position)

import copy
agent.actual = copy.copy(player1.position)

agent.actual = copy.deepcopy(player1.position)

All these methods always return, as before, two different values:
>> [[180.0, 200.0]]
>> [[200.0, 200.0]]

Comment: `agent.actual` is a list containing *a reference to* another list, which is mutable. Changes to `player1.position` are therefore reflected in `agent.actual`. If you don't want that behaviour, see the dupe for how to create a copy instead.
Comment: Thank you, finally I understand it. Sorry for duplicating, I did it because of my ignorance about Python making a reference instead of a copy, not because I didn't looked it up accurately.
Comment: @jonrsharpe I tried as suggested in the question you mentioned, but as you can read in my EDIT the results are not different.
Answer: Player.position is list, which means it's mutable type. If you put this list in another list, Python makes a reference to it, not a copy.
When you add/delete/change items in the list, it will change everywhere the reference is hold.
You need to make a copy when you assign to agent.actual. Look at the copy module in Python or restructure your code (Hint: tuple is immutable type)
Comment: Thank you, As you can see in my EDIT, I tried what you suggested with the `copy` module, but it doesn't work.
Comment: @MauroComi don't hold positions in list, use tuples. Copying lists will cause troubles in corner cases. Tuples are immutable.


Why are the cello strings tied with a shoelace?

Question: At the beginning of this video you can see that the strings of the cello have been tied with a shoelace right below the bridge.

What is the purpose of this? Is it just decoration or is it something like a string eliminator:
Comment: My money's on a homebrew wolf eliminator, but I rather doubt it works very well.  Considering that it's on a $50 cello http://smile.amazon.com/Cecilio-CCO-Purple-Student-Bridge-Strings/dp/B004ZBAV3G/ref=sr_1_1?ie=UTF8&qid=1442064575&sr=8-1&keywords=purple+cello    It could be holding everything together.
Comment: Found on [this Talkbass thread](http://www.talkbass.com/threads/wolf-tone-tail-piece-adjustment.696809/), the following quote: "Well on my homemade wire tailpiece, when there was no damping, they would ring in sympathy the same note as the open G-string. I solved this by weaving a flat shoelace around the wires near the bottom." I really think this is Carl's answer because all I did was web search based on his comment.
Answer: As others have written and commented, this is almost surely meant to dampen sympathic vibrations between strings. Some players do not welcome this kind of resonance when they play. Others welcome them.
As someone noticed, this is looks like a cheap color cello and you can see its bridge. A skilled luthier can select and fit the bridge (and the length of strings between the bridge and the tailpiece) in ways that dampen, enhance or balance the resonance between strings.
The table is probably quite stiff with the colored finish, the bridge seems to me very done quickly, with thick feet, short head, small side peaks. I cannot judge the bass bar and sound post but the bridge and all the rest may contribute a lot to a significant part of the string vibrations not going through the table and back, therefore a tendency to put the other strings in vibration.
Answer: If it has any function at all, it would be to damp any vibration in that section of the string.  It doesn't look like it would be very effective.  It's also quite possible it used to hold a tag.


Complex analysis prerequisites for Apostol's Modular Functions

Question: I have studied Apostol's Introduction to Analytic Number Theory and it didn't much required Complex Analysis. However, I have studied Complex Analysis from the greatest book Churchill's Complex Variables but unfortunately it doesn't cover prerequisites for Apostol's second book Modular Functions and Dirichlet Series in Number Theory. It uses some knowledge of Complex Analysis which I couldn't find any good book for that.
Not a duplicate question, I am definitely looking for a very readable (esp for self-learning) advance book in Complex Analysis which covers ALL the prerequisites for Apostol's second book.
Any right suggestion would be much appreciated.
PS I tried Remmert's two volumes books on the subject as they looked nice at first (with good amazon reviews (why?!)) but instead of giving detailed rigorous proofs they were obsessed with history and the German quotes!
Added - the book gives an quick review or none at all to the following subjects as if the reader already knows about them:
doubly periodic functions in Complex Space
The Fourier expansions in Complex Space
Proof of "If is modular and not identically zero, then in the closure of the
fundamental region $R_{\Gamma}$, the number of zeros of f is equal to the number of poles" is not understandable.
Comment: What is something that appears in Apostol that you find doesn't appear in any introductory complex analysis text?
Comment: @davidlowryduda, I added the main problems of the book in my question.
Answer: I'm quite sure that with these introductory lecture notes (they starts from the the definition of holomorphic function and arrive to residue theorem/argument principle) and with these more advanced lecture notes, you will be able to tackle any topic involving complex analysis.


Migration Issue from Postfix

Question: I am currently using postfix, recently I have started to migrate the mailbox to Google App for Work.
While I changed the MX record from provider, it should be great that all outcoming emails to my domain, has been diverted to the google mail service, BUT the email send internally (Some users using another domain) is delivered to the mailbox in postfix.
May I know any solution to divert those email to Google App for Work?
Answer: For temporary solution, you can setup transport_maps and list all of migrated domains in there. For example
# main.cf
transport_maps = hash:/etc/postfix/gapps, other/transport/maps

# /etc/postfix/gapps
example.com smtp:example.com
example.org smtp:example.org

The point of using tranport_maps is you force the email next-hop. Instead delivering it to local mailbox, you force postfix to deliver it based on MX record of your GApps domain. So, it would override the routing rule in postfix.

For permanent solution you need remove the migrated domains from mydestination, virtual_alias_domains, virtual_mailbox_domains parameter and put it in relay_domains paremeter.
relay_domains = example.com example.org

References:

Postfix Address Classes Readme
This tutorial by  Scott Hebert
This postfix question of SF
Comment: Thanks. For temp. solution, Is that I should excluded the domain (hosted in Google) from the domain list in transport_maps, right?
Comment: See, the updated answer above. You need list all GApps domain to transport_maps
Comment: If this answer your problem, consider to [accept it](http://serverfault.com/help/someone-answers). It's the Server Fault way of saying thank you :)
Comment: Hi, I found that there is a statement for transport_map
"transport_maps = mysql:/etc/postfix/mysql-transport.cf"
May I have the advise? thanks
Comment: Put gapps before mysql. `transport_maps = hash:/etc/postfix/gapps, mysql:/etc/postfix/mysql-transport.cf`
Comment: Masegaloeh, Thank you for your help.
I have tried your solution provided but no luck, I cannot send the email from by IMAP client (Mail app in MAC), and outcoming email cannot be delivered to my server.
Comment: Actually my solution above was very general as I don't know your existing configuration. Maybe You  need to modify it so it won't break your system

Migration Issue from Postfix


What hook can I use for post-save actions?

Question: I've found several posts with this same question, but the answers always diverge to some other topic or they don't work for my needs.
After a node is saved, I want my custom module to insert a row into one of my custom tables which contains the nid of the new node. When hook_node_submit is called, the nid doesn't exist yet. hook_node_insert does have the nid, but only gets called the first time the node is saved, but I potentially need to take action each time the node is saved.
My plan is to separate my functionality into its own method and implement both hooks, but I'm trying to learn to do things properly with Drupal and so I'm wondering if there is a hook that actually fits with my requirements.
Is there a hook which is called every time a node is saved and called AFTER the save is complete?
Comment: you can use the combination of `hook_node_insert` and `hook_node_update` (acts every time a node is being saved).
Comment: `hook_node_insert()` and `hook_node_update()` are the correct hooks for this, `hook_node_submit()` is called during the form submission process before the node is saved. Don't forget you need to clear caches any time you add/remove a hook
Comment: Yes, I described that in the post above.
Comment: @KennyWyland you mentioned `hook_node_submit` but not `hook_node_update`?
Comment: You will get the node nid in `hook_node_insert` as well as in `hook_node_update`.
Comment: I've written a module that provides what yor're looking for. You can find the details here : https://stackoverflow.com/questions/5308465/post-save-callback
Answer: The correct way of doing this is to respect Drupal's db transaction.
If you want to do any database action in hook_node_insert you must consider that the node is not actually saved yet. which means if it fails and rolled back you end up with orphaned data in your database. So Drupal database API should be used and not direct query execution.
If you want to update the node itself in hook_node_insert, you can't because it's not possible to use node_save since the node is not yet saved in the database and calling it causes exception. One solution is to use register_shutdown_function function and passing nid, you can use node_load to make sure that it's really saved and then do any other action you may want on the new node.
function your_module_node_update($node){
  if($node->type == 'your_node'){
    register_shutdown_function('_your_module_post_insert',$node->nid);
  }
}

function _your_module_post_insert($nid) {
  $node = node_load($nid);
  if ($node) {
      node_save($node);
  }
}

UPDATE: You can also find an easier alternative here https://stackoverflow.com/a/24035797/1726778
Comment: Good idea. Though it's better to use `drupal_register_shutdown_function()`. But be aware that both of them seem to have problems in php-fpm environment, see http://drupal.stackexchange.com/q/102603/10667
Comment: I was having trouble checking if a node was acting as an Organic Group `og_is_group`. I think I was getting false negatives for this reason.
Answer: Looking at the comments on the question, it looks like the obvious solution is to use the hooks hook_node_insert() and hook_node_update().
I actually just did something very similar as described in the answer: I wrote a function that does what I want to be done every time a node is created/updated and then call it from each of those hooks. It works just fine.
Comment: @leymannx The downside in that method is it would only be called when a node is inserted/updated via a specific form. In my case, I needed to catch all additions and modifications of nodes.
Comment: That's cool! You could even do this from `hook_form_alter()`. https://drupal.stackexchange.com/a/221041/15055
Answer: There is a new drupal module, Hook Post Action, which adds the post save hooks. Let's try it to see if it works.
Comment: I tried using it in our project. Initially, it worked, but after some time we noticed some memory leaks being supposedly caused by it. Not sure if it's something in our D7 application though.


Get xml Values in Pig Latin

Question: I am using pig latin for a large XML dump. I am trying to get the value of the xml node in pig latin. The file is like
< username>Shujaat< /username>
I want to get the input Shujaat. I tried piggybank XMLLoader but it only separates the xml tags and its values also. The code is
register piggybank.jar;

A = load 'username.xml' using org.apache.pig.piggybank.storage.XMLLoader('username')
as (x: chararray);

B = foreach A generate x;

This code gives me the username tags also and values too. I only need values. Any idea how to do that? I found out regular expression but didnt know much?
Thanks
Answer: The example element you gave can be extracted with:
B = foreach A GENERATE REGEX_EXTRACT(x, '<username>(.*)</username>', 1)
      AS name:chararray;

A nested element like this:
  <user>
    <id>456</id>
    <username>Taylor</username>
  </user>

can be extracted by with something like this:
B = foreach A GENERATE FLATTEN(REGEX_EXTRACT_ALL(x,
     '<user>\\n\\s*<id>(.*)</id>\\n\\s*<username>(.*)</username>\\n\\s*</user>'))
     as (id: int, name:chararray);

 (456,Taylor)

You will definitely need to define a more sophisticated regex that suits all of your needs, i.e: handles empty elements, attributes...etc. Another option is to write a custom UDF that uses Java libraries to parse the content of the XML so that you can avoid writing (over)complicated, error-prone regular expressions.
Comment: Thanks man. It worked. You rock. Will get back to you when writing the UDF using java. :)
Comment: A warning for using regex to parse XML. See http://stackoverflow.com/questions/701166/can-you-provide-some-examples-of-why-it-is-hard-to-parse-xml-and-html-with-a-reg
Comment: @AlexanderTorstling You're right, using regex is not the preferred way to parse xmls. That's why I mentioned that writing a UDF would be more definitely a better approach

Get xml Values in Pig Latin


Google script batching rows over multiple sheets to speed up processing

Question: I've been trying to figure out some of the batching techniques in Google Sheets that came so easily in Excel.
On a workbook with 22 tabs, the first 12 tabs are named for the months of the year. I'm trying to auto-hide certain rows that start with a blank in column A only in these first 12 tabs.
My current script is very slow since it is calculating line by line.
I want to select the data on rows 5 through 94 on the first 12 tabs named JAN, FEB, etc., excluding the remaining tabs prior to the calculations being run. I can't seem to get a handle on the array function, if that's even the best technique.
Any help you could offer batching these rows would be greatly appreciated.
function hideRows() {
["JAN", "FEB", "MAR", "APR", "MAY", "JUN", "JUL", "AUG", "SEP", "OCT", "NOV", "DEC"].forEach(function (s) {
    var sheet = SpreadsheetApp.getActive()
        .getSheetByName(s)
    sheet.unhideRow(sheet.getRange('A1:X94'))
    sheet.getRange('A1:X94')
    sheet.getRange('A1:X94')
        .getValues()
        .forEach(function (r, i) {
            if (!r[0]) sheet.hideRows(i + 1)
        });
    });
}

Answer: Sort by column A, batch-hide the rows, then unsort.
function hideRows() {

  var sheet, range, values, row, col95, // col95 will be a helper column that will unsort back to the original order.
      arr95 = []; // this array will hold the values for col95

  for (i = 0; i < 95; i++) {
    arr95.push([i + 1]);
  }

  ["JAN", "FEB", "MAR", "APR", "MAY", "JUN", "JUL", "AUG", "SEP", "OCT", "NOV", "DEC"].forEach(function (s) {
    sheet =  SpreadsheetApp.getActive()
    .getSheetByName(s);

    sheet.insertColumnAfter(1); // this column, used for sorting, will be deleted at the end.
    col95 = sheet.getRange("B1:B95");
    col95.setValues(arr95);

    range = sheet.getRange('1:94'); // don't specify A1:X94 in case you have data in Column Y
    range.sort(1);
    range = sheet.getRange('1:94');
    sheet.unhideRow(range);
    values = range.getValues();

    row = 94; // resetting row to a neutral number
    for (r = 0, i = 0; r < values.length; r++) {
      if (!values[r][0]) {
        row = r + 1;
        break; // as soon as we get an empty A-cell, the row is set and the loop ends.
      }
    }

    if (row < 94) sheet.hideRow(sheet.getRange(row + ':94')); // batch-hiding all rows with an empty col A
    range = sheet.getRange('1:94');
    range.sort(2);
    sheet.deleteColumn(2); // the unsorting column is deleted.
  });
}

This is the batch-hiding line:
    if (row < 94) sheet.hideRow(sheet.getRange(row + ':94')); // batch-hiding all rows with an empty col A

Compared to the original code, this code reduces runtime by a factor of 10x in my tests. The trade-off is that it is not as concise.
Comment: That batch-hiding line may be exactly what I needed to make this work. Thank you so much
Comment: Hey your code ran for 135 seconds. I managed to execute the code in my answer in under 13 seconds. Let me know if it works for you!
Answer: The forEach seems overused here. Also, the script takes a very unusual structure.
Assuming it will always be the first 12 sheets and you either do not care the order the data appears in or have a column you can sort by
function hideRows() {
  var sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets()
  var count

  for (count = 0; count < 12; count++) {
    // sort the sheet by column A ascending to get all empty cells at the bottom
    sheets[count].sort(1,true)

    // find out the last row with data on column A
    var lastRow = sheets[count].getRange('A5:A94').getLastRow()

    // hide all rows bellow the last data row. You need to add 1 for the first empty row and 5 more because we skipped a few when getting the range
    sheets[count].hideRows(lastRow + 6, 94 - lastRow)

   // If you don't care how it's sorted leave it, or sort by the correct column here
  }
}

This should batch everything as much as possible calling SpreadsheetApp as little as possible. I did not test this code, just wrote it straight here, so you might want to check a few things (like wether ascending will put empty rows on bottom or top) and the calculation of hideRows indexes.
EDIT: just checked that getLastRow() will return the last index of the selected range, not the last row with actual data, so if your range is A5:A94 then it will return 94 so it works slightly differently then if it was for a sheet. But you can simply run a for to check for an empty string against the array of values, should still be faster as you only need to find when the values stop.
Comment: thanks so much. this would be perfect if it only affected rows 5 through 94
Comment: @Vytautas the `.getLastRow` method only returns the last row of the *sheet*, so it does not work to get the last row of a range as you intended.
Comment: @AntoineColson that is incorrect. There is a difference between `sheet.getLastRow()` and `range.getLastRow()`. The first one is as you said for the last row in a sheet. However, if you run `getLastRow()` against a `Range` class it will limit the last row to that specific range. You can see these 2: https://developers.google.com/apps-script/reference/spreadsheet/sheet#getLastRow() and https://developers.google.com/apps-script/reference/spreadsheet/range#getLastRow()
Comment: Thank you for pointing this out. Also I just saw your edit to your answer, thanks. I was testing it and obviously it didn't work. Now we know why!


Question regarding Big O:

Question: Let $f$ be a positive function such that $$f(n) = a f\left( \left\lfloor \frac nb\right\rfloor\right) + c$$ holds for every integer n ≥ 1, where a ≥ 1, b is an integer larger than one, and c ∈ R+ .
Prove that $f(n) = O(n^{\log_b a})$ if a > 1, and $f(n) = O(\ln n)$ if
a = 1
I don't understand where to begin, I tried using the formal definition of big O notation but still couldn't progress any further
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Answer: You may want to have a look at some proofs of the Master Theorem as your question seems extremely similar.
Comment: got it, thanks!

Question regarding Big O:


On Apache2/mod_perl2, stderr output doesn't go to logfile nominated by ErrorLog

Question: I like to maintain separate Apache2 logfiles for each of my named virtual hosts, like this:
<VirtualHost *:80>
  ServerName myhost.example.com
  ErrorLog /var/log/apache2/myhost-error.log
</VirtualHost *:80>

The nominated file correctly gathers error-messages generated by Apache2 itself, such as 404s:
[Thu Jul 18 12:07:45 2013] [error] [client 10.0.0.141] File does not exist: /home/mike/myhost/htdocs/favicon.ico

But when myhost runs mod_perl scripts which emit their own logging messages (using print to STDERR, for example), those messages still appear in the global Apache2 error-log, /var/log/apache2/error.log.
This is using Apache 2.2.16 on Debian GNU/Linux 6.0, but I also see the same thing on Ubuntu 12.04.2 LTS and on MacOS 10.5 -- I guess it's standard Apache2 behaviour.
I want my mod_perl scripts' logging to appear in the site-specific error-log just like the 404s and other Apache-level errors. How can I make this happen?
Comment: I get the same issue on Ubuntu 16.04 with Apache 2.4.18
Answer: This is partially solved by overriding warn for your scripts. STDERR still goes to the main Apache error log though.
From Modperl docs:

First, we need to use mod_perl's logging function, instead of CORE::warn

Either replace warn with Apache2::ServerRec::warn:
use Apache2::Log ();
Apache2::ServerRec::warn("the code is smoking");

or import it into your code:
use Apache2::ServerRec qw(warn); # override warn locally
warn "the code is smoking";

Comment: Thanks, this is helpful.

But it seems like the wrong solution to the problem. Surely _all_ stderr output should go to the nominated error-file? Or, if that is not the default (and it's not) there ought to be a simple option to make it so?


Certain GPO settings are not taking effect on Clients

Question: I've been trying to deploy a printer with Print Management on Server 2012 however it never appears on any of the clients after deployment.
And this is not only the case with printers, the same thing happens when I deploy Desktop Shortcuts to the clients.
Basically anything that changes the desktop, adds software, edits components etc does not work but things like disabling the Task Manager seem to work fine.
All clients login with their own DOMAIN\X accounts and GPRESULT /R shows me that the policy has been applied.
GPUpdate [/force] [/boot] runs succesfully.
-
Where can I start debugging this?
I've only been working with AD & DCs since a week ago and it's very unclear to me why this is not working properly.
Comment: Use gpresult /h to generate an HTML report of group policy application.  It will show any errors that occurred while processing the group policy preferences.
Answer: Printers and desktop items are set by the USER policy, not by the MACHINE policy. So make sure that your policies are set to apply to the appropriate scope and have the appropriate permissions.
If you have a policy that is set to be processed by machines, but it only change user-specific items, it the policy will apply, but no changes will result.


Expectation of integral of a stochastic process

Question: Let $r_t$ be governed by a mean reverting process:
$dr_t = K(\theta - r_t)dt + \sigma dW_t  $
where $W_t $ is standard weiner process, and $\sigma , K, \theta$ are constants
Can it be shown that $-log (E[e^{-\int r_sds}]) = \theta (T- \frac{(1- e^{-KT})}{K}) + r_o \frac{(1- e^{-KT})}{K}  $
Comment: A way to do that is first to show the solution for r is gaussian (check the web for Vasicek SDE). Once you have calculated the first 2 moments of this gaussian you are done as your expression is then the mean of a lognormal for random variable.
Comment: That's a good intuition! However, is there also an intuitive way tro suggest that r is in fact gaussian? Since it is a sum of non-iid normals, it need not be normal, right?
Answer: This is the Ornstein-Uhlenbeck equation with general solution
$$x_t = x_0 e^{-K t}+\theta(1-e^{-K t}) + \tfrac{\sigma}{\sqrt{2K}} W_{1-e^{-2 K t}},$$
where $W_t$ is a Brownian motion. So then you are mainly asking to compute
$$E[exp(-\int^{t}  \tfrac{\sigma}{\sqrt{2K}} W_{1-e^{-2 K s}}ds)].$$
The integral is a random variable so we just have to compute its variance. By the "Theorem 8.5.7 (Time change formula for Itˆo integrals)" in Oksendal
$$\int^{a_{t}}v(s)dB_{s}=\int^{t}v(a_{r})\sqrt{a'_{r}}d\tilde{B}_{r}$$
we have
$$\int^{T}  W_{1-e^{-2 K t}}dt=\int^{T}\int^{1-e^{-2 K t}}\sqrt{2K}e^{-Kr}d\tilde{B}_{r}dt$$
and so by the Ito isometry
$$M_{T}:=E\left(\int^{T}  W_{1-e^{-2 K t}}dt\right)^{2}=E\left[\int^{T}\int^{T}\int^{1-e^{-2 K t_{1}}}\int^{1-e^{-2 K t_{2}}}2Ke^{-K(r_{1}+r_{2})}d\tilde{B}_{r_{1}}d\tilde{B}_{r_{2}}dt_{1}dt_{2}\right]$$
$$=\int^{T}\int^{T}\int^{\infty}\int^{\infty}\min(1-e^{-2 K t_{1}},1-e^{-2 K t_{2}})2Ke^{-K(r_{1}+r_{2})}dr_{1}dr_{2}dt_{1}dt_{2}.$$
Therefore, we get
$$E[exp(-\int^{t}  \tfrac{\sigma}{\sqrt{2K}} W_{1-e^{-2 K s}}ds)]=exp\left(\frac{M_{T}}{2}\tfrac{\sigma^{2}}{2K} \right).$$


Help me find an eigenvector

Question: Good evening all! I'm trying to find the eigenvalues and eigenvectors of the following problem
$$
       \begin{bmatrix}
        -10 & 8\\
        -18 & 14\\
         \end{bmatrix}*\begin{bmatrix}
        x_{1}\\
        x_{2}\\
         \end{bmatrix}
$$
I've found that $λ_{1},_{2}=2$ where $λ$ is double eigenvalue of our matrix and now I'm trying to find its eigenvectors. So we have to solve the above system
$$(-10-λ)*x_{1}+8*x_{2}=0    ,    -18*x_{1}+(14-λ)*x_{2}=0$$
So what I do here is to replace $λ=2$ and we have
$$-12*x_{1}+8*x_{2}=0    ,    -18*x_{1}+12*x_{2}=0$$
From here I get that $3*x_{1}=2*x_{2}$ and I think that gives us the eigenvector $\begin{bmatrix}
        2/3\\
        1\\
         \end{bmatrix}$
My book is telling me that the eigenvector for $λ=2$ is $\begin{bmatrix}
        2\\
        3\\
         \end{bmatrix}$ but I can't find the same solution. Can someone help me?
EDIT : I added my own try at solving it.
Answer: You and your book are both correct. If you take an eigenvector $v$ and multiply it by a scalar, you get another eigenvector. Multiply your result by $3$ and you have the book's result.
If you are still not sure, you can try multiplying your vector and the book's vector by the matrix. You will find that both are eigenvectors. The book just adjusted the values so that all the entries are integers.
Comment: I see. Thank you very much
Answer: Both $\begin{bmatrix}
        2\\
        3\\
         \end{bmatrix}$ and $\begin{bmatrix}
        2/3\\
        1\\
         \end{bmatrix}$ are valid eigenvectors for the eigenvalue $\lambda$. The reason is as follows.
For any eigenvalue $\lambda$ of a square matrix $A$, there exist an infinite number of eigenvectors. The eigenspace of $A$ for $\lambda$ is the set of all eigenvectors of $\lambda$ for $A$ with the addition of the zero vector. It is key to note that the eigenspace is a vector space.
Since vector spaces are closed under scalar multiplication, then the book's solution of $\begin{bmatrix}
        2\\
        3\\
         \end{bmatrix}$ is a valid solution since $\begin{bmatrix}
        2\\
        3\\
         \end{bmatrix} = 3\begin{bmatrix}
        2/3\\
        1\\
         \end{bmatrix}$, hence $\begin{bmatrix}
        2\\
        3\\
         \end{bmatrix}$ is an element of the eigenspace since you have found that $\begin{bmatrix}
        2/3\\
        1\\
         \end{bmatrix}$ is an eigenvector.
Another way to see this is by noting that the system you posted has infinite solutions. After finding that $3x_1 = 2x_2$, the substitution of $x_1$ for $2/3x_2$ into the second equation of the system results in $0=0$, implying infinite solutions for the system.
Comment: Yeah you are right. I guess the book used the other eigenvector because it's better for the purpose it is needed. I'm currently studying dynamical systems!

Help me find an eigenvector