Extract the different components of variance in a linear mixed model in R

Question: Consider a mixed model as follows.
library(lme4)
# Load data
data <- structure(list(blk = c(1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3L),
                       gent = c(1, 2, 3, 4, 7, 11, 12, 1, 2, 3, 4, 5, 9, 1, 2, 3, 4, 8, 6, 10L),
                       yld = c(83, 77, 78, 78, 70, 75, 74, 79, 81, 81, 91, 79, 78, 92, 79, 87, 81, 96, 89, 82L),
                       syld = c(250, 240, 268, 287, 226, 395, 450, 260, 220, 237, 227, 281, 311, 258, 224, 238, 278, 347, 300, 289L)),
                  .Names = c("blk", "gent", "yld", "syld"), class = "data.frame", row.names = c(NA, -20L))
data$blk <- as.factor(data$blk)
data$gent <- as.factor(data$gent)

The data is unbalanced.
# Mixed effect model
frmla <- "syld ~ 1 + gent + (1|blk)"
library(lme4)
model <- lmer(formula(frmla), data = data)

model
Linear mixed model fit by REML ['merModLmerTest']
Formula: syld ~ 1 + gent + (1 | blk)
   Data: data
REML criterion at convergence: 73.9572
Random effects:
 Groups   Name        Std.Dev.
 blk      (Intercept)  9.385
 Residual             16.919
Number of obs: 20, groups:  blk, 3
Fixed Effects:
(Intercept)        gent2        gent3        gent4        gent5        gent6        gent7        gent8        gent9
    256.000      -28.000       -8.333        8.000       32.127       43.678      -36.805       90.678       62.127
     gent10       gent11       gent12
     32.678      132.195      187.195

Primarily I want to compare the gent levels by LS means.
library("lmerTest")
lsmeans(model)
Least Squares Means table:
         gent Estimate Standard Error   DF t-value Lower CI Upper CI p-value
gent  1   1.0    256.0           11.2  6.9    22.9      229      283  <2e-16 ***
gent  2   5.0    228.0           11.2  6.9    20.4      201      255  <2e-16 ***
gent  3   6.0    247.7           11.2  6.9    22.2      221      274  <2e-16 ***
gent  4   7.0    264.0           11.2  6.9    23.6      237      291  <2e-16 ***
gent  5   8.0    288.1           18.5  8.0    15.6      245      331  <2e-16 ***
gent  6   9.0    299.7           18.5  8.0    16.2      257      342  <2e-16 ***
gent  7  10.0    219.2           18.5  8.0    11.8      177      262  <2e-16 ***
gent  8  11.0    346.7           18.5  8.0    18.8      304      389  <2e-16 ***
gent  9  12.0    318.1           18.5  8.0    17.2      275      361  <2e-16 ***
gent  10  2.0    288.7           18.5  8.0    15.6      246      331  <2e-16 ***
gent  11  3.0    388.2           18.5  8.0    21.0      346      431  <2e-16 ***
gent  12  4.0    443.2           18.5  8.0    24.0      401      486  <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

In addition I am interested in variance partitioning.
The variance component due to random effect and residual can be estimated as follows.
VCrandom <- VarCorr(model)
print(VCrandom, comp = "Variance")
 Groups   Name        Variance
 blk      (Intercept)  88.083
 Residual             286.250

How to partition the total variance into components due to each of the factors gent and blk along with the residual ? Something similar to the output given by PROC MIXED of SAS, where MSE is computed even when estimation is by ML or REML instead of least squares.
Should I treat the fixed effect as random just for the purpouse of getting variance component ?
frmla2 <- "syld ~ 1 + (1|gent) + (1|blk)"
model2 <- lmer(formula(frmla2), data = data)
model2

VCrandom2 <- VarCorr(model2)
print(VCrandom2, comp = "Variance")
 Groups   Name        Variance
 gent     (Intercept) 4152.08
 blk      (Intercept)  116.11
 Residual              274.92

If there is no random effect, variance components can be estimated using the least squares approach (ANOVA, Sum of squares, MSE).
The package mixlm has provision for variance partitioning using SS in case of mixed models.
library(mixlm)

mixlm <- lm(syld ~ 1 + r(gent) + r(blk), data)

Anova(mixlm, type="III")

Analysis of variance (unrestricted model)
Response: syld
          Mean Sq   Sum Sq Df F value Pr(>F)
gent      5360.49 58965.36 11   18.73 0.0009
blk        638.58  1277.17  2    2.23 0.1886
Residuals  286.25  1717.50  6       -      -

            Err.term(s) Err.df VC(SS)
1 gent              (3)      6 3044.5
2 blk               (3)      6   52.8
3 Residuals           -      -  286.3
(VC = variance component)

               Expected mean squares
gent      (3) + 1.66666666666667 (1)
blk       (3) + 6.66666666666667 (2)
Residuals (3)

WARNING: Unbalanced data may lead to poor estimates

The estimates are different
# Total variance
var(data$syld)

|source   |  model1|  model2|  mixlm|
|:--------|-------:|-------:|------:|
|gent     |      NA| 4152.08| 3044.5|
|blk      |  88.083|  116.11|   52.8|
|Residual | 286.250|  274.92|  286.3|

Can fixed effect variance be extracted using predict function as suggested here In R: How to extract the different components of variance in a linear mixed model! ?
var(predict(model))

Which is the most appropriate method compatible with (RE)ML estimates in lme4 ?
Comment: The [performance](https://cran.r-project.org/web/packages/performance/performance.pdf) package in R  can measure "conditional R2" (random & fixed effects together), "marginal R2" (random effect alone), adjusted ICC (fixed effect alone) & "conditional ICC" (fixed effect controlling by random effect).
The theory behind it is in this paper:
Nakagawa, S., Johnson, P. C., & Schielzeth, H. (2017). The coefficient of determination R 2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. Journal of the Royal Society Interface, 14(134), 2017021
Answer: This isn't exactly an answer, but whenever I see a question about "explained variance" in mixed models, I always think of this email from Douglas Bates, the original author of lme4 and co-author of nlme, on the R-Sig-ME mailing list on Feb 26, 2010, in response to:

I have written some code to implement Gelman & Pardoe's Rsq for an lmer object. It gives some believable results, but it's difficult to be confident because of the translation from Bayesian into frequentist paradigms.
If anyone is interested then I'd be really happy to discuss this off-list and share/develop the code.

I think it is useful to have this email here on CV. This is Bates' reply:

Assuming that one wants to define an R^2 measure, I think an argument
  could be made for treating the penalized residual sum of squares from
  a linear mixed model in the same way that we consider the residual sum
  of squares from a linear model.  Or one could use just the residual
  sum of squares without the penalty or the minimum residual sum of
  squares obtainable from a given set of terms, which corresponds to an
  infinite precision matrix.  I don't know, really.  It depends on what
  you are trying to characterize.
In other words, what's the purpose?  What aspect of the R^2 for a
  linear model are you trying to generalize?
I'm sorry if I sound argumentative but discussions like this sometimes
  frustrate me.  A linear mixed model does not behave exactly like a
  linear model without random effects so a measure that may be
  appropriate for the linear model does not necessarily generalize.  I'm
  not saying that this is the case but if the request is "I don't care
  what the number means or if indeed it means anything at all, just give
  me a number I can report", that's not the style of statistics I
  practice.
I regard Bill Venables' wonderful unpublished paper "Exegeses on
  Linear Models" (just put the name in a search engine to find a copy -
  there is only one paper with "Exegeses" and "Linear Models" in the
  title) as required reading for statisticians.  As Bill emphasizes in
  that paper, statistics is not just a collection of formulas (many of
  which are based on approximations).  It's about models and comparing
  how well different models fit the observed data.  If we start with a
  formula and only ask ourselves "How do we generalize this formula?"
  we're missing the point.  We should start at the model.
In a linear model the R^2 statistic is a dimensionless comparison of
  the quality of the current model fit, as measured by the residual sum
  of squares, to the fit one would obtain from a trivial model.  When
  the current model can be shown to contain a model with an intercept
  term only (and whose coefficient will be estimated by the mean
  response) then that model fit is the trivial model.  Otherwise the
  trivial model is a prediction of zero for each response.  We know that
  the trivial model will produce a greater residual sum of squares than
  the current model fit because the models are nested.  The R^2 is the
  proportion of variability not accounted for by the trivial model but
  accounted for by the current model (my apologies to my grammar
  teachers for having juxtaposed prepositions).
The interesting point there is that when you think of the
  relationships between models you can determine how you handle the case
  of a model that does not have an intercept term.  If you start from
  the formula instead you can end up calculating a negative R^2 because
  you compare models that are not nested.  Such nonsensical results are
  often reported.  (I think it was the Mathematica documentation that
  gave a careful explanation of why you get a negative R^2 instead of
  recognizing that the formula they were using did not apply in certain
  cases.)
It may be that there is a sensible measure of the quality of fit from
  a linear mixed model that generalizes the R^2 from a linear model.  I
  don't see an obvious candidate but I will freely admit that I haven't
  thought much about the problem.  I would ask others who are thinking
  about this to consider both the "what" and the "why".  George
  Mallory's justification of "because it's there" for attempting to
  climb Everest is perhaps a good justification for such endeavors
  (Mallory may have questioned his rationale as he lay freezing to death
  on the mountain).  I don't think it is a good justification for
  manipulating formulas.


Add authentication to wsdl service in Genexus

Question: I'm studying how to create a web service with Genexus 17U10. The first part looks very simple, following this https://wiki.genexus.com/commwiki/servlet/wiki?4210,Web+Services+with+GeneXus I can expose the service and consume it in another project. Now I want to add some kind of authorization but I can't find how to set it. Does genexus manage it somehow? I can't find informations about this
Answer: It is easier and recommended to use REST services and authenticate using GAM.
For SOAP services you have to add extra programming.


I keep getting newRow() (the name of my function) not defined

Question: http://codepen.io/j4ysen/pen/EiDyj
Entering the ingredient fields correctly is supposed to create a new row, however I keep getting an uncaught error saying that the function itself is not defined when the red + button is pressed.
Anyone have any idea what I might be doing wrong?
Answer: On line 67 of your JS, remove the semicolon. Or more appropriately, move it to the end of line 66.
Change this
if(ingredient_name === "" || ingredient_amount === "" || ingredient_unit === "Select" ){
    console.log("One or more fields weren't entered")
};

To this
if(ingredient_name === "" || ingredient_amount === "" || ingredient_unit === "Select" ){
    console.log("One or more fields weren't entered");
}


How to theme a link in a D7 render array?

Question: I'm trying this:

        $content[] = array(
          '#theme' => 'link',
          '#text' => $row->title,
          '#path' => $base_path . 'node/' . $row->nid,
          '#attributes' => array('attributes' => array('title' => $row->title)),
          '#prefix' => '
',
          '#suffix' => '
',
        );

but Drupal is complaining that "Arg 1 passed to drupal_attributes() must be an array, null given" My array looks fine? ...or does it xD
Here is the API doc I'm looking at
Answer: The theme_link() function behaves a bit like the l() function; the attributes array needs to be inside an #options property:
$link = array(
  '#theme' => 'link',
  '#text' => $row->title,
  '#path' => $base_path . 'node/' . $row->nid,
  '#options' => array('attributes' => array('title' => $row->title)),
  '#prefix' => '
',
  '#suffix' => '
'
);

echo render($link);

Comment: [Apparently](http://drupal.org/node/1187032#comment-4711898) it's better to use `'#type' => 'link'` as otherwise some `l()` preprocessing gets missed. Note [the keys differ](http://drupal.org/node/1187032#comment-5534700) depending on the method.
Comment: Who's awesome? Clive is awesome!
Comment: Crap, I was even on that page & didn't see that [comment](http://api.drupal.org/api/drupal/includes%21theme.inc/function/theme_link/7#comment-12939), thanks for your attention to detail!
Comment: @Merrick No probs, I've been stung by this one before myself :)
Answer: Wanted to edit Clive post, but got rejected :(

As Andy pointed out, it's not recommended to use theme_link instead of l().
Using drupal_pre_render_link() is cleaner for renderable array.
 $link = array(
  '#type'  => 'link',
  '#title' => $row->title,
  '#href'  => 'node/' . $row->nid,
);

You can also go for the "easy-quick-fix" solution with #markup (But don't)
$link = array(
  '#markup' => l($row->title, 'node/' . $row->nid),
);

Answer: I think your code should read
'#attributes' => array('title' => $row->title),

The wording of the doc in common.inc seems a little unclear, but see the doc for drupal_attributes() for an example.
Comment: Yeah, I tried that first and got the same error.


Trying to store an equation into a variable

Question: I am trying to figure out how to store the calculated gross pay into the grossPay variable to use it for calculating net pay. Throws me off when using a struct.
#include<string>
#include <iostream>
using namespace std;

struct Workers {
    string name;
    double hourlyWage;
    int hoursWorkedPerDay;
    int hoursWorkedPerWeek;

};
int healthTax = 130;
double otherTaxes = 0.17;
double grossPay;

int main() {
    Workers JohnDoe;
    JohnDoe.name = "John Doe";
    JohnDoe.hourlyWage = 8.50;
    JohnDoe.hoursWorkedPerDay = 6;
    cout<<"- John Doe's weekly earnings - "<< endl;
    cout<<"Hourly rate: $8.50" << endl;
    cout<<"Hours worked per day: 6" << endl;
    cout<<"Gross pay: $" << 7 * (JohnDoe.hourlyWage * JohnDoe.hoursWorkedPerDay)<<endl; // gross pay
    grossPay; // how to store gross pay into this variable?
    cout<<"Net pay: $" << otherTaxes * (grossPay - healthTax) / 100; // net pay
}

Comment: Why you can't just `grossPay= 7 * (JohnDoe.hourlyWage * JohnDoe.hoursWorkedPerDay)`
Comment: `grossPay = 7 * JohnDoe.hourlyWage * JohnDoe.hoursWorkedPerDay`
Answer: I think you may not be understanding how variables work. You don't store the formula, you store the result of calculating the formula:
     double grossPay =  7 * (JohnDoe.hourlyWage * JohnDoe.hoursWorkedPerDay);

Comment: I think he rather means something like a function
Comment: @Saeid I really doubt he is asking about lambdas or function pointers.
Answer: Other answers are right, but a slightly better solution would be a function wich takes a Worker reference as parameter and returns the result:
double getGrossPay(const Workers& worker) {
        return 7 * (worker.hourlyWage * worker.hoursWorkedPerDay);
}

Then simply:
cout<<"Gross pay: $" << getGrossPay(JohnDoe) <<endl; // gross pay

Answer: This is what you hae to do :
double grossPay =  7 * ( JohnDoe.hourlyWage * JohnDoe.hoursWorkedPerDay );

Then you can use it, in this case, print it :
cout<<"Gross pay: $" << grossPay <<endl;

Comment: And what is the difference of your answer with the Neil's answer?
Comment: I'm just saying that after you created the variable, you can use it.


Issue with Raspberry pi 3.0 i2c SMBus

Question: I am new to the raspberry pi and I have been working through some i2c examples. I am receiving an IO Error: [Errno 22]Invalid argument when I run the code below:
<code>import smbus

bus = smbus.SMBus(1)
addr = 0xa0
data = 0x20
bus.write_byte(addr, data)
</code>
Would someone please explain why this might be happening? I can see the device using the i2cdetect in the terminal but my code is failing. Any help would be greatly appreciated.
Answer: This might happen when you use an invalid address in the function.
You can check addr is correct or not. Please have a look on slave address.

Issue with Raspberry pi 3.0 i2c SMBus


Laravel export downloadable CSV file

Question: I am trying to write an endpoint which exports the data from the users table i currently have into a CSV file which is downloaded upon clicking of a button. I have setup the controller, routes, export file, and added in the button within the view but everytime i go to click the button it just directs me to the admin/user/export then gives me a 404 but shouldnt it just stay on the same URL then just add the file to my downloads file?
This is the package i am using for Laravel
https://github.com/maatwebsite/Laravel-Excel
web route
  Route::get('users/export', 'Admin\UserController@export')->name('users.export');

UserExport
class UsersExport implements FromCollection
{
    public function collection()
    {
        return User::all();
    }
}

Controller Function
public function export()
{
    return Excel::download(new UsersExport, 'users.csv');
}

Summarise problem:
Everytime i click the button on the view page it directs to admin/users/export then gives me a 404 when i want it to just download the CSV file for the users.
Some help to see where i am wrong would be helpful!
Thanks.
Comment: Yes it should download the file without redirecting, make sure you are actually accessing the correct route (maybe try a `dd(123);` in your `export()` action). Currently it looks like a route conflict to me, do you have a route similar to this: `Route::get('users/{user}');`?
Comment: Your a star! yeah it was a route problem i had it set to `users` instead of `user` is there a way you can modify the columns
Comment: You can keep it as `users`, I will post an answer.
Answer: The problem is that two of your routes conflict with each other.
Given the following two routes:
Route::get('users/{user}', 'Admin\UserController@show')->name('users.show');

Route::get('users/export', 'Admin\UserController@export')->name('users.export');

Currently Laravel assumes that when you are trying to access users/export you actually want to access users/{user} with export as the route parameter {user}.
Making sure that users/export is registered before users/{user} should solve your issue:
Route::get('users/export', 'Admin\UserController@export')->name('users.export');

Route::get('users/{user}', 'Admin\UserController@show')->name('users.show');

Comment: Thanks for that Remul! have a great day/evening!
Comment: That makes sense thanks for the teaching! do you have much experience with this package can you easily enough add columm headings etc? ill have a look through the documentation but i wasnt sure if it was best to custom write this exporting or use this package to make life easier(was suggested in alot of threads) and the controller less cluttered?
Comment: Yes you can use the `WithHeadings` concern and add your headings, you can read more about it [here](https://docs.laravel-excel.com/3.1/exports/mapping.html#adding-a-heading-row).

Laravel export downloadable CSV file


How to get URL value in drupal?

Question: My URL is something like this http://xyz.com/content/show-cat?catid=2  . Now i want the value of catid (i.e 2) and save it in a variable. I wrote php code like $store=$_Get['catid'];
But this format is not accepted in drupal. How can i implement this? Is there any alternative?
Answer: Try <code>$store=$_GET['catid'];</code> instead as PHP is case-sensitive.
Comment: also, for security when pulling things straight out of $_GET please make sure to stick it in check_plain() or similar before using the value.

How to get URL value in drupal?


Should a goal be disallowed if a player in an offside position blocks the view of the goalkeeper?

Question: Should Germany's first goal by Özil against Norway on the 4th September 2017 have been disallowed if Timo Werner were in an offside position?
In reality, a defender on the far side of the pitch was playing Werner onside, but I'm interested in the hypothetical case where a player in an offside position is blocking the view of the goalkeeper.
Answer: IFAB Laws of the Game - Law 11: Offside - 2. Offside offence states that:

A player in an offside position at the moment the ball is played or
  touched by a team-mate is only penalised on becoming involved in
  active play by:

[...]
interfering with an opponent by:

preventing an opponent from playing or being able to play the ball by clearly obstructing the opponent's line of vision or
[...]

If we assume that German #11 Timo Werner is in clear offside position (which is not the case) and given that he is positioned right in front of the Norwegian goalkeeper obstructing his line of vision, then it means that he causes an offside offence, hence the goal should have been disallowed.
This situation is also mentioned in IFAB Practical Guidelines for Match Officials:

An attacker in an offside position (A) is clearly obstructing the
  goalkeeper's line of vision. The player must be penalised for
  preventing an opponent from playing or being able to play the ball.
Decision: Offside offence

Here is the freeze frame from another point of view, showing how Timo Werner obstructs Jarstein's (Norwegian goalkeeper) line of vision by positioning himself on the same line with Özil and Jarstein:
Answer: As gdrt answer clearly answer hypothetical case of attacker in an offside position and obstructing the goalkeeper's line of vision.
But the referee interpretation play a big role in determining whether that's the case of offside or not. And it is not sure that if a defender on the far side of the pitch was absent it would have been offside.
From this article:

An attacker, who is initially in an offside position, must clearly obstruct an opponent's line of vision to be penalized. If this possible obstruction is unclear, the attacker should not be penalized.

This article explains how viewing angle of referee affects a decision process.

Source: law-11.com
For different examples of this case see this.


How to improve Gaussian Process Regression fit: reducing oscillations and narrowing confidence interval

Question: I'm trying to fit a model y=f(x1, x2) to some measurements using GPR. For each instance (x1i, x2i) there are two measurements of y. I'm using the following kernel:
WhiteKernel + ConstantKernel * RBF
The result is adequate but not quite right yet, there are mainly two problems:

The average predicted value does pass through the measured values but there is some "oscillations" overshoot/undershoot between some points. This is shown with the red arrow in the images below.

The 90% confidence intervals increase too much between some  measurement points. This is shown with the black arrows in the images below.

How should I specify the kernel to get better results? Do you have any tips for me?
Answer: I'm not sure what is problem you're trying to solve here. The prediction intervals are a feature, not a bug. The Gaussian process is a nonparametric, data-based model. The intervals are thin for regions where it saw enough of the data to be certain and broad for low-data regions, where it's uncertain. The width of the interval tells you when the predictions are more precise. To make them thinner, gather more data for the regions.
Alternatively, use a parametric model that makes assumptions about the shape of the curve, hence produces intervals of more uniform width.
Comment: Thanks for your comment. OK, I can understand that I might need more data to make the CI thinner. But what about the over-/undershoot? Can I make the predictions smoother?
Answer: For the over/undershoot issue I recommend trying the Matern kernel, it can be seen as a generalization of a RBF with an extra parameter that controls smoothness. This extra parameter very often helps in such issues.
Comment: Thanks a lot! With the Marten kernel the over/-undershoot issue disappeared and the confidence intervals reduced as well :)!


Finding relative keys of modes

Question: How does one go about finding the relative keys of modes?
For example, determining the relative major of  F# Phrygian or determining the relative locrian of G Mixolydian?
Comment: I would probably first find the tonic in all cases.Seems convenient to me.
Answer: There are a couple of strategies you could use.

Work out the key signature of the starting mode in question & then ascertain the required mode. In your 2 examples, F♯ Phrygian has a key signature of F♯ & C♯, so its relative major is D Major; G Mixolydian has an empty key signature, so the Locrian mode with the same key signature is B Locrian.
Alternatively, use the your knowledge of which scale degrees in the Major scale for the tonics of each mode & work it out from there. You know that the mode mappings on major scale degrees are:
I   (tonic)        = Major/Ionian
II  (super-tonic)  = Dorian
III (mediant)      = Phrygian
IV  (sub-dominant) = Lydian
V   (dominant)     = Mixolydian
VI  (sub-mediant)  = Minor/Aeolian
VII (leading-note) = Locrian

So in your examples, the Phrygian mode is formed on the mediant (3rd degree) of the major scale, so the "relative major" of F♯ Phrygian is 2 scale steps back in the Phrygian mode (F♯→E→D); the Mixolydian mode is formed on the dominant (5th degree) & the Locrian on the leading-note (7th degree), so the "relative Locrian" of G Mixolydian is 2 scale steps up in the Mixolydian mode (G→A→B).
EDIT: additional answer
A third alternative is to learn the relations between the modes. Just like the relation between the major & relative minor is a 6th, all of the other modes can be related in a similar way. Here's how it would look:
Table interval relations between modes & their relatives
  Read across to the starting mode & down to the relative mode.
  The resulting interval is the distance between the tonics of the relative modes.

> Starting Mode:     Major | Dorian | Phrygian | Lydian | Mixolyd | Aeolian | Locrian
Relative Major       same  |↓Maj 2nd| ↓Maj 3rd |↓Prf 4th|↓Prf 5th |↓Maj 6th |↓Maj 7th
Relative Dorian    ↑Maj 2nd|  same  | ↓Maj 2nd |↓min 3rd|↓Prf 4th |↓Prf 5th |↓Maj 6th
Relative Phrygian  ↑Maj 3rd|↑Maj 2nd|   same   |↓min 2nd|↓min 3rd |↓Prf 4th |↓Prf 5th
Relative Lydian    ↑Prf 4th|↑min 3rd| ↑min 2nd |  same  |↓Maj 2nd |↓Maj 3rd |↓Aug 4th
Relative Mixolyd   ↑Prf 5th|↑Prf 4th| ↑min 3rd |↑Maj 2nd|  same   |↓Maj 2nd |↓Maj 3rd
Relative Aeolian   ↑Maj 6th|↑Prf 5th| ↑Prf 4th |↑Maj 3rd|↑Maj 2nd |  same   |↓Maj 2nd
Relative Locrian   ↑Maj 7th|↑Maj 6th| ↑Prf 5th |↑Aug 4th|↑Maj 3rd |↑Maj 2nd |  same

So, in your examples we read across to Phrygian in the top line & then down to the Relative Major row & find the interval between the tonics (key-notes) is a Major 3rd, going down for Phrygian to Major. Similarly, Mixolydian to Relative Locrian is up a Major 3rd.
Comment: @noorav I also play the piano, so I remember (or can quickly tell, simple methods) all key signatures, but it's usually just the tonic.
Comment: Quite helpful. But just wondering does this task often arise in real practice? As a guitar player I find that probably it's practical to view G-Myxolyd. as degree V (dominant) of Cmaj, hence work from Cmaj as basis for all other modes. Or view Fm# as degree III of Dmaj and think of Dmaj as tonal centre (without thinking of key signature, because it's probably not easy to quickly recall that F# Phryg. has two ##s unless one improvises in different Phryg. keys).
Comment: @alexsms as a guitar player, I have the same issue. It wouldn't be that easy to recall the number of sharps in different modes
Comment: @alexsms, I've written up a lookup table for you. I can't think of a way to do this without memorising the relationships. My background being art music, i learned all of these relationships via scales, keys, key signatures & intervals. I can offer no insight into tricks for musicians who don't work with those building blocks as instant recall.
Comment: @DeanRansevycz, thanks. I guess one could use it too to better remember (understand) some chord substitutions.
Comment: @noorav - I've updated the answer, hopefully it's more helpful.
Answer: Given that a 'relative key' is another that contains the same notes, as in A minor is 'relative' to C major.
The simplest has to be go to the parent key of the mode. Taking your example of F# Phrygian we go back to parent D major. So 'relative' modes for this example are E Dorian, G Lydian, A Mixolydian, etc. Thus making the relative Locrian to G Mixolydian B Locrian.
Your G Mixolydian example takes us to C major, which gives us D Dorian, E Phrygian, F Lydian, etc.
Knowing key sigs will help with #/b, and it's important to know those anyway.

Finding relative keys of modes


Why am I getting a 403 in attempting to access view-source:https://ccachicago.pragmatometer.com/admin/static/css/base.css?

Question: I've been trying by hook or by crook to get /admin/static/ served up in an Apache SSL VirtualHost, preferably by Apache (which is forwarding SSL traffic to a Django Gunicorn instance), or barring Apache, at least Gunicorn serving up static content while I work on a better solution.
I'm getting an Apache-served 403, and all the permissions I've checked suggest that the /usr/lib/python2.7/dist-packages/django/contrib/admin/static/ directory (and parents as needed) are readable and executable by the user running the server.
Can you see anything wrong in the VirtualHost below which would explain why Apache isn't serving up the directory in question as an Aliased directory?
<VirtualHost *:443>
    ServerName ccachicago.pragmatometer.com

    Alias /media/ "/home/jonathan/ccachicago/media/"
    <Directory "/home/jonathan/ccachicago/media/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

    Alias /admin/static/ "/usr/lib/python2.7/dist-packages/django/contrib/admin/static/"
    <Directory "/usr/lib/python2.7/dist-packages/django/contrib/admin/static/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

    ProxyPass /media/ !
    ProxyPass /admin/static/ !
    ProxyPass / http://localhost:8080/
    ProxyPassReverse / http://localhost:8080/

     SSLEngine On
     SSLCertificateFile /etc/apache2/ssl/ssl.crt
     SSLCertificateKeyFile /etc/apache2/ssl/ssl.key
    ServerAdmin email@example.com
</VirtualHost>

--UPDATE--
I get the same 403 error page if I comment out the deny/allow lines. The logfile has:
[Mon Jan 27 21:52:34.297099 2014] [authz_core:error] [pid 4818] [client 188.8.131.52:44895] AH01630: client denied by server configuration: /usr/lib/python2.7/dist-packages/django/contrib/admin/static/css

So something in my configuration is apparently not working; I now have:
<VirtualHost *:443>
    ServerName ccachicago.pragmatometer.com

    Alias /media/ "/home/jonathan/ccachicago/media/"
    ErrorLog /var/log/apache2/error.log
    <Directory "/home/jonathan/ccachicago/media/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

    Alias /admin/static/ "/usr/lib/python2.7/dist-packages/django/contrib/admin/static/"
    <Directory "/usr/lib/python2.7/dist-packages/django/contrib/admin/static/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        #Order deny,allow
        #Deny from all
        #Allow from 127.0.0.0/255.0.0.0 ::1/128
        #Allow from 0.0.0.0 ::1/128
        #Allow from all
    </Directory>

    ProxyPass /media/ !
    ProxyPass /admin/static/ !
    ProxyPass / http://localhost:8080/
    ProxyPassReverse / http://localhost:8080/

     SSLEngine On
     SSLCertificateFile /etc/apache2/ssl/ssl.crt
     SSLCertificateKeyFile /etc/apache2/ssl/ssl.key
    ServerAdmin firstname.lastname@example.com
</VirtualHost>

Comment: Your Log snippet shows that you're not accessing the server from 127.0.0.1 or ::1 so the access restrictions you have in place will deny access when they are in force. After making the changes to your config you did remember to restart apache didn't you ?
Comment: I remembered to restart Apache.
Answer: You're only allowing access from 127.0.0.0 and ::1. Are you sure that you are accessing the site via these addresses? I would first disable the access restrictions and see what happens
from there.

Your Log snippet shows that you're not accessing the server from 127.0.0.1 or ::1 so the access restrictions you have in place will deny access when they are in force.
Comment: Thank you; I saw the same 403 in response to "Allow from all". -- Is there more to it than setting "Allow from all"?
Comment: @JonathanHayward: What do you get with no access restrictions ? What do your logs say ?


Drush on production server without Composer

Question: I want Drush 9 on a Drupal 8 production server without having to install Composer. Is there any way to do this?
And if that is not possible, can anyone suggest an alternative to Drush especially for maintenance, updating and syncing the site with dev and staging?
Comment: To be blunt, and ignoring other issues, composer is a ridiculous resource hog. I have some very small D8 sites I want to put on a server or two and be able to ssh in and run the odd drush command. I do not need, nor do I want, the overhead of composer. As the servers and sites are small I am likely to have issues even using composer without running out of memory. I will have very few dependencies to worry about and I am the only person working on the site.
Comment: Not to mention that if you're looking for a way to break your environment, Composer is perfect way to ruin your day. One little mistake, and your environment becomes smoking rubble
Comment: Why not install composer? You can install composer in a project independent way, e.g. inside a drush-specific directory, and that would allow you to not only have drush 9 but also run future updates on it without workarounds.
Comment: Just as a side not, yes composer can be a hog when doing dependency resolution and picking a set of compatible versions.  If you have a lockfile, though, and just do a `composer install`, the overhead is a lot smaller since it just installs what is in the lockfile.
Answer: If you don't want composer in production on your server, you can run the composer commands locally to install Drush, then make sure to push the vendordirectory to the server. Make sure to also commit the composer.json and composer.lock files, so that anyone else working with the codebase can handle composer on their own local machines as well.
In this way you never need composer on the remote production server, but can get drush.
Comment: Anything in your vendor file will be required. If you aren't using Composer on your server, you need to be doing it locally, and then pushing to the server. The files in the vendor folder will be required.
Comment: The vendor directory is set to be ignored in the .gitignore file. It sounds like you need to fix that.
Comment: If I push the vendor directory. Do I need to take further steps to then use drush on the server? So, for example, I have several sites on the server and want to use Drush to manage them globally via aliases. I am guessing this wont work out of the box with only a vendor directory for each site?
Comment: Ideally you should install Drush Launcher (http://docs.drush.org/en/master/install/), but that said, Drush will exist at vender/drush/drush/drush, so you can use it without Drush Launcher if there is some reason why you are not able to install that.
Comment: Is there any point to everything else in the vendor folder then? Shouldn't I delete or gitignore the rest of that folder since there will be no composer on the server?
Comment: Composer doesn't need to live on the web server. Use a build service like Travis or Circle CI.
Comment: This just seems to produce a broken (500 error) website with missing dependencies. Any more information about how to do this?
Comment: You'll need to track down the error. If you are missing files form the vendor directory, it will mean you haven't got the full directory to your server.
Comment: I have tried following composers own docs on this: https://getcomposer.org/doc/faqs/should-i-commit-the-dependencies-in-my-vendor-directory.md However I am getting what I guess is a submodule issue as only the folders of the vendors is bein added not the actual files...
Answer: On the production stage we don't need Composer (neither composer.json nor composer.lock), Composer is used to build your project and to generate the vendors after that it is necessary to pack all the generated files and to deploy them on the target server.
Comment: How do I get Drush on the production server without composer being present? Sorry if this seems like a silly question but the composer and drush guides out there are super confusing on this.
Comment: how exactly do you "pack" the vendors folder?
Comment: Drush is present in your vendors directory that will be deployed in your server, so all you have to do is settings the path to drush /vendors/drush/drush.
Comment: You can use maven or phing to package your web site and then put it in the server.
Answer: If you build out your D8 site locally in your dev environment using Composer, you'll have Drush 9. Then, if you commit your D8 instance (Git) and pull onto your server, you'll have Drush- albeit you'll have to execute using the full path. For example: From my command line on a Ubuntu 18.04, I can execute

$ vendor/bin/drush pml --no-core

My D8 instance on the Production servers contains the following at the root.

Notice that /vendor is committed and present in this repository. And that contains /vendor/bin/drush
Refer to the Drush Documentation for more info.
Hope this helps!


Azure VM Backup of v2 Hosts?

Question: I have 16 VMs running Win Sever 2012. Some were created with ARM templates, some manually in the new portal.
I need to now get them all discovered by Azure Backup so they cab be captured at the "VM level". These new
VMs do not show up in the classic portal and do not show as "discovered".
Does scripting exist that can force the discovery so I can make VM-level backups of these resources?
Thanks.
Answer: V2 VM backup in Resource Group
Its available now
check with this link
https://azure.microsoft.com/en-us/documentation/articles/backup-azure-vms-first-look-arm/
Comment: This is link only answer.
Answer: Unfortunately this is not possible! There is simply no visibility from the Service Management backup to resource management VMs
I have some workarounds in this question.
And according to Azure it is started on the feedback site. It is frustrating that they've released v2 without backup support, but we just need to work around it for now.

Azure VM Backup of v2 Hosts?


Steepest descent with quadratic form converges in 1 iteration

Question: Well I'm stuck on an exercise given:

The steepest descent method is applied to the quadratic form
$$Q(\mathbf{x}) = \tfrac{1}{2}\mathbf{x}^TA\mathbf{x} - \mathbf{b}^T\mathbf{x} + c$$
where $A$, $\mathbf{b}$ and $c$, are matrix, vector and scalar constants. Under what condition on the matrix $A$ does the steepest descent method converge to the exact minimum in 1 iteration, from any initial condition $x_0$? [Hint: If the initial search line $\mathbf{x}_0 + \alpha \mathbf{d}_0$ includes the exact minimum of $Q(\mathbf{x})$, then the method will converge in 1 iteration.]

More specifically, some conditions about the matrix $A$ are asked, seeing this is a multiple choice question, the options:

$A$ is a multiple of the identity matrix
$A$ is diagonal
$A$ is symmetric
$A$ is positive definite
$A$ has only positive eigenvalues
$A$ is equal to $\mathbf{b}\mathbf{b}^T$
It never converges in 1 iteration
It always converges in 1 iteration

Now following the "hint", the exact solution is given by solving $Q'(\mathbf{x}) = 0$:
$$A\mathbf{\tilde{x}} - \mathbf{b} = 0$$
$$\mathbf{\tilde{x}} = A^{-1} \mathbf{b}$$
From the definition of the steepest descent method ($x_{n+1} = x_n - \alpha f'(x_n)$):
$$\mathbf{x}_1 = \mathbf{x}_0 + \alpha (A\mathbf{x}_0-\mathbf{b})$$
Now $\mathbf{x}_1 = \mathbf{\tilde{x}}$ has to be solved, or written out:
$$\mathbf{x}_0 + \alpha (A\mathbf{x}_0-\mathbf{b}) = A^{-1} \mathbf{b} \qquad \forall\ \mathbf{x}_0,\alpha \in \mathbb{R}$$
Now I'm stuck, what can be said about $A$? Geometrically it feels like solution should be easy to be found, an example is a "cone like surface".
Answer: In order to have existence of a minimizer, $A$ must be positive semi-definite and the equation $Ax=b$ must be solvable.
Assume the steepest descent converges in one step,  i.e. $x_1$ is the solution. Then
$$
0=Ax_1-b=A(x_0 - \alpha(Ax_0-b))-b = (I-\alpha A)(Ax_0-b)
$$
hence $Ax_0-b$ has to be an eigenvector if $A$ (or $Ax_0-b=0$).
The reverse direction is also true, you need that $Ax_0-b$ is an eigenvector to a positive eigenvalue of $A$.
Can you find now,
which of the given choices imply one-step convergence?
Answer: Hint: two conditions are important; first, the function $Q(\mathbf{x})$ must actually have a minimum, and the inverse of the matrix $\mathbf{A}$ must exist.
Comment: Invertibility of $A$ is not necessary for existence of minimizers of $Q$.
Comment: I'm sorry but I'm really stuck here, and I don't see how this would solve it? I'm taking for granted teh minimum exists (from the way the question is asked). - As well as that A is invertible.
Comment: @daw In the question "the exact minimum" is mentioned; for a unique solution to exist, matrix $A$ must be non-singular.
Comment: Sorry, I overlooked 'exact' here, and wrote a separate answer that does not rely on the uniqueness of the minimizer.
Comment: OK, thanks for clearing that up.


Unit Root Test - Order of Integration (Johansen)

Question: Let's assume we have 2 variables and test each of them for a unit root with the ADF test. When plotting the data, we can see that it has some up/down movements but is overall clearly trending upwards.
At level, variable A is stationary with "no constant", "constant" and "constant+trend" (p<0.05). Variable B is only stationary with "constant+trend" (p<0.05).
At first differences, both variables are stationary at "no constant", "constant" and "constant+trend" (p<0.05).
Is it o.k. to conclude that Variable A and B are both integrated of the same order I(1), or do we have to assume that Variable B is I(0) with respect to a trend and Variable B is I(1)?
I want to conduct a Johansen Cointegration test and after reading the literature and several posts, I'm still not sure if it's okay test for correlation with the data at level (and a trend).
Answer: Variable A seems to be stationary as whatever the specification ("no constant", "constant", "constant+trend") of the test regression you still reject the unit root hypothesis. Therefore you cannot conclude that A~I(1). Since at least one of the two variable is stationary, you cannot proceed to cointegration analysis (recall that it requires both variables to be integrated to begin with).
Note on proceeding to testing stationarity of A at first differences: you should not do that as A is already stationary in levels. If A is stationary in levels, testing for stationarity in first differences is redundant when your question is whether A should be included in cointegration analysis.
Comment: When you only have two variables one of which is stationary and one of which is integrated, there is no way of obtaining a stationary linear combination of the two --> No cointegration. If both of the variables are stationary, again you cannot talk about cointegration. Only in systems of more than two variables is your above quote meaningful.
Comment: So there's no way to test the order of cointegration of variable A and B?
Comment: Lütkepohl and Krätzig, Applied Time Series Econometrics, 2004:

"Occasionally it is convenient to consider systems with both I(1) and I(0) variables. Thereby the concept of cointegration is extended by calling any linear combination that is I(0) a cointegration relation, although this terminology is not in the spirit of the original definition because it can happen that a linear combination of I(0)variables is called a cointegration relation."
Comment: So it's no possibility to use the level of the I(0) variable and the
first difference of the I(1) variable?
Comment: If you do that, you are working with two I(0) objects. You can use them for a variety of things but not for cointegration analysis. If the variables are not *integrated*, they cannot be *co-integrated* (look at the construction of these words).
Comment: I think this last sentence made it clear.


Uniform convergence of the complex Fresnel integral

Question: Consider the integral $I(\lambda) = \sqrt {\frac {\lambda \mathbb{i}}{\pi}}^n \int_U \mathbb{e}^{-\mathbb{i} \lambda \|x\ - x_0|^2} f(x) \mathbb{d}x, \lambda>0$ and $U$ some open neighbourhood of $x_0 \in \mathbb{R}^n$. It is known that $\lim \limits_{\lambda \to \infty} I(\lambda) = f(x_0)$. Is this limit uniform with respect to $x_0$?
In general, $f$ is taken to be smooth with compact support. May I relax these hypotheses?
Comment: The integral is a linear functional on $f$, so it certainly can't be true that the limit is $1$ for all $f$.  In particular it's $0$ if $U$ is disjoint from the support of $f$.  Perhaps you mean the limit is $f(x_0)$?
Comment: Of course, thank you. Corrected.
Answer: When you say "uniform with respect to $x_0$", I presume you're keeping $U$ and $f$ fixed.    If $x_0$ is not in the closure of $U$, the limit must be $0$.  A function that is $f(x_0)$ for $x_0$ in $U$ and $0$ for $x_0$ outside the closure of $U$ won't be continuous unless $f$ is $0$ on the boundary of $U$, and so it isn't the uniform limit of continuous functions.  But $I(\lambda)$ is a continuous function of $x_0$. So the limit can't be uniform.
EDIT:
OK, take $U = \mathbb R^n$ and suppose, more generally, that $f$ is a Schwartz function.  Then $I(\lambda, x_0)$ (to include the dependence on $x_0$) is the convolution of $f$ with the bounded continuous function
$ g_\lambda(x) = \sqrt{\lambda i/\pi} \exp(-i\lambda |x|^2)$.  Its Fourier
transform is the product of $\widehat{f}$ and $\widehat{g_\lambda}$:
$$ \widehat{g_\lambda}(p) = \exp(i|p|^2/(4 \lambda))$$
Given $\epsilon > 0$, let $\chi_B$ be the indicator function of
a ball $B$ large enough so that
$$\|(1 - \chi_B) \widehat{f} \widehat{g_\lambda}\|_{L^1}  =
\|(1 - \chi_B) \widehat{f} \|_{L^1}< \epsilon/3$$
Take $\lambda$ large enough so $|g_\lambda - 1| < \delta$ on $B$,
where $\delta \|\widehat{f}\|_{L^1} < \epsilon/3$, so that
 $$\|\widehat{f} (\widehat{g_\lambda} - 1)\chi_B\|_{L^1} < \epsilon/3$$
Then $$\|\widehat{f} - \widehat{f} \widehat{g_\lambda}\|_{L^1} \le
\|\widehat{f} - \widehat{f} \chi_B \||_{L^1} +
\| \widehat{f} \chi_B - \widehat{f}\widehat{g_\lambda} \chi_B\|_{L^1}
+ \|\widehat{f}\widehat{g_\lambda} \chi_B - \widehat{f} \widehat{g_\lambda}\|_{L^1}   < \epsilon$$
and then $|f(x_0) - I(\lambda, x_0)| < \epsilon$ for all $x_0$.
EDIT: Looking back at this, it seems all we're actually using about $f$ is that its Fourier transform is in $L^1$,
Comment: Agreed. Nevertheless, it seems that the possibility remains open if $\mathbb{supp} f \subset U$ (and, in particular, for $U = \mathbb{R}^n$). (And yes, $U$ and $f$ are fixed.)


PHP editar imagem

Question: O meu problema é que o ficheiro img_editar.php não esta a mandar as imagens para a base de dados (é para adicionar varias imagens ao mesmo tempo), o ficheiro adicionar imagem funciona muito bem, clico na opção editar e quando mudo por exemplo o nome da imagem sou obrigado a mudar as imagens, só que a mudar as imagens estas não são enviadas para a BD, eu até tenho a função para manter as imagens if($_FILES['img']['size'] > 0) mas não esta a funcionar.
<?php
    include "bd.php";
    include "_config.php";
    include "admin.php";

  $target = "../upload/";
  $target = $target . basename( $_FILES['img']['name']);
  $pic=($_FILES['img']['name']);

 $data=date("ymd");
$query="update portfolio_imagens set nome='".$_POST['nome']."', descricao='".$_POST['descricao']."', cliente='".$_POST['cliente']."', publicacao='".$data."' where id=".$_GET['id'];
    $result=mysql_query($query);
 $picas=0;
    echo $query;
    if($_FILES['img']['size'] > 0){
      //echo "1";
  $query2="update portfolio_imagens set  imagem='".$pic."' where id=".$_GET['id'];
  $result2=mysql_query($query2);
  $picas=1;
}

  if(move_uploaded_file($_FILES['img']['tmp_name'], $target) || $picas=1) //
  {

 echo "

          Notícia criada com sucesso!

";
         echo '';

 }else{
  echo "

          Erro!

";
         echo '';
}
?>

Comment: [Como debugar código em PHP?](http://pt.stackoverflow.com/q/6037/201)
Comment: se fizerem favor metam me isso em modo codigo que nao sei como se faz aqui
Answer: Veja, eu coloquei um if para verificar se o array contendo o nome da imagem existe e se ela retorna algo diferente de vazio. Caso ela retorne é porque o usuário enviou também uma imagem além do nome, descrição etc.. Então ele adiciona uma variavel com o conteúdo para fazer com que adicione um update da imagem no $result e já faz o upload da imagem.
<?php

include "bd.php";
include "_config.php";
include "admin.php";

$target = "../upload/"; //Diretório aonde irá salvar
$target = $target . $_FILES['img']['name']; //Caminho completo
$pic= $_FILES['img']['name']; //Pega nome do arquivo com extensão

$ID = $_GET['id']; //ID

if(isset($_FILES['img']['name']) && !empty($_FILES['img']['name'])){ //Verifica se existe o array name e se ele é diferente de vazio

  $condition_aditional = ", imagem='".$pic."'"; //Linha adicional no UPDATE
  move_uploaded_file($_FILES['img']['tmp_name'], $target); //Move o arquivo de foto
}

$result=mysql_query("UPDATE portfolio_imagens SET nome='".$_POST['nome']."', descricao='".$_POST['descricao']."', cliente='".$_POST['cliente']."', publicacao='".$data."' ".$condition_aditional." WHERE id=".$ID.""); //Executa a Query

  if($result){

 echo "

          Notícia criada com sucesso!

";
         echo '';

 }else{
  echo "

          Erro!

";
         echo '';
}
?>

Comment: @Alisson Acioli, seria interessante você informar a alteração que foi feita.
Comment: nao funciona continua a fazer o mesmo
Comment: sim obrigado pela explicação mas nao está a dar :s

PHP editar imagem


SQL trigger: Change nvarchar size

Question: I have several tables with columns nvarchar(max). I am building a trigger which tries to store the cells in sql_Variant. Nvarchar(max) cannot be stored in sql_variant. How I can check if it is nvarchar(max) and then convert it to nvarchar(4000) so I can avoid this?
EDIT: Most probably the error occurs here:
set @sql = 'select @audit_oldvalue=[' +@Item +'] from #tempTrigT';

Because @audit_oldvalue is sql_variant but [' +@Item +'] from #tempTrigT can be nvarchar(max)
If I edit something in the input table I get this error:

operand type clash nvarchar(max) is incompatible with sql_variant

Tables Structure:
input table

output table

Code where I want to convert:
Select * into #tempTrigT from (select * from deleted where @Action in ( 'U','D')) A UNION (select * from inserted where @Action ='I')

Select @sql = @sql + 'Case when IsNull(i.[' + Column_Name +
'],0) = IsNull(d.[' + Column_name + '],0) then ''''
 else ' + quotename(Column_Name, char(39)) + ' + '',''' + ' end +'
from information_schema.columns
where table_name = 'Forms' and column_name <>'rowguid' and column_name <>'modifieddate'
--Define output parameter
set @ParmDefinition = '@OutString varchar(max) OUTPUT'
--Format sql
set @sql = 'Select @OutString = '
+ Substring(@sql,1 , len(@sql) -1) +
' From dbo.Forms i  ' --Will need to be updated for target schema
+ ' inner join #tempTrigT d on
i.id = d.id'  --Will need to be updated for target schema
--Execute sql and retrieve desired column list in output parameter
exec sp_executesql @sql, @ParmDefinition, @OutString OUT

DECLARE @Items VARCHAR(max)

 set @Items = @OutString;

 DECLARE @Item VARCHAR(50)
 DECLARE @Pos INT
 DECLARE @Loop BIT
 SELECT @Loop = CASE WHEN LEN(@OutString) > 0 THEN 1 ELSE 0 END
 WHILE (SELECT @Loop) = 1
 BEGIN
 SELECT @Pos = CHARINDEX(',', @OutString, 1)
 IF @Pos > 0
 BEGIN
 SELECT @Item = SUBSTRING(@OutString, 1, @Pos - 1)
 SELECT @OutString = SUBSTRING(@OutString, @Pos + 1, LEN(@OutString) - @Pos)
----------------------------------

set @audit_field = @Item;

set @sql = 'select @audit_oldvalue=[' +@Item +'] from #tempTrigT'; --ERROR LINE
EXEC SP_EXECUTESQL @sql,N'@audit_oldvalue sql_variant OUTPUT',@audit_oldvalue OUTPUT -- If inserted @audit_oldvalue gets the new value

set @sql = 'select @audit_value=i.[' +@Item +'] from dbo.Forms i  inner join #tempTrigT d on i.id = d.id';
EXEC SP_EXECUTESQL @sql,N'@audit_value sql_variant OUTPUT',@audit_value OUTPUT

if @Action = 'U'
begin
    insert into [dbo].[AuditTrailForms]([TSid],[TSField],[OldValue],[NewValue],[changedate],[Change_Action],[Change_user],[Columns_Updated])
    select id,@audit_field, @audit_oldvalue, @audit_value,getdate(),@Action, coalesce(ModifiedBy,suser_name()), @Items
    from inserted

end

     END
 ELSE
 BEGIN
 SELECT @Item = @OutString
 SELECT @Loop = 0
 END
 END

Comment: Which dbms are you using?
Comment: @jarih sql server 2012
Comment: Did you try "CONVERT(nvarchar(4000), @nvarcharmaxparam)" ?
Comment: cast(left(yourColumn),4000) as varchar(4000))
Comment: @sarslan please check edited post
Comment: Look: https://stackoverflow.com/questions/626899/how-do-you-change-the-datatype-of-a-column-in-ms-sql
Comment: @MuratÖzcan I dont want to alter the tables. I want to convert the value of the nvarchar(max) column and store it in sql_variant
Comment: add tables structure and trigger script
Answer: You should be able to just cast it as VARCHAR(4000).
Select @sql = @sql + 'Case when IsNull(CAST(i.[' + Column_Name +
'] AS VARCHAR(4000)),0) = IsNull(CAST(d.[' + Column_name + '] AS VARCHAR(4000)),0) then ''''
 else ' + quotename(Column_Name, char(39)) + ' + '',''' + ' end +'
from information_schema.columns
where table_name = 'Forms' and column_name <>'rowguid' and column_name <>'modifieddate'

Or if you want to only do this for those columns (I don't believe it would an issue unless for performance reasons).
Select @sql = @sql + 'Case when IsNull('
    + CASE WHEN CHARACTER_MAXIMUM_LENGTH = -1 THEN 'CAST(' ELSE '' END
    + 'i.[' + Column_Name +  ']'
    + CASE WHEN CHARACTER_MAXIMUM_LENGTH = -1 THEN ' AS VARCHAR(4000))' ELSE '' END +
    ',0) = IsNull('
    + CASE WHEN CHARACTER_MAXIMUM_LENGTH = -1 THEN 'CAST(' ELSE '' END
    + 'd.[' + Column_name + ']'
    + CASE WHEN CHARACTER_MAXIMUM_LENGTH = -1 THEN ' AS VARCHAR(4000))' ELSE '' END +
    + ',0) then ''''  else '
    + quotename(Column_Name, char(39)) + ' + '',''' + ' end +'
from information_schema.columns
where table_name = 'Forms' and column_name <>'rowguid' and column_name <>'modifieddate'

Comment: OP has posted questions regarding this piece of code a number of times albeit different questions but has not accepted an answer in any thus far (that I've seen).  Not sure if OP is clear on what is actually required.  (Apologies if not in right place, not enough rep so can only comment on my own posts)
Comment: thank you for you answers. I made different questions because there were in the same piece of code but different topic. Of course I have accepted questions.
Comment: I tested the 2nd snippet but I receive the same error again. I dont want to convert everything in VARCHAR(4000) thats why I chose snippet #2
Comment: Nothing. But I guess now that the error occurs in this line: set sql = 'select @audit_oldvalue=[' +Item +'] from #tempTrigT';
EXEC SP_EXECUTESQL sql,N'@audit_oldvalue sql_variant OUTPUT',@audit_oldvalue OUTPUT    Where @Item contains the column that change occured. so it fetches the Item column from tempTrigT which is nvarchar(max) and stores it sql_variant
Comment: What is tempTrigT?  The creation of that table not included in your post
Comment: Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/153859/discussion-between-aggicd-and-preql).

SQL trigger: Change nvarchar size


É preciso de um computador MacOS para programar Swift 4 e ObjC?

Question: Estou pesquisando bastante sobre programação de aplicativos para plataforma iOS, e todos os cursos online que encontro os computadores são da linha Apple. Para programar Apple, é preciso ter um computador da Apple?
Answer: De maneira alguma, elas podem ser usadas em diversas plataformas. Essas linguagens não são "da Apple", são apenas mantidas pela Apple.
É claro que se você quer fazer algo para o MacOS ou o iOS complica um pouco fazer em outra plataforma. Não impede, mas dá um pouco de trabalho, especialmente para iOS que exige um MacOS para publicar sua app, mas não para desenvolver.

IDE para Windows.
Visual Studio Code - Swift
Swift for Linux

Sei que o mesmo vale para Objective-C, mas vai de Swift.
Coloquei no GitHub para referência futura.
Comment: @VictorOCV exatamente, é muito mais fácil desenvolver na plataforma que você vai aplicar. Nos links falam alguma coisa sobre fazer algo para iOS no Windows. Com Swift não sei se compensa. Se fizer em C# aí já compensa um pouco mais, principalmente se for fazer para Android também (que de forma geral tem muito mais mercado). No fim precisará de um Mac para publicar por exigência da Apple. Tem aluguel de Mac virtual.
Comment: Ah entendi, mas normalmente então, para programação de apps para apple é utilizado computadores MacOS ?
Comment: Outra coisa, então eu posso programar para apple ios, no windows ?


Stone Cech compactification of cartesian product

Question: Let $X,Y$ be topological spaces. By the universal property of the Stone-Cech compactification, there is a canonical map
$$\Phi: \beta(X\times Y)\to \beta(X)\times \beta(Y).$$
It is not very hard to show that $\Phi$ is always a surjection.
My question: Is it known when $\Phi$ is injective?
I read somewhere that $\beta(X\times Y)\cong \beta(X)\times \beta(Y)$ (through an arbitrary homeomorphism?) if and only if $X,Y$ are pseudocompact. Maybe if these spaces are homeomorphic, the canonical map is automatically injective as well?
Comment: I haven't thought too much about this particular case. But in general, it's possible to have a functor $F : A \to B$ where for all $X, Y \in A$, there is some isomorphism $F(X \times Y) \cong F(X) \times F(Y)$, but with the canonical map $F(X \times Y) \to F(X) \times F(Y)$ never being an isomorphism (and with there actually existing objects in $A$ - obviously if $A$ is empty than the above is trivial). So I doubt you can read anything from the two spaces merely being homeomorphic.
Comment: See https://www.ams.org/journals/tran/1959-090-03/S0002-9947-1959-0105667-4/ where the case of arbitrary products (not merely two factors) is done.
Comment: @GEdgar Are you sure that paper answers the question? In that paper they seem to talk about "arbitrary homeorphism" instead of "the canonical surjection is injective".
Answer: Let $X, Y$ be completely regular topological spaces, consider $X \times Y$ as a subspace of $\beta X \times \beta Y$.
Hence $\beta X \times \beta Y$ is a compactification of $X \times Y$. As was mentioned by the OP, there exists a unique continuous extension $\Phi: \beta(X \times Y) \rightarrow \beta X \times \beta Y$ and $\Phi$ is surjective.
The following are equivalent:

$\Phi$ is injective.
$\Phi$ is a homeomorphism.
$\beta X \times \beta Y$ is "the" Stone-Cech compactification of $X \times Y$, i.e., there exists a homeomorphism $\Psi: \beta(X \times Y) \rightarrow \beta X \times \beta Y$, which extends the embedding.
$X$ is finite or $Y$ is finite or $X\times Y$ is pseudocompact.

Proof: 1 $\Leftrightarrow$ 2 and 2 $\Leftrightarrow$ 3 are obvious (for "$\Leftarrow$" note that $\Phi$ and $\Psi$ coincide on $X \times Y$, hence $\Phi = \Psi$).
3 $\Leftrightarrow$ 4 is the hard part, and essentially due to Glicksberg, see his paper cited above by GEdgar, see also Engelking, General Topology, 3.12.21 (c). (For "$\Leftarrow$" note that, if $X$ is finite, $X = \bigoplus_{i=1}^n \{x_i \}$, then $\beta (X \times Y) \cong \beta(\bigoplus_{i=1}^n Y) \cong \bigoplus_{i=1}^n \beta Y \cong X \times \beta Y \cong \beta X \times \beta Y$.)
It should be noted that if $X \times Y$ is pseudocompact, $X,Y \neq \emptyset$, then, of course, $X$ and $Y$ are pseudocompact. But, in general, not vice versa, see Engelking, General Topology 3.10.19 and the comment after 3.10.25 for a counter-example. See also there 3.10.26 and 3.12.21 (b) for a discussion, when a product of two pesudocompact spaces is pseudocompact, or Glicksberg's paper §5.
Comment: Isn't it possible to have a homeomorphism $\beta(X\times Y)= \beta X \times \beta Y$ that does NOT extend the embedding? I think in Glicksberg's paper, such a homeomorphism need not be the identity map $X\times Y$ and then your answer might contain a flaw.
Comment: No, Glicksberg proves that every continous function $f: X \times Y \rightarrow \mathbb{R}$ has a continous extension to $\beta X \times \beta Y$. This characterizes $\beta X \times \beta Y$ as the Stone-Cech compactication via the embedding. Note, that this theorem, however, does *not* say that, if $\beta(X \times Y) \cong \beta X \times \beta Y$ (for a  homeomorphism not extending the embeddnig), then $X,Y$ are pseudocompact. But I'm not aware of any such example. Would be interesting to know, whether there is one at all.
Comment: To be more precise, $\beta X \times \beta Y$ is *equivalent* to the Stone-Cech compactication, but this is exactly 3. from my answer. See Engelking, General topology 3.6.3.
Comment: So to be completely precise: Glickberg proves the following:

The following are equivalent for topological spaces: (1) $(\beta X \times \beta Y, \kappa: X\times Y \to \beta X\times \beta Y: (x,y)\mapsto (i_X(x),i_Y(y)))$ is a Stone-Cech compactification for $X\times Y$. (2) $X,Y$ are pseudocompact. Is this the correct formulation? I was under the impression that the homeomorphism $\beta(X\times Y)\cong \beta X \times \beta Y$ could be anything, but I didn't read the paper so it appears I was wrong.
Comment: No, not exactly: it doesn't suffice that $X, Y$ are pseudocompact, but their product must be pseudocompact.  On the oteher hand, (1) holds, if $X$ or $Y$ is finite.
Comment: So if I change $(2)$ by $X\times Y$ is pseudocompact, then the statement is true?
Comment: ... or $X$ is finite or $Y$ is finite, i.e., 4. from my answer above.
Comment: At some MSE post, I saw someone claim that $\beta(\mathbb{N}\times \mathbb{N})\ne \beta(\mathbb{N})\times \beta(\mathbb{N})$ (not homeomorphic) because $\mathbb{N}^2$ is not pseudocompact. So this argument needs modification, right? Because it could be true that there exists some other homeomorphism, so we have to exclude this case.
Comment: Yes, indeed, the above does not prove that $\beta(\mathbb{N}^2)$ and $(\beta\mathbb{N})^2$ are not homeomorphic.  However, they are definitely not homeomorphic, since the latter is not extremally disconnected, see Engelking, Gen. top 6.3.21.


Get file type Node.js

Question: Hey everyone I'm trying to get the file type in Node.js. I have to rename the file before it's uploaded for version control and once the file is written then run some processing on it.
I'm aware that I can check the file type on the client but I still think it would be beneficial to do a server side check as well.
Other solutions have popped the file extension off the file name:
return filename.split('.').pop();

But since I'm renaming this file and a user could simply rename a malicious file to a whitelisted extension I'm looking for a solution that actually determines the file type, specifically CSV.
Can anyone point me in the direction of solving this?
Comment: CSV is an entirely text-based formula so it's hard to check if something is actually csv. You can check if it's all ASCII though.
Comment: what file extension do you need to support?
Comment: @ManuelSpigolon just CSV
Comment: You can use `path.extname(fullnameofthefile)`
Answer: To read the file extension you may use the snippet:
const { extname } = require('path')

console.log(extname('foo.csv')) // .csv
console.log(extname('BAR.CSV')) // .CSV

But since I'm renaming this file and a user could simply rename a malicious file to a whitelisted extension I'm looking for a solution that actually determines the file type, specifically CSV.

Usually, to implement this check, you need to read the magic bytes of the file and act accordingly for a limited set of file types but CSV is not in this list.
CSV is a plain text with a defined format, so you could:

read some line an try to parse them
define that the CSV must have a defined header like id,col1,col2,etc
use some tools that try to guess the mime type of the file (like mime-types )

Get file type Node.js


Why does the characteristic function always exist?

Question: I've read that the characteristic function of a probability distribution always exists because it's bounded. However, the characteristic function is still Taylor expanded in terms of the moments of a given probability distribution. Given the the moments don't necessarily exist, why is it that the characteristic function still exists?
Answer: If $X$ is a random variable, then what it means for the characteristic function of $X$ to exist is that the random variable $|e^{itX}|$ must have finite expectation for every $t$. This is automatically true because $|e^{itX}|=1$ for all $t \in \mathbb{R}$, hence $E(|e^{itX}|)=1<\infty$. This is true regardless of whether or not the moments exist. The characteristic function can be Taylor expanded in terms of the moments only if the moments actually exist, but in general they won't.
Comment: No, not at all.
Comment: By definition, the expectation of a complex-valued random variable is said to exist iff the absolute value of both the real and imaginary parts of the random variable have finite integral, which is equivalent to the modulus of the random variable having finite integral.
Comment: Ahh, I see. So the Taylor expansion is not fundamental to the definition of the characteristic function, then?
Comment: @shalop Why do we take the modulus value of the random variable $e^{itX}$ ?


Will batsman caught out if ball touches body first?

Question: If ball touches body first then bat and caught, will batsman be considered out?
Answer: Yes, he will be out caught - see Law 32


ADS1x15 Python library

Question: Let's say I have an ADS1015 ADC converter and a Raspberry Pi. In my project I only need to read the values of an analog sensor.
This is the library written in Python for this ADC converter for Raspberry Pi: ADS1015 github repository
Let's take as an example sampletest.py:
import time
import Adafruit_ADS1x15

adc = Adafruit_ADS1x15.ADS1115()

print('Reading ADS1x15 values, press Ctrl-C to quit...')

print('| {0:>6} | {1:>6} | {2:>6} | {3:>6} |'.format(*range(4)))
print('-' * 37)

while True:
    values = [0]*4
    for i in range(4):
        values[i] = adc.read_adc(i, gain=GAIN)
    print('| {0:>6} | {1:>6} | {2:>6} | {3:>6} |'.format(*values))
    time.sleep(0.5)

I need to know very precisely what's the sample rate I get on my computer. I know that I can set the sample rate of the ADC anywhere from 64 up to 3200, but whenever I read from the computer, there's a slight delay before the data comes.
For example: if I set the sample rate of the chip at 128 and in Python let's say I delay the reading by 1/60 seconds (so we have a theoretical sample rate of 60 sps), what would the actual sample rate on the computer be? (i.e. 59, 57 etc) And does the waiting time get smaller if I choose a higher sample rate of the ADC? (like 256 sps)
At the moment, I don't have the ADC converter to make tests, so I ask you here.
Answer: For the ADS1015 specifically, there exist 2 different modes it can operate in:

Single-Shot
Continues-Conversion

Single-Shot mode
In Single-Shot mode, the ADC is in a power-down state until u ask it for a conversion of the current voltage applied. The ADS1015 then needs to power up (taking roughly 25µs) and samples the voltage.
If you want to determine the sample rate for this mode, you need to add the time it takes for the ADS1015 to perform the conversion.
SR = 1 / (t_python + t_conversion)

If we now decrease the time it takes the ADS1015 to make the conversion, we can increase the sample rate of our python program.
Continues-Conversion mode
In Continues-Conversion mode, the ADS1015 is constantly sampling data at the specified sample rate.
Determining the sample rate for this mode is a bit more complicated than for the Single-Shot case. This is due to the fact that the python program and the ADS1015 are now operating truly asynchronous.
There are now 2 cases:

We want to read where there is a conversion ready
We want to read when the ADS1015 updates the conversion register

Case 1 is pretty simple, as this just means we don't need to wait for the conversion of the ADC:
SR = 1 / (t_python)

Case 2 means we need to wait for the conversion to finish, which in the worst case could take up to t_conversion time
SR = 1 / (t_python + t_remainingConversion)

with
0 < t_remainingConversion < t_conversion

Code example
Your given example is based on the Adafruit library for the ADS1015.
The adc_read function exclusively operates the ADS1015 in Single-Shot mode, which is quite slow, because of the 25µs start up. It also just waits a specified time based on the sample rate for the result of the conversion.
One optimizing that could be implemented is the usage of the ALERT/RDY pin to notify that a conversion is ready.

All the information concerning the ADS1015 can be found in the datasheet, which in general is highly recommended to read (or skim if you want a quick overview) for any electrical component.

ADS1x15 Python library


Determining whether a condition holds for all members of a list

Question: This is something that has eluded me for a while. For example, say I had a list of numbers, and I wanted to check if each number was divisible by EVERY number in another list, how would I do that? All I can think of is:
mylist = [1,2,3,4,5,etc...]
divisible = [1,2,3,4,5,6,etc...]
for i in mylist:
    for x in divisible:
        if i % x == 0:
            <do something>

Obviously, this will only check if any element in mylist is divisible by any element in 'divisible', but not if it's divisible by ALL the elements in that list. How would I go about doing this? Thanks
Answer: You could use all:
>>> i = 2*3*17*23
>>> divs = [2, 17, 23]
>>> all(i % x == 0 for x in divs)
True
>>> all(i % x == 0 for x in [2,13,17])
False

See also any and next:
>>> any(i % x == 0 for x in [5, 17, 101])
True
>>> next(x for x in [5, 17, 101] if i % x == 0)
17

Comment: With my FP hat on, I should be embarrassed for having come up with an procedural solution; this is far cleaner.
Comment: In relation to this, how would I get it to return the numbers from 'i' which is divisible by every number in 'divs', rather than just true?
Comment: @user2253076: usually when you have a separate question it's better to ask that separately.  But a listcomp like `[i for i in mylist if all(i % x == 0 for x in divs)]` should do it.


If two vowels and four consonants are put away from the word MATHEMATICS, how many distinct possible collections of letters are there?

Question: I am a student in middle school, and I would really appreciate if anyone could explain this question in simpler terms.

On the refrigerator, MATHEMATICS is spelled out with $11$ magnets, one letter per magnet. Two vowels and four consonants fall off and are put away in a bag. If the T's, M's, and A's are indistinguishable, how many distinct possible collections of letters could be put in the bag?

I think that if we use complementary counting, we could subtract something from $C(11, 6)$, but I do not know what that something would be…
Comment: You need to look at the set of consonants and the set of vowels separately, then multiply the results.  Keep in mind that it is the number of times a repeated letter occurs that matters, not which copy of that letter is used.
Answer: If the number of collections of two vowels is $A$, and that of four consonants is $B$, then the answer is $A·B$.
First to calculate $A$. There are four vowels: A, A, E, I. If two A's are selected, then there is only one collection, i.e. (A, A). If only one A is selected, then there are two collections: (A, E), (A, I). If no A is selected, then there is one collection: (E, I). Thus $A = 4$.
Next to calculate $B$. There are seven consonants: M, M, T, T, H, C, S.
Case 1: Three consonants are selected from H, C, S. To complete the collection, one more consonant has to be selected from M, M, T, T, so it is either an M or a T. Thus there are$$
C(3, 3) × 2 = 2
$$
collections.
Case 2: Two consonants are selected from H, C, S. To complete the collection, two more consonants have to be selected from M, M, T, T, so it is can be (M, M), (M, T) or (T, T). Thus there are$$
C(3, 2) × 3 = 9
$$
collections.
Case 3: One consonant is selected from H, C, S. To complete the collection, three more consonants have to be selected from M, M, T, T, so it is can be (M, M, T) or (M, T, T). Thus there are$$
C(3, 1) × 2 = 6
$$
collections.
Case 4: No consonant is selected from H, C, S. Then there is only one collection, i.e. (M, M, T, T).
Therefore, $B = 2 + 9 + 6 + 1 = 18$.
The final answer id $4 × 18 = 72$.
Answer: It's easier to count the number of ways to leave two vowels and three consonants on the refrigerator in a random order.
Let's start with the consonants. We can construct a sum equal to 3 as just 3 or as 2+1 or as 1+1+1. The first partition doesn't work, because we don't have three identical letters. It has to be either one pair and one single or three singles. Then
$$N_{\text{consonants}} =
\binom {2 \color{gray}{\small\text{ pairs}}}
 {1 \color{gray}{\small\text{ pair}}} \times
\binom {4 \color{gray}{\small\text{ singles}}}
 {1 \color{gray}{\small\text{ single}}} +
\binom {5 \color{gray}{\small\text{ singles}}}
 {3 \color{gray}{\small\text{ singles}}} = 18.$$
Now we have to do the same for the vowels and multiply.
Answer: We can use casework for this.
For the vowels, there are two identical letters and two distinct letters. So, the number of ways for the vowels is $1+$3C2$=4$.
The $1$ is because there is one way to choose two identical vowels. The 3C2 is because there are that many ways to choose $2$ objects out of $3$ distinct vowels. Note that it is $3$ and not $4$ because it is distinct objects.
Similarly, the number of ways for the consonants is $1+2*$4C2$+$5C4$=1+12+5=18$.
So, the total number of ways is $4*18=\boxed{72}$.
Answer: We count the number of ways to choose the vowels and the consonants separately. There are four vowels, of which two are As. If there are no As, then we must choose both the remaining vowels, so there is $1$ choice; if there is one A, then we can choose the remaining vowel in $2$ ways; and if there are two As, then there are no vowels left to choose, so there is $1$ choice. This makes $1 + 2 + 1 = 4$ distinct pairs of vowels.
There are seven consonants, of which two are Ts and of which two are Ms. Since we must choose four consonants, we must use at least one of the Ts and Ms.
If we use one T and no Ms, we have only $1$ choice (use the three remaining consonants); the same is true if we use one M and no Ts.
If we use both Ts and no Ms, there are $\tbinom{3}{2} = 3$ choices for the two remaining consonants; the same is true if we use both Ms and no Ts, or if we use one T and one M.
If we use both Ts and one M, there are $\tbinom{3}{1} = 3$ choices for the single remaining consonant; the same is true if we use both Ms and one T.
Finally, if we use both Ts and both Ms, there are no more letters left to choose, so we get $1$ more choice.
In total, we have $2(1) + 5(3) + 1 = 18$ distinct collections of consonants.
Therefore, the number of distinct collections of letters is $4 \cdot 18 = \boxed{72}.$


Justification of Amazon EC2 Performance

Question: I have a .jar file that represents a server which receives over TCP an image in bytes (of size at most 500 kb) and writes it file. It then sobels this image and sends it over TCP socket to the client side.
I ran it on my laptop and it was very fast. But when I put it on Amazon EC2 server m1.large instance, i found out it is very slow - around 10 times slower. It might be the inefficiency in the code algorithm but in fact my code is nothing but receive image (like any byte file) run the sobel algorithm and send.
I have the following questions:
1- Is it normal performance of Amazon EC2 server- I have read the following links link1 and link2
2- Even if the code is not that efficient, the server is finally handling a very low load (just one client), does the "inefficient" code justify such performance?
3- My laptop is dual core only...Why would the amazon ec2 server have worse performance that my laptop? How is this explained?
Excuse me for my ignorance.
Comment: How good is your laptop? An m1.large has 4 EC2 Compute Units. Per AWS, "One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor." Many modern laptops would exceed that performance level.
Comment: You won't know until you analyze the performance and figure out what's taking so long.
Comment: You can start by logging. Log how long it takes to read the data, how long it takes to process the data, and how long it takes to write the data. That will at least narrow it down to one out of three places.
Comment: Personally, in my experience the Amazon servers perform on par with physical servers.  You must be having some other issue.
Comment: I am using a windows instance. Can you help me pls how i should analyze the performance?
Answer: Like djangofan said, I have found AWS servers to perform on par with physical servers. You can try profiling the server jar using VisualVM (this should be fairly straightforward if you remote desktop into the Windows instance).

Justification of Amazon EC2 Performance


Why do we consider measurable function when dealing with abstract integration?

Question: Let $[0,\infty]$ be equipped with the order topology. (That is, it is a subspace of the standard topology on the extended real)
Let $(X,\mathfrak{M},\mu)$ be a measure space.
Let $f:X\rightarrow [0,\infty]$ be a measurable function, which means that the inverse of each borel set of $[0,\infty]$ under $f$ is an element of $\mathfrak{M}$.
In measure theory, we define it's integration as the supremum of $\sum_{x\in s(X)} x\mu(s^{-1}(x))$ where $s$ is a simple measurable fuction such that $s≦f$.
The point is, the role of $\mu$ only acts on those simple functions $s$, not $f$ directly.
What would go wrong if one defines integration for an arbitrary function $f$?
That is, what's wrong with defining an integration of an arbitrary function $f$ as the supremum of $\sum_{x\in s(X)} x\mu(s^{-1}(x))$ over simple meadurable functions $s≦f$?
Answer: Given a Vitali $E$ set in $[0, 1]$, there are no measurable subsets of $E$ with positive measure (see, e.g. here). Hence if we want to integrate $f = \chi_E$, then we would necessarily have $\int f dm = 0$. But one of the "nice" properties of Lebesgue integration is that any nonnegative function with zero integral must be zero almost everywhere - but this implies $E$ has measure $0$, contradicting that it isn't measurable.
So all of a sudden, a lot of results that rely on the idea of equality almost everywhere, or a property holding up to sets of measure $0$ would break down.

Furthermore, the idea that integration and measure are related would break down fundamentally, were this the case. We'd really like to have that
$$\int \chi_A d\mu = \mu(A)$$
for measures; in fact, it's a general result that the function $A \mapsto \int \chi_A f d\mu$ is a measure for measurable functions $f$ (or taking $f$ identically $1$, giving back the measure $\mu$). But if every function is integrable, every set is measurable; but having regularity of the measure requires that some sets aren't measurable.
Comment: This 100% resolves my question. Thank you very much:)
Comment: You're very welcome. It's a good question to ask.
Comment: "... but having regularity of the measure requires that some sets aren't measurable" _or_ [AC](http://en.wikipedia.org/wiki/Axiom_of_choice) fails. $\hspace{.77 in}$


Quais os limites SQL Server Express e MySQL?

Question: Quais os limites do SQL Server Express e MySQL?
Quantas bases são possíveis? Qual o tamanho de cada base? O tamanho dos arquivos índices?
Comment: Qual a versão do SQL Express?
Comment: Dê uma olhada no [tour]. Você pode aceitar uma resposta se ela resolveu seu problema. Você pode votar em todos os posts do site também. Alguma lhe ajudou mais? Precisa que algo seja melhorado?
Answer: Ambos não possuem limite de bases de dados.
O SQL Server Express pode ter bases de até 10GB cada. A versão completa não possui este limite. O MySQL só tem limite prático do sistema operacional que poderá ser 2TB em alguns casos, mas muito mais em outros. Isso pode ser o limite do tamanho de cada tabela dependendo da forma como está utilizando. Tabelas podem ser particionadas.
O SQL Server (incluindo o Express) pode criar até 999 índices secundários tradicionais por tabela e o MySQL pode criar 64 índices secundários por tabela, mas nunca crie tantos assim. O tamanho do arquivo de índice depende do tamanho da base.
Lembre-se que o SQL Server Express possuem limites por imposição comercial. A principal delas é o tamanho da memória que pode ser usada para buffers e outras operações que é de apenas 1GB.
A maioria dos limites de quase todos os principais softwares de banco de dados são suficientes para quase todas as aplicações possíveis. Hoje os limites são mais de hardware ou de praticidade.
Coloquei no GitHub para referência futura.
Comment: Não sei se isso ainda existe, mas se não me engano o SQL Server Express também possuía alguma restrição em relação ao número de processadores e/ou cores utilizados.
Comment: Existe sim, só pode 1 processador. Eu não coloquei porque não foi perguntado especificamente sobre isto. EU considero uma limitação inferior. Pela minha experiência, p/ os volumes que ele é usado, isso não costuma atrapalhar. A limitação de 1GB da memória complica muito.


Bases for exterior powers

Question: I've seen the following claim several times:

If $V$ is a vector space over $K$ with basis $\{e_1,\ldots,e_n\}$ then the basis to the kth exterior power of $V$ is given by the elements $$\{e_{i_1}\wedge e_{i_2}\wedge\cdots\wedge e_{i_k} \mid 1 \le  i_1 < i_2 < \cdots < i_k \le n\}$$

Now, given some basic facts on exterior powers, it's not hard to show that the above set actually spans the $k$th exterior power. On the other hand, I've never seen a complete proof of the linear independence of this set. So my question is: What is the simplest way to show that $\{e_{i_1}\wedge e_{i_2}\wedge\cdots \wedge e_{i_k} \mid 1 \le  i_1 < i_2 < \cdots < i_k \le n\}$ is linearly independent?
Comment: Careful; you use $k$ for both the underlying field and for an integer.
Answer: We induct on $n - k$.
Suppose there exists a nontrivial linear dependence among the pure tensors you list in $\Lambda^k(V)$. Then not all of the tensors share the same components $e_i$, so there is some $i$ such that some tensor does not contain $e_i$ (and some other tensor does). Take the exterior product with $e_i$; this gives a linear dependence among a smaller number of pure tensors in $\Lambda^{k+1}(V)$. By repeating this argument we see that it suffices to show that no single pure tensor can be equal to zero. If $k < n$, then it suffices to take the exterior product with the rest of the basis so that we reduce to the case $k = n$. If $k = n$, it suffices to use any of the standard proofs that the determinant exists.
Comment: Or perhaps in one single step, take the exterior product with the pure tensor formed by taking the complement of the indices in any one of the pure tensors.
Answer: If you want an easy access reference, this book, which is available for free online has a proof of the statement you're looking for in section 2.3.2, basically it's their "lemma 3".

Bases for exterior powers


What is the 'WebsiteGetObject' operation in S3?

Question: I'm seeing a large amount of WebsiteGetObject operations appearing relatively sporadically in the cost explorer of one of our apps. I'm assuming it's distinct from GetObject but can find no mention of it in the API docs, Google or on searching this site.

Does anyone have any information on this?
Answer: These would be requests through the static hosting endpoint for the bucket, rather than accessing the bucket directly.


Read line by line and push to queue, asynchronously in node.js

Question: I'm new to the whole async / promise concept and trying to figure out how to optimize a lambda I'm writing in node.js to execute actions in parallel.
I'm reading line by line from a long file (about 100k rows) and I want to push a message to SQS for each row. I need to wait for all of those processes to return before the function finishes executing.
The code below works but it seems to read every line in the file before the inner promise that writes to the queue begins executing, then at the end it pushes all of the message to the queue because of the call to Promise.all
Is there a way to run these actions in parallel? The file read has to be sequential but I'd expect to see calls to the queue mixed in there.
exports.queueUpdates = async (filePath) => {
  return new Promise((resolve, reject) => {
    const rl = readline.createInterface({
      input: fs.createReadStream(filePath),
      crlfDelay: Infinity
    });

    var queuePromises = [];

    rl.on('line', (line) => {
      console.log("read line", line);
      var message = exports.queueMessageForLine(line); // This function returns the message to be sent to SQS as a JSON Object.
      if (message !== null) {
        console.log("Pushing message", message);
        queuePromises.push(
          sqs.sendMessage(message).promise()
          .then((result) => {
            console.log("Enqueued message", message, result);
            return message;
          })
          .catch((err) => {
            console.error("Failed adding message to queue", err);
            return message;
          })
        );
      }
    }).on('close', () => {
      console.log("file read");
      Promise.all(queuePromises).then((results) => {
        resolve(results)
      })
    }).on('error', (err) => {
      console.error(err, "Error in reading the file contents");
      reject();
    });
  });
};

The output I'd expect here if it was doing what I want would be something like:
read line, line 1
read line, line 2
Enqueued message, message 1
read line, line 3
Enqueued message, message 2
Enqueued message, message 3
etc.

All mixed together.
Comment: I would expect rl to read a fairly large chunk of filePath at once and then trigger a number of "line" events at about the same time.
Comment: That's exactly what was happening. I was trying it locally w/ a test file but when i just pushed it up to AWS and ran the whole thing its queueing the way I was hoping. Thanks @James
Answer: After more investigation this is actually behaving the way I wanted. The issue was what James noted in the comments - readline pulls in multiple lines in bulk, then dispatches events. When testing w/ a smaller file locally it just didn't have enough content to go beyond a single read - resulting in a sequential execution. After pushing it up to AWS and reading the full 100k+ line file it clearly executes in the right order.


ASP.NET - Disable website dlls from being created in bin folder

Question: I have created a new ASP.NET (.net framework) website (not core) and when i run the project i get the following files created in the bin folder - website.dll, website.dll.config, website,pdb.
1) Are these always created for websites?
I would prefer not to have them because when i want to make a change to a .cs file the dll needs to update which will reset all sessions.
2) How can this be done?
I have previous asp.net websites and these files never got created so i am a little confused. old net versions don't do it?
FYI Project settings - .NET Framework 4.7.2 and the output type is class library (other options are console application and console application.
Comment: Yes, on WebApp projects (not websites) you get the DLL compiled in the bin folder. This is the application, you can't deploy without these files. Maybe you want to use the Website type of project, but I think that the template isn't present in newer VS versions.
Comment: IIS can only use compiled DLL files in Asp.Net applications. When the website is running, the DLL file in the system folder cannot be deleted. It means it is occupied by IIS Worker Process. If you delete the DLL, you will not be able to deploy.
Answer: You have two types of asp.net webform projects.
A)
A asp.net web site.
These sites are thus not really a visual studio project. When you create a asp.net web site, then you don't use file->open->project, but use file->open web site.
With such above site, each asp.net page will have a corresponding cs, or vb.net page for the code behind. And you can modify ONE page (say the markup, or code), hit save, and you are done. Such pages (and the code) is compiled on the fly by IIS (the web server).
A lot of people do like the above setup, since in many cases, if the web server is on your same network, you can open the live site directly. Make some changes, hit ctrl-s, and you are done. Of course EVERY page used thus results in a .dll being created.
So, above is still a option you can use (and what you likey were using in the past)
However, there is a 2nd choice,
Asp.net web site application.
The key word/term here is "application".
In a asp.net web site application, then you have a standard sln (project file), and you have to re-build and re-compile the WHOLE site before a new update or deploy.
this means that even changing ONE line of code behind forces you to do a FULL re-deploy of the web site. However, while this is somewhat more "pain" for a new deploy, it is still a far more controlled environment.  This choice also means that visual studio (vs) does the compiling BEFORE deploying.
There are significant advantages to this setup. You can include multiple projects and assemblies (and get FAR BETTER compile time resolution). So, the more advanced a developer, the more say using git, and adopting use of class library's and objects?
And the more compile time checking, and the more you adopt class objects and liraies for your devleopment cycle? The more you perfer the applcation approch to software development. As noted, there is that ONE BIG penalty for the enjoyment of far better managment of referances, compiling, and management of a larger project - you lose the "one quick and dirty" deployment option.
So, you have to pre-compile the site/code, and then run it. IIS does not do the compiling (and more important than the management of those other external code libraries). In most cases, with a web site, to add new assemblies to your project, the .dll's will be placed in the bin file. (and I don't like that at all).
I far perfer when I do a clean project, that the WHOLE bin folder and ANY dll's are 100% blowen out of the water, and don't exist anymore. If you use a web site, then you will and MUST have a hodge podge of .dll's strewn all over the place. And EVERY single page with code behind creates a .dll.
So, while your hands are somewhat "more tied" with a asp.net web site application, from a developer point of view, the advantages far outweigh the additional work/effort to deploy.
I have zero, but BEYOND zero idea why you feel or don't like the idea of all of your referenced assemblies at compile time being managed for you, and they all get compiled, and placed in the bin folder for you. Be it a desktop applcation, a console application, or whatever? The hallmark of the compile process was and is to gather up all of your referenced assemblies and .dll's and dump/place/compile/manage/put them into the bin folder for you.
Better yet? When you publish, you can choose options to "merge" all of the .dll's into one dll. This really is much like using a linker for software development.
The other big advantage, is you can say develop with new rosyln compile features. (free form text for sql is really nice). Since vs is doing the compile for you, then you do NOT have to ensure that the web site and IIS requires the advanced compiler options.
So, much of which one you perfer comes down to:
Do you value developer features, better compile time resolving of assemblies, having VS compile the code for you, or do you want to toss out the code to the web site, and have IIS do the compile?
The above also means that your web site will require the source code when using the web site option. This can be someone of a issue for some developers, or even security. Since any modifying of such web pages will automatic trigger IIS to re-compile that page + code for you.
With a asp.net web site application, then no server IIS compile of your code occurs, and better yet, at compile time the .cs (or vb) pages are stripped out, source code is stripped out, and the source code pages (code behind) NEVER is placed on the server. You thus ONLY get the .dll's and the source aspx pages, but NOT the source code pages published on the server.
As of vs version 2022, BOTH template options are supported. So, if you want to use or go back to use web site as opposed to application development? You can make that choice. Just use file->open webs site, and don't use the .sln (project file) anymore.
And you can when creating a new web site choose "asp.net web site", or choose the preferred (by many) the "asp.net web site application".
So, for a lot of sites - especially those a bit older, often developers choose the web site option - the deployment and update is really far less hassle and is done with far greater ease then a application choice. Despite this greater ease of making small updates to the web site, I still far prefer the application choice, since it has far better options in terms of referencing, compiling, and just overall general application management features. Might not matter for your case, and if the current site was a web site (not web site application), then I do suggest you continue and keep that code and site "as is", as opposed to converting to a application.
It seems strange that your panties are all twisted up over having some .dll's appear in the bin folder at compile time, as opposed to a HUGE MOUNTIN of .dll's created for EVERY web page + code behind as what occurs when using a web site.
This compile time and dumping of .dll's into the bin folder? It how all console, desktop, and more .net applications has worked for over 20 years of .net. Can't possibly be a surprise to any .net developer, and in fact most will be confused and miss this approach to software when using a web site - since now IIS is doing the code compile - and not you nor is visual studio.
So above should clear up the difference between the two choices, and why you are seeing a different behavior from past projects. Sounds like that past project was a web site, and not a web site application.
Comment: Thank you so much for your explanation. One question i have is for a website if i change one of the .cs files this will not force everyone's session to reset (maybe the people using that .cs file). With the application option if i make a change, recompile and update, would this mean that everyone's session is reset? I would think so which is a big disadvantage. Keen to hear your thoughts.
Comment: Well, without question the application choice means a app_pool re-start. However, if you change one line of code and update - then ALSO the web site will trigger a app_pool re-start. So, in both cases, session() will be lost. However, if you use sql server-based session management, then session will remain intact even with a App pool re-start. So, in both choices, app_pool will be re-set and thus both lose session unless you using sql server-based session management.


Each is not working as supposed with Handlebar

Question: In an app that use Handlebar 4.0.3, I have this simple template :
{{#each certificates}}

    {{this}}

{{/each}}

And I use compile it like this :
result = template(certificates: ['test'])

I have this result :

    [object Object]

I think I should have "test" instead of "[object Object]".
I tried with this template :
{{#each certificates}}

  {{this}}
  {{name}}

{{/each}}

And this JavaScript :
result = template({ certificates: [{name: 'Name'}]})

And I have tyhis result :

    [object Object]

As you can see, {{name}} gives nothing instead of "Name".
Am I doing something wrong?
Edit 1
I found the problem.
I have this html :

        {{#each site.certificates}}

          {{this.name}}

        {{/each}}

When I do this $('#modal-site').html(), I have this result :
{{#each site.certificates}}

{{/each}}

    {{this.name}}

With the each is outside so handlebars cannot work. Do you have an idea of the solution ?
Comment: Try `{{this.name}}` in HTML (for that second implementation of template).
Comment: Your second implementation should work as is https://jsbin.com/sicijiwuba/edit?html,js,console
Comment: @RoyMiloh, I edited my question
Answer: For the second implementation of your template, you should use {{this.name}} in HTML if you want to access the string data.
Comment: Make a jsfiddle of entire solution. I can't see that `site` object in your script that would have a `certificates` array within.
Answer: Use the following syntax:
{{#each certificates as |cert|}}

  {{cert}}

{{/each}}