How to store datetime when timezone is different from UTC (Laravel)

Question: My app timezone is set to 'America/Montreal'.
I have a two datetime fields 'start' and 'end', each casted to datetime using laravel $casts property:
protected $casts = [
    'start' => 'datetime',
    'end' => 'datetime'
];

When I create an instance of my model with the following data:
MyModel::create(
                [
                    'start' => "2022-02-08T20:45:58.000Z", // UTC time
                    'end' => "2022-02-08T20:45:58.000Z",
                ]
            );

The created model keeps the same time (20:45) but the timezone is set to America/Montreal:
 App\MyModel {#4799
     id: 44,
     created_at: "2022-02-08 15:49:02",
     updated_at: "2022-02-08 15:49:02",
     start: 2022-02-08 20:45:58,
     end: 2022-02-08 20:45:58,
   }

when I access the start and end attributes I get the same time but with America/Montreal timezone like this:
// accessing 'start' attribute of the instance I just created
Illuminate\Support\Carbon @1644371158 {#4708
 date: 2022-02-08 20:45:58.0 America/Montreal (-05:00),

}
The only way I found to get it to work correctly is to manually setTimezone before saving:
    MyModel::create(
                [
                    'start' => Carbon::parse("2022-02-08T20:45:58.000Z")->setTimeZone(config('app.timezone')),,
                    'end' => Carbon::parse("2022-02-08T20:45:58.000Z")->setTimeZone(config('app.timezone')),,
                ]
            );

which is, I think, repetitive, isn't setting the app timezone enough ? Is there any better way to do this ? I know that I'm supposed to set my app timezone to UTC (which is what I do usually) but this projet has already a lot of data with this timezone and I'm not sure how to convert it.
Thank you.
Comment: Thank you for your answer. Yeah I did, but I would need to define one for every datetime attribute in every model, I hope that there is a better way to do this
Comment: That's the problem, it stores it without doing timezone conversion to the app timezone and then retrieves it as if it was the app timezone, which is clearly a bug imo
Comment: The point of setting it as a datetime in `$casts` is so you can pass `DateTime` or `Carbon` objects, not strings. Also why do you say "the timezone is set to America/Montreal" when you clearly show the date as "2022-02-08 20:45:58.0 +00:00"?
Comment: How about using a mutator on model: https://laravel.com/docs/8.x/eloquent-mutators#defining-a-mutator
Comment: I see, have you tried this `protected $dateFormat = 'U';` on your model, from the same docs page under date casting section, though I've not tried this myself, seems like what you need
Comment: If the above doesn't help, I would make use the a dynamic trait, so you don't have to do it manually, and the following SO answer does exactly that: https://stackoverflow.com/a/48371850/4311336
Comment: @miken32 My bad, I've just correct the example.
I'm not sure I understand what do you mean, I think that $casts are used so if you pass a string it's casted to a Datetime (mutator) and when you retrieve it from the database the string is casted to Datetime (accessor)
Comment: @AbdulRehman thank you but I don't think that "protected $dateFormat = 'U';" solves the problem. The SO answer is interesting though.
Comment: If you are passing a string, you may as well just pass a string the database understands, and not ask Laravel to cast it. The point of `$casts` is so you can do things like `$model->start = Carbon::now()` instead of `$model->start = '2022-02-09 12:43:42'`. The database doesn't store time zones, it will assume you're storing in whatever time zone it's configured for, which by default is your server's time zone.
Answer: Laravel will just store the date in the model doing $date->format('Y-m-d H:i:s') which just uses the original hour/time of the date, but without preserving the timezone information.
Then when it retrieves it, because it's just a string with no timezone info, it will cast it to a carbon date with the app timezone (generally UTC)
This creates a disparity because you do not get the same value from the getter than the one you sent to the setter, if your date had a different timezone than the app
In simpler words, this is basically what happens
Carbon\Carbon::parse(Carbon\Carbon::parse('2022-11-08 00:00', 'America/Montreal')->format('Y-m-d H:i:s'), 'UTC');

Carbon\Carbon @1667865600 {#4115
   date: 2022-11-08 00:00:00.0 UTC (+00:00), // As you can see it is UTC,
  // which is ok because the database does not store the timezone information,
  // but the time is 2022-11-08 00:00 and should be 2022-11-08 05:00:00 in UTC
}

// This would yield the correct result
Carbon\Carbon::parse(Carbon\Carbon::parse('2022-11-08 00:00', 'America/Montreal')->setTimezone('UTC')->format('Y-m-d H:i:s'), 'UTC');

This is a very controversial problem in laravel which doesn't have a sane and expected handling of dates in model, which should have been to convert to the app timezone before casting the date to a string without timezone info, it was marked as "intended behavior"
To palliate this, you can make your own model extension overriding the setAttribute method and extend from this class instead to convert all the dates automatically to your app timezone
<?php

namespace App;

use DateTimeInterface;
use Carbon\CarbonInterface;
use Illuminate\Database\Eloquent\Model as BaseModel;

class Model extends BaseModel
{

    /**
     * Set a given attribute on the model.
     *
     * @param  string  $key
     * @param  mixed  $value
     * @return mixed
     */
    public function setAttribute($key, $value)
    {
        if ($value instanceof CarbonInterface) {
            // Convert all carbon dates to app timezone
            $value = $value->clone()->setTimezone(config('app.timezone'));
        } else if ($value instanceof DateTimeInterface) {
            // Convert all other dates to timestamps
            $value = $value->getTimestamp();
        }
        // They will be reconverted to a Carbon instance but with the correct timezone
        return parent::setAttribute($key, $value);
    }
}

Also do not forget to set your database timezone to your app timezone, otherwise if you store dates in timestamp instead of datetime you may get errors when trying to insert a date because the date may fall in a daylight saving time
In your config/database.php
    'connections' => [
        'mysql' => [
            //...
            'timezone'  => '+00:00', // Should match app.timezone
            //...

If you didn't do that before, you will need to migrate all your dates, here is a migration that does just that
<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

class ConvertAllTimestampsToUtc extends Migration {
    public static function getAllTimestampColumns(): array
    {
        $results = DB::select(
            "SELECT TABLE_NAME, COLUMN_NAME from information_schema.columns WHERE DATA_TYPE LIKE 'timestamp' AND TABLE_SCHEMA LIKE :db_name", [
            'db_name' => DB::getDatabaseName()
        ]);

        return collect($results)->mapToGroups(fn($r) => [$r->TABLE_NAME => $r->COLUMN_NAME])->toArray();
    }

    public static function convertTzOfTableColumns($table, $columns, $from = '+00:00', $to = 'SYSTEM')
    {
        $q = array_map(fn($col) => "`$col` = CONVERT_TZ(`$col`, '$from', '$to')", $columns);
        DB::update("UPDATE `$table` SET " . join(', ', $q));
    }

    /**
     * Run the migrations.
     */
    public function up(): void
    {
        foreach (self::getAllTimestampColumns() as $table => $cols) {
            self::convertTzOfTableColumns($table, $cols);
        }
    }

    /**
     * Reverse the migrations.
     */
    public function down(): void
    {
        foreach (self::getAllTimestampColumns() as $table => $cols) {
            self::convertTzOfTableColumns($table, $cols, 'SYSTEM', '+00:00');
        }
    }
};


Content type within content type?

Question: I have such a situation:
I have 4 content types:
city (name)
cinema (name, city_ref, hall_count)
movie (name, description)
session (movie_ref, cinema_ref, hall_number, date_time)
city_ref, movie_ref, cinema_ref are connecting content types with Node Reference module.
After that with Views and EVA I can aggregate some movie listings, etc.
What I want to do is to make site content manager register multiple sessions while entering a movie description, i.e.:
there are number of cities and cinemas created and fixed. But the number of movies may change and content creator opens the form and enters name of the movie, description and then has something like multiple sessions: in the same form content admin can enter session, which automatically has reference to this movie, add reference to the cinema, hall number, date AND (if needed) he could ADD additional session with the same duplicated info, but maybe different hall number/cinema and date.
How could it be solved? Is there any way to connect those content types to make it a bit easier to enter all the data and still have all the relations working?
Answer: This sounds like the kind of thing to Node References and the Node Relationships module in D6 or the References module for D7.  Both of those suggest other related modules that might helpful as well.
Comment: Could you give a bit more info about that? I see references module, but how in more exact way it could be used in this case?

Content type within content type?


Modules and performance

Question: I know that disabled modules do not effect performance. I also know that typically the number of modules you have enabled will effect the speed of you site. One individual on Drupal.org said you should never have more than 40 modules enabled. I found this strange because I would assume some modules are more intensive than others.
Is it simply the number of modules that somehow effect performance, (because the hook is always called), or is it really dependent on what you are doing in the module?
Comment: See http://drupal.stackexchange.com/questions/1949/does-having-more-enabled-modules-make-the-site-slower
Answer: It depends on the module.
You can create a module which has a simple info file and a blank module file, compare that to views which has loads of file and implement lots of hooks and create it's own as well. Obviously this requires more memory than a blank module.
Performance depends on a lot of factors. A module can use a lot of memory but not slow the side down much while the reversed can be true as well. Usually the bottleneck is r/w to the database. A single bad query can take seconds to perform, most tasks in code is not that slow. Iterating over a lot of objects and munching data can, however, take a lot of time.
So it's hard to say how a module will effect performance, but it's always a good idea to limit the amount of modules used when possible. I haven't heard of a thumb rule though and to be able to make one, you would need to know the hosting environment.
Comment: Great answer, that makes sense, there is also interesting info in the link that mikeytown2 posted. The includes withstanding, it seems to me that if you want certain functionality on your site, and that requires installing certain modules, it's worth throwing some additional hardware at it.

Modules and performance


A problem on order of intersection of two subgroups

Question: Consider the multiplicative group $S=\{z:|z|=1\}\subset \mathbb C$. Let G and H be subgroups of order  8 and 10 respectively. If n is the order of $G \cap H$ , then what is n ?

n=1
n=2
$3\leq n\leq 5$
$n\geq 6$

I am completely stuck on this problem ,please someone help me.Thanks.
Comment: Lagrange's theorem is definitely something you want to use here.
Comment: According to Lagrange theorem  ,it is 2 .am i right?
Comment: 2 or 1, it remains to show that $n\neq 1$...
Comment: If n=1,then order of G and H are relatively prime ,which is impossible. is it?
Comment: This is not the reason (two subgroups of a group might have the same cardinal but have a trivial intersection), using Lagrange's theorem the only conclusion you get is that $n$ divides both $8$ and $10$ whence divides $2=pgcd(8,10)$. The thing is that in $S$ you have exactly one subgroup of order $n$ which is the subgroup generated by some primitive $n$-root of the unity. You should use this to identify $G$ and $H$ and then find an explicit element which is in the intersection of both leading, indeed, to $n=2$...
Answer: If $S$ is a finite group with $n$ members and identity $1$ then $\forall x\in S (x^n=1)$ (Lagrange). So $x\in G\cap H\implies x^8=1=x^{10}\implies x^2=1\implies x\in \{-1,1\}.$ These implications are one-way. For the other direction we have $x\in G\iff x^8=1$ and $x\in H\iff x^{10}=1.$ So $G\cap H=\{-1,1\}.$
Comment: You mean $\{-1,1\}$ at the very end.


Como poderia reduzir este código?

Question: def hello_quantum(string):
    """
    hello_quantum: cad. caracteres --> lógico
    funcao permite jogar o jogo praticamente, permite o uso das portas para manipular o tabuleiro
    com o objetivo de igualar o tabuleiro inicial ao tabuleiro introduzido

    """
    a = eval(string.replace(":",",")) # transforma o string num tuplo, onde primeiro elemento eh o tabuleiro,
                                                                    #  e segundo eh o numero jogadas maximas
    num_jogadas = a[1]
    t_objetivo = str_para_tabuleiro(str(a[0]))
    counter = 0
    t_inicial = tabuleiro_inicial()
    tabu = []

    print("Bem-vindo ao Hello Quantum!\nO seu objetivo e chegar ao tabuleiro:")
    print(tabuleiro_para_str(t_objetivo))
    print("Comecando com o tabuleiro que se segue:")
    print(tabuleiro_para_str(t_inicial))

    while num_jogadas > counter and not (t_objetivo == tabu):
        in_porta = input("Escolha uma porta para aplicar (X, Z ou H): ")

        in_lado = input("Escolha um qubit para analisar (E ou D): ")

        if in_porta == "X":
            if in_lado == "E":
                tabu = porta_x(t_inicial,"E")
            else:
                tabu = porta_x(t_inicial,"D")
            print(tabuleiro_para_str(tabu))

        elif in_porta == "Z":
            if in_lado == "E":
                tabu = porta_z(t_inicial,"E")
            else:
                tabu = porta_z(t_inicial,"D")
            print(tabuleiro_para_str((tabu)))

        elif in_porta == "H":
            if in_lado == "E":
                tabu = porta_h(t_inicial,"E")
            else:
                tabu = porta_h(t_inicial,"D")
            print(tabuleiro_para_str(tabu))

        counter += 1

    if tabuleiros_iguais(tabu,t_objetivo):
        print("Parabens, conseguiu converter o tabuleiro em %s jogadas!"%(counter))
        return True

    else:
        return False

Função feita com abstração de dados, por isso chama várias outras funções.
Answer: @user133264, sua pergunta é vaga, quando pergunta como reduzir o código. Pois reduzir pode significar fazer em menos linhas, reduzir tempo de processamento, reduzir número de loops (que acaba se relacionando com os últimos dois pontos), etc.
Mas estou assumindo reduzir como fazer um código mais enxuto.
Primeiro ponto é que você não detalha as outras funções sendo chamadas, mas algo que já reduziria a quantidade de código absurdamente, seria:
1. adicionar um dicionário para mapear a relação entre in_portas e funções
2. checar se in_lado e in_portas receberam um valor válido
3. trocar todos os if, elif e else por uma única linha de código:
portas = {
            'X': porta_x,
            'H': porta_h,
            'Z': porta_z
        }
lados = ["E", "D"]

if in_porta not in portas or in_lado not in lados:
     continue

tabu = portas[in_porta](t_inicial,in_lado)

Acredito que aqui você já tem um grande ganho da ineficiência de loops que havia criado. Quando ao restante do código, precisaria de mais detalhes das funções para tornar ainda mais eficiente, mas me parece que você estava buscando algo para otimizar os loops mesmo (caso não, me avise)

Como poderia reduzir este código?


Difference between predictive and prognostic

Question: I hear the terms predictive and prognostic factors getting thrown around a lot (usually in the context of biostatistics) and I am very unsure of what the difference between them is.  Could someone please give a definition of each, an example of the two, and ultimately how they are different from each other?
Comment: A brief comment, at the moment I don't have the time to make a quality answer (traveling). a prognostic factor is a factor associated with prognosis, in medical studies usually overall survival or progression free survival. A predictive factor is a factor which can help predict an outcome, this outcome could be related to prognosis (and so is also prognostic) or other endpoints, such as objective response in radiologic evaluation or development of a side effect.
Answer: This is an excellent question because the two are being terribly misused in personalized medicine and biomarker studies.  The correct definition of the two, at least when it comes to data, is the same.  Here is how the terms are being misused in personalized/precision medicine: prognostic is taken to mean predictive and predictive is taken to mean interaction, i.e., the ability to predict differences in treatment effectiveness over values of patient covariates.  The correct terminology should be: predictive/prognostic for ability to forecast outcomes, and modeling interactions/differential treatment effect for the other.


Mac OS X: java.lang.ClassNotFoundException: com.sun.java.browser.plugin2.DOM

Question: I am trying to use the new LiveConnect features introduced in Java 6 Update 10.
Code looks like this (copied from the applet tutorial):
<code> Class<?> c = Class.forName("com.sun.java.browser.plugin2.DOM");
 Method m = c.getMethod("getDocument", java.applet.Applet.class);
 Document document = (Document) m.invoke(null, this);
</code>
But all I am getting is a ClassNotFoundException for the entry-point class.
This on the Mac, 10.6, with both Firefox and Safari.
<code>Java Plug-in 1.6.0_22
Using JRE version 1.6.0_22-b04-307-10M3261 Java HotSpot(TM) 64-Bit Server VM
</code>
Is this not implemented on the Mac? Or do I need to configure something?
All I need to do is get and set the value of form elements on the page, so
I would be fine with an older (pre-6u10) API if that works better.
Comment: `deployJava.isPlugin2()` also returns `false` :-(
Comment: I hope `plugin.jar` is in your classpath .
Comment: There is one in `/Library/Java/Home/lib/` and it does have `com/sun/java/browser/plugin2/DOM.class`
Answer: Found the answer in this thread:
Java 6 on the Mac does include Plugin2, but it is not the default. You can enable it in the Java Preferences by checking "Run applets in their own process".
I'll use the older API for better compatibility.
Answer: According to the release notes http://www.oracle.com/technetwork/java/javase/index-135519.html mac is not supported. Have a look at the "Supported Platforms and Browsers" on that link... I can only see Windows, Linux and Solaris. Not too nice of them!
Comment: I saw that, but I think (and hope) that this is only because Oracle does not make the JDK and browser plugin for the Mac, but Apple does. So support would be from Apple instead. Of course, they have recently declared that they no longer want to support it, either...
Answer: Look at the classes in:

/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home/lib/plugin.jar

which on my machine is an alias for

/Library/Java/JavaVirtualMachines/1.6.0_22-b04-307.jdk/Contents/Home/lib/plugin.jar

The com.sun.java.browser.plugin2.DOM class does exist there.
It might be different on your machine since I installed the extra developer JDK packages while most machines will only have the JRE. The path might also be different on your computer if you haven't installed all the latest Apple Java updates. Apple made a change to the filesystem location of the JVM in their last Java update.


How to investigate arrayFlip errors in Commerce

Question: I get periodic errors telling me that something expects a parameter to be an array. But the messages are too vague, and don't tell me which piece of code is feeding that information to Drupal. Here's an example:
<code>Warning: array_keys() expects parameter 1 to be array, null given in inline_conditions_field_widget_form_validate() (line 353 of /Users/newman/Sites/visionart/sites/all/modules/inline_conditions/inline_conditions.field.inc).
Warning: array_filter() expects parameter 1 to be array, null given in inline_conditions_field_widget_form_validate() (line 353 of /Users/newman/Sites/visionart/sites/all/modules/inline_conditions/inline_conditions.field.inc).
Warning: array_flip() expects parameter 1 to be array, null given in inline_conditions_field_widget_form_validate() (line 353 of /Users/newman/Sites/visionart/sites/all/modules/inline_conditions/inline_conditions.field.inc).
Warning: array_intersect_key(): Argument #1 is not an array in inline_conditions_field_widget_form_validate() (line 353 of /Users/newman/Sites/visionart/sites/all/modules/inline_conditions/inline_conditions.field.inc).
</code>
Most of these are in connection with array_flip(). I can see that this one in particular is being flagged but inline_conditions, bit it's such a core module that it's impossible to find out what's triggering it. How can I do that and how can I fix?
Answer: BTW, I think you're mistaken, inline conditions isn't a core module. In fact looks like the latest version is 7.x-1.0-alpha7. Also, notice that these are warnings, not errors. It's likely that they won't effect a site set up with production error handling.
The way I've dealt with that is through debugging on copy of the site running in a local environment with xdebug. If you haven't set that up yet there are a bunch of ways you can do it, here's a few:

Here's a link for how to do it using Homebrew. That's my reply but read the others there, they've got good info.
you can sign up for a free account on Acquia and use their dev desktop software. Here's some info on doing that. (Seems like I'm plugging my own posts, but thats what I know.) I've gotten
There's a similar app from Kalamuna called Kalabox that specifically is set up to work with Pantheon, but I was able to get to work standalone. Let me know if you need help with this, I'll try to get time to put more details up.

Be ready to be frustrated getting a local environment with debugging up and running, it takes some doing, but it's well worth the effort.
Once you've got debugging happening, you can put a breakpoint on line line 353 of
/sites/all/modules/inline_conditions/inline_conditions.field.inc. If you can reproduce the error, then just hit continue until just before you expect the error to occur. (If you can't, you'll have to do this each time it breakpoint is reached.) From the breakpoint, you'll be able to look at the parameters that are referenced on that line and see what's expected to be an array that isn't. My setup using Netbeans allows me to hop to earlier parts of the code by checking the stack. This will allow you to see just where the parameter was originally set.
Sorry this isn't just an easy answer, but getting yourself set up will be worth it in the long run.
Comment: Just checking in on this old post. FWIW, I've switched from Netbeans to PHPstorm and it's a lot more functional. **CONS:** Takes a while to get up to speed; it costs a yearly subscription fee. **PROS:** it gives daily hints which makes it easier to learn; there's good online help built in; every hint you learn makes your programming _much_ easier and faster (**NOTE**: I get no kickback from PHPStorm.)
Answer: This may be related to issues with Commerce discounts (possibly related to AJAX?), if so, make sure you'e using the latest version of Commerce Discount.
Also clear all your caches and re-try. If problem persist, report a bug against the module it-self.


How do I get a user profile's picture from entity_metadata_wrapper()?

Question: I am using entity_metadata_wrapper() for getting values from the user profile.
This is my code:
$user_wrapper = entity_metadata_wrapper('user', $uid);
$user_name = $user_wrapper->name->value();

How do I get the user picture field from $user_wrapper?
Answer: You can get the information about user picture indirectly as follows:
$user_picture = $user_wrapper->value()->picture;

I said indirectly, since the picture object itself is not an entity wrapper anymore, so you cannot chain it further using entity_metadata_wrapper functions. However, it should have all the information you need about the user picture, like the fid, filename, uri, filemime, filesize, metadata, and any other fields that could be attached to a file, for instance using File Entity module.
Comment: If you want it wrapped, you need to apply this patch I proposed a while back: https://www.drupal.org/node/2224645


is Shapiro Wilk test insensitive on the tails?

Question: Is Shapiro Wilk test insensitive on the tails of sample distribution? I've read such statement on a paper but I cannot figure out why from its test statistic W. Could someone help me understand this?
Comment: See my answer [here](http://stats.stackexchange.com/questions/90697/is-shapiro-wilk-the-best-normality-test-why-might-it-be-better-than-other-tests/129826#129826) to a similar question. It also has the references to papers with empirical research, comparisons to different alternatives including long-tailed distributions.
Comment: Which paper did you read this in?
Answer: The situation is complicated, but the results tend to the opposite of this claim: for moderate dataset sizes $n$, the Shapiro-Wilk test is more sensitive in the tails than elsewhere.

Quantifying sensitivity
I take "sensitive" to mean the extent to which the results vary when values in the dataset are perturbed.  (Another possible interpretation is that "sensitivity" is meant in terms of the power of the test to detect deviations from the tail behavior of a Normal distribution.  However, since "sensitivity" and "power" are common, well-understood statistical terms with distinct meanings, this second interpretation does not seem appropriate.)
Generically, consider the test "results" (which usually would be taken as a p-value) to be some function $f$ of the ordered data $x$.  Then we might want to define the sensitivity of $f$ to the $i^\text{th}$ element of $x$ to be
$$\frac{d}{dx_i} f(x_1,x_2,\ldots, x_n).$$
There are a few problems with this, however.  First, $f$ might not be differentiable.  Second, sensitivity to extremely small changes might be less relevant than sensitivity to larger changes.  To cope with these complications we may (1) use directed finite differences to explore changes in $f$ when $x_i$ is separately increased and decreased and (2) obtain these differences for deviations which are appreciable compared to the spread of the data.  To this end, given a deviation $\delta\ge 0$ let
$$s_\delta^{\pm i} f = \frac{f(x_1,\ldots,x_{i-1},x_i\pm\delta\sigma, x_{i+1},\ldots, x_n) - f(x_1,x_2,\ldots, x_n)}{\delta\sigma}$$
(where $\sigma$ is a standard measure of the spread of $x$, such as its standard deviation) and define the sensitivity of $f$ to be the vector of absolute difference quotients
$$(|s_{\delta/2}^i| + |s_{\delta/2}^{-i}|, i=1, 2, \ldots, n).$$
That is, each data value is displaced upwards and downwards by amounts $\delta/2$ times the overall spread.  The sensitivity is the total absolute relative change, reflecting a net deviation of $\delta\sigma$ centered at the data.
Assessing sensitivity of distributional tests
Sensitivity can vary with the dataset.  Should we be assessing it when the data conform to the null hypothesis or when they are far from the null?  Both assessments can be informative.  But for distributional tests we face the complication that the alternative often is not even parameterizable: although the null hypothesis might be that the data are sampled from a Normal distribution, the alternative would be that they are sampled from any distribution.
A thorough study would look at many alternatives and many sample sizes.  Below I report on results for three sample sizes, $n=4, 12, 36$, which are typical of datasets where the Shapiro-Wilk test is used, and for the null (a Normal distribution), a short-tailed alternative (a Uniform distribution), a long-tailed alternative (an Exponential distribution), and a bimodal alternative (a Beta$(2,2)$ distribution).  In each case I make the dataset look as much like its parent distribution as possible.  This is accomplished by computing the quantiles of the distribution at $n$ probability plotting points (spaced according to Filliben's formulas, aka "Weibull plotting points").
For reference I have applied the same analysis to a variant of the Kolmogorov-Smirnov test.  For this variant I first recenter the data, because (at least for the alternatives) the K-S test will not be a realistic comparison.  With the recentered data, both tests often produce comparable p-values and those p-values range from $1$ down to $0.0003$, covering a useful range of possibilities.
Results
The sensitivities for $\delta=1$ are plotted on logarithmic axes against the data indexes (ranks).  The results for the S-W test are shown in red with filled circles; those for the K-S test are in blue with filled triangles.  (Sensitivities of zero are plotted at $10^{-12}$.)

The S-W test tends to be slightly more sensitive to data in the tails (i.e., where the ranks are close to $1$ or to $n$) than in the middle, except for very small datasets.  The K-S test, by contrast, tends to be extremely sensitive to a small number of data in one or both tails, at least once the dataset size is sufficiently large.  Clearly these tests are telling us different things about the shapes of the datasets.
By and large, the S-W test has substantially larger sensitivities than the K-S test.  The reasons for this are complicated, but note especially that two distributional tests cannot be compared based on sensitivity alone: you should also consider the p-values at which these sensitivities are measured.
Code
The R code used to produce these results follows.  It is structured to be easily modified to extend the study in any desired direction: different sample sizes, different dataset distributions, and different distributional tests.
filliben <- function(n) {
  a <- 2^(-1/n); c(1-a, (2:(n-1) - 0.3175)/(n + 0.365), a)
}
sensitivity <- function(x, f, delta=1, ...) {
  s <- delta * sd(x) / 2
  e <- function(i) {u <- rep(0, length(x)); u[i] <- s; u}
  f.x <- f(x)
  sapply(1:length(x), function(i) f(x + e(i)) - f.x) / abs(s)
}
sensitivity.abs <- function(x, f, delta, ...) {
  abs(sensitivity(x, f, delta/2, ...)) + abs(sensitivity(x, f, -delta/2, ...))
}

delta <- 1
beta <- function(q) qbeta(q, 1/2, 1/2) # A bimodal distribution
par(mfrow=c(3, 4))
for (n in c(4, 12, 36)) {
  x <- filliben(n)
  for (f.s in c("qnorm", "qunif", "qexp", "beta")) {
    # Perform the tests.
    y <- do.call(f.s, list(x))
    y <- (y - mean(y))
    cat(n, f.s, shapiro.test(y)$p.value, ks.test(y, "pnorm")$p.value, "\n")

    # Compute sensitivities.
    shapiro.s <- sensitivity.abs(y, function(x) shapiro.test(x)$p.value, delta)
    ks.s <- sensitivity.abs(y, function(x) ks.test(x, "pnorm")$p.value, delta)
    shapiro.s <- pmax(1e-12, shapiro.s) # Eliminate zeros for log plotting
    ks.s <- pmax(1e-12, ks.s)           # Eliminate zeros for log plotting

    # Plot results.
    plot(c(1,n), range(c(shapiro.s, ks.s)), type="n", log="y",
         main=f.s, xlab="Rank", ylab=paste0("Sensitivity, n=", n))
    points(shapiro.s, pch=16, col="Red")
    points(ks.s, pch=24, bg="Blue")
    lines(shapiro.s, col="#801010")
    lines(ks.s, col="#101080", lty=3)
  }
}

Answer: I don't think it's particularly insensitive; I'd say it's more sensitive there than the Lilliefors test, for example, and I'm having trouble thinking of another comparable goodness of fit test in wide use* that I think will be much more sensitive to the tails.
If we look at power comparisons of goodness of fit tests (on which there are numeorus papers), the Shapiro Wilk generally performs very well in a wide variety of situations, including some that I'd regard as relating to "sensitivity to tails".
*  well, apart perhaps from the Anderson-Darling$^\dagger$ which might beat it on tail sensitivity in some cases.
Edit: I've been through a number of power comparison studies, including against symmetric heavy tailed alternatives and alternatives with small amounts of contamination by outliers (the two most obvious ways to look at 'sensitivity to tails') and the Shapiro-Wilk does extremely well, generally outperforming even the Anderson-Darling on this task (a task at which the A-D should be expected to excel).
$\dagger$ suitably adapted to the parameter estimation inherent in testing normality without specifying the parameters, of course - see the discussion in the book Goodness of fit techniques by D'Agostino and Stephens
[Do the authors say how this sensitivity was measured or what it was compared to? Does the paper give any justification or context for the claim at all?]


Identify Webserver & Script of a website

Question: I have got two simple questions

How can I tell what server is a website on? I remember I used to read the HTTP Host Header to identify the type of server. Is there any tool to do it?

2a. A lot of the website have the page extension .html and you just know they are not html. How can I tell what programming language is behind them?
2b. For ASPX, I think IIS can map the extension, so it will show HTML instead of ASPX, right?
Cheers
Answer: 1.
Yes, you can check the http header tag "SERVER". Example of responses:
-Microsoft-IIS/6.0
-GFE/1.3
-Server Apache/2.2.11 (Ubuntu) PHP/5.2.6-3ubuntu4.2 with Suhosin-Patch
You can also check "X-Powered-By" on some servers, example:
-PHP/5.2.6-3ubuntu4.2
-ASP.NET
You can do this in firefox/firebug for example. Go to NET pick a request, select headers and check under response headers. You could do this is Fiddler to or any other http sniffer.
2a)
See my first answer
2b)
Yes you can map .html or anything as a "asp.net" extension, meaning that the extension will be handled by the web application. Common use is that you have a httphandler that catches that extension in web.config.
Not sure what your endgoal of these questions are.. or rather to what purpose, maybe we could answer better then.
Answer:
Look at the HTTP headers. This works as long as the Server admin hasn't disabled them (which he usually doesn't).

Try http://kalender-365.de/ip/get-http-header.php
2a. This actually works with all servers and all extensions. Some Interpreters - such as e.g. PHP - send a special created-by HTTP header (which can be disabled, however).
Comment: Actually, you only can hope you get an http header. There is no other even remotely reliable way of knowing.
Comment: Thanks very much about Q1.

Question 2a. Do you mean it can only tell by looking at the HTTP Header? I wanna know site like this one www.australianidol.com.au, what's behind it? I can see it's Apache Server, so it's can be anything but ASPX.

Cheers


strange behaviour in output

Question: I have a problem with my program to get it run correctly.
here is the code:
#include <stdio.h>
#include <stdlib.h>  // for EXIT_SUCCESS and EXIT_FAILURE
#include <ctype.h>
#include <string.h>

void ReadFile(FILE* file) {
    unsigned lines = 0;

    int braces      = 0;
    int curlyBraces = 0;
    int comments    = 0;

    int c;
    char* line = 0;
    unsigned col = 0;

    while ((c = fgetc(file)) != EOF) {
        if(c == '\n') { // new line
            lines++;

            printf("%4d: {%d} (%d) /*%d*/ |%s\n", lines, curlyBraces, braces, comments, line);
            free(line); line = 0;
            col = 0;
        } else {
            // add character to line
            line = (char*)realloc(line, (col+1)*sizeof(char));
            if (line == 0) {
                fprintf(stderr, "error reallocating memory");
                return;
            }
            line[col] = c;
            col++;

            if (c == '(') {
                braces++;
            } else if (c == ')') {
                braces--;
            } else if (c == '{') {
                curlyBraces++;
            } else if (c == '}') {
                curlyBraces--;
            } else if (c == '/') {
                if (fgetc(file) == '*') {
                    comments++;
                } else {
                    fseek(file, -1, SEEK_CUR);
                }
            } else if (c == '*') {
                if (fgetc(file) == '/') {
                    comments--;
                } else {
                    fseek(file, -1, SEEK_CUR);
                }
            }
        }
    }
}

int main(int argc, char** argv) {
    short lines = 0, words = 0, chars = 0;

    /* check for arguments */
    if (argc == 1) {
        fprintf(stderr, "usage: %s filename\n", argv[0]);
        return EXIT_FAILURE;
    }

    /* open file */
    FILE* file = fopen(argv[1], "r");
    if(file == 0) {
        fprintf(stderr, "error open file '%s'\n", argv[1]);
        return EXIT_FAILURE;
    }

    ReadFile(file);

    if (fclose(file) == EOF) {
        fprintf(stderr, "error in fclose()\n");
        return EXIT_FAILURE;
    }

    return EXIT_SUCCESS;
}

on the output i get some weird output, like realloc overwrites some data...
i already tried strcat, but i can't use constant chars. so i have to use realloc.
here is a short excerpt of the output
P 68: {1} (0) /*0*/ |   / open file *
  69: {1} (0) /*0*/ |   FILE* file = fopen(argv[1], "r");
  70: {2} (0) /*0*/ |   if(file == 0) {
rn EXIT_FA(0) /*0*/ |       fprintf(stderr, "error open file '%s'\n", argv[1]);
r open fi (0) /*0*/ |       return EXIT_FAILURE;
  73: {1} (0) /*0*/ |   }
} 74: {1} (0) /*0*/ |

maybe there is another way to reaize this? with fgets() i could get the whole line, but increments the pointer in the file and i have to give fgets() a count of chars. so that wouldn't be the perfect solution.
Answer: You need to terminate your string before you print it out:
    lines++;
    line[col] = 0; // new
    printf("%4d: {%d} (%d) /*%d*/ |%s\n", lines, curlyBraces, braces, comments, line);

Unfortunately line[col] is out of bounds here. So you need realloc line before adding the terminator like this:
        lines++;
        line = (char*)realloc(line, (col+1)*sizeof(char));
        if (line == 0) {
            fprintf(stderr, "error reallocating memory");
            return;
        }
        line[col] = 0; // new
        printf("%4d: {%d} (%d) /*%d*/ |%s\n", lines, curlyBraces, braces, comments, line);

Also, do you know about ungetc? You can replace those fseek(-1) with ungetc.
Comment: thanks, i forgot about the termination.
but does i really need to reallocate memory for the terminating char?
if i initiate the pointer with (char*)malloc(sizeof(char)) would also work?
Comment: The terminating char goes on the end of the line so you need to treat it just like any other character.
Comment: If you changed 'realloc(line, (col+1)*sizeof(char));' to 'realloc(line, (col+2)*sizeof(char));' you would haven't to realloc for the final terminator.

strange behaviour in output


Mysterious Jboss shutdown

Question: We have a several application deployed in Jboss server. From last few month its observed that jboss servers gets killed without logging any error or exception neither in any application log nor in Jboss server log. The crash is not periodic and happens randomly. The crash is very silent (similar to kill -9 )and its become extremely difficult to debug the root cause.From our analysis one thing sure its not out of memory issue.We analysed the syslog as well, nothing suspicious found. Is possible to force VM to generate error log before the process crash due to any reason.
Any help would be appreciated.
Comment: Your question is "Is possible to force VM to generate error log before the process crash due to any reason?" but the question isn't asked until the end. Nor does the title indicate what your question is. You should edit the title and the question to be more clear.
Comment: We used to have very similar problem in our system - the server would simply "die" if it was idle for a longer period of time (by "idle" I mean no user activity). Our workaround was to create a simple scheduled bean that would print the current time to `System.out` every 10 minutes or so. Not very nice, but we haven't had enough time to trace the root cause... and the server is alive since then, so it is at least somewhat improved.
Answer: The JVM logs fatal errors to hs_err_<pid>.log files
You can change the location of the fatal log file using java -XX:ErrorFile=/var/log/java/java_error%p.log
Answer: I experienced similar problem using JBoss EAP 6 in Linux enviroment, when system memory is almost gone the S.O kill some process, normally the file hs_err_pidxxx.log  with crash report are generated inside bin directory of JBoss. Other case that process is killed is when you are using JBoss with DC mode, when instance stay long time without communication with DC, the process is finalized, I dont know why.

Mysterious Jboss shutdown


How can I test TCP socket status in Perl?

Question: I've got a TCP socket which reads data. When an error occurs when reading the data, I return an undef (NULL) value. Errors can be caused by badly formatted messages or broken sockets. Can someone tell me if there is a specific function which returns the status of a socket?
Comment: What do you mean by status?  Whether it's opened or closed?
Comment: Whether the socket is in a valid state, meaning that the tcp connection is active
Answer: There are three ways to detect whether the socket is open or closed, but neither of them are 100% full proof.
The first is to attempt a read on the socket as follows:
my $ret = recv($sockfd, $buff, 1, MSG_PEEK | MSG_NOWAIT);

If the socket has went through an orderly shutdown, i.e. the peer called shutdown for writing or called close AND the FIN packet has arrived then this call will result in a 0 length read indicating a closed socket.  This also helps if your peer application crashed since the OS will close the connection and send a FIN. However, if your peer machine has crashed or your peer application has locked up this won't help you since each end of the connection maintains independent state.
A second way to detect a broken connection is by probing your peer.  If you send a 0 length packet to your peer ( which is should be able to handle ) and the application has crashed then you send a second 0 length packet your application will get the SIG_PIPE signal indicating a broken pipe.
Another way to deal with this issue is to use an application level heartbeat in which the peers periodically send a heartbeat packet to each other indicating that they are alive and functioning properly.
One last option is to use the SO_KEEPALIVE socket option, although this is of limited use since it will only detect a broken socket after approximately 2 hours of inactivity.
If you really must know fairly quickly when a connection is broken, then the most reliable option is probably going to be the application level heartbeat.
Comment: It's actually `MSG_DONTWAIT` flag name
Answer: doh!, the answer was obvious in retrospect, use the connected call.
$socket = IO::Socket::INET(localhost, 1000);
die "no connection" unless $socket -> connected();
$socket -> send("your face here for $20");
die "socket is dead" unless $socket -> connected();
$socket -> recv($data);

Comment: This is probably not what you want.  All this call will tell you is if the initial connection succeeded or if there has been a graceful shutdown of the connection by the your own program or the peer.  If there are other problems with the connection such as the peer computer crashed, or the peer application locked up, or the peer computer is unavailable due to network problems this won't help you.  Basically, this won't catch any of the error conditions you're likely looking to catch.
Comment: True, but in my use case, I know that the message decoding process has failed anyway, what I needed was a way to distinguish between badly formatted messages and broken sockets.
Comment: also, btw, the connected call is best used in conjunction with the select function.


Docker - Access host /proc

Question: This is a duplicate of a post I have created in the docker forum. Thus I am going to close this / the other one once this problem is solved. But since no one answers in the docker forum and my problem persists, I'll post it again, looking forward to get an answer.
I would like to expose a server monitoring app as a docker container. The app I have written relies on /proc to read system information like CPU utilization or disk stats. Thus I have to forward the information provided in hosts /proc virtual file system to my docker container.
So I made a simple image (using the first or second intro on docker website: Link) and started it:
docker run -v=/proc:/host/proc:ro -d hostfiletest

Assuming the running container could read from /host/proc to obtain information about the host system.
I fired up a console inside the container to check:
docker exec -it {one of the funny names the container get} bash

And checked the content of /host/proc.
Easiest way to check it was getting the content of /host/proc/sys/kernel/hostname - that should yield the hostname of the vm I am working on.
But I get the hostname of the container, while /host/proc/uptime gets me the correct uptime of the vm.
Do I miss something here? Maybe something conceptual?
Docker version 17.05.0-ce, build 89658be running on Linux 4.4.0-97-generic (VM)
Update:
I found several articles describing how to run a specific monitoring app inside a containing using the same approach I mentioned above.
Update:
Just tried using an existing Ubuntu image - same behavior. Running the image privileged and with pid=host doesn't help.
Greetings
Peepe
Comment: Still the same problem. /prochost/sys/kernel/hostname yields the container host name as well as /prochost/mounts yields container mounts.
Comment: Check https://hub.docker.com/r/frapsoft/atop/ and `docker run --pid=host`
Answer: The reason of this problem is that /proc is not a normal filesystem. According to procfs, it is like an interface to access some kernel data and system information. This interface provides a file-like structure, so it can make people misunderstand that it is a normal directory.
Files in /proc are also not normal files. They are empty (size = 0). You can check by yourself.
$ stat /proc/sys/kernel/hostname
File: /proc/sys/kernel/hostname
Size: 0             Blocks: 0          IO Block: 1024   regular empty file

So the file doesn't hold any data, but when you read the file, the kernel will dynamically return to you a corresponding system information.
To answer your question, /proc/sys/kernel/hostname is just an interface to access the hostname. And depending on where you access that interface, on the host or on the container, you will get the corresponding hostname. This is also applied when you use bind mount -v /proc:/hosts/proc:ro, since bind mount will provide an alternative view of /proc. If you call the interface /hosts/proc/sys/kernel/hostname, the kernel will return the hostname of the box where you are in (the container).
In short, think about/proc/sys/kernel/hostname as a mirror, if your host stands in front of it, it will reflect the host. If it is the container, it will reflect the container.
Answer: I know its a few months later no but I came across the same problem today.
In my case I was using psutil in Python to read disk stats of the hosts from inside a docker container.
The solution was to mount the whole host files system as read only into the docker container with -v /:/rootfs:ro and specify the path to proc as psutil.PROCFS_PATH = '/rootfs/proc'.
Now the psutil.disk_partitions() lists all partitions from the host files system. As the hostname is also contained within the proc hierarchy, I guess this also works for other host system information as long the the retrieving command points to /rootsfs/proc.

Docker - Access host /proc


Table Tennis exercises by yourself and without a table?

Question: Is it possible to improve your table tennis skills by yourself without a table as well?
If so are there exercises that one could do to improve your table tennis skills by yourself?
Comment: @MarcusSwope i thought about that but i'd rather use a table tennis racquet ;). I guess i could do that to the end of the bat.
Comment: I would take a look here for some ideas http://sports.stackexchange.com/questions/50/recommended-drills-to-help-keep-your-eye-on-the-ball
Comment: Update: found a youtube video on this - http://www.youtube.com/watch?v=i8Pv8vWdyL8&feature=relmfu
Comment: Found another exersize: http://www.youtube.com/watch?v=iOvEOFO9YYM&feature=g-vrec&context=G29d6775RVAAAAAAAAAA
Answer: One article discussed various solo table tennis training drills. Topics included:

Serving Practice
Shadow Play and Ghosting
Solo Footwork Drills
Video Footage Analysis
Table Tennis Robots

Each topic had several practical pointers and tips.
One table tennis forum post dealt with this question and had two suggestions:

Shadow play (mentioned in the first article above)
Purchase a return board

Regarding the return board, one poster noted:

return boards have proper rubber on them, and can actually help you,
  unlike folding the table up

These resources suggest that there's a wide range of drills that one can do without purchasing any equipment - at minimum shadow play and footwork drills can be done without any equipment. At the high end, buying a robot or doing video analysis can be costly, but can provide good returns if solo practice is very important.
Comment: Ah great idea return board


Qual é a diferença entre as keywords extends e implements em java?

Question: Gostaria de saber quando é que devem ser usadas e como é que podemos distingui-las. Sei que basicamente implements, significa que essa classe implementa uma classe ou um interface. E extends poderá aceder aos métodos da classe base.
Comment: `extends` serve somente para herança, quando você quer que uma determinada classe "estenda" suas funcionalidades para uma outra, de modo que ela possa usar essa funcionalidade original ou adaptar para sua necessidade. `implements` é somente para interface, pois esta funciona como um contrato, e você precisa "implementar" na classe os métodos que a interface tiver.
Comment: Só uma nota, @diegofm, para quem está criando interfaces. `extends` também pode ser usado no contexto de hierarquia de interface https://docs.oracle.com/javase/tutorial/java/IandI/nogrow.html
Comment: @diegofm, mas extends poderá ser usado tanto em uma classe derivada como uma classe base? ou apenas uma classe derivada é que poderá ter extends?
Comment: @jorgesaraiva não entendi sua pergunta.
Comment: @jorgesaraiva, creio que esse artigo pode te ajudar : http://www.tiexpert.net/programacao/java/heranca.php
Answer: extends
Usa-se extends quando você deseja aplicar herança á sua classe.
Quando falamos que a classe A estende a classe B, significa que A herda alguns (ou todos) métodos e atributos da classe B.
Os únicos métodos e atributos que não são herdados são os que possuem o modificador de acesso private.
Pode-se aplicar diversos níveis de herança. Por exemplo:
class A extends B { }
class B extends C { }
class C extends D { }

Ao fazer isso, os membros com modificador de acesso public das classes D, C e B são acessíveis na classe A. Os membros com modificador de acesso protected da classe B também são acessíveis na classe A.
Em Java é possível herdar apenas de uma classe. Não existe herança múltipla.

implements
Usa-se implements quando você deseja implementar uma interface.
Não implementa-se classes. Implementa-se apenas interfaces.
Uma interface "firma um contrato" entre classes em que define comportamentos (métodos) que devem ser sobrescritos pela classe que os herda (se essa for uma classe concreta).
Uma interface pode conter métodos e constantes. Constantes em Java são definidas pelas palavras static e final. Ex: public static final String NOME = "xpto";.
Os métodos de uma interface não tem corpo. Eles tem apenas a sua definição. Ex:
public interface Imprimivel {
    public void imprime();
}

Pode-se também aplicar herança entre interfaces (exclusivamente):
public interface Monstro {
    public void ameacar();
}

public interface MonstroPerigoso extends Monstro {
    public void destruir();
}

public class Godzilla implements MonstroPerigoso {
    @Override
    public void ameacar() {
        //implementação
    }

    @Override
    public void destruir() {
        //implementação
    }
}

Vale a leitura desses capítulos da apostila da Caelum: Herança e Interfaces
Comment: Só faltou citar o que foi mencionado pelo  @JeffersonQuesado, sobre interfaces tambem puderem usar extends entre si apenas.
Comment: @diegofm atualizado
Answer: implements é usado quando uma classe implementa uma interface, extends é quando uma classe estende uma outra classe (concreta ou abstrata) ou quando uma interface estende uma outra interface. Ou seja, a keyword extends é usada para quando um tipo (classe ou interface) é derivado do seu mesmo tipo, e o implements é sempre que deseja fazer uma implementação, no caso uma classe implementa uma interface.
Possibilidades de uso das keywords:
class Classe {}
class SubClasse extends Classe {}

interface Interface {}
interface SubInterface extends Interface {}

class SubClasse2 implements Interface {}
class SubClasse3 extends SubClasse implements Interface {}

A interface nunca tem implementação portanto ela nunca pode usar o implements, ela só pode usar o extends, e ela também nunca pode derivar de uma classe, pois caso contrário estaria herdando implementações para si própria, algo que por definição não faz sentido para uma interface.
As duas keywords representam que a classe ou interface está herdando todo os métodos e atributos e consequentemente tendo acesso a eles, mas ainda existem os modificadores de acesso que podem por exemplo declarar métodos e atributos como private para que eles somente pertençam apenas a classe que o declarou.
Por a interface nunca possuir implementação não faria sentido que ela em alguma hipótese tenha um método ou atributo que somente ela use (privado), pois tudo que a interface faz é declarar métodos e atributos para que seja implementados por classes.
Exemplo:
interface Interface2 {
    //private int i = 0;  <<< ERRADO! apenas public, static e final são permitidos
}

class Classe2 {
    public int i = 0;
    private int j = 1;
}

class SubClasse4 extends Classe2 {
    public void imprime() {
        System.out.println(i);
        //System.out.println(j); <<< ERRADO! O atributo não está visível para a subclasse
    }
}


Is the existence of a linear operator between two vectors always possible?

Question: I have some questions to gain insigth on some core concepts of linear algebra:
Supose $V(\mathbb{C}^n)$ is a complex vector space, $\{v,u\}\in V$ are arbitrary distinct vectors with $v≠0$. Can we always find a linear operator $\tau:V\rightarrow V$ such that $\tau(v)=u$?
If it is the case, can we always find such an operator on a real vector space?
Does this implies that every linear transformation has an eigenvalue?
Comment: Why would this imply that every linear transformation has an eigenvalue?
Comment: One needs to assume $v$ is nonzero, unless the case happens to be $u$ is also zero.  A linear transformation on a real vector space may have no real eigenvectors.
Comment: For the first part, split into cases (according to whether $\{\vec v, \vec u\}$ are linearly dependent or not.  If they are independent, extend to a basis.
Answer: Yes, it is always possible to find a map such that $v \mapsto u$, except when $v = 0$ because a linear transformation always maps the zero vector to itself. Indeed, you can take $v$ as the first basis vector of $V$ and maps it to $u$, such that the vector $u$ will appear as the first column of the matrix representation of $\tau$.
However, the eigenvalues of such a map are not necessarily real; indeed, the case $v = (1,0), u = (0,1) \in \mathbb{R}^2$ are linked by the linear operator $\tau = \begin{pmatrix} 0 & -1 \\ 1 & 0 \end{pmatrix}$, whose eigenvalues are $\pm i$.
If you want another concrete example, you can take $\tau = \frac{uv^\dagger}{\|v\|^2}$ for instance. The same map works for the real case when replacing $v^\dagger$ with $v^T$. Rotations constitute good candidates too.
Comment: Thanks for your answer, this was very helpful.
Comment: Shouldn't the $u$ vector on your example be $u=(0;-1)$?
Comment: @SimónFlavioIbañez
$$
\tau v =
   \begin{pmatrix}
      0 & -1 \\
      1 & 0
   \end{pmatrix}
   \begin{pmatrix}
      1 \\
      0
   \end{pmatrix}
=
   \begin{pmatrix}
      0 \\
      1
   \end{pmatrix}
= u
$$


Android - Firebase Deeplink URL is lost when we call the deeplink URL from another website instead of passing it directly in href tag

Question: We are using hybrid platform - kony for mobile application(android and IOS).
I have configured dynamic links in firebase dynamic links section. When we add the links directly in our email buttons as href tag and if user clicks that link in gmail from mobile browser it's working without any issues in both android and ios. Received dynamic link URL in mobile app and able to navigate user to particular screen based on URL params.
But for tracking purpose(Analytics), on click of that email button we are redirecting the user to our vendor website from there it is getting redirected to firebase dynamic URL. By doing in this way, IOS is working as expected, user is taken to particular screen.
But android has issues. App is opened in normal mode instead of deeplink mode. dynamic link is missing. So we couldn't navigate user to that particular screen.
I have added valid SHA-1 and SHA-256 release keys as well in my firebase project.
I have valid assetlinks.json file for the domain which i created. It's a custom domain with xxx.page.link(below one for reference).
URL which is used in href tag: http://t.XXX.com/click_stats/?ID=xxxx&ContactID=xxxxx&Tracking=98c68752ea4f4af1ab9922178ff608c1&StepId=128262
Firebase Deeplink URL: https://xxx.page.link/frmOffers
So 1st link will be added in email buttons href tags-  when user clicks 1st link,we will capture analytics and redirect to 1st link. Android app is opening but not in deeplink mode. Link url params are missing.
Please check and help. If IOS is working why android is not working.
<code>[{"relation":["delegate_permission/common.handle_all_urls"],"target":{"namespace":"android_app","package_name":"com.xxx.xxx","sha256_cert_fingerprints":"xxxx"}}]
</code>
Answer: you need to write an Kony FFI for this we handled this kind of use case recently and solved via FFI.


How to prove Mean Squarred Error (MSE)

Question: I would like to prove this equation of Mean Squared Error (MSE):

m is the number of training instances.
X is a m × n matrix containing all the feature values (excluding
labels) of all instances in the dataset.
X^T is the transpose of matrix X.
w is the model's parameter vector, containing the bias term w0 and the feature weights w1 to wn.
y is the vector of target values containing y(1) to y(m) values (training instances).

The above function is the gradient of the minimum squared error loss function:

where,

m is the number of training instances.
hw is the hypothesis function, using the model parameters w.
x(i) is a n × 1 vector of all the feature values (excluding the label) of the i-th instance in the dataset, and y(i) is its label (the desired output for this instance)

I would really appreciate any clue for a start at least or any relevant link where I could find more information.
Comment: I'm confident we can help, but please explain what all your symbols mean so we don't guess incorrectly.
Comment: @BDEngineer Firebug's done a nice job outlining the mechanics of this solution (+1). If you're looking for broader treatments, we have several threads with additional resources: https://stats.stackexchange.com/search?q=matrix+calculus
Comment: @whuber Yes you are right..Let me update the question
Comment: Do you know matrix calculus? It's a straightforward result from that. Also, by your result, $h$ should be a linear function
Comment: @Firebug I am a totally newbie in mathematics. Since you wrote matrix calculus, I will search of it. However, I would really appreciate it if you provide more input on that.
Comment: No need to search for that, I'll try to provide a simpler answer. It's just that, if you knew it already it would be simpler :)
Comment: @Sycorax Thank you for the comment. I have accepted the answer.
Answer: First, let's identify what we mean here:
$$\text{MSE}=\frac{1}{m}\sum (X_iW-Y_i)^2$$
And the gradient is a vector of partial derivatives:
$$\nabla_W\text{MSE}=\left[\frac{\partial \text{MSE}}{\partial W_j} \right]$$
$$\frac{\partial \text{MSE}}{\partial W_j}=
\frac{1}{m}\frac{\partial}{\partial W_j}\sum (X_iW-Y_i)^2=\\
\frac{1}{m}\sum 2(X_iW-Y_i)\frac{\partial}{\partial W_j}(X_iW-Y_i)
$$
Remember that $X_iW = \sum_k x_{ik}W_{k}$
This last derivative is then simply:
$$\frac{\partial}{\partial W_j}(X_iW-Y_i) = \frac{\partial}{\partial W_j}(\sum_k x_{ik}W_{k}-Y_i)=x_{ij}$$
Then, recognizing that the resulting sum is a vector product:
$$\frac{\partial \text{MSE}}{\partial W_j}=
\frac{1}{m}\sum 2(X_iW-Y_i)x_{ij} = \frac{2}{m} x_j^T(XW-Y)
$$
Where $x_j$ is the $j$-th column of $X$
Then reorganizing into the columns of the gradient we can recognize that it's a matrix product, arriving at the result:
$$\nabla_W\text{MSE}=\left[\frac{2}{m} x_j^T(XW-Y) \right]_j = \frac{2}{m}X^T(XW-Y)$$

Using matrix calculus:
$$\text{MSE}=\frac{1}{m}\epsilon^T\epsilon =\frac{1}{m} (XW-Y)^T(XW-Y)$$
$$\frac{\partial \text{MSE}}{\partial W}=
\frac{1}{m} \frac{\partial}{\partial W}(XW-Y)^T(XW-Y)=\\
\frac{1}{m} \left(\frac{\partial (XW-Y)^T}{\partial W}(XW-Y) + \left((XW-Y)^T\frac{\partial (XW-Y)}{\partial W}\right)^T\right)=\\
\frac{1}{m} \left(X^T(WX-Y) + ((WX-Y)^TX)^T\right) = \frac{2}{m} X^T(WX-Y)$$

How to prove Mean Squarred Error (MSE)


Filling object with CSV file returning error

Question: I Have a CSV file, that contains the following information:
Santa Catarina,Florianópolis,São José,Biguaçu,Palhoça
Rio grande do Sul,Porto alegre,,,
Paraná,Curitiba,Londrina,Ponta Grossa,

Every first data from each line, is a state, and the following are the cities of this respective state.
I have two objects, State and City, State(Estado) have Name and an ArrayList of cities, and City(Cidade) have a bunch of attributes.
To read my CSV, here it's my code:
BufferedReader r = new BufferedReader(new FileReader("C:\\Users\\Pedro Sarkis\\Desktop\\ex3.csv"));

ArrayList<Estado> estados = new ArrayList<>();
ArrayList<Cidade> cidade = new ArrayList<>();

// String estados2[];
int i = 1;
String line = r.readLine();
try {
    while (line != null) {
        //  System.out.println("Line " + i + ": " + line);
        String[] campos = line.split(",");

        for (int j = 1; j < campos.length; j++) {
            Cidade c = new Cidade();
            c.setNome(campos[j]);
            cidade.add(c);
        }

        Estado e = new Estado(campos[0], cidade);
        estados.add(e);

        cidade.clear();

        line = r.readLine();
        i++;
    }
} finally {
    r.close();
}

The problem is that I just can't limit the Cities to their respective states.
I was testing using .clear() to kinda reset my list after each while, but it's not working, because it resets all of my past data, and without using .clear(), all my States receive all the cities.
Answer: Using clear() will not work in this case, since previous elements in the list still point to the same object. Thus, it will change the value of the previous elements too. Change your code from
cidade.clear();

to
cidade = new ArrayList<>();

Comment: If the solution works for you, make sure you upvote the answer/ mark as solved.
Answer: Each instance of Estado needs to have a brand new List.  If you pass the same List object to every Estado construction, they are all sharing the same List object.  Calling clear() does not make a new or different List object, it just removes elements from the same List object.
There are two ways you can accomplish this.
First approach: You can change the Estado class to use an object-oriented practice known as defensive copying.  The Estado class would copy the List argument given to its constructor, so other code is not able to change the Estado instance by altering the List.  This way, only calling methods of Estado can alter the Estado instance.  This allows us to say that the Estado class encapsulates its data, by having exclusive control over its own state.
public class Estado {
    private String state;

    private List<String> cities;

    public Estado(String state,
                  List<String> cities) {

        this.state = state;

        // Copying the List, so any later modifications cannot affect
        // this instance.
        this.cities = new ArrayList<>(cities);
    }
}

Second approach:  Create a new city ArrayList for every line you read.
while (line != null) {

    String[] campos = line.split(",");

    cidade = new ArrayList<>();

    // ...


TcpClient reconnect to an already open connection

Question: I'm working with TcpClient to communicate with a hardware device.
The communication to the device may drop for a period of 30 seconds or so, as part of a testing process. This is fine and sometimes intended.
The problem begins when I'm sending data to the device while communication is down. Because I'm using TcpClient, I'm getting an IO exception and the connection is dropped. The connection at the device side is still open though.
How can I:

Reconnect to the open connection at the device? creating a new TcpClient will create a new connection at the device side and is unwanted...
Perhaps Make TCP retransmissions take longer then 30 seconds ?(windows 7)
Answer: Your best method is to exchange a session identifier or have some other way to track connections and have code to handle resumes. You can increase the value of your SendTimeout Property, but the receive side could still end up timiing out the connection on its end.
You cannot reopen a specific connection with TcpClient once it is closed. The only other way you might do this (raw sockets code) seems to me to be more trouble than it's worth.


Migrating vserver from Linode elsewhere

Question: I need to find an alternative to Linode due to payment issues (if you're interested mainly because Linode only accepts credit cards and I don't want to use them). However, my biggest issue with migration is actually moving all the services I have on my vserver. Assuming I get a new vserver somewhere with the same distro I have on Linode (Fedora 22), what's the safest way to perform the migration without having to setup everything again?
My initial thought was to somehow create a disk image of my Linode vserver and [somehow] deploy it on the new vserver. Anyone with experience doing this? What do you recommend?
Comment: Take this opportunity to deploy your server and applications(s) using a configuration management tool. It will be a fair amount of work the first time around, but after completion, you'll be able to repeatably deploy your infrastructure wherever you'd like in minutes.
Comment: @EEAA any suggestions for a cmt then?
Comment: Ansible is my preference. Salt is worth looking at, though, as well.
Answer: disclaimer: I work for Linode
To migrate your stuff, there're a few ways you can do it. You can make an image of your entire disk (this is EVERYTHING, including data files, DB, OS, etc). One popular way to do this is to boot into Rescue Mode (Finnix) and dump your disk to an IMG file used DD. You can then download the image over SSH - Block-level Backup
You can also migrate everything at the file-level. This will allow you to pick and choose what you want to keep, as well as likely being a smaller 'backup' that you'll have to carry over to your new hosting provider. We have a guide on several tools to do this including Rsync, tar, etc - File-level Backup
Lastly, I don't know what your specific issue was but Linode also accepts debit cards. If a bank is your issue, you can use the PayPal Debit Mastercard or even Visa gift cards. Regardless, I hope you're able to accomplish what you need.
Comment: thanks for the help. The problem is simple. In Germany there are no visa debit cards and credit cards are expensive as hell. The only solution given to me in a Linode ticket was to pay 6months by check which is a terrible solution since my bank charges my eyes and legs for a check in USD.
Comment: I will look into your suggestion of PayPal debit card. Sounds like a possibility if not too expensive.
Comment: I'm not sure if in Germany the rules are different but for me, the PayPal debit card is free. It uses your balance in your PayPal account. You'd likely need to pay the VAT fees but that's about it.

Migrating vserver from Linode elsewhere


Is there anything wrong with my codes?

Question: One of my app worked and crashed at thread0, step 35, mem address 0x00002c60
35  MyAppName                           0x00002c60 0x1000 + 7264

So I called command atos to locate the crash point as below:
atos -arch armv6 -o MyAppName.app/MyAppName 0x00002c60

it returns
-[AppDelegate setGPictureArray1:] (in MyAppName) (AppDelegate.m:9)

I show all codes before the line
//AppDelegate.h
#import <UIKit/UIKit.h>
#import "RootViewController.h"
@class RootViewController;
@interface AppDelegate : NSObject <UIApplicationDelegate> {
    UIWindow *window;
    IBOutlet RootViewController *rootViewController;
    NSMutableArray * gPictureArray1;
    NSMutableArray * gPictureArray2;

}

@property (nonatomic, retain) NSMutableArray * gPictureArray1;
@property (nonatomic, retain) NSMutableArray * gPictureArray2;
@property (nonatomic, retain) IBOutlet UIWindow *window;
@property (nonatomic, retain) RootViewController *rootViewController;
@end

 //AppDelegate.m

#import "AppDelegate.h"
#import "RootViewController.h"
@implementation AppDelegate

@synthesize window;
@synthesize rootViewController;
@synthesize gPictureArray1;//**it looks like the crash happens here**
@synthesize gPictureArray2;

I can find there is any problem.
Welcome any comment.
Comment: The problem is where you're setting `gPictureArray1` or perhaps what you're setting it to.
Answer: I would guess that either the value you are trying to set that property as is an invalid pointer (pointing to a deallocated instance) or that the existing value is invalid.
Create the following function in the implementation file and put a breakpoint in it so you can track who is calling the setter:
-(void)setGPictureArray1:(NSMutableArray*)array
{
    [gPictureArray1 release];
    gPictureArray1 = array;
    [gPictureArray1 retain];
}

Each time it is called check the value of "array" and "gPictureArray1" to make sure it is pointing to something valid. If it is called too many times you can also use NSZombies:
http://www.cocoadev.com/index.pl?NSZombieEnabled
Comment: this crash file returned as Apple Rejection Email. But the app works well on my ipad(os4.3.2, deploy target os 3.0, 4.2, 3.31) and ipod (os 3.1.3, deploy target os 3.0, 3.1.3), but it crash when just start on Apple appstore review device (os 4.3.1), I can not duplicate crash on my device

Is there anything wrong with my codes?


Calculate the area between $x^2 = py$,$ x^2 = qy$, and $y = ax$, $y = bx$!

Question: So how do you calculate the area between $x^2 = py$,$ x^2 = qy$, and $y = ax$, $y = bx$?
I have a solution, it might be bad, so that is why I'm asking.
So I drew the area, supposing that $b>a$ and $p>q$.
The points of intersections in the area we are looking for are: $ap$, $bp$, $aq$,$bq$.
Let $I$ be the area. So I suppose that
$$I=\int_{ap}^{bp}\int_{bx}^{\frac{1}{p}x^2}\,\mathrm{d}y\,\mathrm{d}x+\int_{bp}^{aq}\int_{ax}^{bx}\,\mathrm{d}y\,\mathrm{d}x+\int_{aq}^{bq}\int_{\frac{1}{q}x}^{bx}\,\mathrm{d}y\,\mathrm{d}x$$
Is this right?
Thanks in advance!!
Comment: Actually two different possibilities, depends which is bigger, $ap$ or $bq$
Comment: no. In either case, one of the integrals, the $x$ endpoints, goes either from $bq$ to $ap$ if that is increasing order, otherwise $ap$ to $bq.$ Please draw two graphs, one with $a=2, p=5, b=q=3,$ the other graph with $a=2, p=4, b=q=3.$
Comment: Does my answer cover one of the possibilities?
Answer: Let's assume that $ b > a > 0 $ and  $ p > q > 0 $. Solve all with respect to $ y $ and you get : $ y=ax, y=bx, y=(1/q)x^2, y=(1/p)x^2 $. By drawing up these functions with the way we defined the constants, the calculation of the area will go like that :
1) Find the intersections :
$ ax=(1/q)x^2 \Leftrightarrow x=aq $ (calculating the others) : $x=aq, x=ap, x=bp, x= bq $
2) Looking carefully on the plot and making sub-areas, the total area is :
$\int_{aq}^{bq}((1/q)x^2 - ax) dx + \int_{bq}^{bp} (bx - (1/p)x^2)  dx - \int_{bq}^{ap} (ax - (1/p)x^2)  dx $
3) The  condition is that $ ap > bq $ . Otherwise, the solution changes but is in the same way of thinking.
The exercise is demanding because you need to see and subtract a small area between $ax $ and $ (1/p)x^2 $. It took some time to make a good plot and notice the small things. I hope I haven't done any silly mistake with the limits of integration. Feel free to ask me anything that you may want to ask or point out something that doesn't seem correct.
Comment: Thanks! Now I feel like, I understand this, but I need be more careful with this.


What is the number of possible simple directed graphs of $n$ elements without 2-cycles and self-loops?

Question: Let $A$ be the adjacency matrix of a graph with $n$ vertices. Let $a_{ij}$ denote the entry in the $i$-th row and $j$-th column.
For a given $n$, how can we compute the number of possible networks, $f(n)$, such that

There are no self-loops: $a_{ii}=0$
There are no 2-cycles: $a_{ij}=1\implies a_{ji}=0$
there is at most one edge between two vertices: $a_{ij}\in\{0,1\}$

To get some intuition, I wrote a script that generates all such networks for $n=3,4,5$ and got $27,729,59049$ possible networks respectively. I did some reverse engineering and got that
$$f(n)=3^{\frac{n(n-1)}{2}}$$
However, I don't understand why is this the case. It is possible, of course, that the function I deduced is wrong and that it only works for those three integers.
Can someone shed some light on whether this formula is correct and why is this the case?
Comment: Do you allow multiple edges between two vertices?
Comment: @ParclyTaxel right, I forgot to clarify, I will edit the question. Thank you for pointing it out! :)
Answer: Assuming all vertices are labelled, for each of the $\binom n2=\frac{n(n-1)}2$ pairs of vertices there are three possibilities: there is an edge from $A$ to $B$, there is an edge from $B$ to $A$, there is no edge between $A$ and $B$. There are no other restrictions, so the number of different graphs is $3^{n(n-1)/2}$.


SPF permerror reported by Hotmail

Question: Our server has been receiving abuse reports from Hotmail, because of an auth-failure. One of our sending servers has been blacklisted and Hotmail is refusing delisting. We have investigated a lot, and maybe the error on the SPF record might be the cause. But Hotmail isn't providing any userfull feedback, apart from 'abuse reports' on another (non-blacklisted) server.
Feedback-Type: auth-failure
User-Agent: XMR/2.2
Version: 1.0
Original-Mail-From: <firstname.lastname@example.com>
Arrival-Date: Tue, 29 Sep 2015 22:49:33 -0700
Message-ID: <firstname.lastname@example.com>
Authentication-Results: hotmail.com; spf=permerror (sender IP is 22.214.171.124; identity alignment result is pass and alignment mode is relaxed) firstname.lastname@example.com; dkim=none (identity alignment result is pass and alignment mode is relaxed) header.d=sp.nl; x-hmca=none firstname.lastname@example.com
Source-IP: 22.214.171.124
Reported-Domain: sp.nl
DKIM-Domain: sp.nl

But when I test the SPF record (http://tools.bevhost.com/spf/), the combination appears to be valid. So I'm wondering if these are false positives by Hotmail, or if there is something else I'm overlooking.
The diagnostic tool appears to find two(?) identical SPF records:
v=spf1 ip4:18.104.22.168/27 ip4:126.96.36.199/29 a mx a:mail.sp.nl a:listserver.sp.nl a:aegir.sp.nl a:www.sp.nl a:mail1.parlement.nl a:mail2.parlement.nl ip4:126.96.36.199 ~all

Answer: I see two records, and worse, they are not identical:
;; ANSWER SECTION:
sp.nl.                  3600    IN      TXT     "v=spf1  ip4:18.104.22.168/27 ip4:126.96.36.199/29 a mx a:mail.sp.nl a:listserver.sp.nl a:aegir.sp.nl a:www.sp.nl a:mail1.parlement.nl a:mail2.parlement.nl ip4:220.127.116.11 ~all"
sp.nl.                  3600    IN      TXT     "v=spf1  ip4:188.8.131.52/27 ip4:22.214.171.124/29 a mx a:mail.sp.nl a:gazpacho.sp.nl a:listserver.sp.nl a:aegir.sp.nl a:www.sp.nl a:mail1.parlement.nl a:mail2.parlement.nl ~all"

I have highlighted the difference that I see.  Given that RFC 4408 makes it fairly clear that you should only have one SPF record, I can see how two different records would confuse some receivers.
Work out what should be in your record, and make sure there's only one of it.
And while you're at it, make sure you've listed all your servers, then terminate your record with -all; ~all is completely pointless as an SPF policy, and in some cases worse than useless (some admins here consider it a sign of an actively-spammy sender).
Comment: ~all is good in staged deployment if you are using dmarc to control reject vs quarantine, most forwarders ignore SPF anyway
Comment: This was helpful, thanks! We also discovered a 'Duplicate netblock authorization' in the SPF. With this [diagnostic tool](https://dmarcian.com/spf-survey) - (Still blacklisted though, and waiting for a response from Microsoft).
Comment: Changing the ~all to -all is 'complicated'. As some of our users want their mail forwarded.
Comment: I agree; it's just that SPF is totally pointless if you *don't*.  Noone cares which servers *should* send mail from you; we want to know which ones *shouldn't*, so we can reject it.  You have to indicate that you have confidence in the completeness of your record by setting `-all`.  If that requires changing your users' behaviour, then it's time to start doing so - or don't bother with SPF, and save yourself some hassle.

SPF permerror reported by Hotmail


Expected value of type Entity for association field got "string" instead

Question: I'm trying to insert into my Reviews table, within my controller I have:
public function indexAction(Request $request, $id)
{
    if($id != null)
    {
        // Create a new Review entity
        $review = new Review();

        $form = $this->createForm(ReviewType::class, $review,[
            'action' => $request->getUri()
        ]);

        $form->handleRequest($request);

        if($form->isValid()) {

            $manager = $this->getDoctrine()->getManager();
            $review->setPosted(new \DateTime());
            $review->setBookID($id);
            $review->setUserID($this->getUser());
            $manager->persist($review);
            $manager->flush();

        }

        return $this->render('ReviewBookBundle:Book:index.html.twig',
            ['form' => $form->createView());
    }
}

However on the line $review->setBookID($id); i get this error:
Expected value of type "Review\BookBundle\Entity\Book" for association field "Review\ReviewsBundle\Entity\Review#$bookID", got "string" instead.

How do i overcome this issues? Since I've tried creating a Book entity and setting the bookID and then passing the Book entity into the $review-setBookID, like so:
$review->setBookID($book);

but that still doesn't work?
Comment: In your Review entity, do you have a "setBook" method, which could have one argument (Book Entity) ?
Comment: @LucasDelobelle That is a good idea, let me try that!
Comment: @LucasDelobelle Could you please guide me to implementing this please?
Comment: @LucasDelobelle Could you please guide me to implementing this please?
Comment: Let's try $review->setBook($book); where $book is the Book itself (instance of Book Entity), not the id. Doctrine is smart, would do the link via the ManyToOne/OneToMany association you must have set
Comment: @LucasDelobelle Within the setBook function what should i set $book to?
Comment: You must have a $book field in your review Entity, so maybe you can try $this->book = $book in your setBook method
Comment: @LucasDelobelle I get this error: Annotation @ORM\OneToMany is not allowed to be declared on method Reviewr\ReviewsBundle\Entity\Review::setBook(). You may only use this annotation on these code elements: PROPERTY.
Comment: Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/131919/discussion-between-ktov-and-lucas-delobelle).
Comment: Look at LugiHaue answer, it should work this way :) don't write the annotation on the setBook method but on the $book attribute. You also have to set a link annotation on your Book entity where you must have a reviews attribute
Comment: @LucasDelobelle It doesn't work :/
Answer: Can you try this:
Add this in your Review Entity:
/**
 * @var Book
 * @ORM\ManyToOne(targetEntity="YourBundle\Entity\Book", inversedBy="review", fetch="LAZY")
 * @ORM\JoinColumn(name="book_id", referencedColumnName="id")
 */
protected $book;

/**
 * @return Book
 */
public function getBook()
{
    return $this->book;
}

/**
 * @param $book
 */
public function setBook($book)
{
    $this->book = $book;
}

And use:
$review->setBook($book);

$book must be instance of Book Entity
Edit
Book Entity:
public function __construct() {
    $this->reviews = new ArrayCollection();
}

/**
 * @var Review
 * @ORM\OneToMany(targetEntity="YourBundle\Entity\Review", fetch="LAZY")
 * @ORM\JoinColumn(name="review_id", mappedBy="book", referencedColumnName="id")
 */
protected $reviews;

/**
 * @return Review
 */
public function getReviews()
{
    return $this->reviews;
}

/**
 * @param Review $review
 */
public function addReview(Review $review)
{
    $this->reviews->add($review);
}

/**
 * @param Review $review
 */
public function removeReview(Review $review)
{
    $this->reviews->removeElement($review);
}

Comment: OneToMany mapping on field 'book' requires the 'mappedBy' attribute.
Comment: Shouldn't the inversedBy for the book be "reviews" rather than "review"
Comment: Yes, I think this should be "reviews", since it's the name of the target entity's attribute (here it is the "reviews" attribute located in "YourBundle\Entity\Book"). Does it work ?
Answer: You should use $review->addBook() and pass book object not ID.


Correlation of parts of two variables that miminize correlation with another variable

Question: I have three data vectors $A$, $B$ and $C$ that are all more or less correlated.
What I want is a meaningful definition for a measure of the degree of correlation between those parts of $A$ and $B$ that are not correlated with $C$.
I do apologize if this is not fully understandable. I struggle to come to an easier formulation of the problem. However I can give an example:
Example: Say $A$, $B$, $C$ could all be binary, i.e. only contain the values 0 and 1. I then would want to know the number of positions where $A$ and $B$ are 1 and $C$ is 0.
As a side note: How does one "uncorrelate" variables, i.e. given two variables $A$ and $B$ how to split $A$ into a part that is maximally correlated with B and another that is maximally uncorrelated?
Comment: I added the [tag:least-squares] tag to this question because it provides the answer!
Comment: Can you elaborate on this short notice?
Answer: Correlation, according to its usual definition, is the cosine of the angle between vectors.  Least squares regression decomposes a vector into a component within the linear span of a given set of vectors and an orthogonal component (the "residuals" or "errors").  Those two components, being at a right angle to each other, have zero correlation.  Therefore,

A meaningful measure of the "degree of correlation between those parts of $A$ and $B$ that are not correlated with $C$" is the correlation of the residuals of $A$ and $B$ with each other when regressed separately against $C$.
The part of $A$ that is maximally correlated with $B$ is the projection of $A$ onto $B$ (the least-squares fit) and the part that is "maximally uncorrelated" is the residual of the regression of $A$ against $B$.

The example in the question suggests that "correlation" may be used there in a less conventional but undefined sense.  Note that the residuals of the regressions of $A$ and $B$ on $C$ when all are binary will usually not be binary.
Comment: Thank you for the answer, I think it solves my problem. According to it, I'll now correlate the components of $A$ and $B$ that are both orthogonal to $C$.


Do clients use the IP of the default gateway for something other than ARP?

Question: When I ping an external unknown IP, I see the sent ICMP packet with the default gateway's MAC address as destination, as expected. The gateway's IP address however is not used anywhere - it's just that single packet which naturally has the external unknown IP as the destination.
So why do I (or DHCP) need to configure the IP address of the default gateway on the client? Is it really only to fire that single ARP request to find the gateway's mac address?
I know what a default gateway is. What surprises me is that the default gateway is always identified by an IP address, when in fact all you really need is its MAC address. For all I know, you could build a functioning gateway that doesn't even have an IP address.
Comment: `So why do I (or dhcp) need to configure the ip address of the default gateway on the client?` - All traffic that isn't destined for the local network goes to and through the Default Gateway.
Comment: Possible duplicate of [Does Default Gateway IP = Router IP?](https://serverfault.com/questions/186645/does-default-gateway-ip-router-ip)
Comment: Yes it is mostly ND and interface discovery. I think it also uses the Ip to compare ICMP redirect origin (if secure_redirect is used)
Answer: MAC address is defined for an interface. If an interface card at the default gateway router is changed, or the router itself is changed, MAC address changes.
An alternative would be to use a fixed MAC address on the gateway, that is, configuring the same MAC address on the interface of the new hardware. However, this is not always possible.
Therefore using only MAC address for the default gateway is not practical and therefore IP address is used.
This is also consistent with regards to OSI network layer division, where OSI layer 3 always handles routing, and layer 3 has its own address mechanisms.
Answer: When a routing table entry is created the gateway IP address is used to decide which network interface the packets have to be send on. And after that the gateway IP address is used for ND or ARP to learn the MAC address of the destination.
There are other ways to specify which network interface a routing table entry is used for. So using the IP address for that could be avoided if you really wanted to. In that case the gateway IP address would only be used for ND or ARP.
It may seem redundant, but remember that every time a layer of indirection is introduced, it means there is something which can be changed (more or less dynamically) for added flexibility. The mapping from IP address to MAC address can change, for example if you have redundant gateways.
Though bypassing the ND/ARP step is possible, it is not directly supported by much software. The reason it isn't directly supported is that the standard way of doing things happens to be more useful.
If you really wanted to bypass the ND/ARP step the way you could do it is by manually creating an entry in the ND/ARP cache and use that. It would allow you to use a gateway knowing only its MAC and not its IP address, but that is a very uncommon scenario.


How to fit and center-align Pixi.js stage in the maximized canvas element?

Question: I have prepared a complete jsfiddle for my question -

For a board game I am trying to use jQuery UI buttons on both sides and assign the rest of the screen to the Pixi drawing area.
I have found a suggestion by a Pixi.js developer to scale the stage, so that its children are scaled automatically and below I am trying to implement it:

$(function() {
  var WIDTH = 400;
  var HEIGHT = 400;
  var RATIO = WIDTH / HEIGHT;

  var scaleX = 1,
    scaleY = 1,
    offsetX = 0,
    offsetY = 0,
    oldX, oldY;

  var app = new PIXI.Application({
    width: WIDTH,
    height: HEIGHT,
    view: document.getElementById('pixiCanvas'),
    backgroundColor: 0xFFFFFF
  });

  app.stage.interactive = true;
  app.stage.on('pointerdown', onDragStart)
    .on('pointerup', onDragEnd)
    .on('pointerupoutside', onDragEnd)
    .on('pointermove', onDragMove);

  var background = new PIXI.Graphics();
  for (var i = 0; i < 8; i++) {
    for (var j = 0; j < 8; j++) {
      if ((i + j) % 2 == 0) {
        background.beginFill(0xCCCCFF);
        background.drawRect(i * WIDTH / 8, j * HEIGHT / 8, WIDTH / 8, HEIGHT / 8);
        background.endFill();
      }
    }
  }
  app.stage.addChild(background);

  var bunny = PIXI.Sprite.from('https://pixijs.io/examples/examples/assets/bunny.png');
  bunny.anchor.set(0.5);
  bunny.scale.set(2);
  bunny.x = WIDTH / 2;
  bunny.y = HEIGHT / 2;
  app.stage.addChild(bunny);

  function onDragStart(ev) {
    var stagePoint = ev.data.getLocalPosition(app.stage);
    if (bunny.getBounds().contains(stagePoint.x, stagePoint.y)) {
      bunny.alpha = 0.5;
      oldX = stagePoint.x;
      oldY = stagePoint.y;
      this.data = ev.data;
    }
  }

  function onDragMove() {
    if (this.data) {
      var stagePoint = this.data.getLocalPosition(app.stage);
      bunny.x += stagePoint.x - oldX;
      bunny.y += stagePoint.y - oldY;
      oldX = stagePoint.x;
      oldY = stagePoint.y;
    }
  }

  function onDragEnd() {
    if (this.data) {
      bunny.alpha = 1;
      var stagePoint = this.data.getLocalPosition(app.stage);
      this.data = null;
    }
  }

  $(':button').button();
  $('#scaleCheck').checkboxradio();

  window.onresize = function() {
    if (!$('#scaleCheck').prop('checked')) {
      return;
    }

    var canvasRatio = $('#pixiCanvas').width() / $('#pixiCanvas').height();

    if (canvasRatio > RATIO) {               // if canvas is too wide
      scaleX  = 1 / canvasRatio;
      scaleY  = 1;
      offsetX = 0;
      offsetY = 0;
    } else {
      scaleX  = 1;
      scaleY  = canvasRatio;
      offsetX = 0;
      offsetY = 0;
    }

    app.stage.scale.set(scaleX, scaleY);
    app.stage.position.set(offsetX, offsetY); // how to center-align?
  }

  window.onresize();
});
body {
  margin: 0;
  padding: 0;
}

#mainDiv {
  width: 100%;
  height: 100vh;
  background: #FFF;
  display: flex;
}

#leftMenuDiv {
  text-align: center;
  background: #FCC;
}

#rightMenuDiv {
  text-align: center;
  background: #CFC;
  flex-shrink: 1;
}

#canvasDiv {
  flex-grow: 1;
}

#pixiCanvas {
  width: 100%;
  height: 100%;
  background: #CCF;
  box-sizing: border-box;
  border: 4px red dotted;
}

    <button id="btn1">Button 1</button>

    <button id="btn2">Button 2</button>

    <button id="btn3">Button 3</button>

    <canvas id="pixiCanvas"></canvas>

    <input id="scaleCheck" type="checkbox">
    Fit and center Pixi stage

    <button id="btn4">Button 4</button>

    <button id="btn5">Button 5</button>

    <button id="btn6">Button 6</button>

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.css">

My problem with the above code is the following -
When you first run my code, it works. But after you set the checkbox and resize browser window, dragging will break.
Answer: I have been able to solve my problem by the following code:

$(function() {
    var WIDTH = 400;
    var HEIGHT = 400;
    var RATIO = WIDTH / HEIGHT;

    var scaleX = 1, scaleY = 1, offsetX = 0, offsetY = 0;

    var app = new PIXI.Application({
        width: WIDTH,
        height: HEIGHT,
        view: document.getElementById('pixiCanvas'),
        backgroundColor: 0xFFFFFF
    });

    var background = new PIXI.Graphics();
    for (var i = 0; i < 8; i++) {
        for (var j = 0; j < 8; j++) {
            if ((i + j) % 2 == 0) {
                background.beginFill(0xCCCCFF);
                background.drawRect(i * WIDTH / 8, j * HEIGHT / 8, WIDTH / 8, HEIGHT / 8);
                background.endFill();
            }
        }
    }
    app.stage.addChild(background);

    var bunny = PIXI.Sprite.from('https://pixijs.io/examples/examples/assets/bunny.png');
    bunny.interactive = true;
    bunny.buttonMode = true;
    bunny.anchor.set(0.5);
    bunny.scale.set(2);
    bunny.x = WIDTH / 2;
    bunny.y = HEIGHT / 2;
    bunny.on('pointerdown', onDragStart)
         .on('pointerup', onDragEnd)
         .on('pointerupoutside', onDragEnd)
         .on('pointermove', onDragMove);
    app.stage.addChild(bunny);

    function onDragStart(ev) {
        this.data = ev.data;
        this.alpha = 0.5;
    }

    function onDragMove() {
        if (this.data) {
            var newPos = this.data.getLocalPosition(this.parent);
            this.x = newPos.x;
            this.y = newPos.y;
        }
    }

    function onDragEnd() {
        if (this.data) {
            this.alpha = 1;
            this.data = null;
        }
    }

    $(':button').button();
    $('#scaleCheck').checkboxradio();

    window.onresize = function() {
        if (!$('#scaleCheck').prop('checked')) {
            return;
        }

        var canvasRatio = $('#pixiCanvas').width() / $('#pixiCanvas').height();

        if (canvasRatio > RATIO) {          // if canvas is too wide
            scaleX = RATIO / canvasRatio;
            scaleY = 1;
            offsetX = (1 - scaleX) * WIDTH / 2;
            offsetY = 0;
        } else {
            scaleX = 1;
            scaleY = canvasRatio / RATIO;
            offsetX = 0;
            offsetY = (1 - scaleY) * HEIGHT / 2;
        }

        app.stage.scale.set(scaleX, scaleY);
        app.stage.position.set(offsetX, offsetY);
    }

    window.onresize();
});
body {
    margin: 0;
    padding: 0;
}

#mainDiv {
    width: 100%;
    height: 100vh;
    background: #FFF;
    display: flex;
}

#leftMenuDiv {
    text-align: center;
    background: #FCC;
}

#rightMenuDiv {
    text-align: center;
    background: #CFC;
    flex-shrink: 1;
}

#canvasDiv {
    flex-grow: 1;
}

#pixiCanvas {
    width: 100%;
    height: 100%;
    background: #CCF;
    box-sizing: border-box;
    border: 4px red dotted;
}

    <button id="btn1">Button 1</button>

    <button id="btn2">Button 2</button>

    <button id="btn3">Button 3</button>

    <canvas id="pixiCanvas"></canvas>

    <input id="scaleCheck" type="checkbox" checked>
    Fit and center Pixi stage

    <button id="btn4">Button 4</button>

    <button id="btn5">Button 5</button>

    <button id="btn6">Button 6</button>

<link rel="stylesheet" href="//cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.css">

The main change for me has been to make the draggable sprite interactive and add drag listeners to it - instead of the app.stage.
Maybe app.stage could be interactive as well as in my original code? I think there is some matrix calculation required there in that case, that is why the math is difficult to figure out.


Gluing lemma for convex functions?

Question: Let $A, B$ are two open intervals of real numbers such that $A\cap B\neq\emptyset.$ Suppose there are two convex functions $f: A\to\mathbb{R},$ and $g: B\to\mathbb{R}$ satisfying $f(x)=g(x)$ for all $x\in A\cap B.$ Then the function $$ F(x) =
\begin{cases}
f(x),  & \text{if $x\in A\setminus B$} \\
g(x), & \text{if $x\in B$}
\end{cases}$$ ought to be convex. If both $f, g$ are differentiable (or twice differentiable) the proof is straightforward. I suppose a proof without this assumption can be given via the three slopes rule, but did not succeed yet.
Can somebody give a proof or a reference for this fact.
Answer: The statement is true for open intervals $A, B$ because convexity is a local property for functions defined on open intervals.
Every $x \in A \cup B$ has a neighborhood $U$ such that $\left.F\right|_U =\left.f\right|_U$   or $\left.F\right|_U =\left.g\right|_U$, so $F$ is locally convex and therefore convex.

Gluing lemma for convex functions?


Are planes without $n$ points isomorphic as algebraic varieties for different n?

Question: Denote $\mathbb A^d_n=\mathbb A^d \setminus \{x_1, \ldots, x_n\}$ (the algebraic variety over the field $k$). Then $\mathbb A^1_n$ are not isomorphic over $k$ for different $n$, probably because the multiplicative group of global invertible functions is $k^* \times \mathbb Z^n$. For $d \geqslant 2$, $\mathbb A^d_n$ is affine iff $n=0$ (by "algebraic Hartog's lemma"). The questions are the following:

Are $\mathbb A^1_n$ isomorphic for different choices of points?
For $d \geqslant 2$, are $\mathbb A^d_n$ not isomorphic for different $n$?
Is there a general method to compare varieties, explicitly covered by affine ones?
Comment: The automorphism group of $\mathbb{A}^1$ acts 2-transitively, so you can remove any two points you like. I don't know whether you can remove any three points you like, though.
Answer: For two different choices  $(x_i)$ and $(y_i)$ of your $n$ points  the varieties $X=\mathbb A^1 \setminus \{x_1, \ldots, x_n\}$ and $Y=\mathbb A^1 \setminus \{x_1, \ldots, x_n\}$ will not be isomorphic as soon as  $n\geq 3$.  Here is why:
Any isomorphism $f:X\to Y$ is a birational map and thus extends to a biregular  automorphism $F:\mathbb P^1\to \mathbb P^1$.  (Hartshorne, Chapter I, Proposition 6.8)
 That automorphism should biject $\{\infty,x_1\,x_2,x_3\}$ onto $\{\infty,y_i,y_j,y_k\}$ , which is impossible in general.
Indeed,  the cross-ratio $(\infty,x_1;x_2,x_3)$ of the four points $\infty,x_1\,x_2,x_3$ will not be equal to  $(\infty,y_i;y_j,y_k)$ in general (even  allowing for some permutation of those points) while any automorphism of $\mathbb P^1$ must preserve the cross-ratio of four given points.