NodeJS - SHA256 Password Encryption

Question: I'm currently learning about encryption and password safety in NodeJS. I'm working with a current example that currently is using PBKDF2, I'd like to switch this out to use SHA256 instead. Is this possible and/or make sense? How would I go about it?
var crypto = require('crypto');

var len = 128;

var iterations = 13000;

module.exports = function (pwd, salt, fn) {
  if (3 == arguments.length) {
    crypto.pbkdf2(pwd, salt, iterations, len, fn);
  } else {
    fn = salt;
    crypto.randomBytes(len, function(err, salt){
      if (err) return fn(err);
      salt = salt.toString('base64');
      crypto.pbkdf2(pwd, salt, iterations, len, function(err, hash){
        if (err) return fn(err);
        fn(null, salt, hash);
      });
    });
  }
};

Comment: This is a fairly good summary of password hashing: https://crackstation.net/hashing-security.htm
Comment: _Please_ do not use sha256 for password hashing https://security.stackexchange.com/questions/211/how-to-securely-hash-passwords
Comment: This may help you http://stackoverflow.com/questions/13714103/hashing-a-password-using-sha256-and-net-node-js
Comment: https://www.npmjs.com/package/sha256 Check this module
Answer: If wanted to generate sha256 hashes, then you'd have to drop the iterations and length property as those are specific to pbkdf2. You would then use crypto.createHash() which uses OpenSSL to generate hashes. That being said, the types of hashes you can generate are dependent on the version of OpenSSL that you have installed.
var crypto = require('crypto');
var hash = crypto.createHash('sha256').update(pwd).digest('base64');

Your specific implementation might look like this:
var crypto = require('crypto');
module.exports = function(pwd, fn) {
  var hash = crypto.createHash('sha256').update(pwd).digest('base64');
  fn(null, hash);
};

Comment: The hash is still `sha256`, it's just in a different encoding. The person who asked the question also used `base64` as his encoding, therefore, I answered like so.
Comment: Actually to generate a password hashed in sha256, you have to use digest 'hex':
var hash = crypto.createHash('sha256').update(pwd).digest('hex');
Comment: Yeah, it's just that I guess some people don't manage encoding and want a result in basic SHA256 (such as me), I used some time to understand it so I just wanted to help for the nexts :p
Comment: Hi @user740189,

`SHA` is a one way algorithm, a hashing algorithm. It is not meant to be reversed and can't be. A hashing algorithm is meant to represent data in a unique way. It is not meant to store data.
Comment: Hi, This is nothing but encryption. Can you please let me know how to decrypt?.

NodeJS - SHA256 Password Encryption


Text categorization/classification for small scale text

Question: I'm looking into a way to classify/categorize sentences into pre-defined categories (around 10-15). Yes, indeed sentences, not articles or paragraphs.
Given the average length of articles are not too long (2-10 normal pages) and number of articles (tens-hundreds) are relatively small, it is quite a small scale problem. However, accuracy is much more important.
Because I am quite new in this field, I started by looking into some introductory papers and few generic open source projects (e.g. WEKA, GATE and LingPipe ). However, what I found so far are pieces too hard to put together to fit my purpose.
What specific algorithm/software/resource do you recommend on this problem?
Thanks in advance.

I did find a tool called TagHelper which is quite suitable for few aspects of my purpose and am still exploring it.
But I still expect more insights and suggestions. Thanks!
Answer: In my limited experience short texts don't make things appreciably harder.  Word count data is hopelessly sparse anyway, so vigorous regularisation of some sort is needed anyway.  Basically, you've got a document classification problem with really small documents.
Since you're already looking at Lingpipe, you may find the book Text Analysis with Lingpipe helpful.  It's a work in progress, but the basics of classification are there.  For a more introductory exposition, without accompanying software package, there also Manning et al.'s Introduction to Information Retrieval (don't be fooled by the title).
Since 'accuracy is important' the precision, recall, and confusion matrix discussions in these texts will be important to understand.  You may find that some categories simply cannot be reliably distinguished, a fact which is important to communicate and/or that one of recall and precision is much more important to you than the other, so you'll want to threshold your classifier decisions carefully.
Comment: Hi there, thanks for these very useful suggestions! Any further recommendation about the tools? E.g. you mentioned Lingpiepe, will that be more suitable for this purpose than, say, WEKA or GATE or others. Because I am not familiar with them. Thanks.
Comment: My impression is that GATE is more for folk interested in Information Extraction *system* construction. And personally I always found WEKA a bit awkward to use, and somewhat behind the times on algorithms.  But these are very subjective judgements - you should try each one out.  Most have tutorials you can work through.
Answer: A very good resource that's pretty easy to understand is the O'Reilly book "Natural Language Processing with Python" by Bird, Klein and Loper. Also have a look at the NLTK toolkit that's used by that book.
Comment: Hi Dave, thanks for your reply. Though I do not have access to the book, I visited NLTK website. "Classifier" or "Categorize", etc, keywords are searched, but no relevant pages show up. Could you give more specific hints on using text classification using NLTK?
Comment: You can actually find the book online at http://www.nltk.org/book. Look at Chapter 6.
Answer: I've used SVMLight for a similar text classification problem!


Am I Double-Dipping on my Hypothesis Test?

Question: I am testing a large number of simultaneous hypotheses (~300), with each hypothesis test trying to determine whether a given test statistic differs significantly from zero.
Am I cheating the multiple hypothesis framework if I first throw out all test statistics whose estimates are obviously very close to zero, and then run my multiple hypothesis test (with alpha=0.05) on the remainder?
For example, suppose I started with 300 test statistics with values ranging from -8 to 8. Suppose further that 200 of these statistics have values in the range [-1,1]. Would it then be acceptable to throw away these 200 test statistics (which I know will not reach significance in any subsequent hypothesis test), and only test the remaining 100 statistics with a Bonferroni-corrected p-value cutoff of $\frac{0.05}{100}$ (instead of $\frac{0.05}{300}$)? Or am I double-dipping here?
Comment: Straight up double dipping.
Comment: Perhaps consider [false discovery rate](http://www.biostathandbook.com/multiplecomparisons.html)?
Comment: Before using any 'correction' for multiple comparisons you should decide whether the nature of your questions require attention to false positive error rates at the expense of more false negative errors. Contrary to the impression that might be gained from many answerers, there are many situations in which 'correction' will throw away interesting results for no good reason.
Comment: @MichaelLew My situation does require having some guarantees about the number of false positives, but as Penguin_Knight pointed out, I'm actually more interested in the false discovery rate than in the family-wise error rate
Answer: With each statistic, you're still checking whether it would exceed the critical value. For a given test statistic, knowing that it doesn't exceed the critical value because its absolute value is less than 1 doesn't alter the fact that you looked at its size and figured out whether that statistic fell in the rejection region.
Which is to say, you still did a test.
Consider the following scenario. We have a large collection of t-statistics (say 100 of them), all with df between 55 and 60, on each of which we need to perform a two-tailed t-test. We each take the collection of statistics and find out which ones were rejections:
A) For each test statistic, I find the relevant critical value and either reject or fail to reject the null. Let's say I decided "not reject" on 30 of them and "reject" on 70 of them. This is how things normally work, and clearly I did 100 tests.
B) You happen know that the critical value for $\nu=60$ is above 2, and it's higher still for $\nu<60$. So first you pull out all the cases with $|t|<2$ and put those aside; you don't even calculate the corresponding critical value for those cases because clearly they can't exceed it. Now for the remaining 71 cases you calculate the critical value and reject 70 of them.
When you took approach B did you really do 29 fewer tests than I did under scheme A? You still started with 100 test statistics, and you still classified them as 'reject' or 'not reject' -- you just split that decision up into two stages -- (i) don't reject any with $|t|<2$, and (ii) check to see which of the remaining cases are rejections.
[Or consider it this way: we know the first approach is correct. If you come up with any scheme that organizes the same work differently and it comes up with a different count of how many tests you did, you must have counted wrong.]
Comment: For future readers: I found this to be a great explainer on multiple hypothesis testing in general: http://statweb.stanford.edu/~joftius/slides/testing.pdf


Best Way For Users To Upload Data From Android App To Web Server Database

Question: I'm writing an app that uses a formula to calculate a number.  The formula however is only an estimate, so I want to get actual results from users so I can improve it.  What is the best way to handle this?
Answer: I'd build a REST API on your web server, and have the Android app feed data back to it so the API can store it in your central database.
http://www.youtube.com/watch?v=xHXn3Kg2IQE
http://www.slideshare.net/sullis/connecting-to-web-services-on-android/
How you create the API will depend on what platform the API is being built on.


How to sql-sync prefixed multi-site installations within the same database with drush?

Question: I am running several Drupal 7 multi-sites that share same code base as well as the same database with prefixed tables per multi-site (some tables are shared too).
I chose this setup, because i need several sites, each representing a region. They share the same code but not the same content. (This is why i didn't chose Domain Access). The reason, why all multi-sites share the same database instance with prefixed tables is because i need cross-multi-site queries to aggregate global information. A context-switch within one database (by changing the table prefix whith db_set_active) is much "cheaper" than initializing a new database connection for every of the N regions.
Now I need to sync the data of one multi-site with the data of another multi-site within the same database. I thought Drush could easily do the job with:
drush sql-sync [from multi-site1] [to multi-site2]

But i recognized, that the sync is done per database and not based on prefixed tables... That sounds logical!
So the command above would sync *[same_database]* with *[same_database]*, but what i need is a script that runs like this:
sync [all prefixed1_ tables] with [all prefixed2_ tables] within same database

Now my question: Has anyone worked on synchronization of multi-sites within the same database before, or has anyone a hint on best practice for my synchronization-needs?
Edit: I found out, what my needs really are. Drush's sync is not a real "sync" it is more like a "export from and import to". So it is quite more simple than a complete synchronization-task with "insert", "merge" and "delete". So i am going to write my own drush's script to fulfill my needs.
Comment: "Best" is a weasel word. Best for what? Do you want to keep many branches of one devel codebase together? Keep testing site in the same database you use for production? What's your reason to have multiple sites with same codebase and content?
Comment: Sorry for my vague formulation on best practice, i edited it. The reason for my setup is well evaluated and i can't change it anymore.
Comment: I'm not accusing you bad organisation and I do not advise to change it. It's just that without knowledge what those sites are to each other no one will be able to reliably answer about practices. For example, if one of the pages is production, and second is preproduction for testing content, and you need to update 1, 2 nodes at time, solution will be totally different than if one page is devel branch X and the other devel branch Y and you regularly delete, create and synchronize 10k nodes.
Comment: Sorry for that, i didn't meant to offend you with my short comment. I go and explain myself and edit my question once again.
Comment: You did not offend me. I'm just trying to understand what you need to try to provide you with answers.
Comment: You should separate questions about your particular multisite configuration and put them into another question, and leave this one about sql-sync and prefixed database tables.
Answer: sql-sync does not support prefixed database tables.  There is work going on in the issue queue to support removal of cache tables via wildcard expressions. Perhaps this code could be adapted to support prefixed tables as well, if a motivated contributor would write and submit a patch.
Comment: Same question was already discussed on drush's project page as well: http://drupal.org/node/625870. Anyway, i accept this answer, because i have to do it myself ;)
Answer: Your safest, if not fastest, bet is to use Feeds. You can export data that needs to get synchronized from one site using Views, maybe Views Data Export. This little tool will create XML. It can be created on cron run and stored on your server's HDD. Then Feeds can use it to import nodes that needs importing.
Direct database manipulation is faster, but you are asking for trouble with node IDs and revisions, unless all you want is 1:1 content copy.
Comment: Thanks for your advise on Feeds, I am going to use it for cross-region content exchange as well... but this happens on demand, triggered by content-authors, who can import content from other regions to their own region. Beside that, yes, i need a 1:1 content copy that ... however... fulfills the needs of drush's regular sql-sync job. I am going to edit my question again...


Dúvida com variável global em um controller

Question: Eu possuo um método no meu controller que é chamado através do ajax. Ele popula uma variável global dentro desse controller.
Existe alguma forma de eu acessar essa variável global já populada pelo primeiro método se eu chamar outro método ajax na mesma página e controller?
O objetivo final: Essa variável seria uma lista grande de dados que são gerados randomicamente. Então eu quero no primeiro método gerar. A pessoa vê os dados e se ela der o ok ela volta ao controller com esses mesmos dados e faz outra ação com eles.
Comment: Já tentou por em uma TempData ? não entendi certo seu contexto
Comment: Dê mais detalhes do que deseja fazer. Ás vezes, pode ter mais de uma solução para o seu caso. Exemplo: salvar a informação no localStorage do navegador, se usar Angular você pode usar o `$rootScope` (mas a informação se perde no _refresh_ da página), etc
Comment: Fiquei com a msm dúvida quando ele postou, se era angular ou não porém é do lado do server que ele está utilizando e não do client a variavel global.
Comment: Por que você precisa dessa variável global? As vezes o seu erro está na forma como tenta resolver o problema.
Answer: Não é possível fazer por variável global.
Para cada requisição o controller é instanciado, então sua variável global irá perder o valor.
Uma boa prática para fazer isso é através de Session state ou cookies.
Uma forma de fazer isso como uma gambiarra é tornar sua variável global como static, porém não é recomendado usar um estado global mutável em uma aplicação web.
Answer: Isto é uma baita gambiarra. O que você pode fazer é instalar o Redis e gravar este "estado" por usuário numa chave fora da aplicação ASP.NET MVC.
O acesso a esta variável é feito por este client.
Comment: Estava procurando exatamente isso . :)


JS loop replaces every value from the list, but it should replace only the one I'm changing

Question: I have 2 lists of inputs. One has already loaded values. I want to be able to type something in the other list so it would appear in the first list and replace the already existing value.

  <input type="text" class="ingredient1" value="a11">
  <input type="text" class="ingredient2" value="a22">
  <input type="text" class="ingredient3" value="a33">
  <input type="text" class="ingredient4" value="">
  <input type="text" class="ingredient5" value="x00123">
  <input type="text" class="ingredient6" value="">
  ...
  (24 inputs total)

  <input type="text" class="name-calc1">
  <input type="text" class="name-calc2">
  <input type="text" class="name-calc3">
  <input type="text" class="name-calc4">
  <input type="text" class="name-calc5">
  <input type="text" class="name-calc6">
  ...
  (24 inputs total)

$(".name-calc-col").children().on('input', function(){
    var name = [];

    for (let x = 1; x <= 24; x++) {
        name[x] = $(".name-calc" + x).val();
        name[x] = name[x] ? name[x] : "";

        $(".ingredient" + x).val(name[x]);
    }
});

When I type something in the .name-calc-col input its value replaces .ingredient(x) value as it should, but it also deletes every already existing value (in the primary-list inputs) that were there before (so it won't delete the value that was given from the .name-calc-col input - when I type something in name-calc1 and name-calc2, it will replace ingredient1, ingredient2 values as it should and delete ingredient3 a33 value).
I know that the loop takes EVERY value from the .name-calc-col list (also the empty ones) and replaces the values from the other list.
How can I make it so it won't replace all of the values for the list, but only the one I'm changing?
Comment: you can refer to the current element inside your event callback as `$(this)`
Comment: I know, I've been trying to do that with $(this), but can't figure it out.
Comment: `name[x] = name[x] ? name[x] : "";` is unnecessary because it's the same as `name[x] = name[x]`
Comment: Extract the "id" (`x`) from the class name or use [`$.index()`](https://api.jquery.com/index/)
Answer: Using $(this) to access the current element. This approach explicitly sets the target's class inside input elements:

$(".name-calc-col").children().on('input', function() {
  // GET CURRENT INPUT VALUE
  const name = $(this).val();

  // GET TARGET ELEMENT CLASS
  const targetClass = $(this).attr('data-target')

  // SET VALUE ON .ingredient TARGET
  $("." + targetClass).val(name);
});

  <input type="text" class="ingredient1" value="a11">
  <input type="text" class="ingredient2" value="a22">
  <input type="text" class="ingredient14" value="a33">

  <input type="text" class="name-calc1" data-target="ingredient1">
  <input type="text" class="name-calc2" data-target="ingredient2">
  <input type="text" class="name-calc3" data-target="ingredient14">

Comment: When I type in, for example, name-calc15 it doesn't replace ingredient15 value, but something else's. Inputs in both lists can have no value.
Comment: use second approach, that way you can set exactly the class name you want to target
Comment: @Andreas true, I removed that snippet
Comment: The second solution (now the first one) works just fine. Thank you!


Numbering row in mysql

Question: I want to numbering selected row in mysql. My meeting table structure:
id | date
4  | 2014-09-29
5  | 2014-10-06
6  | 2014-10-13
7  | 2014-10-20
8  | 2014-10-27
9  | 2014-11-03
10 | 2014-11-10

So i only want to select the date row that greater from today and the result should be like:
id | date        | meeting number
8  | 2014-10-27  | 5
9  | 2014-11-03  | 6
10 | 2014-11-10  | 7

Comment: Where do the values 5,6,7 come from? What real world value does "meeting number" have?
Comment: the meeting_number should be generated by sql query
Answer: You can use a self join to number the rows in order of date
select t1.id, t1.date, count(*) meeting_number
from mytable t1
join mytable t2 on t1.date >= t2.date
where t1.date > current_date()
group by t1.id

Another way using a variable in case you have multiple meetings on the same date
select id, date, meeting_number
from (
    select *,
    (@meeting_number := @meeting_number + 1) meeting_number
    from mytable cross join (select @meeting_number := 0) t1
    order by date
) t1 where date > current_date()

Comment: It do not work on me, sorry, actually i have several meeting in my table, so this code return index number the row before it selected, i want the index number after WHERE condition(meeting_id = x AND date > curdate()).
Comment: Thx for fast response Fuzzy, the meeting number do not increase, all of row returning 1
Answer: I would not go for a solution that returns SQL results exactly as you desire.  Why? Because it will likely involve a complex query that is not able to leverage indexes properly and thus is poorly performing.  Let me give you a few alternate options.
First.  Simply use two queries:
SELECT COUNT(id) As row_count
FROM meetings
WHERE meeting_date < ?

And read the value for row_count. Or simply
SELECT id
FROM meetings
WHERE meeting_date < ?

And use a num_rows type functionallity to get number of rows (no need to read out the result set at all).
This will give you the number of meetings that don't meet your criteria. Store this number in your application then do you regular query
SELECT id
FROM meetings
WHERE meeting_date >= ?
ORDER BY meeting_date ASC

And simply start incrementing from the previously stored value to get desired meeting_number for display.
These two queries will likely be much faster than a more complex query where incrementation is done in result set.
As yet another suggestion, you might also consider something like this to get all the data you need in one query
SELECT
    CASE
        WHEN meeting_date < ? THEN 0
        ELSE id
    END AS identifier,
    CASE
        WHEN meeting_date < ? THEN '2000-01-01' /* Or some sufficiently old date */
        ELSE meeting_date
    END AS meeting_date,
    COUNT (id) AS id_count
FROM meetings
GROUP BY identifier
ORDER BY meeting_date ASC

This would give a result set lik:
identifier | date        | id_count
0          | 2000-01-01  | 4
8          | 2014-10-27  | 1
9          | 2014-11-03  | 1
10         | 2014-11-10  | 1

Here, the id_count value from the first row would represent the number of meetings with date less than desired date.  You would use this value as basis for incrementation in your application similar to what is descried with the two query approach above.

Numbering row in mysql


Proof that $n^2 \not\in \omega(2^n)$

Question: I'm trying to prove that $n^2 \not\in \omega(2^n)$ and I have to do it from definition.
$f(n) \in \omega(g(n)) = \left\{f(n)| \forall c>0, c \in \mathbb{R}, \exists n_0 \in \mathbb{N}, \forall n \ge n_o  (f(n) \ge c g(n)) \right\}$
So we're trying to prove that
$n^2 \ge c 2^n$.
Using limits I can prove that $\exists c, n^2 \not\ge c 2^n$
$$\frac{n^2}{2^n} \ge c$$
$$\lim_{x \to \infty} \frac{n^2}{2^n} = 0$$
$$0 \ge c$$
However I am wondering is there a way to prove it without the us of limits?
Answer: The result that you're showing is actually stronger:
You've proved that $n^2 \in o(2^n)$ which is equivalent to $\lim_{n\rightarrow\infty} n^2/2^n = 0$.
Since these asymptotic notions are defined in terms of limits, you'll most likely will be using limits (at least implicitly) for proving such statements.

Proof that $n^2 \not\in \omega(2^n)$


Choice of a vector for supporting hyperplane theorem

Question: I'm having trouble relating the content on these notes  to these ones from MIT OCW  here.
Specifically, the question I'm having is the first set of notes describes the specific half space where $a^Tx \leq a^T x_0$ and the second concludes that any one of the half spaces must include $C$.
Generally, I'm having trouble visualizing this for all $a$. Does the theorem assume the same $a$ for all boundary points?
Answer: Both statements are equivalent. For any given point on the boundary, there is a hyperplane $a^Tx=k=a^Tx_0$ such that your set is in one of the half-spaces defined by it. You can always assume it is the "positive" half-space $a^Tx\ge k$, otherwise you replace $a$ by $-a$. Yes, the linear form $a$ depends on the point, it is different at every point, in general.


php Adding numbers in a variable and a string

Question: I want to be able to add a number from a variable and from a string together. The variable $userinfo->pilotid will always be different depending on a User ID, for example it maybe 1.
I'd like to introduce an offset to this variable, by always adding 1000 to that number. How can I add the both with this?
Auth::$userinfo->code . '' . Auth::$userinfo->pilotid

Something along the lines of...
$offset = '1000';
echo Auth::$userinfo->code . '' . Auth::$userinfo->pilotid + $offset

Would the above work or am I mixing up things?
Comment: yes why not use correct concatenation that's all
Comment: It echoes `10001000`. `$userinfo->pilotid` when printed is 1.
Answer: You can use parenthesis so that you add the offset rather than concatenating the offset:
echo Auth::$userinfo->code . '' . (Auth::$userinfo->pilotid + $offset);

Answer: When concatenating, you don't need to include the '', and you should wrap the addition operation in parenthesis.
$offset = 1000;
echo Auth::$userinfo->code . (Auth::$userinfo->pilotid + $offset);

The '' was superfluous as it added nothing. PHP has type juggling, so you can concatenate an integer directly onto a string. You just need to perform the addition separately, ergo the parenthesis.


Why is the correlation between two coordinate axes measured by cosine?

Question: In factor analysis, I believe, the correlation between two factors is measured by the cosine of degree between the factors.
For instance, the correlation between two factors below is cosine θ, the degrees between factor 1 and factor 2 (or simply cosine degrees between two axes).

I would appreciate any help!
Comment: In analytic geometry this is the *definition* of the cosine!  So, look to an account of analytic (coordinate) geometry for explanations.
Comment: Not very familiar with factor analysis, but according to its Wikipedia page, the expectation of each factor is $\mathbf{0}$, so this formulation definitely has something to do with the fact that $\text{Cov}(\mathbf{f}_1,\mathbf{f}_2) \equiv \mathbb{E}\sum_i(\mathbf{f}_1-\mathbb{E}\mathbf{f}_1)_i(\mathbf{f}_2-\mathbb{E}\mathbf{f}_2)_i = \mathbb{E}\sum_i{\mathbf{f}_1}_i{\mathbf{f}_2}_i=\mathbb{E}(\mathbf{f}_1\cdot\mathbf{f}_2)$
Comment: Much appreciated!
Comment: In the subject space, variables are vectors. If variables had been centeted (as usually in factor analysis), the cosine of the angle between them equals their correlation. The factors (or principal components) extracted - we usually draw them as axes (dimensions) of the latent space where the variables having produce them lie. But the factors are readily seen as (latent) variables too, i.e. they posess variance and so can be drawn as vectors (arrows) of a length, coinciding with those axes. As factors are centered variables too, their correlation = cosine between those axes.
Comment: Subject space https://stats.stackexchange.com/a/192637/3277. Variables as vectors and factors as axes (easy to turn into vectors) in the subject space https://stats.stackexchange.com/a/119758/3277. Pic in footnote 2 showing correlating factors "covering tightly" correlating variables https://stats.stackexchange.com/a/151688/3277
Answer: There is a least squares explanation of this.
Let $y$ be one factor (it doesn't matter which) and $x$ be the other.  Ordinary least squares regression of $y$ against $x$ (without an intercept!) gives the slope estimate
$$\hat\beta = \frac{y \cdot x}{|x|^2}$$
where $y\cdot x$ is the usual dot product (equal to the matrix product $y^\prime x$ if you like) and $|x|^2 = x\cdot x$ is the squared length of $x$ (assumed to be nonzero).
Thus, the least squares fit is
$$\hat y = \hat\beta x = \left(\frac{y \cdot x}{|x|^2}\right)x.$$
The vectors $y$ and $\hat y,$ along with their common origin, form a triangle.  This is a right triangle with hypotenuse subtended by $y.$  Its right angle is at $\hat y.$  One way to tell is to use the fact that right angles are formed only between two orthogonal vectors: that is, their dot product is zero.  So we compute the dot product of the two legs $y - \hat y$ (aka the residual) and $x:$
$$ (y - \hat y) \cdot x =  y \cdot x - \left(\frac{y \cdot x}{|x|^2}\right)x \cdot x =  y\cdot x - \left(\frac{y \cdot x}{|x|^2}\right)|x|^2 = y\cdot x - y\cdot x = 0.$$
(These are called the Normal Equations.)
The most elementary definition of the cosine of the angle between $x$ and $y$ is the ratio of the length of the leg $\hat y$ to the length of the hypotenuse $y:$
$$\cos \theta_{xy} = \frac{|\hat y|}{|y|} = \frac{\bigg|\frac{y \cdot x}{|x|^2}\bigg|\,|x|}{|y|} = \frac{|y\cdot x|}{|x||y|} = \frac{|y\cdot x| / n}{\sqrt{\left(\frac{|x|^2}{n}\frac{|y|^2}{n}\right)}} = \big|\rho(x,y)\big|,$$
the absolute value of the correlation coefficient, assuming $y$ and $x$ have been centered (that is, the components of each one sum to zero).  $n$ is the common length of the vectors.
This is actually the unsigned version; by removing the absolute value around $y\cdot x$ we obtain its generalization to signed angles.
Most of the expressions appearing here have standard names in statistics.  Here's a brief translation table.  It continues to assume $x$ and $y$ are centered.
$$\begin{array}{ll}
\text{Expression} & \text{Name} \\ \hline
y & \text{Response or Dependent variable} \\
x & \text{Explanatory or Independent variable}\\
x\cdot (y - x\hat\beta)= 0 & \text{Normal equations for }\hat\beta\\
\hat y & \text{Fitted or predicted value}\\
y - \hat y & \text{Residual}\\
\hat \beta & \text{Estimated coefficient (slope)}\\
y\cdot x\,/\,n& \text{Covariance}\\
|y|^2\,/\,{n},\ |x|^2\,/\,n & \text{Variances}\\
|\hat y|^2\,/\,|y|^2& R^2,\text{ the coefficient of determination}\\
|\hat y|\,/\,|y| & |\rho|,\text{ the absolute Pearson correlation; also, the size of the cosine}\\
y\cdot x\,/\,(|y||x|) & \text{Correlation coefficient }\rho\\
\arccos\left(y\cdot x\,/\,(|y||x|)\right) & \text{(Signed) angle }\theta_{xy}\\
\hline
\end{array}$$
Comment: Wows thanks for the detailed explanation. Much appreciated!!


What does "scale" mean in this code about SIR model?

Question: This code is in C#, and I don't understand the purpose of the scale variable.
S = S + dS * scale
I = I + dI * scale
R = R + dR * scale
t = t + dt * scale

Is the purpose of this scale, to scale the graph horizontally? I think it should be horizontal, but I don't quite get it. Also, how can we do this in general, for instance, the function $f(x) = x^{2}$?
Answer: Since dt equals 1, scale is just the time step in Euler's method.
It would perhaps have been clearer to get rid of the scale variable, rename dS to dS_dt (for $dS/dt$) and similarly for dI and dR, define dt to be equal to whatever scale was (0.1 in this case), and write the code as
S = S + dS_dt * dt
I = I + dI_dt * dt
R = R + dR_dt * dt
t = t + dt

Your question about $f(x)=x^2$ doesn't really make sense in this context. (Not to me, at least.)


Install Multiple version of Cuda

Question: I have an ubuntu 18.04 VM system with Cuda 10.2 already installed.
I have to run a training of a coda on a GPU, but when I run it I get some errors like:
Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64:/usr/local/cuda-10.2/lib64:

So I think I have to install Cuda 10.0.
Is it possible to have multiple version of Cuda installed? How can I add Cuda 10.0?
I want to run my training on Nvidia GPU
Edit: I succeed Installing Cuda 10.0, downloaded Cudnn 7.4.2, extracted the .tgz file in the cuda-10.0 folder. Now I got this:
I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
how can I solve this?
Comment: i think u can only install the higher version.( there should be an error) at least i can remember in windows but having multiple versions is fine.
Answer: CUDA supports installation of multiple versions at the same time. Here is the CUDA 10.0 download archive link: https://developer.nvidia.com/cuda-10.0-download-archive
Once you have installed CUDA, you can specify for your code to look for CUDA 10.0 libraries by defining environment variable LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64.
Comment: also, if you are installing an "older" version of CUDA than one you already have, keep your "newer" GPU driver.  Deselect the option to install any older driver.  The rest of the "older" CUDA toolkit will work fine with your "newer" GPU driver.

Install Multiple version of Cuda


Is it possible to calculate trigonometric functions with the imaginary exponential

Question: By Eulers identity $e^{i \theta}=\cos(\theta)+i \sin(\theta)$ sine and cosine can be written in exponential form as
$\sin(\theta)=\frac {e^{i \theta}-e^{-i \theta}}{2i}$ and $\cos(\theta)=\frac {e^{i \theta}+e^{-i \theta}}{2}$.
Could you calculate specific values of the trigonometric function with these formulae? My guess is that the complex exponential can only be calculated using Euler's identity so you have to know the values of sine / cosine to begin with. Is there any method to calculate the value of sine / cosine using the identity above? Is there any reason why it isn't (is?) possible?
Comment: You could use powers of $e^{i\theta}$ for instance. $\left(e^{i\theta}\right)^n = e^{in\theta}$ are used to compute $\cos(n\theta)$ and $\sin(n\theta)$
Comment: The development by Euler was to observe that the complex function $f(z)=\sum_{j=0}^{\infty}z^j/j!$  satisfies $f(z_1)f(z_2)$ for all complex $z_1,z_2,$ and also $f(x)=e^x$ for all $x\in \mathbb R.$  So it is reasonable to call this function $e^z.$  And then we obtain $e^{\pm ix}=\cos x \pm i\sin x$ for real $x.$  So calculating $e^{\pm ix}$ for real $x $ is the same thing as calculating its real and imaginary parts $\cos x$ and $\pm  \sin x.$
Answer: You can evaluate it at imaginary values:
$$\sin(i)=\frac{e^{-1}-e}{2i}$$
$$\cos(i)=\frac{e^{-1}+e}2$$
But not much else.

Good for deriving some formulas though:
$$\cos^2(x)=\left[\frac{e^{ix}+e^{-ix}}2\right]^2=\frac{e^{2ix}+2+e^{-2ix}}4=\frac{\cos(2x)+1}2$$
Comment: Don't forget that the imaginary values lead to the hyperbolic functions, as well.
Comment: @CyeWaldman Of course they do :P
Answer: For me, the best use of Euler's formula arises in the rotation of coordinates, that is,
$$
x'=x\cos\theta+y\sin\theta\\
y'=-x\sin\theta+y\cos\theta
$$
In complex coordinates this transform and it inverse are simply
$$z'=z\,e^{i\theta}\text{ and }z=z'\,e^{-i\theta}$$
Basically, anything you can do in Cartesian coordinates can be done in the complex plane, and frequently much simpler (in my opinion) .
Answer: Technically, you can use the Maclaurin series of the exponential function to evaluate sine and cosine at whatever value of $\theta$ you want. But you will find that when you simplify the resulting expressions, they turn out to be exactly the same expressions as the Maclaurin series of sine and cosine themselves. So you have not made the situation any easier.
Answer: There are many many things you can use the imaginary exponential for without knowing much if anything about sine and cosine of a particular value. Perhaps the most well-known example is this:
$$ 1 = e^0 = e^{i\theta}e^{-i\theta} = (\cos \theta + i \sin \theta)(\cos \theta - i \sin \theta) = \cos^2 \theta + \sin^2 \theta. $$
Furthermore, the imaginary exponential is no more useless than sine and cosine themselves in as much as numerics are concerned. We forget, but, sine and cosine don't come for free. However, the algebraic structure of the imaginary exponential is far more lucid than that of sine and cosine. Unfortunate it is that teach them more we do not.


Webpack complains of a typescript definition file import - unless I add a `d.ts` ending

Question: I have the weirdest situation where my webpack build complains of a single instance of importing a file where it cannot find the file, unless I add a .d.ts or .d ending in the the import literal.
ERROR in ./src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx
Module not found: Error: Can't resolve '../../types/schedule-type-interfaces' in '/home/myuser/dev/nicecorp/nicecorp-frontend/src/screens/homeward-questionnaire-schedule'
 @ ./src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx 33:0-66 380:10-20 384:10-20 388:10-20 392:10-20 396:10-20 400:10-20 404:10-20 408:10-20 412:10-20 416:10-20
 @ ./src/components/one-offs/MainGuiLayout.tsx
 @ ./src/components/one-offs/AppWrapper.jsx

The file is imported in 3 files, and it is only an issue in this single file:
$ git grep schedule-type-interfaces
src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx:import { ButtonProp, Question, QuestionId } from '../../types/schedule-type-interfaces'
src/screens/homeward-questionnaire-schedule/QuestionSelector.tsx:import { Question } from '../../types/schedule-type-interfaces'
src/utils/questionnaire-util.ts:import { ButtonProp, Question } from '../types/schedule-type-interfaces'

Adding this single diff will make the build pass with flying colors:
$ git diff
diff --git a/nicecorp-frontend/src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx b/nicecorp-frontend/src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx
index 0a2a3e366..4d3def526 100644
--- a/nicecorp-frontend/src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx
+++ b/nicecorp-frontend/src/screens/homeward-questionnaire-schedule/HomewardQuestionnaireScheduleScreen.tsx
@@ -32,7 +32,7 @@ import { Encounter, Patient, QuestionnaireSchedule } from '../../types/domain-ty
 import { CancellationCriteriaEnum } from '../../types/enums'
 import { nicecorpError } from '../../types/other-types'
 // TODO: find out why we need the extension for webpack not to fail
+import { ButtonProp, Question, QuestionId } from '../../types/schedule-type-interfaces.d'
-import { ButtonProp, Question, QuestionId } from '../../types/schedule-type-interfaces'
 import { connectWithDataLoader } from '../../utils/composer'
 import {
   dayNameMap,

This makes no sense to any of us, so I am reaching out in hope of becoming smarter. So it can find the file no problem if skipping the *.ts ending, but I have to specify *.d.ts for it to not fail. So weird. This is the directory in which the file lives - no name clashes:
$ ls -1 src/types/
base-type-interfaces.d.ts
base-types.js
completable-types.js
domain-type-interfaces.d.ts
domain-types.js
dom-types.js
enums.js
other-type-interfaces.d.ts
other-types.js
questions-types.js
react-specific-types.js
README.md
schedule-type-interfaces.d.ts

Comment: Show us the content of the file that triggers the error. It's quite possible that you're using an `import` from that module in a manner that necessitates it have an implementation but you have no `.js` corresponding to it. Adding the `.d` is definitely wrong. You really don't want to do that.
Comment: That was actually right on the money, and you can see this by looking at the file listing! This file name was created by someone else. This is actually not a correct use for a `*.d.ts` file, because there is no accompanying `*.js` file! I renamed all the offending `*.d.ts` files to `*.ts` and presto, it worked.
Answer: *.d.ts files are supposed to add typings for existing javascript code. We simply used them for creating interfaces, for some weird historical reasons, and people forgot why in the process, creating new ones.
Renaming these files to *.ts was what fixed the issue.
Comment: I'm glad you got everything working. Just a slight correction. `.d.ts` files only need an accompanying implementation if they export declared values, i.e. `const` or `class` or `function` etc. However it is best to stick to `.ts`/`.tsx` files as you suggest regardless.
Comment: Yeah, and it has worked for the most part, up until this peculiar situation. No idea what made it different, but sticking with the most standard convention fixed it.
Comment: If you are curious to know why it failed, you can look at `HomewardQuestionnaireScheduleScreen.tsx` and see how the imports from `base-type-interfaces` are used. If any of them are used in value position, it would fail.
Comment: They were only used as types in interface declarations. I don't see how renaming the file they were declared in from `*.d.ts` to `*.ts` should matter at all, really, since there were no other changes needed to make the compiler stop complaining, so I put this one in the "compiler quirks" category.


LSTM network window size selection and effect

Question: When working with an LSTM network in Keras. The first layer has the input_shape parameter show below.

model.add(LSTM(50, input_shape=(window_size, num_features), return_sequences=True))

I don't quite follow the window size parameter and the effect it will have on the model. As far as I understand, to make a decision the network not only makes use of current windowframe but also the information about past windows stored in the network.
So is the window size more of a tool for saving memory / computing requirements? Or does it have a big impact on a model?
Answer:
So is the window size more of a tool for saving memory / computing requirements? Or does it have a big impact on a model?

It's both! Imagine you have a long text like War and Peace. Back-propagating from the end of the text to the beginning is a huge effort because the text is so long. Most of the effect of the update will pertain to the most recent time-steps, because the previous words in a sentence are most relevant for what you're predicting (the next word). On the other hand, imagine the most extreme truncation, which only looks back 1 time step. This won't allow the model to learn any long-term dependencies because the model focuses exclusively on the most recent time-step.
Picking a good window size is important, but fine-tuning (e.g. choosing between 64 and 65) isn't necessary -- pick a window that's "large enough" to learn longer dependencies, and call it a day.
The term of art for truncating the number of time steps in a recurrent neural network is "truncated back-propagation through time."
Comment: I don't understand your question. Batch size and window size are two different concepts.
Comment: thanks for your answer. I am still a little confused, apologies I am a slow learner. As I understand it the batch size dictates how many observations are grouped together for back propagating. I also read that you don't take advantage of the long term memory any more or less by changing batch size so are we saying that  it is the window size that specifies how long the model should look back to find a relationship?
Comment: But I when I read the answer to another of my questions https://stats.stackexchange.com/questions/385911/how-the-long-term-memory-works-in-an-ltsm-network it appears to me that nothing is even thrown out of the model so that through back propagating the long term memory shows which i think is linked to the batch size
Comment: Apologies I keep confusing myself. I thought batch size was basically the window size and didn't really understand why the window size required because the batch size dictates when the model updated the weights in the model. Think I need to start again and understand the difference between Batch size & window size


Trying to figure out a way to find consecutive elements in a list

Question: What I'm trying to do is: for every instance of two consecutive vowels in a list, the second one will become accented, but only if the previous one is unaccented. Still new to Python, this is what I have now:
sample = ['b' 'e', 'a', 'a', 'e']
vowels = "ae"

def accenting(list):
    vowel_changes = {
        "a": "á",
        "e": "é",
    }
    for i in range(0, len(list)):
        if list[i] in vowels and list[i + 1] in vowels:
            vowel_to_change = list[i + 1]
            for replacement in vowel_changes:
                if replacement in vowel_to_change:
                    vowel_to_change = vowel_to_change.replace(replacement, vowel_changes[replacement])
            print(vowel_to_change)
    return list

print(accenting(sample))

The goal is to have something like: ['b' 'e', 'á', 'a', 'é']. Right now it returns 'á', 'á', and 'é'. So the two things I'm having trouble figuring out are 1) putting the replacement back into its respective spot in the list, and 2) having the function change every other vowel instead of each one. I think I might need to cut up and create a new list for 1, and maybe have another condition for 2? Not sure.
Comment: 1) You just need to assign to that index in the list. What's so hard about that? 2) Fixing point 1 will fix this. However, the code you've posted raises an IndexError for `list[i + 1]`. You can [edit] the question to fix it. See [mre] and [ask] for advice.
Comment: Also you're missing a comma in `sample`.
Comment: BTW `list` is a bad variable name since it shadows the builtin.
Answer: Lookahead can be messy, especially with edge conditions. What if list is zero or the last element isn't a vowel and list[i + 1] goes off the end of the list? Another option is to track state as you go.
sample = ['b', 'e', 'a', 'a', 'e']

def accenting(sample_list):
    vowel_changes = {
        "a": "á",
        "e": "é",
    }
    was_unaccented_vowel = False
    for i, c  in enumerate(sample_list):
        if c in vowel_changes:
            if was_unaccented_vowel:
                sample_list[i] = vowel_changes[c]
                was_unaccented_vowel = False
            else:
                was_unaccented_vowel = True
        else:
            was_unaccented_vowel = False
    return sample_list

print(accenting(sample))

That code changed the original sample list. You could turn it into a generator with the yield statement and make a new copy
sample = ['b', 'e', 'a', 'a', 'e']

def accenting(sample_list):
    vowel_changes = {
        "a": "á",
        "e": "é",
    }
    was_unaccented_vowel = False
    for c in sample_list:
        if c in vowel_changes:
            if was_unaccented_vowel:
                c = vowel_changes[c]
                was_unaccented_vowel = False
            else:
                was_unaccented_vowel = True
        else:
            was_unaccented_vowel = False
        yield c

print(list(accenting(sample)))

I also fiddled around with a state machine although I'd be hard pressed to claim its better
sample = ['b', 'e', 'a', 'a', 'e']

vowel_changes = {
    "a": "á",
    "e": "é",
}

def _state_no_accent_check(c):
    if c in vowel_changes:
        return _state_accent_check, c
    else:
        return _state_no_accent_check, c

def _state_accent_check(c):
    return _state_no_accent_check, vowel_changes.get(c, c)

def accenting(sample_list):
    state = _state_no_accent_check
    for c in sample_list:
        state, c = state(c)
        yield c

print(list(accenting(sample)))

Answer: There are 2 problems with this code:
The first and most important is that it raises an IndexError (at least in Python3.7 which I used) because on line 10 you try to access list[i+1] while you are at the end of the list, so you fall off the end of the list and raise and Exception. In other words, it tries to access list[5] when the highest index in the list is 4, and so it doesn't know what to do and crashes. I would suggest fixing this by looking backward instead of forward. By that I mean: instead of checking whether list[i+1] is a vowel, check whether list[i-1] is a vowel. That way, you can start from index 1 since index 0 can't be a consecutive vowel anyway. Keep in mind that you'll have to pay attention not to accent the third vowel in a row; I'll let you think about how to fix that.
The second is that you're way overthinking how to replace the letter in the list. You can simply say list[i] = vowel_changes[replacement]
Answer: There's a few changes here that we can make to both make it do what you want and to remove extra lines of code that aren't actually needed.
def accenting(lst): # Use lst instead of list. list is a python reserved keyword.

First change the loop so we don't go out of range on it list.
Since we are looking ahead, we want to stop one before the length of the list so we can look at the next one
for i in range(0, len(list)-1):

We need to change the condition for checking lst[i+1]. You had it checking if lst[i+1] was in vowels. But that doesn't mean it's a consecutive. It could be the case of ae next to each other. You already check lst[i] is in vowels, so we can just check if lst[i+1] is equal to that.
Then the if statement can be expanded with another and to cover the condition of not giving the accent to multiples.
In the parenthesis we check that i is not equal to 0 so that the whole parenthesis will not count for the first letter in the array, because there are no letters before it. Then we check if the letter before is equal to the current letter.
if lst[i] in vowels and lst[i+1] == lst[i] and (i != 0 and lst[i-1] != lst[i]):

Inside of the if statement we can get rid of all the variables and the second loop. Since we know the location in lst we can directly assign it. We also know the key we want the value for in vowels, so there is no need for the loop.
lst[i] = vowels[lst[i]]]

After that you can just return the list.
The total code looks like this.
sample = ['b' 'e', 'a', 'a', 'e']
vowels = "ae"

def accenting(lst):
    vowel_changes = {
        "a": "á",
        "e": "é",
    }
    for i in range(0, len(list)-1):
        if lst[i] in vowels and lst[i+1] == lst[i] and (i != 0 and lst[i-1] != lst[i]):
            lst[i] = vowels[lst[i]]]

    return lst

Answer: You can use a dictionary to represent the required letter mapping and zip to pair each character with its predecessor.  Using a list comprehension you can get the result in one line:
mapping  = { ("e","e"):"é", ("a","a"):"á" }

sample   = ['b','e', 'a', 'a', 'e','c','e','e','z']
accented = [ {(a,b):b,**mapping}[(a,b)] for a,b in zip([""]+sample,sample) ]

print(accented)
# ['b', 'e', 'a', 'á', 'e', 'c', 'e', 'é', 'z']

Answer: Here's a simple implementation that mutates the list.
sample = ['b', 'e', 'a', 'a', 'e']
vowel_changes = {"a": "á", "e": "é"}

# Start from index 1 and use items (i-1, i) to avoid an IndexError
for i in range(1, len(sample)):
    c, d = sample[i-1], sample[i]
    # Check if the two items are both unaccented vowels
    if all(x in vowel_changes for x in [c, d]):
        # Accent the second vowel
        sample[i] = vowel_changes[d]

print(sample)  # -> ['b', 'e', 'á', 'a', 'é']

If you wanted to put this in a function, you could use list.copy() to avoid mutating the input.


Why is it necessary to use edi constraint in this inline assembly?

Question: centos 6.5 64bit vps, 500MB ram  gcc 4.8.2
I have the following function that works only if I use edi as the constraint to hold the string pointer.  If I try to use any other register or constraintg or q etc, it segfaults.
BUT this problem only occurs when both link time optimization and o3 are used together.  If o2 it's fine.  If I don't use -flto, it's fine.  But both together then the only register I can use that doesn't crash is edi
gcc -flto
CFLAGS=-I. -flto -std=gnu11 -msse4.2 -fno-builtin-printf -Wall -Winline -Wstrict-aliasing -g -pg -O3 -lrt -lpthread

It seems like there might be some sort of register clobbering going on or something else.  I'm really at a loss to understand why and how to fix this.  Another interesting aspect is the generated assembly puts rdi into rdx before using the pointer but if I try to use either register as the input constraint... it segfaults!  If it fails under aggressive compiling options it suggests to me either the compiler is stuffing up somehow, or more likely I'm doing something wrong.
char *sse4_strCRLF(char *str)
  {
  __m128i M = _mm_set1_epi8(13);
  char *res;
  __asm__ __volatile__(
     "xor %0,%0\n\t"
     "sub $1, %1\n\t"
"1:" "sub $15,%1\n\t"
    ".align 16\n\t"
"2:" "add $16, %1\n\t"
     "pcmpistri  $0x08,(%1),%2\n\t"
     "ja 2b\n\t"
     "jnc 2f\n\t"

     "cmpb $10,1(%1,%%rcx)\n\t"
     "jne 1b\n\t"
     "add %%rcx,%1\n\t"
     "mov %1,%0\n\t"
"2:"
     :"=q"(res)
     :"edi"(str),"x"(M)  //<-- if use anything except edi, it segfaults
     :"rcx"
     );
return (char*) res;
}

Disassembled output:
00000000000002e0 <sse4_strCRLF>:
2e0:   55                      push   rbp
2e1:   48 89 e5                mov    rbp,rsp
2e4:   e8 00 00 00 00          call   2e9 <sse4_strCRLF+0x9>
2e9:   66 0f 6f 05 00 00 00 00 movdqa xmm0,[rip+0x0]          # 2f1 <sse4_strCRLF+0x11>
2f1:   48 89 fa                mov    rdx,rdi   //<--- puts rdi into rdx!
2f4:   48 31 c0                xor    rax,rax
2f7:   48 83 ea 01             sub    rdx,0x1
2fb:   48 83 ea 0f             sub    rdx,0xf
2ff:   90                      nop
300:   48 83 c2 10             add    rdx,0x10
304:   66 0f 3a 63 02 08       pcmpistri xmm0,[rdx],0x8
30a:   77 f4                   ja     300 <sse4_strCRLF+0x20>
30c:   73 0d                   jae    31b <sse4_strCRLF+0x3b>
30e:   80 7c 0a 01 0a          cmp    byte[rdx+rcx*1+0x1],0xa
313:   75 e6                   jne    2fb <sse4_strCRLF+0x1b>
315:   48 01 ca                add    rdx,rcx
318:   48 89 d0                mov    rax,rdx
31b:   5d                      pop    rbp
31c:   c3                      ret

Comment: I don't think "edi" means what you think it does.  Looking at the [docs](https://gcc.gnu.org/onlinedocs/gcc/Machine-Constraints.html) (scroll down to x86 family): "e" means "32-bit signed integer constant", "d" means the rdx register, and "i" means "An immediate integer operand".  By specifying all 3, you are allowing the compiler to choose among these 3 options, not (as I suspect you intended) selecting the edi register.
Comment: Another important point: As the [docs](https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#InputOperands) say, you must NOT modify input parameters.
Comment: I didnt know about the prohibition on modifying input parameters!  I thought that would solve it but alas no.  I changed the code so that %1 is moved into a register and all operations are done on that register but still same problem
Comment: Are you going to post the new code?
Answer: @David Wohlferd gave me the answer. It was 2 dumb mistakes I was making due to ignorance and assumptions.  The below code is modified such that the input variable char pointer is not modified by the routine.  It's copied into a register and that register is used.  Also I was mistakenly thinking I could directly specify a particular register as opposed to a b etc.
gcc still seems to be fussy about what constraints I use.  e.g. If I use a and b for res and str respectively, it compiles fine but segfaults on running.  But using S and D seems to work fine.
@David Wohlferd, I'd like to credit you as the answerer but I don't think I can do that to a comment.
char *sse4_strCRLF(char *str)
   {
   __m128i M = _mm_set1_epi8(13);
   char *res;
   __asm__ __volatile__(
     "xor %0,%0\n\t"
     "mov %1,%%rdx\n\t"
     "sub $1,%%rdx\n\t"
"1:" "sub $15,%%rdx\n\t"
     ".align 16\n\t"
"2:" "add $16, %%rdx\n\t"
     "pcmpistri  $0x08,(%%rdx),%2\n\t"
     "ja 2b\n\t"
     "jnc 2f\n\t"

     "cmpb $10,1(%%rdx,%%rcx)\n\t"
     "jne 1b\n\t"
     "add %%rcx,%%rdx\n\t"
     "mov %%rdx,%0\n\t"
"2:"
     :"=S"(res)
     :"D"(str),"x"(M)
     :"rcx","rdx"
     );
  return (char*) res;
}

Comment: Without having run it, what about something more like (sorry about the formatting): `__asm__ ("xor %[res],%[res]\n\t" "sub $1, %[str]\n" "1:\t" "sub $15,%[str]\n\t" ".align 16\n" "2:\t" "add $16, %[str]\n\t" "pcmpistri $0x08,(%[str]),%[M]\n\t" "ja 2b\n\t" "jnc 2f\n\t" "cmpb $10,1(%[str],%%rcx)\n\t" "jne 1b\n\t" "add %%rcx,%[str]\n\t" "mov %[str],%[res]\n" "2:\t" : [res] "=&r" (res), "=&r"(junk) : [str] "1" (str), [M] "x"(M) : "memory", "cc", "rcx");`
Comment: Why I think this is better: Allow compiler to select registers instead of forcing specifics, use fewer registers, memory clobber.  Oh, and you'll need to add `char *junk;`.
Comment: This could be better written.  And I worry about just assuming rcx will contain str.  And not using the memory clobber even though you are accessing memory.  But if you are happy with it...
Comment: I did read somewhere that pcmpistri returns the result in rcx, so I'm not making an assumption this time. And I don't think I'm modifying memory here so I don't think a memory clobber is a problem.If I'm mistaken, do let me know.  I'm still learning.
Comment: I'm sure there's a more elegant way to write it but it's written for performance.  Like fitting common loops into a cache line etc  If there is a way to make it faster I would love to hear it.
Comment: Doh!  My bad (about rcx).  I don't using pcmpistri and I assumed.  You aren't modifying memory, but you are reading it.  Depending on how str is being created, parts of it could still be in registers.  Faster is always a crapshoot.  Humans are bad at writing efficient asm anymore, so you always have to time it.
Comment: Tested your code and it compiles to 3 bytes shorter than mine so that's a plus.  Seems to work fine under my stress tests and testing with `__rdtscp` suggests similar or possibly very slightly better performance.  Cheers for that.  It's given me something to study and learn from.


How to solve logarithms equations

Question: I've some problems with solving of those logarithms equations:
1) $\ln(x+1)\le x$
2) $x^2-8\ln(x)=0$
Comment: By solve what do you mean? numerical or Lambert functions? or some other analysis
Comment: @Chinny84 just finding the X
Answer: For 1 consider the function
$$
f(x)=\ln(x+1)-x
$$
defined over $(-1,\infty)$, whose derivative is
$$
f'(x)=\frac{1}{x+1}-1=-\frac{x}{x+1}
$$
Can you find the absolute maximum?
For 2 consider the function
$$
g(x)=x^2-8\ln x
$$
defined over $(0,\infty)$, whose derivative is
$$
g'(x)=2x-\frac{8}{x}=\frac{2(x^2-4)}{x}
$$
Can you find the absolute minimum? What can you deduce?

How to solve logarithms equations


How do I change the global site url when serving Drupal from behind a reverse proxy?

Question: I want the site name to be the FQDN instead of the machine's internal name (ie: localhost), especially in tokens.
I'd like to be able to change the value once (like say in an env var) so it shows up any time the site's url needs to be used, say in automated emails.
aggravating factor: we're using IIS7
Answer: inside settings.php:
<code>global $base_url;
$base_url = 'http://my.domain.name';
</code>


Why is my select criteria not allowing me to append HTML "td" attributes to a D3 generated HTML Table?

Question: I'm trying to construct a sortable table using D3.  In the debugger, I can watch the DOM objects be constructed, right down to the table body records ().  However, when I attempt to use a for loop to append table data elements () to each table record, nothing is appending.  I can't tell if the problem is my select approach, the syntax for the append, or the for look.
Any thoughts?
The original code is at: http://bl.ocks.org/Guerino1/8203388
My code is:
//THIS IS THE LAST THING THAT SEEMS TO WORK FINE...

    // Select the HTML tbody element and append an HTML tr element to create body records
    var densityBody = d3.selectAll("#density_body").selectAll("tbody.tr")
      .data(totalsArray)
      .enter()
      .append("tr")
        .attr("id", function(d, i) { return "density_body_record-" + i; })
        .on("mouseover", bodyRecordMouseOver)
        .on("mouseout", bodyRecordMouseOut);

// THINGS START TO BREAK INSIDE THIS FOR LOOP...

    for(var z = 0; z < totalsArray.length; z++){
      // Select the HTML tr elements in the tbody and append HTML td elements
      var dataSelectString = "#density_body_record-" + z;
      var densityBodyDataRecords = d3.selectAll(dataSelectString).selectAll("tr.td") // HERE!!!
        .data(totalsArray[z])
        .enter()
        .append("td")
          // Append text to each column
          .text(function(d) { return d;

          });

Thanks for any help you can offer.
Answer: For starters, you're using an invalid selector: "tr.td" will select table row (
) elements that have a class of "td".  What you intended was "tr > td", or table data () elements that are direct children of table row elements.  Same problem in the first section -- you're adding rows to a selection that D3 thinks is table body elements.
Then, as @Barnab said, there is the the fact that you're joining one object (containing the whole row's data) to your selection, not an array of individual data elements.  His suggestion works, but at the expense of hard-coding in all the column names in an order that isn't accessible outside that one line of code.  If you add data categories or delete them, you have to go monkeying around with this code and with any code for creating column header labels.
One alternative would be to use the d3.entries(object) method, which will return an array, where each element of the array is an object representing a key and value pair from the original object.  However, the method doesn't guarantee the order that the key-value pairs will appear in the array it creates, so you would have to sort the array returned by d3.entries() based on the key value and a custom sort function.
Another approach is to define the order of columns as it's own array (possibly read in or calculated from the data source), and then use that to create the array of table-cell data objects from the row data objects.
I've got an example at the end, but first you'll have to read up on using nested selections for table rows and then table cells, as a much more elegant alternative to using a for loop to scan through the rows.
The trick is how to go from having one data object for the entire row to having an array of objects for each column, when you want those data objects to be based on both the row data and the column data array.
I created this example for someone else, but it shows how to use a customizable function to combine row and column data with nested selections:
http://fiddle.jshell.net/un58H/
(The source of the function I was trying to explain.)
Except in your case, you wouldn't want to create a new object from the row and column data, but to use the column data to access the associated value from the row data object.  Like this:
http://fiddle.jshell.net/un58H/2/
Note a few other things about the fiddle:

by declaring an array of column keys, you can use that array as the data to populate the column headers;
the order in which data is declared in the object is irrelevant; the final order is determined by the column names array;
if some rows don't have values for a particular column, you can leave them out of the data object and just check for null values when using the data object in each cell.
Comment: Great answer that gives you big picture and helps navigate through the maze of objects and table rows, columns, and cells.
Comment: Thanks Amelia.  Quick question.  You mentioned that my selection was wrong because "tr.td" selects all table records and elements that have a "class" value of "td".  May I ask what the right syntax is to select all table data  elements within a table record ?  -- thanks.
Comment: `"tr > td"` for direct children, `"tr td"` for all descendents.  See: https://developer.mozilla.org/en-US/docs/Web/Guide/CSS/Getting_started/Selectors
Comment: I should add a warning that "tr > td" finds *all* table data elements that have a table row as a parent, regardless of whether that table row is part of your current selection; this can be an issue if you have nested tables (or nested lists, nested groups, etc.)  In that case, you'd need classes as well to distinguish between your nesting levels.  See [this discussion](https://github.com/mbostock/d3/pull/1618#issuecomment-28076822) on the d3 github.
Answer: Try this code for the second .data call :
.data([totalsArray[z].Name, totalsArray[z].Relationships, totalsArray[z].Percentage, totalsArray[z].Density])

With your code totalsArray[z] is an object and I don't see how D3 can iterate over its attributes. In my implementation the return d part in .text will iterate over the array passed in .data.


CentOS: command not linked/working

Question: Recently installed CentOS and many commands are not working:
<code>[root@zrebirth zeno]# quotacheck
bash: quotacheck: command not found
[root@zrebirth zeno]# adduser
bash: adduser: command not found
[root@zrebirth zeno]# warnquota
bash: warnquota: command not found
[root@zrebirth zeno]# edquota
bash: edquota: command not found
</code>
But they all exist, so they must not be linked:
<code>/sbin/quotacheck
/usr/sbin/adduser
/usr/sbin/warnquota
/usr/sbin/edquota
</code>
I'm sure there are more. Is there a way I can quickly fix all of these (even the ones I don't know) without doing each one manually?
Answer: It appears that that you just need to modify your $PATH. The following article goes over how to do that.
Cyberciti
On my default CentOS install, it only has /bin as my path. Therefore if I want to type any other commands that are outside of the /bin path, I need to either fully enter the command such as "/sbin/command" or add those paths as well.
<code>[blah~]$ cat .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export PATH
</code>
Comment: It's completely up to you. In my opinion leaving these paths out is another layer of security. If the user wants to use a command such as adduser or lets use the example rm. If they have to specifically put in /sbin/rm or something to that affect it makes them more aware of what they are doing. However there is also a convenience factor depending on how much you use them. I personally just type in all of the paths I need.
Comment: How do I apply those $PATH settings to any and every once at once?
Comment: You can modify /etc/profile.

PATH= $PATH:/sbin:/bin:/usr/local/bin:/usr/local/sbin/
Comment: Thanks. Should I just give every user those path values, even though they can't use adduser etc?
Answer: From centos documentation(http://wiki.centos.org/TipsAndTricks/BecomingRoot):

Commands for regular users are mostly located in /usr/local/bin, /usr/bin, and /bin. However, root commands are mostly located in /usr/local/sbin, /usr/sbin, and /sbin and root's PATH reflects this.When you become root by using 'su -', you also adopt root's PATH whereas using just 'su' retains the original users PATH, hence why becoming root using just 'su' and trying to run a command located in /usr/local/sbin, /usr/sbin, or /sbin results in a 'command not found' error. For a more detailed explanation, see the bash manual page (man bash), particularly the section on INVOCATION and login shells.

You need to use "su -" and not "su".
Comment: Hm but I was logging into SSH directly with root.

CentOS: command not linked/working


Looking for a strategy to optimize a big body field on a node

Question: I'm using Drupal 8.4.x
I have just one node with a big text on its body field (about 2.5 MB). Looking a strategy to optimize:

Page load. Does exist any "content lazy loading" module? (I can see one for D7 but without a stable release).
I'll like to create revisions on this node. I'm gonna use Diff module to see changes and also to enable visitors to see those changes. This a consuming task on a big node and not sure how to cache/optimize those Diffs.

Note: I've tought about spliting the content in different nodes (like a book) and then use something like Views Infinite Scroll but I'll find hard to implement 2 with this aproach.
Answer: Using the Paragraphs module is probably the current go-to solution for a more manageable setup than a large body field. Functionally, it's right in between a single text field and completely separate nodes. It enables you to create more manageable chunks of content that replace the traditional body field. Each chunk can be simple text, or more complex things. Each "paragraph" is an entity with fields, and the node references those entities. When editing, it feels like a more focused interface. The content creator stays on the node, so the complexity is hidden in the background.
Another strength it has is good support for revisions. In fact, it requires the Entity Reference Revisions module, and its project page suggests using the Diff module.
Comment: Thanks for the suggestion, I'm sure it will be clear for the editor/content creator and great to know that it supports revisions and diff. How to handle 1.? I'll suppose that is possible to create a View of paragraphs and use Views infite scroll in order to lazy load the content and increase page load speed?
Comment: @Yaazkal I think you're right, that using a view to lazy-load the paragraph items will be what is needed. The Views Infinite Scroll (https://www.drupal.org/project/views_infinite_scroll) module can take care of the lazy loading. It's well supported and has a stable release, currently at 1.5. Everything I see related to lazy loading is for images, videos, iframes, etc. Not for arbitrary content. It would be cool to have a lazy-load paragraphs field formatter, but I don't think one exists yet.
Comment: thanks so much for the suggestion. Paragraphs made the administration of the big content pretty easy; it is now much faster for the editor to manage the content of the node. By now it's an easy fix and I'll look forward how to make it interesting from the user perspective. Happy new year!


Do finitely generated ideals have a "dimension"?

Question: In our algebra class, we recently talked briefly about finitely generated modules, and the instructor referred to how even if a $R$-module admits a basis, we can easily have that different bases of the module are not in bijection with each other. This brought to mind a different question: For a ring $R$ and a finitely generated ideal $I$, need all minimal generators of $I$ be in bijection? That is, if $S, T \subseteq I$ are two finite sets such that $I = (S) = (T)$, and moreover such that any proper subset of $S$ or $T$ will not generate $I$ (i.e. if $S$ and $T$ are minimal), then does there exist a bijection between $S$ and $T$? I raised the question with a friend in my class who's often interested in these kinds of questions, and he brought up that any (left) ideal $I$ would carry a natural structure as a (left) $R$-module, so if there existed a finite ideal $I$ that would induce a module with non-bijective bases, then the answer would be to the negative, but I have no intuition for whether that would be possible. If the answer is in general to the negative, then is there a natural assumption that could be made about $R$ so that the conjecture does hold?
Comment: It's possible (although I wouldn't really call it "easy") for two finitely generated free modules over a ring to have different size bases. Rings which do this are said to lack the Invariant Basis Number property. I only know two families of examples for this.
Answer:
For a ring $R$ and a finitely generated ideal $I$, need all minimal generators of $I$ be in bijection? That is, if $S, T \subseteq I$ are two finite sets such that $I = (S) = (T)$, and moreover such that any proper subset of $S$ or $T$ will not generate $I$ (i.e. if $S$ and $T$ are minimal), then does there exist a bijection between $S$ and $T$?

This isn't even true for $\mathbb Z$. Consider $(1)$ and $(2,3)$.
Comment: It's as easy as $1,2,3$ `¯\_(ツ)_/¯`


Probability - Randomizing seats

Question: Really basic probability question that I want answered,
There is a class room with 3 students and 3 seats. Each student selects a seat and sits on it. The next day, the instructor randomly picks a seat for each student. What is the probability that each student is assigned a different seat from the one they sat on yesterday? (i.e: Everyone is in a different seat).
The way I approached the problem was to find the probability of the students sitting in the same seat. If there are 3 students, A, B, C, and the instructor is assigning them to the seats one by one, then
P(A is assigned a different seat than yesterday) = 2/3
P(B is assigned a different seat than yesterday) = 1/2
P(C is assigned a different seat than yesterday) = 1
P(Everyone is assigned a different seat) = 2/3 * 1/2 * 1 = 1/3
Is this the correct way of doing this?
Answer: You have to be careful, because the parts aren't independent - certainly the probability that A gets a different seat is $\frac{1}{3}$. But there are two possibilities - either he gets B's seat (in which case B has a 100% chance of getting a different seat, and C has a 50% chance) or he get's C's seat (in which case B has a 50% chance of getting a different seat, while C has a 100% chance).
Now in this situation it still works, because you can choose to look at the person whose seat A didn't get, and you get the answer you've found. But for larger groups, this won't work - for example, with 4 people you can have it so that there's a loop of people sitting in each other's seats (e.g. A sits in B's seat, B sits in C's, C sits in D's and D sits in A's), or you can have it so that two pairs of people each swap seats (so A and B swap, and C and D swap), and a simple approach to the probability won't work.
In fact, there's a bit of a trick to calculating it, and the details are explained in the Wikipedia article on Derangement, which is what this is usually referred to as.
Answer: Your method fortuitously gives the correct value, but is not actually correct.   Student $A$ can be in the wrong seat by being in either seat B or C.   If A is in seat B then B may equally likely be in seat A or C, otherwise if A is in seat C then B may equally likely be in seat A or B.
$\begin{align}\mathsf P(A\in\{B_0, C_0\}) &= 2/3\\ \mathsf P(B\in\{A_0,C_0\}\mid A\in\{B_0, C_0\}) &= {\mathsf P(B\in\{A_0,C_0\}\mid A\in\{B_0\})\mathsf P(A\in\{B_0\}\mid A\in\{B_0,C_0\}) \\ + \mathsf P(B\in\{A_0,C_0\}\mid A\in\{C_0\})\mathsf P(A\in\{C_0\}\mid A\in\{B_0,C_0\}) } \\ & = 1\cdot \tfrac 12+\tfrac 12\cdot \tfrac 12\\ & = \tfrac 34 \end{align}$
And so on.
A simpler method:

There are several equally likely ways to arrange three students, of which, only some have no student in their original position.   Let us have a look.
$$\left\{\rm \underset\checkmark A\underset\checkmark B\underset\checkmark C, \underset\checkmark A\underset\times C\underset\times B, \underset\times B\underset\times A\underset\checkmark C, \underset\times B\underset\times C\underset\times A, \underset\times C\underset\times A\underset\times B, \underset\times C\underset\checkmark B\underset\times A\right\}$$

Probability - Randomizing seats


Overfitting detecting in Binary Classification Problem

Question: I have a binary classificaiton problem with positive label proportion is very small, 1% at most of the cases. So, recall, precision and f score for the positive label is used to evauluate my classifiers.
However, the commom overfitting detecting method uses the metric MSE, accuracy  to detect the situtaion. MSE, accuracy is useless in my situation.
So, is there any methods to detecting overfitting in the small proporation positive label binary classification problem?
Comment: Do you have enough positive label results to do cross validation?
Comment: I have 10,000,000+ samples, so 1% mean 100,000+ positive labels, I think  it enougth
Answer: The "gold standard" is log loss ("crossentropy loss" in some circles).
$$
\text{Log Loss}\\
L(y,\hat p)=
-\dfrac{1}{N}\sum\left(
y_i\log(\hat p_i) +
(1-y_i)\log(1-\hat p_i)
\right)
$$
The $\hat p=(\hat p_1,\dots ,\hat p_N)$ is the vector of predicted probabilities, not the classifications based on a threshold.
Like the log loss, Brier score is a strictly proper scoring rule that assesses the predicted probabilities instead of categorical classifications, and it is equivalent to MSE. Thus, I disagree that MSE is useless in this setting.
Despite the issues with accuracy, precision, recall, and F-score, if you have reason to believe those to be the metrics of interest, those could be viable. If you dislike such metrics in this situation because basically every prediction is the same category (with $99$:$1$ imbalance, every predicted probability might be below the software-default cutoff of $0.5$, and such predictions would not be surprising), then these metrics are revealing issues in your predictions that you want to catch and correct before you use the model for real.


What happens if I subscribe to the same topic multiple times? (Python, Google Pubsub)

Question: If I have the following code, will anything bad happen? Will it try to create new subscriptions? Is subscribe an idempotent operation?
subscriber = pubsub_v1.SubscriberClient()

def f(msg):
    print(msg.data)
    print(msg)
    msg.ack()

def create_subscriptions():
  restults = [] # some sql query
  for result in results:
    path = self.subscriber.subscription_path('PROJECT', result)
    subscriber.subscribe(self.path, callback=f)

while True:
    time.sleep(60)
    create_subscriptions()

I need to be able to update my subscriptions based on when people create new ones. Is there any problem with this approach?
Comment: The doc for this API doesn't say if subscribing multiple times to a particular path within a single program is idempotent?  Can you just try it...subscribe to a topic multiple times and then see if you get duplicate messages?
Comment: I mean, I tried it. I don't get the message twice or anything, but not sure.
Answer: You should avoid repeatedly calling "subscribe" for the same subscription -- even though you will most likely not increase the number of duplicate messages that are delivered, you would create multiple instances of the receiving infrastructure. This is both inefficient, and defeats some of the flow control properties that Pub/Sub provides, since these are only computed per instance of the subscriber; i.e. it can cause your subscriber job to run out of memory and fail, for example.
Instead, I would suggest keeping track of which subscribers you've already created. Note that the "subscribe" method returns a future that you can use for this purpose, or to cancel the message receiving when necessary. You can find more details on the documentation.
Comment: "Receiving infrastructure" in this context means the classes in the client library that manage connections to the server (and the connections themselves) and manage the lifetime of messages received by the subscriber client.
Comment: What do you mean by "receiving infrastructure"? I feel that's the part from the documentation that's missing. For example, it seems that while using this in a Django project, a new database connection is created for every single subscription created which is unexpected.


postfix - dns error - can't resolve my A or MX record

Question: I've installed postfix on my arch linux pc. I use systemd-resolved and systemd-networkd to establish a network connection.
I want to use postfix as a satellite system to send some mails if my raid systems is broken.
So, when I want to send some test mails with echo "Body" | mail -s "Header" email@example.com postfix write in my logs, that he can't resolve my domain after a A or MX record

Okt 21 22:16:54 markus-pc postfix/error[17574]: F1C6E2E0C10: to=, orig_to=, relay=none, delay=360970, delays=360969/0.57/0/0.02, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=smtp1.example.com type=AAAA: Host not found, try again)
Okt 21 22:16:54 markus-pc postfix/error[17569]: 0448E2E0C0E: to=, relay=none, delay=360970, delays=360969/0.57/0/0.02, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=smtp1.example.com type=AAAA: Host not found, try again)
Okt 21 22:16:54 markus-pc postfix/error[17577]: F420B2E0B46: to=, relay=none, delay=360970, delays=360969/0.57/0/0.02, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=smtp1.example.com type=AAAA: Host not found, try again)
Omt 21 22:16:54 markus-pc postfix/error[17572]: F257B2E0C12: to=, orig_to=, relay=none, delay=360970, delays=360969/0.57/0/0.02, dsn=4.4.3, status=deferred (delivery temporarily suspended: Host or domain name not found. Name service error for name=smtp1.example.com type=AAAA: Host not found, try again)

But when I tried it with dig, I get a valid response.
How can I fix this dns error, that my local postfix installation can find my mailserver and can send him some mails?
Volker
Comment: At first glance it appears that your system is only trying to do IPv6 lookups, indicated by the  `AAAA` records and the domain may not be configured for those. When you use dig to test  you may get to see IPv4 A records instead, which probably are the default? If it is your own domain, adding  the actual ipv6 records may be a solution.
Comment: Yes, it's my domain and my server has an A, AAAA and MX record. My mailserver runs fine since 2 years. I can't understand why my local postfix installation on my arch linux pc can't lookup the records.
Comment: Have you tried using `traceroute` or `mtr` to verify that you actually have dual stack connectivity to the internet?
Comment: Okay, I tried it with `traceroute` and `mtr`. It's a little bit crazy because with traceroute he want to resolve my valid ipv6 address but he goes to a wrong ipv6 address. The resolved address and the target address are totally different.
Comment: Okay, I have fixed this error. Now it's the right ipv6 address. I such tried to change `internet_protocols` to `ipv4` in postfix' main.cf, I get the same error.
Answer: Have you tried to double check your

/etc/resolv.conf

and

/var/spool/postfix/etc/resolv.conf

to find any issues about name resolution?
The AAAA record refers to IPV6 addresses, maybe you just ran into a misconfiguration of your DNS/network!
Comment: So, I have this two records in my resolv.conf
nameserver 192.168.178.1
nameserver fd00::ca0e:14ff:febf:46c9

I can't find a resolve.conf file under /var/spool/postfix/etc. I don't have the directory /etc/ in /var/spool/postfix. Must I create it and symlink it to the origin resolv.conf?
Comment: @VolkerRaschek When you see an IP address starting with `fd00::` there is 99.99999% chance it was configured by a person who did not know what they were doing. Where did that address come from? It may be that your router is misbehaved or that your ISP is sending it a bad configuration.
Comment: The address with `fd00::` come from systemd-resolved. I've added now manually the ipv6 address from my router in `etc/systemd/resolv.conf`. It start with `2003:a:c7d:`.


Changing files on the fly in Linux (writing to input file on a pipe)

Question: How to change a file on the fly along a pipe?
I'm probably looking for a way to buffer a file at the beginning of a pipe, which in contrast to this:
    cat foo.txt | grep bar > foo.txt

... would preserve the input data from destruction by the pipe itself. Is there such a buffer in stock?
Answer: Try using sponge from moreutils like this:
sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

It collects the whole input before writing to it's output.
Comment: I never knew about sponge, that's a fantasic tool that solves an annoying problem
Answer: I would guess sed still might create the temp file, but the following might do what you want?  (Using strace on this might show you if sed creates a temp file or not).
sed -i '/bar/!d' foo.txt

The exclamation inverts the match, d is for delete, so this removes all lines that don't have bar in them.
Answer: Use >> to preserve the contents.

cat foo.txt | grep bar >> foo.txt

Now that will append to the file.
AFAIK, there is no direct way to prepend data to a file in shell. If you want to prepend you may need to use a temporary file in between.
Answer: Depending on how complex your command line is, you may get mileage out of
cat foo.txt | grep bar | tee -a foo.txt

Answer: The best way to do this is to remember how files work in unix -- the file exists as long as there is a link to the file (directory or process opening the file).
So, open the file, delete that directory entry, then run your process that writes to a new directory entry with the same name but linked to a different inode.
Lastly, the nice thing about this is that it works for any sort of pipeline operation, not just things that can do the temporary file buffer hack.
{ rm file ; sed -e s/this/that/g > file ; } < file

A word of warning -- this may cause disaster if you must have your rewrite be an atomic operation.  unix doesn't offer clean ways of locking files, especially not at shell level.  This was an issue in the dark ages if you were doing something like editing a password file on a busy system that was running something like NIS.  Any time more than one process is reading / writing to a file, be very careful if your system is busy or important.
The only operations that are 100% certain to be atomic are directory entry manipulations -- rm / ln / mv (on the same filesystem).
So now things get longer and uglier..
The math part of this requires a posix shell or similarly extended bourne shell
c=0
    while
      f=$SECONDS
      test -f file.$c.tmp
    do
      c=$(($c+1))
    done

  grep stuff < file > file.$c.tmp && mv file.$c.tmp file

Answer: Ooops, UUOC, useless use of cat :) you don't need cat there :
grep something somefile > someotherfile

As sekenre said, there is sponge. Or maybe you may want to try perl inline edition ( -i switch ) :

print only matching output:

perl -ni -e 'm/foo/g and print ' bar.txt

replace "foo" with "bar" in the file :

perl -pi -e 's/foo/bar/g' bar.txt


Does Hartshorne assume curves are irreducible when proving that pullback along a finite morphism of nonsingular curves preserves linear equivalence?

Question: The question is how it can be proved the statement in the title:
Let $f: X\to Y$ be a finite morphism of nonsingular curves.
Then the pullback morphism $f^*:\operatorname{Div} Y \to \operatorname{Div} X$ preserves linear equivalence.
I have found this very same topic answered here:
Let $f: X \to Y$ be a finite morphism of nonsingular curves. Then $f^*: \operatorname{Div} Y \to \operatorname{Div} X$ preserves linear equivalence.
In my opinion the answer linked above (and consequently the proof) is correct as long as one assumes that at least the curve $Y$ is irreducibile. So my doubt is: is it so obvious that we are dealing with irreducibile curves at the point that the book doesn't even specify it?  In such case why?
Or am I wrong about the fact that the proof assumes the irreducibility of   ? In such case I am not able to justify this part of the proof:
"[...] On the other hand, for each $Q_i$
we can find an open set of
on which $g=u\cdot g_i^{v_{Q_i(g)}}$ [...] "
since I cannot understand why it must exist an open neighborhood in which both $g$ and $g_i$ are regular.
I hope my doubt is clear, thanks in advance.
Comment: Hi, welcome to MSE. Please use MathJax for all math in your post - I noticed you had used it for some of the math, but not all of it, so I took the liberty of fixing it this time. I also adjusted your title to try and get a little closer to what you were asking about (your confusion about whether irreducibility was assumed). Hopefully the title change is better in your eyes - feel free to keep improving it if you like :)
Comment: Thanks you! @KReiser
Answer: Yes, we are dealing with irreducible curves at that point in the book. From just before remark II.4.10.1 (page 105 in my copy):

Definition. An abstract variety is an integral separated scheme of finite type over an algebraically closed field $k$. If it is proper over $k$, we will say it is complete.
Remark 4.10.1. From now on we will use the word "variety" to mean "abstract variety" in the sense just defined. We will identify the varieties of chapter I with their associated schemes, and refer to them as quasi-projective varieties. We will use the words "curve", "surface", "three-fold", etc., to mean an abstract variety of dimension 1, 2, 3, etc.
Comment: Thanks you for the answer, unfortunately I'm still missing the point. I was aware of the fact that we were dealing with abstract varieties, and now I assume that this implies that they are irreducibile. I don't understand why.
Comment: @ChiaraM an integral scheme is irreducible and reduced (see for instance [here](https://math.stackexchange.com/questions/2082845/show-a-scheme-is-integral-if-and-only-if-it-is-irreducible-and-reduced)). Integral is a part of the definition of an abstract variety.
Comment: really, really thank you, now it is clear.
Comment: @ChiaraM you're welcome. If this resolves your question, you may wish to consider [accepting this answer](https://math.stackexchange.com/help/someone-answers#:~:text=Choose%20one%20answer%20that%20you,the%20answer%2C%20at%20any%20time._) by clicking the checkmark.


Library dependancies not loading

Question: The dependency core/jquery.ui is not loading into the theme. jQuery is loaded as it is included by default but jquery.ui refuses to load.

#theme.libraries.yml
popup:
  version: 1.x
  js:
    js/popup.js: {}
  dependancies:
    - core/jquery
    - core/jquery.ui

#theme.info.yml

libraries:
  - theme/popup

I have also tried attaching the library using the code below, but still no luck.

// theme.theme
function theme_page_attachments_alter(&$page) {
  $route = \Drupal::routeMatch()->getRouteObject();
  $is_admin = \Drupal::service('router.admin_context')->isAdminRoute($route);
  if(!$is_admin) {
    $page['#attached']['library'][] = 'theme/popup';
  }
}

I have seen multiple references to core/jquery.ui throughout drupals files. So am a little confused.
I feel like i must be missing something obvious.
jQuery UI will load in fine if you are logged in, but again i believe this is included by default for admin users.
Answer: For those that are trying to use the built in jquery ui you must include -core/drupal.dialog as a dependency in your theme.libraries.yml for it to work
Answer: You made a typo in your theme.libraries.yml, dependancies should be dependencies.

Library dependancies not loading


How to perform a random-effects model to include more than one category?

Question: Sample Panel Data

Year
Firm_ID
Region
Industry
ROE
ROA
Tobin_Q
ESG
Leverage
Age
Size

2012
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
12.19
6.22
1.17
51.24
1.45
1.58
6.51

2013
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
11.88
6.06
1.12
55.37
1.47
1.59
6.56

2014
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
11.82
5.69
1.17
57.85
1.51
1.6
6.63

2015
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
11.3
5.05
1.05
48.76
1.5
1.61
6.7

2016
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
13.22
5.39
1.13
52.48
1.48
1.62
6.78

2017
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
12.07
4.56
1.18
41.74
1.44
1.63
6.85

2018
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
12.95
4.72
1.28
53.31
1.43
1.64
6.91

2019
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
11.63
4.35
1.42
42.56
1.46
1.65
7

2020
1
Asia/Pacific Rim
Oil_and_Gas_Refining_and_Marketing
9.37
3.63
1.22
41.74
1.47
1.66
7.07

2012
2
EMEA
Electric_Utilities
14.4
5.95
1.1
77.27
1.55
1.11
4.39

2013
2
EMEA
Electric_Utilities
12
5.03
1.2
72.31
1.59
1.15
4.37

2014
2
EMEA
Electric_Utilities
30.2
14.1
1.24
66.53
1.51
1.18
4.33

2015
2
EMEA
Electric_Utilities
33.56
18.75
0.94
66.53
1.42
1.2
4.36

2016
2
EMEA
Electric_Utilities
3.65
2.22
0.98
67.77
1.37
1.23
4.34

2017
2
EMEA
Electric_Utilities
6.53
3.96
1.07
64.46
1.35
1.26
4.34

2018
2
EMEA
Electric_Utilities
6.77
3.82
1.23
65.7
1.43
1.28
4.35

2019
2
EMEA
Electric_Utilities
11.94
6.48
1.28
66.12
1.46
1.3
4.37

2020
2
EMEA
Electric_Utilities
14.06
4.49
1.08
57.02
1.27
1.32
4.76

2012
3
Americas
Oil_and_Gas_Storage_and_Transportation
19.13
6.91
1.89
11.57
1.65
1.11
4.56

2013
3
Americas
Oil_and_Gas_Storage_and_Transportation
18.29
6.83
2.17
12.81
1.64
1.15
4.6

2014
3
Americas
Oil_and_Gas_Storage_and_Transportation
16.75
6.38
2.1
12.81
1.66
1.18
4.67

2015
3
Americas
Oil_and_Gas_Storage_and_Transportation
13.15
5.25
1.64
14.46
1.66
1.2
4.69

2016
3
Americas
Oil_and_Gas_Storage_and_Transportation
11.87
4.98
1.67
19.01
1.66
1.23
4.72

2017
3
Americas
Oil_and_Gas_Storage_and_Transportation
12.55
5.25
1.64
19.01
1.65
1.26
4.74

2018
3
Americas
Oil_and_Gas_Storage_and_Transportation
17.98
7.49
1.52
28.51
1.66
1.28
4.76

2019
3
Americas
Oil_and_Gas_Storage_and_Transportation
18.89
7.74
1.6
34.3
1.66
1.3
4.79

2020
3
Americas
Oil_and_Gas_Storage_and_Transportation
15.37
6
1.29
33.88
1.68
1.32
4.81

For example, my fixed effects model is using the code:
dummyvar = lm(ROE ~ ESG + Leverage + Age +
    Size + factor(Industry) + factor(Region),
    data=Updated_Age_Data)

Whereby the industry and regions are the dummy variables. ROE is the dependent, ESG is my independent, and leverage, age, and size are my control variables.
How do I apply the same concept to a random effects model?
Would either of these be the correct code? As I want the output to show the significance of ROE based on region and industry type.
random1  <- plm(ROE ~ ESG + Leverage + Age +
    Size + Region + Industry,
    index=c("Firm_ID", "Year"), model="random",
    data=Updated_Age_Data)

OR
random2  <- plm(ROE ~ ESG + Leverage + Age +
    Size + Industry + Region,
    data=Updated_Age_Data, model="random")

List of acronyms/explanations:

ROE = Return on Equity
ESG - Proxy for corporate social responsbility
ROE is my dependent variable
ESG is my independent variable
Age, Size and leverage are my control variables
Comment: Welcome to stack! Please @Emma, try to give a toy dataset to provide a minimal reproducible example (see [mcve]) so that people here can gives you answer that can be tested and reproduced by otehrs
Comment: @denis Hi, apologies for that. I just tried to include a sample in, my unsure how to format it nicely in the snippet code as per the link you sent.
Comment: try to use `dput` on your dataset
Answer: As you do not give a dataset, I will try to give you some guidance without having the possibility to test.
I think you need to have a look at the lme4 and lmerTest packages for the mixed effect linear regressions.
From what I understand, Industry and Region are your cluster variables. Your regression with random intercept would look like
lmer(ROE~ESG+Leverage+Age+Size+(1|Industry)+(1|Region),
     data = Updated_Age_Data)

For random slopes of your independent variable:
lmer(ROE~ESG+Leverage+Age+Size+(1 + ESG|Industry) + (1 + ESG|Region),
     data = Updated_Age_Data)

You could also want to use GEE from the gee package or geepack package, but these do not allow for 3 levels random models. If you want just 2 level random variable, gee would do the trick too (without estimating the variance of intercepts and slopes)
Comment: Hi, apologies if I'm wrong. But I did some tests which told me to use the random-effect model. Would the mixed effect linear regression model not be something entirely different? Thank you for the reply nevertheless!
Comment: mixed-effect and random effect are two ways of naming the same thing actually, see https://en.wikipedia.org/wiki/Mixed_model
Comment: Sorry, I should have worded it better. I'm looking to just get the results using just the random-effects model, rather than mixing the two! The fixed-effects code was there just for example haha.
Comment: If you found this answer helpful, then please consider
[upvoting](https://stats.stackexchange.com/help/why-vote)
and/or
[accepting](https://stats.stackexchange.com/help/accepted-answer)
it.