Need help understanding what a natural log transformation is actually doing and why specific transformations are required for linear regression

Question: I'm taking an online "Intro to AI" course for which I'm doing some azure machine learning labs.  This course is largely about how to apply azure ML solutions and, while there is an "essential math for ML module", it doesn't do any sort of deep dive into the math/statistics. But I want to understand what's happening mathematically to a basic level. (I've taken an intro to statistics, that's it so far.)  Below are some details about some specific things I'd like to understand more, and my specific questions.
At a particular point in the "training a classification model" lab, there is this text: "The distribution of the Age column in the diabetes.csv dataset is skewed because most patients are in the youngest age bracket. Creating a version of this feature that uses a natural log transformation can help create a more linear relationship between Age and other features, and improve the ability to predict the Diabetic label. This kind of feature engineering as it's called is common in machine learning data preparation."
Then the instructions show you how to use Azure ML Studio to apply the natural log operation to the age column in the data set:

And after doing that, the data includes the original/raw age data and the transformed age data:
Question #1: What is that transformation actually doing? I don't mean the nitty gritty math, but what is it doing conceptually?
Question #2: The next general question is around why is a transformation necessary.  On that, I did some research and found this article (https://www.r-statistics.com/2013/05/log-transformations-for-skewed-and-wide-distributions-from-practical-data-science-with-r/) that describes which type of log transformations to use in a few scenarios.  Here's a snip of text from the article:
"The need for data transformation can depend on the modeling method that you plan to use. For linear and logistic regression, for example, you ideally want to make sure that the relationship between input variables and output variables is approximately linear, that the input variables are approximately normal in distribution, and that the output variable is constant variance (that is, the variance of the output variable is independent of the input variables). You may need to transform some of your input variables to better meet these assumptions."
I don't understand why any of that is necessary.  I've broken my question down:
For linear and logistic regression, for example, you ideally want to make sure that:
the relationship between input variables and output variables is approximately linear – why?
the input variables are approximately normal in distribution- why?
the output variable is constant variance (that is, the variance of the output variable is independent of the input variables – why?
Could someone help with information for those whys, or point me to material that would help?
Comment: One intuitive way to think about what the log transformation is doing is to look at the distribution of Age - it's right skewed, so many of the ages are 'clumped together' at the lower end of the age scale, while fewer ages are spread out at the higher end of the age scale. When you log transform the distribution of Age, you 'stretch out' the clumped values to make it easier to see what is going on.  In other words, the log transformed age values at the lower end of the scale are better separated from each other, instead of crowding each other.
Comment: To give you a concrete example, let's say that all predictor variables in your model except for Age are continuous and have roughly symmetric distributions.  You fit a linear regression model including all the predictors as they are (i.e., untransformed)  to your data and look at the distribution of the model residuals. If this distribution reflects the skeweness of the Age distribution, then you'd be entitled to think that log-transforming Age prior to including it in your model would resolve the skewness of the residual distribution.
Comment: I hate it when questions such as yours are put on hold or marked as duplicate, as we cannot answer them properly.  So we're reduced to having to just add comments instead of answers!
Comment: Look at the high spike in the distribution of Age (a clear sign of clumping or crowdedness). That spike was attenuated in the distribution of the log-transformed Age.
Comment: Strictly speaking, transforming Age before including it in your model is not necessary. In fact, you could fit your model using the original Age variable first and then check the model diagnostics for any evidence that a transformation of Age might improve the model fit. People tend to transform predictor variables such as Age when they have skewed distrubutions because they think that would lead to an improved model. Whether or not that is the case is debatable. Additionally, transforming variables can complicate the interpretation of the model.
Comment: Both linear regression and logistic regression can accommodate nonlinear effects (e.g., a quadratic effect of Age).  Such effects can easily be accommodated in your model by using polynomial regression terms (e.g., include both Age and Age^2 in your model, where Age^2 is the squared value of Age).  However, you might want to consider including orthogonal polynomial regression terms to avoid collinearity between predictors (e.g., Age and Age^2 may be highly correlated, which would create problems with model interpretation).
Comment: Another way to incorporate nonlinear effects in linear and logistic regression models is to use so-called GAM modelling (generalized additive modelling).  So it is a myth that these models can only include linear effects! In reality, these models should include whatever effects are best supported by the data. If your model best supports a linear effect of Age on the response variable, all else being the same, then you should proceed accordingly. If your your model best supports a quadratic effect of Age on the response variable, all else being the same, then go for that.
Comment: Perhaps this will help https://stats.stackexchange.com/questions/18844/when-and-why-should-you-take-the-log-of-a-distribution-of-numbers .
Comment: You ask a lot of good questions.  Most of them have been answered in more focused threads that duplicate parts of yours.  But if any remain that are not answered there, please consider paring down your post to focus on those that still need replies. One keyword for a search is "Box-Cox."
Comment: I know some people get bugged by "thanks" and the like, but thank you so much for this, [Isabella Ghement](https://stats.stackexchange.com/users/200312/isabella-ghement).
Answer: There's a lot here to break down.  I hate to say it, but some of the advice in your course is quite misguided and wrong.

What is that transformation actually doing? I don't mean the nitty gritty math, but what is it doing conceptually?

The math here is pretty simple.  You have a bunch of measurements of people's age that you would like to use as a feature in predicting some other measurement (looks like the probability of something happening).  You're simply creating a new feature which is the logarithm of the original feature.  I'll explain why you would want to do this below.

For linear and logistic regression, for example, you ideally want to make sure that: the relationship between input variables and output variables is approximately linear – why?

This is a structural assumption of the linear and logistic regression models.  I'll focus on linear regression, because its a bit simpler, but the same thing holds for logistic regression.
The linear regression model makes predictions by building a formula based on the data you feed into the algorithm.  All prediction models work this way, but linear regression is distinguished by building the simplest possible formula.  If $y$ is the thing you are trying to predict, and $x_1, x_2, \ldots$ are the features you are using to predict it, then the linear regression formula is:
$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_k x_k$$
Here, the $\beta_i$'s are just numbers, and the job of the algorithm is to determine what numbers work best.
Notice that if you vary one of the $x$'s, and look at how the output $y$ changes as a result, you'll get a line.  This is a direct consequence of the way the linear regression model works.  If you want it to give you sensible results, then you need to make sure this drawing lines assumption is at least approximately true.

the input variables are approximately normal in distribution- why?

This is simply wrong.  Linear regression works fine even if the distribution of the input variables is highly non-normal.  What is important is the relationship between the inputs and outputs, not the distribution of the inputs themselves.
This is what I meant by the advice the course being misguided.  You don't transform input variables because their distribution is skew, you transform them so that the linear shape the model is trying to draw through your data is reasonable
For example, here is a scatterplot I found online of a country's GDP vs. its average life expectancy (attribution is in the image):

Clearly, drawing a line through the scatter plot is completely unreasonable, so the linear regression equation:
$$ \text{Life Expectancy} = \beta_0 + \beta_1 \text{GDP} $$
is a bad choice for the data.  On the other hand, it looks like a logarithmic relationship is reasonable, so something like:
$$ \text{Life Expectancy} = \beta_0 + \beta_1 \log(\text{GDP}) $$
looks like it would work a lot better.  This is the type of situation where transforming the GDP measurements with a logarithm is a good idea.  But it has nothing to do with the distribution of GDP. You can't tell it's a good idea by drawing a histogram of GDP, it's about the relationship between GDP and life expectancy.

the output variable is constant variance (that is, the variance of the output variable is independent of the input variables – why?

This is a deeper issue of a different nature than the others.  For prediction models, it doesn't really matter, so if you're focusing on learning to build good predictive models don't worry about it for now.
As a summary, this assumption is intended to support the computation of the sampling distribution of parameter estimates.  For example, if you want to say something like "the probability that I would collect data in which the relationship between log(GDP) and Life Expectancy is greater than what I actually observed, even when the there is truly no relationship, is very, very small" you need to be able to compute the sampling distribution of the parameter estimates.  There are various assumptions that allow this to be done, and this constant variance assumption is one them.
That said, if you're only trying to make predictions, this isn't really relevant.  And in no case is the distribution of the input data assumed to be normal, that's just a misconception.


How to remove double quotation marks when converting from excel to text file?

Question: Every time I use this code, for some reason, double quotation marks appear for cell's output from beginning to ending. Anyway how I can remove these quotation marks?
I tried using a basic VBA, where it copies data from a certain column and converts it to a txt file.
Dim s As String, FileName As String, FileNum As Integer

  ' Define full pathname of TXT file
  FileName = ThisWorkbook.Path & "\2019.con"

  ' Copy range to the clipboard
  Range("A2", Cells(Rows.Count, "A").End(xlUp)).Copy

  ' Copy column content to the 's' variable via clipboard
  With New DataObject
     .GetFromClipboard
     s = .GetText
  End With
  Application.CutCopyMode = False

  ' Write s to TXT file
  FileNum = FreeFile
  If Len(Dir(FileName)) > 0 Then Kill FileName
  Open FileName For Binary Access Write As FileNum
  Put FileNum, , s
  Close FileNum

Actual Result:
"CONT=10"
"."
"."
"."
Desired Result:
CONT=10
.
.
.

Answer: You should use Print not Write if you don't want the quote marks
Comment: Also consider opening the file `For Output` if you're not writing a binary blob to it
Comment: How would I write it? @Cryostasys
Comment: Change
` Open FileName For Binary Access Write As FileNum
  Put FileNum, , s
  Close FileNum`
To
`Open FileName For Output As #1
Print #1, s
Close #1`
Comment: `s Close #1` is giving me error. do I need the s? @Cryostasys
Comment: I tried all this, still showing the double quotation marks
Comment: The comment didn't let me line space things properly for display.
The line structure for VBA should go by line numbers:
1 `Open FileName For Output As #1`
2 `Print #1, s`
3 `Close #1`
 The s is the copy of the data that you made in the original code.


Docker sempre verifica se a imagem local é de versão igual ou inferior a do repositório oficial Docker Hub?

Question: Ao levantar uma nova aplicação, não se tem a imagem localmente, então o Docker vai buscar a imagem que é a base do container. Uma vez que foi feito o download da imagem, Docker sobe o container. A aplicação está lá funcionando. Mas caso exista localmente a imagem que foi base para um outro container, se for antiga, como o Docker trata essa parte ao levantar a nova aplicação? Pega a imagem antiga, atualiza e roda o novo container? Gostaria de entender esse processo todo.
Answer: Vamos começar pelo comando run. Ao rodar docker run ubuntu o docker irá verificar se há localmente uma imagem do ubuntu utilizando a tag :latest. Caso haja o docker irá criar um container utilizando esta imagem, se não ele irá realizar o docker pull ubuntu:latest e, após isso, criar o container.
Ao rodar o comando docker pull ubuntu ele irá verificar se há uma imagem do ubuntu com a tag :latest, em caso negativo ele irá baixar a imagem. Caso, ao rodar o docker pull ele encontre uma imagem local com a tag :latest então irá baixar a assinatura da imagem(um hash em sha256) que está la no DockerHub e irá comparar com a assinatura da sua imagem local, se forem iguais ele imprime no console algo como:
status: Image is up to date for ubuntu:latest

Caso sejam diferentes ele irá baixar a nova imagem e marcará a antiga com a tag <none>, ficando algo parecido com isto:
Repository            tag                 Image ID            CREATED             SIZE
ubuntu               latest              113a43faa138        2 weeks ago         81.2MB
ubuntu               <none>              452a96d81c30        7 weeks ago         79.6MB

Comment: quando uma imagem fica com essa tag `` não serve de nada, só ocupa espaço em disco pois o docker não utiliza ela para gerar um novo container. Se você atualizar uma imagem o docker sempre utilizará a mais nova(a imagem local com a tag especificada) para criar novos containers.
Comment: Uhmmm. Show! Daí o novo projeto irá usar a última versão. Se eu levantar o projeto antigo, ele também rodará com base na última?  O  fica tipo estacionado, sem utilidade? Sei que seria outra pergunta. Mas valeu!


SQL Pivot Table with MAX

Question: I am stuck with this. I developed an SQL script tu obtain these information:-
Result
Here are the codes:-
SELECT [COST UNIT],[GL ACCOUNT]+ ' ' + [GL], SUM([CLOSING BALANCE]) AS [TOTAL GL CLOSING BALANCE] FROM
(
    SELECT AB.StartDate AS [DATE], AB.FreeTextField_04 AS [COST UNIT], IT.GLAccountDistribution AS [GL ACCOUNT], GL.Description AS [GL], ST.ItemCode AS [ITEM CODE] ,
    ST.Description AS [DESCRIPTION],
    IT.UserField_03 AS PACKAGING, ST.StockOnHandUOM AS [U/MEA.], IT.CostPriceStandard AS [STD COST PRICE], ST.PhyStkTakeQty AS [STOCK QTY],
    CAST(ROUND((ST.PhyStkTakeQty * IT.CostPriceStandard ),2) AS DECIMAL(12,2)) AS [CLOSING BALANCE]
    FROM [001].DBO.CS738281StkTake ST INNER JOIN [001].DBO.Absences AB ON ST.ReqID = AB.ID
    INNER JOIN [200].DBO.Items IT ON ST.ItemCode = IT.ItemCode
    INNER JOIN [200].DBO.GRV_GLAccounts GL ON IT.GLAccountDistribution = GL.GLAccount
    WHERE AB.StartDate = '2017-08-31 0:00:00.000'
    --AND AB.FreeTextField_02 in ( '1CTY','bsp')
    AND IT.GLAccountDistribution IN (12010010,12010020,12010030,12010040,12010041,12010042,12010043,12010050,12010060,12010080,12030010)
    --ORDER BY [GL ACCOUNT],[ITEM CODE]
) tmp
GROUP BY [GL ACCOUNT], [GL], [COST UNIT]
ORDER BY [COST UNIT],[GL ACCOUNT]

My objective is to do pivot table to look like this:-
Required Result
I tried many times but couldn't work. Can anyone help? Thank you so much :)
Answer: Use the PIVOT table operator with MAX like this:
WITH CTE
AS
(
    SELECT [COST UNIT],[GL ACCOUNT]+ ' ' + [GL], SUM([CLOSING BALANCE]) AS [TOTAL GL CLOSING BALANCE] FROM
    (

         SELECT AB.StartDate AS [DATE], AB.FreeTextField_04 AS [COST UNIT], IT.GLAccountDistribution AS [GL ACCOUNT], GL.Description AS [GL], ST.ItemCode AS [ITEM CODE] ,
            ST.Description AS [DESCRIPTION],
            IT.UserField_03 AS PACKAGING, ST.StockOnHandUOM AS [U/MEA.], IT.CostPriceStandard AS [STD COST PRICE], ST.PhyStkTakeQty AS [STOCK QTY],
            CAST(ROUND((ST.PhyStkTakeQty * IT.CostPriceStandard ),2) AS DECIMAL(12,2)) AS [CLOSING BALANCE]
            FROM [001].DBO.CS738281StkTake ST INNER JOIN [001].DBO.Absences AB ON ST.ReqID = AB.ID
            INNER JOIN [200].DBO.Items IT ON ST.ItemCode = IT.ItemCode
            INNER JOIN [200].DBO.GRV_GLAccounts GL ON IT.GLAccountDistribution = GL.GLAccount
            WHERE AB.StartDate = '2017-08-31 0:00:00.000'
            --AND AB.FreeTextField_02 in ( '1CTY','bsp')
            AND IT.GLAccountDistribution IN (12010010,12010020,12010030,12010040,12010041,12010042,12010043,12010050,12010060,12010080,12030010)
            --ORDER BY [GL ACCOUNT],[ITEM CODE]

    ) tmp
    GROUP BY [GL ACCOUNT], [GL], [COST UNIT]
)
SELECT *
FROM
(
    SELECT GL, [COST UNIT], [TOTAL GL CLOSING BALANCE]
    FROM CTE
) AS t
PIVOT
(
  MAX([TOTAL GL CLOSING BALANCE])
  FOR [COST UNIT] IN([1CTY], [AMCP], [AMPO])
) AS p

Comment: Hi Dimitri, thank you for your feedback. I have tried your script and it returns an error:-

    Msg 8155, Level 16, State 2, Line 1
    No column name was specified for column 2 of 'MyCTE'.

SQL Pivot Table with MAX


What are the fewest passing yards by quarterbacks in an NFL playoffs win?

Question: At the end of the Tennessee Titans victory against the Baltimore Ravens in the divisional round of the 2019-2020 playoffs, the commentators announced that Ryan Tannehil became the first Quarterback to throw for less than 100 yards in back to back playoff victories. Tannehil threw for 72 yards against the New England Patriots in the wild card round. And he threw for 88 yards against the Ravens.
This makes me wonder what are the 5 lowest passing yards by winning quarterbacks in any NFL playoff game? I'm considering quarterbacks who started the game and finished it (includes those who were benched at the end of the game if the team was winning by a big margin)
Answer:
Joe Flacco - BAL, 34 yards, Wild Card - Baltimore Ravens at New England Patriots - January 10th, 2010
Craig Morton - DAL, 38 yards, Divisional Round - Detroit Lions at Dallas Cowboys - December 26th, 1970
Mark Brunell - WAS, 41 yards, Wild Card - Washington Redskins at Tampa Bay Buccaneers - January 7th, 2006
Otto Graham - CLE, 43 yards, Divisional Round - New York Giants at Cleveland Browns - December 17th, 1950
Dieter Brock - LAR, 50 yards, Divisional Round - Dallas Cowboys at Los Angeles Rams - January 4th, 1986

Source - may require a subscription to view
Comment: Had to research each game log.  And there's no field you can set to sift it out.  Glad to help.
Comment: Looking at your source, it appears there have been instances where the QB threw for 17, 21, 31, and another 34 yards, which would be less than (or equal) to the list in this answer.
Comment: You are not considering your own criteria.  My list contains only starters that played a full (or nearly full) game
Comment: That's great! The sourced table doesn't have any indication if the QBs started the game or not.


Overriding radio button rendering

Question: I have a List (text) field on a comment type. This field is required and only one option may be chosen, so a radio button is how it will be rendered. I would now like to change the way the radio buttons are rendered to allow for additional necessary styling. I have never done this before.
Currently each element is styled as such:

  <input data-drupal-selector="edit-field-list-1" type="radio" id="edit-field-list-1" name="field_list" value="1" class="form-radio">
  Item #1

But, let's say that I wanted to add an HTML element inside this
 or move one around. I have created a module which adds a widget for this field type. Once this widget is selected, I have tried modifying the field using a number of different methods, but to no avail. I have been making sure to regularly clear the cache since I have been using the .module file and a template.
I've tried using a template file, radios.html.twig and adding it using various hooks, such as hook_theme_registry_alter() and hook_theme(). I've tried using hook_form_alter().
I have never modified the rendering of an element like this before and the documentation all seems to be different enough from what I am trying to do so as to only add confusion.
Could anyone please tell me how to accomplish this? I'm happy to add clarification as needed.
Additionally, is using a Field Widget the best way to accomplish this? It seems the correct way to me, since I only want it to affect the input element on the comment creation/edit form, but I'm open to being wrong.
Comment: I'm not sure on your structure, but if you're unable to access this field through template files because of it being produced by a module, you can do this quite easily with javascript.

https://developer.mozilla.org/en-US/docs/Web/API/ParentNode/append
Answer:
Additionally, is using a Field Widget the best way to accomplish this?

I would say yes, it's more work than a hook, but it makes things configurable most times hidden in a form alter hook. The result is the same. You can add to the form array additional render elements like div containers or attributes to the input element itself. Based on this add a theme suggestion, in case you want to add more markup in a template. See this answer from @Kevin.

Overriding radio button rendering


Which statistical models are suitable for prediction with clickstream data?

Question: I'm a Statistics student, and I'm thinking of writing my master's thesis on clickstream data analysis.
For my analysis I have a pretty big dataset (80 million rows), each of them being a click "impression". The dataset is from a news website and includes information such as:

User ID when logged on the website
User ID when NOT logged on the website (like a cookie)
Time of the visit (hour and date)
Link of the visited webpage
"Section" of the link (for example, Sport, News, ... with many categories)
Number of clicks which led the user to land on that page

What I'd like to do with this data is find the probability that a new user would click on a given new article, in order to recommend the user what to read next. I have in mind something like a score.
Doing my research I found out that a common way to tackle this type of issues is with association rules, path analysis or collaborative filtering.
What I'd like to know is: is it possible to approach the problem with "classic" data mining/machine learning techniques? I'm talking about GLMs, decision trees, neural networks, ... and other similar algorithms for supervised learning.
I ask the question because being each row an impression I have some "path" for each user and I'm not sure if it would not be statistically correct to apply one of the models I mentioned.
Comment: What is it you want to know about these data? Without that, how can an analysis be chosen?
Comment: You are absolutely right, how silly of me to forget such a fundamental detail. Post edited with this information.
Answer: I was tackling the exact same problem in my previous project. In this case, it is a food ordering company, so the data is clickstream and the target variable is ordered/not ordered. So, these are some techniques which were used and have been proven effective:

Tree-based models: As tree-based models are non-parametric in nature, they can be used for such problems. The approach can be scaled up to using random forests also.
Market Basket Analysis: This takes into account the entire journey of the user till they make the order, and till a time threshold for the ones who didn't. These models are notoriously popular to be not scalable, as they do full data scans every computation, and multiple computations are needed to model these. The faster version is the 'FP-growth' algorithm.

Here is a very related post from the DataScience SE site, which has more models and experiences from other users.
Answer: Yes, you can.
One thing to consider is engineering features from your data set that make the best use of the data you have collected - the raw variables don't look all that great. So, for example, from the website address you may be able to discover the site's host country. Or given the user id's, you may be able to group users into frequent users and occasional users, or (given the timestamp) night owls and day time users. Another thing to consider is how many different areas of the site a user visited.
An example of something that could be similar to your problem is the Avito competition on kaggle, which had a very similar target and arguably overlapping variables (admittedly said with incomplete understanding of your data) where the winner used gradient boosting (effectively a sophisticated ensemble of decision trees) and manually engineered variables, likely somewhat similar to what I described above. See http://blog.kaggle.com/2015/08/26/avito-winners-interview-1st-place-owen-zhang/ (the entropy based features Zhang mentions could include something like the number of different areas a user visited as suggested above)
Comment: Thank you very much, Robert. I know the Avito competition, but in that case there was a binary target variable (click/no click), while that's not the case for my data. The only information I have on the articles is about their section (~500 sections in total), so I don't have a very clear idea on how to create a target variable here.
What I'd like to do is use the variables I have (with some engineering too, as you said) to answer the question: "when a user comes to the website and reads some articles, what should I suggest him next?"
Do you have any idea on how to "binarize" this problem?


JQuery Overlay Opacity and Text Problem

Question: I have an unordered list show a bunch of logos as blocks. When the user hovers over any logo I'm using a div to create a semi-transparent overlay, but on top of the overlay I would like to put some text with a button.
Everything is working fine, but when I hover over the item the text tends to look like it's behind the overlay. If however I set the overlay opacity to 1 (opaque) the text shows up just fine. Either that or the text is adopting the opacity of the overlay.
Here's the HTML:

Click here for more information.

Company

        Description...

Locations: Toronto - Canada, Brisbane - Australia

Click here to see more

The div with the class set to portfolioMore is supposed to be displaying above the div overlay.
Is there any known problem with showing an opaque div over a semi-transparent div? Or is this a problem with my code somewhere?
Thanks
Jacques
And here is the CSS:
.portfolioMore
{
    background: black url(Portfolio.png) -194px -704px;
    //height:46px;
    //width:46px;
    margin-left:0;
    margin-top:-65px;
    position:inherit;
    display:none;
    text-align:center;
    font-family:arial;
    font-size:9pt;
    color:white;
    opacity:1;
    -moz-opacity:1;

}

and the second CSS part:
.overlay
{
    background-color:#000;
    opacity:0;
    display:none;
}

Comment: Can you post the CSS that goes along with this?

There shouldn't be any problem with what you're trying to do.  One thing that might help is putting  after the  in the markup.  Source order shouldn't matter here, but in some cases and browsers it does.
Comment: can you show the CSS rules of the overlay and portfolioMore elements?
Comment: Hey guys, I've added the two CSS rules to the original post. Thanks for the feedback. I'm about to try Pat's suggestion.
Answer: Try explicitly setting the stacking order of your div's by setting position: relative on both of them and giving a z-index:
.portfolioMore {
    position: relative;
    z-index: 2;
    /* other css declarations */
}

.overlay {
    position: relative;
    z-index: 1;
    /* other css declarations */
}

Comment: position: relative didn't seem to change anything, but position:absolute on the overlay rule fixed it. It seems I've gone for an overly complicated route here. In fact over all the years that I've been working with HTML CSS I've never gotten to fully understand Absolute vs Relative, now I see Fixed being used, but it apparently has bugs. Got any good resources to fully explain positioning in CSS?
Comment: Glad to hear you got it fixed and I do indeed: http://www.barelyfitz.com/screencast/html-training/css/positioning/
Comment: mother of all things green, what a an absolutely perfect tutorial. Pat, that's the best it's ever been explained to me. Thanks a bunch, really appreciate it.
Comment: Not a problem - once you get your head around CSS positioning/floating, CSS only layouts get a lot easier.

JQuery Overlay Opacity and Text Problem


How to play the trills in Chopin's Waltz in A minor Op 43 No 2

Question: In Chopin's Waltz Op 43 No. 2 in A minor are some trills in the left hand. How many trills should be played in this bar and where to stop the trill, on E or on F?
Also since the right hand starts while we are still trilling with the left hand, on which note of the trill should we start with the right hand? For me it seems natural to end the trill on E because the next tone will be a D#, but then we have an odd numbers of trills if we start on E which does not work rhythmically?
Thank you for your answers.
Answer: You would start the trill on E, then end on the D# eighth note. For trills, you generally don't play them rhythmically like you are suggesting. They are not coupled or aligned with the other notes. Therefore, you don't need to consider what part of the trill you are on when the notes in the right hand come in.
Comment: @Matriz you are correct, I believe. It should start on E. I indeed was thinking of Baroque practice, but I just checked and it looks like things changed! I will edit the answer.
Comment: It depends on the context: tempo, length of the trill, general stylistic traits of the composer/period, etc. You really just need to listen to some of the countless recordings there are on places like YouTube so you can get a sense of what the possibilities are. I've been playing for over 30 years, so it's not even something I am conscious of anymore. It has just become a matter of taste.
Comment: Hi, thanks for the answer. Why should the trill start on F? I thought trills in the romantic era started from the lower note unlike the baroque era.
Comment: I have another little question. Are the amount of notes in a trill in general odd (meaning you land on the note where you started) or are there sometimes trills which have an even number of notes?
Comment: My pleasure and good luck with the piece!


Example of response variables that are/ are not statistically independent

Question: I have read that in regression there is no need for the response variable (independent variable) to be statistically independent. The source is
https://en.wikipedia.org/wiki/Multinomial_logistic_regression, under assumptions: "As with other types of regression, there is no need for the independent variables to be statistically independent".
I understand what it means to be statistically independent, but I cannot think of examples of when the response variable would/ would not be statistically independent.
Could you give an example of a statistically dependent response variable vs. a statistically independent response variable?
Comment: Could you please post a reference where you have seen this? "Independence" requires multiple variables to be discussed.
Comment: $y_t = f(y_{t-1}) + \varepsilon_{t}$? Conditions today are a function of conditions yesterday?
Answer: Example of independent response:  Exam results are recorded for a class of 30 students. We might use a t-test, or a linear regression model for such data.
Example of non-independent response: Exam results are recorded for students in several classes. Results for students in one class are more likely to be similar to each other than results for students in other classes, perhaps due to different teachers. To account for this non-independence, we might use a linear mixed effects model, or use a linear regression model with fixed effects for the class identifier.
Comment: Does this answer your question ? If so please consider marking it as the accepted answer. If not, please let us know why. Also, if you haven't already, please consider upvoting it.


Overriding abstract methods with values, a good practice?

Question: In Scala, fields and methods belong to the same namespace so a field can easily override a method. That's a very attractive Scala feature. For example, it allows fixing a class method to an specific reference at some point of a classes hierarchy:
abstract class AC { def x: Int }
class C extends AC { override val x = 10 }

Suppose there is a second Base's attribute calculated form x:
abstract class AC { def x: Int; val y: Int = x*2 }
class C extends AC { override val x = 10 }

val v = (new C).y

One could expect v value to be 20 however it is 0. Ok, what is happening is that AC constructor is being called before C's, hence, x (which is now an attribute) isn't set at that point yet and the constructor uses the default value for integers (I may be wrong).
What strikes me is that with:
class C extends AC { override def x = 10 }

val v = (new C).y

v equals 20.
I have a hunch that the difference comes from the fact that methods get resolved in a different way than values. Can somebody give an explanation for the behaviour difference? more details on how Scala constructors work?
Wouldn't it be more coherent to have the same behaviour in both cases?
Concerning the question's title: Isn't it hazardous to override methods as values? Some client code could do that for a library class without noticing that dependant attributes could get invalid values thus generating bugs.
Comment: I think the better practice would be to make `y` lazy or a def when it references something that may be overridden. Though I'm not sure why the second initialization has problems. It's hard to find anything in the spec about how overriding a def with a val is handled.
Answer: The fields of the superclass are instantiated before the fields of the subclass. So given
abstract class AC {
  def x: Int
  val y: Int = x * 2
}

class C extends AC {
  override val x = 10
}

new C will first instantiate the fields of AC, in this case y and then the fields of C, i.e., x. Because Int values are 0 before initialization, y = x * 2 will be 0.
Now, if you override x as a method (def), then C has no fields to be initialized and y = x * 2 calls the method x which returns 10, hence y takes the value 20. Update: To make it clear, methods do not belong to a single instance, but are shared among all instances of the same class. Hence, the method x is created at compile time not when the instance of C is created.
The thing I consider dangerous here is that y in AC is a val; doing so forces y to be instantiated before the fields of the subclasses are initialized, forming a potential forward reference. If you want to refrain from implementing y as a method, you can implement it as a lazy val, meaning it is evaluated exactly once, namely on its first use, in this case only after the object is fully instantiated.
abstract class AC {
  def x: Int
  lazy val y: Int = x * 2
}

class C extends AC {
  override val x = 10
}

println((new C).y) // prints 20, because y is initialized on its first use

By the way: The same problem occurs if you override a val as a val.
class AB {
  val x = 10
  val y = x * 2
}
class B extends AB {
  override val x = 4
}

println((new B).y) // prints 0, because the overridden `x` is not yet initialized

Comment: As described in my question, I´d expect `y` to be 0 when `x` is an attribute; What surprises me is that `y` isn't 0 when `x` is a method. As I can understand from your answer when `y` is instantiated by `AC`'s constructor `x` method is called, being its implementation the provided by the subclass.
Comment: Your edit tells me I might be misexpressing: Of course methods do not belong to a single instance, that isn't the question here. It is that, as far as I currently understand, the fact of `y` getting the right value when `x` is a method may be explained by the fact that `AC` constructor is calling to `C` (CLASS) implementation of `x`.
Comment: @PabloFranciscoPérezHidalgo Yes, when the method `x` is called, the right implementation is chosen based on the runtime type, here it is an instance of `C`. Albeit the example is for c++, the Wikipedia page on [VTable](https://en.wikipedia.org/wiki/Virtual_method_table) has a nice example of a possible memory layout. I assume the general idea is also true for Scala. Also note that Scala creates a getter-method for every `val`, making overriding `def` as `val` possible. Does this answer the question?


Who are on the field in a field goal or PAT situation in American football?

Question: In American football, a long snapper, a holder, and a kicker are required in a field goal or one-point conversion situation.
Who are the other eight players on the field in these situations? If most of the offensive team stay on the field, who are the three players that are removed?
Answer: 8 big guys, who have one job: don't let the defense block the kick. There would be no specific overlap with the offensive starters (e.g. the starting left tackle wouldn't necessarily stay in to play left tackle), but obviously the skills that offensive linemen have are very closely related to the skills you want here.
Answer: The first 2-3 (4-6 players total) on each side of the snapper are usually offensive linemen. The last player, or 2, on each side is usually player types that fit a Tight End mold. Large, strong players with blocking ability that are also more athletic and more likely to be able to catch (for fake kicks).
Outside of the linemen, they're not typical positions. Different teams call them different things based on the teams utilization of them (say between the OL and TE types).


How do I remove a driver completely on a Windows XP client machine?

Question: I have a client (XP SP3) machine where newer drivers have been installed.
Unfortunately while these (audio) drivers fixed mic boost, they have introduced problems such as no input working and random unnamed devices in applications that select the audio device.  I need to revert the drivers.
However, rollback does not work (something about the drivers weren't backed up) and manually selecting the older driver is futile as XP just ignores it, stating, in its infinite wisdom, that the current driver is better and so I'd have no reason to want to use the older driver.
I know on printer drivers I can just delete - is there an option for other drivers?
Btw. I have done some research and can't seem to find a definite answer so I think this will be of help to plenty of others as well.
Answer: Windows uses a number of factors to decide what is the "best" driver, such as hardware ID matching (INFs that match more specific hardware IDs are "better"), driver date, driver version, and whether the driver is signed. How Setup Selects Drivers has the details if you're interested.
As for how to remove the newer driver, if it has an installer listed in Add/Remove Programs, then by all means use that. If not, you should be able to manually choose the older driver by right-clicking on the device in Device Manager, selecting "Update Driver...", "Install from a list or specific location", "Don't search. I will choose the driver to install", etc. Or did you already try that? (As well as selecting "Uninstall..."?)
If all else fails, you might have some success using the devcon command that ships with the Windows Driver Kit. You can remove a driver package using devcon dp_delete, or force Windows to choose a different driver using devcon update. There's also an older version of devcon available for direct download (the WDK is pretty large).
If you resort to manually deleting the driver's .inf file from %SystemRoot%\inf, be sure to delete the corresponding .pnf file as well, because Windows uses that to cache some/all of the .inf information.
Comment: Thanks - great answer.  I tried the manual install first but I think it kind of just uninstalled the device not the driver->device attachment.  Your answer is great though for when the standard approach fails.
Answer:
Here are some good pointers for uninstalling pesky .inf files (Microsoft link): http://support.microsoft.com/kb/813449
Something to try if your driver uses a .dll would be to unregister the driver's dll: http://www.xp-vista.com/other/how-to-unregister-dll-files
It's worth a try, just make sure you 'cd' to the directory containing the dll.

Another useful tool for these kinds of problems is the Change Analysis Diagnostic tool: http://support.microsoft.com/kb/924732
Answer: Have tried devcon, but it says that there is no dp_delete command, and when investigating things I found a easier way.
As some people say, you can delete .inf and pnf. What they don't tell is how to locate those.
First, open device manager as administrator, right click on device and choose properties, on driver tab, click on details, and you would get the .sys filename of the driver.
Then you must locate the correct .inf and pnf to delete, they are not the same name as the .sys.
Follow instructions from http://winhlp.com/node/199 , which in brief are:

cmd as admin, cd to inf folder: cd %windir%\inf
Using the name of the .sys file (without the extension) instead of "WG311T", look for the inf file with:

for %a in (oem*.inf) do find /i "WG311T" %a >>out.txt

Open out.txt with notepad, and look which oem*.inf file is your driver.
Delete .inf and .pnf oem file with (substitude "oemXX" for the name of oem file where your driver was found):

attrib -h -r -s oemXX.*
del oem12.* Reboot

Follow original link http://winhlp.com/node/199 for detailed instructions.
How to know the name of the driver to look for isn't explained there, remember that it's the same as the .sys file (without the extension).
Tested removing bcmwl6.sys driver from Broadcom wifi succesfully.
Comment: Driver can also be registered at `HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services` registry subkey for autoload. This key also needs to be removed.
Answer: You should find corresponding .inf file and delete it from windows\inf directory
Comment: Thanks - the related files will just then be ignored right?
Answer: The KB article that l0c0b0x mentioned includes a link to an API call that can be used on Windows XP and later (on Windows 2000 you delete .pnf and .inf files manually).
SetupUninstallOEMInf()

The SetupUninstallOEMInf function
  uninstalls a specified .inf file and
  any associated .pnf file. If the .inf
  file was installed with a catalog for
  signing drivers, the catalog is also
  removed. A caller of this function
  must have administrative privileges,
  otherwise the function fails.
Answer: Hmm, I booted into safe mode and managed to resolve this by making sure I uninstalled the device and interrupted XP before it tried automatically reinstalling it when I scanned for hardware changes.  Still interested in how to remove a driver from being detected in automatic driver install though.


How to create code that searches for a word in a .txt file and counts the amount of times it occurs in C?

Question: FYI I am fairly new to coding in C. I am looking for a way to search a .txt file for a certain word that is inputted by the user and counts the number of times it occurs within the .txt file. I currently have the .txt file in a 2D array. I have no idea how to even start. Please help!
Comment: no it wouldnt be
Comment: how would I count the number of times the word occurs in the .txt file if I used strcmp? Doesn't strcmp only give back a boolean value (so true or false)?
Comment: I'm voting to close this question as off-topic because "I don't know where to start" is not a question.
Answer: You can start with the standard library function strstr. Read the description, and think about how you'd apply it to your 2D array (of, presumably, char*).


Meaning of design matrix in context of Bayesian B-Spline regression?

Question: I'm learning about modeling B-Splines using PyMC3. The design matrix of splines (apparently) can become quite complicated, so it's easier to delegate this construction to an API, Patsy. In the context of B-Splines, I don't understand what the formula-like string is supposed to do.
The tutorial I'm following (see section 4.74 in https://github.com/pymc-devs/resources/blob/master/Rethinking_2/Chp_04.ipynb) uses Patsy's dmatrix in order to construct its design matrix. I don't quite "get it."
I've created some synthetic data, namely, columns, year and value. In the below synthetic data, I've used one knot per observation and degree of 0 (no observation is influenced by more than 1 spline) to hopefully keep things simple.
Synthetic data
fake = pd.DataFrame({'year':[1,2,3],'value':[1,1,1]})
k = [1,2,3]

Without intercept:
dmatrix(
    "bs(year, knots=knots, degree=0, include_intercept=False) ",
    {"year": fake.year.values, "knots": k},
).view()
>>>
array([[1., 1., 0., 0.],
       [1., 0., 1., 0.],
       [1., 0., 0., 1.]])

Without intercept -1:
dmatrix(
    "bs(year, knots=knots, degree=0, include_intercept=False) -1",
    {"year": fake.year.values, "knots": k},
).view()
>>>
array([[1., 0., 0.],
       [0., 1., 0.],
       [0., 0., 1.]])

With intercept:
dmatrix(
    "bs(year, knots=knots, degree=0, include_intercept=True) ",
    {"year": fake.year.values, "knots": k},
).view()
>>>
array([[1., 0., 1., 0., 0.],
       [1., 0., 0., 1., 0.],
       [1., 0., 0., 0., 1.]])

With intercept -1:
dmatrix(
    "bs(year, knots=knots, degree=0, include_intercept=True) -1",
    {"year": fake.year.values, "knots": k},
).view()
>>>
array([[0., 1., 0., 0.],
       [0., 0., 1., 0.],
       [0., 0., 0., 1.]])

API references:
https://patsy.readthedocs.io/en/latest/spline-regression.html
https://patsy.readthedocs.io/en/latest/API-reference.html
From what I can gather, the default behavior is the first column containing all 1's. By including -1 at the end of the "formula-like" string object, this column of ones if removed. Likewise, by default, the second column is all zeroes, and include_intercept=False removes this column of zeroes.
I'd like to know, for the context of spline regression: (A) what is the columns of zeroes (intercept?) for and why is it all zeroes? Likewise, (B) what is the column of all ones for? (I would have guessed that ones represented the intercept, but I guess this is wrong.)
Answer: The column that is giving you all zeros doesn't always give all zeros.  I never use it, but I see that if I change year to include 0, the bs-intercept column has a one in it:
patsy.dmatrix(
    "bs(year, knots=knots, degree=0, include_intercept=True) ",
    {"year": [0,1,2,3], "knots": k}).view()

array([[1., 1., 0., 0., 0.],
       [1., 0., 1., 0., 0.],
       [1., 0., 0., 1., 0.],
       [1., 0., 0., 0., 1.]])

The first column is the more traditional "intercept" term in regression modeling, where if you didn't have the spline part making things confusing, you would be predicting $y \sim \beta_0 + \beta_1x_1 + \ldots \beta_p x_p$.
It helps me to think of bs as a fancy alternative to including nonlinear transformations of $x$---instead of $x^2$ and $x^3$ to try to represent a nonlinear response, I can use bs(x) to do a better job with less typing.  For me a picture is worth a thousand words, so here is one that I find useful:
xx = np.linspace(-5,5,100)
df = patsy.dmatrix(
        "bs(xx, knots=[0,2], degree=3, include_intercept=False)",
        return_type='dataframe')
df.index = xx

df.plot()
plt.legend(loc=(1.01, .01));


Decomposing isometries in $\mathbb R^2$

Question: My geometric intuition is a bit weak, and I am trying to understand some basic examples of isometries in the Euclidean plane.
How can you write the following two isometries as a composition of first a function which can be either the reflection across the $x$-axis or the identity map, then a rotation about the origin, and finally a translation?
(i) The reflection across the line $y=3x$.
(2) The translation by $(2,2)$ followed by the reflection across $y=3x$.
I'd really appreciate an explanation.
Answer: For the first one, you start with a reflection on the $x$-axis and then you apply a clockwise rotation of $2\arctan(3)$ radians around the origin. Then you apply the null translation.
For the second one, start with a reflection on the $x$-axis and then you apply a clockwise rotation of $\arccos\left(-\frac45\right)$ radians around the origin. Then you apply the translation by $\left(-\frac25,\frac{14}5\right)$.
Comment: How did you get the number $2\arctan(3)$? And for the second one, are you accounting for the reflection across $y=3x$?
Comment: It was chosen so that each point of the line $y=3x $ is mapped into itself.
Comment: I still don't see how you get your answers. Could you please explain a bit more?
Comment: @CuriousKid7 The slope of the line $y=3x$ is $3$. After reflecting it on the $x$-axis, ye get the line $y=-3x$, whose slope is $-3$. The way of mapping this line back to the line $y=3x$ through a clockwise rotation around $(0,0)$ is by choosing a rotation of $2\arctan(3)$ radians.
Comment: I see, but for the second isometry I mention, isn't your decomposition the same as a translation by $(2,2)$? How do you include the reflection across $y=3x$ that follows it?
Comment: @CuriousKid7 I misunderstood your second isometry and shall edit my answer.
Comment: @CuriousKid7 Please do the math. The reflection on the $x$-axis is the map $(x,y)\mapsto(x,-y)$. The matrix of the rotation that I mentioned is $\bigl(\begin{smallmatrix}-4/5&-3/5\\3/5&-4/5\end{smallmatrix}\bigr)$.

Decomposing isometries in $\mathbb R^2$


Intercepted XHR - run function before raising load

Question: On a web page with a extension-injected custom XMLHttpRequest class, I need to intercept and modify certain responses before the load event is raised on the rest of the document. Right now, my modification code reacts to the load event. How would I use a function to modify the response before the load event was fired?
This is the code I am using:
let oldXHROpen = window.XMLHttpRequest.prototype.open;
window.XMLHttpRequest.prototype.open = function (method, url, async, user, password) {
    var originalresp = this;
    //Do something with the method, url and etc.
    originalresp.addEventListener('load', async function () {
        //Do something with the response text
        //***This function needs to execute before load is raised for the rest of the document***
        value = '<note>This is a test XML document.</note>';
        parser = new DOMParser();
        apiresponsexml = parser.parseFromString(value, 'application/xml');
        //Replace data in the response
        Object.defineProperties(originalresp, {
            'response': {
                value: value
            },
            'responseXML': {
                value: apiresponsexml,
            },
            'responseText': {
                value: value,
            }
        });
    });
    return oldXHROpen.apply(originalresp, arguments);
};

This question is a continuation of this previous thread.
Answer: Override onreadystatechange (fired before load/onload) inside send():
function plantXhrHook() {
  let origStateChange;

  function modifyResponse(e) {
    if (this.readyState === 4) {
      const value = 'foo';
      let xml;
      Object.defineProperties(this, {
        response: {value},
        responseText: {value},
        responseXML: {
          get() {
            if (typeof xml === 'undefined')
              xml = new DOMParser().parseFromString(value, 'application/xml');
            return xml;
          },
        },
      });
    }
    if (origStateChange) {
      origStateChange.apply(this, arguments);
    }
  };

  const origSend = XMLHttpRequest.prototype.send;
  XMLHttpRequest.prototype.send = function () {
    origStateChange = this.onreadystatechange;
    this.onreadystatechange = modifyResponse;
    origSend.apply(this, arguments);
  };
}

const script = document.createElement('script');
script.textContent = `(${plantXhrHook})()`;
document.documentElement.appendChild(script);
script.remove();

Since other extensions or page scripts may also hook XHR, it would make sense to run your code in a content script with "run_at": "document_start".
Comment: Thanks! I'll give this a try when I have more time to add in my other code.


cakephp full find/read depth

Question: I'm basically trying to extend the depth of my query by returning the Users and Comments for both my Project and Update tables.
Site visitors can view a Project, which has many Updates listed below it.
Users can leave comments on both the initial Project AND the Updates below, and each update and project is submitted by any users.
I have the following relationships set up, but these could be wrong:
Project.php (Model)
    belongsTo           - User
    hasMany             - Update, Comment

Update.php (Model)
    belongsTo           - Project, User
    hasMany             - Comment

Comment.php (Model)
    belongsTo           - Project, Update
    hasAndBelongsToMany - User

User.php (Model)
    hasMany             - Project, Update, Comment

The view in my ProjectController.php is as follows:
public function view($id = null, $slug = null) {
    $this->Project->id = $id;
    $this->set('Projects', $this->Project->read());
}

But this only exports an array containing:
Project
    User        (belonging to Project)
    Comments    (belonging to Project)
    Updates     (belonging to Project)

But I would like this full result, containing the comments to everything and the users for those comments:
Project
    User        (belonging to Project)
    Comments    (belonging to Project)
        User        (belonging to each Comment)
    Updates     (belonging to Project)
        User        (belonging to Update)
        Comments    (belonging to Update)
            User        (belonging to each Update Comment)

In the database, comments are tied to users by one ID (user_id) and contain no more details about the user than that.
Answer: Have you tried toggling $this->Project->recursive before calling read()?
Depth   Description
-1      Cake fetches Group data only, no joins.
0       Cake fetches Group data and its domain
1       Cake fetches a Group, its domain and its associated Users
2       Cake fetches a Group, its domain, its associated Users, and the Users' associated Articles

http://book.cakephp.org/1.3/view/1063/recursive

cakephp full find/read depth


Php in JavaScript in Laravel

Question: I have JS code:
var pobTeamId = document.getElementById('team_a_id').value;
var query = "<?php echo Sport::find(Team::find(pobTeamId)->sport_id)->id; ?>";

I need insert value pobTeamId in variable query.
I don't know how I can add this variable. I trying using this:

...Team::find(pobTeamId)...

...Team::find($pobTeamId)...

...Team::find(?>"pobTeamId"<?php)...
but Laravel returned only errors.
Comment: You can't. You're going the wrong direction. PHP (processed on the server) can output javascript. Javascript (processed on the client generally) can't output PHP (at least not how you're trying to do it) You can send and receive information from PHP through ajax calls.
Answer: Do it like this:
<form method="get" action="{{ route('get_sport_id') }}">
    <input id="team_a_id" value="" name="team_a_id"/>
    <button type="submit"> Fetch </button>
</form>

Then in your controller:
public function getSportID()
{
    $sport_id = Sport::find(Team::find(request()->get('team_a_id')->sport_id)->id;

    return back()->with('sport_id', $sport_id);
}

With a corresponding route that's something like this:
Route::get('/sport-id', 'SportController@getSportID')->name('get_sport_id');

Now your view will have access to $sport_id. Of course, you should check isset($sport_id) before attempting to use it in the view.
Answer: You approach is wrong! PHP won't be able to get the value of pobTeamId.
Use ajax to send the value to the Controller
var pobTeamId = document.getElementById('team_a_id').value;

// Initiate an Ajax either on page load or on button click
$.ajax({
    url: '', // path you defined in your routes file
    type: '' // either POST or GET
    data: {
         "pobTeamId": pobTeamId
    },
    success: function (data) {
    }
});

and in the Controller you would have access to the pobTeamId
public function yourFunction(Request $request)
{
    $pobTeamId = $request->input('pobTeamId');

    $sport_id = Sport::find(Team::find($pobTeamId)->sport_id)->id;
}

you would need to reference the Sport Model in your controller and add an appropriate route

Php in JavaScript in Laravel


How to prove that $C = cc^T$ is not convex?

Question: Consider the primary optimization problem
$$\begin{array}{ll} \text{minimize} & f_0(c,C)\\ \text{subject to} & C = cc^T\end{array}$$
at this paper. $c$ can only take discrete values, except zero. In equation $(9)$, the author wrote that the constraint is not convex. How can I prove the non-convexity of this equality constraint?
Comment: I think you should drop the discreteness requirement.  Discrete implies non-convex, unless some definitions are tweaked.
Answer: Suppose $c \neq 0$ and $C=cc^T$. Then $C = (-c)(-c)^T$ and so $(C,c), (C,-c)$ are feasible. However the average $(C , 0)$ is not feasible, hence the feasible set is not convex.
Comment: I do know know what convexity means in the context of discrete parameters.
Comment: thanks for answer,  does your answer hold for discrete $c$ that can't take zero value?
Answer: Saying "the constraint is convex" means that if the constraint is satisfied at two points, the constraint has to be satisfied at any convex combination of those points. In particular, it must be satisfied at the midpoint of those two points. Now, this constraint is satisfied at $c=\begin{pmatrix}1\\0\end{pmatrix},C=\begin{pmatrix}1&0\\0&0\end{pmatrix}$ and at $c=\begin{pmatrix}-1\\0\end{pmatrix},C=\begin{pmatrix}1&0\\0&0\end{pmatrix}$, but not at the midpoint $c=\begin{pmatrix}0\\0\end{pmatrix},C=\begin{pmatrix}1&0\\0&0\end{pmatrix}$, so it isn't convex.
Comment: thanks for answer, does your answer hold for discrete $c$ that can't take zero value?
Comment: You'd need to modify it, but I suspect so. But convexity doesn't play well with discrete values, you can't have a convex constraint for a discrete problem. The continuous relaxation either will or won't be convex, I suspect it won't be convex.


How to get the insrted row data in nodejs-mysql insert query?

Question: i have a simple insert query that looks like that:
let sql = "INSERT into users(name, date) VALUES ('"+name+"', '"+date+"')";
  connection.query(sql, (err, result) => {
      if(err) {
          res.status(500);
      } else {
        res.send(result) // we have here an object that has only the inserted id
      }
  });

which i really want is getting the inserted row data not just the id without making another select query to get them.
is there is a way to make that happens in one query?
Comment: `mysql` doesnt return data on insert. You could use a query builder or an ORM http://knexjs.org/#Builder-returning
Answer: If you are using:
// include mysql module var mysql = require('mysql');
The mysql module won't return data on the insert's query. It's returning:

Number of rows affected, Number of records affected with the warning, Message

If you wanna get data that you inserted, you should use a query builder or ORM like Sequelize. Sequelize Documentation.
Answer: You can get last inserted id using the code below:
SQLconnnection.query(sql, (err, result) => {
      if(err) {
          console.error(err);
      } else {
         console.log(result) ;
         /*Output=>{affectedRows: 1
                    changedRows: 0
                    fieldCount: 0
                    insertId: 1 =>Last inserted ID Here
                    message: ""
                    protocol41: true
                    serverStatus: 2
                    warningCount: 0}*/
       console.log(result.insertId);//=>last inserted id get
      }
  });

Answer: You can use this to retrieve the
let sql = "INSERT into users(name, date) VALUES ('"+name+"', '"+date+"') ; SELECT * FROM users WHERE id = SCOPE_IDENTITY()";
  connection.query(sql, (err, result) => {
      if(err) {
          res.status(500);
      } else {
        res.send(result) // we have here an object that has only the inserted id
      }
  });

SCOPE_IDENTITY()  Returns the last identity value inserted


How to find the speed and acceleration of a spaceship moving exponentially towards the speed of light

Question:
Taylor is in a spaceship that begins to move from rest and whose velocity tends exponentially to the value $c$ (the speed of light) over time. After one second, Taylor's speed was $-10$ m/s. Determine the speed function, depending on the time! (Hint - look at the function in the form $c×a^x+b$, which could meet the described criteria.) Determine the acceleration felt by Taylor during this movement! At what point will it be greater - at the beginning or the end of the movement?

Did I solve it correctly?
First, let's understand the problem. We are given that the velocity of the spaceship tends exponentially to the value $c$ over time. This suggests that the velocity function could be of the form $v(t) = c \times (1 - e^{-kt})$, where $v(t)$ is the velocity at time $t$, $c$ is the speed of light, $k$ is a constant, and $e$ is the base of the natural logarithm.
We are also given that after one second, Taylor's speed was $10$ m/s. We can use this information to find the value of $k$.

Step 1:
Find the value of $k$.

We know that $v(1) = 10$ m/s. Substituting these values into the velocity function gives us: $10 = c\times(1 - e^{-k})$. We can solve this equation for $k$. However, we don't know the value of $c$. Since $c$ is the speed of light, we can assume it to be approximately $3\times10^8$ m/s. Substituting this value into the equation gives us: $10 = 3\times10^8\times(1 - e^{-k})$.
Solving this equation for $k$ gives us:
\begin{align} e^{-k} = 1 - 10/(3\times10^8) &\implies -k = \ln(1 - 10/(3\times10^8)) \\
&\implies k = -\ln(1 - 10/(3\times10^8))
\end{align}

Step 2:
Find the velocity function.

Now that we have the value of $k$, we can substitute it back into the velocity function to get:
$$v(t) = c\times(1 - e^{-kt}).$$

Step 3:
Find the acceleration function.

The acceleration is the derivative of the velocity with respect to time.
So, we can find the acceleration function by differentiating the velocity function:
$$a(t) = \frac{\Bbb dv}{\Bbb dt} = c \times k \times e^{-kt}.$$

Step 4:
Determine when the acceleration is greater.

The acceleration is a decreasing function of time (since $e^{-kt}$ decreases as $t$ increases), so the acceleration is greater at the beginning of the movement.
So, the velocity function is $v(t) = c\times(1 - e^{-kt})$ and the acceleration function is $a(t) = c \times k \times e^{-kt}$. The acceleration is greater at the beginning of the movement. The velocity function is $v(t) = c\times(1 - e^{-kt})$ and the acceleration function is $a(t) = c \times k \times e^{-kt}$. The acceleration is greater at the beginning of the movement.
Comment: This is a poorly conceived problem. It sets up a scenario in which relativistic physics clearly applies, then (apparently) wants you to solve it using only classical mechanics. It's a serious framing error.
Answer: The form of the function $ca^t+b$ in the hint is not exactly what you want. If you have a function $v(t)=ca^t+b$, where $c$ is a known constant, $v(0)=0$, and $v(1)=10$, then your solution is correct. But that's not a realistic velocity function, because, since $a=(10+c)/c$, $\lim_{t\to\infty} v(t)=\infty>c$, which is both impossible and does not satisfy the statement of the problem that his velocity tends to $c$. It is also not physically realistic, because acceleration should decrease over time, but you have (correctly) stated that your $v$ function has increasing derivative.
I noticed that the hint says a function of the form $ca^t+b$ could solve the problem. I would recommend trying a function of the form $-ca^t+b$. Actually, you could try a function of the form $v(t)=ma^t+b$ and solve for $m$, $a$, and $b$ using $v(0)=0$, $v(1)=10$, and $\lim_{t\to\infty}v(t)=c$, in which case you'll find that $m=-c$.
After that, everything you did with the function once you found it is good.


Vertical vectors on a principal bundle

Question: Let
$$\pi:P\longrightarrow M$$
be a principal $G$-bundle, and consider, for each $p\in P$, the map
$$\begin{matrix}l_p:&G&\longrightarrow&P\\&g&\longmapsto &p\cdot g\end{matrix}$$
let $x=\pi(p)$, and since $l_p(e)=p$ we get the sequence of vector spaces
$$0\longrightarrow T_e G\longrightarrow T_pP\longrightarrow T_xM\longrightarrow 0$$
and so vertical vectors are defined as the image of $d_el_p$, or (and here is where I have trouble) as the kernel of the next map, which means that the sequence must be exact.
In my notes (Fibre bundles and Chern-Weil theory by Johan Dupont) there is no proof of this fact and it's driving me crazy, I don't know how to write things so I can "see" what those linear maps are or how they work on a particular vector. Any idea or reference will be appreciated.
Answer: I don't think you need an explicit description of those maps to conclude. Consider the compositon $\pi\circ l_p$. For any $g\in G$ it holds that $\pi(l_p(g))=\pi(p\cdot g)=\pi(p)=x$, so $\pi\circ l_p$ is a constant map. Hence, we have that $d(\pi\circ l_p)=0$, so $0=d\pi_p\circ(dl_p)_e$, which means that $\mathrm{im}((dl_p)_e)\subset\ker(d\pi_p)$. Can you close the argument yourself?
Geometric edit
Consider first the trivial principal G-bundle $P=M\times G$ with the trivial $G$-action. In this case, viewing the tangent space to $P=M\times G$ at $p=(m,g)$ as the $T_pM\oplus T_gG$, the exactness of the sequence is clear immediately, and the map $dl_p$ is just $v\mapsto(0,dL_g(v))$, with $L_g:G\to G$, $h\mapsto gh$. Anyways, if you just "draw" $M\times G$ as a product with the base horizontally a line and $G$ attached to $M$ at each point as a vertical line, then the tangent vectors to $G$ are actually "vertical", and I think this is where the terminology comes from, and it should serve as a sufficient interpretation/intuition. The vertical vectors are simply precise those which don't point non-trivially with respect to $M$ at all.
Comment: If anything it still unclear, please say so.
Comment: Your argument is pretty clear but I did not manage to see the other content yet. I can't see geometrically what it means for a vector in $T_pP$ to be zero through the map $d\pi_p$.
Comment: I don't have time to edit my answer now, but try thinking of P as MxG (which is locally true), then it should be clear.
Comment: In this lecture https://www.youtube.com/watch?v=jFvyeufXyW0&list=PLPH7f_7ZlzxTi6kS4vCmv4ZKm9u8g5yic&index=21 the above idea is explained more carefully and clearly. I found this enlightening.

Vertical vectors on a principal bundle


PHPUnit Symfony4 creating and testing Custom Validator

Question: My goal is to build custom validator and test it with PHPUnit.
PasswordComplexity.php
<?php

namespace App\Components\Validator\Constraint;

use Symfony\Component\Validator\Constraints\Regex;

/**
 * Class Password
 * @package App\Components\Validator\Constraint
 * @Annotation
 */
class PasswordComplexity extends Regex
{
    public $message = 'The password "{{string}}" does not meet the password policy requirements.';
    public $pattern = [
        '/.{8,}/',
        '/\d+/',
        '/[a-z]+/',
        '/[A-Z]+/',
        '/[!@#$%\-_*+=]+/'
    ];

    /**
     * {@inheritdoc}
     */
    public function getRequiredOptions()
    {
        return [];
    }
}

PasswordComplexityValidator.php
<?php
declare(strict_types=1);
namespace App\Components\Validator\Constraint;

use Symfony\Component\Validator\Constraints\RegexValidator;

class PasswordComplexityValidator extends RegexValidator
{
}

PasswordComplexityValidatorTest.php
<?php
declare(strict_types=1);
namespace App\Tests\Components\Validator\Constraint;

use App\Components\Validator\Constraint\PasswordComplexity;
use App\Components\Validator\Constraint\PasswordComplexityValidator;
use PHPUnit\Framework\MockObject\MockObject;
use PHPUnit\Framework\TestCase;
use Symfony\Component\Validator\Context\ExecutionContext;
use Symfony\Component\Validator\Context\ExecutionContextInterface;
use Symfony\Component\Validator\Violation\ConstraintViolationBuilderInterface;

class PasswordComplexityValidatorTest extends TestCase
{
    public function testRequiredOptions()
    {
        $validator = new PasswordComplexity();
        $this->assertEquals([], $validator->getRequiredOptions());
    }

    /**
     * @dataProvider validPasswordProvider
     */
    public function testValidation($value)
    {
        $constraint = new PasswordComplexity();

        /** @var ExecutionContextInterface|MockObject $context */
        $context = $this->getMockExecutionContext();
        $context->expects($this->never())->method('buildViolation');

        $validator = new PasswordComplexityValidator();
        $validator->initialize($context);

        $validator->validate($value, $constraint);
    }

    /**
     * @dataProvider invalidPasswordProvider
     */
    public function testValidationFail($value)
    {
        $constraint = new PasswordComplexity();

        /** @var ExecutionContextInterface|MockObject $context */
        $context = $this->getMockExecutionContext();
        $context->expects($this->once())
            ->method('buildViolation')
            ->with($constraint->message)
            ->willReturn($this->getMockConstraintViolationBuilder());

        $validator = new PasswordComplexityValidator();
        $validator->initialize($context);
        $validator->validate($value, $constraint);
    }

    public function validPasswordProvider()
    {
        return [
            ['jXb8p$cn'],
            ['!MC6NcuS'],
            ['!3%Sy6iF'],
            ['XyiWmw2#'],
            ['r@5e#hSY'],
        ];
    }

    public function invalidPasswordProvider()
    {
        return [                // conditions not meet:
            ['YQ(GJ)&'],        // at least 8 characters
            ['LuAqA=uX'],       // at least 1 digit (0 to 9)
            ['{OG>:@I1'],       // at least 1 small case letter
            [']su-(+\a'],       // at least 1 upper case letter
            ['CJuCyAGc'],       // at least 1 special character
        ];
    }

    private function getMockExecutionContext()
    {
        $context = $this->getMockBuilder(ExecutionContext::class)
            ->disableOriginalConstructor()
            ->getMock();
        return $context;
    }

    private function getMockConstraintViolationBuilder()
    {
        $constraintViolationBuilder = $this->getMockBuilder(ConstraintViolationBuilderInterface::class)->getMock();
        $constraintViolationBuilder
            ->method('setParameter')
            ->willReturn($constraintViolationBuilder);
        $constraintViolationBuilder
            ->method('setCode')
            ->willReturn($constraintViolationBuilder);
        $constraintViolationBuilder
            ->method('addViolation');
        return $constraintViolationBuilder;
    }

I have errors like:
here were 10 errors:

\Tests\Components\Validator\Constraint\PasswordComplexityValidatorTest::testValidation with data set #0 ('jXb8p$cn')
  preg_match() expects parameter 1 to be string, array given

I have no idea how to pass array samples to validator.
Comment: Do you have individual array indexed regex so that you may show specific errors ?
Comment: If so, a quicker way is to get all the shortcomings at once, then present that list of lacking characters to the user in a single string. This saves a lot of time and gives the user a complete list of what is missing in a single try. Example; if `^(?=.*(\d))?(?=.*([a-z]))?(?=.*([A-Z]))?(?=.*([!@#$%\-_*+=]))?.{8,}$` doesn't match the message is `Minimum 8 characters`. If it does match, check each capture group and construct a string: `If (match[1] == NULL ) error += "Requires at least 1 digit\n"; If (match[2] == NULL ) error += "Requires at least 1 lower case letter\n";`
Comment: con't: `If (match[3] == NULL ) error += "Requires at least 1 upper case letter\n";  If (match[4] == NULL ) error += "Requires at least 1 special character !@#$%\-_*+= \n"; if ( length( error ) > 0 ) print error;` Doing it this way removes the _annoying_ habit programmers have of piecemealing out info.
Comment: Hi sln, thanks for the idea! I have implemented this in my code witch little changes in pattern array and validate function.
Answer: I have fixed this.
My regex patterns didn't match the sample data, changed to:
public function invalidPasswordProvider()
{
    return [                // conditions not meet:
        ['J#$j#2'],        // at least 8 characters
        ['LuAqA=uX'],       // at least 1 digit (0 to 9)
        ['$OG3W4I1'],       // at least 1 small case letter
        ['3su-a+aa'],       // at least 1 upper case letter
        ['CJuC$AGc'],       // at least 1 special character
    ];
}

Pattern variable removed from PasswordComplexity.php
public $pattern = [
    '/.{8,}/',
    '/\d+/',
    '/[a-z]+/',
    '/[A-Z]+/',
    '/[!@#$%\-_*+=]+/'
];

PasswordComplexityValidator.php has been filled with validate function and all necessary data:
<?php
declare(strict_types=1);
namespace App\Components\Validator\Constraint;

use Symfony\Component\Validator\Constraint;
use Symfony\Component\Validator\Constraints\RegexValidator;

class PasswordComplexityValidator extends RegexValidator
{
    const PASS_REQ_1 = '/.{8,}/';           // at least 8 characters
    const PASS_REQ_2 = '/\d+/';             // at least 1 digit (0 to 9)
    const PASS_REQ_3 = '/[a-z]+/';          // at least 1 small case letter
    const PASS_REQ_4 = '/[A-Z]+/';          //  at least 1 upper case letter
    const PASS_REQ_5 = '/[!@#$%\-_*+=]+/';  //  at least 1 special character (e.g.: !@#$%-_*+=)

    public static $patterns = [
        self::PASS_REQ_1,
        self::PASS_REQ_2,
        self::PASS_REQ_3,
        self::PASS_REQ_4,
        self::PASS_REQ_5,
    ];

    public function validate($password, Constraint $constraint)
    {
        foreach (self::$patterns as $pattern) {
            if (!preg_match($pattern, $password, $matches))
            {
                $this->context
                    ->buildViolation($constraint->message)
                    ->setParameter('{{ string }}', $password)
                    ->addViolation();
            }
        }
    }
}

Changed also
/**
 * @dataProvider invalidPasswordProvider
 */
public function testValidationFail($value)
{
    $constraint = new PasswordComplexity();

    /** @var ExecutionContextInterface|MockObject $context */
    $context = $this->getMockExecutionContext();
    $context->expects($this->once())

to
$context->expects($this->atLeast(1))

to test password if not match more than one regex.
Test passed, I feel that everything is now as should be.
Regards,
J


How can I pad a hexadecimal digit with 0 in java?

Question: I'm trying to pad a hexadecimal digit with 0's in the beginning so that the length of it is always 4. For example, the FF is going to be padded as 00FF. I have tried to use String.format("%04d", number) but it didn't work as my hexadecimal digit is actually a string.
I have also tried using StringUtils.leftPad() but for some reason, IntelliJ couldn't detect it.
Please note my count is going to change, but I haven't represented it here as I'm yet to implement it. This is my sample code. Thanks.
    public static void main(String[] args) {
        int count = 0;
        String orderID = Integer.toHexString(count).toUpperCase();
        System.out.println(String.format("%04d", Integer.valueOf(orderID)));

    }

Comment: `String orderID = Integer.toHexString(count).toUpperCase(); String.format("%04d", Integer.valueOf(orderID))` ... What? Just get rid of the first line and pass in `count` to `String.format`. There is no reason to go from `int` to `String` back to `int` just to format another `String`.
Comment: Use `format` directly to format the integer to hex (use `X` instead of `d` in the format string).
Comment: Possibly related: [Integer to two digits hex in Java](https://stackoverflow.com/q/8689526) -> `System.out.printf("%04X", count);`. You can add `%n` at the end of format to generate line separator similarly to what `println` does: `System.out.printf("%04X%n", count);`
Comment: @tkausl, that was a method that I tried, of course, it isn't correct, and in the end, I need a string, not an int.
Answer: String.format("%04x", 255)

with output
00ff

(or use X for uppercases hex digits)
As @user16320675 point, more info about Formatter here.
Comment: it worked. I couldn't find about `X` before, thank you for your answer!


Why is meteor.js synchronous?

Question: Doesn't code take an efficiency hit by being synchronous?  Why is coding synchronously a win?
I found these two links in doing some research: http://bjouhier.wordpress.com/2012/03/11/fibers-and-threads-in-node-js-what-for/, https://github.com/Sage/streamlinejs/
If the goal is to prevent spaghetti code, then clearly you can have asynchronous code, with streamline.js for example, that isn't a callback pyramid, right?
Answer: You have to distinguish two things here:

Synchronous functions like node's fs.readFileSync, fs.statSync, etc. All these functions have a Sync in their names (*). These functions are truly synchronous and blocking. If you call them, you block the event loop and you kill node's performance. You should only use these functions in your server's initialization script (or in command-line scripts).
Libraries and tools like fibers or streamline.js. These solutions allow you to write your code in sync-style but the code that you write with them will still execute asynchronously. They do not block the event loop.

(*) require is also blocking.
Meteor uses fibers. Its code is written in sync-style but it is non-blocking.
The win is not on the performance side (these solutions have their own overhead so they may be marginally slower but they can also do better than raw callbacks on specific code patterns like caching). The win, and the reason why these solutions have been developed, is on the usability side: they let you write your code in sync-style, even if you are calling asynchronous functions.
Jan 25 2017 edit: I created 3 gists to illustrate non-blocking fibers:
fibers-does-not-block.js, fibers-sleep-sequential.js, fibers-sleep-parallel.js
Comment: "They do not block the event loop." The statement is misleading as fibers have to block the event loop in critical sections. It is up to developer to decide what  the critical sections are and use fibers appropriately.
Comment: @ashim. This is wrong: fibers are non-preemptive; context switches between fibers are implemented by libcoro and are completely decoupled from the event loop.
Comment: I am confused. If I put i/o operations inside a fiber, between fiber.run() fiber.yield() commands, then the rest of the program is going to wait until that section of code executed. How it is not blocking? Am I misunderstanding something?
Comment: Your fiber will start the I/O operation and then call `Fiber.yield()`. This yield will save the context and transfer control to the main loop  which will process all the callbacks, as usual. When your I/O callback eventually fires, it will call `fiber.run()` from the main loop to resume execution of your fiber where it yielded before. The event loop is never blocked; it runs while your code _appears_ to block on `Fiber.yield()`. Same thing with `async/await`: main loop runs everywhere your code _awaits_.
Comment: I created a small gist to illustrate callbacks firing during `Fiber.yield()`: https://gist.github.com/bjouhier/7d660269b6fae13b7e33081d391c665d
Comment: That is a great example! You might consider adding it to your original answer. Other people will benefit too. One more question though. What if we have 2 update methods to a mongodb (https://docs.mongodb.com/manual/reference/method/db.collection.update/). If we do fiber.run() only in the callback, then we block execution only after the db has been modified. Such situation is a race condition. How can fibers help here?
Comment: I don't understand what you mean by "race condition". I've created 2 more gists: one for sequential execution: https://gist.github.com/bjouhier/ab384311dc10d67e2599b367eecc3fa2  and one for parallel execution: https://gist.github.com/bjouhier/b26e69ff5ecec58dd8d6e62c5d3ebcc4. Notice how the "tick" keeps running in both cases. If you have several fibers that access a shared resource, you may need a synchronization primitive. For this I'm using a small "funnel" utility. See https://github.com/Sage/f-promise/blob/master/README.md#funnel
Answer: The code is not "synchronous" when using something like streamlinejs. The actual code will still run asynchronously. It's not very pretty to write lots of anonymous callback functions, thats where these things helps.

Why is meteor.js synchronous?


Dúvida sobre operador de decremento

Question: Tenho essas 2 linhas na minha função, mas o que seria a linha abaixo?
$idade--;

Só o return não ia ser a mesma coisa?
$idade--;
return $idade;

Só queria entender para que seria esta linha. Não gosto de usar algo que não sei pra que serve.
Código completo
Answer:
Só o return não ia ser a mesma coisa?

Não!!!
O $idade-- subtrai 1 do valor de $idade. Equivale a:
$idade = $idade - 1;

Portanto, se você retirar o $idade--, o valor retornado pela sua função será sempre 1 a mais do que ela retorna hoje.
Se quiser, pode até fazer ambas as coisas na mesma linha, mas aí precisaria usar --$idade em vez de $idade--:
return --$idade;

Isso porque o -- no final primeiro devolve o valor (atual) da variável, e depois decrementa. Já no começo ele primeiro decrementa e depois retorna. O mesmo é válido para os operadores de incremento ++ (no início ou no final).
Comment: Entendi a função agora, thank
Comment: Todo ano ele soma +1, $idade-- faz ele -1 se o mês não passou e etc.
Comment: `$idade = $idade -1`, sem o `m`, não?
Comment: @CaioFelipePereira Ops, valeu, eu não tinha visto.
Comment: Disponha. Estamos aí pra isso
Answer: Assumindo que $idade é um inteiro, $idade-- é igual a $idade = $idade - 1, assim como $idade++ é $idade = $idade + 1.
Isto é um operador de decremento. Veja mais no manual do PHP
Comment: Tentei buscar algo no manual, mas sem saber o nome é difícil, obrigado.

Dúvida sobre operador de decremento


possible number of sheets for a Moebius band covering

Question: Let M be the Moebius band, identified by the quotient of $[0,1]\times [0,1]$ by the equivalence $(x,0) \sim (1-x,1)$.
Let $p: M\to M$ be a covering and $n$ its number of sheets.
Find the possible values of $n$.
what I did:
$M$ is path-connected so it is connected, all the fibers have the same cardinality.
$M$ is also compact so $n$ is finite.
My intuition is that any $n\in \Bbb Z$ would be valid, as the possible coverings are $\Bbb R/n\Bbb Z\cong \Bbb R/\Bbb Z\times \Bbb Z/ n\Bbb Z$.
Thank you for help and comments.
Comment: Something of a side note, but that isomorphism you write is not correct. The left hand side is isomorphic to the circle, while the right is $n$ disjoint circles.
Comment: This is closely related to what we talked about in your [other question](https://math.stackexchange.com/questions/3122907/is-every-path-connected-covering-of-the-moebius-strip-a-galois-cover/3122957#3122957). Remember that the universal cover of $M$ is $\tilde{M}=\mathbb{R}\times [0,1]$ and the deck transformations act by translating and flipping. All the connected coverings of $M$ are quotients of $\tilde{M}$ by subgroups of $\mathbb{Z}$, so you should try to determine for which values of $n$ the quotient $\tilde{M}/n\mathbb{Z}$ is homeomorphic to $M$.
Answer: I think only an odd number of sheets are possible.
This is a great example where the general theory leads to interesting computational results in particular cases: we can determine possible coverings of the form $M\to M$ by first determining all connected coverings of $M$, and then detecting which ones have total space homeomorphic to $M$.

Classification Theorem: For any path-connected, locally path-connected, semi-locally simply-connected space $X$ there is a bijection between isomorphism classes of connected covering spaces of $X$ and conjugacy classes of subgroups of $\pi_1(X)$. (See for example Theorem 1.38, page 67.)

This works by constructing a universal covering $\tilde{X}\to X$ so that the quotients $\tilde{X}/H$ represent all the connected coverings of $X$ as $H$ varies over conjugacy classes. The covering $\tilde{X}$ is characterized up to covering space isomorphism by being simply-connected.
Universal covering space of $M$: Recall $M\sim S^1$ so $\pi_1(M)\cong \mathbb{Z}$. The universal covering space of $M$ is $\mathbb{R}\times [0,1]$ and the action of $\mathbb{Z}$ is given by $n\cdot (x, t)= \big(x+n, f^{n}(t)\big)$ for $n\in\mathbb{Z}$ and where $f\colon [0,1] \to [0,1]$ is the "flip" homeomorphism given by $f(t)= 1-t$. (Visually, think about $\mathbb{R}\times[0,1]$ as an infinite strip of tape that you're applying to the Möbius strip, which is alternating "front" and "back" sides.)
Quotients of $\tilde{M}$: Every connected covering of $M$ is a quotient of the form $(\mathbb{R}\times [0,1])/n\mathbb{Z}$. For each $n$ a fundamental domain of the quotient is $[0,n]\times[0,1]$, and the quotient only depends on how we identify the subspaces $\{0\} \times [0,1]$ and $\{n\}\times [0,1]$. When $n$ is odd then $f^{n} = f$ so we identify the ends using a flip, and hence the quotient is homeomorphic to $M$; on the other hand if $n$ is even then $f^{n}=id$ and so the quotient is actually the cylinder $(\mathbb{R}/n\mathbb{Z})\times [0,1]$.
Since these quotients make up all of the possible connected coverings of $M$, it follows that coverings of the form $M\to M$ can have any odd number of sheets.
Comment: Yes that's true. Since we already know $M$ is locally path-connected, so will be any covering. Therefore every connected covering of $M$ is already path-connected.
Comment: Thanks @William for your thorough answer. I see now why you say connected instead of path-connected. It boild down to the same as local path-connectedness is inherited by the coverings
Answer: Even though the question is already answered, I found an alternate argument that there are no even-sheeted covers using the first Stiefel-Whitney class of $TM$.
Recall that the tangent bundle of $M$ is non-orientable so $w_1(TM)\neq 0 \in H^1(M;\mathbb{Z}/2\mathbb{Z})$. An $n$-sheeted covering $p_n\colon M\to M$ is a local diffeomorphism and hence induces a bundle map $TM\to TM$, so by naturality of characteristic classes we have $w_1(TM) = p_n^*(w_1(TM))$. But the covering also restricts to an $n$-sheeted covering $S^1\to S^1$, which on cohomology $H^1(S^1;\mathbb{Z}) \to H^1(S^1;\mathbb{Z})$ induces multiplication by $n$. Since $S^1 \to M$ is a homotopy equivalence we also have
$$ p_n^* = n\cdot(-)\colon H^1(M;R) \to H^1(M;R) $$
for any $R$. In particular if such a cover exists when $n = 2k$ is even then $$w_1(TM) = p_n^*(w_1(TM)) = 2k \cdot w_1(TM) = 0$$ which contradicts $w_1(TM) \neq 0$.
Edit: I'm currently trying to modify this argument into a proof of the following:

Conjecture: If $M$ is a non-orientable smooth manifold then there are no even-sheeted coverings of the form $M\to M$.

Update: This general conjecture is NOT true. Consider the double-cover $$p\colon S^1\times \mathbb{RP}^2\to S^1 \times \mathbb{RP}^2$$ given by $p(z, x) = (z^2, x)$.
Comment: Thank you @William interesting alternate argument! I only studied some elementary  De Rham cohomology, so not able to understand all of it but will read about the ring cohomology. I am interested in any references.


What is the difference between overfitting and "not learning"

Question: I am trying to build a Random Forests (RF) model using around 2000 observations and a number of features (can be 50 or can 1000, I still do not know which features are to be used).
One way to evaluate the learned model is by plotting true vs. predicted values from the test set. Suppose I observe the following scenario:

On the x-axis we have the true values and on the y-axis the predicted ones. On the left we have the training set and on the right the test set.
For training we can assume the $R^2$ (regression problem) to be around 0.9 while for testing it's around 0.3/0.4. Furthermore, the Mean Absolute Error (MAE) on the test set is around 0.4 while the model predicting just the mean has a MAE of around 0.6 (with values of the target variable between 0 and 5).
My question is then: how can we say whether the model is overfitting or if the chosen features are simply not enough to make the model learn some relation with the target variable? Is it the same problem but stated in a different manner?
One way to avoid observing such a huge difference between training and test sets, is by increasing, for instance, the minimum number of samples required to be at a leaf node. This would decrease the training score to 0.7 and increase the test score to 0.5. But still, what if the problem is with the selected features?
Comment: How did you obtained the predictions on the training sample? By simply plugging the samples into the model or via OOB?
Comment: @bi_scholar I predict the target using the fitted model and the training samples.
Answer: This is overfitting, and should not be dealt with by manipulating just one aspect of prediction development such as leaf node size.  You have to look at all the possible tunings that can be done with RF, and these tuning parameters are critical.  But this avoids the main problem: RF may require up to 200 events (or 200 observations in the continuous Y case) per candidate feature.  So your sample size is very low for RF.  See this.
I am not clear why you don't know the number of candidate features in your problem.
However you decide to go forward, it is important to have unbiased estimates (e.g., from an independent holdout sample or using rigorous cross-validation) of both predictive discrimination (e.g., $R^2$) and absolute predictive accuracy (smooth calibration curve and scatterplot as you did).
Answer: Random Forest models 'overfit' by definition, however this seldom has an effect on their predictive power. When you are passing a training sample down it's corresponding random forest model, the sample will end up in the exact same terminal nodes it ended up during training. Therefore, the above left plot depicts the deviation of the sample value from the mean values of samples which share a terminal leave node with it.
Random Forests are a form of NN model and thus the conventional definition of overfitting, i.e. performance of the model on train vs. test samples, does not really make sense, as the later will almost always be lower.
Overfitting is generally not an issue in context of Random Forests and increasing the node size, pruning the trees, etc. might actually hurt the model, as it can increase the bias.
Selecting meaningful model features is another topic but again, Random Forests are relatively robust against noisy features, as the uninformative ones will simply be not selected during a split. There exist many ways of feature selection (e.g. forward/backward selection based on certain variable importance measures) but those are mostly used for model interpretation purpose.
Comment: Thanks for the answer, but I'm not sure what I can take from it! Does it mean that my descriptors are not meaningful for the task?
Comment: My point was that the performance of a random forest on it's training samples is rarely a useful metric and one should not worry about, as RFs overfit by definition.
If you want to find out whether the predictions are meaningful you should look into variable importance techniques. One approach might be to shuffle the values of a specific variable and check whether the performance of the model decreases significantly.


Create Drupal 7 Multi Site on Mac OSX 10.10

Question: I want to create a Drupal Multisite so I can share my theme and users between two sites. I want to set it up locally before I upload it on a webhost.
I am using Mac 10.10. However, many of the tutorials seem to be written for older versions of Mac OSX and don't work properly.
For example, this tutorial suggests the following:
Edit /etc/apache2/http.conf to add the following:
NameVirtualHost 127.0.0.1:80
<VirtualHost 127.0.0.1:80>
DocumentRoot /Users/YOURSITE/Sites/drupal
ServerName drupal1.localhost
</VirtualHost>

<VirtualHost 127.0.0.1:80>
DocumentRoot /Users/YOURSITE/Sites/drupal
ServerName drupal2.localhost
</VirtualHost>

Then edit  /etc/hosts to add the following:
127.0.0.1     drupal1.localhost
127.0.0.1     drupal2.localhost

You can now find your site at: http://drupal1.localhost & http://drupal2.localhost
When I do this, I just get the message "It Works" I am unable to go to the Druapl installation page to start installing the site.
I've had a search for other tutorials, but they all seem to be written for MAMP.
How can I fix it, without using MAMP?
Comment: Did you restart apache after making changes to http.conf?
Comment: @PatrickRyan. Yes, I did "sudo apachectl restart"
Comment: Did you edit the `DocumentRoot` and the `` to point to your Drupal installation?  I'm referring to the DocRoot and Directory outside of your Vhost
Comment: @PatrickRyan Which file would I edit that in. Is it located in /etc/. I am quite new to this, so I'm not sure which file path to look in. Thanks!
Comment: It should be in the same file, http.conf
Comment: @PatrickRyan In my  /etc/apache2/http.conf there is only the text I have put in my Opening post. How do I add a Document Root. Also, do I just paste in `` or do I need to add more text in between the directory tags.
Comment: Your http.conf should have a lot more than just your code in it by default.  At the very least, above all your vhost settings add `DocumentRoot "/Users/YOURSITE/Sites/drupal"` and `AllowOverride All`
Comment: @PatrickRyan Does YOURSITE contain the name of the Drupal folder that will have the multisite, or is it the Mac Account user name?I've tried both and when I visit the URL I get: 404 Not Found.
Comment: It needs to be the exact path to the folder that all the drupal files are in
Comment: @PatrickRyan Yes, I've done that but now the http://drupal1.localhost & http://drupal2.localhost says "Not Found" instead of "it Works". For the multi-site: If I go to the root Druapl directory, I get the installation page. Do I have to install the main Drupal site, before I can install the child sub sites?
Comment: I wrote an answer covering everything since comments are getting messy
Answer: It sounds like you don't have your directories configured properly.
Assuming that you have this file structure:
/users/
    /drupal/
        /sites/
            /drupal1.localhost/
                /settings.php
            /drupal2.localhost/
                /settings.php

Note that the folders under /sites must match the http.conf server name.  (unless alias's are set up).
With the above structure, your http.conf would look like this:
DocumentRoot "/users/drupal"

<Directory "/users/drupal">
    AllowOverride All
</directory>

NameVirtualHost 127.0.0.1:80

<VirtualHost 127.0.0.1:80>
    DocumentRoot "/users/drupal"
    ServerName drupal1.localhost
    <Directory "/users/drupal">
        Require all granted
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

<VirtualHost 127.0.0.1:80>
    DocumentRoot "/users/drupal"
    ServerName drupal2.localhost
    <Directory "/users/drupal">
        Require all granted
        AllowOverride All
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

Note that the document root points to the base Drupal folder, not /sites.  Drupal will handle this part automatically.
Your hosts file would remain the same:
127.0.0.1     drupal1.localhost
127.0.0.1     drupal2.localhost