On the connection of ODE's and control theory

Question: I have a hard time seeing where an ODE and its solution or a system of ODE's and its solution comes into play in control theory.
As far as I understand we generally consider som kind of mix of the "phase/state space" idea and the in- and out-put idea when we consider the control problem.
I cant find a worked example involving a box plot, the state space model and the ODE, where the states, in- and out-put etc are specified in detail. Most examples seem to just say that the equation describes this problem but not in which way.
I am therefore looking for such an example!
The state space model takes the following form in my book
\begin{equation}
\frac{dx}{dt}=f(x,u), y=h(x,u)
\end{equation}
Comment: Does your book describe a mass spring damper system as an example? that shall be typical...We could take $\vec{x} = (x,\dot{x})$ (where $x$ is the displacement) as two state variables which defines the "state of the system", so that the governing equation $m\ddot{x}+b\dot{x}+kx = u(t)$ can be converted to a state space model. (I could expand it to an answer if your book doesn't have that)
Comment: @LiChunMin it does!
Answer: Control theory concerns itself with dynamic systems. How are dynamic systems classically modelled? By a set of ordinary differential equations. Usually, the simplest type of control system you would encounter in a textbook on the subject matter is a spring-mass-damper system, as @Li Chun Min mentioned in the comments. Such a one-dimensional system, in an ideal world, would be governed by the following differential equation:
$$
M\frac{d^2y(t)}{dt^2} + C\frac{dy(t)}{dt} + Ky(t) = u(t) \quad\Rightarrow\quad M\ddot{y} + C\dot{y} + Ky = u
$$
You can think of it this way: if you apply a force $u(t)$ on the mass, its displacement will be $y(t)$. If you are wondering where this equation came from, it can be derived in classical mechanics by application of Newton's second law. The notation that I have used above is slightly different than what you would typically find in physics books: $m\ddot{x} + c\dot{x} + kx = f(x)$. This is because $f(t)$ and $x(t)$ are the input and output of your system, respectively, and they have their own notation in control theory. Finally, the transfer function of the system can be derived to be:
$$
G(s) = \frac{1}{Ms^2 + Cs + K}
$$
And now the output can be related to the input as follows:
$$
Y(s) = G(s)U(s)
$$
Now you can control your system, job done. Classical control, however, has limitations. For example, it gets tedious or nigh impossible to model systems with multiple inputs and outputs. Modern control uses state-space representation to get around those limitations. The idea is that the system has a new property called state, $x(t)$. The current state of the system and the input, $u(t)$, determine the next state, $\dot{x}(t)$. Likewise, the output, $y(t)$, depends on the current state and the input. Putting all of this information in equations yields the model in your question:
$$
\begin{align}
\dot{x} &= f(x,u) &= Ax + Bu\\
y &= h(x,u) &= Cx + Du
\end{align}
$$
Working with the same system as above (spring-mass-damper), you would have two state variables because the system is of second order. The state-space model would then be:
$$
\begin{align}
\begin{bmatrix}
\dot{x}_1 \\
\dot{x}_2
\end{bmatrix} &= \begin{bmatrix}
0 & I \\
-M^{-1}K & -M^{-1}C
\end{bmatrix} \begin{bmatrix}
x_1 \\
x_2
\end{bmatrix} + \begin{bmatrix}
0 \\ M^{-1}
\end{bmatrix} u \\
y &= \begin{bmatrix}
1 & 0
\end{bmatrix} \begin{bmatrix}
x_1 \\
x_2
\end{bmatrix}
\end{align}
$$
And there you have it. The same system modelled both classically and in state-space.
Comment: Very nice. The leap that bothers me is in some sense what @Li Chun Min adress in his comment. I.e how these to representations are related. The book I am reading (Astrom and Murray) mentions that the first form you give  "was repalced" by the latter.
Comment: Here is a link, http://www.cds.caltech.edu/~murray/books/AM05/pdf/am08-complete_22Feb09.pdf. Thier statement is on page 32
Comment: Sorry, I do not completely understand what you are asking. Is it how to translate a classical model to its state-space representation? Or perhaps the physical meaning of these representations? Be aware that the same system could be modelled in different ways when in comes to state-space. Not only do you have a choice of state variables, but there are different *forms* of the state-space representation.
Comment: I am trying to figure out how they are connected and how thier different parts are releted to eachothers representations.Differential equations have states etc aswell, but I cant seem to get things right here. Also I dont see a natural place for the $x$ or how the $x$ comes into play into the first differential equation. In the end these two diffrent things seem to be very diffrent and not at all the same, which I think the authors are impling them to be.
Comment: Both of them seem quite similar to me. They related a change in input to a change in output. Classically, this change is modelled as a differential equation, or a set thereof. Remember that differential equations are defined exactly for that purpose: to describe the rates of change of some quantities.

The state-space model is just a slightly different beast. By introducing an intermediate variable, namely the internal state, we are able to capture better the properties of the system. For example, ODE can describe only LTI systems.
Comment: At the end of the day, they are absolutely not the same thing, but they do describe the same thing. I think this is the crucial distinction.
Comment: right!! The ODE in $y$ is connected directly to the output while the state space model is related to change in $x$ which is not the output per say right?
Comment: Something like that, yes. In state-space, the output is dependent both on the internal state and on the input: $y = h(x,u)$. Therefore knowing a change in the input signal is not enough information to determine the change in the output. You also need to determine how the internal state has changed. But no matter how you model the system (ODE or state-space) an identical change in the input signal will result in the same change to the output of the system.


Is most higher level math written in English?

Question: There are many math journals published in English. Are there a lot of math journals written in Chinese, Russian, German, French, etc. they have no English translation? Are they only accessible to those who speak and read those languages? Could there be non-English written mathematics at the higher level that would not have an English counterpart? Could there be math in the world that would be "new" to English speakers?
Answer: The short answer is that there are old articles that are not available in English (especially in Russian, German or French), but nowadays basically everything is written in English, with very few exceptions (there are still some articles written in French for instance). I guess the main exception would be thesis manuscripts, that are not uncommonly written in the local language of the PhD student (especially in countries where this is mandatory).
Comment: So do Chinese universities and Russian universities require their phd student to write their thesis in English?


H2O Deep Learning R

Question: H2O Deep Learning is running regression by default even though I have ensured that the target variable is a factor (with only two levels). Any leads on how to resolve this ?
Below is the code :
<code>dnn_mod <-
  h2o.deeplearning(x = 2:321,  # column numbers for predictors
                   y = 322,   # column number for label
                   training_frame = sdcs_data, # data in H2O format
                   activation = "TanhWithDropout", # or 'Tanh'
                   input_dropout_ratio = 0.2, # % of inputs dropout
                   hidden_dropout_ratios = c(0.3,0.3,0.3), # % for nodes dropout
                   balance_classes = FALSE,
                   hidden = c(150,150,150),
                   epochs = 500,
                   #standardize = TRUE,
                   epsilon = 1.0e-5,
                   loss = "CrossEntropy",
                   stopping_rounds = 50,
                   stopping_metric = "AUC")
                   #classification = TRUE)
</code>
Comment: what do you mean by "is running regression"?
Comment: Defaults to running a regression model rather than classification. Parameters like the CrosseEntropy loss don't make sense in which case and throw an error.
Comment: So don't set loss="CrossEntropy"....
Comment: I don't think that is the point. The problem is not the error, its that H2O doesn't run classification.
Comment: Some data and some more code would be nice , aka a [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example)
Answer: If you want to run classification, then your response variable must be encoded as a "factor" (aka "enum") type.  See this R code example from the H2O Deep Learning booklet.  This is the case for all H2O algorithms.
Comment: Per my question, I have already ensured that the response is encoded as a "factor".
Comment: Are you sure that column 322 is the response column?  Please paste the output of `h2o.describe(sdcs_data[,322])` or `h2o.getTypes(data)[[322]]`.  It should say "enum".
Comment: Additionally, you can explictly set `distribution = "bernoulli"` and if you have mistakenly not converted your response to a factor, then you will see a message that includes the following: `"bernoulli distribution is not allowed for regression."`
Comment: Lastly, if you are seeing the error, `"For CrossEntropy loss, the response must be categorical."`, then that means that you have not converted the response to a factor.

H2O Deep Learning R


Determinant of the Jacobian of a holomorphic mapping of several complex variables

Question: I am reading from Degree Theory by N. Lloyd, and in one section he writes about the degree of a holomorphic map of several complex variables. I am unsure about one of the steps in a proof he gives.
The setup:
$\mathbb{C}^n$ is the vector space of complex $n$-tuples $z=(z_1,\ldots,z_n)$. $D$ is a bounded, open subset of $\mathbb{C}^n$. $\mathscr{H}(\bar{D})$ is the vector space of holomorphic mappings of $\bar{D}$ into $\mathbb{C}$; that is, $\phi=(\phi_1,\ldots,\phi_n)$ is in $\mathscr{H}(\bar{D})$ if each $\phi_j = u_j+iv_j$ is a holomorphic function of $n$ complex variables, so they satisfy the Cauchy-Riemann equations on some neighborhood of $\bar{D}$:
$$
\frac{\partial u_k}{\partial x_j} = \frac{\partial v_k}{\partial y_j},~~~
\frac{\partial u_k}{\partial y_j} = -\frac{\partial v_k}{\partial x_j},~~~
(k,j=1,\ldots,n).
$$
Now, let $p\in D$, $p\notin \partial D$. Consider a point $\zeta$ such that $\phi(\zeta)=p$. The matrix of $\phi'(\zeta)$ as a mapping of $\mathbb{C}^n$ into itself with respect to the standard basis is given by $A=(\alpha_{kj})$, where
$$
\alpha_{kj} = \frac{\partial u_k}{\partial z_j} = \frac{\partial u_k}{\partial x_j}-i\frac{\partial u_k}{\partial y_j}.
$$
So far so good.
Now, Lloyd claims that the determinant $\det \phi'(\zeta) \geq 0$. There isn't much explanation as to why, other than to consider the Jordan normal form. In fact, this doesn't even seem true for $n=1$. I know of the result that the square of this determinant is the determinant of the corresponding real Jacobian matrix, but I'm not sure if that's what the author is referring to.
I don't have much of an idea of what the the eigenvalues of a Jacobian of a holomorphic map of several complex variables should be. Would someone please explain to me why the determinant should be nonnegative? (I have minimal knowledge of several complex variables, and it is not a focus in this book.)
(More context, in case it helps: we are proving the following theorem:

Suppose $D$ is an open bounded subset of $\mathbb{C}^n$, and $\phi\in\mathscr{H}(\bar{D})$. If $p\notin\phi(\partial D)$, then $\deg(\phi,D,p)\geq 0$.

Here $\deg(\phi, D, p)$ is the degree of $\phi$ with respect to $D$ and $p$.)
Answer: Earlier in that section, the author writes: "$d(\phi,D,p)$ is defined by viewing $\phi$ as a map from a subset of $\mathbb R^{2n}$ into $\mathbb R^{2n}$." Although $A=\phi'(\zeta)$ is introduced as a complex $n\times n$ matrix, Lloyd implicitly switches to the real point of view, because the purpose of computing the Jacobian is to find the degree of $\phi$.
One does not have to go as far as the Jordan form; bringing $A$ (over complex numbers) to triangular form is good enough. When considered as a real matrix, this matrix has $2\times 2$ blocks along the main diagonal: a complex diagonal entry $\lambda=\alpha+i\beta$ is now represented by the block $$\begin{pmatrix} \alpha & -\beta \\ \beta & \alpha \end{pmatrix}$$
It is easy to see that the real determinant of $A$ is the product of determinants of these blocks, which leads to $\det_{\mathbb R}A=|\det_{\mathbb C}A|^2$.

what  the eigenvalues of a Jacobian of a holomorphic map of several complex variables should be.

They can be any $n$-tuple of complex numbers: consider the map $(z_k)\mapsto (w_k)$ defined by $w_k=\lambda_k z_k$. More generally,  every complex matrix is the Jacobian matrix of the complex linear transformation it defines.


Tomcat8 doesn't detect Letsencrypt renewed certificate (but Apache does)

Question: I have an Ubuntu 17.10 Server with Apache 2.4.27 and Tomcat 8.5.21, both serving through HTTPS thanks to a Letsencrypt Certificate. It's been working fine for 3 months and automatic renewal (with default cron) worked. But after the original expiration date, my Apache is still working fine using the renewed certificate, but Tomcat8 is still using the expired certificate, so I get the Insecure Connection warning on my browser.
I tried force-reload on my Tomcat8 service and even rebooting the whole server without success.
This is my Tomcat8 configuration for SSL on server.xml:
<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
    SSLEnabled="true" scheme="https" secure="true" clientAuth="false"
    sslProtocol="TLSv1.2" keyAlias="tomcat"
    keystoreType="JKS"
    keystoreFile="/etc/letsencrypt/live/example.com/MyDSKeyStore.jks"
    keystorePass="PASSWORD" />

If I try:
openssl s_client -connect 127.0.0.1:443 -showcerts

I see my Apache, effectively using the renewed certificate, but for Tomcat8, on port 8443:
openssl s_client -connect 127.0.0.1:8443 -showcerts

[...] verify error:num=10:certificate has expired [...]

Any idea on how can I make my Tomcat use the renewed certificate?
Update:
Thanks to comments, I've seen that the Tomcat keystore on /etc/letsencrypt/live/example.com/MyDSKeyStore.jks was not touched by the renewal. So I guess the real problem here is how to automatically update this file when my certificate renews.
Comment: what is the timestamp of `/etc/letsencrypt/live/example.com/MyDSKeyStore.jks`? Are you sure that this file is supposed to be updated by the standard Let's Encrypt renewal script?
Comment: You're right @AlexD timestamp on the file is the date of creation, not the day the renewal took place... I'll make a search on how to update this file. Thanks!
Answer: Standard Let's Encrypt certbot doesn't support JKS key store format, so your /etc/letsencrypt/live/example.com/MyDSKeyStore.jks file was created manually and needs to be updated after each renewal. You can try using certbot renew --deploy-hook to do this, check its documentation for details.
Comment: You're right. I remember creating the `jks`... I'll take a look at the docs you linked to automatize this process.


Insertion of 2016-08-29-23.17.147253 timestamp data in db2 table

Question: How can I insert 2016-08-29-23.17.147253  in db2 timestamp column?
INSERT INTO TEST  VALUES (2015,2016-08-29-23.17.147253,  5000 , 0, 0, 0, 0, 0, 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP )

CURRENT_TIMESTAMP is working fine, 2016-08-29-23.17.147253 not
Answer: Your format of timestamp seems incorrect. It must be "A timestamp is a seven-part value representing a date and time by year, month, day, hour, minute, second, and microsecond". Also, enclose the timestamp value in single quotes in INSERT statement. For more infor check :
http://www.ibm.com/support/knowledgecenter/SSEPEK_10.0.0/intro/src/tpc/db2z_datetimetimestamp.html
Answer: Try with quote like this:
    INSERT INTO TEST  VALUES (2015, '2016-08-29-23.17.147253',  5000 , 0, 0, 0, 0, 0, 0, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP )

Comment: Only adding the apostrophes as string delimiter [to the second expression in the VALUES() from the OP] is of little help, without also a proper format of the TIMESTAMP string constant [to avoid sqlcode=-180].  See [accepted] answer by bp89.


Git authentication Failed when cloning into TFS private Repo

Question: I am using TFS for my repos and all are private . moreover i have also added hostFiles into my computerOS because that is for extra security and also set my username and password in credential manager . But still no luck and whenever i use  git clone then the following error happens
fatal: Authentication failed for

kindly Help
I have tried using credential managers and have rechecked my credentails multiple times . Reinstalled Git but am unable to understand what is causing the issue
Comment: You can try the suggestions in [this ticket](https://stackoverflow.com/questions/51424051/git-clone-authentication-failed-for-url) and check if it works.
Comment: trying but not sure what am doing wrong its been 2 days since am stuck at this .
Comment: You should not have to setup a hosts file. Can you turn on tracing in git and the gcm to see what the error is? What version of TFS/Azure DevOps? Is your laptop domain joined?
Comment: https://github.com/git-ecosystem/git-credential-manager/blob/main/docs/environment.md#gcm_trace
Comment: What version of git? What version of git credential manager? Is git configured to use OpenSSL or Windows Secure Channel (SChannel)?
Comment: git version 2.42.0
Comment: Moreover am my machine was using a vpn before and now we configured to work with another VPN .  my machine is using one password for all the things like for window login , mail etc
Answer: well i have found a solution after spending much hours on this . i just reinstalled GIT and then upon installation i skipped the part for installing credential manager and i selected None. After that when i clone the Repo the Git asked for Username ad password and now everything is working Fine
Comment: It's great to know you can resolve the issue and thanks for sharing your solution. In order to let others who have the similar question with you know this answer more clearly, it's suggested that you accept the answer according to [How does accepting an answer work](https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work/5235#5235). Thanks!
Comment: yes sure . as i was stuck for two days and will definitely do it


Converter um ficheiro de texto para PDF sem gravar em um local físico

Question: Bom eu fiz um método para converter um ficheiro texto em PDF, porém ele salva o PDF na maquina. Queria que que o método retorna-se um array sem ter de gravar o PDF na maquina nem ter de ler o ficheiro novamente.
public byte[] Converte(string path)
{
    FileInfo fl = new FileInfo(path);
    fl.GetAccessControl(); //Dando acesso total ao fileinfo.
    DirectoryInfo pir = new DirectoryInfo(fl.FullName.ToString());
    string[] lines = System.IO.File.ReadAllLines(path);  // Ler todas linhas do documento
    Document doc = new Document(PageSize.A4);//criando e estipulando o tipo da folha usada
    doc.SetMargins(40, 40, 40, 80);//estibulando o espaçamento das margens que queremos
    doc.AddCreationDate();//adicionando as configuracoes
    string caminho = fl.FullName.Replace(".txt", ".pdf"); // dando replace na extensão
    PdfWriter writer = PdfWriter.GetInstance(doc, new FileStream(caminho, FileMode.Create));
    doc.Open(); //Abrindo documento
    string dados = ""; //Criando string de dados para passar dados.

    foreach (string line in lines)
    {
        Paragraph paragrafo = new Paragraph(dados); // Criando paragrafo para receber os valores das linhas.
        paragrafo.Add(line.ToString()); // Passando valor da linha do txt para pdf
        doc.Add(paragrafo); //Adicionando ao pdf
    }

    doc.Close(); // Fechando e salvando documento

    //Criando um Array de Bytes para o pdf.
    FileStream stream = System.IO.File.OpenRead(caminho);
    byte[] fileBytes = new byte[stream.Length];
    stream.Read(fileBytes, 0, fileBytes.Length);
    stream.Close();

    return fileBytes;
}

Answer: Substitua o FileStream por um MemoryStream:
using(MemoryStream memStream = new MemoryStream())
{
    PdfWriter writer = PdfWriter.GetInstance(doc, memStream);
    // o seu codigo
    return memStream.ToArray();
}


How to use the LSTM model for multi-step forecasting?

Question: I developed a time series model with LSTM. I can't use it for predicting stock price in future days. I want to use it for predicting stock price for next year and plot it. How to use it for forecasting stock price in future (next year)?
df=pd.read_csv('foolad.csv')
df=df.set_index(pd.DatetimeIndex(df['Date'].values))

data=df.filter(['Close'])
dataset=data.values

training_data_len=math.ceil(len(dataset)*0.8)
scaler=MinMaxScaler(feature_range=(0,1))
scaled_data=scaler.fit_transform(dataset)
scaled_data

training_data=scaled_data[0:training_data_len , :]

xtrain=[]
ytrain=[]
n = 60

for i in range(n,len(training_data)):
    xtrain.append(training_data[i-n:i , 0])
    ytrain.append(training_data[i,0])

xtrain , ytrain = np.array(xtrain) , np.array(ytrain)
xtrain=np.reshape(xtrain , (xtrain.shape[0],xtrain.shape[1],1))
xtrain.shape

model=Sequential()
model.add(LSTM(50,return_sequences=True,input_shape=(xtrain.shape[1],1)))
model.add(LSTM(50,return_sequences=False))
model.add(Dense(25))
model.add(Dense(1))

model.compile(loss='mean_squared_error',optimizer='adam')

model.fit(xtrain,ytrain,epochs=1,batch_size=1)

test_data=scaled_data[training_data_len - n : , :]
xtest=[]
ytest=dataset[training_data_len : , :]
for i in range(n , len(test_data)):
    xtest.append(test_data[i-n : i , 0])

xtest=np.array(xtest)
xtest=np.reshape(xtest , (xtest.shape[0],xtest.shape[1],1))

prediction=model.predict(xtest)
prediction=scaler.inverse_transform(prediction)

#for future 360 days what can I do?....

Answer: One way of doing it is to feed the forecasts back to the model as inputs: at each step you update the input sequence by dropping the oldest value and adding the latest forecast as the most recent value. This is schematically illustrated below, where n is the length of the input sequence and T is the length of the time series.

The code below shows how you could implement this approach for your LSTM model and plot the results.
import numpy as np
import pandas as pd
import yfinance as yf
import tensorflow as tf
from tensorflow.keras.layers import Dense, LSTM
from tensorflow.keras.models import Sequential
from sklearn.preprocessing import MinMaxScaler

# download the data
df = yf.download(tickers=['^IXIC'], period='5y')
y = df['Close'].fillna(method='ffill').values.reshape(- 1, 1)

# scale the data
scaler = MinMaxScaler(feature_range=(0, 1))
scaler = scaler.fit(y)
y = scaler.transform(y)

# generate the training sequences
n_forecast = 1
n_lookback = 60

X = []
Y = []

for i in range(n_lookback, len(y) - n_forecast + 1):
    X.append(y[i - n_lookback: i])
    Y.append(y[i: i + n_forecast])

X = np.array(X)
Y = np.array(Y)

# train the model
tf.random.set_seed(0)

model = Sequential()
model.add(LSTM(50, return_sequences=True, input_shape=(X.shape[1], X.shape[2])))
model.add(LSTM(50, return_sequences=False))
model.add(Dense(25))
model.add(Dense(1))

model.compile(loss='mse', optimizer='adam')
model.fit(X, Y, epochs=100, batch_size=128, validation_split=0.2, verbose=0)

# generate the multi-step forecasts
n_future = 365
y_future = []

x_pred = X[-1:, :, :]  # last observed input sequence
y_pred = Y[-1]         # last observed target value

for i in range(n_future):

    # feed the last forecast back to the model as an input
    x_pred = np.append(x_pred[:, 1:, :], y_pred.reshape(1, 1, 1), axis=1)

    # generate the next forecast
    y_pred = model.predict(x_pred)

    # save the forecast
    y_future.append(y_pred.flatten()[0])

# transform the forecasts back to the original scale
y_future = np.array(y_future).reshape(-1, 1)
y_future = scaler.inverse_transform(y_future)

# organize the results in a data frame
df_past = df[['Close']].reset_index()
df_past.rename(columns={'index': 'Date'}, inplace=True)
df_past['Date'] = pd.to_datetime(df_past['Date'])
df_past['Forecast'] = np.nan

df_future = pd.DataFrame(columns=['Date', 'Close', 'Forecast'])
df_future['Date'] = pd.date_range(start=df_past['Date'].iloc[-1] + pd.Timedelta(days=1), periods=n_future)
df_future['Forecast'] = y_future.flatten()
df_future['Close'] = np.nan

results = df_past.append(df_future).set_index('Date')

# plot the results
results.plot(title='NASDAQ')

See this answer for a different approach.
Comment: Wouldn't be better to train a model that used N historical points and predicts Y future points instead of a model that is predicting only one future value?what do you think?
Comment: Why do you fit the model on the whole X and not on a train set?
Comment: Why not using multi-step forecasting?
Comment: Same question: why not just make do a multi-step forecasting model? Is there a significant difference than doing a recursive approach?
Answer: there are multiple ways to do this ill explain three ways
first one is to use Recursive Forecasting approach
second one is to use different Window Slicing to predict different time stamp
third one the lagged values approach uses past observations (lagged values) as input features for forecasting future time points. In your example, using t-3, t-2, and t-1 to forecast t, t+1, t+2 is an instance of this approach.


Classification Problem with large number of predictors and huge training data

Question: I have a classification problem with over 700 predictor variables and training data with 200,000 observations. There is only one response variable.
I have a feature selection problem. I am looking for techniques to determine out of the 700 predictors, which are relevant for the model. I am aware of some techniques such as regularization (Lasso and Ridge) and PCA, but not sure which technique would be feasible for my dataset.
Also, are there any packages in R that can help me achieve this?
Thanks,
Gaerik
Answer: As it is a supervised problem, you can use Fisher Score for feature selection. It helps you to prune the dataset distinguishing which variables affect more to your labels (in order to differentiate them).
For feature extraction you can use LDA(Linear Discrimnant Analysis) that finds a projection that maximizes the distance between labeled groups and helps the posterior classifier algorithm. It is also useful for data visualization.
These are two examples for feature selection and feature extraction approaches. They can be useful with supervised problems but exist lots of other algorithms that help with the classification task.
Comment: Thanks! Apologies for my lack of limited understanding of this subject - I believe that the Fisher score will only provide me the set of predictors that could be important if used independently. How do I check if a combination of variables, say a and b, that are not important predictors by themselves but good predictors when used together?
Comment: Hi! When you apply any feature extraction technique the resulting projection  can be translated to a new set of features ( a kind of "combination"). You can skip the selection part(to preserve all the features) and tackle directly the extraction one. I don't know if it helps, if I'm honest I don't know any selection technique that performs what you want. Hope somedoby else could answer...
Comment: Understood! That really helps. I have some direction now and should be good to get started and try different things
Answer: There is one package Boruta in r which basically use random forest alongwith with wrapper approach to calculate the importance of different features. Feature Importance is calculated using permutation importance. You need to have "ranger" package as well to install Boruta package.
Following link might be helpful:
https://cran.r-project.org/web/packages/Boruta/Boruta.pdf
Comment: Thanks! Does does this approach provides only the most important independent features OR does it provide a set of features that work well as a combination?
Also,  with over 700 predictor variables and 200000 observations, is this computationally heavy?
Comment: This thing will provide you the importance of all the features. It will calculate score for each feature and will sort them in descending order based on there score. It might be computationally heavy.


Power rule Question

Question: From my understanding, we can't use the power rule when $x$ is being modified. So how can we use power rule on derivative of $(x+1)^2$? I know that we could expand it and find the derivative of each part (and add these together), but I receive the same answer when bring down the two and multiply through. Is this just a coincidence?
Comment: Here is a [tutorial](https://math.meta.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference) on how to typeset mathematics on this site.
Comment: Chain rule.....
Comment: No... if $\dfrac{d}{dx}(x+1)^2$ wasn't equal to $\dfrac{d}{dx}(x^2+2x+1)$ then you'd break math.
Comment: What you did works only when the derivative of the enclosed function is equal to $1$. That won't work the same if you only use the power rule to differentiate  $(5x+3)^2$, or $(x^3+5)$
Comment: @Dude156 You are using power *and* chain rules. When you are using power rule, you are not touching base $(x+1)$ but when differenting $(x+1)$ in chain rule as a second necessary operation you *can* change it.
Answer: Chain rule, let $y=x+1$,then \begin{align}\frac{d}{dx}(x+1)^2&=\frac{d}{dx}y^2\\&=\frac{d}{dy}y^2\frac{dy}{dx}\\&=2y \frac{dy}{dx}\\&=2y\\&=2(x+1) \end{align}
Of course, eventually, we no longer write down $y$ explicitly after sufficient practice.
Comment: So why can't we use a similar method when finding the derivative of sec x?
Comment: hmmm... what do you mean by similar method when finding $\frac{d}{dx} \sec x$?
Comment: Ok so say we write $sec(x)$ as $1/cos(x)$ now we replace cos(x)=y. Now we take the derivative of $y^(-1)$. This gives us $-y^(-2)$. Thus substituting back cos(x) we get $-sec(x)^2$. This is incorrect though, as the correct answer should be $tan(x)*sec(x)$
Comment: You also need to take the derivative of $\cos x$, so the derivative is $$\frac{-\sin x}{-\cos(x)^2} = \tan x \cdot \sec x$$  So $$(\sec x)' = (\frac 1{\cos x})' = (\cos x)'\cdot (\cos^{-1} x)' = \tan x \cdot \sec x$$
Answer: The answer is that you can use the power rule sometimes if the $x$ is being modified. To be as thorough as possible, I'll assume you haven't learned about Chain Rule yet.
Remember that when you find the derivative of, say, $x^3$, you're finding the rate of change of that function. Now take something like $(3x^2+1)^4$; it makes sense that the rate of change of this should depend not only on that $x^4$ nature of the function, but also on the rate of change of that part inside the parentheses. There is a more specific way to say this, called Chain Rule.
The Chain Rule says that when you have a composition of two functions, that is, $f(g(x))$, then the derivative is given by
$$\frac{d}{dx}f(g(x))=g'(x)f'(g(x))\text{, or }\frac{dg}{dx}*\frac{df}{dg}$$
This is more often remembered as "the inside times the outside". The proof for this is a bit involved, but Wikipedia has 3 different versions if you're interested.
For example, let's take the derivative of that function from before, $(3x^2+1)^4$. Chain rule says we take the derivative of the inside, $6x$, and multiply it by the derivative of the outside (ignoring the inside), which is $4(3x^2+1)^3$. This gives us our answer:
$$\frac{d}{dx}(3x^2+1)^4=6x*4(3x^2+1)^3=24x(3x^2+1)^3$$
Now we can return to your observation. Let's apply chain rule to your function, $(x+1)^2$:
$$\frac{d}{dx}(x+1)^2=1*2(x+1)^1=2x+2$$
Now you can see why the Power Rule appears to work on this: when we use Chain Rule, the derivative of the inside is just 1, so it doesn't affect the result. Instinctively, this makes sense, since the whole point of Chain Rule is to account for how much the inside changes, and $x+1$ changes just as fast as $x$.
Notice that there isn't anything special about $x+1$. We could have used anything which has a derivative of $1$, like $x+3$, or even $x+999,999$. But be careful; as soon as the derivative inside is no longer $1$, like with $2x+1$ or $\frac{x}{2}+3$, Chain Rule is no longer trivial.
EDIT: As an addendum, I'll demonstrate chain rule using the example from the comments on another answer, $\sec(x)$.
$$\frac{d}{dx}\sec(x)=\frac{d}{dx}(\cos(x))^{-1}=-\sin(x)*-1(\cos(x))^{-2}=\frac{\sin(x)}{\cos^2(x)}=\tan(x)\sec(x)$$
Answer: Claim
$$
\frac{d}{dx}(x+c)^n=n(x+c)^{n-1}
$$
for all $c\in\mathbb{R}$. Indeed by the binomial theorem
$$
(x+c+h)^n=(x+c)^n+n(x+c)^{n-1}h+\binom{n}{2}(x+c)^{n-2}h^2+\dotsb+h^n
$$
so
$$
\begin{align}
\lim_{h\to0}\frac{(x+c+h)^n-(x+c)^n}{h}&=\lim_{h\to0}\,[n(x+c)^{n-1}+\binom{n}{2}(x+c)^{n-2}h+\dotsb+h^{n-1}]\\
&=n(x+c)^{n-1}
\end{align}
$$

Power rule Question


Anchor links to paragraph content

Question: I have some drupal 8 sites that make heavy use of the Paragraphs and Bootstrap Paragraphs modules.
Some of my users want to be able to create anchor style links to specific paragraph items on pages. For example; a user may wish to link directly to a specific accordion that is halfway down the page. They may wish to do this either from the body of the same page or with an anchor style hyperlink from elsewhere on the site. eg: mysite.com/mypage#myanchor
Other than adding a "name" field and creating a specific "anchor link" twig override for every paragraph type and field is there a more straightforward way of doing this?
In messing around with the anchor_link module It seems it can only find anchors in the body of a node. If you add an anchor to say, the body of an accordion the link function cannot see it.
Answer: First you need to set up an ID for the paragraphs. You can do this in oko_preprocess_paragraph():
/**
 * Implements template_preprocess_paragraph().
 */
function THEME_preprocess_paragraph(&$variables) {
  $variables['paragraph_id'] = $variables['paragraph']->id();
}

Now, you can output the ID of the paragraph using {{ paragraph_id }} in any paragraph templates you have. So you can copy paragraph.html.twig to your theme, and change this:

To this:

Comment: For a one file solution you could also add the ID to `attributes` in preprocess or access {{ paragraph.id }} in the template directly.
Comment: Nowdays, id must be collected with id() method, so it becomes: {{ paragraph.id() }}.
Comment: Nice. You should put that as an answer, I'll upvote it.

Anchor links to paragraph content


Deserialize variable meta object with serde

Question: I am looking an elegant way to deserialize the following input:
{
  "products": [

    {
      "id": 1,
      "ptype": "Clothes",
      "description": "some data about clothes",
      "metadata": {
        "colors" : ["blue", "green"],
        "web": false,
        "size": 2
      }
    },
    {
      "id": 4,
      "ptype": "Food",
      "description": "text for foods",
      "metadata": {
        "country": "France",
        "wine": true
      }
    },
    {
      "id": 12,
      "ptype": "EmptyPlaceholder",
      "description": "nothing at all",
      "metadata": {
      }
    }
  ]
}

The json contains an array of products. A product can be  identified by a ptype field. According to the type of the field the metadata object differs. For example if the ptype is Food, then the metadata for food will be a string (country) and a boolean value (wine). So the products have some fields in common, id, ptype and description and some metadata. I want to deserialize this JSON file in a Vec<Product>.
So far I have used the following code:
use serde::{Deserialize};
use serde_json::Result;

#[derive(Deserialize, Debug)]
struct ClothesData {
    colors : Vec<String>,
    web : bool,
    size: u32,
}

#[derive(Deserialize, Debug)]
struct FoodData {
    country: String,
    wine: bool,
}

#[derive(Deserialize, Debug)]
struct EmptyData {

}

#[derive(Deserialize, Debug)]
enum Metadata {
    ClothesData,
    FoodData,
    EmptyData,
}

#[derive(Deserialize, Debug)]
enum  Ptype {
    Clothes,
    Food,
    EmptyPlaceholder
}

#[derive(Deserialize, Debug)]
struct Product {
    id: u32,
    ptype: Ptype,
    description: Option<String>,
    metadata: Metadata,

}

I am not sure how to proceed from here and I would like to ask if serde crate can do this "automatically".
Answer: This corresponds to the "adjacently tagged" serde enum representation:
use serde::Deserialize;
use serde_json::Result;

#[derive(Deserialize, Debug)]
struct Clothes {
    colors: Vec<String>,
    web: bool,
    size: u32,
}

#[derive(Deserialize, Debug)]
struct Food {
    country: String,
    wine: bool,
}

#[derive(Deserialize, Debug)]
struct Empty {}

#[derive(Deserialize, Debug)]
#[serde(tag = "ptype", content = "metadata")]
enum Kind {
    Clothes(Clothes),
    Food(Food),
    EmptyPlaceholder(Empty),
}

#[derive(Deserialize, Debug)]
struct Product {
    id: u32,
    description: Option<String>,
    #[serde(flatten)]
    kind: Kind,
}

#[derive(Deserialize, Debug)]
struct Root {
    products: Vec<Product>,
}

fn main() -> Result<()> {
    let data = r#"
    {
        "products": [

            {
              "id": 1,
              "ptype": "Clothes",
              "description": "some data about clothes",
              "metadata": {
                "colors" : ["blue", "green"],
                "web": false,
                "size": 2
              }
            },
            {
              "id": 4,
              "ptype": "Food",
              "description": "text for foods",
              "metadata": {
                "country": "France",
                "wine": true
              }
            },
            {
              "id": 12,
              "ptype": "EmptyPlaceholder",
              "description": "nothing at all",
              "metadata": {
              }
            }
        ]
    }"#;

    let root: Root = serde_json::from_str(data)?;
    println!("{:#?}", root);

    Ok(())
}


How do I temporarily pause, stop, block, or deny users from logging into my Windows Server 2016 application server during maintenance?

Question: I have an application server into which multiple users log in daily, but I want to keep them off while I am working on some maintenance tasks. How can I do that?
Comment: disable the remote access and use the local console from yozr virtual environment. i used to disable the primary network card and use the management interface to have my maintenance
Answer: The trick is to open the Local Group Policy Editor on your application server and deny them access there. You can then restore access when finished. To do so, simply,

Click Start
Type in Run while the Start menu is up
Select Run from the Apps menu
Type in gpedit.msc on the Run dialog box's Open field and press Enter
When the Local Group Policy Editor window appears, navigate using the left hand side tree menu to Computer Configuration | Windows Settings | Security Settings | Local Policies | User Rights Assignment
In the list of policies that appear on the right hand side, scroll down to the policy titled, "Deny access to this computer from the network" and double-click it

Click Add User or Group... in the pop-up dialog box that appears

Type in the individual username you wish to block, or better yet, the name of an applicable Active Directory group whose members you wish to block, and click Check Names

You can add multiple users or groups here, and once you've spelled the user or group names right and are done, just click OK

All the users you've blocked can have their access easily restored by simply following the same instructions above and on step 8, simply remove their names from the list!


Convert dataframe types to Python types or just a way to serialize

Question: I want to take a generic dataframe (just an example):
my_data_frame = pandas.dataFrame({"numeric":5}, index=[0])

and just get [{"numeric": 5}], then serialize it and send it over the wire.
I tried doing this:
values = my_data_frame.to_dict('records')

which does convert it to [{"numeric": 5}], but the 5 is an 'numpy.int64', and isn't serializable. Is there some way I can get a dataframe into this format, but with Python types instead of Numpy types, and in a generic enough way it works for all dataframes (may have strings or datetimes instead of numeric)
Answer: You can just use the data frames' to_json method which directly converts the data frame to json, so you don't have to worry about the serialization process:
my_data_frame.to_json(orient="records")
# '[{"numeric":5}]'


Como importar arquivo MDB ler e gravar no SQL

Question: Estou tentando importar um arquivo MDB, ler este arquivo e gravar no SQL.
Já estou conseguindo importar o arquivo e ler, só não consigo ainda pegar estes dados e gravar no SQL.
Se reparar no código eu até consegui mostrar os dados em um Lista.
Segue abaixo o meu código:
<code>protected void Btn_Importar(object sender, EventArgs e)
        {
            if (FileUpload1.HasFile)
            {
                if (FileUpload1.PostedFile.ContentType == "application/msaccess")
                {
                    string filename = Path.GetFileName(FileUpload1.FileName);
                    FileUpload1.SaveAs(Server.MapPath("~/Upload/") + filename);
                    Label1.Text = "File uploaded successfully!";
                    //ReadMdb();
                    Insert();
                }
            }
        }

public void Insert()
        {
            string strFile = Server.MapPath("~/Upload/teste.mdb");
            string connectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + strFile;
            var myDataTable = new DataTable();
            using (var connection = new OleDbConnection("Provider=Microsoft.JET.OLEDB.4.0;" + "data source=" + strFile))
            {
                connection.Open();
                var query = "SELECT * FROM BOLETO";
                var command = new OleDbCommand(query, connection);
                var reader = command.ExecuteReader();

                Conexaocs con = new Conexaocs();
                con.Conexao();

                SqlCommand sqlComm = new SqlCommand("INSERT INTO BOLETO (CODIGO,NF_CONTA,TEXTO) VALUES (@CODIGO, @NF_CONTA, @TEXTO)", conn);

                //if (reader.HasRows)
                //{
                //    while (reader.Read())
                //    {
                //        //ListBox1.Items.Add(reader.GetInt32(0).ToString() + " - " + reader.GetString(1));
                //    }

                for (int i = 0; i < reader.FieldCount; i++)
                {
                    sqlComm.Parameters.AddWithValue("@CODIGO", reader[i]);
                    sqlComm.Parameters.AddWithValue("@NF_CONTA", reader[i]);
                    sqlComm.Parameters.AddWithValue("@TEXTO", reader[i]);
                    sqlComm.ExecuteNonQuery();
                }

                connection.Close();
            }
        }
</code>
Eu coloquei apenas estas colunas para teste.
Answer: Tente o seguinte. Só não esqueça de adaptar seu INSERT com todas as colunas/campos disponíveis.
<code>     public void Insert()
     {
        string strFile = Server.MapPath("~/Upload/teste.mdb");
        string connectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" + strFile;
        var myDataTable = new DataTable();
        using (var connection = new OleDbConnection("Provider=Microsoft.JET.OLEDB.4.0;" + "data source=" + strFile))
        {
            connection.Open();
            var query = "SELECT * FROM BOLETO";
            var command = new System.Data.OleDb.OleDbCommand(query, connection);
            var reader = command.ExecuteReader();

            Conexaocs con = new Conexaocs();
            con.Conexao();

            if (reader.HasRows)
            {
                while (reader.Read())
                {
                    // Prepara um comando para cada insert
                    SqlCommand sqlComm = new SqlCommand("INSERT INTO BOLETO (CODIGO,NF_CONTA,TEXTO) VALUES (@CODIGO, @NF_CONTA, @TEXTO)", conn);

                    // Loop com os campos
                    for (int i = 0; i < reader.FieldCount; i++)
                    {
                        // Adiciona um parametro para cada campo
                        sqlComm.Parameters.AddWithValue("@" + reader.GetName(i), reader[i]);
                    }

                    // Executa com os parâmetros adicionados
                    sqlComm.ExecuteNonQuery();
                }

            }

            connection.Close();
        }
    }
</code>


Is there a statistical differing interpretation between match and fit?

Question: I've seen the terms matching/match and fitting/fit used almost interchangeably when discussing models goodness of fit, observed vs simulated data matching, etc.
Is there a difference in interpretation between these terms that should be taken into account or are they more or less synonyms (in the strict statistical sense)?
Comment: Do you have any citations where this occurs?  I am interested in seeing where they actually interchange the two.
Comment: Hi Dan, I don't have _specific_ citations since this question was mostly based on my own recollections of having seen it somewhere and my own need to clearly separate the definition of those words. You can check this link http://goo.gl/7n0rJC for a list of articles with both words used in the abstract of astronomic articles.
Answer: Those terms mean different things (at least in the statistics community) and should not be used interchangeably.

Matching is a statistical technique which is used to evaluate the effect of a treatment by comparing the treated and the non-treated units in an observational study or quasi-experiment (i.e. when the treatment is not randomly assigned). The goal of matching is, for every treated unit, to find one (or more) non-treated unit(s) with similar observable characteristics against whom the effect of the treatment can be assessed. By matching treated units to similar non-treated units, matching enables a comparison of outcomes among treated and non-treated units to estimate the effect of the treatment without reduced bias due to confounding.

Fitting is the process of constructing a curve, or mathematical function, that has the best fit to a series of data points, possibly subject to constraints. Curve fitting can involve either interpolation, where an exact fit to the data is required, or smoothing, in which a "smooth" function is constructed that approximately fits the data. A related topic is regression analysis, which focuses more on questions of statistical inference such as how much uncertainty is present in a curve that is fit to data observed with random errors. Fitted curves can be used as an aid for data visualization, to infer values of a function where no data are available, and to summarize the relationships among two or more variables.


Recursively Create Binary Tree at Specific Depth

Question: I want to create a binary tree with a specific depth.
My code so far creates binary trees till a specific maximum depth, but also creates trees with lower maximum depths. I will illustrate my problem below.
My code so far (Method is called create):
public class BT<E>{
   E value;
   BT<E> left, right;

   public BT(E value)
   {
      this.value=value;
   }

   public BT (E value, BT left, BT right)
   {
      this.value = value;
      this.left = left;
      this.right = right;
   }

   private static final String[] random = {"a","b","c","d"};
   public static BT create(int depth) {
      if (depth > 1 && random.nextBoolean())//if not at the maximum depth
      //choose randomly a,b,c,d - nextBoolean ensures it is not balanced all the time
      {
         String random = OPERATORS[random.nextInt(OPERATORS.length)];
         return new BT(operator, create(depth-1), create(depth-1));//recursively create tree
      }
      else {//when node is a leaf node, make it X
         String t = "x";
         return new BT(t);
      }
   }
}

The problem:
If I input 3 as the depth it should create trees like this ONLY:
    (c)              (d)         (a)
 (b)   (a)         (b)        (d)   (c)
(x)(x)(x)(x)     (x)  (x)             (x)

My code currently creates the trees as above but also includes trees like:
(x)       (c)           (d)
             (x)      (x) (x)

Clearly these aren't of depth 3. I do not want this. I only want the former without these exceptions.
Could somebody have a look at my code and tell me what I am doing wrong and how to correct it.
Comment: Your definition of depth is inconsistent with the way that I have seen it defined.  Depth is the maximum length (or number of edges) from root to a leaf.  The trees that you say should be displayed when depth = 3 actually have a depth of 2.
Answer: the easiest approach would be to create a wrapper function which calls your create tree function.
The function will create a random free using your function, test to see if it has the needed length of depth X. If it is, it will return; if not, it will call your function again until it finds a tree with correct depth
Answer: The problem is:

random.nextBoolean()

in this line:
if (depth > 1 && random.nextBoolean())

You're intending to create unbalanced trees with this statement, but this also has a very high chance of creating trees without any maximum depth branches. All you need is for false to be returned a few times in a row early on in the recursion and you have your undesired result. Given that there's a 50% chance for this to occur it will happen more than you think! To guarantee at least one branch with maximum depth you need a little more processing.
One suggestion could be to pre-determine one or more branches in your tree that will reach maximum depth, and leave the others to chance. As you're using a binary tree this can be done with booleans:
int depth = 3;
boolean[] branch = new boolean[depth];
for (int i = 0; i < branch.length; i++) {
  branch[i] = random.nextBoolean();
}

Once you have this, generate your tree as you're doing now. Once complete, move through your tree as the boolean[] dictates, using true for left and false for right. If you find that the child is missing for a particular boolean, and you haven't hit maximum depth, create that child and continue. This isn't a particularly attractive way of doing it, but it's cheap and fast.
Once this is working you can add more complexity - more guaranteed branches, and additional branches coming from those - in order to flesh out the tree. As your maximum depth increases it might be wise to lower the chance of ending a branch, as otherwise you'll end up with one long maximum depth branch and loads of smaller ones.
Answer: As already mentioned the problem is in this line:
if (depth > 1 && random.nextBoolean())

What you have to do is:

Control the evaluation order:
Consider return new BT(operator, create(depth-1), create(depth-1));
The left tree will always be evaluated first and then the right tree.
Let the trees create leafs, if one of them has reached the desired depth:
I created a boolean variable finished for this purpose

So first we will create a path with the desired depth. Then we will let the other paths either create a Tree or a Leaf.
    private boolean finished;

    public BT<E> create(int depth){
        if(depth<=0) { // One tree is at desired depth
            finished = true;
            return new BT<E>(null);
        }
        if(finished) // Let the other trees set a Leaf.
            if(random.nextBoolean())
                return new BT<E>(null);

        BT<E> temp = create(depth-1); // Control evaluation
        if (random.nextBoolean()) // evaluate left and right evenly distributed
            return new BT<E>(null, temp, create(depth-1));
        else return new BT<E>(null, create(depth-1), temp);

Note that the created tree will rather be sparse than dense, because of
if(finished)
    if(random.nextBoolean())
        return new BT<E>(null);

You might want to make the random variable depend on the number of current paths.


How to make React Redux async action return a promise?

Question: I have a redux action function that returns a promise. Inside that function body, there is another async function that returns a callback when it finished the call. Then I call this function in another place chaining it with .then() but when debugging it in the browser with breakpoints, the promise function exists after the first two lines of code
return new Promise((resolve, reject) => {
    return (dispatch, getState) => {

is it because of the second return statement? This is the react/redux code so I must have a second return statement for my async Thunk redux action. How can I achieve this? So that I could call it in another redux action and chain it with handleProfileImageUploadToS3().then(() => {...}) ?
Full function body:
export const handleProfileImageUploadToS3 = () => {
  return new Promise((resolve, reject) => {
    return (dispatch, getState) => {
      const settingsState = getState().BusinessSettings
      const userState = getState().User
      const imageUpload = true

      if (!settingsState.get('logoImage') || settingsState.get('logoImage') === null) {
        reject('no image selected')
        return
      }
      Utilities.uploadFileToS3(
        imageUpload,
        settingsState.get('logoImage'),
        'apiurl',
        `profiles/${userState.get('id')}`,
        (error, url) => {
          if (error) {
            dispatch(uploadProfileSettingsImageError(error))
            return
          }
          dispatch(updateProfileImageUrlAfterUpload(url))
          resolve(url)
        }
      )
    }
  })
}

Answer: I'm not seeing your whole codebase but there are some red flags in this function that I'm concerned about.

The BusinessSettings and User properties of your state seem to be non-serializable objects with a get() method.
If the S3 upload gives an error then you dispatch and return but you never resolve or reject the Promise.
The rejection from reject('no image selected') is unlikely to be caught anywhere.
It seems like an un-uploaded image should be something that is stored in your UI state and passed as an argument rather than stored in Redux.  You would want to store the URL after it's uploaded.  This also removes the need for that rejection.

You are addressing two separate concerns in this function and I would recommend that you separate the two.
First is that you have a function Utilities.uploadFileToS3 which uses a success/failure callback and you want to convert it to an async function (one that returns a Promise).
I would make a helper that takes just the arguments which vary and ignores those that are constant (like 'apiurl').
const asyncUploadFileToS3 = (image, userId) => {
  return new Promise((resolve, reject) => {
    Utilities.uploadFileToS3(
      true,
      image,
      "apiurl",
      `profiles/${userId}`,
      (error, url) => (url ? resolve(url) : reject(error))
    );
  });
};

Now that you have that part settled, you can approach writing the thunk in a more typical way.  You can return a Promise by .then() chaining or by making the function async and using a try/catch block.  By we don't need to wrap the whole thing in a new Promise because we've dealt with that on the asyncUploadFileToS3 function.
You can return a result from the thunk and chain it, but I'm not sure what actually makes the most sense here.
export const handleProfileImageUploadToS3 = (image) =>
  async ( dispatch, getState ) => {
    const userId = getState().user.id;
    try {
      const url = await asyncUploadFileToS3(image, userId);
      dispatch(updateProfileImageUrlAfterUpload(url));
      return "this is the result";
    } catch (error) {
      dispatch(uploadProfileSettingsImageError(error));
      return "there was an error";
    }
  };

export default function App() {
  const dispatch = useDispatch();

  const avatar = useSelector((state) => state.user.imageUrl);

  const onClick = () => {
    const image = new Blob();
    dispatch(handleProfileImageUploadToS3(image)).then(
      // logs "this is the result" or "there was an error"
      (returned) => console.log("finished", returned)
    );
  };

  return (

      <button onClick={onClick}>Upload</button>
      {avatar ?
Avatar URL: {avatar}
 :
No Avatar
}

  );
}

Code Sandbox Demo
Comment: Thanks for a comprehensive answer. There is a lot to absorb.
"It seems like an un-uploaded image should be something that is stored in your UI state and passed as an argument rather than stored in Redux." I actually do that. I store the image blob in a local state and only save it to redux store on button click to save. I save the File object in redux store that I use to upload to S3, get the url back, update the store from File object to a generated URL from S3 and then call another action to save that URL to database.
Answer: After some more research I found out that you must return another function from initial Promise function before returning new Promise to have access to Redux dispatch and getState methods, instead of returning it as anonymous function right inside new Promise() code block which then just exits the function execution. Here is the full code:
export const handleProfileImageUploadToS3 = () => (dispatch, getState) => {
  return new Promise((resolve, reject) => {
    const settingsState = getState().BusinessSettings
    const userState = getState().User
    const imageUpload = true

    if (!settingsState.get('logoImage') || settingsState.get('logoImage') === null) {
      reject('no image selected')
      return
    }
    Utilities.uploadFileToS3(
      imageUpload,
      settingsState.get('logoImage'),
      'apiurl',
      `profiles/${userState.get('id')}`,
      (error, url) => {
        if (error) {
          dispatch(uploadProfileSettingsImageError(error))
          return
        }
        dispatch(updateProfileImageUrlAfterUpload(url))
        resolve(url)
      }
    )
  })
}

Comment: Do you need a `resolve` or `reject` inside of the `if(error)`?  Seems like it might hang there forever.  If it were me personally I would address the two issues separately.  Those issues being 1) you need to create a `Promise` wrapper around the  `Utilities.uploadFileToS3` function since it's not `async` and 2) you need to create a thunk dispatch that uses that function.  When you `reject` here with "no image selected", is that caught anywhere?
Comment: `settingsState` is a piece of your Redux state, but you are calling `settingsState.get('logoImage')` which is very strange to me because your Redux state should be raw serializable data.  It should not have a `get()` method.  (same for `userState.get('id')`).
Comment: @LindaPaiste It doesn't hang in `Utilities.uploadFileToS3() `callback, by using breakpoints I found that the function was exeting at the second line of code which is return `(dispatch, getState) => {}` I'm still not sure why though. And `Utilities.uploadFileToS3()` is an async function on its own but it uses callback instead of promises. Should I use reject inside its callback as well if it fails?
"2) you need to create a thunk dispatch that uses that function" - what is the difference between using it as I do right now versus dispatching it as an action of its own? Works well as it is now.
Comment: @LindaPaiste I'm not sure regarding the redux store calling, we do it like this all over the project and I was introduced to this project very recently, so I'm not sure what are the good practices of redux. This is how that reducer looks:
`const getInitialState = () => {
  return Map({
    logoImage: '',
    brandName: '',
    website: '',
    countryCode: '',
    currency: '',
    vat: '',
  })
}`
Comment: It is not a good idea to use a `Map`.  Just store that  object without the `Map` around it.  [With Redux, our application state is always kept in plain JavaScript objects and arrays. That means you may not put other things into the Redux state - no class instances, built-in JS types like Map / Set Promise / Date, functions, or anything else that is not plain JS data.](https://redux.js.org/tutorials/fundamentals/part-3-state-actions-reducers#designing-the-state-structure)
Comment: Per your other comment about separating the two issues, that's more of a code style / design thing.  It's not inherently a problem but I think the way I've written it in my answer is better.


Calculate effects on survival time from Cox proportional hazard model?

Question: Novice stats question here:
When I run a cox model over daily survival data as a function of 3 covariates, the results:
Fixed coefficients
             coef exp(coef)    se(coef)      z       p
var1  -0.02306065 0.9772032 0.004961884  -4.65 3.4e-06
var2  -0.11755938 0.8890877 0.007519678 -15.63 0.0e+00
var3  0.01145542 1.0115213 0.032680554   0.35 7.3e-01

show that var1 and var2 increase survival time and the effects are statistically significant. I further interpret this to mean that an additional unit increase in var1 reduces the daily hazard by a factor of 0.977 on average -- that is, by 2.3 percent. Similarly, each unit increase in var2 reduces the hazard by a factor of 0.889, or 11.1 percent.
My question is: can I convert these interpretations into something like

each unit increase in var1 increases mean survival time by n days
on average, or
each unit increase in var2 increases the probability that survival will exceed 2 days by p%

?
Answer: The advantage in Cox regression--that you don't need to specify a form for the baseline hazard as a function of time--becomes a disadvantage in this case.
You can't make general statements about mean survival times or survival probabilities at specific times from the coefficients of a Cox model unless you also specify the corresponding baseline hazard. This isn't particularly difficult, as survival software will typically provide ways to make predictions based on the model and the empirical baseline hazard that the model estimated. But it's not so simple as you had hoped.
Even with parametric survival models there often won't be a simple formula depending just on the covariate values. In those models the full set of coefficients (including those estimated for the parameters of the baseline survival function) would contain the needed information.
If you prefer to think in terms of relative times to failure instead of instantaneous hazards, you might consider instead an accelerated failure time (AFT) parametric model based for example on a Weibull or lognormal distribution. Except with a Weibull you would no longer be talking about hazard ratios as other AFT baseline functions don't follow proportional hazards. If the model fits well, however, the coefficients for the covariates are easier to interpret in terms of changes of expected event times.
Answer: You cannot derive absolute changes or differences in time directly from the hazard ratios. You can create fitted mortality or survival curves and consider differences in survival or mortality at a common time point or compare the time until a common cumulative mortality or survival is encountered.
While reductions in hazard are commonly referred to as percentages, these ratios represent multiplicative change in risks that are often non- linear. I recommend avoiding a percentage interpretation.


Actual source line in ORA-06550 message

Question: When developing an Oracle PL/SQL procedure in Toad, I encounter errors like

ORA-06550: line 97, column 25: PLS-00330: invalid use of type name or subtype name
ORA-06550: line 97, column 9: PL/SQL: Statement ignored

These given line numbers and column numbers seem to have little or nothing to do with source code line numbers.  Where can I find tips or tricks that will help me identify the actual source line or actual code that caused the error?  This is especially important in situations like the above (which is the result of a 700+ line source) where the error message doesn't provide clues as to details of the actual problem.  (By the way, line 101 in that source is the third of a series of comment lines, and is followed by a blank line.)
(Note that this is not [yet] in a stored procedure; it is being developed and executed from source at this time.  Not sure if that makes a difference since at present there is no database in which we developers have authority to CREATE PROCEDURE.)
Sample code:
SET SERVEROUTPUT ON;

DECLARE
TYPE bendemo_hdr IS RECORD
(
    recid      CHAR(1)
  , client     CHAR(30)
  , filedesc   CHAR(30)
  , seqnum     CHAR(2)
  , crtdate    CHAR(20)
  , srtdate    CHAR(8)
  , stpdate    CHAR(8)
);

TYPE bendemo_record IS RECORD
(
    recid        CHAR(1)
  , ssn          CHAR(9)
  , empnum       CHAR(15)
  , eeflag       CHAR(1)
  , titlecode    CHAR(6)
  , fname        CHAR(30)
  , mname        CHAR(30)
  , lname        CHAR(30)
  , namesuffix   CHAR(6)
  , prefname     CHAR(30)
  , dob          CHAR(8)
  , dod          CHAR(8)
  , dverdte      CHAR(8)
  , dauddte      CHAR(8)
  , ddtesrc      CHAR(6)
  , gender       CHAR(1)
  , martstat     CHAR(6)
  , mstateffdt   CHAR(8)
  , lang         CHAR(1)
  , citzcde      CHAR(6)
  , vipflag      CHAR(1)
  , kyeeflag     CHAR(1)
  , orghrdte     CHAR(8)
  , lathrdte     CHAR(8)
  , adjhrdte     CHAR(8)
  , jobtitle     CHAR(30)
  , precntmthd   CHAR(6)
  , mailpref     CHAR(6)
  , phnepref     CHAR(6)
  , emalpref     CHAR(6)
  , hmaddrefdt   CHAR(8)
  , hmaddr1      CHAR(60)
  , hmaddr2      CHAR(60)
  , hmaddr3      CHAR(60)
  , hmaddr4      CHAR(60)
  , hmaddcity    CHAR(60)
  , hmaddstate   CHAR(60)
  , hmaddzip     CHAR(10)
  , hmaddcnty    CHAR(3)
  , hmphcnty     CHAR(6)
  , hmphnbr      CHAR(16)
  , hmphext      CHAR(4)
  , mbphcnty     CHAR(6)
  , mbphnbr      CHAR(16)
  , faxcnty      CHAR(6)
  , faxnbr       CHAR(16)
  , hmemail      CHAR(100)
  , wkaddrefdt   CHAR(8)
  , wkaddr1      CHAR(60)
  , wkaddr2      CHAR(60)
  , wkaddr3      CHAR(60)
  , wkaddr4      CHAR(60)
  , wkaddcity    CHAR(60)
  , wkaddstate   CHAR(60)
  , wkaddzip     CHAR(10)
  , wkaddcnty    CHAR(3)
  , wkphcnty     CHAR(6)
  , wkphnbr      CHAR(16)
  , wkphext      CHAR(4)
  , wkemail      CHAR(100)
  , hiresrc      CHAR(30)
  , bgnunit      CHAR(30)
  , qdroflag     CHAR(1)
  , hiresrcdt    CHAR(8)
);

TYPE bendemo_trlr IS RECORD
(
    recid      CHAR(1)
  , client     CHAR(30)
  , filedesc   CHAR(30)
  , reccount   CHAR(9)
  , field1     CHAR(15)
);

demo_hdr    bendemo_hdr;
demo_rec    bendemo_record;
demo_trlr   bendemo_trlr;
i           NUMBER;

PROCEDURE dump_hdr_rec IS
BEGIN
    DBMS_OUTPUT.put(bendemo_hdr.recid);
    DBMS_OUTPUT.put(bendemo_hdr.client);
    DBMS_OUTPUT.put(bendemo_hdr.filedesc);
    DBMS_OUTPUT.put(bendemo_hdr.seqnum);
    DBMS_OUTPUT.put(bendemo_hdr.crtdate);
    DBMS_OUTPUT.put(bendemo_hdr.srtdate);
    DBMS_OUTPUT.put(bendemo_hdr.stpdate);
    DBMS_OUTPUT.put_line('<');
END dump_hdr_rec;

PROCEDURE dump_demo_rec IS
BEGIN
    --      IF l_output IS NULL THEN
    DBMS_OUTPUT.put(bendemo_record.recid);
    DBMS_OUTPUT.put(bendemo_record.ssn);
    DBMS_OUTPUT.put(bendemo_record.empnum);
    DBMS_OUTPUT.put(bendemo_record.eeflag);
    DBMS_OUTPUT.put(bendemo_record.titlecode);
    DBMS_OUTPUT.put(bendemo_record.fname);
    DBMS_OUTPUT.put(bendemo_record.mname);
    DBMS_OUTPUT.put(bendemo_record.lname);
    DBMS_OUTPUT.put(bendemo_record.namesuffix);
    DBMS_OUTPUT.put(bendemo_record.prefname);
    DBMS_OUTPUT.put(bendemo_record.dob);
    DBMS_OUTPUT.put(bendemo_record.dod);
    DBMS_OUTPUT.put(bendemo_record.dverdte);
    DBMS_OUTPUT.put(bendemo_record.dauddte);
    DBMS_OUTPUT.put(bendemo_record.ddtesrc);
    DBMS_OUTPUT.put(bendemo_record.gender);
    DBMS_OUTPUT.put(bendemo_record.martstat);
    DBMS_OUTPUT.put(bendemo_record.mstateffdt);
    DBMS_OUTPUT.put(bendemo_record.lang);
    DBMS_OUTPUT.put(bendemo_record.citzcde);
    DBMS_OUTPUT.put(bendemo_record.vipflag);
    DBMS_OUTPUT.put(bendemo_record.kyeeflag);
    DBMS_OUTPUT.put(bendemo_record.orghrdte);
    DBMS_OUTPUT.put(bendemo_record.lathrdte);
    DBMS_OUTPUT.put(bendemo_record.adjhrdte);
    DBMS_OUTPUT.put(bendemo_record.jobtitle);
    DBMS_OUTPUT.put(bendemo_record.precntmthd);
    DBMS_OUTPUT.put(bendemo_record.mailpref);
    DBMS_OUTPUT.put(bendemo_record.phnepref);
    DBMS_OUTPUT.put(bendemo_record.emalpref);
    DBMS_OUTPUT.put(bendemo_record.hmaddrefdt);
    DBMS_OUTPUT.put(bendemo_record.hmaddr1);
    DBMS_OUTPUT.put(bendemo_record.hmaddr2);
    DBMS_OUTPUT.put(bendemo_record.hmaddr3);
    DBMS_OUTPUT.put(bendemo_record.hmaddr4);
    DBMS_OUTPUT.put(bendemo_record.hmaddcity);
    DBMS_OUTPUT.put(bendemo_record.hmaddstate);
    DBMS_OUTPUT.put(bendemo_record.hmaddzip);
    DBMS_OUTPUT.put(bendemo_record.hmaddcnty);
    DBMS_OUTPUT.put(bendemo_record.hmphcnty);
    DBMS_OUTPUT.put(bendemo_record.hmphnbr);
    DBMS_OUTPUT.put(bendemo_record.hmphext);
    DBMS_OUTPUT.put(bendemo_record.mbphcnty);
    DBMS_OUTPUT.put(bendemo_record.mbphnbr);
    DBMS_OUTPUT.put(bendemo_record.faxcnty);
    DBMS_OUTPUT.put(bendemo_record.faxnbr);
    DBMS_OUTPUT.put(bendemo_record.hmemail);
    DBMS_OUTPUT.put(bendemo_record.wkaddrefdt);
    DBMS_OUTPUT.put(bendemo_record.wkaddr1);
    DBMS_OUTPUT.put(bendemo_record.wkaddr2);
    DBMS_OUTPUT.put(bendemo_record.wkaddr3);
    DBMS_OUTPUT.put(bendemo_record.wkaddr4);
    DBMS_OUTPUT.put(bendemo_record.wkaddcity);
    DBMS_OUTPUT.put(bendemo_record.wkaddstate);
    DBMS_OUTPUT.put(bendemo_record.wkaddzip);
    DBMS_OUTPUT.put(bendemo_record.wkaddcnty);
    DBMS_OUTPUT.put(bendemo_record.wkphcnty);
    DBMS_OUTPUT.put(bendemo_record.wkphnbr);
    DBMS_OUTPUT.put(bendemo_record.wkphext);
    DBMS_OUTPUT.put(bendemo_record.wkemail);
    DBMS_OUTPUT.put(bendemo_record.hiresrc);
    DBMS_OUTPUT.put(bendemo_record.bgnunit);
    DBMS_OUTPUT.put(bendemo_record.qdroflag);
    DBMS_OUTPUT.put(bendemo_record.hiresrcdt);
    DBMS_OUTPUT.put_line('<');
END dump_demo_rec;

PROCEDURE dump_hdr_trlr IS
BEGIN
    DBMS_OUTPUT.put(bendemo_trlr.recid);
    DBMS_OUTPUT.put(bendemo_trlr.client);
    DBMS_OUTPUT.put(bendemo_trlr.filedesc);
    DBMS_OUTPUT.put(bendemo_trlr.reccount);
    DBMS_OUTPUT.put(bendemo_trlr.field1);
    DBMS_OUTPUT.put_line('<');
END dump_hdr_trlr;
BEGIN
demo_bendemo_hdr.recid := 'x';
demo_bendemo_hdr.client := 'Client';
demo_bendemo_hdr.filedesc := 'Descr';
demo_bendemo_hdr.seqnum := 'a';
demo_bendemo_hdr.crtdate := 'Created';
demo_bendemo_hdr.srtdate := 'Sorted';
demo_bendemo_hdr.stpdate := 'stop';
dump_demo_hdr(l_output, demo_hdr);
END;

Comment: No sample.  The question is not about this particular code - it is about future self-service, and knowing how to find the failed line number.
Answer: Without seeing an example of the sort of code you are running it is hard to be sure if this answer is relevant, but the line number specified in the message relates to the overall block of PL/SQL being compiled, and ignores any blank lines or comments before that line.  For example if your script says (with line numbers added for clarity only):
1
2 -- test script
3
4
5 begin
6
7     rubbish;
8
9 end;

Then when you try to run it you will get the error:
ORA-06550: line 3, column 3:
PLS-00201: identifier 'RUBBISH' must be declared ...

It says line 3 not line 7 because "rubbish;" is the 3rd line of the PL/SQL being compiled.  The blank lines and comments above that do not count.  Blank lines and comments within the block do count however.
EDIT
So, applying my rule to the code sample you posted we can ignore the 2 lines above DECLARE and so "line 97" is the 99th line:
DBMS_OUTPUT.put(bendemo_hdr.recid);

But bendemo_hdr is a TYPE not a variable, and so it makes no sense in this statement, hence the error.  It is analagous to writing:
DBMS_OUTPUT.put(CHAR(1)); --!!!

What you probably meant was:
DBMS_OUTPUT.put(demo_hdr.recid);

Comment: I didn't mean to be obstinate w.r.t. sample code.  I have updated the original question with a sample after stripping unrelated data.  (I updated line numbers of error message also, to match.)  I do now see your point about not being sure that the answer is relevant, and apologize for the situation.
Comment: Thank you; that makes good sense, and I have accepted your answer. This is my first time to try to use TYPE ... RECORD, and I think I need to re-review its use. But in any case, this answer is invaluable to me, since I have been baffled from the beginning when it comes to these line numbers. (At the top of any source, I'll often have many lines of comments about a procedure's use and purpose. Shame on me, trying to document things! ;) )

Actual source line in ORA-06550 message


What is the PDF for the minimum difference between a random number and a set of random numbers

Question: I have a list (lets call it $ \{L_N\} $) of N random numbers $R\in(0,1)$ (chosen from a uniform distribution). Next, I roll another random number from the same distribution (let's call this number "b").
Now I find the element in the list $ \{L_N\} $ that is the closest to the number "b" and find this distance.
If I repeat this process, I can plot the distribution of distances that are obtained through this process.
When $N\to \infty$, what does this distribution approach?
When I simulate this in Mathematica, it appears as though it approaches an exponential function.  And if the list was 1 element long, then I believe this would exactly follow an exponential distribution.
Looking at the wikipedia for exponential distributions, I can see that there is some discussion on the topic:

But I'm having trouble interpreting what they are saying here. What is "k" here? Is my case what they are describing here in the limit where $n\to \infty$?
EDIT: After a very helpful helpful intuitive answer by Bayequentist, I understand now that the behavior as $N \to \infty$ should approach a dirac delta function. But I'd still like to understand why my data (which is like the minimum of a bunch of exponential distributions), appears to also be exponential. And is there a way that I can figure out what this distribution is exactly (for large but finite N)?
Here is a picture of what the such a distribution looks like for large but finite N:

EDIT2:
Here's some python code to simulate these distributions:
<code>%matplotlib inline
import math
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
numpoints = 10000
NBINS = 1000

randarray1 = np.random.random_sample((numpoints,))
randarray2 = np.random.random_sample((numpoints,))

dtbin = []

for i in range(len(t1)):
    dt = 10000000
    for j in range(len(t2)):
        delta = t1[i]-t2[j]
        if abs(delta) < abs(dt):
            dt = delta
    dtbin.append(dt)

plt.figure()
plt.hist(dtbin, bins = NBINS)
plt.show()
</code>
Comment: The exponential may be a reasonable approximation, but your minimal distance will certainly not be truly exponentially distributed: the exponential has unbounded support, but your distance is bounded between 0 and 1.
Comment: I'm pretty confused right now :/ Can you also share the code that you used to generate this picture?
Comment: Okay, I added some python code that can produce those images.
Comment: That looks to me like it's going to converge to Dirac delta. Have you tried bigger sample sizes (100k, 1m, 10m...)? (make sure the range of x-axis is consistent) The only difference between my code and yours is that you allow the distances to be negative. If you take absolute value of the distances your plot will look like mine.
Comment: Also, the number of b's (let's call it nSim) doesn't have to be as large as N! If you also make nSim = N = 1m, your code will take forever to run. You can try fixing nSim = 10k, and observe how the distribution changes shape as N goes from 10k to 100k.
Comment: To address your edit, look at (1) the range on your x axis, and (2) the fact that @Bayequentist uses absolute difference while you're using actual difference. If you use absolute difference, you'd get the same shape as Bayequentist did, and looking at the range of your x axis, you're looking at **very** small numbers just like they are.
Comment: There is an aspect of the question that is not really clear. You write: "If I repeat this process" ... But, when you repeat the process, do you keep the value of $b$ that was generated or do you generate a new $b$ each time? It sounds like you intend the latter - can you please confirm.
Comment: @wolfies, I generate a new value of b each time. You can see that in the simulation code: I generate two random lists. Then for each element in the second list, I find the minimum distance between all the elements in the first list and one element in the second list. Then I collect each of the obtained distances in this process. (And that is what I'm plotting as a simulated PDF)
Comment: (To Bayequentist & duckmayr): I'm in agreement with both of you. The absolute value does not make much of a difference in the problem, and the function should approach a delta function for infinitely large N. But what I was trying to understand was how I could understand the PDF for such a system, and I made the mistake in thinking that I would be satisfied with an answer that shows the pdf's limit for infinitely large N, while I really needed for finite but large N. Sorry if such a pivot was a moving target, but hopefully the extended discussion will be helpful for other future readers.
Answer: If you had been looking for the distance to the next value above, and if you inserted an extra value at $1$ so this always had an answer, then using rotational symmetry the distribution of these distances $D$ would be the same as the distribution of the minimum of $n+1$ independent uniform random variables on $[0,1]$.
That would have $P(D \le d) =  1-(1-d)^{n+1}$ and so density $f(d)=(n+1)(1-d)^n$ when $0 \le d \le 1$.  For large $n$ and small $d$ this density can be approximated by $f(d) \approx n e^{-nd}$, explaining the exponential shape you have spotted.
But your question is slightly more complicated, as you are interested in the signed distance to the nearest value above or below.  As your Wikipedia link shows, the minimum of two i.i.d. exponential random variables with rate $\lambda$ is an exponential random variable with rate $2\lambda$.  So you need to change the approximation to the density to reflect both the doubled rate and the possibility of negative values of $d$.  The approximation actually becomes a Laplace distribution with $$f(d) \approx  n e^{-2n|d|}$$ remembering this is for large $n$ and small $d$ (in particular the true density is $0$ unless $-\frac12 \lt d \lt \frac12$).  As $n$ increases, this concentrates almost all the density at $0$ as in Bayequentist's response of the limit of a Dirac delta distribution
With $n=10^6$ the approximation to the density would look like this, matching the shape of your simulated data.

Comment: So if you multiply the difference by n (e.g. rescale by the appropriate amount so the expectation stays bounded away from 0 and infinity), then it does converge to a two sided exponential distribution.
Answer: When $N \to \infty$, $L_N$ contains all real numbers in $(0,1)$. Thus, the distance from any number in $(0,1)$ to the closest number in $L_N$ will approach 0 as $N \to \infty$. The distribution of distances approaches the Dirac delta distribution as $N \to \infty$.
Here are some simulations:

Here's a code snippet:
<code>n <- 100000
Ln <- runif(n)

nSim <- 10000
distances <- rep(0,nSim)
for (i in 1:nSim){
  b <- runif(1)
  distances[i] <- min(abs(Ln-b))
}
hist(distances,main="N=100000")
</code>
Comment: When $N\to\infty$, $L_N$ contains all real numbers in $(0,1)$. That seems doubtful to me, since the set $\{L_1,L_2,L_3,\dots\}$ is countable, so we get $N$ as "countable infinity", while the set of "all real numbers in $(0,1)$" is uncountable. So, maybe it's better to say "contains a number that is arbitrary close to any real number in $(0, 1)$" (i.e. a dense subset of $(0, 1)$)?
Comment: That is correct - a subset of a countable set cannot be uncountable. This answer was never meant to be a rigorous answer anyway. I was just trying to provide a useful and easy-to-understand intuition. Henry's answer is much more complete and rigorous than mine.
Comment: Thanks for the intuitive answer! I think this answers my question about what happens when $N \to \infty$, but I'm still hoping to have an understanding of why my data looks exponential. I'll upload an edit with a picture so you can see it.
Comment: Also, if my edit is a bit too much of a "moving target" for you, then I can rewrite that particular part as a separate question. I'll happily accept your answer if that's the case.
Comment: This was my thought when I first read the question, that the answer was obviously 0. The average distance obviously gets smaller for each added element, and there's no obvious way that it can reasonably approach anything but 0
Answer:
is there a way that I can figure out what this distribution is exactly (for large but finite N)?

The difference of two standard Uniform random variables is Triangular(-1,0,1) with pdf $1-|x|$ defined on $(-1,1)$.
Distance is the absolute value of the difference which has pdf say $f(x)$:

Repeating the exercise $n$ times and taking the minimum distance is equivalent to finding the minimum $(1^{\text{st}})$ order statistic wrt the parent pdf $f(x)$, which is given by:

where I am using the OrderStat function from the  mathStatica package for Mathematica to automate the nitty gritties, and where the domain of support is (0,1). The solution has a Power Function distribution with pdf of form $g(x) = a x^{a-1}$.
The following diagram compares a plot of the exact pdf of the minimum distance just derived $g(x)$ (red dashed curve) ... to a Monte Carlo simulation (squiggly blue curve), when the sample size is $n=10$:

Simulation: As you are using Mathematica for simulation, here is the code I am using for the data simulation in Mathematica:
<code>  data = Table[Min[Abs[RandomReal[{}, 10] - RandomReal[]]], 20000];
</code>
Answer: For you to get a number larger than $d$ as your result, all numbers in your sample have to be $d$ away from $b$. The probability of that happening for any individual $x_0$ is just the probability mass outside the range $b \pm d$. Call that $p_{outside}$. The probability of that happening for all $x_i$ in your sample is $(p_{outside})^N$. If $x_i$ are chosen uniformly from the unit interval, then $p_{outside}$ for $b$ more than $d$ from the boundary will be $1-2d$, and that gives $p_{outside}^N = (1-2d)^N$. For large $N$ and small $d$, that can be approximated by $e^{-2Nd}$.
Answer: Imagine you first draw the last one and denote it X. This does not change the problem formulation at all. For any $X_i \in L_N, i=1,...,N$, we know that $Y_i := |X-X_i|$ has some distribution (you may or may not want to compute this) and that $Y_i$ are iid given $X$. From Wikipedia, we know that the CDF of their minimum is
$$
F_{min}(y) = 1 - [1-F_Y(y)]^N.
$$
For any fixed $y$, we know $F_Y(y) > 0$ for any $y > 0$ and $F(y) = 0$ otherwise. Take $N \to \infty$ and you get a CDF which is identically one for $y > 0$ and identically zero otherwise. This is a delta function centered at zero, like all simulations above show. This holds for any $x \in (0,1)$ so the convergence holds always (albeit with varying convergence rates, perhaps).


Audit.Net - Find all audits of a given entity

Question: I am using Audit.Net with Entity Framework Data Provider.
I am using one database for the application data and a different database (and DbContext) for the audits.
In the core configuration, I am seetting the the explicit mapper as in this example:
config.AuditTypeExplicitMapper(m =>
{
    m.Map<Client, ClientAudit>();
    m.AuditEntityAction<IAuditEntity>(SetAuditEntity);
});

Everything works fine. However later on I would like to load an entity and then see all audits for that entity.
What I easily do is query the Audit context with the Id of the entity. But then I need to know before hand the mappings from my Application context to my Audit context.
What I mean is. Find that for the Client entity, the ClientAudit entity is used. Does Audit.Net offers this possibility?
I tried looking directly in the Audit.Net code and documentation, but I found nothing that would give me this direct result.
Searching online I found no similar information. Seems I am the first to try to accomplish this.
Comment: It looks like `IAuditEntityMapping` doesn't expose any accessors to read the mappings you've set so your best bet might be to persist those mappings yourself (maybe in your own config).
Answer: If you use the Explicit Mapper, you can get the mapped type by using the AuditTypeMapper property in the EF data provider, for example:
var dataProvider = (EntityFrameworkDataProvider)Audit.Core.Configuration.DataProvider;
var auditType = dataProvider.AuditTypeMapper.Invoke(typeof(Client), null);

Comment: I also wonder if there is a neat way to get all audits related to a specific entity?
Comment: I don't understand what you mean by a neat way. A query to the database should not be complicated. Anyway, you can create a discussion [here](https://github.com/thepirat000/Audit.NET/discussions)


Intranet online document collaboration tool for text and/or spreadsheet documents?

Question: I'm looking for a collaboration tool similar to Google Docs but with one major difference:
 be able to run the collaboration document editing tool on intranet - nobody from outside the intranet should be able to access the data.
I'm aware about SharePoint but I heard that it is very expensive.
Also, wikis are good but they are missing two important features: concurrent editing and  spreadsheet support.
Answer: WSS is free.
http://www.microsoft.com/downloads/details.aspx?FamilyId=D51730B5-48FC-4CA2-B454-8DC2CAF93951&displaylang=en
Comment: Probably I will take a look at it after I install Windows 2003 Server on a virtual machine.
Comment: We use it internally. It's not the full blown product like Sharepoint Portal Server or MOSS, but it's fairly feature rich.
Comment: WSS is a great solution.  Also note that spreadsheets on fileshares already support concurrent editing however I usually strongly discourage it, If you are doing concurrent editing in excel 9 times out of 10 you.are using the wrong app.  I try to push the users over to sql server express or (if I have no other choice ) access.  See "Use a shared workbook to collaborate" in excel help for details
Comment: ...for some definition of "free."  You still need a Windows Server box to run it on.
Comment: @ktower: True, but to carry this to it's logical conclusion: you need a physical host machine which is not free, you need power which is not free, you need a location to house it in which is not free, etc., etc. In the spirit of my answer WSS is free. ;)


Forecasting with two or more causal factors using the Holt-Winters method (in R)

Question: Is there something similar to the Holt-Winters forecasting method in R, which can be used to model two or more explanatory factors?
Comment: What do you mean by "a forecast with 2 or more factors"?
Comment: @Stephan Kolassa i have my data..and there are 2 or more factors where i want to import in an order.the factors affect my data..so i guess that i have to include them to have a better forecast
Answer: You could in principle do one of two things:

Extend the Exponential Smoothing framework to include additional components for your causal variables, which get updated along with the level, trend and seasonal components.
Regress your time series on your causal factors, the model residuals using Exponential Smoothing.

There is no implementation in R that I know of. Here are Rob Hyndman's thoughts on the topic, along with pointers to papers where he did something along these lines.
Alternatively, you could use ARIMAX modeling, i.e., ARIMA with eXternal or eXplanatory variables. Look at the forecast package and the auto.arima() function. You can feed external information into its xreg parameter.


Delphi XE3 WM_HOTKEY How to tell when HotKey is Released ?

Question: I am trying to program a "cough button" in a communications program that does not always have focus. I have the code working to mute and unmute the mic (MMDevApi) and it works perfect. I set up a global hot key and this works perfect to set the mute. Now the problem. How do I tell when the Hotkey is released ? I tried a timer as shown in the code but it has an odd behavior. I Hold down my hotkey and the mic mutes immediately then after the interval of the timer it unmutes for a what appears to be half the interval of the timer and then it mutes and stays muted. Without the timer it mutes and stays muted perfectly. I really don't want ( or think it would be any good ) to have to hit a second key to unmute the mic.
//here is my register hot key code !
CoughKeyWnd := AllocateHwnd(CoughKeyWndProc);
CoughKey := GlobalAddAtom('CoughKey');
  if CoughKey <> 0 then
    RegisterHotKey(CoughKeyWnd, CoughKey, MOD_CONTROL, VK_OEM_3);

//the procedure
procedure TForm1.CoughKeyWndProc(var Message: TMessage);
begin
 if Message.Msg = WM_HOTKEY then
  begin // to prevent recalling mute
    if CoughOn = FALSE then
 begin
   CoughOn := True;
   CoughOff.SetMute(1,@GUID_NULL);
 end;
  Timer1.Enabled := FALSE;
  Timer1.Enabled := True;
end
   else
    begin
      Message.Result := DefWindowProc(CoughKeyWnd, Message.Msg, Message.WParam, Message.LParam);
    end;

//and finally the ontimer !
procedure TForm1.JvTimer1Timer(Sender: TObject);
begin
  CoughOff.SetMute(0,@GUID_NULL);
  Timer1.Enabled := False;
  CoughOn := False;

end;

Comment: I do not think this is possible at all. `WM_HOTKEY` notification send on press but not on depress. I'm going to add a `[winapi]` tag to broaden audience of your Q to confirm my thought.
Comment: You don't get that information.
Comment: I'm struggling to understand why you need to detect key up. Why can't you just respond to the hot key?
Answer: You'll see that kind of behavior if your timer expires before the second WM_HOTKEY is retrieved but does not expire retrieving consecutive messages. The time frame between the first and second message is greater than the time frame between consecutive messages. This is because keyboard delay is greater (~250ms typical) than keyboard repeat interval.
To make your approach work, increase your timer interval, something like twice the keyboard delay for instance. You can use SystemParametersInfo to get an approximation for keyboard delay. Or employ a minimum time frame for the microphone to stay in muted state, and only after then start watching for repeated hotkey messages to re-enable your timer. Still, this method will be somewhat unreliable, hotkey messages might be delayed for whatever reason. Better use GetKeyState in your timer handler to test if the keys are still down.
You can install a keyboard hook or register for raw input when the hotkey is hit if you don't want to use the timer.
Comment: Definitely use a keyboard hook, or register for Raw Input, instead of using a hotkey at all. They will give you both key down and up notifications.


Exim - restrict which addresses a user can send from

Question: By default, exim allows any user to send out an email with any "From:" address they wish, which can lead to users spoofing each other. How do I restrict which addresses each system user can send from? For example the user james should be able to send from james@localhost, but also email@example.com and email@example.com.
I want all system users to be able to send email from their SYSTEM_USER@HOSTNAME, but if they attempt to send from another address I want to look up that address in a text file to see if it is allowed.
I currently have a list, but it is for what addresses a system user will receive from. But it could still work. Given the example email@example.com the data is stored in the file /etc/exim4/virtual/my-domain.com where the content looks like:
me : james@localhost

If we could reuse the existing files that would be great. But any suggestions would be helpful. Thank you in advance.
Comment: how are they siending  (SMTP or /usr/bin/sendmail)?,  which from address (envelope sdender or from heeader)?
Comment: Both, and both :-) I'd like to write a rule which covers all cases.
Comment: Basically you need a couple of ACL rules (cf. a [similar question](https://serverfault.com/q/999324/530633), but for recipients) to check the envelope sender and the `From` and `Sender` headers.
Comment: Yes thanks, sounds like it. Even though I have years of experience writing code in various languages, I just can't wrap my head around exim configurations. Any chance you could get me started?
Answer: Exim4 already does some standard checks on the envelope sender, From: and Sender: of the locally submitted messages (cf. trusted users). Since you want to extend this validation you should:

For messages submitted through sendmail (i.e. exim4 -bm), add to the main configuration section:
# allow untrusted users to set an envelope sender
untrusted_set_sender = *
# don't delete the Sender: header
local_sender_retain = true
# don't add Sender: header
local_from_check = false

For messages submitted through SMTP you need to add the control = submission/sender_retain modifier. E.g. Debian has in the MAIL acl a rule:
accept
  authenticated = *
  control = submission/sender_retain
  control = dkim_disable_verify

To configure your check you need to add a couple of ACL rules at different stages of mail processing. Since these rules need to be applied to different ways of message submission you can create a new named ACL (I assume that all your /etc/exim4/virtual/<domain> files return username@localhost as value):
# Requires system user id in $acl_arg1
acl_check_sender:

  # Users with default domain
  accept
    condition = ${if eq{$sender_address}{$acl_arg1@$qualify_domain}}
    condition = ${if eq{${address:$h_from:}}{$acl_arg1@$qualify_domain}}
    condition = ${if match_address{${address:$h_sender:}}{:$acl_arg1@$qualify_domain}}

  deny
    ! condition = ${if and{\
        {exists{/etc/exim4/virtual/${domain:${address:$h_from:}}}}\
        {eq {${acl_arg1}@localhost}\
          {${lookup {${local_part:${address:$h_from:}}}\
            lsearch{/etc/exim4/virtual/${domain:${address:$h_from:}}}\
          }}\
        }\
      }}
    message = Spoofed From: header.

  # Most messages don't have a Sender: header, but if it is not empty, check it.
  deny
    ! condition = ${if or{\
        {eq {$h_sender:}{}}\
        {and{\
          {exists{/etc/exim4/virtual/${domain:${address:$h_sender:}}}}\
          {eq {${acl_arg1}@localhost}\
              {${lookup {${local_part:${address:$h_sender:}}}\
                lsearch{/etc/exim4/virtual/${domain:${address:$h_sender:}}}\
              }}\
          }\
        }}\
      }}
    message = Spoofed Sender: header.

  deny
    ! condition = ${if and{\
        {exists{/etc/exim4/virtual/$sender_address_domain}}\
        {eq {${acl_arg1}@localhost}\
          {${lookup {$sender_address_local_part}\
            lsearch{/etc/exim4/virtual/$sender_address_domain}\
          }}\
        }\
      }}
    message = Spoofed envelope sender.

  accept

For the syntax of string expansion (quite full of braces), check chapter 11 of Exim's documentation. You can use exim4 -bem <message> <expansion_string> to test them (the envelope sender is passed in the message like the mbox format).
You can apply it to different situations:

For messages submitted through exim4 -bm, add to the acl_not_smtp ACL the following rule:
deny
  ! acl = acl_check_sender ${sender_ident}

For messages submitted by authenticated users through SMTP add to the acl_smtp_data ACL something like:
deny
  authenticated = *
  ! acl = acl_check_sender ${authenticated_id}

Some programs submit messages using a non-authenticated SMTP connection to localhost (or exim -bs). For them you can add to the acl_smtp_data ACL:
deny
  hosts = : localhost
  ! acl = acl_check_sender ${sender_ident}

and install some identd on the server.
Comment: Thanks so much! That got me on the right track. You switched your arguments round though in the statements, but I was able to sort that. Also, if no file existed for a domain it would temporarily fail rather than reject, but I fixed that too. I'll post my solution separately soon.
Comment: Things I can't fix: If you set `untrusted_set_sender = *`, then for some obscure reason exim no longer populates $authenticated_id, it's just bank. So unless you can tell me another way to find the local username for a non-smtp email then it is impossible to filter locally sent mail. For the moment I don't use `acl_check_sender` for non-smtp mail and have just disabled being able to set custom From: addresses for local mail (if you want to do that, use smtp).
Comment: And last, but not least. The setup works fine, except if blocks all mail **unless** they have a file in the the `/virtual/` folder. My intention was to use the `virtual` folder only for aliases in addition to a user's @localhost account. I've been trying to think how to write a condition to not match if the mail is from localhost, but I don't know the variable to use to get the local system name. Any ideas?
Comment: Thanks, I edited the messed up rules and added a rule for the default domain (which is in the variable `$qualify_domain`).
Comment: You should probably post your answer as an **answer** instead of modifying the question.
Comment: Thanks, I was hoping you would so I could mark your answer correct. I see you've added the file exists check as a separate condition. That was my first attempt, but Exim ANDs conditions, don't we need this lack of file existance to be ORed with the other rule? I haven't actually tried your code yet. Can you confirm for me if you want the conditions in your new deny case to be ORed or ANDed? Thank you
Comment: I just tried out your new rule, it works perfectly! Thank you! So there's just the question of how to find $authorized_id for trusted local users (currently it is blank).
Comment: Apparently for **trusted users**  the `$authenticated_id` variable is unset, but `$sender_ident` isn't. I corrected the example in the answer.
Comment: When I enable the not_smtp rule, I suddenly become unable to send any mail from the mail command (whether changing the from or not). Looking in the log, for each attempt there is an entry which lists the from firstname.lastname@example.com, but then the next line has F=<> with the message `Error while reading message with no usable sender address`. Any ideas?
Comment: Try adding `log_message` to the `deny` rules to debug it.