Two-component Cox proportional hazards model Simulation

Question: In an article that I have been reading, they have a simulation study:

In this simulation, we generate $T_i$ from the following
group-specific linear transformation model: $$H(T_i) = \beta_{k,1}
X_{i,1} + \beta_{k,2} X_{i,2} + \varepsilon_i, i = 1, 2, \ldots,  n;
 \quad k = 1, 2 $$ where $ H(t) = \log\left(2(e^{4t} - 1)\right)$ and
$\varepsilon_i $follows the standard extreme-value  distribution. In
this case, the linear transformation model is  equivalent to the Cox
proportional hazards model. We generate samples from a two-component
Cox proportional hazards model with mixing weights $ \pi_1 =
 \frac{1}{3}$, $\pi_2 = \frac{2}{3}$, and $ \beta_1
 = (-3, -2)^T$, $\beta_2 = (1, 1)^T$. The covariates $X_i$ are generated from a multivariate normal distribution with a mean of zero
and a first-order autoregressive structure $ \Sigma = (\sigma_{st})$
with $ \sigma_{st} = 0.5^{|s - t|}$ for $ s, t = 1, 2$. The censoring
time is generated from a uniform distribution on $[0, C]$,  where $C$
is chosen to achieve censoring proportions of $5\%$ and $25\%$.

My question is, how would I generate the survival time? Here is my approach:
Transformation Functions: I start by inverting the transformation model to simulate survival time.
$$
        H(t) = \log(2(e^{4t} - 1)), \\
        H^{-1}(y) = \frac{1}{4} \log\left(\frac{e^y}{2} + 1\right).
  $$
Model Parameters:
The model involves two components with mixing weights
$$\pi_1 = \frac{1}{3}, \quad \pi_2 = \frac{2}{3},
$$
and parameter vectors $$ \beta_1 = (-3, -2)^{\top}, \quad \beta_2 = (1, 1)^{\top}$$
Covariate Generation:
$\Sigma$ is the covariance matrix of the covariate variables, and therefore, the covariate is generated from a multivariate normal (MVN)
$$\Sigma = \begin{pmatrix} 1 & 0.5 \\ 0.5 & 1 \end{pmatrix}, \\
        X \sim \text{MVN}(\mu = (0, 0)^{\top}, \Sigma)$$
Group Assignment and Survival Time Simulation:
$$\text{group}_i = \begin{cases} 1 & \text{if } U_i < \pi_1 \\ 2 &  \text{otherwise} \end{cases}$$, where
$$ U_i \sim \text{Uniform}(0, 1)$$
$$X\beta = \begin{cases} \beta_1 \cdot X_i + \epsilon_i & \text{if group}_i = 1
\\ \beta_2 \cdot X_i + \epsilon_i & \text{if group}_i = 2 \end{cases}.
$$
where         $$\epsilon_i \sim \text{Extreme Value Distribution}(0, 1)$$
Edit:
Generating survival times to simulate Cox proportional hazards models

To generate event times from the proportional hazards model, we can
use the inverse probability method (Bender et al., 2005): if $V$
is uniform on $(0, 1)$ and if $S(\cdot \,|\, \mathbf{x})$ is the
conditional survival function derived from the proportional hazards
model, i.e. $$ S(t \,|\, \mathbf{x}) = \exp \left( -H_0(t)
  \exp(\mathbf{x}^\prime \mathbf{\beta}) \vphantom{\Big(} \right) $$
then it is a fact that the random variable $$ T = S^{-1}(V \,|\,
  \mathbf{x}) = H_0^{-1} \left( - \frac{\log(V)}{\exp(\mathbf{x}^\prime
  \mathbf{\beta})} \right) $$  has survival function $S(\cdot \,|\,
  \mathbf{x})$. This result is known as ``the inverse probability
integral transformation''. Therefore, to generate a survival time $T
  \sim S(\cdot \,|\, \mathbf{x})$ given the covariate vector, it
suffices to draw $v$ from $V \sim \mathrm{U}(0, 1)$ and to make the
inverse transformation $t = S^{-1}(v \,|\, \mathbf{x})$.

In this article titled "On the linear transformation model for censored data" this is written:

Recently Cheng, Wei & Ying (1995, 1997) developed a class of simple inference procedures for semiparametric linear transformation models with censored survival data. Specifically, let $T$, $C$ and $Z$ denote the failure time, the censoring time and the $p \times 1$ covariate vector. Let $h(\cdot)$ be an unknown increasing function. A linear transformation model is
\begin{equation}
h(T) = Z^T\beta + \epsilon, \quad (1)
\end{equation}
where $\epsilon$ has a completely known density $f$ and distribution function $F$, and $\beta$ is the vector of unknown regression coefficients. If $F(s) = 1 - \exp \{ - \exp(s) \}$, an extreme value distribution, (1) is the proportional hazards model (Cox, 1972). Note that, if $S_Z(t)$ is the survival function of $T$ for given $Z$, then (1) can be rewritten as
\begin{equation}
g\{S_Z(t)\} = h(t) - Z^T\beta, \quad (2)
\end{equation}
where $g^{-1}(\cdot) = 1 - F(\cdot)$.

So based on the above Cox model can be written as
\begin{equation}
\log[-\log \{S_Z(t)\}] = h(t) + Z^T\beta,
\end{equation}
Now, if I were to find $S_{Z}^{-1}$,
\begin{equation}
S^{-1}(u) = h^{-1}(\log(-\log(u)) - Z^T\beta)
\end{equation}
In my question notition, the survival time will be generated as,
\begin{equation}
T=S^{-1}(V|X) = H^{-1}(\log(-\log(V)) - X^T\beta)
\end{equation}
where $H^{-1}(y)= \frac{1}{4} \log\left(\frac{e^y}{2} + 1\right)$ and $V \sim U(0,1)$
Update
Thank you, @Lukas Lohse, for your answer. Also, thank you, @EdM, for taking a look at the question. I found an article that used a similar model for a different reason, and R code for the simulation is available online (which is in the second simulation setup). You see in their simulation how survival time is set up. I am wondering how they came up with this setup and it seems more accurate.
library(survival)
library(mvtnorm)

H_inv <- function(y){
  1/4 * (log(exp(y)/2 + 1))
}

b1 <- c(-3, -2)
n <- 10^5
replications <- 1000

First:
coefficients_m1 <- matrix(NA, nrow = replications, ncol = 2)
for (i in 1:replications) {
  X <- rmvnorm(n, sigma = rbind(c(1, 0.5), c(0.5, 1)))
  lin_pred <- as.vector(X %*% b1)
  error <- log(-1*log(runif(n)))
  times <- H_inv(error - lin_pred)

  m1 <- coxph(Surv(time = times, event = rep(TRUE, n)) ~ X[, 1] + X[, 2])
  coefficients_m1[i, ] <- coefficients(m1)
}

mean_coefficients_m1 <- colMeans(coefficients_m1)
bias_m1 <- mean_coefficients_m1 - b1

list(mean_coefficients = mean_coefficients_m1, bias = bias_m1)

results
$mean_coefficients
[1] -2.913310 -1.942282

$bias
[1] 0.08669012 0.05771758

Second :
coefficients_m2 <- matrix(NA, nrow = replications, ncol = 2)
for (i in 1:replications) {
  X <- rmvnorm(n, sigma = rbind(c(1, 0.5), c(0.5, 1)))
  lin_pred <- as.vector(X %*% b1)
  temp = rexp(n)
  times = as.numeric(0.5*log(2*temp*exp(-lin_pred)+1.0))

  m2 <- coxph(Surv(time = times, event = rep(TRUE, n)) ~ X[, 1] + X[, 2])
  coefficients_m2[i, ] <- coefficients(m2)
}

mean_coefficients_m2 <- colMeans(coefficients_m2)
bias_m2 <- mean_coefficients_m2 - b1

list(mean_coefficients = mean_coefficients_m2, bias = bias_m2)

results
 $mean_coefficients

 [1] -2.99789 -1.99855

 $bias
 [1] 0.002110146 0.001449510

Comment: $H(t)$ is defined differently in the two papers, probably accounting for the slight difference in bias in recovering the assumed coefficients. The second approach samples directly from the minimum extreme value distribution instead of using the inverse probability method.
Comment: To make sure there is no confusion, could you please edit the question to include a link to the "article that I have been reading" where you first mention it, in the first sentence? Sometimes there are things elsewhere in an article that help clarify a specific quote.
Comment: https://www.sciencedirect.com/science/article/abs/pii/S0304407622001841
Answer: While I'm interested in seeing what @EdM has to say, this question seems like a straightforward yes, your approach works. I have coded up a version, limited to one group, in R and both the coefficients get recovered and we can find $H(t)$ in the baseline hazard.
library(survival)
library(mvtnorm)

H_inv <- function(y){
  1/4 * (log(exp(y)/2 + 1))
}
b1 <- c(-3, -2)
# large n for the simulation so we can check the coefficients
n <- 10^5
# simulate
X <- rmvnorm(n, sigma = rbind(c(1, 0.5), c(0.5, 1)))
cov(X)
colMeans(X)

lin_pred <- as.vector(X %*% b1)
error <- log(-1*log(runif(n)))
times <- H_inv(error - lin_pred)

m1 <- coxph(Surv(time = times, event = rep(TRUE, n)) ~ X[, 1] + X[, 2])
summary(m1)

result:
            coef exp(coef)  se(coef)      z Pr(>|z|)
X[, 1] -2.912249  0.054353  0.007981 -364.9   <2e-16 ***
X[, 2] -1.943424  0.143213  0.005986 -324.7   <2e-16 ***

Looking at the baseline hazard:
# H(t) is the (log) cumulative baseline-Hazard,
bhz <- basehaz(m1)
plot(bhz[, 2], log(bhz[, 1]))
curve(log(2*(exp(4*x) - 1)), add = T, col = 2)

result:
Comment: Nice demonstration that the proposed method can work (+1). I might have approached this by sampling directly from the extreme value distribution (see [this page](https://stats.stackexchange.com/a/591966/28500)), but that's not what the OP requested.
Comment: @EdM I have made some update on the question section, please check them out. Thank you!


How to make a python class not exposed to the outside?

Question: I'm relatively new to Python.
When I did C/C++ programming, I used the internal classes quite often. For example, in some_file.cc, we may implement a class in the anonymous namespace to prevent it from being used outside. This is useful as a helper class specific to that file.
Then, how we can do a similar thing in Python?
Comment: Possible duplicate of [Data Hiding in Python Class](http://stackoverflow.com/questions/9025027/data-hiding-in-python-class)
Comment: Python is almost always editable by text editor, so security measures like this aren't near as useful
Comment: anonymous namespaces around c++ class definitions in `.cc` files aren't to prevent the class from being used outside that TU (it already can't be). It's to prevent ODR-violations if the same class name is used in another TU with a different class definition.
Answer: class MyClassOuter:
    def __init__(self,...):
        class MyClassInner:
              def __init__(self,...):
                 pass
        self.my_class = MyClassInner(..)

would only have MyClassInner available inside the __init__ function of MyClassOuter
likewise you could put it inside a function
def my_class_factory(arg1,arg2,...):
    class MyClass:
        def __init__(self,arg1,arg2,...):
           print "OK??"
    return MyClass(arg1,arg2,...)

Comment: What about `MyClassOuter.__init__.__code__.co_code`?
Answer: Python code doesn't have any such equivalent for an anonymous namespace, or static linkage for functions. There are a few ways you can get what you're looking for

Prefix with _. Names beginning with an underscore are understood
to be for internal use to that python file and are not exported by
from * imports. it's as simple as class _MyClass.
Use __all__: If a python file contains a list a list of strings
named __all__, the functions and classes named within are
understood to be local to that python file and are not exported by
from *.
Use local classes/functions. This would be done the same way you've
done so with C++ classes.

None these gets exactly what you want, but privacy and restricting in this way are just not part of the language (much like how there's no private data member equivalent). Pydoc is also well aware of these conventions and will provide informative documentation for the intended-to-be-public functions and classes.


Views relationship question

Question: I have an Article node type where Expert node's can be referenced via autocompletion (as in: X Expert was consulted for this article)
I created a views for internal use, listing all the experts with exposed filters, easing up the journalist workflow to find suitable experts to contact for certain situations.
I'd like to list under each expert the Article nodes that they were referenced in, but as to how, I'm baffled, I attempted to create a relation with:
<code>Content: Expert (field_expert_nid) - reverse
Relate each Content referencing the node through Expert.
</code>
But that's about how far I've gotten, any help or suggestions how to tackle this greatly appreciated :)
Answer: I had similar situation and resorted to using viewfield. Works like a charm. Recommended.
Comment: Doesn't that module just enable a Field in content type to be rendered as a view?

Unless this field can be used in view, to show a subview hmm.
Comment: Tested & works.
Comment: As per your question, you probably mean a similarly named module called Views Field View.
Comment: Nope. Adding this Viewfield into a Content Type also enables any View displaying the content show that Viewfield - so basicly you have views rows that have views in them. :)

Views relationship question


Dynamically calculating textWidth in tableCells become wrong after scrollToRowAtIndexPath

Question: When I tried to scroll table to the bottom using this way
  let ip = NSIndexPath(forItem: array.count - 1, inSection: 0)
  chatTableView.scrollToRowAtIndexPath(ip, atScrollPosition: .Top, animated: true)

Cell width calculating:
private var leadingConstant: CGFloat = 30

@IBOutlet weak var leadingConstraint: NSLayoutConstraint! {
   didSet {
      leadingConstant = leadingConstraint!.constant
      }
   }

   private func resizeCell() {
     let labelTextWidth = chatTextLabel?.intrinsicContentSize().width
        guard labelTextWidth < lWidth else {
          return
        }
        leadingConstraint?.constant = leadingConstant + lWidth -  labelTextWidth!
   }

I will get the wrong calculation of textWidth in cells
Comment: Dude where is the code for cell width calculaton ????
Comment: I think we need to see implementation of textWidth calculation
Comment: and why atScrollPosition: .Top???? when you want to scroll bottom :P shouldn't it be .Bottom ???
Comment: The calculation added
Answer: I have the issue with resizing label so I nee just to do    chatTextLabel?.updateConstraints() after setting up the text
var chatMessage: ChatMessage! {
    didSet {
      setContent()
    }
}

func setContent() {
    chatTextLabel.text = chatMessage.message
    chatTextLabel?.updateConstraints()  // here

    let labelTextWidth = (chatTextLabel?.intrinsicContentSize().width) ?? 0
    let labelTextHeight = chatTextLabel?.intrinsicContentSize().height

    guard labelTextWidth < originWidth && labelTextHeight <= singleLineRowheight else {
      trailingConstraint?.constant = trailingConstant
      return
    }
    trailingConstraint?.constant = trailingConstant + (originWidth - labelTextWidth)

  }


Datetime on an X Axis in Bokeh

Question: I'm having trouble understanding how datetime works in Bokeh.
My code is outputting milliseconds on the x axis, which seems to be the default if the axis type is 'datetime', but I would like to have it separated by every 5 years.
How can I accomplish this in Bokeh? The dates are before 1950 so I can't use a timestamp. When I exclude the datetime axis type, it displays every single value as a tick on the x axis. When I try to change the inputted data as a datetime object, Bokeh rejects it.
I've read other posts on the subject, as well as Bokeh's documentation on
DatetimeTickFormatter, which doesn't seem to be working for me either.
Datetime axis in Bokeh
Bokeh FixedTicker with Custom Datetime/Timestamp values
https://docs.bokeh.org/en/latest/docs/reference/models/formatters.html?highlight=formatters#module-bokeh.models.formatters
All my code for the plot is below:
<code>r = requests.get('https://raw.githubusercontent.com/FreeCodeCamp/ProjectReferenceData/master/GDP-data.json')
    data = json.loads(r.text)

chart_data = {
    'date': [a[0] for a in data['data']], #this is the problem data
    'gdp': [a[1] for a in data['data']],
}

source = ColumnDataSource(chart_data)
hover = HoverTool(
    tooltips=[
        ('GDP', '@gdp{$0,0.00}'),
        ('Date', '@date{%Y - %B}'),
    ],
    formatters={
        'date' : 'datetime',
    }
)

bar = figure(x_range=chart_data['date'],
             x_axis_type='datetime',
             plot_height=750,
             plot_width=1000,
             tools='wheel_zoom, reset, save',
             title=data['source_name'])
bar.vbar(x='date', top='gdp', width=0.9, source=source)
bar.add_tools(hover)

#visual settings

bar.xaxis.formatter = DatetimeTickFormatter(years = ['%Y'])
bar.xaxis.major_label_orientation = 'vertical'
bar.xaxis.minor_tick_line_color = None
bar.xgrid.grid_line_color = None

bar.yaxis.formatter = NumeralTickFormatter(format='$0,0.00')
bar.ygrid.grid_line_color = None

show(bar)
</code>
Thank you for the help, this has been driving me crazy!
Comment: What, exactly, is `chart_data['date']` ?
Comment: Thanks for helping! If you look at the json in the link, the date format is '%Y-%m-%d'. An example is: '1947-01-01'.

https://raw.githubusercontent.com/FreeCodeCamp/ProjectReferenceData/master/GDP-data.json
Comment: Passing a sequence of strings results in a Categorical range, which is why a datetime axis will not work, and why all values are shown (Categorical axes show all factors).  I'd suggest converting to real datetime objects, which Bokeh will handle. Then if you don't want the standard datetime ticker you could use, e.g `YearsTicker`
Comment: So I converted to a datetime object, but Bokeh is giving me a ValueError. "ValueError("Unrecognized range input: '%s'" % str(range_input))". I changed:
        'date': [datetime.strptime(a[0], '%Y-%m-%d') for a in data['data']]

Is this incorrect?
Comment: Don't set `x_range` anymore. With real datetime objects, the default `DataRange1d` is what you'd want.
Comment: Yes!! The x_range seems to have been the issue. So, when I set that range it results in a Categorical axis, which was throwing off my datetime objects?
Comment: Yes, setting `x_range` to a list of strings results in a Categorical `FactorRange`, which expects coordinates to string factors, not date times.
Comment: One last thing - my hovertool isn't showing up at all on the graph now. Does changing from a Categorical type break this in any way?
Answer: Thanks to help from bigreddot. My issue was that my plot was acting as a Categorical range due to me setting the x_range property, instead of defaulting to datetime.

Datetime on an X Axis in Bokeh


Analytic number theory books after Apostol

Question: I am planning to learn some classical results on analytic number theory. I have read  Apostol's Introduction to Analytic Number Theory, but nothing about algebraic number theory. Can anyone recommend some more advanced books with further results on analytic number theory?
Comment: Maybe you can peruse http://math.stackexchange.com/questions/153022/a-good-reference-to-begin-analytic-number-theory?rq=1, http://math.stackexchange.com/questions/29056/supplemental-number-theory-text-to-montgomery-and-vaughan
Answer: Are you aware that Apostol: Introduction to Analytic Number Theory. continues with Volume 2, Modular Functions and Dirichlet Series in Number Theory.
Answer: Yes, Davenport's book (Multiplicative Number Theory) is great, as is Serre's (Course in Arithmetic) Once you read those, check out Iwaniec/Kowalski.


XSL template structure for choosing latest event

Question: I have a list grouped by Region and it currently shows all the items for each city. I want to reduce to only the most recent advisory for each city.
I have tried to use an xsl:for-each statement but I am messing up the names/parameters.
List is called mlc
The list contains the fields:
Title
City
Region
Advisory
DateCreated
TT (calculated number field to find the number of minutes from the DateCreated to end of today which I intended to use the smallest to find the most recent)
I have the list grouped by Region and it currently shows all the items for each city. I want to reduce to only the most recent advisory for each city.
Comment: Could you post a sample of your XML, as that will make things clearer. Thanks!
Answer: I had a co-worker help me and the code goes like this. City is sorted ascending. Created is sorted descending.
<xsl:if test="not(preceding-sibling::Row[1]/@City) or preceding-sibling::Row[1]/@City!=@City">
Now each city is shown only once and it is the most recent advisory.


How to create a plot on python using Matplotlib, where all of the zeros from the data set are emitted?

Question: def plot_exam_versus_test_no_zeros(filename):
    """scatter plot, emitting zeros
    """
    marks = np.loadtxt(filename, delimiter=',', skiprows=1, usecols=[2, 3])
    zeros = marks[marks[:, :]==0] = np.nan
    marks_test = marks[:, 0]
    marks_exam = marks[:, 1]
    axes = plt.axes()
    axes.plot(marks_test, marks_exam, "go", label="markers")
    axes.set_xlabel("Test")
    axes.set_ylabel("Exam")
    plt.title("COSC121: Exam versus Test")
    plt.show()

plot_exam_versus_test_no_zeros("marks.csv")

This is the extent of my code so far. My plot looks identical to what it should, but there are a few differences in the test, as highlighted in yellow.
Thanks in advance.
Here's some photos...
https://i.stack.imgur.com/LVHJ6.jpg
https://i.stack.imgur.com/de4JK.jpg
https://i.stack.imgur.com/G1KD2.jpg
Comment: consider filter data before plotting?
Answer: No need to over complicate ~
def plot_exam_versus_test_no_zeros(filename):
    """scatter plot, emitting zeros
    """
    marks = np.loadtxt(filename, delimiter=',', skiprows=1, usecols=[2, 3])
    marks = marks[marks.all(axis=1),:]
    axes = plt.axes()
    axes.plot(marks, "go", label="markers")
    axes.set_xlabel("Test")
    axes.set_ylabel("Exam")
    plt.title("COSC121: Exam versus Test")
    plt.show()

plot_exam_versus_test_no_zeros("marks.csv")

Comment: Thanks for the reply. Sorry, I forgot to mention that we are not allowed to use any loops or list comprehensions.
Comment: Okay, fixed it to be a `numpy` way, this is what I thought should be possible in the first place :')
Comment: Hmm, I still can't get it to work


How can I rward Mails with Postfix and Dovecot to a specific Mailbox?

Question: I can't figure out how to forward mail withhPostfix and Dovecot to a specific Mailbox.
Sender Mail Address --> Mailbox of Receiver
At moment the mails are transfered into the INBOX of specific Users.
Postfix configuration:
Main.cf
lmtp_destination_recipient_limit = 1
mailbox_transport = lmtp:unix:private/dovecot-lmtp
virtual_alias_maps = hash:/etc/postfix/_virtual
virtual_mailbox_base = /var/vmail
virtual_uid_maps = static:10000

Virtual_alias_maps
@sender.org                        firstname.lastname@example.com

Dovecot:
mail_location = maildir:~/Maildir
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    special_use = \Drafts
  }
  mailbox Junk {
    special_use = \Junk
  }
  mailbox Trash {
    special_use = \Trash
  }
  prefix =
  separator = /
  type = private
}
protocols = imap imap lmtp
service auth {
  unix_listener /var/spool/postfix/private/auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0600
    user = vmail
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0666
    user = postfix
  }
}

Answer: The virtual_alias_maps command is designed to redirect an incoming message by recipient, not by sender, so yours would suggest user@recipient as a catchall address for mails directed to *@sender. Note that the file name must be the same as specified in main.cf, so /etc/postfix/_virtual in your case and you need to call postmap after every update.
If you want to reroute an incoming mail based on the sender address, you could try the following:
In main.cf modify the option smtpd_recipient_restrictions so that it contains check_sender_access hash:/etc/postfix/sender_access
Then create /etc/postfix/sender_access with a content like
sender.org REDIRECT firstname.lastname@example.com

(See the postfix manual entry of Access(5) for details on the file syntax.)
After that, run postmap /etc/postfix/sender_access, reload/restart your postfix and from now on, all incoming mail from the domain sender.org should be delivered to email@example.com, regardless of the TO, CC and BCC field.
Answer: thanks for the answer. You are right. Its the recipient not the sender.
Let me clarify my question.
How can I realise the following constalation?
Mail -> recipient address -> mail account account -> specific Mailbox


Deno - access web apis

Question: I am bundling my Deno code for the web using deno bundle and I know the browser has RTCPeerConnection from the WebRTC API which I would like to use.
I thought Deno was aiming to have web compatibility, so I don't see why WebRTC is not planned to be implemented.
But since it isn't at the moment, how can I tell Deno to trust that this global variable exists and to compile?
Answer: Using deno bundle will actually run fine for many web APIs that are supported by the Deno language.
My temporary solution for other apis such as crypto is the following:
declare crypto: any;

This article might be useful to look at too (see the 'Compiling and bundling' section).

EDIT: in order to support a wide variety of environments, dependency injection can be a great solution in many cases.

Deno - access web apis


Android SQLite auto increment

Question: I currently have a table called User which has a id column which is created as
'INTEGER PRIMARY KEY'
Lets say I have created two users so the table has id 1 and 2
If I delete the second user and create a third the id is 2, I need this to be 3
So it seems Android is selecting the next available id, how can I change this to its more like  a sequence number?
Regards
Comment: but u need primary key and this is working fine.why u need this ?
Answer: Make it INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL. Here's what the docs say:

If a column has the type INTEGER PRIMARY KEY AUTOINCREMENT then... the ROWID chosen
  for the new row is at least one larger than the largest ROWID that has
  ever before existed in that same table.
The behavior implemented by the AUTOINCREMENT keyword is subtly
  different from the default behavior. With AUTOINCREMENT, rows with
  automatically selected ROWIDs are guaranteed to have ROWIDs that have
  never been used before by the same table in the same database. And the
  automatically generated ROWIDs are guaranteed to be monotonically
  increasing.
Comment: I want `AUTOINCREMENT` without `PRIMARY KEY`.Because I assign the primary key is another column, I am using `_ID INTEGER AUTOINCREMENT NOT NULL` this make error.Please help me bro.
Comment: @reegan29 AUTOINCREMENT can ONLY be used with INTEGER PRIMARY KEY. As per *Because AUTOINCREMENT keyword changes the behavior of the ROWID selection algorithm, AUTOINCREMENT is not allowed on WITHOUT ROWID tables or on any table column other than INTEGER PRIMARY KEY. Any attempt to use AUTOINCREMENT on a WITHOUT ROWID table or on a column other than the INTEGER PRIMARY KEY column results in an error.* [SQLite Autoincrement](https://sqlite.org/autoinc.html)
Answer: SQLite AUTOINCREMENT is a keyword used for auto incrementing a value of a field in the table. We can auto increment a field value by using AUTOINCREMENT keyword when creating a table with specific column name to auto incrementing it.
The keyword AUTOINCREMENT can be used with INTEGER field only.
Syntax:
The basic usage of AUTOINCREMENT keyword is as follows:
CREATE TABLE table_name(
   column1 INTEGER AUTOINCREMENT,
   column2 datatype,
   column3 datatype,
   .....
   columnN datatype,
);

For Example See Below:
Consider COMPANY table to be created as follows:
sqlite> CREATE TABLE TB_COMPANY_INFO(
   ID INTEGER PRIMARY KEY   AUTOINCREMENT,
   NAME           TEXT      NOT NULL,
   AGE            INT       NOT NULL,
   ADDRESS        CHAR(50),
   SALARY         REAL
);

Now, insert following records into table TB_COMPANY_INFO:
INSERT INTO TB_COMPANY_INFO (NAME,AGE,ADDRESS,SALARY)
VALUES ( 'MANOJ KUMAR', 40, 'Meerut,UP,INDIA', 200000.00 );

Now Select the record
SELECT *FROM TB_COMPANY_INFO
    ID      NAME            AGE     ADDRESS             SALARY
    1       Manoj Kumar     40      Meerut,UP,INDIA     200000.00

Comment: thanks... I needed "INTEGER PRIMARY KEY   AUTOINCREMENT" for autoincrement field .. and got it.
Comment: Shoot, I had `AUTOINCREMENT` in the wrong spot. Thanks for the example!
Answer: If speaking for ANDROID, yes, above answers are correct, except naming of the id column.
   database.execSQL("CREATE TABLE IF NOT EXISTS "
                + TableName
                + " ( rowid INTEGER PRIMARY KEY AUTOINCREMENT,  Raqam VARCHAR, ChandBor INT(3));");

It looks like in Android it should be named as 'rowid'.
And with Cursor you need to instantiate it like:
Cursor cursorLcl = database.rawQuery("SELECT *," + TableName + ".rowid AS rowid" + " FROM " +
                TableName, null);

Otherwise it didnt work for me. I don't know why it so.
Answer: Just remember for android when writing tot the database (ie. executing),
do
INSERT INTO TABLE_NAME (param1name, param2name) VALUES (param1,param2)
and there is no need to add a place holder for the auto increment. It will add it by itself when adding a record. If you do not declare the params that you will put in, you will get the error x amount of variables expected and you only gave x-1, and this is because you are not supposed to give any place holding value for the auto increment column

Android SQLite auto increment


Simple program throws FileNotFound Exception for relative path

Question: A dumb question, but after a lot of googling, I still can't get it to work.
    System.out.print("Start");

    File file = new File("TestFile.txt");

    Scanner scanner = new Scanner(file);

When I run this, I get a FileNotFound exception.  I'm running on NetBeans, and I have placed the TestFile.txt inside the same package as the main class.  When I use the absolute path, it works.  I can see the see the file next to the .class file inside the build folder.  I've tried using the command line, and it doesn't work either.
Comment: *and I have placed the TestFile.txt inside the same package as the main class* - That's the issue. If you are accessing the file like this, you should put it in the `user.dir`. You can test it with `System.out.println(System.getProperty("user.dir"))`
Answer: Relative file paths are not relative to the location of the class file containing the code. They're relative to the directory from which the java executable is started.
Just like, when executing
ls foo

or
dir foo

ls/dir look for the foo directory in the current directory, and not in the directory where the ls executable or dir executable is located.
So, similarly, when executing
java com.foo.bar.MainClass

if the MaiClass file opens the file "foo.txt", it will look for it in the directory where you were when executing the command java com.foo.bar.MainClass.
Answer: If you use relative paths the folder that has the file must be in the classpath, so you can put the file in the root folder (if you run the code from a jar file), or add the root folder to your classpath.
If you want to keep the file in that folder you have to include it into the classpath in order to get it work.
Also, the relative path for a java command line program is the root folder, so you can construct the relative path from that point too.
Answer: You should put the text file in the 'working directory' of your application.  And if you don't want to do that, you should include the folder that contains this file in class path.


Create a custom block type using macros

Question: In avr-gcc you can something like this:
ATOMIC_CODE{
   cout << "Here I can do stuff that is very time sensitive\n";
}

Unfortunately this is a #define which uses a special gcc atribute, which I would like to avoid.
So a workaround is this:
void enableInterrupts(){ std::cout << "Interupts Enabled\n"; }
void disableInterrupts() { std::cout << "Interupts Disabled\n"; }
class Scoped{
    void (*cleanup)();
    bool incremented;
    public:
        Scoped(void (*clean)(),void (*before)()) : cleanup(clean),incremented(false){
            before();
        }
        ~Scoped(){
            cleanup();
        }
        void operator ++(){
            incremented = true;
        }
        bool operator!(){
            return !incremented;
        }

};
#define ATOMIC for (Scoped x(&enableInterrupts,&disableInterrupts); !x; ++x)

//Later in main.cpp
ATOMIC{
    /*do stuff*/
    std::cout << "This is atomic code\n";
}

The only problem is that it relys on the destructor getting called immediately (which I am not sure is a reliable method).
So is it guaranteed that the destructor will get called immediately or can the compiler destruct the object whenever it feels like it?
Comment: If the compiler conforms to the language standard, then it has to do what the language standard tells it to do and the program behaves the way the standard says. If the compiler does something different, then it does something different.
Comment: @KerrekSB Which is what?
Comment: You mean you're asking what the lifetime of an object is in C++?
Comment: @KerrekSB I am asking that once the block ends does the object get destroyed imediatly (so i suppose yes)
Comment: Contemplate perhaps the fact that the class template `std::lock_guard` exists.
Comment: @KerrekSB I usually don't use std classes (except cout)....But does that mean that the object is immediately destroyed
Answer: Yes, it is guaranteed that the destructor will be called immediately.
https://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization
You're thinking about languages that do lazy mark and sweep garbage collection.  (ok, maybe you're not)
Comment: No I am not....I was only wondering when the destructor was called to a non pointer object
Comment: Automatic variables are cleaned up immediately when they go out of scope.

Create a custom block type using macros


Notation for generic voicing

Question: Is there any common notation for a generic voicing of a chord? For instance, I could specify as a kind of voicing template:

"Root, 3rd and fifth in fourth octave, plus root in the fifth octave"

And apply this to any triad.
So is there a common notation for such a template?
Comment: Take a look at https://music.stackexchange.com/a/58341/71178
Answer: There's the standard root-3rd-5th making a root position, then 1st inversion with 3rd-5th-root up an octave, and 2nd inversion, with 5th at the bottom-root (up an octave from original)-3rd(up as well).Go up another step and it's back to root.
As there are so many combinations of the three notes available on several instruments, I don't believe there is a name. Guitar gets quite complex with voicings, with some chords doubling a particular note, others having 3 of the same, others missing out notes (often 5ths). Then on keys, you could end up with dozens of voicings for the same three notes.Apart from seeing the dots written down, I don't think there are labels.
Answer: possible voicings of a 3 note chord include those 3 halfsteps in ANY and ALL of the 7.3 octaves.
your example is ok for when the 1,3,5 are only used once.  But maybe you want 3 separate bass notes in 3 of the lower octaves, plus a 3rd in octave 4, and 2 5ths in octaves 5 and 6.
You could have that 3 note chord playing on 21-ish possible notes or any combination.  And of course, you can omit notes of a chord and still call it that chord.
So usually spec'ing the arrangement for a chord is listing each individual octave that's playing it.  And of course, rhythm comes into play and some of the notes aren't held for the full duration or start at the same time.  So the arrangement is usually just plain spec'd in notation.
Answer: There is a general distinction between "closed" and "open" voicings of chords -- what you gave as an example looks like a closed voicing (where the notes of the chord are as close as they can be). When creating an open chord voicing using many notes spread over a number of octaves, it is often good to follow the spacing of the overtone series:  use octaves at the bottom, then add fifths and fourths, then thirds and sevenths, etc. (At the top of the frequency scale, many composers favor octaves again as an acoustic reinforcement.)
Comment: That sounds like good advice for choosing a voicing, but the question is about notation for the voicing, once chosen.

Notation for generic voicing


ARIMA Modeling on specific time series

Question: i have some time series example that i want to predict, lately i was applying
ARIMA and Bayesian Forecasting method. But i really am interested in making a further best ARIMA Model (or just knowing some suggestion about the time series), so here is my time series data (it is as far only almost 2 years)
data: purchase of Alpara Kapl medicine from Feb 2015 - Oct 2017
0 2 0 0 0 1 2 3 3 0 0 2 0 1 0 0 3 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0
From these time series, i get best model of ARIMA is ARIMA(0,0,0) which is White Noise, but then there are other models (that is likely more redundant) from ARIMA(0,0,0) which is these (with the calue of AICc):
ARIMA(2,0,2) with non-zero mean : Inf
ARIMA(0,0,0) with non-zero mean : 97.09692
ARIMA(1,0,0) with non-zero mean : 98.46206
ARIMA(0,0,1) with non-zero mean : 98.42101
ARIMA(0,0,0) with zero mean     : 104.5138
ARIMA(1,0,1) with non-zero mean : 100.3878
there is ARIMA(0,0,1) the best second model,
but when using at forecasting, White Noise will give a flat forecasting (because its definition is not suitable for forecasting as i known so far), so is it a good way to implement ARIMA(0,0,1) for this reason? or maybe is there some other clue to define other model? any thoughts is welcomed thankyou :)
Answer: Your data suggest the need for an Intermittent Demand solution ...  . I use AUTOBOX to identify a useful model identifying a three period interval between demands . It uses a sophisticated i.e. robust  extension of the Croston method https://stats.stackexchange.com/search?q=user%3A3382+croston which deals with anomalies (one in this case ).  . The useful model is here . Note that the forecasts can be simply integerized but are shown for completeness purposes.
Comment: I know that this is an very old question, but thankyou sir Irish for the insights :)
Comment: you are very welcome ..

ARIMA Modeling on specific time series


Can't get results from a IQueryable

Question: I have the following code in Linq to Entity:
var policy= from pol in conn.Policy
            where pol.Product.DESCRIPTION=="someProduct"
            SELECT pol;

Then, the table Policy, has some dependencies for a table called Entity. If I do this:
foreach(Policy p in policy){
    if(!p.Entity.IsLoaded) p.Entity.Load();
    IEnumerable<Entity> entities= from ent in p.Entity
                                  Where ent.EntityType.DESCRIPTION=="SomeEntityType"
                                  select ent;
    Console.Writeline(entities.ElementAt(0).NAME);
}

It says, "Object not set to an instance", but if I do:
foreach(Policy p in policy){
    if(!p.Entity.IsLoaded) p.Entity.Load();

    foreach(Entity et in p.Entity)Console.Write(et.NAME);

}

It works!
Can anyone tell me why?
Thank you, Best regards.
Answer: I'd say it's because your LINQ statement isn't returning any values, so entities is null.
Your second example works because you're not trying to filter the results.
Do you have some entities that have the DESCRIPTION set to exactly the string (including case)?

Can't get results from a IQueryable


Drush SSH Port 22 Error

Question: I've been trying for quite some time now to get my drush to play nicely with my Drupal configuration and I'm having some problems. I'm attempting to perform cc all and uli on my Drupal directory and I keep getting the following error:
ssh: connect to host site.dev port 22: Operation timed out

The command I'm running is "drush @site.dev cc all" and "drush @site.dev uli"
Whenever I run uli I get a slightly longer version of the same error:
The external command could not be executed due to an application     [error]
error.
Illegal string offset 'site' backend.inc:1030                        [warning]
The command could not be executed successfully (returned: ssh:       [error]
connect to host site.dev port 22: Operation timed out
, code: 255)
current() expects parameter 1 to be array, null given                [warning]
user.drush.inc:415
parse_url() expects parameter 1 to be string, array given            [warning]
exec.inc:427
 does not appear to be a resolvable hostname or IP, not starting     [warning]
browser. You may need to use the --uri option in your command or site
alias to indicate the correct URL of this site.

I'm not really sure where to start with getting this to work. I've poked around in my Vagrant configuration and I'm able to SSH on port 22, so I don't really know where else to look to correct this error.
This is what my alias configuration looks like:
Vagrant.configure("2") do |config|
  mc_settings = {
    :domain => 'site.dev',
    :docroot => '/home/vagrant/docroot',
    :host_docroot => '../docroot',
    :database_name => 'site_dev',
    :vbox_name => 'site_dev'
  }

Comment: First things first: are you able to connect to your server using ssh? And, when drush is telling you to try `--uri`, have you tried it? Did it make any difference?
Answer: It's really hard to answer specifically, since it really depends on how you have set your vagrant up.
Long story short
You need to setup your drush alias so that it's using SSH settings to access your vagrant box. The issue is probably that your vagrant box doesn't expect connections on port 22 or that you haven't setup the correct IP/domain for SSH connection in your alias.
If you want to use Vagrant for Drupal development I can recommend parrot. It has all you want for Drupal developetment, uses puppet, which makes it pretty easy to change PHP versions etc, and has Varnish, Solr etc.
If you are using parrot you should be able to access the Vagrant box by doing.
ssh firstname.lastname@example.com

Assuming that your site is located on the site.dev url.

Drush SSH Port 22 Error


How do I get the mock of the current user?

Question: I'm trying to get a mock of current user.
$user = \Drupal\user\Entity\User::load(\Drupal::currentUser()->id());

For this, I extend UnitTestCase class and load the User class methods.
$methods = get_class_methods('\Drupal\user\Entity\User');

Since load() is a User class method, I have created a mock of the User class, disabled its constructor, and set all methods of this class, including load() which is the method I actually need.
$user = $this->getMockBuilder('\Drupal\user\Entity\User')
  ->disableOriginalConstructor()
  ->setMethods($methods)
  ->getMock();

Since the method is set now, I'm trying to call this method once and not sure what it will return exactly.
$user->expects($this->once())
  ->method('load')
  ->will($this->returnValue('admin'));

Now, while running this test, I got an error.

Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException:
You have requested a non-existent service "entity.manager".

What's the need of injecting the entity.manager service?
Although load() is a method of the entity storage interface, it also exists in the User class.
Why does this error come up?
Can I load the current user without mocking the entity.manager class?
The actual method for which I'm doing unit testing is the following.
public static function get_ctype_field_value($field) {
  $user = user_load(\Drupal::currentUser()->id());
  $ulm = get_type_id($user);
  $local_market = \Drupal\node\Entity\Node::load($ulm);
  $llf = '';
  if ($local_market) {
    // Get the field value from the LM.
    if ($local_market->hasField($field)) {
      $localmarket_field = $local_market->$field->value;
    }
  }
  return $localmarket_field;
}

Answer: In Drupal, unit tests (UnitTestCase or \PHPUnit_Framework_TestCase, sometimes) are really best when you have a class that doesn't have a lot of dependencies or when you can really predict what the dependencies do so you can mock them.  Typically, you mock individual methods, and not try to do everything.  When you find yourself mocking too much, or with complicated mocks, it is best to rework your test as a kernel test (KernelTestBase or KTB for short).
A KTB will boot a minimal Drupal environment, and then you can enable modules and choose which config and schema you want installed (enabling the module doesn't automatically do this).  These are still PhpUnit tests, so you can run that that way and have full access to providers and other PhpUnit features.  Many modules in core also define extensions of KTB to provide setup and additional features (eg, EntityKernelTestBase and ViewsKernelTestBase).
KTB tests are not as fast as UTC tests, but still quick and not as slow as functional tests (BrowserTestBase).  Core has tons of examples.
In your case, since it seems like you need users and nodes, you really want a KTB.  Since both are entities, there is really a lot you would need to mock and you will go down the rabbit hole quickly trying to do this because of the dependencies.
See also Types of tests in Drupal 8.
A few answers to your specific questions.
Users are content entities (so are nodes), which use a lot of services.  And, they aren't all properly injected (the base Entity class uses a lot of helper methods that access the global \Drupal class to get services).  In addition EntityManager is deprecated and got broken up in different services; so a lot of its methods are really helper functions for the other services.  To to a UTC, you would have to start mock these helper functions.  I do not think there is a way around this.
Based off your test snippet, you are really best starting off with a KTB.


Three-way interaction of two categorical factors and a continuous factor with random error in GAM

Question: I am modeling a three-way interaction with a random error in GAM using mgcv package. The data is from a plant growth experiment with 2 categorical predictors f1 (mois; moisture = 3 levels) and f2 (spec; plant species = 4 levels), a continuous x1 predictor (temp_cont; temperature = 5 levels of a warming treatment treated as continuous in the model), and block, f3 (4 levels).
Datapoints with geom_smooth(loess) line for temperature - response_var relationship at different factor levels:

The aim is to determine (1) whether the relationship of plant growth (and carbon fluxes) with temperature is nonlinear or not (we anticipate it to be non-linear) and (2) whether the shape (slope) of the temperature response curves differ between plant species.
I would first fit a three-way interaction model and then compare this model to a model without the 'species' term using the models' AICs.
I struggle with fitting the model which also accounts for the block effect, and also how to check if this model fits the data well, and after reading many online posts, my latest try is:
gam(response_var ~ spec*moist+
                s(temp_cont, by = spec, k = 5, bs = 'tp')+
                s(temp_cont, by = moist, k = 5, bs = 'tp')+
                s(temp_cont, by = spmo, k = 5, bs = 'tp')+
                s(block, bs = 're'),
                data = data_var,
                method = 'REML')

spmo is the interaction term between spec and moist
(data_var %>% mutate(spmo = as.factor(interaction(spec, moist, drop = TRUE))))
Is this model correctly and sufficiently taking into account the block effect? When I plot the smooths for the different levels of combinations of the interacting categorical factors, some of the plots look strange and edf numbers does not reflect the non-linearity of those relationships plotting the raw data. I wonder whether the three-way interaction model requires the two-way interactions (i.e., s(temp_cont, by = spec, k = 5, bs = 'tp') and s(temp_cont, by = moist, k = 5, bs = 'tp')).
The original model as per the script above:

Updated model without the two-way interactions:

I have read about options to account for different levels of wiggliness for the levels of a categorical factor included in the model and am not certain that I understood it correctly. What will be the way to determine whether different levels of wiggliness for a categorical factor in te model are needed?
Then, based on the edf parameter of the model, I can determine the linearity of relationships between temperature and response variable at different levels of a categorical factor (whether it is the three-way interaction or the reduced model).
I would also compare such a model with the same linear model (lme4 model) using AIC to further show that GAM would be preferred over a linear model, if this is the case. Will such a comparison be valid? I have read that the AICs are differently calculated for the GAM vs. the lme4 model.
The other issue is that the overall model adjusted.r2 is 0.9 (for the three-way interaction gam model shown above), which suggests overfitting, but we are not concerned about this as we are not trying to predict the response for any other plant species and we want to fit the model to this data and asses whether the relationship is nonlinear or not (non-linearity is the novelty of our research and we aim to show that it exists).
The link to the original data:
https://drive.google.com/file/d/1HiDbZULJeDSOw8zbmseT2iqR513JQLni/view?usp=drivesdk
I would greatly appreciate constructive comments and recommendations for my model improvement. Many thanks.
--------------------------additional questions after the initial response--------------------------
The location-scale Tweedie with the power parameter allowed to vary with the 'moist' factor is a more appropriate choice than without this allowance, as you have shown in your initial response. I'm curious about the decision-making process in determining which factor should govern the variation of the power parameter ('moist' or 'spec' in this case). To explore this, I compared model, designated as m5 in your response, with another model where the power parameter varied with the 'spec' factor (and smoothly with temp_cont). After comparing both models using their AIC values, your selected model exhibited a lower AIC than the other model (518.9 vs. 522.6). Is this method consistent with your approach for selecting the factor influencing the power parameter of the Tweedie distribution? Alternatively, is there any indication on the selection of the factor when screening the plots produced by plotting the model?
My overall aim is to determine whether the response variable exhibits a nonlinear relationship with temperature and whether additional predictor variables ('spec' and/or 'moist') affect the shape of this relationship (or slope of this relationship, in the case when the linear relationship is preferred). Initially, I fitted the three-way interaction model with all the other 2-way interaction terms, denoted as the m5 model to address this. By systematically evaluating AIC values for models with individual interaction terms removed, I determined that the three-way interaction model (m5 model) provided the best fit, retaining all the other terms. Am I then right to consider the m5 model as the more appropriate model?
You also showed the GAM model with the Gamma family which has the shape parameter allowed to vary with moist and smoothly in temp_cont(model m7) has the lowest AIC (although within 2 AIC units of the AIC of m5). Which of the models do you prefer and why?
I recently learned online that an effective degree of freedom (edf) equal to 1 or lower indicates a linear relationship, while edf greater than 1 suggests nonlinearity. The model parameters of model m5 include 'edf', 'edf1', and 'edf2', with 'edf2' being identical to 'edf' values. Upon extracting the 'edf' values from the model, none exceeded 1. Does this simply imply that all relationships within the model are linear and not wiggly?
Additionally, I fitted a linear model using gam function, which will be the linear equivalent of the m5 model and compared AIC values between this model and the m5 model.
m5_linear <- gam(list(resp.var ~ spec * moist * temp_cont+
                   s(block, bs = 're'),
                   ~ 1,
                   ~ moist + s(temp_cont, k = 5)),
            data = data_var,
            family = twlss(),
            method = 'REML')

This linear model exhibited a higher AIC compared to the GAM model (545 vs. 518, respectively). Have I made this comparison correctly? What does the discrepancy between the fact that AIC would prefer the GAM model wile all 'edf' values are equal to 1 or lower signify?
I'm uncertain about which parameters to consider when assessing GAM model fit. For the m5 model, I've examined the normal Q-Q plot, which displays deviations from a straight line at both ends.

In my interpretation, these deviations might still be acceptable. Would you agree with this assessment?
Which other model checks should be done?
The 'k.check' function gives this output which does not look like standard output from the k.check function with smooth interaction terms:
                        k'       edf  k-index p-value
s(temp_cont)             4  3.520319 1.139948  0.9775
s(temp_cont,spec)       15  7.000248       NA      NA
s(temp_cont,moist)      10  4.059748       NA      NA
s(temp_cont,spec,moist) 30 13.342120       NA      NA
s(block)                 4  2.567347       NA      NA
s.2(temp_cont)           4  2.206703 1.139948  0.9875

Answer: A problem with the first formulation is that there is nothing in the way the factor by smooths are set up that makes the smooths generated by the different terms orthogonal to one another and that can have implications for model identification.
A better way might be to fit your model as
library("readr")
library("mgcv")
library("dplyr")
library("gratia")

data_var <- readr::read_csv("~/Downloads/data_var.csv") |>
  mutate(block = factor(block),
  spec = factor(spec),
  moist = factor(moist))

m <- gam(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
  data = data_var,
  family = Gamma(link = "log"),
  method = 'REML')

which does two things:

we create a series of smooth-factor interactions that are orthogonal to lower order effects (i.e. the first order interaction is orthogonal to the main effect smooth s(temp_cont), and the second order interaction is orthogonal to the first order and main effects smooths), and
we fit the model with a more appropriate random component via family = Gamma(link = "log")

This model produces

and
> summary(m)

Family: Gamma
Link function: log

Formula:
response_var ~ s(temp_cont, k = 5) + s(temp_cont, spec, k = 5,
    bs = "sz") + s(temp_cont, moist, k = 5, bs = "sz") + s(temp_cont,
    spec, moist, k = 5, bs = "sz") + s(block, bs = "re")

Parametric coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  1.02427    0.03801   26.95   <2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Approximate significance of smooth terms:
                           edf Ref.df       F  p-value
s(temp_cont)             3.263  3.708 180.002  < 2e-16 ***
s(temp_cont,spec)        7.049  7.701  18.191  < 2e-16 ***
s(temp_cont,moist)       5.428  6.128 112.672  < 2e-16 ***
s(temp_cont,spec,moist) 13.052 13.633   2.988 0.000432 ***
s(block)                 2.356  3.000   3.697 0.003353 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

R-sq.(adj) =  0.896   Deviance explained = 87.8%
-REML =  326.5  Scale est. = 0.073748  n = 238

I used Gamma(link = "log") because your concentration response can't be negative and therefore has a mean-variance relationship and the Gamma is a reasonable starting point to model such data.

What will be the way to determine whether different levels of wiggliness for a categorical factor in te model are needed?

You already did that; by definition, the by smooths and the sz smooths I use allow for different wigglinesses as each smooth gets it's own smoothness parameter. If you want them to share a single smoothness parameter than you need to use the id = 1 for example:
m2 <- gam(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz', id = 1)+
    s(temp_cont, moist, k = 5, bs = 'sz', id = 2)+
    s(temp_cont, spec, moist, k = 5, bs = 'sz', id = 3)+
    s(block, bs = 're'),
  data = data_var,
  family = Gamma(link = "log"),
  method = 'REML')

which results in far few smoothing parameters being estimated in m2 because I set all the smooths for a give s() call the same id
r$> m$sp
             s(temp_cont)        s(temp_cont,spec)1        s(temp_cont,spec)2
             8.807430e-02              1.695839e+00              2.810039e+04
       s(temp_cont,spec)3        s(temp_cont,spec)4       s(temp_cont,moist)1
             1.803149e+00              7.822437e-05              2.702676e-05
      s(temp_cont,moist)2       s(temp_cont,moist)3  s(temp_cont,spec,moist)1
             6.038187e-01              4.021637e+00              3.395644e+04
 s(temp_cont,spec,moist)2  s(temp_cont,spec,moist)3  s(temp_cont,spec,moist)4
             2.008389e-01              2.922855e-01              3.173709e-04
 s(temp_cont,spec,moist)5  s(temp_cont,spec,moist)6  s(temp_cont,spec,moist)7
             5.505667e-01              3.936865e+00              6.172809e-05
 s(temp_cont,spec,moist)8  s(temp_cont,spec,moist)9 s(temp_cont,spec,moist)10
             4.159845e-05              2.151653e+04              1.870382e+01
s(temp_cont,spec,moist)11 s(temp_cont,spec,moist)12                  s(block)
             2.412331e+04              2.915978e+04              1.625096e+01

r$> m2$sp
           s(temp_cont)       s(temp_cont,spec)      s(temp_cont,moist) s(temp_cont,spec,moist)
           9.183549e-02            3.698062e+00            6.679556e-01            1.990463e+03
               s(block)
           1.672849e+01

But doing so results in a worse fit
> AIC(m, m2)
         df      AIC
m  36.03123 577.4602
m2 33.93994 580.5186

I would also compare such a model with the same linear model (lme4 model) using AIC to further show that GAM would be preferred over a linear model, if this is the case. Will such a comparison be valid? I have read that the AICs are differently calculated for the GAM vs. the lme4 model.

You should ideally fit the linear model using {mgcv} and gam() and then compare the two gam() models. You don't say what linear mixed model you want to fit, so I can't help with the gam() equivalent, but it is usually possible to get the same result as lmer() and glmer() with gam() except for the use of correlated random effects.

The other issue is that the overall model adjusted.r2 is 0.9 (for the three-way interaction gam model shown above), which suggests overfitting

Your model is not over fitted, nor is my version of it.
That's not to say there aren't problems - it seems the implied mean variance relationship of the Gamma is not appropriate for these data and a Tweedie random component does a better job (the variance scales as $\mu^{1.2}$ in the models below while the Gamma assumes it scale as $\mu^{2}$, for example.
# Fit as a Tweedie instead of a Gamma GAM
m3 <- gam(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
  data = data_var,
  family = tw(link = "log"),
  method = 'REML')

# Fit a location-scale Tweedie, where we allow the power parameter
# to vary with `moist`
m4 <- gam(list(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
    ~ 1,
    ~ moist),
  data = data_var,
  family = twlss(),
  method = 'REML')

# Fit a location-scale Tweedie, where we allow the power parameter
# to vary with `moist` and smoothly in `temp_cont`
m5 <- gam(list(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
    ~ 1,
    ~ moist + s(temp_cont, k = 5)),
  data = data_var,
  family = twlss(),
  method = 'REML')

Each of the Tweedie models fits better than the Gamma
> AIC(m, m2, m3, m4, m5)
         df      AIC
m  36.03123 577.4602
m2 33.93994 580.5186
m3 39.47912 535.0788
m4 36.38935 522.2200
m5 37.75694 518.8516

although I didn't try similar location-scale-shape models for the Gamma, which could be done via the gammals() family:
# Fit a location-scale-shape Gamma, where we allow the shape parameter
# to vary with `moist`
m6 <- gam(list(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
    ~ moist),
  data = data_var,
  family = gammals(),
  method = 'REML')

# Fit a location-scale-shape Gamma, where we allow the shape parameter
# to vary with `moist` and smoothly in `temp_cont`
m7 <- gam(list(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
    ~ moist + s(temp_cont, k = 5)),
  data = data_var,
  family = gammals(),
  method = 'REML')

and it does seem that allowing the shape parameter of the Gamma to vary with moist and smoothly in temp_cont that we get the better fit and model diagnostics:
> AIC(m, m2, m3, m4, m5, m6, m7)
         df      AIC
m  36.03123 577.4602
m2 33.93994 580.5186
m3 39.47912 535.0788
m4 36.38935 522.2200
m5 37.75694 518.8516
m6 39.80994 524.5466
m7 42.57224 517.0190

I'm not suggesting you go that far, but I think you are worrying about the wrong things with the concern about whether you are over fitting, whether to use varying or the same smoothing parameters etc, when there are bigger problems such as your fitting a model that is objectively wrong for the type of response you are modelling - assuming it is conditionally Gaussian when you can't have negative concentrations is a bigger issue than worrying about how many smoothing parameters you end up using.

A few comments on the additional questions raised by the OP in their edited question.

I'm curious about the decision-making process in determining which factor should govern the variation of the power parameter ('moist' or 'spec' in this case)

Beyond noting that this was a 5 minute quick look at your data and not a proper analysis (that last bit is up to you to do, or I am available at standard consultancy rates ;-)) the main things to note here are

building a distributional model is quite difficult and there often aren't good rules for how to go about building them.
there is much less information in the data about these other distributional parameters than there is about the mean. Hence, all else equal you will typically want to keep the linear predictors for the other distributional terms simpler than the linear predictor for the location/mean parameter.
Matteo Fasiolo's mgcViz package has some tools to help with diagnosing which linear predictors to enchance through predictors.

More specifically, I think I tried moist and spec in both the linear predictors for the $\sigma$ and $p$ parameters, but once moist was included spec didn't seem that useful. A model with moist * spec seems to be a slight improvement over the one I mentioned (m5)
m8 <- gam(list(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
    ~ 1,
    ~ moist * spec + s(temp_cont, k = 5)),
  data = data_var,
  family = twlss(),
  method = 'REML')

but it seems this is mainly due to a single species. So, perhaps it is best to keep spec in the model as that reflects the design of the experiment. It also seems to lead to a more precise estimate of the smooth effect of temp_cont in the linear predictor for $p$.
And for completeness, here's the same model with the gammals() fit:
m9 <- gam(list(response_var ~ s(temp_cont, k = 5) +
    s(temp_cont, spec, k = 5, bs = 'sz')+
    s(temp_cont, moist, k = 5, bs = 'sz')+
    s(temp_cont, spec, moist, k = 5, bs = 'sz')+
    s(block, bs = 're'),
    ~ 1,
    ~ moist * spec + s(temp_cont, k = 5)),
  data = data_var,
  family = gammals(),
  method = 'REML')

and there's isn't much to choose between the two, with the twlss() version (m8) performing slightly better
> AIC(m, m3, m4, m5, m6, m7, m8, m9)
         df      AIC
m  36.03123 577.4602
m3 39.47912 535.0788
m4 36.38935 522.2200
m5 37.75694 518.8516
m6 39.80994 524.5466
m7 42.57224 517.0190
m8 49.24810 514.6135
m9 50.75916 516.0505

IIRC adding terms to the linear predictor for $\sigma$ started to lead to computational issues so I just focused on $p$.

I recently learned online that an effective degree of freedom (edf) equal to 1 or lower indicates a linear relationship, while edf greater than 1 suggests nonlinearity.

That's not always strictly true with {mgcv; using select = TRUE we can often get EDF < 1 while have non-linear estimated functions.
I'm not sure what you were extracting but the EDF of the smooths are all > 1 in m5:
> edf(m5)
# A tibble: 6 × 2
  .smooth                  .edf
  <chr>                   <dbl>
1 s(temp_cont)             3.52
2 s(temp_cont,spec)        7.02
3 s(temp_cont,moist)       4.02
4 s(temp_cont,spec,moist) 13.4
5 s(block)                 2.57
6 s.2(temp_cont)           2.21

If you're just looking at what is in m5$edf1 or m5$edf2 then what you are looking at are the contributions to the EDF of the model/smooth per model basis function. These should, IIRC, all be <= 1 as we've penalised the coefficients for these basis functions and they can't be greater than 1 as each basis function contributes max 1 degree of freedom to the model - it would be 1 in an unpenalized model. What you need to do is to sum the EDF by smooth, which is what the gratia::edf() function did to generate that output.

Additionally, I fitted a linear model using gam function, which will be the linear equivalent of the m5 model and compared AIC values between this model and the m5 model.

Note that this model isn't completely linear — you left the smooth in the $p$ linear predictor. But otherwise, yes, this would be a linear equivalent of the model.

What does the discrepancy between the fact that AIC would prefer the GAM model wile all 'edf' values are equal to 1 or lower signify?

There is no discrepancy; you don't seem to have computed the EDF for the smooths correctly. Look at the EDF in the output from summary(m5) to see that the EDFs are > 1.

In my interpretation, these deviations might still be acceptable. Would you agree with this assessment?

Not normally, no. But this QQ-plot is assuming that the residuals are Gaussian. Ideally we'd use reference bands to guide our choice of reference quantiles, but the methods available for that don't work with twlss() it seems.
We see somewhat similar deviations for the gammals() model (m9), especially in the upper tail.

However, this discrepancy is well with the range of discrepancies from the 1:1 line we see if we simulate residuals from the model, which by definition must have the correct distribution. As shown by the reference band. I'm not 100% sure why the more nuanced methods aren't working for twlss() - they don't work for all the families in mgcv, but the necessary random number generator is in mgcv.

The 'k.check' function gives this output which does not look like standard output from the k.check function with smooth interaction terms

This is typical. The test that k.check() does doesn't work for bs = "re" and also it seems for the bs = "sz" basis, I suspect this is because of the way this basis is implemented.
For further diagnostics, I would see the mgcViz package.
Comment: holy cow that's a thorough answer.
Comment: Thank you very much for your response. I added my further questions to the original post below the dashed line.
Comment: @RadimSarlej I've added some responses to the new questions, but if further clarification is needed, consider doing that in new questions that can focus on a specific statistical issue at a time.
Comment: @Gavin Simpson - I am still digesting it all. To make sure I understand correctly, the model with bs = 'sz' does not include the main effect of factors in the model formula as opposed to a model with bs = 'tp' specified in the smooths. I understand that the main effects are still calculated for such a model with bs = 'sz'. Am I right?


"Add an item to a list" using rule just works for plain text

Question: I created a rule with an action which "Add an item to a list".
The value of the item to add is this (also see my picture bellow)
Voucher: [node:title].
The code for this voucher is [codigo-otorgado:value].
View the voucher: [node:url]

This item will be added to a field of the user which I created at admin/config/people/accounts/fields The details of this field are:
Name: My voucher codes
Type: Long text
Widget: Text area (multiple rows)
Text processing: Plain Text
Number of values: Unlimited

So far so good. When the rule is fired all is working right, but The problem is that the text is shown as plain text. As you can see there is a url and breaks lines that I need to print out.
I guess the first suggestion will be to change the Text processing of the field  to Filtered Html or Full Html, and so I did.
But then the tokens ([node:url], [node:title]..) inside the value doesn't work. The tokens are printed as it, instead of their values.
I know that Rules is very mature module and I'm sure there must be a way to do this. What am I missing here?
Answer: You have saved your textarea field as Plain text, change it to Filtered Text (in Text formats admin settings you should check allowed tags):
Comment: Ok, I get the select list to change to Full Html. I works, but tokens are not working in this format. I found the token filter module which is useful to get global tokens but is not useful for node tokens like [node:url]
Comment: Please try next module then: http://drupal.org/project/token_insert


Finding the mean of right-censored values

Question: I wanted to check how fast an average colleague of mine is able to complete a puzzle, so I ran some experiments. The problem is they weren't done in exactly 50 minutes, I always got bored and moved on to the next person. Hence my data is right-censored at 50. Let's say it looks like this:
Aaron: 34
Betty: X
Cecil: X
Dave: 18
Elias: 47
Fredric: 39
Greta: 37
Howard: X
Ingrid: X
Joy: 29
Kristine: 34
Lou: 32
Margret: X
Nigel: 40

An X means that they did not complete it in 50 minutes.
How do I use this data to find how fast on average person in my company is in completing the puzzle? To be more clear: How do I find the mean or the median of the above data with a bootstrap confidence interval? I don't want to just throw out the data where the person did not complete the task within 50 minutes, but I don't know how to incorporate those data points in my analysis.
Comment: The mean can abviously not be computed without further assumptions and the median should be no problem, as long as less then half of the population has no `X`.  Just replaxe `X` with a very large number (if you are doing this in `R` I suggest `Inf`) and compute the median. Depending on the number of `X` you might also consider a trimmed mean.
Comment: @Bernhard. Thanks. It may happen that more than half of the population has 'X'. Does that mean that I cannot draw any conclusions?
Comment: Intuitively, without math, what do you expect the mean of $(3, 5, 9, X, X, X, X, X, X)$ to be? More or less then $20$?
Answer: The bootstrap (alone) will not help you identify the mean in the presence of censoring, and will not be much use with the median given a dataset like the one you supplied; you may be able to get some kind of lower bound on the median in this case with a bootstrap, but you can do that nonparametrically without the need for bootstrapping.
To estimate the mean would require some kind of parametric assumption*.
If you observe fewer than half the times to completion then the median would have a similar problem.
This is a standard kind of problem in survival analysis; once a suitable parametric distributional model is chosen, estimation is relatively easy with canned routines. I've done this in R; it provides a number of distributions by default but it's possible to add further distributions.
First we create a dataset with time under observation and a column to denote the censoring status (first few rows shown below). Here I fit an intercept-only Weibull model to the times
  subject time completed
1   Aaron   34         1
2   Betty   50         0
3   Cecil   50         0
4    Dave   18         1
5   Elias   47         1
6 Fredric   39         1

(weib.fit <- survreg(Surv.puz~1,puzzle,dist="weibull"))
Call:
survreg(formula = Surv.puz ~ 1, data = puzzle, dist = "weibull")

Coefficients:
(Intercept)
   3.879441

Scale= 0.3010924

Loglik(model)= -40.8   Loglik(intercept only)= -40.8
n= 14

Here's the Kaplan-Meier survival curve (nonparametric), along with 95% CI, and the fit from the above model:

The sample median is 39.5, the fitted median is 43.34, the fitted mean is 43.43. You can see from the nonparametric CI that we can get a lower bound on the median without a parametric assumption (where the horizontal green dots hit the vertical black dashes) but there's not a large enough proportion of complete times to get an upper bound on the median. The parametric fit can provide CI-type bounds on both the mean and median (though I haven't given any here).
You could try a variety of parametric assumptions if you don't want to be too reliant on any one assumption, but you're still reliant on whatever set of distributions you consider -- there's no way to avoid some assumption about the upper tail beyond what you observed.
* well it might be possible to proceed with some assumption that sufficiently regularizes the problem without being fully parametric (obtaining some set of bounds, presumably), though I am not sure right now what form that assumption might take.
Here's the code that I used. First read in and set up data:
puzzle <- read.table(stdin(),colClasses=c("character","numeric"),
             na.strings="X",sep=":",col.names=c("subject","time"))
Aaron: 34
Betty: X
Cecil: X
Dave: 18
Elias: 47
Fredric: 39
Greta: 37
Howard: X
Ingrid: X
Joy: 29
Kristine: 34
Lou: 32
Margret: X
Nigel: 40

 puzzle$completed <- as.numeric(!is.na(puzzle$time))
 puzzle$time <- with(puzzle,ifelse(is.na(time),50,time))

Then load the survival library and create the required survival object,
fit the Weibull and plot the curves etc
 require(survival)
 (Surv.puz <- with(puzzle,Surv(time,event=completed)))
 (weib.fit <- survreg(Surv.puz~1,puzzle,dist="weibull"))

 plot(survfit(Surv.puz~1),xlim=c(0,100))
 lambda <- exp(weib.fit$coefficients)
 k <- 1/weib.fit$scale
 f <- function(x) pweibull(x,k,lambda,lower.tail=FALSE)
 curve(f,0,100,col="blue",add=TRUE)
 weib.mean <- lambda*gamma(1+1/k)
 weib.median <- lambda*log(2)^(1/k)
 puz.median <- median(puzzle$time)

 abline(h=c(0,1),col="darkred",lty=3)
 segments(0,.5,max(weib.median,puz.median),.5,col="green4",lty=2)
 segments(weib.median,0,weib.median,.5,col="green4",lty=2)
 segments(puz.median,0,puz.median,.5,col="green4",lty=2)
 arrows(weib.mean,-.03,weib.mean,0,length=0.05,col="green4")
 c(data.median=puz.median,weib.median=weib.median,weib.mean=weib.mean)

Comment: +1 I... I just can't help it! The white background of graphs clashes really hard with the faint graph-paper background of the site... I am repeatedly compelled to edit graphs for transparency... there's probably a 12-step program for this kind of thing. ;)
Comment: @Alexis Well, I'll leave it as it is for your sake, but personally I prefer the white background to having the page background showing through -- it's not enough to bother me, but I like the "clean paper" look of the white background on plots. On my monitor the difference is pretty subtle though.
Comment: How are you able to get the diagram and the data (sample median and fitted median) using the above R code?
Comment: I didn't show every bit of R code I used, but this can be worked out from the parameters estimates above using information in the help of the relevant functions. In terms of the parameterization of the Weibull given at [Wikipedia](https://en.wikipedia.org/wiki/Weibull_distribution), $\hat\lambda=\exp(\tt{Intercept})$ and $\hat{k}=1/\tt{Scale}$, from which the maximum likelihood estimates for the median and mean may be computed by substitution. You can draw a Weibull survivior function via a simple call to `pweibull`. The Kaplan-Meier 'curve' is drawn via `plot.survfit`.
Comment: ... I only kept the code a few days; I'd have to regenerate it from scratch. Actually, I see a small error in my copy of your data, which I will also fix. (now done. Apologies to Alexis; I'll try to fix up the transparency later.)
Answer: You can use interval regression on a constant, since you have a mixture of point data, which can be expressed as $[t_1,t_2]$ where $t_1=t_2 < 50$, and right-censored data, where completion time is $[50,+\infty)$. In this case, the expected duration is 43 minutes, instead of the 40 you get from ignoring the right-censored folks. This model relies on the error being normally distributed with mean zero and homoskedastic variance. This is the sort of strong parametric assumption that Glen_b is rightly concerned about.
Here's the example in Stata:
. clear

. input str8 name str2 time

          name       time
  1. "Aaron" 34
  2. "Betty" X
  3. "Cecil" X
  4. "Dave" 18
  5. "Elias" 47
  6. "Fredric" 39
  7. "Greta" 37
  8. "Howard" X
  9. "Ingrid" X
 10. "Joy" 29
 11. "Kristine" 34
 12. "Lou" 32
 13. "Margret" X
 14. "Nigel" 40
 15. end

.
. gen t1 = cond(missing(real(time)),50,real(time))

. gen t2 = cond(missing(real(time)),.,real(time))
(5 missing values generated)

. intreg t1 t2, nolog

Interval regression                             Number of obs     =         14
                                                   Uncensored     =          9
                                                   Left-censored  =          0
                                                   Right-censored =          5
                                                   Interval-cens. =          0

                                                LR chi2(0)        =       0.00
Log likelihood =  -40.91248                     Prob > chi2       =          .

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _cons |   43.20936   4.073829    10.61   0.000      35.2248    51.19391
-------------+----------------------------------------------------------------
    /lnsigma |   2.634876   .2589092    10.18   0.000     2.127423    3.142329
-------------+----------------------------------------------------------------
       sigma |   13.94158   3.609604                      8.393212    23.15773
------------------------------------------------------------------------------

If you still want to boostrap, that can also be done:
. bs, reps(100): intreg t1 t2, nolog
(running intreg on estimation sample)

Bootstrap replications (100)
----+--- 1 ---+--- 2 ---+--- 3 ---+--- 4 ---+--- 5
..................................................    50
..................................................   100

Interval regression                             Number of obs     =         14
                                                   Uncensored     =          9
                                                   Left-censored  =          0
                                                   Right-censored =          5
                                                   Interval-cens. =          0
                                                Replications      =        100

                                                Wald chi2(0)      =          .
Log likelihood =  -40.91248                     Prob > chi2       =          .

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
       _cons |   43.20936    4.94787     8.73   0.000     33.51171      52.907
-------------+----------------------------------------------------------------
    /lnsigma |   2.634876   .2534669    10.40   0.000      2.13809    3.131662
-------------+----------------------------------------------------------------
       sigma |   13.94158    3.53373                      8.483219    22.91203
------------------------------------------------------------------------------

Comment: Cool, thank you! I was going to experiment with this myself, but I saw that Strata is proprietary software. Do you have any pointers on how I can do this in Python or R?
Comment: Try the `censReg` package in R. This simple case is also called a Tobit model.


structural break

Question: Say I have the equation:
$y_t = \alpha + \beta*\text{time}$
I'm looking for a structural break test that allows both parameters ($\alpha$ and $\beta$) to potentially change with a structural break. However rather than test these out individually, or jointly, I want a SB test that, say detects a break or series of breaks, and tells you whether the break thus found was $\alpha$ break, a $\beta$ break, or both.
Is there such a test -- that can identify a break and then the type of break?
Comment: You have just defined a research agenda for about a decade. Look at Andrews 1993 and Andrews and Ploberger, and sup tests and CUSUM tests. There are estimators of break dates but usually just for one break. The most rudimentary estimators rely on you eyeballing the break and then see whether it's there.
Comment: "identify break/classify break type" and "test" are not synonyms. You may like to consider whether your interest is more on identifying the break (and type) or on formal testing.
Answer: As pointed out in the comments, you are asking about two different things: testing for a structural break in each of the parameters and estimating (or dating) the structural breaks, if any.
As for testing: The Nyblom-Hansen test would be a natural candidate here. It's a test based on the scores (i.e., contributions to the gradient of the model) and was first proposed by Nyblom (1989, Journal of the American Statistical Association, 84, 223-230) for a single parameter, later extended by Hansen (1992, Journal of Policy Modeling, 14, 517-533), and shown to by trend-resistant by Ploberger and Krämer (1996, Journal of Econometrics, 70, 175-185). Hansen (1992) proposes to apply this test jointly to all parameters (in your case: intercept and slope) but also separately to each parameter. As I have pointed out in Zeileis (2005, Econometric Reviews, 24, 445-466) the latter is fine but you should apply a simple Bonferroni correction to account for carrying out two separate tests (one for each parameter). This would give you what you want: separate tests for intercept and slope, either of which can or cannot be significant.
As for dating: I wouldn't know of an algorithm that estimates separate breakpoints in the two parameters. Usually, breakpoints are selected such that all parameters in the model change simultaneously, e.g., as in Bai & Perron (2003, Journal of Applied Econometrics, 18, 1-22). If you apply that method and get multiple breakpoints, you can also judge easily whether each change is driven more by the intercept or the slope. An idea to formalize separate breakpoint estimates would be an EM-type approach, i.e., estimate the breaks in the intercept given the breaks in the slope and vice versa and then iterate until convergence. But as I said above, I haven't seen something like this in the literature, yet.

structural break


creating filter by fields

Question: I'm having a content type publications and researchers associated with these publications. I want to create a filter by the name of the researchers.  I created a relationship in the view, and I created a contextual filter, but it's not working.
Right, now the filter is not showed in the view. If I create a sort criteria based on the researcher it's not working.
Answer: I create a entity relationship between in both content types, by creating a content type reference. Then, in the view I created a relationship the user. I created a contextual filter in the view using the user name.

creating filter by fields


If A and B are ideals of a ring, show that A + B = $\{a+b|a \in A, b \in B\}$ is an ideal

Question: If A and B are ideals of a ring, show that A + B = $\{a+b|a \in A, b \in B\}$ is an ideal
I have the ideal test but no clue as to what to do with it:

$a-b \in A$ whenever $a,b \in A$
$ra$ and $ar$ are in A whenever $a \in A$ and $r \in R$
Answer: You may be getting confused because you are using $A$, $a$, and $b$ to mean many different things.
We want to show that $A+B$ is an ideal. That is, we want to show that whenever $x,y\in A+B$, then $x-y\in A+B$; and whenever $x\in A+B$ and $r\in R$, we have $rx\in A+B$.
So let us take $x,y\in A+B$. Since $x\in A+B$, we can find $a_1\in A$ and $b_1\in B$ such that $x=a_1+b_1$; and since $y\in A+B$, we can find $a_2\in A$ and $b_2\in B$ such that $y=a_2+b_2$. So then
$$x-y = (a_1+b_1)-(a_2+b_2).$$
Now, can we rewrite it so that it is the sum of something in $A$ and something in $B$? Remember that $A$ and $B$ are ideals.
Likewise, if $x\in A$ and $r\in R$, then we can find $a_1\in A$ and $b_1\in B$ such that $x=a_1+b_1$. Then
$$rx = r(a_1+b_1).$$
Can you show that this can be written as the sum of something in $A$ and something in $B$?
You also forgot to check whether $xr$ is in $A$ (you did not say your ring is commutative, so you need to check both products). Same idea should work.
Comment: + 0.7 for the first sentence. + 0.3 for the rest :)
Comment: I think you meant $x-y=(a_1+b_1)-(a_2+b_2)$
Comment: @JosephSkelton: Thank you; indeed I did.
Answer: So, let $a+b, c+d \in A+B$ for $a,c \in A$ and $b, d \in B$. Then $$(a+b)-(c+d)=a+b-c-d=(a-c)+(b-d)\;.$$ Since $A$ and $B$ are ideals then $a-c \in A$ and $b-d \in B$, so $(a+b)-(c+d) \in A+B$.
Then consider $r(a+b)$ for some $r \in R$. Then $r(a+b) = ra+rb$. Since $A$ and $B$ are ideals then $ra \in A$ and $rb \in B$; thus $r(a+b) \in A+B$. Make a similar argument for $(a+b)r$ ...
Therefore, $A+B$ is an ideal.
Answer: Let $x,y\in A+B$.  We want to show that $x-y\in A+B$.
First, $x = a+b$, for some $a\in A$, $b\in B$, and $y = c+d$, for some $c\in A$, $d\in B$.  Then $x - y = (a-c) + (b-d) \in A+B$, since by assumption, $A$ and $B$ are ideals, and hence $a-c\in A$, for $a,c\in A$, and $b-d\in B$, for $b,d\in B$.
Let $r\in R$, and $x\in A+B$.  Then $x = a+b$ for some $a\in A$, $b\in B$.  Then $rx = ra + rb$, and by assumption, this is in $A+B$, since $ra\in A$, and $rb\in B$.  Similarly, $xr = ar + br \in A+B$.  This proves that $A+B$ is an ideal.


Quando usar método mágico __contructor ou set e get?

Question: Minha dúvida é com relação construtor, por exemplo, tenho uma classe com nome, idade. O correto é usar __construtor para passar valores a eles ou usar set e get?
Answer: Primeiro você tem que responder se ter esses dados inicializados na instância da classe são fundamentais para considerá-la consistente desde o seu início.
A função do construtor é criar uma instância em um estado válido desde o início, ou seja, não deixar ter acesso até que todos os dados importantes sejam inicializados.
Eu vou acreditar que neste caso esses dois dados são necessários e portanto devem ser inicializados pelo construtor.
Mesmo assim nada impede de ter setters e getters para eles também, principalmente getter que faz algo diferente do que o construtor faz. Assim como mesmo tendo setter nada impede de usar a propriedade referente a ele no construtor, mesmo que ele não seja fundamental. É mais comum colocar nos construtores as propriedades essenciais para a classe mas também há casos que pode-se colocar propriedades extras no construtor para facilitar a vida do programador usuário da classe.
Mas se pesquisar aqui, nem sempre eles devem ser usados.
Coloquei no GitHub para referência futura.
Comment: Pode, claro, se estes dados não precisam ser alterados depois, não só pode como deve evitar seu uso.
Comment: Conforme eu lhe disse se esse `geraValor()` for estático, pode. Ou se ele for externo a esta classe. Mas também como eu disse, estou me baseando em linguagens decentes. Como o PHP é a linguagem *mainstream* mais mal planejada que eu já vi, pode até ser que ele permita chamar qualquer coisa indiscriminadamente jogando no lixo a vantagem do construtor que é não permitir acesso à instância até que ela esteja completa.
Comment: se eu passar os dados no momento da instanciação da class, posso evitar de usar set se eu quiser, isso ?
Comment: Mais uma pergunta. Posso passar função da própria classe no construtor ? ex public function ____construct($nome, geraIdade()), é permitido fazer isso ?
Comment: Não pode porque a declaração do construtor ou qualquer função ou método só permite ter parâmetros. Não pode executar nada. Mas você pode fazer isto dentro corpo do construtor sob certas circunstâncias. Às vezes me confundo com o que pode em cada linguagem, não me lembro bem do PHP mas acho que só poderia chamar métodos estáticos da classe, afinal se for chamar um método que depende da instância inicializada o que ainda não aconteceu dentro do construtor. Só não garanto porque o PHP faz uma maluquices que ninguém consegue explicar. Método estáticos não teria porque impedir. Teste e veja por si.
Comment: ok, mto obrigado. Perguntei isso pq tenho um atributo que necessita ser instanciado no construtor, um gerador de id único, dai estou tentando resolver isso.
Comment: mas posso dentro do construtor chamar um método que retorna um valor para um atributo, sem referência-lo nos parâmetros ?
Comment: ex.: ... ' __construct($val){ $this->atr1 = $val; $this->atr2 = geraValor(); } '
Comment: blz, entend. obrigado mais uma vez
Comment: Quando está em dúvida, testa. Quando tiver problema, posta aqui. Se acha que tem erro mesmo funcionando, também.
Comment: testei e funcionou, mas fiquei na dúvida se isso é correto


What is the probability of getting at least 4 correct when guessing 12 people's Zodiac signs, where each person has a different sign?

Question: Let's say an astrologer is tasked with guessing the star signs of 12, for him, unknown students. The astrologer knows that all 12 star signs are represented among the 12 students, i.e. a single star sign will belong to one and only one student.
The astrologer can only guess one star sign per person. Additionally, he will have to assign each star sign to someone, i.e. each single star sign cannot be assigned to more than one student.
Given this premise, what is the probability that the astrologer guesses the correct star sign for at least 4 students?
Edit: I am not a probability student, I am just curious about how well the astrologer fared in this video. I started by thinking that the probability of randomly guessing each one correctly would be (1/12)(1/11)(1/10)...(1/2), but I'm not even sure if that is correct and I don't know where to start to answer my question.
Comment: This looks like an assignment for a class that has just discussed inclusion-exclusion formulas (https://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle#A_special_case). It is easier to calculate the probability of the astrologer guessing correctly for at least 0, at least 1, at least 10, or at least 11 students, and maybe you can work from there in one direction or the other.
Comment: This looks like a HW question. Those are OK here, but you should show what you have tried so far.
Comment: Research [partial derangements](https://en.wikipedia.org/wiki/Rencontres_numbers), also known as the *problème des rencontres.*
Comment: Your $(1/12)(1/11)(1/10)...(1/2)$  is correct for the probability of randomly guessing each one of the 12 correctly
Answer: Assuming there is no information about the students that is known to correlate with their star-sign (allowing this information to be used in the guess), this is an instance of a famous antique probability problem, which is variously called the "problem of coincidences" (or "problème des rencontres" in French), the "hat-check problem", the "secret-Santa problem", or the "classical matching problem".  In the general version of this problem there are $n$ distinct things arranged in a random order and you have to guess the true order of the things.  We are then interested in the distribution of the number of correct "matches".  (This problem can also be framed in many equivalent ways that sound slightly different to what I have described but are probabilistically equivalent.)
The distribution of the number of matches in this problem (your correct guesses) has been derived and discussed by a number of authors, going back to Mortmont (1708, 1713), and it is closely related to derangments and partial derangments.  You can find a detailed analysis of this distribution and a useful generalisation of it in O'Neill (2023).  The distribution has probability mass function given by:
$$\text{Match}(k|n) = \frac{!(n-k)}{k!(n-k)!} = \frac{1}{k!} \sum_{i=0}^{n-k} \frac{(-1)^i}{i!}
\quad \quad \quad \quad \quad
k = 0,...,n.$$
You can compute the probabilities from the matching distribution using the various probability functions in the stat.extend package.  The cumulative distribution function can be computed with the pmatching function.  In your problem you have $n=12$ things and you want to compute the probability of at least four matches, so you can use the following code:
library(stat.extend)
1 - pmatching(3, size = 12)
[1] 0.01898819

We see from this that the probability of at least four correct guesses is $0.01898819$, so it will happen around one in 52 or 53 times over the long-run.  If  you prefer, you can compute the probabilities for each possible number of correct guesses using the dmatching function to get the same answer.
You might also find it useful to note that the linked paper also examines a generalisation of this distribution, where we allow for some non-zero probability that the person can correctly guess an outcome from contextual information.  This would allow you to examine probabilities that would arise if your guesser has some other contextual information about the students that is related to star-signs, that allows them to guess "better than random".
Answer: Not sure I got it, but I lay out my process, which might help.
Let the set integers from 1 to 12 be the zodiac signs.
The space of guesses is $12!$, i.e. the permutations of this set.
A guess is correct when the number falls in the same place as it was originally.
Now, you want to know in how many cases every guess is wrong, one is correct, two are correct, and three are correct. Those four probabilities should be summed and subtracted from 1 to get your answer.
The probability that no sign is assigned to the right person is a derangement, and the number of derangements can be computed as a sub factorial $ !n = n!/e$ rounded to the closest integer.
What about the others?
The probability of having a single guess right is to have a derangement of $11$ and a correct guess, and you can have 12 different guesses, so you need to multiply by that. I think this can be easily extended to 2 and 3 correct guesses to finally get your answer.


Can the core Profile module (it is deprecated) be used? And how will I upgrade?

Question: The profile core module is being deprecated in D7.  It exists and operates in D7, but is hidden and can only be enabled with drush or a custom module.  Drupal.org recommends that users now use Profile2, a custom module, but my attempts to do so have met with WSOD on several attempts. In fact, after enabling the module, it will crash the site if you try to disable it.  The problem links to a core issue, but even with the recommended patch, Profile2 still has other problems I have not been able to correct yet.
In the meantime, I enabled Profile, and it has desirable features I would like to implement.  My concern is that whatever I do may be lost in the future.  Since I cannot load Profile2, I do not know if it builds on Profile or I will have to reconstruct everything later.
Comment: you can now extent the core user fields in D7 and might not need profile2 at all. Do you have any special requirements?
Comment: What is causing your WSOD? If you haven't done so, follow the instructions at http://drupal.org/node/158043 to resolve the problem.
Comment: Thanks for asking.  Profile2 is causing it, but by circumstances outside their control.  The cache clearing sequence has a problem that they have been chasing for months.  There was a recent patch that fixes that, but as soon as I tried again, apparently profile 2 had not installed fields in the database, it needs.  I have a report in with them.
Answer: First of all, it's important to understand something MotoTribe has already mentioned. The most important functionality of "Profile" is now in for Drupal 7 new "users are entities" concept. Since this allows you extend users with fields, the profile module was considered redundant.
If I recall correctly, the main reason for it remaining in Core, was as a transition path from D6 Profile. In D8 I don't believe Profile will exist at all, and since it's already deprecated, I very much doubt there will be an upgrade path from D7 profile.
If we instead follow the trail of issues that follow from yours on the Profile2 queue, we find a Core issue, which is so far advanced that it has been applied to D8, and applies cleanly against D7 with full test coverage. I would apply that D7 patch, and try with Profile2 again.
Comment: Thanks,  I did.  I can now disable the module without crashing the site, but run into a database table problem I think may be Profile2.  All I am trying to do is understand what these modules did to see if I need their functionality to make a user profile work better. I've enabled Profile to look at it and it handles some things such as private and public information, input forms etc. which would be nice to use. Without it, it looks like putting together a custom  module may be the only way.