What is $f'(x)$ if $f(x)$ is a cumulative function

Question: Let me preface by saying that I am completely ignorant of the math covered in Calculus 2, which for me includes integration. All I know is that the integral is the area under a curve, which I understand as the sum total of all data points up to a given value of $x$, and that the derivative of an integral is just the original function.
I'm working on a problem where I'm given $f '(t)$, and I know that $f(t)$ is the number of errors made by a shift worker from $t = 0$ to $t$. To me, $f(t)$ sounds like an integral, since it describes the total errors made by the worker till $t$. My question then is, would $f'(t)$ be some function (let's say $g$), whose integral is $f(t)$?
The difference in my analysis is that $g(t)$ would be the number of errors made at the instant $t$, while a typical derivative is be the rate that errors are occurring at $t$.
Comment: You say you are given the function f'(t) if it is given it must mean something measurable, since you can not make errors in an exakt moment t, you alway would need a time interval $\Delta t$ to make errors. Can you give an example how f'(t) is given and what exactly it means?
Comment: I'm not allowed to give out information directly, but f(t) is the total number of errors made from t = 0 to t, and f'(t) is the "instantaneous rate of change" of errors made.
Comment: Typically this $f(t)$ would change discretely (by jumps at each instant when an error is made), and so its derivative is not an ordinary function like you are used to. There is a concept for describing the derivative of a function with jumps, but I think at your level explaining this would be more confusing than illuminating.
Comment: By discretely do you mean something like a step function? Would it be easier to describe the antiderivative for this particular problem, since I'm given the derivative?
Answer: First of all I'll assume that this is a continous probability case and not discrete one, because if it's a discrete probability the integral interpretation doesn't apply. I'll also define cumulative function as $\mathbf F(t)$ and it's derivative as $\mathbf F'(t) = f(t)$.
Please note that nothing below applies if you are given an event probability table or a cumulative function that looks like this:
$$\mathbf F(t) = \begin{cases}
0.2, \ 0\leq t \leq 2\\
0.6, \ 2\leq t \leq 4\\
1, \ 4\leq t
\end{cases}
$$
This is a discrete case and no integral can be applied to it.
By definition, a cumulative function $\mathbf F(t)$ returns a probability that something is less than or equal to $t$. In your case I suppose it would return a probability that there was no more than $t$ errors made by a worker. Taking a derivative of that function, in some sense, would give you a rate of change, but more importantly the derivative of cumulative function $\mathbf F(t)$ is probability density function $f(t)$. That is, by taking a derivative you still get a function.
You can find online images of cumulative and probability density functions plotted together. They tell slightly different story. While $\mathbf F(t)$ tells the probability of something happening no more than $t$, the probability density allows you to take a look at the probability that some value of $t$ lies in some region $[a,b]$. That probability is in fact defined as the integral $\int_a^b f(t) \ dt$.
You can ask "What is the probability that a worker makes at least 3 and no more than 5 mistakes?". You can calculate the integral $\int_3^5 f(t) \ dt$ to find out.
Alternatively, you can make use of that definition and only use the cumulative $\mathbf F(t)$. Since $\mathbf F(t)$ tells the probability that your worker performas no more than $t$ mistakes, asking $\mathbf F(5) - \mathbf F(3)$ would produce the same result as integrating. To make it more obvious $\mathbf F(5) - \mathbf F(3) = \int_0^5 f(t) \ dt \ - \int_0^3 f(t) \ dt = \int_3^5 f(t) \ dt$, thanks to the properties of an integral.
Using the language of math, here you're considering the probability that your worker performs anywhere from 0 to 5 mistakes (which of course includes making  anywhere from 0 to 3 mistakes) and you subtract the probability that he makes 0 to 3 mistakes, leaving only the probability to make 3, 4 or 5 mistakes.
Comment: Yes, as I pointed out f'(t) and f(t) are related via said integral. The derivative actually tells you what are the most probable values of t (number of errors), where as the cumulative f(t) tells you the probability that there is no more than t errors made. Plot both of them on the interval of t = [0,10] to make it more visual, use wolframalpha if you aren't sure how to integrate that function. Also I'll say that a function doesn't "describe" an integral, an integral is just a number like 3 or 7. Functions, though, can be related via integral if h(x) = g'(x), then integral(h(x)) = g(x) + C.
Comment: I am almost certain that f(t) is continuous (not discrete) in this example, but I don't believe that it  gives the probability that an error will occur. f'(x) is based on a  data model following an individual's performance over the course of their shift. The analysis of the data is used to predict future trends, but the data itself is discrete. I'm sure some aspects of this problem were simplified to keep to the scope of the course.
Comment: I think you should provide a bit more details on things that you are given in this problem and things that you want to find. Depending on specifics, it may go either way - discrete or continous. It's difficult to suggest something other than exact definitions of what a cumulative F(t) and probability density f(t) are. I'm also not sure - are you actually given any function or is it just pure data? For example a probability distribution function could look something like this f(t) = g * e^(-g*t), where g is some constant and t is a variable.
Comment: Again, I'm mostly curious wether f(t) function describes an integral, and what the implications of that would be in my analysis of f'(t) (as opposed to the derivative of a typical function).
Comment: The weird thing is that both of these functions f'(t) and f(t) should be divided by 10 to make sense of the probability interpretation, but I guess these are up to semantics of the assignment.
Comment: Thank you. I think this is the best answer I can get with what I'm working with, so I'll run with it. If the function looks unusual, it's because I'm going a little outside the scope of the course in my analysis, (mostly just out of curiosity).
Answer: If you can plot f'(t) you just measure the area under the curve  from 0 to t to obtain f(t). So I still strongly doubt that one can measure the  "instantaneous rate of change"


Algorithm for Trading Resources

Question: I'm trying to find the best algorithmic solution to the following problem. It's a real-world problem, but I'm going to present it in an abstract manner.
There is a community of 1000 people. Each user is provided with a set number of tickets. There are four types of tickets (each one corresponds to a different event). However, some people are willing to make trades (for example, I want one A-ticket and am willing to give up two B-tickets). Moreover, some people have extra tickets that they are willing to give away for nothing (for example, I'll give away two C-tickets to whoever wants them). Assuming that I know what each person is willing to give away / trade, how do I satisfy the most number of people?
I tried Googling, but I don't know how to word this problem to avoid getting results related to algorithmic trading of financial instruments.
Thanks.
Comment: You need to define "satisfy". Do people get satisfied if they have at least tickets to some subset of {A,B,C,D} or a certain number of each type at minimum? Also if a person is willing to give away or trade, and you can't satisfy that person, is that person still willing to give away the ticket?
Comment: This algorithm not sufficiently well-defined. You do not say what it means to "satisfy" a person, whether they start with more than one type of ticket, etc. I would suggest editing the question; it may be closed soon otherwise.
Comment: Good point. So a person is satisfied when he or she ends up with the exact tickets (in terms of both quantity and type) that he or she wanted. The people who do not want to make any trades or are willing to give tickets away are always considered satisfied.
Comment: Just noticed the second question. So a person is only willing to make a trade if he or she can be satisfied by the trade.
Comment: When I added barter to my search keywords I started picking up papers on kidney exchanges, which is at least not financial instruments. That looks like a hard problem. One of the papers said research in combinatorial auctions wasn't quite applicable to them - might perhaps be to you.
Answer: Given that it has multiple dimensions, it is likely an NP-complete problem. It has parallels to a multi-dimensional knapsack problem.
Therefore, I recommend trying a backtracking approach.
Start with everybody involved in the trade.
Sort the people who cause the most deficit descending (here you can weight the deficit caused by each ticket type by the shortfall in each ticket).
Then in a backtracking manner, kick the person causing the next highest deficit person out of the trade.
Repeat until you either have no more deficit in any ticket (record as possible answer), or you have kicked everybody out.
When that happens, backtrack 1 step and continue (if you already tried kicking the highest deficit, kick the next highest deficit causing person).
Repeat until end or you run out of time. Get the optimal answer out of the possible answers you found.
If the problem is too hard, it will probably run out of time. Otherwise, this algorithm should give you a reasonable answer (perhaps near to optimal).
How well this method works depends on how generous/greedy the people are, how many people there are, and how fast your computer is.
Comment: Thanks! Especially for the reference to the knapsack problem.
Answer: Look for a bipartite minimum weigth matching problem. The idea is to find the shortest distance from i to j using vertices 1 .. k only.

Algorithm for Trading Resources


Decrease a column of values by a given percentage in R

Question: I have a dataset with a column of values (in dollars) that needs to be decreased by a percentage.
For example:
df=data.frame(matrix(0,nrows=4,ncol=2))
colnames(df) <- c("ID", "Cost")
df$Cost <- c(2500, 1600, 1200, 6000)
df$ID <- seq(1:4)
How would I decrease the "cost" column by 5%?
Any help would be greatly appreciated!
Comment: Answered, but I'm afraid the question is not in scope in this site and it could at best be migrated to Stack Overflow.
Answer: You just need:
df$Cost <- df$Cost * 0.95

Answer: In R the columns of dataframes must be the same data type. Meaning, if you run as.numeric(df$Cost) or as.integer(df$Cost) one of those must be true to perform arithmetic operations over them. If they are, you can perform a single operation over an entire column, which is very efficient and easy to read. For your example, I would write the following:
percent.decrease <- 5
df$Cost <- df$Cost * (100 - percent.decrease)/100

This way you can more easily change the percent decrease rather than doing the calculation elsewhere to arrive at 0.95 because mistakes happen.


Master-Detail relationship not cascading on insert in ADO.NET

Question: I attempting to handle the insert of a new master record and associated child records in Winforms using IronPython (developing in SharpDevelop 4) against SQL Server, although I think this is a general ADO.NET problem.  The form only ever has one master record (booking) and many detail records (lines). If the master record already exists I can insert child records without issue, but my relationships do not seem to be cascading if a new booking record is added.
Specific code - firstly I add my relationship as follows:
parentColumn = self._dataSetBookings.Tables["booking"].Columns["intBookingID"]
childColumn = self._dataSetBookings.Tables["lines"].Columns["intBookingID"]
bookingRelationship = DataRelation("mediabooking",parentColumn,childColumn,True)
self._dataSetBookings.Relations.Add(bookingRelationship)
self._dataSetBookings.EnforceConstraints = True

intBookingID is an identity column on the booking table
My apply update procedure looks like this, note the event added on row updated. The sql commands used are effectively 'select * from  where intBookingID = @intBookingID)
# update booking
adapter = SqlDataAdapter()
adapter.RowUpdated += self.AdapterUpdateBooking
cmd = adapter.SelectCommand = self.conn.CreateCommand()

cmd.CommandText = self.bookingSqlCmd
parameter = cmd.Parameters.Add("@intBookingID", SqlDbType.Int)
parameter.Value = self.BookingID

builder = System.Data.SqlClient.SqlCommandBuilder(adapter)
adapter.InsertCommand = builder.GetInsertCommand()
adapter.UpdateCommand = builder.GetUpdateCommand()
adapter.Update(self._dataSetBookings,"booking")

# update lines
adapter = SqlDataAdapter()
cmd = adapter.SelectCommand = self.conn.CreateCommand()
cmd.CommandText = self.lineSqlCmd
parameter = cmd.Parameters.Add("@intLineBookingID", SqlDbType.Int)
parameter.Value = self.BookingID

builder = System.Data.SqlClient.SqlCommandBuilder(adapter)
adapter.InsertCommand = builder.GetInsertCommand()
adapter.UpdateCommand = builder.GetUpdateCommand()
adapter.UpdateCommand = builder.GetDeleteCommand()
adapter.Update(self._dataSetBookings,"lines")

And finally the RowUpdated event - which only fires once against the booking row so we can obtain the identity of the record just inserted
    def AdapterUpdateBooking(self, sender, e):
    cmd = self.conn.CreateCommand()
    #cmd.CommandText = "SELECT SCOPE_IDENTITY()" - this does not work!
    cmd.CommandText = "SELECT @@IDENTITY "
    newBookingID = cmd.ExecuteScalar()

    if newBookingID != System.DBNull:
        for bookingRow in self._dataSetBookings.Tables["booking"].Rows:
            bookingRow["intBookingID"] = newBookingID

Tracing this through when the updates are applied the correct ID value is retrieved by the "SELECT @@IDENTITY" call for a new record (but SCOPE_IDENTITY returns null, which I do not understand and may be a clue?). However when the value is applied to the intBookingID field in the Master (booking) table it does not cascade update intBookingID field in the details (lines) table and the insert fails.
However if the Master (booking) record was loaded from the database, and new lines added, the the relationship does work, the intBookingID field correctly set, and the insert succeeds.
I feel I must be missing something trivial, but what am I doing wrong?
Answer: Have you coded for cascading? I don't see it. Try this.
ForeignKeyConstraint oFKey;
oFKey = new ForeignKeyConstraint("BookingForeignkey", parentColumn, childColumn);
oFKey.DeleteRule = Rule.Cascade;
oFKey.UpdateRule = Rule.Cascade;
oFKey.AcceptRejectRule = AcceptRejectRule.Cascade;
self._dataSetBookings.Tables["booking"].Constraints.Add(oFKey);
self._dataSetBookings.EnforceConstraints = True

You might want to read this:
DataTable Constraints (ADO.NET)
EDIT: I dont understand this part
if newBookingID != System.DBNull:
        for bookingRow in self._dataSetBookings.Tables["booking"].Rows:
            bookingRow["intBookingID"] = newBookingID

Why are you assigning all the rows 'newBookingID'? From this I am guessing primary keys might not have been defined for datasets. If they were, the assignment line must have thrown some error.
Comment: Doesn't a foreign key relationship get assigned automatically though when you assign a DataRelation?
Comment: I doubt. A DataRelation is just a simple relation. You can check this out in the Dataset designer. A foreignkey constraint has to be explicitly declared.
Comment: I did, and it does.  Actually I found the problem was a extraneous record insert and nothing to do with this code - but answer accepted because what you say is largely correct.


Can predicate and propositional logic be 'mixed'?

Question: Motivation
I wanted to write out a simple statement from linear algebra in a very formal, predicate logic way. The statement is that given two vector spaces $V \subset W$ over a field $\mathbb{F}$, $V$ is a subspace iff for any two vectors, any linear combination of them is contained in $V$.
I initially wrote $$\forall a \in V \forall b \in V \forall \alpha \in \mathbb{F} \forall \beta \in \mathbb{F} \big(V \text{ is a subsapce iff } a\alpha + b\beta \in V\big)$$
I realised this was wrong, and it should be (as confirmed by this set of notes I found)$$V \text{ is a subsapce iff } \forall a \in V \forall b \in V \forall \alpha \in \mathbb{F} \forall \beta \in \mathbb{F} \big( a\alpha + b\beta \in V\big)$$
It felt wrong in the first instance that the universal quantifiers extend over the statement that $V$ is a subspace.
Question
However, my issue is with proving these two are different. The way I see it, it's akin to trying to showing that $$\forall x P(x) \iff Q \text{ and } \forall x (P(x) \iff Q)$$ are different statements. Here I write $P(x)$ to be analogous to arbitrary vectors/scalars, over which we use a universal quantifier, and I write $Q$ as a statement that is independent of the variable $x$, to represent the statement that a vector is a subspace. To my mind, $Q$ is like a propositional statement, that is either simply either true or false.
But I am failing to to prove the two are not equivalent statements (I also tried showing that they ARE the same statements). In particular, I tried writing out truth tables, but got stuck and came across this post which explains that quantifiers do not work with truth tables.
Are these statements equivalent? And how do you go about proving it? (I realise my discussion of the logic here is likely shaky at best, so apologies if I'm under obvious misconceptions.)
Answer: You're right that these statements are different.
The quickest way to see this is by looking at the claim
$$
\forall a,b \in V . \forall \alpha, \beta \in \mathbb{F} . (V \text{ is a subspace} \iff \alpha a + \beta b \in V)
$$
If this is true, then (basically by definition) we should have
$$
(V \text{ is a subspace} \iff \alpha a + \beta b \in V)
$$
for every choice of $a,b \in V$ and $\alpha, \beta \in \mathbb{F}$. (Do you see why?)
In particular, this must be true when $a,b = 0$ and $\alpha, \beta = 0$. But in this case the claim reads
$$
(V \text{ is a subspace} \iff 0 \in V)
$$
which we know is false!

Said in the more abstract terms you adopts later, you're exactly right that
$$
\forall x \in X. (P(x) \iff Q) \quad \quad (1)
$$
is different from
$$(\forall x \in X . P(x)) \iff Q \quad \quad (2)$$
and the same argument works (as long as the domain $X$ has more than one element).
In formula $(1)$ we're saying that $P(x) \iff Q$ is true for every $x \in X$. So in particular, this tells us that $P(x_1) \iff Q \iff P(x_2)$ so that $P$ is constant on its domain $X$.
In formula $(2)$ we're saying that $Q$ is true exactly when $P(x)$ is always true. This is what we usually want to talk about.

But how can we formally prove that $(1)$ and $(2)$ are different? We need to exhibit a countermodel. We actually already have a countermodel from our first example of vector subspaces, but here's a small one you can really get your hands on:
Let $X = \{a,b\}$, with $P(a) = \top$, $P(b) = \bot$, and $Q = \bot$.

Can you show that this model thinks $(1)$ is false?
Can you show that this model thinks $(2)$ is true?
From here, conclude that $(1)$ and $(2)$ must not be provably equivalent. If they were, they would have to be equivalent in all models, but here we've built a model where they differ.

I hope this helps ^_^
Comment: The idea of a countermodel is new to me, so this is excellent! Thank you!


Which JDBC drivers support scroll sensitive/insensitive properly?

Question: I just found out that the Postgres Java JDBC driver does not really support the SCROLL_SENSITIVE/SCROLL_INSENSITIVE modes using streaming, but instead simulates those modes by loading the full result set into client memory all at once. For queries with a big result set, that can lead to an unexpectedly huge memory usage, especially in a language like Java with little support for unboxed values. When using FORWARD_ONLY mode, the driver streams the results as expected.
(details; From my understanding this is a limitation of the Postgres wire protocol, though the driver could maybe in theory work around that by converting queries into explicit cursors.)
To prevent such surprises in the future, I'm wondering how wide spread such behavior is and which other common JDBC drivers do not implement real scrolling where that would be expected, so I know to watch out for that if I happen to use one of those drivers. Which other common JDBC drivers do not support all of the scrolling modes in a streaming fashion?
Comment: JDBC is filled with differences between databases and sometimes even between drivers (not driver versions) for the same database. It's too broad to start listing them here.
Comment: Your question is rather unclear. Are you looking to find other drivers that do the same as the driver you've specified here? Or alternative drivers that do not? Certainly, all drivers are not created equal. You might try [the ones from my employer](http://uda.openlinksw.com/odbc-postgres/). Databases differ too; [Virtuoso comes from my employer](http://virtuoso.openlinksw.com/).
Comment: I edited the question to clarify the intent. I'm looking for drivers that do not implement streaming scrolling similar to the postgres driver, so I know to watch out for such behavior if I happen to be using one of those drivers.
Answer: MySQL/MariaDB is slightly worse than Postgres. Like Postgres it will buffer the entire query result by default. Also like Postgres it can only do forward scrolling without buffering, but unlike Postgres the database connection is occupied while a query result has not been exhausted and cannot be used for other queries at the same time. Any locks and other resources are also held until the result set has been exhausted, so leaving a result set open for a long time is not recommended. ref
Oracle streams results by default, but the server or network protocol does not support scrolling backwards. The JDBC driver emulates scrolling by caching the results if SCROLL_INSENSITIVE or SCROLL_SENSITIVE is set. So other than the default this is similar to Postgres. ref


Prove that $\left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right)\geq4$ with $a>0, b>0 , c> 0$ and $d>0.$

Question:
Prove that
  $\left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right)\geq4$
  with $a>0, b>0 , c> 0$ and $d>0.$

My attempt:
$$\begin{align*}\left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right)& =  \dfrac{abcd+b^2c^2+a^2d^2+abcd}{abcd}\\
    & =\dfrac{b^2c^2+a^2d^2+2abcd}{abcd}\\
     &=\dfrac{b^2c^2+a^2d^2+2abcd}{abcd}\\
&=\dfrac{(ad)^2+(bc)^2+2(ad)(bc)}{abcd}\\
&=\dfrac{(ad+bc)^2}{abcd}\end{align*}$$
I don't know how to continue from this.
Can someone help me?
Comment: One can start with AM-GM; $ (b/a+d/c)(a/b+c/d) \ge =2\sqrt{(bd)/(ac)} \times 2\sqrt{(ac)/(bd)} = 4$
Comment: Cauchy-Schwarz inequality.
Answer: Use AM-GM. $\frac{ad + bc}{2} \ge \sqrt{abcd}$. Squaring both sides, you get the answer. A tiny tip: If everything is positive, and you have an inequality, think about AM-GM once at least.
Answer: Taking a different approach entirely, note that
$$\left({b\over a}+{d\over c}\right)\left({a\over b}+{c\over d}\right)=1+{ad\over bc}+{bc\over ad}+1$$
Thus, letting $ad/bc=x$ and noting that $x\gt0$, we see that
$$\left({b\over a}+{d\over c}\right)\left({a\over b}+{c\over d}\right)\ge4\iff x+{1\over x}\ge2\iff x^2-2x+1\ge0\iff(x-1)^2\ge0$$
(Note, the stipulation $x\gt0$ is important is multiplying both sides of the inequality $x+1/x\ge2$ by $x$ to get to $x^2+1\ge2x$.)
Answer: Now,     $$\frac{(ad+bc)^2}{abcd}-4=\frac{a^2d^2-2abcd+b^2c^2}{abcd}=\frac{(ad-bc)^2}{abcd}\geq0.$$Also, by C-S
$$\left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right)\geq\left(\sqrt{\frac{b}{a}\cdot\frac{a}{b}}+\sqrt{\frac{d}{c}\cdot\frac{c}{d}}\right)^2`=4$$
Answer: To continue from
$\tag 1 \left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right) = \dfrac{(ad+bc)^2}{abcd}$
set
$\quad u = ad$
and
$\quad v = bc$
Then substituting in the rhs of $\text{(1)}$, we have
$\quad \dfrac{(u+v)^2}{uv} \ge 4 \text{ iff } (u-v)^2 \ge 0$
Note that if we let $a,b,c,d \in \Bbb R$ satisfy $abcd \gt 0$ then
$\quad  \left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right) \ge 4$
and if $abcd \lt 0$ then
$\quad  \left(\dfrac{b}{a}+\dfrac{d}{c}\right)\cdot\left(\dfrac{a}{b}+\dfrac{c}{d}\right) \le 4$
Answer: Let $x=a/b$, $y=c/d$, you'll get $2+x/y+y/x$.  Now use that for any positive number the sum of that number and its reciprocal is at least $2$.


Aplicativo android gerando a exception: androidx.fragment.app.Fragment$InstantiationException

Question: Meu aplicativo está em produção e eu estou com um problema em uma pequena parte dos dispositivos, ainda não consegui identificar o motivo ou ao menos reproduzir o problema em um ambiente de teste, realizei algumas pesquisas buscando mais informações, mas sem sucesso, gostaria que a comunidade me direcionasse para possíveis situações que podem está gerando esse problema.
O meu aplicativo é simples, é um aplicativo de notícias e quando possui noticias novas o aplicativo notifica os usuários.
Os dispositivos que apresentaram o erro:

Samsung Galaxy A20 (a20), 2816MB RAM, Android 9
Samsung Galaxy J4 (j4lte), 2048MB RAM, Android 9
Motorola Moto G (5th Gen) (cedric), 2048MB RAM, Android 8.1
Samsung Galaxy J4 Core (j4corelte), 1024MB RAM, Android 8.1

Como eu não sei quais são as partes do código que estão relacionadas o problema, não coloquei nenhuma uma classe especifica, porém se alguém quiser é só solicitar nos comentários, por exemplo eu acredito que esteja relacionado a notificação.
O erro foi capturado direto da google play console.

java.lang.RuntimeException:
at android.app.ActivityThread.performLaunchActivity
  (ActivityThread.java:3121)   at
  android.app.ActivityThread.handleLaunchActivity
  (ActivityThread.java:3264)   at
  android.app.servertransaction.LaunchActivityItem.execute
  (LaunchActivityItem.java:78)   at
  android.app.servertransaction.TransactionExecutor.executeCallbacks
  (TransactionExecutor.java:108)   at
  android.app.servertransaction.TransactionExecutor.execute
  (TransactionExecutor.java:68)   at
  android.app.ActivityThread$H.handleMessage (ActivityThread.java:1955)
  at android.os.Handler.dispatchMessage (Handler.java:106)   at
  android.os.Looper.loop (Looper.java:214)   at
  android.app.ActivityThread.main (ActivityThread.java:7078)   at
  java.lang.reflect.Method.invoke (Native Method)   at
  com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run
  (RuntimeInit.java:494)   at com.android.internal.os.ZygoteInit.main
  (ZygoteInit.java:965)
Caused by: androidx.fragment.app.Fragment$InstantiationException:
at androidx.fragment.app.Fragment.instantiate (Fragment.java:538)
  at androidx.fragment.app.FragmentContainer.instantiate
  (FragmentContainer.java:57)   at
  androidx.fragment.app.FragmentManagerImpl$6.instantiate
  (FragmentManagerImpl.java:2851)   at
  androidx.fragment.app.FragmentState.instantiate
  (FragmentState.java:85)   at
  androidx.fragment.app.FragmentManagerImpl.restoreSaveState
  (FragmentManagerImpl.java:2488)   at
  androidx.fragment.app.FragmentController.restoreSaveState
  (FragmentController.java:195)   at
  androidx.fragment.app.FragmentActivity.onCreate
  (FragmentActivity.java:288)   at
  androidx.appcompat.app.AppCompatActivity.onCreate
  (AppCompatActivity.java:106)   at xxx.xxxx.xxxxx.MainActivity.onCreate
  (MainActivity.kt:70)   at android.app.Activity.performCreate
  (Activity.java:7339)   at android.app.Activity.performCreate
  (Activity.java:7330)   at
  android.app.Instrumentation.callActivityOnCreate
  (Instrumentation.java:1275)   at
  android.app.ActivityThread.performLaunchActivity
  (ActivityThread.java:3101)
Caused by: java.lang.NoSuchMethodException:
at java.lang.Class.getConstructor0 (Class.java:2328)   at
  java.lang.Class.getConstructor (Class.java:1725)   at
  androidx.fragment.app.Fragment.instantiate (Fragment.java:523)

Parte da classe para onde o problema aponta
MainActivity.kt
override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState) // Linha 70
        setContentView(R.layout.activity_main)

        ...
}

Outras classes que podem está relacionadas com o problema
PagerAdapter.kt
class PagerAdapter(val fragmentActivity: FragmentActivity, private var lista: MutableList<Blog>) : FragmentStateAdapter(fragmentActivity) {

    override fun getItemCount(): Int {
        return lista.size+1
    }

    override fun createFragment(position: Int): Fragment {

        return if(position<lista.size)
            Tab(lista[position])
        else
            OutraTab() // tab fixa no final

    }

}

Tab.kt
 class Tab(private val blog: Blog) : Fragment() {

        ...

        override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View? {
            val view = inflater.inflate(R.layout.tab, container, false)

        ...

        }

}

OutraTab.kt
   class OutraTab : Fragment() {

        override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?, savedInstanceState: Bundle?): View? {
            val view = inflater.inflate(R.layout.tab_outra, container, false)

        ...

        }

    }

Obs. OutraTab.kt é semelhante ao Tab.kt, só que sem o parâmetro blog ela contem conteúdo fixo.
Comment: @LSA " Essa depreciação pode ocasionar o erro relatado?". Não use métodos depreciados. O padrão é um só Android, mas cada fabricante implementa os endpoints da API segundo seu hardware e talvez, pode ser por um outro motivo qualquer, o fabricante para economizar, memória,  mão de obra ou tempo desenvolvimento com material obsoleto decida não implementar os métodos depreciados em determinadas versões do OS e por isso acontece do código funcionar em algumas marcas/modelos e em outros marcas/modelos não funcionar.
Comment: Sua pergunta está muito ampla, você poderia discutir algo assim no meta (se não estou enganado). Aqui as perguntas precisam ser objetivas, descrevendo um problema de forma que seja respondido diretamente e não de forma argumentativa.
Comment: Entendo, sei que as informações que eu tenho são insuficientes para resolver, é essa talvez a dificuldade para resolver esse problema, eu ainda não consegui reproduzi esse erro em um ambiente de teste, eu ainda não sei qual o fluxo que gera esse erro, porém acredito que alguém aqui na comunidade pode identificar algum erro de fluxo ou de lógica.. hoje eu vou  realizar novos testes e depois vou postar alguns trechos do código
Comment: Tudo bem, esperamos detalhes para te ajudar e também, deixar respostas úteis para pessoas que futuramente tenham o mesmo problema.
Comment: O método [`androidx.fragment.app.Fragment.instantiate`](https://developer.android.com/reference/androidx/fragment/app/Fragment#instantiate(android.content.Context,%20java.lang.String,%20android.os.Bundle)) foi depreciado.
Comment: @AugustoVasques é verdade estava utilizando no meu PagerAdapter o FragmentStatePagerAdapter que está depreciado, mudei para o FragmentStateAdapter, mudei também o ViewPager para ViewPager2 que estava também depreciado,  o Android Studio não me alertou sobre isso, como eu não conseguir reproduzir o erro no cenário de testes, eu mudei também uma parte do fluxo do meu app, pra ver se o erro vai persistir e agora estou colocando em produção e vou observar se o erro vai continuar. Essa depreciação pode ocasionar o erro relatado?
Comment: Acho que avançamos com relação a esse problema, a dúvida também era que o problema apontava para a linha 70 do MainActivity, a linha 70 é  super.onCreate(savedInstanceState), só que até então não tinha sentido, porém agora percebemos que tinha um problema na interface (ViewPager depreciado), geralmente quando se tem algum problema na interface essa é a linha que é apontada pelo erro, alguém me corrija se eu estiver equivocado..
Comment: Ok, o app já está produção, vou aguardar até os próximos dias para ver se o problema foi resolvido, se foi eu vou postar aqui o que foi feito exatamente no código para que possa ajudar outras pessoas com o mesmo problema.
Comment: As referencias da classe R as vezes se perdem e ocasiona em problema do RunTime. Mas vendo o erro, parece estar relacionado ao dado de acesso que esta a instanciar e não esta conseguindo exibir na ActivityThread. Tentou um novo build da aplicação?
O aplicativo esta dando erro apenas nesses modelos citados?
Usa API de acesso para exibir a noticias?
Comment: Eu pensei que substituindo os componentes depreciados o problema estaria resolvido, porém o problema continua, nessa nova versão lançada teve um erro igual no dispositivo Motorola Moto G (5S) (montana) android 8.1, porém eu estou tendo outra dúvida e não sei os efeitos colateral desse tipo de implementação, eu vou ter que abrir outra pergunta para tirar essa dúvida e depois voltar aqui, se não vai fugir do tema. Eu observei também que outro aplicativo que eu gerencio possui o mesmo erro.
Comment: @Nellonidas eu fiz um novo build, mas o erro continua, esse erro ele acontece de vez em quando ainda não conseguir reproduzir em cenário de teste, eu só consigo ver que ainda não foi sanado com os relatórios de falhas do google, provavelmente ocorre em outros dispositivos também, as notícias são carregadas direto dos sites através da biblioteca Jsoup
Comment: Imagino que ainda não tenha resolvido o problema. Todos seus fragmentos possuem um construtor com 0 argumentos (construtor padrão)? Se não, o problema pode ser esse (e então posso elaborar uma resposta sobre)
Comment: @RafaelTavares, ainda não, mas estou buscando corrigir, só o Fragmento OutraTab que possui 0 argumentos no construtor padrão, eu atualizei a resposta e coloquei as classes
Answer: Resumo
Seus fragmentos precisam de um construtor padrão (sem argumentos) para que o Android consiga recriá-los quando necessário (quando mudar a orientação do celular, por exemplo).
Explicação
Quando ocorre alguma mudança de configuração no Android (como mudança na orientação de retrato pra paisagem, ou mudança de idioma), a sua Activity é recriada.
No processo de recriação, os Fragments que serão recriados terão uma chamada Construtor(), ou seja, o construtor sem argumentos. O fato de não ter esse construtor gera o seguinte erro (que você indicou):
Caused by: java.lang.NoSuchMethodException:
at java.lang.Class.getConstructor0 (Class.java:2328) at java.lang.Class.getConstructor

De acordo com a documentação sobre alterações de configuração do Android:

Algumas configurações de dispositivo podem mudar durante o tempo de execução, como orientação da tela, disponibilidade do teclado e ativação do modo de várias janelas. Quando ocorre uma alteração, o Android precisa reiniciar a execução de Activity (onDestroy() é chamado, seguido de onCreate()).
O comportamento de reinício foi projetado para ajudar o aplicativo a se adaptar a novas configurações atualizando automaticamente o aplicativo com recursos alternativos que correspondam à configuração do dispositivo.

Solução com o Padrão de Nova Instância (NewInstance)
Esse padrão permite que você passe argumentos para um fragmento sem precisar criar um construtor que não seja o padrão. O padrão existe tanto em Java quanto em Kotlin, mas abaixo explico o processo no Kotlin. Na documentação de Fragments esse padrão é utilizado.

Crie um método novaInstancia para instanciar seu Fragment;
Receba os argumentos necessários nesse método e passe para o onCreate através de arguments;
No método onCreate, armazene os valores como atributos da sua class Fragment.

Código do fragmento:
private const val ARG_MEU_ATRIBUTO = "meuAtributoId"

class MeuFragmento: Fragment() {
    private var meuAtributo: String? = null

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        arguments?.let {
            meuAtributo = it.getString(ARG_MEU_ATRIBUTO)
        }
    }

    companion object {
        fun novaInstancia(valorDoAtributo: String?) =
                MeuFragmento().apply {
                    arguments = Bundle().apply {
                        putString(ARG_MEU_ATRIBUTO, valorDoAtributo)
                    }
                }
    }
}

Para instânciar seu fragmento:
val meuFragmento = MeuFragmento.novaInstancia("texto")

NewInstance com o Android Studio
Quando você for criar um fragmento novo, deixe a opção "Include fragment factory methods?" marcada e o código será gerado com o padrão descrito acima.

Código gerado:
// TODO: Rename parameter arguments, choose names that match
// the fragment initialization parameters, e.g. ARG_ITEM_NUMBER
private const val ARG_PARAM1 = "param1"
private const val ARG_PARAM2 = "param2"

/**
 * A simple [Fragment] subclass.
 * Use the [BlankFragment.newInstance] factory method to
 * create an instance of this fragment.
 */
class BlankFragment : Fragment() {
    // TODO: Rename and change types of parameters
    private var param1: String? = null
    private var param2: String? = null

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        arguments?.let {
            param1 = it.getString(ARG_PARAM1)
            param2 = it.getString(ARG_PARAM2)
        }
    }

    override fun onCreateView(inflater: LayoutInflater, container: ViewGroup?,
                              savedInstanceState: Bundle?): View? {
        // Inflate the layout for this fragment
        return inflater.inflate(R.layout.fragment_blank, container, false)
    }

    companion object {
        /**
         * Use this factory method to create a new instance of
         * this fragment using the provided parameters.
         *
         * @param param1 Parameter 1.
         * @param param2 Parameter 2.
         * @return A new instance of fragment BlankFragment.
         */
        // TODO: Rename and change types and number of parameters
        @JvmStatic
        fun newInstance(param1: String, param2: String) =
                BlankFragment().apply {
                    arguments = Bundle().apply {
                        putString(ARG_PARAM1, param1)
                        putString(ARG_PARAM2, param2)
                    }
                }
    }
}

Comment: Entendi, vou realizar as modificações no código e assim que eu verificar que o problema foi corrigido eu aceito a resposta como correta.
Comment: @LSA A resposta acima resolveu o seu problema? Precisa de ajuda com algo mais?
Comment: Aparentemente sim, porém preciso de mais uns 2 a três 3 dias para que a maior parte do usuários do aplicativo já esteja utilizando essa última versão, agora me responde, se possível, o fragmento precisa de um construtor vazio para recriado nesses cenários, porém o contexto fornece os argumentos que a situação pede automaticamente ?  ou se precisa-se recriar, o parâmetro  ARG_PARAM1 seria "param1", ou como no teu exemplo ARG_MEU_ATRIBUTO seria "meuAtributoId" ? Eu precisaria criar um valor "default" no lugar do "meuAtributoId" para quando isso ocorresse ?
Comment: Você não precisa indicar um valor padrão. Mesmo após o fragmento ser recriado, o `arguments` continua passando o valor do `novaInstancia` no `onCreate`. Você pode testar isso colocando um `Log.d()` para checar os valores passados
Comment: Perfeito, parece que o problema foi resolvido, cerca de 95% dos usuários estão utilizando o app agora e não tive mais relatos do google play com relação a esse problema.


Field groups not showing when nested using field type HTML

Question: I'm using Drupal 7.34, Profile2 module, Field groups 7.x-1.4.
When building a form with nested field groups and using the fieldset field type, my form renders properly.
When I configure my form like this:

It renders into this, showing nothing but the Title and Save button:

There is nothing in the generated HTML as well, i.e. not hidden div's or something. The HTML markup just isn't there.
When I configure my form like this (changed nested fieldgroup element from HTML to Fieldset):

It renders into this, showing all:

My goal is to have some nested HTML elements, which I am able to style in a theme. Am I doing something wrong with nested HTML elements?
Thanks,
Eric
Comment: This question appears to be off-topic because it is about [a bug found in a module](https://www.drupal.org/node/2311789) - these are not allowed here, you should post them in issue queue. There you would get pointed to existing bugreport and provided patch. Patch seems to be already included in dev version, too.
Comment: Ah thanks. I did not find this bug report (and believe me: I tried) so I posted it here.
Answer: the solution and patch can be found here:
https://www.drupal.org/node/1503094#comment-11981615


Expand a div's width untill all the children fit

Question: I'm trying to make a simple slider so I'm using a div (#container) to hold the items that I need to slide (this div will have overflow:hidden), a div (#slider) that I want to move so that all the items will be moved together, and divs that fit in the slider.
I want the slider's width to expand until all the items fit to the same row with float:left.
I have this jsFiddle : http://jsfiddle.net/rBFPL/5/ and I would need some help to make the red divs on the same row and the cyan div expand its width.
Can you help me with this?
Answer: Update: http://jsfiddle.net/HCYPD/3/
<code>#container {
    width : 400px;
    height : 200px;
    position : relative;
    background : yellow;
    overflow: hidden;
}

#slider {
    margin : 5px;
    position : absolute;
    background : cyan;
    width: auto;
    height : 190px;
    white-space: nowrap;
}

#slider div {
    margin : 20px;
    width: 100px;
    height: 150px;
    position : relative;
    display: inline-block;
    background : red;
}
</code>
Comment: That's almost what I need, But isn't there a way to do it using only css?
Comment: I'll have to add dynamically loaded items there, and their width will not be the same..
Comment: Posted an update. Haven't tested across browsers, but I believe it should work.
Comment: Updated again. Does this work for you? Tested on Firefox fine here.
Answer: I've made this jsFiddle , I think it does what you say, I've only tested on chrome.
Update: Second attempt
<code>#container {
    background : yellow;
    padding: 5px;
    white-space: nowrap;
    overflow: hidden;
    position : fixed;
    width : 400px;
}
#slider {
    background : cyan;
    position : fixed;
}
#slider div {
    margin : 20px;
    width: 100px;
    height: 150px;
    position : relative;
    float : left;
    background : red;
}
</code>
Comment: No... I need the yellow div to stay with fixed width and the cyan one to be wider than the yellow one


Como editar dados de um Struct utilizando uma função

Question: Eu quero criar uma função editar que receba como parâmetro por referência o vetor de músicas. (usando ponteiros)
O usuário deve escolher o número da música e digitar novamente os dados daquela posição do vetor.
Eu criei o struct, já estou recebendo os valores e estou tocando. Mas não sei como editar os valores. Alguma pessoa para me ajudar a iniciar essa parte?
    #include <stdio.h>
    #include <stdlib.h> //
    #include <stdbool.h>
    #include <locale.h>
    #include <string.h>

      struct registry_of_music  {
        char name[50];
        char artist[60];
        char url[80];
    };
    struct registry_of_music music[9];

    int main() {
    int i;
    printf("\nRegistry of Music\n\n\n");

        for(i = 0; i <= 3;i++ ){
              printf("Name of Music: ");
              fflush(stdin);
              fgets(music[i].name, 50, stdin);

              printf("Name of Artist: ");
              fflush(stdin);
              fgets(music[i].artist, 60, stdin);

              printf("URL of Internet: ");
              fflush(stdin);
              fgets(music[i].url, 80, stdin);
        }

        int op;
        do
        {
            printf("1 - Play\n");
            printf("2 - Edit\n");
            printf("3 - Exit\n");
            printf("Please enter a value:");
            scanf("%d", &op);
                    switch(op) {
                case 1: play();
                        break;
                case 2: edit();
                        break;
                case 3: printf("Bye\n");
                        break;
                default: printf("Try Again\n");
            }
        } while (op!=3);

      getch();
      return(0);
}
    void play(){
    int i;
                    for(i = 0; i <= 3;i++ ){
                      printf("Name ...........: %s", music[i].name);
                      printf("Artist .....: %s", music[i].artist);
                      printf("URL .....: %s", music[i].url);
                    }
}

    void edit(){}

Answer: Algumas observações antes de começar:

Não coloque as funções depois do main sem antes dele ter as definições/assinaturas
Se a sua estrutura já é global, pois foi declarada acima do main, então não precisa de a passar a nenhuma função, pois consegue aceder diretamente. Se olhar com atenção é o que acontece na função play.
Não inclua mais headers que os estritamente necessários.
Evite usar funções dependentes de sistema operativo como o getch

Posto isto, o primeiro passo é alterar o vetor para que não seja global:
int main() {
    struct registry_of_music music[9];
    ...

Que agora lhe dará erro de compilação na função play. Para corrigir, basta alterar a função para receber o vetor tal como pretende fazer na função editar:
void play(struct registry_of_music *music /*<--aqui*/) {
    //código da função
}

E agora quando chama a função, passa o endereço do vetor que é apenas music:
int main() {
    ...
        switch(op) {
        case 1:
            play(music);
//                  ^---
            break;

Agora para o editar é idêntico, mas passando a posição da musica a alterar:
case 2:
    printf("Please enter the position of the music to edit:");
    int edit_index;
    scanf("%d", &edit_index);
    edit(music, edit_index);
    //              ^----
    break;

E a função ficaria assim:
void edit(struct registry_of_music *music, int index) {
    printf("Editing music %s", music[index].name);
    printf("New Name: ");
    fflush(stdin);
    fgets(music[index].name, 50, stdin);

    printf("New Artist: ");
    fflush(stdin);
    fgets(music[index].artist, 60, stdin);

    printf("New URL: ");
    fflush(stdin);
    fgets(music[index].url, 80, stdin);
}

Como nota final, as leituras que está a fazer com fgets deixam o \n dentro das strings, que é o motivo pelo qual não tem de o fazer manualmente. Já respondi sobre como contornar isso noutras perguntas, tal como nesta.
Código com todas as alterações para referência:
#include <stdio.h>

struct registry_of_music  {
    char name[50];
    char artist[60];
    char url[80];
};

void play(struct registry_of_music *music) {
    int i;
    for(i = 0; i <= 3; i++ ) {
        printf("Name ...........: %s", music[i].name);
        printf("Artist .....: %s", music[i].artist);
        printf("URL .....: %s", music[i].url);
    }
}

void edit(struct registry_of_music *music, int index) {
    printf("Editing music %s", music[index].name);
    printf("New Name: ");
    fflush(stdin);
    fgets(music[index].name, 50, stdin);

    printf("New Artist: ");
    fflush(stdin);
    fgets(music[index].artist, 60, stdin);

    printf("New URL: ");
    fflush(stdin);
    fgets(music[index].url, 80, stdin);
}

int main() {
    struct registry_of_music music[9];
    int i;
    printf("\nRegistry of Music\n\n\n");

    for(i = 0; i <= 3; i++ ) {
        printf("Name of Music: ");
        fflush(stdin);
        fgets(music[i].name, 50, stdin);

        printf("Name of Artist: ");
        fflush(stdin);
        fgets(music[i].artist, 60, stdin);

        printf("URL of Internet: ");
        fflush(stdin);
        fgets(music[i].url, 80, stdin);
    }

    int op;
    do {
        printf("1 - Play\n");
        printf("2 - Edit\n");
        printf("3 - Exit\n");
        printf("Please enter a value:");
        scanf("%d", &op);
        switch(op) {
        case 1:
            play(music);
            break;
        case 2:
            printf("Please enter the position of the music to edit:");
            int edit_index;
            scanf("%d", &edit_index);
            edit(music, edit_index);
            break;
        case 3:
            printf("Bye\n");
            break;
        default:
            printf("Try Again\n");
        }
    } while (op!=3);

    return(0);
}

Comment: Sua explicação foi perfeita. Entendi todos os passos de forma sucinta. Muito obrigado por esse seu tempo e por sua explicação, me ajudou demais.
Comment: @spw Sem problema. Estamos cá para ajudar :)


Correcting for multiple post hoc tests in factorial ANOVA

Question: I have done a two-way repeated-measures ANOVA with factor A having 3 levels and factor B having 5 levels.  I am doing post hoc comparisons between all 3 levels of factor A within each of the 5 levels of factor B, resulting in 15 comparisons. I am not explicitly doing any post hoc comparisons between the levels of factor B.
Assuming I use Bonferroni to correct for alpha inflation (disregarding whether or not it is the best correction method) would I need to correct by the number of tests I am actually doing (15), or the number of all possible comparisons, i.e. 45, including the 30 comparisons between the levels of factor B within each level of factor A? I have seen different opinions on which of these comparisons should be regarded as a family.
Thanks in advance!
Answer: no, you only correct for the ones you actually test. Basically, the correction is only to punish "hypothesis fishing". If you would correct for all possible ones, data sets with many variables or groups would always deal with underpowerdness, because their alpha would be lower just by having a large range of possible comparisons.
Comment: Thanks for your response! That I understand. However, as I understand it, post hoc tests are usually corrected exactly because they are somewhat a fishing expedition, as they are not hypothesis-guided, like planned comparisons. So there's, by definition, no specific reason to carry out only a subset of all possible comparisons, so that we should act as if we would do all of them (even if we don't di them explicitly). Where do I err?


Comparing two dates by week in a .NET entity framework LINQ query?

Question: Is there an easy way of comparing weeks in two dates like this...
var times = (from d in db.TimeSet
             where d.EventDate.Week) == DateTime.ThisWeek &&
             d.Employee.Username == username
             select d).ToList();

Comment: 6 questions, 0 accepted answer. I'm sure that you *WON'T* get an answer.
Comment: Actually, if you check the profile now at least, that's 4 accepted answers. Not so bad.
Comment: does that mean you don't know or just bored?
Comment: that means you have no respect for the time & knowledge that other people care to share with you.
Comment: so ticking a green icon gives respect? hmm, never realised that. Thanks to people like you scouring this site for non respectful users, I have now realised my terrible ways and endeavour to correct them ;)
Answer: public static int GetWeekNumber(DateTime date)
{
        CultureInfo ci = CultureInfo.CurrentCulture;
        int weekNum = ci.Calendar.GetWeekOfYear(date, CalendarWeekRule.FirstFourDayWeek, DayOfWeek.Monday);
        return weekNum;
}

and
var times = (from d in db.TimeSet
             where GetWeekNumber(d.EventDate) == GetWeekNumber(DateTime.Now) &&
             d.Employee.Username == username
             select d).ToList();

You might want to look into CalendarWeekRule to pick the one that fits your needs.
Comment: LINQ to Entities does not recognize the method 'Int32 GetWeekNumber(System.DateTime)' method, and this method cannot be translated into a store expression.
Comment: The only thing I had to edit was DateTime.Now instead of DateTime.ThisWeek. Otherwise, it seems to compile for me.


Updating the lasso fit with new observations

Question: I am fitting an L1-regularized linear regression to a very large dataset (with n>>p.) The variables are known in advance, but the observations arrive in small chunks. I would like to maintain the lasso fit after each chunk.
I can obviously re-fit the entire model after seeing each new set of observations. This, however, would be pretty inefficient given that there is a lot of data. The amount of new data that arrives at each step is very small, and the fit is unlikely to change much between steps.
Is there anything I can do to reduce the overall computational burden?
I was looking at the LARS algorithm of Efron et al., but would be happy to consider any other fitting method if it can be made to "warm-start" in the way described above.
Notes:

I am mainly looking for an algorithm, but pointers to existing software packages that can do this may also prove insightful.
In addition to the current lasso trajectories, the algorithm is of course welcome to keep other state.

Bradley Efron, Trevor Hastie, Iain
  Johnstone and Robert Tibshirani,
  Least Angle Regression, Annals
  of Statistics (with discussion)
  (2004) 32(2), 407--499.
Answer: The lasso is fitted through LARS (an iterative process, that starts at some initial estimate $\beta^0$). By default $\beta^0=0_p$ but you can change this in most implementation (and replace it by the optimal $\beta^*_{old}$ you already have). The closest $\beta^*_{old}$ is to $\beta_{new}^*$, the smaller the number of LARS iteration you will have to step to get to $\beta_{new}^*$.
EDIT:
Due to the comments from  user2763361 I add more details to my original answer.
From the comments below I gather that user2763361 suggests to complement my original answer to turn it into one that can be used directly (off the shelves) while also being very efficient.
To do the first part, I will illustrate the solution I propose step by step on a toy example. To satisfy the second part, I will do so using a recent, high quality interior point solver.
This is because, it is easier to obtain an high performance implementation of the solution I propose using a library that can solve the lasso problem by the interior point approach rather than trying to hack the LARS or simplex algorithm  to start the optimization from a non-standard starting point (though that second venue is also possible).
Note that it is sometimes claimed (in older books) that the interior point approach to solving linear programs is slower than the simplex approach and that may have been true a long time ago but it's generally not true today and certainly not true for large scale problems (this is why most professional libraries like cplex use the interior point algorithm) and the question is at least implicitly about large scale problems.
Note also that the interior point solver I use fully handles sparse matrices so I don t think there will be a large performance gap with LARS (an original motivation for using LARS was that many popular LP solvers at the time were not handling sparse matrices well and these are a characteristic features of the LASSO problem).
A (very) good open source implementation of the interior point algorithm is ipopt, in the COIN-OR library. Another reason I will be using ipopt is that it has has an R interface, ipoptr. You will find more exhaustive installation guide here, below I give the standard commands to install it in ubuntu.
in the bash, do:
sudo apt-get install gcc g++ gfortran subversion patch wget
svn co https://projects.coin-or.org/svn/Ipopt/stable/3.11 CoinIpopt
cd ~/CoinIpopt
./configure
make
make install

Then, as root, in R do (I assume svn has copied the subversion file in ~/ as it does by default):
install.packages("~/CoinIpopt/Ipopt/contrib/RInterface",repos=NULL,type="source")

From here, I'm giving a small example (mostly from the toy example given by Jelmer Ypma as part of his  R wraper to ipopt):
library('ipoptr')
# Experiment parameters.
lambda <- 1                                # Level of L1 regularization.
n      <- 100                              # Number of training examples.
e      <- 1                                # Std. dev. in noise of outputs.
beta   <- c( 0, 0, 2, -4, 0, 0, -1, 3 )    # "True" regression coefficients.
# Set the random number generator seed.
ranseed <- 7
set.seed( ranseed )
# CREATE DATA SET.
# Generate the input vectors from the standard normal, and generate the
# responses from the regression with some additional noise. The variable
# "beta" is the set of true regression coefficients.
m     <- length(beta)                           # Number of features.
A     <- matrix( rnorm(n*m), nrow=n, ncol=m )   # The n x m matrix of examples.
noise <- rnorm(n, sd=e)                         # Noise in outputs.
y     <- A %*% beta + noise                     # The outputs.
# DEFINE LASSO FUNCTIONS
# m, lambda, y, A are all defined in the ipoptr_environment
eval_f <- function(x) {
    # separate x in two parts
    w <- x[  1:m ]          # parameters
    u <- x[ (m+1):(2*m) ]

    return( sum( (y - A %*% w)^2 )/2 + lambda*sum(u) )
}
# ------------------------------------------------------------------
eval_grad_f <- function(x) {
    w <- x[ 1:m ]
    return( c( -t(A) %*% (y - A %*% w),
               rep(lambda,m) ) )
}
# ------------------------------------------------------------------
eval_g <- function(x) {
    # separate x in two parts
    w <- x[  1:m ]          # parameters
    u <- x[ (m+1):(2*m) ]
    return( c( w + u, u - w ) )
}
eval_jac_g <- function(x) {
    # return a vector of 1 and minus 1, since those are the values of the non-zero elements
    return( c( rep( 1, 2*m ), rep( c(-1,1), m ) ) )
}
# ------------------------------------------------------------------
# rename lambda so it doesn't cause confusion with lambda in auxdata
eval_h <- function( x, obj_factor, hessian_lambda ) {
    H <- t(A) %*% A
    H <- unlist( lapply( 1:m, function(i) { H[i,1:i] } ) )
    return( obj_factor * H )
}
eval_h_structure <- c( lapply( 1:m, function(x) { return( c(1:x) ) } ),
                       lapply( 1:m, function(x) { return( c() ) } ) )
# The starting point.
x0 = c( rep(0, m),
        rep(1, m) )
# The constraint functions are bounded from below by zero.
constraint_lb = rep(   0, 2*m )
constraint_ub = rep( Inf, 2*m )
ipoptr_opts <- list( "jac_d_constant"   = 'yes',
                     "hessian_constant" = 'yes',
                     "mu_strategy"      = 'adaptive',
                     "max_iter"         = 100,
                     "tol"              = 1e-8 )
# Set up the auxiliary data.
auxdata <- new.env()
auxdata$m <- m
    auxdata$A <- A
auxdata$y <- y
    auxdata$lambda <- lambda
# COMPUTE SOLUTION WITH IPOPT.
# Compute the L1-regularized maximum likelihood estimator.
print( ipoptr( x0=x0,
               eval_f=eval_f,
               eval_grad_f=eval_grad_f,
               eval_g=eval_g,
               eval_jac_g=eval_jac_g,
               eval_jac_g_structure=eval_jac_g_structure,
               constraint_lb=constraint_lb,
               constraint_ub=constraint_ub,
               eval_h=eval_h,
               eval_h_structure=eval_h_structure,
               opts=ipoptr_opts,
               ipoptr_environment=auxdata ) )

My point is, if you have new data in, you just need to

update (not replace) the constraint matrix and objective function vector to account for the new observations.
change the starting points of the interior point from
x0 = c( rep(0, m),
        rep(1, m) )
to the vector of solution you found previously (before new data was added in). The logic here is as follows. Denote $\beta_{new}$ the new vector of coefficients (the ones corresponding to the data set after the update) and $\beta_{old}$ the original ones.  Also denote $\beta_{init}$ the vector x0 in the code above (this is the usual start for the interior point method). Then the idea is that if:

$$|\beta_{init}-\beta_{new}|_1>|\beta_{new}-\beta_{old}|_1\quad(1)$$
then, one can get $\beta_{new}$ much faster by starting the interior point from
$\beta_{old}$ rather than the naive $\beta_{init}$. The gain will be all the more important when the dimensions of the data set ($n$ and $p$) are larger.
As for the conditions under which inequality (1) holds, they are:

when $\lambda$ is large compared to $|\beta_{OLS}|_1$ (this is usually the case when $p$, the number of design variables is large compared to $n$, the number of observations)
when the new observations are not pathologically influential, e.g. for example when they are consistent with the stochastic process that has generated the existing data.
when the size of the update is small relative to the size of the existing data.
Comment: Any libraries you know that can do this without needing to edit the source code?
Comment: Thanks, but I am afraid I don't follow. LARS produces a piecewise-linear path (with exactly $p+1$ points for the least angles and possibly more points for the lasso.) Each point has its own set of $\beta$. When we add more observations, all the betas can move (except $\beta^0$, which is always $0_p$.) Please could you expand on your answer? Thanks.
Comment: @aix: do you want to update the whole lasso path or just the solution ? (i.e. is the sparsity penalty fixed?).
Comment: I was looking to update the entire path. However, if there's a good way to do it for a fixed penalty ($\lambda$ in the formula below), this may be a good start. Is this what you are proposing?

$$ \hat{\beta}^{lasso} = \underset{\beta}{\operatorname{argmin}} \left \{ {1 \over 2} \sum_{i=1}^N(y_i-\beta_0-\sum_{j=1}^p x_{ij} \beta_j)^2 + \lambda \sum_{j=1}^p |\beta_j| \right \} $$
Comment: @aix. Yes, it all depends on the implementation you use and the facilities you have access to. For example: if you have access to a good lp solver, you can feed it with the past optimal values of $\beta$ and it'll carry the 1-2 step to the new solution very efficiently. You should add these details to your question.
Comment: This method doesn't seem correct.
Yes, it will start with last estimates of $\beta$'s, but it will then find new $\beta$'s that are optimal _only_ for few newly added datapoints, not for the whole sample observed so far. So if you have added 10 new points then this will give you $\beta$'s optimal for these particular 10 points which may be very different from the whole sample $\beta$'s.
Comment: @Kochede: no because *all* the points are used in the new optimization. The idea is that you have to visit less edges of the simplex before finding the good one because you already start from one close to the good one.
Comment: @user2763361: I recommend COIN-OR but any interior point solver should be able to do it. I think I understand what you mean now. I will update my answer during the day.


Get the current ProductVariation from the formElement of a custom widget in the "Add to cart" form

Question: I've a custom field in my Order Item, i need the value of a field of my ProductVariation in the widget of my custom field.
Let's say that my Variation has a "remote ID" field, when i select one of the Variation in the Add to cart form, in my custom field i've to retrieve that ID and use it to do some things.
If i've only one Variation everything works great doing something like that:
<code>  public function formElement(FieldItemListInterface $items, $delta, array $element, array &$form, FormStateInterface $form_state) {
    $purchased_entity = $items->getEntity()->getPurchasedEntity();

    $my_cusrom_id = $purchased_entity->get('my_variation_field')
      ->first()
      ->getValue()['value'];
  }
</code>
The only problem is that if I have multiple variations, when i chose one from the select, in this code always the first one is returned from the getPurchasedEntity method, and i see no informations in $form_state or $form of something that could be useful.
Is there a way to retrieve the selected variation from there? seems strange that's not possible...
Answer: At the moment i ended by getting the value directly from the $_POST, sounds a bit hacky but can't find another way to do that...


Move maven module between parent projects in eclipse

Question: I'm having a maven multi module structure as shown below
ParentApp1
   - childApp1
   - childApp2
   - childApp3
TestParent
   - tchild1
   - tchild2
   - tchild3

After importing maven projects to eclipse it imports properly and individual projects are created for child projects. I want to move tchild1 module under ParentApp1 parent in the above strucute. After drag and drop the dependencies are not resolved. Even i tried with update Maven project in eclipse. Can anybody tell me is it possible to move modules between projects by drag and drop ? If yes, please provide me an any suggestion or reference else please tell me the steps to move the modules between maven projects
Comment: Im not sure about Drag and drop .so childApp1 need tchild1 dependency right? or you want to move tchild1 inside ParentApp1?
Comment: Did the parent POM reference change when you moved the child from one parent to the other?
Comment: do you have Pom.xml in ParentApp1 right just add the module under  ../tchild1 tchild1 module should be same parentapp1 location
Answer: You need to add the project on the pom.xml of the parent(ParentApp1) And also change the parent on the tchild1 pom.xml
ParentApp1 pom.xml
    <modules>
<module>tchild1</module>
</modules>
TChild1 pom.xml
    <parent>
<artifactId>ParentApp1</artifactId>
<groupId>...</groupId>
<version>...</version>
</parent>
Comment: I tried & when i am trying to build, it throws Child module C:\Users\xxxx\test_workspace\Parentapp1/tchild1 of C:\Users\xxxxtest_workspace\Parentapp1\pom.xml does not exist. In Actual project structure parent modules contains modules. But after changing in pom the parentapp1 doesn't contain tchild1 module.
Comment: Also you need to copy the tchild1 module inside the Parentapp1. This is the ParentApp1:
   - childApp1
   - childApp2
   - childApp3t
   - tchild1. And run maven update proyect


What does "min" mean?

Question: I was doing a problem based on constructing an interval centered on $c$ which is a subset of a separate interval $(a,b)$ , where $c \in (a,b)$ and $a$ and $b$ are real numbers.
The issue with trying to create an interval like $I = (c-(c-a), c+(c-a))$ is that $c$ may be closer to $b$ than to $a$, and thus adding the difference between $c$ and $a$ would exceed $b$, and $I$ wouldn't be a subset of $(a,b)$.
I was looking at the solution manual and it said: Let $r=\min (c-a, b-c)$, and let $I = (c-r, c+r)$.
From this it seems that $\min$ means select $r$ to be the smaller of the two numbers in the parentheses? I've never seen this operation before and it's not in the textbook. Is it just shorthand for :

Let $r$ be an element of the real numbers : if $(c-a)>(b-c)$ then $r = b-c$ . If $(c-a)<(b-c)$ then $r=c-a$. If $(c-a)=(b-c)$ then $r=c-a=b-c$ ?
Comment: Min means Minimum.
So yes, it's a function that, taken two elements, gives you the minimum of those.
Comment: This is the minimum function, and just takes the value of the smallest value in the given set: $\min(2,5)=2$, $\min(-1,-8,3)=-8$
Answer: Yes, minimum of the two numbers in the parentheses.
Answer: Note that the minimum is not always well-defined. For example, given the interval $(0,1)$, there is no smallest element in the interval, since we may find numbers in the interval arbitrarily close to zero. So the minimum (and maximum) are not always well defined. We may use infimums and supremums to fix this problem.
When learning about the min and max operations, I think it's important to know their limitations.
Answer: "Min" means minimum, so in your case it's saying what's the minimum of the two values in (c-a, b-c).

What does "min" mean?


Shipping method and shipping service from a placed order

Question: I am trying to get the shipping method and the shipping service values after i place an order on a Drupal 8 commerce website so i can use it inside an email im sending the website's owner (not the receipt for the client). I know Swift email would be an option but i have to continue with what i have. So far the owner of the website does receive an email but i need to add more information to it like the shipping method and shipping service.
I've got as far as finding out that $order->shipments->entity  will give me, among other values, these 2 fields shipping_method and shipping_service.
Shipment {#4521 ▼
  #values: array:19 [▼
    "shipment_id" => array:1 [▶]
    "type" => array:1 [▶]
    "uuid" => array:1 [▶]
    "order_id" => array:1 [▶]
    "package_type" => array:1 [▶]
    "shipping_method" => array:1 [▼
      "x-default" => "2"
    ]
    "shipping_service" => array:1 [▼
      "x-default" => "PRIORITY_OVERNIGHT"
    ]

Where do I find the values of the shipping method value whose ID is 2 and the shipping service value whose ID is PRIORITY_OVERNIGHT?
Answer: Since shipping_method is an entity reference field, you can use $order->shipments->entity->shipping_method->entity to get the referenced entity.
$order->shipments->entity->shipping_service is not a entity field, so you get its value with $order->shipments->entity->shipping_service->value.


Associar duas listas em python

Question: Associar duas listas, uma lista com nome de carro e outra lista de cores e cada uma contendo n elementos:
listaCarros = ["gol","uno","corsa","palio","idea"]
listaCores = ["branco","verde","preto","cinza","azul"]

com a saída neste formato:
"O gol é branco"
eu fiz o seguinte:
n=0

while n <= 5:

    print("\nO", end=' ')
    print(listaCarros[n],end=' é ')
    print(listaCores[n])
    n = n + 1

deu certo, porém apareceu essa mensagem:
O Traceback (most recent call last): "caminho do arquivo"
line 286, in
    print(listaCarros[n],end=' é ')
IndexError: list index out of range
teria um jeito melhor de fazer isso?
Comment: @Miguel é bem simples.... Serie  listaCarros = ["gol","corsa","palio"]  e outra de cores do mesmo jeito. E no caso associar as duas só para imprimir naquele formato, deu certo aquele jeito que fiz, porém está dando essa msg de IndexError e se por acaso tem algum jeito melhor pra fazer isso
Comment: Estou a constriur uma resposta William
Answer: A condição <= 5 está errada, pois se as listas têm 5 elementos só têm entre 0 e 4 indices (pois estes começam em 0), experimenta nessa versão do código while n < 5, ou melhor, while n < len(listaCarros): .
Dito isto, aqui fica uma versão mais pythonica e com menos linhas, o 'jeito melhor':
listaCores = ['azul', 'vermelho', 'verde', 'laranja', 'branco', 'preto']
listaCarros = ['mazda', 'toyota', 'honda', 'pegeot', 'mercedez', 'BMW']
text = ''
for carro, cor in zip(listaCarros, listaCores):
    text += '\nO {} é {}'.format(carro, cor)
print(text)

DEMONSTRAÇÃO
Para fazer como estavas a fazer (com while) usando este exemplo (precisas de ter o cuidado de as listas pussuírem o mesmo número de elementos, ou construir o ciclo com base na lista com menos elementos):
listaCores = ['azul', 'vermelho', 'verde', 'laranja', 'branco', 'preto']
listaCarros = ['mazda', 'toyota', 'honda', 'pegeot', 'mercedez', 'BMW', 'saf', 'safs']

n = 0
text = ''
min_list = min(len(listaCores), len(listaCarros))
while n < min_list: # fazes isto caso nao tenhas a certeza que as listas tem o mesmo num de elementos
    text += '\nO {} é {}'.format(listaCarros[n], listaCores[n])
    n += 1
print(text)

DEMONSTRAÇÃO
Comment: isso mesmo...  legal... é porque estou iniciando agora no python e por exemplo no C, se eu colocar printf("O %s eh %s", arrayCarro[i], arrayCor[i]); e imprimo certinho no padrão...  mas ein python queria ver se tinha outro jeito de formatar essa saida sem ser daquele jeito que fiz... vou testar desse jeito que tu fez!
Comment: entendi mano! muito obrigado.
Comment: De nada @WilliamHenrique , adicionei demonstrações para veres, de ambas as construções

Associar duas listas em python


MySQL Many-to-Many Relationship Select Conditions

Question: I have two tables books and tags which are in a many-to-many relationship via the table books_tags. What I want to do is select all books that have one or more tag(s) like '%tag%'. No problem. However, I also have a group_concat field which I want to be filled with all the tags that are related to the selected book.
My select query basically looks like this:
select
  flbdb_books.id,
  flbdb_books.title,
  flbdb_tags.id,
  flbdb_tags.name,
  group_concat(distinct concat('["', flbdb_tags.id, '","', flbdb_tags.name, '"]') separator ',') as tags

from
  flbdb_books

join flbdb_books_tags
  on flbdb_books.id = flbdb_books_tags.book_id
join flbdb_tags
  on flbdb_tags.id = flbdb_books_tags.tag_id

group by flbdb_books.id;

This delivers the desired result if I don't have any conditions. I use PHP and json_decode to read the tags field.
I have tried three different ways to filter the results, but none of them works.

use where:
where flbdb_tags.name like '%poetr%'
however, this means that the tags field (the one that is assembled using group_concat) is only filled with this tag
use having
having flbdb_tags.name like '%poetr%'
this does not return all books tagged poetry, because in some cases the poetry tag is hidden behind another tag due to the group by
use having on the group_concat field
having tags like '%poetr%'
this would actually do the trick, but I need to do the case-insensitive searching, and upper(tags) doesn't work

I hope it is clear what I want to do.
Thanks
Comment: Why doesn't your #3 solution work? doing a like search like that is case-insensitive to begin with. There is no difference between using `'%poetry%'`,`'%POETRY%'`, or even `'%pOeTrY%'`
Comment: This may be a good case to use `inner_join`.
Comment: @Brian I don't know why, but like is not case-insensitive in this case. Maybe it's because of the group_concat?
Answer: Will it work for you?
select
flbdb_books.id,
flbdb_books.title,
COUNT(CASE
WHEN flbdb_tags.name LIKE '%poetr%' THEN 1
END) as num_poetry,
group_concat(distinct concat('["', flbdb_tags.id, '","', flbdb_tags.name,
'"]')         separator ',') as tags
from
flbdb_books

join flbdb_books_tags
on flbdb_books.id = flbdb_books_tags.book_id
join flbdb_tags
 on flbdb_tags.id = flbdb_books_tags.tag_id

group by flbdb_books.id
HAVING num_poetry >0;

Answer: Not sure I clearly understand, but I'd try to wrap your original select and do the filtering in the wrapper select
select * from (
select
  flbdb_books.id,
  flbdb_books.title,
  flbdb_tags.id,
  flbdb_tags.name,
  group_concat(distinct concat('["', flbdb_tags.id, '","', flbdb_tags.name, '"]') separator ',') as tags
from
  flbdb_books
join flbdb_books_tags
  on flbdb_books.id = flbdb_books_tags.book_id
join flbdb_tags
  on flbdb_tags.id = flbdb_books_tags.tag_id
group by flbdb_books.id) inner_select
where upper(inner_select.name) like '%herpderp%'

Answer: There is a relational query that does exactly what you need.
The insight is that the filter tag is your starting point, and you actually need to join from the tag to the book back on to tags.
select
 books.id,
 books.title,
 tags.id,
 tags.name,

 group_concat(distinct concat('["', tags.id, '","', tags.name, '"]') separator ',') as tags

from flbdb_tags filter_tag

join flbdb_books_tags filter_book_tags
  on filter_tag.id = filter_book_tags.tag_id

join flbdb_books books
  on filter_book_tags.book_id = books.id

join flbdb_books_tags books_tags
  on books.id = books_tags.book_id

join flbdb_tags tags
  on tags.id = books_tags.tag_id

where filter_tag.name like '%poetr%'

group by books.id;

You need to join twice, not once since the tag you filter is a different set of data then all tags for a book.


Computing number of business days between start/end columns

Question: I have two Dataframes

facts:

columns: data, start_date and end_date

holidays:

column: holiday_date

What I want is a way to produce another Dataframe that has columns:
data, start_date, end_date and num_holidays
Where num_holidays is computed as: Number of days between start and end that are not weekends or holidays (as in the holidays table).
The solution is here if we wanted to do this in PL/SQL. Crux is this part of code:
    --Calculate and return the number of workdays using the input parameters.
    --This is the meat of the function.
    --This is really just one formula with a couple of parts that are listed on separate lines for documentation purposes.
    RETURN (
        SELECT
        --Start with total number of days including weekends
        (DATEDIFF(dd,@StartDate, @EndDate)+1)
        --Subtact 2 days for each full weekend
        -(DATEDIFF(wk,@StartDate, @EndDate)*2)
        --If StartDate is a Sunday, Subtract 1
        -(CASE WHEN DATENAME(dw, @StartDate) = 'Sunday'
            THEN 1
            ELSE 0
        END)
        --If EndDate is a Saturday, Subtract 1
        -(CASE WHEN DATENAME(dw, @EndDate) = 'Saturday'
            THEN 1
            ELSE 0
        END)
        --Subtract all holidays
        -(Select Count(*) from [dbo].[tblHolidays]
          where  [HolDate] between @StartDate and @EndDate )
        )
    END

I'm new to pyspark and was wondering what's the efficient way to do this? I can post the udf I'm writing if it helps though I'm going slow because I feel it's the wrong thing to do:

Is there a better way than creating a UDF that reads the holidays table in a Dataframe and joins with it to count the holidays? Can I even join inside a udf?
Is there a way to write a pandas_udf instead? Would it be faster enough?
Are there some optimizations I can apply like cache the holidays table somehow on every worker?
Answer: Something like this may work:
from pyspark.sql import functions as F

df_facts = spark.createDataFrame(
    [('data1', '2022-05-08', '2022-05-14'),
     ('data1', '2022-05-08', '2022-05-21')],
    ['data', 'start_date', 'end_date']
)
df_holidays = spark.createDataFrame([('2022-05-10',)], ['holiday_date'])

df = df_facts.withColumn('exploded', F.explode(F.sequence(F.to_date('start_date'), F.to_date('end_date'))))
df = df.filter(~F.dayofweek('exploded').isin([1, 7]))
df = df.join(F.broadcast(df_holidays), df.exploded == df_holidays.holiday_date, 'anti')
df = df.groupBy('data', 'start_date', 'end_date').agg(F.count('exploded').alias('business_days'))

df.show()
# +-----+----------+----------+-------------+
# | data|start_date|  end_date|business_days|
# +-----+----------+----------+-------------+
# |data1|2022-05-08|2022-05-14|            4|
# |data1|2022-05-08|2022-05-21|            9|
# +-----+----------+----------+-------------+

Answers:

Is there a better way than creating a UDF...?

This method does not use udf, so it must perform better.

Is there a way to write a pandas_udf instead? Would it be faster enough?

pandas_udf performs better than regular udf. But no-udf approaches should be even better.

Are there some optimizations I can apply like cache the holidays table somehow on every worker?

Spark engine performs optimizations itself. However, there are some relatively rare cases when you may help it. In the answer, I have used F.broadcast(df_holidays). The broadcast sends the dataframe to all of the workers. But I am sure that the table would automatically be broadcasted to the workers, as it looks like it's supposed to be very small.
Comment: Looks promising, I wonder if I'll have any memory problems with explode. My data is in 71 columns. Will try and update.


Units and Zero Divisors in different Rings

Question: I am aware about these algebraic structures which are always contained in each other
$Rings\supset Commutative\ rings\ with\ unity\supset Integral\ Domains\supset fields$
There have been many questions in my mind about the existence of units and zero divisors
Is it possible that in an ID we have elements which are not units?
Is it possible to have a commutative ring with unity which have elements

that are both units and zero divisors
that are units but not zero divisors
that are not units but zero divisors
that are not both

how can we prove or negate each statement
Answer: A very interesting result about domains and units, which you should try proving yourself, is that every finite integral domain is a field. Hence if we are looking for a domain with non-unit elements, we should look at infinite domains. One example is $\mathbb{Z}$, which is an ID but the only units are $1$ and $-1$.

A unit cannot be a zero divisor: let $a, a^{-1}, b \in R$, such that $a.b=0, a \neq 0$. Then, multiplying by $a^{-1}$ on the left, we see that $a^{-1}.a.b=0 \implies b=0$, hence $a$ is not a zero-divisor. Note that we didn't need the fact that multiplication is commutative.
Every non-zero element of a field is a unit but not a zero-divisor.
In $\mathbb{Z}/ \mathbb{6Z}$, which is not a domain, $\bar{2}.\bar{3}=\bar{0}$ but neither are units in $\mathbb{Z}/ \mathbb{6Z}$
Again, we take a look at infinite IDs, like $\mathbb{Z}$ where there are no zero divisors and no units apart from $1$ and $-1$.


Would my 12 string guitar break if i added pairs of strings instead of octaves and removed the last low-E string?

Question: I've heard of lots of famous musicians doing this, for example Ariel Camacho. And their guitars never break. I'm wondering if I added pairs of strings and removed the last low-E string, would it break?
Answer: All strings on a guitar are at approximately the same tension, +/- 15% ish. The fact that you put, say, an extra ordinary A string in place of the octave A won't make much difference, as the octave string is much smaller gauge but is also tuned up an octave, so will end up with about the same tension as its partner. Look up 'string tension' and you'll find exact figures for given open string gauges tuned to standard.
There isn't much point in leaving off the last bottom E, in fact, that may cause imbalance in the neck, due to everything else being pretty well the same tension as original, but less so on the topside of the neck.
A potential problem may be the nut slots. A bonus may be the intonation is more accurate!
That apart, you say loads of folk have done it, so the only other  criterion is will your guitar take it. That, we cannot vouch for here.
Comment: "will your guitar take it." Unfortunately, the answer to this is "there's one sure way to find out."
Comment: I agree with your statement about potential tension imbalance if you leave off one of the low E strings.  But read the answer I just posted describing the problem that might be encountered with a unison tuned pair of same diameter low e strings and see if you agree.
Comment: @RockinCowboy - good thought about vibrating into each other. A change of nut should sort it, particularly for the lower three pairs, but actual dimensions I can't help with. No 12 strings available right now! However, it's only that half so measuring the amount the, say, E string moves at 12th fret would be a good guide to where the other of the pair would live. I recommend the lightest strings regardless..! If it stays a problem, there's always a sort of Nashville tuning variation, maybe leaving the octave 6 in place. It won't make that much difference to the strummed sound.
Comment: my concern would be that the neck would bow a little - you'd need to add truss rod tension
Answer: Tim is correct about the tension.  As long as you tune each pair of strings in unison vs octaves, you will not alter the overall tension significantly by using pairs of strings that are each the same gauge.
My concern would be the string spacing and the nut slot.  A standard 12 string guitar will have the nut slot cut for a pair of strings of different diameter (to be tuned an octave apart) for all but the two highest strings (pair of high e strings and pair of b strings which are to be tuned in unison vs octaves).
So for the pair in the Low E position the nut would have one slot sized for a standard low E string paired with a narrower nut slot for the lower diameter octave E string that would be part of a standard 12 string set.
 Pictured is a standard 12 string guitar at the nut.
Two issues will likely arise if you tried to put two standard low E strings on  a 12 string guitar with a standard 12 string nut. First, the nut slot meant for the octave string would be too narrow for the standard low e string.
Certainly you could file the nut slot and make it wider.  But here is the second problem that I believe you might encounter.  The larger diameter wound strings such as the Low E, have a wider oscillation pattern than the smaller diameter strings (the arc of the vibration).  That is the reason the bridge on a guitar is higher on the bass side.  The bass strings must sit higher off the fretboard than the treble strings to accommodate the greater amount of string oscillation or you would encounter fret buzz.
So if you put two low E strings too close to one another, I would expect that they would hit each other when strummed or plucked at the same time.  If you put them too far apart, you will need really fat fingertips to press them both down at the same time if you were playing for example, a standard open G chord formation fretting both the Low E pair and the A pair.
That might be why you were thinking of using only one low E string.  But you might have a similar issue with the next pair of bass strings (the A position).
It's possible that you might need to have a qualified guitar tech or luthier make you a custom nut slot to provide for both the width of the slots for your unison bass strings and adjust the spacing on the pairs of unison bass strings to find the point where they don't buzz against one another but are still close enough together to fret them both with one fingertip.
That said, you would want to be sure to start with a guitar that had at least a 1 7/8" wide nut so you would have room for a slightly wider string spacing.  Some 12 string guitars have a narrower 1 3/4" nut.
FWIW Ariel Camacho reportedly used a Takamine P3DC 12 string (part of Takamines "Pro Series").  But any 12 string with a 1 7/8" nut should be adaptable to your desire to use two pairs of standard strings instead of a standard 12 string set.
I would recommend using the lightest gauge standard strings you can find to minimize the occurrence of the unison bass strings buzzing against each other and reduce the spacing that might be necessary to avoid it.
Good luck.
Answer: What could also work is: use an A string as a second low E string, a D string for a second low A string etc etc. The strings are thus less tense and this could reduce the probability of damaging the guitar.
What I also did is tune the guitar down a whole step or even a minor third.


VMware publish new system to second webserver and make it the live server?

Question: HI I've got a Vmware vsphere4 environment that is now running my core systems (web, mail etc) and we have been plagued over the years by a few faulty software publishes to our webserver that has then taken it out of action whilst the developers resolve the issue.
I was keen to find a way where I can take a clone of our live webserver within Vmware vsphere4, publish and test this server on the night and when we are happy swap it over with the current server. However after some searching and digging this process does not seem to be a quick, simple or even fun process.
Am i missing something or over thinking this entire process?
As so far I have this as the current process, clone the server, boot it up not connected to the Vm Network, re-configure all of the IP addresses for the box including changing the hostname. restart the vm and connect it to the network.
Publish the updated software to this server and test it.
bring down the live server for a holding page and then repeat the process in step 1 but this time putting all the ip addresses and hostnames back as they were.
Thanks for any advice
Comment: VMware is a company, not a product. Please edit your question (and its tags) to clarify what exactly you're working with.
Comment: @ErikA all done and updated.
Answer: While doing something like this with cloned VMs is possible, it's likely overkill.
I'd recommend the following:

Bring up a load balancer in front of two (or more) webservers. Depending on how your application is written, you'd either have requests balanced between the two servers or directed to a single server.
When it comes time for an upgrade, drop one server out of the load balancer's rotation, then upgrade that server.
Once that server is upgraded and tested as functional, add it back into the rotation.
Repeat steps 2 and 3 with the other server.
Answer: Why are you cloning it every time? Typically, I handle this by cloning the production server once every 6 months or so just to keep things tidy. Then use some kind of configuration management tool like SCCM to make sure that the config for the servers stays the same. Run through your deploy process once on the test server and verify everything, then if it works, just run through it again on the production server. If you make a task for it in SCCM, you'll only have to run through the steps once.
The production server shouldn't need to be cloned every single time since it's configuration should be made to mirror that of the live server.


Javascript regex causing browser hanging

Question: I'm trying to parse a stacktrace with regex, but it seems that it has either some major performance issues, or something more horribly wrong. When I have it run in code, the browser seems to hang forever, tho when I run it in the console, it takes maybe 10 or 15 seconds to return. I'm testing in chrome.
Here's an example:
"Object.printStackTrace.implementation.createException (file:///F:/billysFile/code/javascript/nodejs/deadunit/browserPackage/deadunit.browser.gen.umd.js:942:19)"
  .match(/^(((new )?[a-zA-Z_$][\w$]*(.[a-zA-Z_$][\w$]*)*)\(\)@((?:http|https|file):\/\/[^\s)]+|javascript:.*)(:(\d*):(\d*))|((new )?[a-zA-Z_$][\w$]*(.[a-zA-Z_$][\w$]*)*) \(((?:http|https|file):\/\/[^\s)]+|javascript:.*)(:(\d*):(\d*))\)|((new )?[a-zA-Z_$][\w$]*(.[a-zA-Z_$][\w$]*)*) (\(native\)))$/)

and here's the original regex before composition:
"Object.printStackTrace.implementation.createException (file:///F:/billysFile/code/javascript/nodejs/deadunit/browserPackage/deadunit.browser.gen.umd.js:942:19)".match(CHROME_STACK_LINE)

var IDENTIFIER_PATTERN_ = '[a-zA-Z_$][\\w$]*';
var URL_PATTERN_ = '((?:http|https|file)://[^\\s)]+|javascript:.*)';

var CHROME_FILE_AND_LINE = URL_PATTERN_+'(:(\\d*):(\\d*))'
var CHROME_COMPOUND_IDENTIFIER = "((new )?"+IDENTIFIER_PATTERN_+'(\\.'+IDENTIFIER_PATTERN_+')*)'

var CHROME_ANONYMOUS_FUNCTION = CHROME_COMPOUND_IDENTIFIER+'\\(\\)'+'@'+CHROME_FILE_AND_LINE
var CHROME_NORMAL_FUNCTION = CHROME_COMPOUND_IDENTIFIER+' \\('+CHROME_FILE_AND_LINE+'\\)'
var CHROME_NATIVE_FUNCTION = CHROME_COMPOUND_IDENTIFIER+' (\\(native\\))'

var CHROME_FUNCTION_CALL = '('+CHROME_ANONYMOUS_FUNCTION+"|"+CHROME_NORMAL_FUNCTION+"|"+CHROME_NATIVE_FUNCTION+')'
var CHROME_STACK_LINE = new RegExp('^'+CHROME_FUNCTION_CALL+'$')

Am I doing something horribly wrong here? It doesn't look to me like catastrophic backtracing..
Comment: Congratulations! You do indeed have a pathological regular expression of some sort. `(.[a-zA-Z_$][\w$]*)*` could be the culprit.
Comment: Maybe I should just use PEG..
Comment: Maybe you should split on `" "`. Depending on what you're trying to extract.
Comment: I think changing . to \. might actually have solved it. I really was trying to match a dot, not anything - oops. BB in a second with results!
Comment: You need to re-evaluate what comes after 'new' in that regex. This `( . [a-zA-Z_$] [\w$]* )*` dot in there is a major problem.
Comment: It did in fact solve it! Thanks so much for your help! Good eye. Write it up as an answer and I'll accept it.
Answer: This ( . [a-zA-Z_$] [\w$]* )* dot in there is a major problem.
Change it to \. maybe?
This is your expanded regex.
Get a tool like RegexFormat4 to look at your
large regex's.
 ^
 (
      (
           ( new\  )?
           [a-zA-Z_$] [\w$]*
           ( . [a-zA-Z_$] [\w$]* )*
      )
      \(\)@
      (
           (?: http | https | file )
           :// [^\s)]+
        |  javascript: .*
      )
      (
           :
           ( \d* )
           :
           ( \d* )
      )
   |  (
           ( new\  )?
           [a-zA-Z_$] [\w$]*
           ( . [a-zA-Z_$] [\w$]* )*
      )
      \ \(
      (
           (?: http | https | file )
           :// [^\s)]+
        |  javascript: .*
      )
      (
           :
           ( \d* )
           :
           ( \d* )
      )
      \)
   |  (
           ( new\  )?
           [a-zA-Z_$] [\w$]*
           ( . [a-zA-Z_$] [\w$]* )*
      )
      \
      ( \(native\) )
 )
 $


Forward Shibboleth Environment Variables to Tomcat via Apache

Question: I am using Shibbolethv2.3 with Apache web server and Tomcat application server. I am using Apache as a reverse proxy using mod_proxy.so. I am not able to forward the Shibboleth environment variables from Apache to Tomcat. I am able to forward the attributes in the headers but as already mentioned in the wiki this approach is not safe.
I have tried forwarding the environment variables by the following directive :
SetEnv AJP_username ${username}
then at the Java side I can access the attribute by : request.getAttribute("username");
The strange thing here is that, I get a different value instead of the one set by Shibboleth. I get the Windows account name as a result. If I use any other attribute name, I get a null value.
I have searched a lot and have run out of options. Please guide me towards the right solution.
My setup details :

Shibboleth version : 2.3
OS : Windows XP SP3
Webserver : Apache 2.2
Application Server : Tomcat 6
Proxy module : mod_proxy.so
Comment: Make sure that your `attribute-map.xml` is set up correctly, sharing it might help.
Answer: Make sure you are proxying through using ajp in the httpd.conf
ProxyPass /example ajp://example.org/example

And in the shibboleth2.xml in the ApplicationDefaults section we need to add
attributePrefix="AJP_"

So it looks something like
<ApplicationDefaults id="default" policyId="default"
    entityID="https://idp.example.org"
    REMOTE_USER="eppn persistent-id targeted-id"
    signing="false" encryption="false" attributePrefix="AJP_">

Check out this question on the shib mailing list:
http://groups.google.com/group/shibboleth-users/browse_thread/thread/2bdd3e272baf49a2?pli=1
Comment: Why the prefix?
Comment: Prefix seems to be needed but the attributes are available without prefix in Java.


Using preprocess node variable in page.tpl.php

Question: I've set up a couple of variables in my template_preprocess_node(). How can I access them within a page template? I can access the $node variable, but it doesn't seem to have been preprocessed.
How can I force preprocessing?
Answer: The node variable for page templates is set in template_preprocess_page
http://api.drupal.org/api/drupal/includes--theme.inc/function/template_preprocess_page/6
You should override that function in your theme
Comment: I know, but since I've already overridden preprocess_node, I'd rather prefer not to duplicate the code, and use the preprocessed node within the page.
Comment: Actually I can answer that myself.  Yes you can.  In Drupal 7 they seem to be available as `$variables['node']->field_example`.
Comment: Wrap your code in a function, I'm pretty sure that's the only way to go.
Comment: @Eelke: In `template_preprocess_page` can you access fields of the node?


Como definir um alias global no BASH?

Question: Estou utilizando o Terminal no Linux e estou cansado de ficar repetindo os mesmos comandos toda hora.
Por exemplo, eu uso um comando praticamente o dia inteiro, daí eu crio um alias para poder facilitar as coisas.
Exemplo:
  alias tt="php artisan tinker --env=local"

  alias artisan="php artisan --env=local"

Porém, toda vez que o Terminal é encerrado, esses atalhos "vão para o saco". Daí eu tenho que definir tudo de novo quando abro o Terminal de novo  ou reinicio o meu Ubuntu;
Como posso fazer para deixar esses Aliases globais? Tem alguma forma de fazer isso?
Comment: Podem falar que não é "dentro do escopo", mas que uso isso todo dia para programar, uso!
Answer: Você pode acrescentar essas linhas no arquivo /etc/profile, Assim que o usuário efetuar o login, as informações são carregadas.
$ sudo vim /etc/profile
ou
$ sudo nano /etc/profile

profile:
#...
alias tt="php artisan tinker --env=local"
alias artisan="php artisan --env=local"

Para aplicar as alterações, deve efetuar o logout.
Comment: Apenas relembrando @Brumazzi DB que basta fazer `$source /etc/profile` ou `#source /etc/profile` que já atualiza e torna desnecessário efetuar o logout.
Comment: @ArmandoMarquesSobrinho, sim mas isso só se aplica no terminal que foi feito o `source`, caso abra outro terminal a configuração não é aplicada, por isso a sugestão do logout.
Comment: sem problemas, foi só para implementar! mas pelo que eu saiba se abrir outro terminal ele ja vem carregado, pelo menos no meu com Scientific Linux, mas vou testar numa maquina que tenho aqui das crianças ver filmes e que usa o ubuntu.
Comment: opa, tens razão @Brumazzi DB
testei lá no ubuntu, precisa mesmo efetuar logout
mas no meu com RHEL nunca precisei
Comment: @ArmandoMarquesSobrinho, Algumas distribuições vem com o `.bashrc` ou o `.profile` pré-configurados, provavelmente o seu já importa as configurações diretamente do `/etc/profile` por causa dessas configurações.
Comment: Isso seria global para todos os usuários? Será que é isso mesmo que ele quer?
Answer: Coloca os teus aliases dentro do arquivo /home/$USER/.bashrc .
Este arquivo é executado todas as vezes que você abrir um terminal gráfico.
Adicione, preferencialmente, os seus aliases no final do arquivo.
Por exemplo, você pode usar o nano para editar esse arquivo:
 sudo  nano /home/$USER/.bashrc

Aí você pode adicionar qualquer alias:
artisan="php artisan --env=local"

Como definir um alias global no BASH?