Loop/Iterate through date range while inserting

Question: So I have data being imported into a table (let's call is RAWDATA) in the following format:
EMPID  | STARTDATE  | ENDDATE    | TOTALHOURS | TOTALWAGES
ABC123 | 01-01-2013 | 01-28-2013 | 160.0      | 1800.00
XYZ987 | 01-01-2013 | 01-31-2013 | 200.0      | 2500.00

I need to take that data, and put it in a different table (EMPDATA) in the following format:
EMPID  | DATE       | HOURS | WAGES
ABC123 | 01-01-2013 | 5.71  | 64.29
ABC123 | 01-02-2013 | 5.71  | 64.29
ABC123 | 01-03-2013 | 5.71  | 64.29
...... | .......... | ....  | .....
XYZ987 | 01-01-2013 | 6.45  | 80.66
XYZ987 | 01-02-2013 | 6.45  | 80.66
XYZ987 | 01-03-2013 | 6.45  | 80.66
...... | .......... | ....  | .....

My thought is to do a DATEDIFF between STARTDATE and ENDDATE to figure out how many days (in this case: 28) to spread the hours and wages over, then for each day, insert a row that contains the average hours and wages worked per day. This will all be done with a trigger on the RAWDATA table. I'm just not sure how to iterate from STARTDATE to ENDDATE in a trigger.
EDIT:
I should also state that the data being imported does not always have the same start/end date for each row. I have updated the first table example to indicate this.
Comment: Do you need to handle just one row, or multiple rows?
Comment: Multiple rows. I have updated the question
Comment: And what is the maximum date range you need to support? 31 days, or longer? And is this SQL Server 2000, 2005, 2008, 2008 R2, 2012?
Comment: @D.R. The demo included in my answer also shows multiple rows.
Answer:
Create a date table and use a JOIN.
Calculate days between startdate and enddate
Divide totalhours and totalwages by calculated days.

Here's my solution:
SELECT a.empid, b.dd AS date,
  CAST(a.totalhours AS decimal) / (DATEDIFF(day, startdate, enddate) + 1) AS hours,
  CAST(a.totalwages AS decimal) / (DATEDIFF(day, startdate, enddate) + 1) AS wages
FROM wages a
INNER JOIN dates b ON dd BETWEEN a.startdate AND a.enddate

Result
|  EMPID |       DATE |         HOURS |          WAGES |
--------------------------------------------------------
| ABC123 | 2013-01-01 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-02 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-03 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-04 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-05 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-06 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-07 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-08 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-09 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-10 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-11 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-12 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-13 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-14 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-15 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-16 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-17 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-18 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-19 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-20 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-21 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-22 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-23 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-24 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-25 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-26 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-27 | 5.71428571428 | 64.28571428571 |
| ABC123 | 2013-01-28 | 5.71428571428 | 64.28571428571 |
See the demo
Comment: @swasheck, sdlect into willnot work in a trigger, it only works if the table does not exist.  YOu need to add insert to the top of this with your table name and fields. The author of this is right, we shouldn't do your work for you.
Comment: i need to have coffee before i start commenting/answering on SO
Comment: @D.R. that depends, what is your specific objection to a date table?
Comment: @D.R. Date table is really your **best** solution. Just take a quick peak at the performance charts in [this article](http://www.sqlperformance.com/2013/01/t-sql-queries/generate-a-set-3) .
Comment: I consulted with the dba and he is OK with a date table so I have used your solution! Thanks!
Comment: But I'm inserting the data into another table, not just generating a different result set
Comment: @D.R. Please see [INSERT with SELECT syntax](http://msdn.microsoft.com/en-us/library/ms188263%28v=sql.105%29.aspx). That should help do the job.
Comment: @PolishPrince The question specifically asked for an INSERT statement.
Comment: By far the best solution. triggers need high performat code, not iteration or recursion. I'd upvote a million times if I could.
Comment: @swasheck SELECT INTO will create a new table. Not bad, but should be mentioned. And yeah, thanks for the syntax, I missed the whole SQL Server part.
Comment: @PolishPrince I'd also like to thank you for showing me SQL Fiddle! That's a great site!
Comment: Any solution where a date table is not required?
Comment: @AaronBertrand FWIW, you converted me to a date table. I owe you a beer.
Comment: I assume for a trigger, the only change I need to make is to add the INSERT part and JOIN it on the "INSERTED" table as well so it only does it for rows that have been inserted
Comment: Here's a weird one. I put the sample data into the database, and I'm getting 6.93 and $67.67 for avghours and avgwages for row 1 and 7.67 and $84.33 as the averages for row 2! This is bizarre...going to go through it with a fine-tooth comb!
Comment: Darn order of operations! I forgot the parenthesis around the DATEDIFF(day,startdate,enddate)+1 so it was dividing by the wrong number, then adding one! all fixed now!
Comment: @D.R. Sounds to me like the `datediff` is getting miscalculated. If you can, throw the sample data into a fiddle with the query you're using.
Comment: Good call on the date table. Saved me a lot of time!
Answer: Assuming you're using SQL Server 2005 and higher, you can use a Recursive CTE for this to create your dynamic date ranges (vs using a lookup table).  This uses GROUP BY on the empid field and assumes a single row for each empid:
WITH RecCTE AS (
  SELECT empid, startdate
  FROM rawdata
  UNION ALL
  SELECT R.empid, DATEADD(day,1,R.startdate)
  FROM RecCTE R JOIN rawdata RD ON R.startdate < RD.enddate
  )
INSERT INTO EMPDATA
SELECT R.EmpId, R.StartDate, T.TotalHours/R2.cnt, T.TotalWages/R2.cnt
FROM RecCTE R
  JOIN (SELECT empid, totalhours, totalwages
        FROM rawdata
       ) T ON R.empid = T.empid
  JOIN (SELECT EmpID, COUNT(*) cnt
        FROM RecCTE
        GROUP BY EmpID) R2 ON R.EmpID=R2.EmpId

SQL Fiddle Demo
There might be a way to make this a little more efficient, but this should get you going in the right direction.  For example, you could use a COUNT OVER Function to return the count vs the subquery:
WITH RecCTE AS (
  SELECT empid, startdate
  FROM rawdata
  UNION ALL
  SELECT R.empid, DATEADD(day,1,R.startdate)
  FROM RecCTE R JOIN rawdata RD ON R.startdate < RD.enddate
  )
INSERT INTO EMPDATA
SELECT R.EmpId, R.StartDate,
    T.TotalHours/COUNT(1) OVER (PARTITION BY R.EmpId),
    T.TotalWages/COUNT(1) OVER (PARTITION BY R.EmpId)
FROM RecCTE R
  JOIN (SELECT empid, totalhours, totalwages
        FROM rawdata
       ) T ON R.empid = T.empid
ORDER BY R.StartDate

Comment: Thanks for the answer, but I was able to use PolishPrince's answer! I appreciate the help though
Comment: @D.R. -- not a problem -- glad they could help.  Just wanted to offer an alternative to having to create and maintain a separate dates table.  Best regards!
Comment: What is everyone's big objection to "having to create and maintain a separate dates table"? Is it really that hard to build a table, once, populate it, once, and not have to maintain it at all (until our calendar changes)? Is the concern performance? (Such a table will be tiny, and if you reference it enough, it will always be in memory.) I don't comprehend why such a useful table is met with such resistance.
Comment: @AaronBertrand -- I have no issue with the accepted answer (I gave it a +1).  I was trying to supply an alternative (as OP asked for).  In my consulting career, there have been several instances/clients where I can NOT create a new table (permissions for example), and therefore must utilize other means to solve the problem (in this case, a recursive CTE).  Best regards.
Comment: @sgeddes right, the question wasn't directed at you, a little more general than that. People constantly scoff at the suggestion to create a table and I guess I don't like to see that scoffing encouraged. :-)
Comment: Seeing how useful a date table can be, you almost wish MSSQL and/or other DB software included a system-wide table for dates (or some function that would act as the same)


Is incremental data loading supported in Alibaba QuickBI? If supported then how can we achieve it?

Question: How to incrementally load data in Quick BI from the external database without any third party ETL tool?
Because there is no ETL support as of now.
Answer: Again you need to wait for the Data Lake Analytics to be launched. This will not only solve your requirement, but this will also bridge the gap between the BI products and the DB's (the incremental support)


HTTP 400 483 with Apache ProxyPassMatch and RewriteCond

Question: I keep getting an error of:
apache_1_da0b6d97082f | 172.24.0.1 - - [16/Sep/2019:15:58:52 +0000] "GET /logi HTTP/1.1" 400 483 "-" "curl/7.64.0"

But I have no idea why? It seems the request does not get passed through to the php-fpm socket.
ServerName mainapp.local

<VirtualHost *:80>
    ServerName mainapp.local

    DocumentRoot /var/www/html/public/
    CustomLog /dev/stdout combined
    LogLevel debug

    RewriteEngine On

    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_FILENAME} !-d
    RewriteRule ^(.*)$ index.php [L,QSA]

    ProxyPreserveHost On
    ProxyPassMatch "^/(.*\.php(/.*)?)$"           "unix:/this/path/does/not/matter.sock|fcgi://localhost/var/www/html/public"

    <Directory "/var/www/html">
        Options +Indexes +FollowSymLinks +IncludesNOEXEC -MultiViews
        AllowOverride All
        Require all granted
        DirectoryIndex index.php
    </Directory>

</VirtualHost>

It works correctly (HTTP 200) without the Rewrite directives - but obviously I do need this functionality.

If I enable LogLevel debug rewrite:trace8:
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] init rewrite engine with requested uri /test
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] applying pattern '(.+)' to uri '/test'
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] RewriteCond: input='/test' pattern='!-f' => matched
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] RewriteCond: input='/test' pattern='!-d' => matched
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] rewrite '/test' -> 'index.php?p=/test'
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] split uri=index.php?p=/test -> uri=index.php, args=p=/test
... mod_rewrite.c(483): [client 172.24.0.1:50934] 172.24.0.1 - - [localhost/sid#7f20f4ad15e8][rid#7f20f40e50a0/initial] local path result: index.php

Here is the Dockerfile I used to build the image:
# FROM debian:buster

FROM debian@sha256:903779f30a7ee46937bfb21406f125d5fdace4178074e1cc71c49039ebf7f48f

ARG DEBIAN_FRONTEND=noninteractive

RUN apt update
RUN apt install -y apache2 libapache2-mod-fcgid

COPY ./mainapp_vhost.conf /etc/apache2/sites-available/000-default.conf

RUN a2enmod actions proxy proxy_ajp proxy_http proxy_fcgi rewrite
RUN mkdir /var/lib/php-fcgi

RUN chmod 644 /var/log/apache2

ENV APACHE_UID 33
ENV APACHE_GID 33

RUN ln -sf /dev/stdout /var/log/apache2/access.log
RUN ln -sf /dev/stderr /var/log/apache2/error.log

RUN mkdir /var/www/html/public

RUN apachectl configtest

ENTRYPOINT ["/usr/sbin/apache2ctl"]
CMD ["-D", "FOREGROUND"]

Comment: Check the Apache error log.
Comment: I did attach the appropriate out from the error/access log.
Comment: I see an access log entry, but the debug trace entries in your question don't correspond to it. Please look further.
Comment: Either my configuration is not logging the errors - or Apache isn't writing anything to the error log. Is there some way to verify?
Comment: Look for the `ErrorLog` directive in the Apache configuration and ensure that it does something sensible . Or at least you can find out what it was doing.
Comment: ```
root@304cf01845b4:/# grep 'Error' /etc/apache2/apache2.conf
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a
ErrorLog ${APACHE_LOG_DIR}/error.log
```
Comment: So it sounds like it's ended up in a file somewhere in your container. The default directory being `/etc/httpd/logs`
Comment: To my understanding it's going to `/var/log/apache2/error.log` which I then point to the docker's /dev/stderr - so I don't think I'm dropping any errors here.
Comment: Well, something definitely got dropped. If you made a request and got a 400 Bad Request response from Apache, there will be a corresponding error log entry explaining why the error was returned. Try reproducing it. Of course, there won't be an error log entry if the 400 response came from your app, so you should consider that possibility as well.
Comment: I appreciate your suggestions - but I'm not sure how much further I could reproduce this. If you try build the docker image, and just send a request to that process - you should get exactly the same output as I do.
Answer: The error occurs most probably due to a malformed URL - even though it reports just a vague Bad Request error.

It was probably the missing /....
RewriteRule ^(.*)$ /index.php [L,QSA]

Instead of
RewriteRule ^(.*)$ index.php [L,QSA]

Why is my rewrite rule resulting in "400: Bad Request"?


Derivation of Closed-Form Gompertz-Makeham Life Expectancy

Question: I am trying to replicate the derivation of the life expectancy formulas from Missov & Lenart (2013). They propose that the following:

Theorem 2. Life expectancy of the Gompertz–Makeham mortality distribution at age $x$ is:
$$e_{GM}(x) = \frac{1}{b} e^\frac{a}{b} \left( \frac{a}{b} \right)^\frac{c}{b} \Gamma \left( -\frac{c}{b} , \frac{a}{b}e^{bx}\right) \qquad a,b > 0, \; c\geq 0$$
where $\Gamma(s,z) = \int_z^\infty t^{s-1} e^{-t} dt$ denotes the upper incomplete gamma function

Their proof goes as follows:

Proof. The individual force of mortality and the survival function for a Gompertz–Makeham homogeneous population is given by (1) and (2) for z = 1, i.e
$$\mu_{GM}(x) = a ^{bx} + c \qquad \qquad S_{GM}(x) = \exp \left[ -\frac{a}{b} (e^{bx} - 1) -cx \right]$$
Remaining life expectancy at age $x$ can be calculated by
$$e_{GM}(x) = \int_x^\infty S_{GM}(t, y) dt \tag{B1}$$

So far I can follow, except for the added $y$ argument to the survival function. Nevertheless, they continue:

Substituting $s = e^{bt}$ and denoting $a = \frac{a}{b}$ reduces (B.1) to
$$e_{GM}(x) = \frac{e^a}{b} \int_{e^{bx}}^\infty e^{-as} q^{-\frac{c}{b}-1} dq$$

Here I have a problem: we seem to subsitute using $s$, but a variable $q$ appears. My intuition is that thefirst step here is to plug in the survival function as $\exp \left[ -\frac{a}{b} (e^{bt} - 1) -ct \right]$ and substitute for $a$, which allows us to draw $e^a$ out of the integral. But the we need to substitute for $t$, so what happens to $ct$? And how is $q$ defined?
To be complete, they the proof continues:

Finally we substitute $u = as$ and get
$$e_{GM}(x) = \frac{e^a}{b}[a]^\frac{c}{b} \int_{ae^{bx}}^\infty e^{-u} u^{-\frac{c}{b}-1} du = \frac{1}{b} e^\frac{a}{b} \left( \frac{a}{b} \right)^\frac{c}{b} \Gamma \left( -\frac{c}{b} , \frac{a}{b}e^{bx}\right)$$
Answer: As happened previously, formulating the question and thinking about it helped to find a solution. Here it is:
We start from
$$\begin{aligned}
e(x) &= \int_x^\infty \exp \left[ -ct - \frac{a}{b} (e^{bt}-1) \right] \mathsf{d}t \\
&= \int_x^\infty \exp \left[ -ct - \frac{a}{b} e^{bt} + \frac{a}{b} \right] \mathsf{d}t \\
&= \exp\left( \frac{a}{b} \right) \int_x^\infty \exp \left[ -ct - \frac{a}{b} e^{bt} \right] \mathsf{d}t
\end{aligned}$$
Now, define $q(t) = e^{bt}$, so that we can apply integration by substitution:
$$\frac{\mathsf{d} q}{\mathsf{d} t} = q'(t) = b e^{bt} \qquad \Leftrightarrow \qquad \mathsf{d} q = \mathsf{d} t \cdot q'(t)$$
This might be somewhat loose on notation, but it works (I'm not a mathematician). Nevertheless, we can continue:
$$\begin{aligned}
\exp\left( \frac{a}{b} \right) \int_x^\infty \exp \left[ -ct - \frac{a}{b} e^{bt} \right] \mathsf{d}t &= \exp\left( \frac{a}{b} \right) \int_x^\infty \exp \left[ -ct - \frac{a}{b} e^{bt} \right] \frac{q'(t)}{q'(t)}\mathsf{d}t \\
&= \exp\left( \frac{a}{b} \right) \int_x^\infty e^{-ct} \exp \left[-\frac{a}{b} e^{bt} \right] \frac{1}{b}e^{-bt} \;\; \mathsf{d} t \cdot q'(t) \\
&= \frac{1}{b}\exp\left( \frac{a}{b} \right) \int_x^\infty \exp \left[-\frac{a}{b} e^{bt} \right] \left(e^{bt} \right)^{-\frac{c}{b} - 1} \;\; \mathsf{d} t \cdot q'(t)
\end{aligned}$$
Where we used the fact that $\left(e^{bt} \right)^{-\frac{c}{b} - 1} = e^{-bt - ct}$. We then get to the first step in the proof by substituting in $A = a/b$ (less confusing with a capital A) and the definition of $q(t)$, while also setting the lower bound to $q(x)$:
$$\begin{aligned}
\frac{e^A }{b} \int_{e^{bx}}^\infty e^{-Aq} q^{-\frac{c}{b} - 1} \;\; \mathsf{d} q
\end{aligned}$$
So $s$ seems to have been $q$ all along. We can then continue, by defining $u(q) = Aq$ so that:
$$\frac{\mathsf{d} u}{\mathsf{d} q} = u'(q) = A \qquad \Leftrightarrow \qquad \mathsf{d} u = \mathsf{d} q \cdot u'(q)$$
and perform a second round of integration by substitution.
$$\begin{aligned}
\frac{e^A }{b} \int_{e^{bx}}^\infty e^{-Aq} q^{-\frac{c}{b} - 1} \;\; \mathsf{d} q &= \frac{e^A }{b} \int_{e^{bx}}^\infty e^{-Aq} q^{-\frac{c}{b} - 1} \frac{A^{\frac{c}{b}}}{A^{\frac{c}{b}}}\;\; \frac{u'(q)}{u'(q)}\mathsf{d} q \\
&= \frac{e^A }{b} A^{\frac{c}{b}} \int_{e^{bx}}^\infty e^{-Aq} q^{-\frac{c}{b} - 1} A^{-\frac{c}{b}} A^{-1}\;\; u'(q)\cdot\mathsf{d} q \\
&= \frac{e^A }{b} A^{\frac{c}{b}} \int_{e^{bx}}^\infty e^{-Aq} q^{-\frac{c}{b} - 1} A^{-\frac{c}{b} -1}\;\; u'(q)\cdot\mathsf{d} q \\
&= \frac{e^A }{b} A^{\frac{c}{b}} \int_{e^{bx}}^\infty e^{-Aq} u^{-\frac{c}{b} - 1} \;\; u'(q) \cdot\mathsf{d} q \\
&= \frac{e^A }{b} A^{\frac{c}{b}} \int_{Ae^{bx}}^\infty e^{-u} u^{-\frac{c}{b} - 1} \;\; \mathsf{d} u
\end{aligned}$$
The last step to the proof is then simply to apply the definition of the upper-incomplete gamma function.


Pyspark py4j PickleException: "expected zero arguments for construction of ClassDict"

Question: This question is directed towards persons familiar with py4j - and can help to resolve a pickling error.  I am trying to add a method to the pyspark PythonMLLibAPI that accepts an RDD of a namedtuple, does some work, and returns a result in the form of an RDD.
This method is modeled after the PYthonMLLibAPI.trainALSModel() method, whose analogous existing relevant portions are:
  def trainALSModel(
    ratingsJRDD: JavaRDD[Rating],
    .. )

The existing python Rating class used to model the new code is:
class Rating(namedtuple("Rating", ["user", "product", "rating"])):
    def __reduce__(self):
        return Rating, (int(self.user), int(self.product), float(self.rating))

Here is the attempt So here are the relevant classes:
New python class pyspark.mllib.clustering.MatrixEntry:
from collections import namedtuple
class MatrixEntry(namedtuple("MatrixEntry", ["x","y","weight"])):
    def __reduce__(self):
        return MatrixEntry, (long(self.x), long(self.y), float(self.weight))

New method foobarRDD In PythonMLLibAPI:
  def foobarRdd(
    data: JavaRDD[MatrixEntry]): RDD[FooBarResult] = {
    val rdd = data.rdd.map { d => FooBarResult(d.i, d.j, d.value, d.i * 100 + d.j * 10 + d.value)}
    rdd
  }

Now let us try it out:
from pyspark.mllib.clustering import MatrixEntry

def convert_to_MatrixEntry(tuple):
  return MatrixEntry(*tuple)

from pyspark.mllib.clustering import *
pic = PowerIterationClusteringModel(2)
tups = [(1,2,3),(4,5,6),(12,13,14),(15,7,8),(16,17,16.5)]
trdd = sc.parallelize(map(convert_to_MatrixEntry,tups))

# print out the RDD on python side just for validation
print "%s" %(repr(trdd.collect()))

from pyspark.mllib.common import callMLlibFunc
pic = callMLlibFunc("foobar", trdd)

Relevant portions of results:
[(1,2)=3.0, (4,5)=6.0, (12,13)=14.0, (15,7)=8.0, (16,17)=16.5]

which shows the input rdd is 'whole'.  However the pickling was unhappy:
5/04/27 21:15:44 ERROR Executor: Exception in task 6.0 in stage 1.0 (TID 14)
net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict
(for pyspark.mllib.clustering.MatrixEntry)
    at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
    at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:617)
    at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:170)
    at net.razorvine.pickle.Unpickler.load(Unpickler.java:84)
    at net.razorvine.pickle.Unpickler.loads(Unpickler.java:97)
    at org.apache.spark.mllib.api.python.SerDe$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(PythonMLLibAPI.scala:1167)
    at org.apache.spark.mllib.api.python.SerDe$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(PythonMLLibAPI.scala:1166)
    at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
    at scala.collection.AbstractIterator.to(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
    at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:819)
    at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:819)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1523)
    at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1523)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
    at org.apache.spark.scheduler.Task.run(Task.scala:64)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:212)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)

Below is a visual of the python invocation stack trace:
Answer: I had the same error as I was using MLlib, and it turned out that I had returned a wrong datatype in one of my functions. It now works after a simple cast on the returned value. This might not be the answer you're seeking but it is at least a hint for the direction to follow.
Comment: I no longer work on that project - and so can not verify. However it seems like a reasonable consideration so have upvoted.
Answer: Had the same issue, several times.
numpy types don't have implicit conversions to pyspark.sql.types.
Make a simple explicit conversion to the native type system.
In my case it was:
float(vector_a.dot(vector_b))

Answer: I received this error using Spark Version >= 2.0.
Spark is transitioning MLlib fucntionality to the newer ML namespace.
As a result there are two types of SparseVector:
ml.linalg.SparseVector and mllib.linalg.SparseVector
Some MLlib functions still expect the older mllib kind
from pyspark.ml.linalg import Vectors
# convert ML vector to older MLlib vector
old_vec = Vectors.fromML(new_vec)

HTH
Comment: This was extremely helpful - thank you! Only thing is that in version 2.1.1 `fromML` no longer seems to exist, so I had to create the object manually by doing `pyspark.mllib.linalg.SparseVector(sv.size, sv.indices, sv.values)`, where `sv` was my `pyspark.ml.linalg.SparseVector` object.


Automorphism group of a graph = Automorphism group of that graph's adjacency matrix?

Question: Is  automorphism group (or set) of a graph $G$  equal to the automorphism group (or set) of adjacency  matrix of $G$?
Example: $G_1, G_2$ are  separate graphs  where    $G_1^{\pi}= G_2$ and $ G= \bar G_1 \cup \bar  G_2$. Also, $\pi$ swaps only $2$ vertices of $G_1$, say, it swaps $k^{th}$ vertex with $(k+1)^{th}$ vertex.
So, the adjacency matrix of $G_1 \neq$ the adjacency matrix of $G_2$.
We will have this dissimilarity in $G$ also.
$G$ is made of 2 non equal matrices of $\bar G_1, \bar  G_2$. There are only two dissimilar columns/ rows. They  are  $k^{th}$, $(k+n)^{th}$ and  $(k+1)^{th}, (k+1+n)^{th}$ vertex.  Remaining all pairs of rows/columns $(i,i+n)$ are same where $i \neq k, k+1$.
Now, consider  a permutation $\sigma$  of $G$ that  swaps $k^{th}$ vertex of $G$with $(k+1+n)^{th}$ vertex of $G$.  Note that, $k^{th}$ vertex is in  $G_1$ and  $(k+1+n)^{th}$ vertex is in $G_2$.
here, the adjacency matrix of $G^{\sigma} \neq$ the adjacency matrix of $G$.
So, Automorphism group of a graph $\neq$ Automorphism group of that graph's adjacency matrix.
Is it correct?
This post is the source of my argument.

If $A$ is a matrix and $\pi$ is an automorphism of $A$, then $A^{\pi} =A$.
Comment: @User12345   ,  if $A$ is a matrix and $\pi$ is an automorphism of $A$, then $A^{\pi} =A$.
Comment: If you're swapping the $i,j$ vertexes of a graph, this is equivalent to swapping the $i,j$ rows and $i,j$ columns (in either order) of its adjacency matrix.
Comment: @Axoren  yes, but it will happen  in $G_1$ But when you apply $\sigma$  it will move one of i , j
Comment: Are the vertex sets disjoint between $G_1$ and $G_2$? If so, you can simply combine their adjacency matrices as $\left[\matrix{A_1 & 0 \\ 0 & A_2}\right]$. This becomes the adjacency matrix of $G$.
Comment: @Axoren  yes I can do that but the question is does the auto: of G  keeps the matrix same?
Comment: @Axoren  note that $\sigma$ is an auto but it does not keep the matrix of G same.
Comment: I think I see the issue now. You don't have a notion of equivalence between adjacency matrices other than the matrix equivalence, while I'm considering the adjacency matrices as belonging to an equivalence class if they correspond to isomorphic graphs. Would that not be sufficient?
Comment: @Axoren , have a look at this question  http://math.stackexchange.com/questions/1797292/how-to-verify-an-automorphism-of-a-graph?lq=1
Comment: @Axoren   ,  you may think, permutation of the vertices of a graph corresponds to permuting the rows and columns of the adjacency matrix. It is an automorphism of the graph if and only if this leaves the adjacency matrix unchanged. But this is not case in my example given in the post. Moreover, How you verify an automorphism ?
Comment: Was a bit late on the answer, but @MorganRodgers got to the meat of it quite quickly.
Answer: Your definition of automorphism is a bit too strong, I feel. Leaving something "unchanged" in the sense that the numbers in each cell of a matrix is a VERY strong notion of equality. However, you're not using that definition of equality for graphs.

As corrected by Morgan Rodgers, strict equality actually works fine. The adjacency matrices are indeed unchanged under automorphisms when the adjacency matrices are constructed with the same vertex set and the vertex $v_i$ corresponds to column and row $i$ always.
Everything below still holds, but any mention of row-column swapping may be trivially true.

If a graph undergoes an automorphism, it has been turned into another graph with equivalent structure.
$G_1 = (V_1, E_1)$
$G_2 = (V_2, E_2)$
$f(G_1) = G_2$ is an automorphism if and only if:

There exists a bijection $g : V_1 \to V_2$ such that $(x, y) \in E_1$ if and only if $(g(x), g(y)) \in E_2$

This is not the same as "$f(G_1) = G_2$ if and only if $V_1 = V_2$ and $e \in E_1 \iff e \in E_2$", which would be true equality. This stricter definition is closer to the one you were holding for adjacency matrices. Essentially, each index in each of the vertex and edge is "unchanged", therefore under this equality any swapping at all is not an automorphism. This is not what you intend at all, I'm fairly certain.
If you consider that you're already constructing equivalence classes between graphs (otherwise, the only automorphism on graphs is the identity map), then you can similarly construct an equivalence class on the adjacency matrices.
You define a new $=$ relation such that $A_1 = A_2$ if and only if $A_2$ can be attained by a sequence of $(i,j)$ row-column swaps (swap the $i,j$ rows and $i,j$ columns) from $A_1$. For any finite graph, this is verifiable exhaustively by trying every possible sequence of swaps to check equality. This equality relation is far more relaxed than cell-by-cell equality and it gives you exactly what you need to verify automorphisms between the graphs these adjacency matrices represent.
This equality relation is consistent with any automorphism on your graph. If your automorphism swaps the $i,j$ vertices with each other, this bijectively maps to an automorphism in your adjacency matrices by swapping the $i,j$ row-columns. The reverse also holds following similar logic.
Any automorphism on your graphs can be factored into a minimal chain of single-swap automorphisms. Any automorphisms on your adjacency matrices can be constructed by composition of single-swap automorphisms (closure property of automorphisms). As such, there is a unique automorphism on the graphs for each automorphism on the adjacency matrices. The reverse is also true following similar logic.
We now have a bijective map from the automorphism group of the graphs to the automorphism group of their adjacency matrices. Therefore, the groups are equivalent.
Comment: @MorganRodgers That depends on how the adjacency matrix is constructed. If the rows and columns are labeled by vertices, row 1 will always belong to $v_1$. However, if swapping $v_1$ with $v_2$ changes this labeling instead, then you will in fact have no changes made to the matrix. I assumed the former, where row $i$ was associated with $v_i$.
Comment: Nice answer,  i am  aware of   equivalence relation, but i was asking if adjacent matrices are equal, how do I verify it, I mean practically
Comment: @MorganRodgers I see, all the row-column swapping is unnecessary. In the end, it doesn't change the result at all. It's a shame I didn't see that when writing all that up.
Answer: Each permutation $g \in S_n$ can be represented by an $n \times n$ permutation matrix $P(g)$ defined by $P(g)_{ij} = 1$ iff $g$ maps $j$ to $i$, and $P(g)_{ij}=0$ otherwise.
Let $A$ denote the adjacency matrix of the graph $G$. It can be verified that a permutation $g$ is an automorphism of the graph $G$ if and only if $P(g)^{-1}AP(g) = A$.  In other words, a permutation $g$ of the vertex set preserves adjacency (and non-adjacency) in the graph $G$ if and only if the permutation matrix $P(g)$ preserves the adjacency matrix $A$.
In this sense, the automorphism group of a graph and of its adjacency matrix are equal. When a permutation $P(g)$ is applied to an adjacency matrix $A$, you need to make sure $P(g)$ is multiplied on both sides of $A$, so that the same permutation is effected on both the columns and the rows of the adjacency matrix.  Recall that pre-multiplying a matrix $A$ effects row operations/permutations on the matrix $A$, while post-multiplying a matrix effects column operations.
Comment: @Jim If you have an automorphism $\sigma$ of $G$ that swaps the $k$  and $(2k+1))$th vertices, then $P(\sigma)$ will preserve the adjacency matrix also.  So it is not true that the adjacency matrices of $G$ and $G^\sigma$ are not equal. This is because the adjacency matrix of $G$ is $A$ and the adjacency matrix of $G^\sigma$ is $P(\sigma)^{-1}AP(\sigma)$, and if $\sigma$ is an  automorphism then these two matrices will also be equal.
Comment: @Jim In fact, one can _define_ an automorphism of a graph to be a permutation $\sigma$ such that $P(\sigma)^{-1} A P(\sigma)=A$.  This is an equivalent definition for the automorphism of a graph (in terms of the adjacency matrix of the graph rather than in terms of adjacencies in the graph).
Comment: +1 , Thanks for you answer, would you  please  edit your answer   or comment  here about the example I gave in my post?


IONIC4 Angular PWA standalone is not refreshing

Question: I'm trying to create a PWA with IONIC Angular (web).
The application is running on: https://todo-ionic.thehangrycoder.now.sh
When I open the url on an iphone via Safari, it does asks me to add to homescreen.
The icons and splashscreens are also working. So far so good.
Now, when I make a change and deploy again. The standalone version doesn't notify me to refresh it.
But when I open the url on the same iphone device in safari, it does asks me to refresh the application.
Also, when the application starts in the mobile browser, it will perform an api request to get data (https://todo.thehangrycoder.now.sh/todo/).
But when I open the standalone application, it looks like it doesn't do the api request.
This is my nsgw config:
https://github.com/fransyozef/todo-challenge/blob/master/frontend/ionic4/todo-app/src/ngsw-config.json
and my manifest:
https://github.com/fransyozef/todo-challenge/blob/master/frontend/ionic4/todo-app/src/manifest.json
The application is deployed on NOW:
https://github.com/fransyozef/todo-challenge/blob/master/frontend/ionic4/todo-app/now.json
Any suggestions I can try? Or is here something I'm missing?
Comment: What do you mean by the standalone version? is that from the homescreen launch vs url in safari?
Comment: are you expecting a notification as a push notification, or do you mean your own custom on swUpdate code?
Comment: yeah "standalone" is the one that is placed from safari using the add to home screen. Then an icon will placed on the homescreen and when tapped, the application starts as "standalone" (https://developer.mozilla.org/en-US/docs/Web/Manifest).

as for notification. Thats the service worker notification (so NOT push). I subscribe to this event : https://github.com/fransyozef/todo-challenge/blob/276e245b89fc90d0d588e38a322dacb69318f090/frontend/ionic4/todo-app/src/app/app.component.ts#L72
Answer: so I got an workaround (at least for the checking the version number).
I do an interval of 20 seconds to invoke the servic worker checkForUpdate() method.
https://github.com/fransyozef/todo-challenge/blob/25047e2add4ae1c570316a7c51ee6811d8b85bbf/frontend/ionic4/todo-app/src/app/app.component.ts#L42
This wil also reload the whole application (and get new api data). But when there is no new update, it will still not get new api data. Just found out that ios just saves the state of the standalone pwa application as is.


Array Program for Java

Question: package myProjects;

import java.util.*;

public class Acm3 {
    static int numberOfFriends;
    static int rowOfChocolate;
    static int columnOfChocolate;
    static String division;
    static Scanner myScanner = new Scanner(System.in);

    public static void main(String[] args) {
        System.out.println("How many friends you want to distribute?");
        numberOfFriends = myScanner.nextInt();
        System.out.println("How do you want to distribute?");
        division = myScanner.next();
        String[] myArray = division.split("\\s");
        int[] myArray1 = new int[numberOfFriends];
        int i;
        for (i = 0; i <= numberOfFriends; i++); {
             myArray1[i] = Integer.parseInt(myArray[i]);
             System.out.print(myArray1[i]);
        }
    }
}

I am trying to transfer String array to Int Array.I have a string array and want to convert each elements into int and finally into array. Anyone who can help?
Thanks in advance
Comment: Possible duplicate of [Converting a String array into an int Array in java](http://stackoverflow.com/questions/6881458/converting-a-string-array-into-an-int-array-in-java)
Comment: Please format your code next (time indent 4 spaces or use the {} button). Also note that you have a stray semicolon after your loop, so the block in curly braces runs only once, not once for each array element as you might expect.
Comment: What error do you get ? On which line ?
Answer: int[] intValues = Arrays.asList(myArray)
                       .stream()
                       .mapToInt(Integer::parseInt)
                       .toArray();

System.out.println(Arrays.toString(intValues));

Array Program for Java


How to make an uploaded video visible on the node page in drupal 7

Question: I have created a CCK named sample with a field name VIDEOS of type file. I have added a video of CCK type Sample. But it only showed a url after the node is saved.
Instead of displaying the file name how to make the video playable on the same page.
I installed jwplayer in libraries. I tried changing display type From Generic to JWplayer

Was trying since past 2 days using video module, VideoJS. But did not succeeded. Please help.
UPDATE
here is the result which i got when working with Video module

It is returning empty data with Play button on it. No video is displayed
UPDATE
A small progress,Tried Flow Player  flv videos are able to play. but mp4 is not playing. It simply shows a black screen.
Answer: Have you tried using the Video module. You can create field type video insted of type file and then upload and render with the player you have configured.
Comment: Try going to Home » Administration » Configuration » Media » Video » Players and select JWplayer for extensions *.mp4, also you can try the flowplayer module - if so select the flowplayer player for *.mp4 and give it a try.
Comment: Yes i tried. i choose field type Video. let me update my post with a snapshot
Comment: I updated my post please have a look
Comment: Ya sure. I will let u know. Many thanks for your reply.
Comment: I have gone through the formats and changed the player to FLV player and then there is a sub category called jwplayer. But Still the video did not appeared. I think something wrong with jwplayer. IS there any perfect Documentation for JwPlayer. I think there is no stable version for Drupal7
Comment: well if you suspect that there is an issue with the jwplayer, try installing flowplayer and play with it.
Comment: hi @ivan Ivanov i tried flow player. Thanks for the module suggestion, It really worked with flv formats. But mp4 is not working It just displays Black Screen with out controls. What might be the reason. Any thing to work with server configuration?
Comment: hmm, is it possible the video you are trying with to be broken, have you tried with several .mp4 files? because it is kinda strange that the .flv works and .mp4 does not, they are very similar formats from player's prespective.
Comment: dunno what are you trying to achieve but you can try this: clean new drupal installation, install video module then flowplayer, then create a video field and upload some .mp4 videos - it should work 100%, Ive done it some many times. If the video doesn't work it is very likely to be an issue with the environment where you are working or an issue with the video file itslef.
Answer: Have you tried setting the display to "rendered File" in the display settings for the content type? If uploading as a file, might work for you.
Comment: There are only two options Video Player and Video Video Thumbnail, nothing like rendered File @Daniel
Comment: OK, got ya, sorry, I think "Rendered File" requires the  [Media Module](https://drupal.org/project/media) in order to show up as an option. Maybe install that, and check if it helps. I have had good luck just utilizing that module for video work. Just create a field with "field type: file", then under display, rendered file. If you don't want to go that route, what does the rendered source code for your blank video look like (that you posted above)?
Comment: Hi @Daniel i tried using Media Module. No luck. Yes i have selected rendered file option, also experimented with view mode to Original but no use. Displaying same old link. Do we need any libraries or plugin to make the video play?


Keyboard and Voice: What is the keyboard playing?

Question: I've "composed" a melody on a keyboard which sounds nice when I play it.
Later I incidentally also wrote lyrics which I am able to "sing" to the melody.
Due to my lack of understanding I am unsure what the keyboard should now be playing.
The voice is already singing the notes of the melody, repeating them on keyboard somehow seems wrong.
I suppose this is a common scenario, but I cannot wrap my head around that. The keyboard playing the melody sounds great, the voice singing the melody sounds great, but together they don't work. It's like having two lead singers competing.
What are my alternatives? I need to write the harmony for the keyboard?
I tried to learn from songs I know (and like) but I never stumbled upon a keyboard repeating the melody. Some songs either plays something very complex (but still accompanying the melody well) or they fall back to some sort of drum pattern or a beat.
I also tried thinning the melody down for the keyboard but leaving notes out and playing only accent notes yields in basically a beat pattern.
Answer: Repeating the cantus firmus (c.f.) would not have been wrong at all, and it is done a lot. However, there are other options of course:

You just play the harmony, i.e., the keyboard plays moreorless only large choords in some proper way, by both hands.
The right hand plays another melody that you think out. There are several possibilities: You can play a descant, i.e., something that covers the c.f. from above. You can play the alto part, i.e., something that helps support the melody's harmony. It can be a completely unrelated melody that just shares the harmony with the c.f., this way you can create a nice polyphony. You can play long notes when the c.f. is complicated and "fill in the gaps" when the c.f. is simple or is resting.
You can even stop playing at all for large parts of the melody, or combine the methods above. For instance: you play only nice and fancy choords for the verses and you play the c.f. in forte for the refrain. Or whatever you wish.

As for the left hand: I most often just play the bass + some parts of the choords. At some places, it's nice to play a piece of some melody in the lower notes as well, it adds some spices to the masterpiece :)
The options are endless. The best thing to do is to try various things, record them, listen to them and use your judgement to see which ones do you like.


How to Unit test a Directive Angular 2

Question: I have this drop down Directive i want to unit test but i am getting this error in console.I am confused as how to test this Directive standalone
import {Directive, HostListener, HostBinding} from '@angular/core';

  @Directive({
    selector: '[appDropdown]'
  })
  export class DropdownDirective {

    private isOpen:boolean = false;

    @HostBinding('class.open') get opened(){
      return this.isOpen;
    }
    constructor() { }

    @HostListener('click')open(){
      this.isOpen = true;
    }

    @HostListener('mouseleave')close(){
      this.isOpen = false;
    }
  }

Spec file
import {
  it,
  describe,
  expect,
  inject,
  injectAsync,
  beforeEach,
  beforeEachProviders,
  TestComponentBuilder
} from 'angular2/testing';

import {Component} from '@angular/core';

describe('DropDown Diretive', () => {

  let mouseenter;

  beforeEachProviders(() => [
    TestComponentBuilder
  ]);

  beforeEach(() => {
    mouseenter = new MouseEvent('click', {});
  });

  it('Style added', injectAsync([TestComponentBuilder], (tcb: TestComponentBuilder) => {
    let template = '

';
    return tcb.overrideTemplate(TestComponent, template)
      .createAsync(TestComponent)
      .then((fixture) => {
        let div = fixture.nativeElement.querySelector('div');

        fixture.detectChanges();

        expect(div.style.className).toEqual('open');

      });
}));

  @Component({
    selector: 'container'
  })

  class TestComponent { }

I am using Angular 4 Rc 3 as of now Please help new to writing test cases in Angular
Error in Console
Failed: No provider for DirectiveResolver!

Answer: beforeEachProviders is deprecated since 2.0.0 RC4, check this question: How to fix beforeEachProviders (deprecated on RC4)
DropdownDirective needs to be declared in order to be able to used in component templates. (And its selector should be app-dropdown.)
Check the official documentation for more info. https://angular.io/docs/ts/latest/guide/testing.html. I would probably consider using the TestBed in this case.

How to Unit test a Directive Angular 2


How to make create multiple modals on one page with React-Responsive-Modal

Question: I have recently installed the React-Responsive-Modal library and it does work however, I would like to create multiple modals on one page. with each button, I click opening up its own unique modal.
I tried naming the states differently and passing those different states as props to the buttons and modals but it still doesn't seem to work.
here is the code:
 const [open, setOpen] = useState(false);
 const [open2, setOpen2] = useState(false);

  const onOpenModal = () => setOpen(true);
  const onCloseModal = () => setOpen(false);

  const onOpenModal2 = () => setOpen(true);
  const onCloseModal2 = () => setOpen(false);

 return (

  <button onClick={onOpenModal}>Open modal</button>
  <Modal open={open} onClose={onCloseModal} center>

Simple centered modal

      Lorem ipsum dolor sit amet,   Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam
      pulvinar risus non risus hendrerit venenatis. Pellentesque sit amet
      hendrerit risus, sed porttitor quam.

  </Modal>

  <button onClick={onOpenModal2}>Open modal</button>
  <Modal open={open2} onClose={onCloseModal2} center>

Simple centered modal

      Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam
      pulvinar risus non risus hendrerit venenatis. Pellentesque sit amet
      hendrerit risus, sed porttitor quam.

  </Modal>

);
Also, when I use this method, only the first modal puts up even if I click the second button
Answer: You have an error in this line
const onOpenModal2 = () => setOpen(true);

It should be
const onOpenModal2 = () => setOpen2(true);

So that's why the second one didn't open.
Also you have the same error for onCloseModal2


Is there a difference between LASSO regularisation and LASSO penalisation?

Question: I've seen the terms LASSO regularisation and LASSO penalisation  used interchangeably? Is this correct, are they the same thing or what are the differences?
Thanks!
Answer: In mathematics, statistics and physics, regularisation is the process of adding information in order to make an ill-posed problem soluble and well-behaved or to force a problem to exhibit some property known to be satisfied by suitable solutions; it particularly applies to objective functions in ill-posed optimisation problems.
With regard to the lasso, the added information pertains to the $\ell_1-$norm of some parameter vector and, due to convexity and the geometry of the extreme points of a polyhedron specified by a constraint of the form $$ \sum_{j=1}^{p} \lvert \beta_j \rvert \leq t,$$ with $t>0$ large enough, this comes down to specifying that some of the parameters $\beta_j$ vanish.
Whilst having the same meaning, the term penalisation is, to my knowledge, mostly used by statisticians and data scientists ; it has the merit of highlighting the fact that one regularises a problem by penalising solutions that stray from a certain desirable behaviour (e.g. sparsity).
Comment: @Henry I don't think the answer contradicts this in any way. Effectively if we deem "sparse" solutions to be more "regular" in the sense described above, then a 'penaliser' relying on the l1 norm is an effective 'regularizer' (i.e. it is an effective means of achieving more 'regular' solutions), w.r.t. this particular definition of what it means to be 'regular'.
Comment: Ah so *that's* what regularisation means. Thank you.
Comment: I do not think Lasso as such specifies sparsity i.e. that some of the parameters vanish; instead this is common effect of the $\mathcal{l}_1$-norm which is not shared with the  $\mathcal{l}_2$-norm


Jenks Natural Breaks in Python: How to find the optimum number of breaks?

Question: I found this Python implementation of the Jenks Natural Breaks algorithm and I could make it run on my Windows 7 machine. It is pretty fast and it finds the breaks in few time, considering the size of my geodata. Before using this clustering algorithm for my data, I was using sklearn.clustering.KMeans (here) algorithm. The problem I had with KMeans, was finding the optimum K value parameter, but I "solved" it launching the algorithm for different K values and using sklearn.metrics.silhouette_score (here) to find the best K.
My question is: if I tell the Natural Breaks algorithm to find 5 classes (that would be the K), how can I be sure that this is the number of classes that best match my data? How to validate that I am choosing the best number of breaks?
Thanks!
Comment: So that we can objectively determine what "best" means, could you explain the sense in which the classes "match" data?  (Or, really, how you would quantify any degree of mis-matching.)
Comment: Using Silhouette with Jenks should be comparable to using it with kmeans. It's a heuristic and you shouldn't trust it blindly. IMHO the best is to *visualize* your rrsults.
Comment: Whuber: Best, using Silhouette, would mean the number of classes that make the index be closer to 1, according to the definition on sklearn site: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html#sklearn.metrics.silhouette_score

Anony-Mousse: I cannot visualize 20+ variables, prepare maps for that and expect my brain not to mess up with the number of classes. I need to rely on an index that says, "for variable X, the best you can do is to use Y classes". Moreover I need to re-run the analysis several times, the viz approach is slow unfortunately...
Comment: from jenks import jenks: gives following error Traceback (most recent call last): File "", line 1, in
ImportError: cannot import name jenks
Answer: Jenks Natural Breaks works by optimizing the Goodness of Variance Fit, a value from 0 to 1 where 0 = No Fit and 1 = Perfect Fit.  The key in selecting the number of classes is to find a balance between detecting differences and overfitting your data.  To determine the optimum number of classes, I suggest you use a threshold GVF value you desire and use the number of classes that satisfies this value first.
Below is a function to calculate the Goodness of Variance Fit given an array of values to classify and the number of classes selected:
from jenks import jenks
import numpy as np
def goodness_of_variance_fit(array, classes):
    # get the break points
    classes = jenks(array, classes)

    # do the actual classification
    classified = np.array([classify(i, classes) for i in array])

    # max value of zones
    maxz = max(classified)

    # nested list of zone indices
    zone_indices = [[idx for idx, val in enumerate(classified) if zone + 1 == val] for zone in range(maxz)]

    # sum of squared deviations from array mean
    sdam = np.sum((array - array.mean()) ** 2)

    # sorted polygon stats
    array_sort = [np.array([array[index] for index in zone]) for zone in zone_indices]

    # sum of squared deviations of class means
    sdcm = sum([np.sum((classified - classified.mean()) ** 2) for classified in array_sort])

    # goodness of variance fit
    gvf = (sdam - sdcm) / sdam

    return gvf

def classify(value, breaks):
    for i in range(1, len(breaks)):
        if value < breaks[i]:
            return i
    return len(breaks) - 1

For example, consider you decide the GVF should be at least .8, then you could increment the number of classes until the GVF is satisfied:
gvf = 0.0
nclasses = 2
while gvf < .8:
    gvf = goodness_of_variance_fit(array, nclasses)
    nclasses += 1

Comment: Hi, useful post. Can I know how to interpret Gvf = 87% for n_classes = 4? Does it mean with 4 classes, we are able to seperate the data (into bins) correctly by 87%?


How to generate url with language code in drupal7?

Question: Is there any drupal function to return the url with currently used language.
Like: example.com/en
Answer: You can get the URL by using following code
<code>global $language ;
$lang_name = $language->language ;

$url_current = $GLOBALS['base_url']/$lang_name;
</code>
Comment: Not quite...you're dividing strings at the moment
Comment: Following links will be helpful for you  https://api.drupal.org/api/drupal/developer%21globals.php/global/language/7 https://api.drupal.org/api/drupal/includes%21path.inc/function/drupal_get_path_alias/7
Comment: It means there is not a straight drupal function?
Comment: I've never found a straight function for this. If you find it please share.


How can I make logarithmic function first grow faster and then slow down faster?

Question: How can I manipulate a logarithmic function to both grow faster as well as slow down faster?
I need to map a set of values to a range of real numbers between $0$ and $1$ (representing percentages by which to manipulate something). I am using this formula for it ($max > 0$, otherwise the formula would be more complicated):
$$f(x) = \frac {\log (1+x)}{\log (1+max)}$$
lowering the base of the log-function in the dividend makes $f(x)$ grow faster, but then $f(max) = 1$ is not true anymore.
What I am trying to accomplish is that smaller values in my set get mapped to higher percentage values, while higher values get only a slightly higher percentage-increase
Comment: Use $\frac{1}{x}$ ?
Comment: Could you explain, please, why does the lowering of the base of the log-function change $f(max)$? As far as I can see, $\frac{\log_n {(1+max)}}{\log_n {(1+max)}} = 1$ for any $n \in (0; 1) \cup (1; + \infty)$.
Comment: hyperbolic tangent
Comment: When you describe a function only qualitatively, it's hard to find. A differential equation cuts down the options considerably. Given some value of the function then fixes it.
Comment: You can construct such function in a following way: choose any unbounded function $g: [0, +\infty) \rightarrow [0, +\infty)$, such that $g(0) = 0$ and $g(\frac{max}{x} - 1)$ is integrable on $[0, max]$. Then you can define $f(x) = \frac{\int_{[0, x]} g(\frac{max}{t} - 1)d\mu}{\int_{[0, max]} g(\frac{max}{t} - 1)d\mu}$. This function will satisfy the condition "that smaller values in my set get mapped to higher percentage values, while higher values get only a slightly higher percentage-increase", but if you want something more specific, please, do not hesitate to inform us about it.
Comment: @YaniorWeg Thank you for your reply. Concerning your first comment: it changes it only if I change the base in the dividend exclusively. If I change the base in both dividend and divisor the outcome is the same as with natural logarithm.
Answer: You could add increase the additive constant $1$ in both logarithms or, equivalently, multiply both $x$ and $\max$ by some factor.
Comment: Multiplying worked flawlessly, thank you very much!


Finding the integral of $(x^2+4x)/\sqrt{x^2+2x+2}$

Question: Can somebody explain me how to calculate this integral?
$$\int \frac{\left(x^2+4x\right)}{\sqrt{x^2+2x+2}}dx$$
Comment: Try Euler's first substitution:$$\sqrt{x^2+2x+2}=x+t$$
Answer: HINT:
As $\displaystyle\frac{d(x^2+2x+2)}{dx}=2x+2,$
write $$x^2+4x=(x^2+2x+2)+(2x+2)-4$$
Use  $\#1,\#8$ of this
Comment: @amWhy, We have $$\int\sqrt{(x+1)^2+1^2}dx+\int\frac{d(x^2+2x+2)}{\sqrt{x^2+2x+2}}-4\int\frac{dx}{\sqrt{(x+1)^2+1^2}}$$
Answer: Let $x+1=\tan\theta$, so $x=\tan\theta-1$, $dx=\sec^{2}\theta d\theta$, and $\sqrt{x^2+2x+2}=\sqrt{(x+1)^2+1}=\sec\theta$.
Then $\displaystyle\int\frac{x^2+4x}{\sqrt{x^2+2x+2}}dx=\int\frac{(\tan\theta-1)^2+4(\tan\theta-1)}{\sec\theta}\sec^{2}\theta d\theta$
$\displaystyle=\int(\tan^{2}\theta+2\tan\theta-3)\sec\theta d\theta=\int(\sec^{2}\theta-1+2\tan\theta-3)\sec\theta d\theta$
$=\displaystyle\int(\sec^{3}\theta-4\sec\theta+2\sec\theta\tan\theta)\; d\theta$
$=\displaystyle\frac{1}{2}(\sec\theta\tan\theta+\ln|\sec\theta+\tan\theta|)-4\ln|\sec\theta+\tan\theta|+2\sec\theta+C$
$=\displaystyle\frac{1}{2}(x+1)\sqrt{x^2+2x+2}-\frac{7}{2}\ln\left(\sqrt{x^2+2x+2}+x+1\right)+2\sqrt{x^2+2x+2}+C$.


Find all functions such that $af(x-1)+bf(1-x)=cx$, where $a$,$b$ and $c$ are real and $a^2$ is not equal to $b^2$

Question: I tried to find $f(0)$, by taking $x=1$,
$$f(0)=\frac{c}{a+b},$$
Then i deduced that $f(x)$ is linear,
Taking $x=(u+1), af(u)+bf(-u)=(u+1)c$
Similarly, $af(-u)+bf(u)=f(1-u)c$, adding them
$2c/(a+b)=f(u)+f(-u)$
Now take u+1,
$f(u+1)+f(-u-1)=2c/(a+b)$
Equating this with the first,
$f(u+1)-f(u)=f(-u)-f(-u-1)=2c/(a+b)$
this lead me to the conclusion that f(x) is linear,
Comment: If you have shown that $f(x)$ is linear then you know that $f(x)=Mx+B$ for two constants $M,B$.  Now you can just solve for $M,B$.
Comment: But note:  the function $F(x)=x^3$ satisfies $F(x)+F(-x)=0=2F(0)$ so I am not convinced you have shown that your function $f(x)$ is linear.
Comment: $f(x)=\frac c {a+b}$ is a non-zero solution.
Comment: @lulu Notice the condition $a^2\ne b^2$
Comment: @Qurultay  I didn't say that my $F(x)=x^3$ satisfied the OP's functional equation, I am just pointing out that knowing that $F(x)+F(-x)=2F(0)$ does not imply that $F(x)$ is linear, which appears to be a claim that the OP is making.
Comment: @user728189 From $af(u)+bf(-u)=(u+1)c$ you cannot conclude that $f$ is linear.
Comment: If $x=u+1$ then the functional equation gets us $af(u)+bf(-u)=c$.  I don't understand where the $u+1$ term on the right comes from (in your comment).
Comment: Taking $x=(u+1), af(u)+bf(-u)=(u+1)c, Also af(-u)+bf(u)=f(1-u)c, adding them , 2c/(a+b)=f(u)+f(-u), Now take u+1, f(u+1)+f(-u-1)=2c/(a+b), equating this with the first, f(u+1)-f(u)=f(-u)-f(-u-1)=2c/(a+b), this lead me to the conclusion that f(x) is linear
Comment: Did you write your functional equation correctly?
Comment: It is $cx$ on the right side,
Comment: Post edit:  most of the comments are now irrelevant.  But it is still true that knowing that a function $F(x)$ satisfies $F(x)+F(-x)=2F(0)$ does not imply that $F(x)$ is linear.
Comment: Check my previous long comment,
Comment: No.  Please edit your post to indicate your reasoning.   Don't leave detailed computations for the comments.
Comment: I updated my post
Comment: Please edit your post for clarity.  You don't mean $af(-u)+bf(u)=f(1-u)c$ for example.  And once again:  knowing that $F(x)+F(-x)=2F(0)$ does not imply linearity.
Comment: af(x-1)+b(1-x)=cx is the functional equation, i substituted x=u+1, to deduce that f(x) is linear
Answer: Let $x-1=t.$
$$af(x-1)+bf(1-x)=cx \implies af(t)+bf(-t)=c(t+1)~~~(1).$$
Let $t\rightarrow -t$; we get
$$af(-t)+bf(t)=c(-t+1)~~~(2)$$
Let $f(t)=X, f(-t)=Y$, then we have $aX+bY=ct+c, aY+bX=-ct+c$
Solve for $X$
$$X=f(t)=c[\frac{1}{a+b}+\frac{t}{a-b}]\implies f(x-1)=c[\frac{1}{a+b}+\frac{(x-1)}{a-b}]$$
Finally $$f(x)=c[\frac{1}{a+b}+\frac{x}{a-b}]$$
Comment: Why do you have those brackets?


Only group some rows on Ag-Grid

Question: I'm looking at row grouping with Ag-Grid and have got it working in my project (grouping art works by master edition, with child elements being editions). The problem is that some art works are not master editions, nor editions. They're just their own work without being a part of a set.
Right now Ag-Grid creates a group for every single art work. So, those of type "normal" have a "normal" group, etc. I only want grouping behaviour on works of type "master edition". Is there a proper way to do this?
My current idea was to create a custom cell renderer that only renders HTML if params.node.group = true. This feels pretty hacky though so I was wondering if there was a better solution.
Answer: From my understanding of your problem, a tree view could help you:
https://www.ag-grid.com/javascript-grid-tree-data/
You could accomplish that asymetric grouping, by creating a tree path dynamically, in the getDataPath callback in gridOptions.
getDataPath: function(data) {
    return data.type === "master edition" ? ["master edition", data.name]: [data.name];
}

Plunker: https://next.plnkr.co/edit/3jdzvQ1QJmF7xJy1
Comment: Ended up using the master/detail feature instead and embedding a new grid. Works well and I get to have more flexibility with further sorting/filtering of child elements which is great. I'll put you down as the answer though because a tree view would have probably worked too. Thanks!

Only group some rows on Ag-Grid


Como simular um clique no Linkedin com selenium. Problema: id dinâmico / usa ember

Question: Bom dia, estou tentando clicar no botão 'Exibir mais' no campo das competências mas sem êxito algum.
O xpath é dinâmico, e por class também não funciona.
O botão parece estar escondida de algum modo.
Tentei por xpath:
driver.find_element_by_xpath('//*[@id="ember2605"]/button/span[2]/li-icon/svg/path').click()

Obrigado pela ajuda
Answer: Olá, aparentemente o Linkedin te força a usar o Xpath, já que o Selenium (hoje!,24/04/18) não te permite achar por classes compostas.
portanto você vai precisar procurar por:
driver.find_elements_by_xpath("//*[contains(@id,'ember']))

procure todos os elementos que cuja xpath comece com 'ember'. Observe que o uso do Xpath continua o mesmo, então você poderá usar como sempre: adicionando mais mais caminhos a frente.
O que vai se tornar problemático pq vai trazer coisas inúteis junto, mas é o que resolve.
Espero ter ajudado.
Att, Willy do passado.


Posição ScrollTop()

Question: Possuo uma função JavaScript que dá refresh na minha página a cada 15 minutos e estou tentando pegar a última posição da barra de rolagem antes do refresh com o comando jQuery.
$(window).scrollTop();

Função refresh:
var $myVar = setInterval(myTimer, 900000);

        function myTimer() {
            console.log(scrollPosition);
            loadHs();

        }

Sempre que minha página é recarregada ele já me retorna o valor zero, então gostaria de resgatar o último valor a medida que vou rolando o scroll e setar esse valor resgatado depois do refresh para que ele retorne a posição anterior.
Comment: Essa linha `localStorage.setItem("timer", true)` é para quê?
Comment: @Sam esta linha já não faz mais parte do meu código

A pergunta também já foi atualizada.
Comment: Tu vai precisar colocar o valor atual do scroll em um cache pra poder resgatar depois do refresh da página.
Answer: Salve a posição num sessionStorage (cookie temporário que será apagado ao sair da página) e verifique se existe e é diferente de 0 quando a página for recarregada. Então use o método window.scrollTo(x,y) para mover a tela para a posição salva no cookie.
No caso, você só vai alterar o valor do eixo y (vertical), deixando o x em 0:
document.addEventListener("DOMContentLoaded", function(){ // aguarda o carregamento do dom
   var pos = sessionStorage.getItem("posicao"); // atribui o cookie a uma variável
   // se a variável tiver valor, move para a posição salva no cookie
   if(+pos) window.scrollTo(0, pos);
});

var $myVar = setInterval(myTimer, 900000);

function myTimer() {
   var scrollPosition = $(window).scrollTop(); // pega a posição do scroll
   sessionStorage.setItem("posicao", scrollPosition); // salva no cookie
   loadHs();
}

Em vez de sessionStorage, pode usar localStorage, a sintaxe é a mesma. A diferença é que o segundo gera um cookie permanente.
Comment: Obrigado pela ajuda, porém no meu caso ainda continuo recebendo a posição zero
Comment: Tente assim na linha do scrollTo: `if(+pos) setTimeout(function(){ window.scrollTo(0, pos); }, 10);`
Comment: continua retornando zero

acredito que possa está sendo pelo fato de que quando acesso a pagina pela primeira vez, a barra de rolagem inicia na posição 0 e ja salva esse valor.

se isso estiver acontecendo teria que ter um método que ficasse verificando a posição  ?
Comment: Só salva o valor da posição quando chama a função myTimer(). Como vc está recarregando a página?
Comment: Sim, está correto agora estou conseguindo ver a ultima posição quando uso console.log(), porém a barra não volta para a posição anterior.
Comment: vc colocou o setTimeout como falei?
Comment: tava esquecendo isso, fiz os ajustes e esta funcionando corretamente

muito obrigado pela ajuda.

Posição ScrollTop()


Effect of nickel cover on Telecaster bridge pickup?

Question: Traditionally telecaster guitars have the neck pick-up covered in a nickel/chrome cover, and the bridge pickup left with exposed pole pieces.
What is the origin/logic of the nickel cover?
What effect would it have if the same pick-up was used with the cover removed?
Why don't we have a nickel cover on the bridge? How would that affect the sound? (I've never even seen an aftermarket one, do they exist?)
Answer: I have read much about the Telecaster, but I still don't know what Leo Fender was thinking. I know that Seth Lover was thinking the same thing, as PAF humbuckers had covers too. It was only into the 70s when you started seeing pickups with their covers removed. Even Strat pickups are covered, albeit with plastic.
In part, the nickel cover was to make the guitar look better. The bridge pickup isn't covered, but the original design had a chromed cover for the whole bridge. Because guitarists wanted to be able to do things like palm muting, they were generally left off -- Albert Collins is the only guitarists I can think of who used one -- and were used as ashtrays enough that they're called "ashtray bridge covers".
Part of the logic could be the same as the piece of tape on the treble side of Steve Vai's EVO guitar, that a string could get caught in the bobbin and might hurt the windings, but that seems unlikely because it was only in the 80s with the Floyd Rose tremolo that an errant high E could even get there.
Some have removed the cover, and some pickup makers use covers with cut-off tops. The reported effect is brighter tone and more clarity.
Comment: I forgot about the old "ashtray" cover :)


Probability question in Venn diagram

Question: Look at this PDF file's question No. 7( Under National Cyber Olympiad's subheading Logical Reasoning, it's in page 2)
As the question can't be understood without the diagram, I don't think it is worth effort to add the question text, as it would seem incomplete.
So, the answer is said to be 5400. That's (17/50)*10000.   Okay, but why? The question says the least number of people who read two newspapers. The people who read two news papers(different) are 8, 12, 5. And ones who read all three are only 2.
I guessed that 2 is deducted from 8, 12, 5 and the results are then added. But why is 2 subtracted from them? The two who read three also read at least two newspapers, don't they? So shouldn't it be added just once to sum of 8, 12, 5?
Please explain, 9th grader here, if it matters...
Answer: The number of people who read at least two newspapers is $8+12+5+2=27$.  Then $\frac {27}{50}\cdot 10000=5400$

Probability question in Venn diagram


Unable to fetch the data with help of QueryBuilders.termQuery

Question: I am a newcomer to elastic search I was trying to work with high-level Rest Client
I am able to work with CRUD operations, With Search functionality, I got stuck up.
My objective to bring all the data based on the book id start with E106 search criteria
http://localhost:5918/book-elastic/books/book/E106
I added the part of the code below
I am able to get all the data using
QueryBuilders.matchAllQuery()
But I couldn't get a specific field value
QueryBuilders.termQuery("_id",bookId)
I have also shared the screenshot of the both results
Can somebody help me out on the query?
Kindly revert in case of any further information needed.
Thanks in Advance
<code>public Page<BookEntity> findByBookId(String bookId, Pageable pageable) throws IOException{

        SearchRequest searchRequest = new SearchRequest(INDEX);
        searchRequest.types(TYPE);

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.fetchSource(false);
        //searchSourceBuilder.fetchSource(null, new String[]{"excludedProperty"});

        /*MatchQueryBuilder matchQueryBuilder =  new MatchQueryBuilder("id",bookId);
        matchQueryBuilder.fuzziness(Fuzziness.AUTO);
        matchQueryBuilder.prefixLength(3);
        matchQueryBuilder.maxExpansions(7);*/

        searchSourceBuilder.from((int)pageable.getOffset());
        searchSourceBuilder.size(pageable.getPageSize());

        //searchSourceBuilder.query(matchQueryBuilder);
        searchSourceBuilder.query(QueryBuilders.matchAllQuery());
        //searchSourceBuilder.query(QueryBuilders.termQuery("_id",bookId));
        searchRequest.source(searchSourceBuilder);

        SearchResponse searchResponse = restHighLevelClient.search(searchRequest,RequestOptions.DEFAULT);
        SearchHits hits = searchResponse.getHits();
        SearchHit[] objectHits = hits.getHits();
        for (SearchHit searchHit : objectHits) {
            System.out.println("***************************");
            System.out.println("Search Hit :: "+searchHit);
            System.out.println("***************************");
        }

        return null;

    }
</code>
Result Screenshot

matchAllQuery

Input : {"from":0,"size":20,"query":{"match_all":{"boost":1.0}},"_source":false}

Response
Response Value: {"took":5,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":3,"max_score":1.0,"hits":[{"_index":"bookdata","_type":"books","_id":"E106401","_score":1.0},{"_index":"bookdata","_type":"books","_id":"E106403","_score":1.0},{"_index":"bookdata","_type":"books","_id":"E10640","_score":1.0}]}}

Input Values: QueryBuilders.termQuery("_id",bookId)

{"from":0,"size":20,"query":{"term":{"_id":{"value":"E106","boost":1.0}}},"_source":false}

Response
The Response is Null
Answer: matchPhrasePrefixQuery from QueryBuilders solved my issues


Link to static HTML page from within Svelte component

Question: How can I move from a Svelte component to an HTML file using window.location.href?
I tried various combinations, but the HTML can't be found. Do I have to set up routing?
Folder Structure:
<code>project
├── example.html
├── svelteProject
    ├── src
        ├── App.svelte
</code>
Comment: You need routing, juste use svelte-kit.
Comment: Missing context, e.g. how you build and serve the application.
Answer: The most ideal thing would be to use Sveltekit. You could also try to create a full page iframe, not sure how you'd reference your HTML file though.


Is there a name for the propositional tautology (and it's associated rule) $Q\Rightarrow(P\Rightarrow Q)$?

Question: I have the tautology $Q\Rightarrow(P\Rightarrow Q)$. I can prove this intuitionistically:
Rule
  Conclusion
    Q=>(P=>Q)
  Proof
    Suppose
      Q
    Hence
      Suppose
        P
      Hence
        Q
      P=>Q
    Q=>(P=>Q)

This gives the corresponding inference rule:
$$
\frac{Q}{P\Rightarrow Q}\quad\text{[VacuousImplicationIntroduction]}
$$
The proof, which relies on the tautology, is:
Rule (VacuousImplicationIntroduction)
  Premise
    Q
  Conclusion
    P=>Q
  Proof
    Q=>(P=>Q)
    Q
    P=>Q by ModusPonens

I've called this rule 'vacuous implication introduction', which is a pretty poor name. Is there a standard or at least a better one?
Comment: By the way, I've made some progress on my implementation of Propositional Logic since the last time you kindly helped me out: https://github.com/jecs-imperial/Propositional-Logic
Comment: It is basically [Addition](https://en.wikipedia.org/wiki/Disjunction_introduction) : $P \to (Q \lor P)$.
Comment: Let me try to work out why you've written that...
Comment: Presumably the two rules are equivalent...
Comment: In classical logic: YES. In Natural Deduction (Intuitionistically) $\dfrac {P} {Q \to P}$ is simply $\to$-intro.
Comment: The tautology $Q \to (P \to Q)$ was named *Simp* by W&R is their *Principia Mathematica* [2.02]; they proved it from their *Add* axiom: $Q \to (P \lor Q)$ [1.03] with a simple substitution: $\lnot P$ in place of $P$ **and** the fact that in *PM* system they have the definitional abbreviation: $P \to Q$ in place of $\lnot P \lor Q$, that obviously made the system "classical".
Comment: Okay, I cheated and looked at your answers before working out the equivalence myself. My excuse is you hardly gave me any time.
Comment: Well it appears I have one up on Whitehead and Russell then because I haven't had to employ addition or had to equate logical consequence with the material conditional...!
Answer: Different authors have called the formula (not the rule) different names.
I've seen 'positive paradox', 'simplification', and 'Simp'.
I prefer to call the formula [p => (q => p)] or CpCqp in Polish notation, recursive variable prefixing.  Why?  Because under the the rule of detachment (also called modus ponens) it allows you to prefix any formula with a variable.  And since you can do that for any formula, you can recursively prefix each new formula with a variable in a proof sequence with prefixed formulas.  Or perhaps better, it allows to demonstrate metalogically that where 't' indicates a theorem or an axiom, that
$\vdash$Ca$_1$Ca$_2$...Ca$_n$t
CpCqp may look dull, but taking it as the only axiom under the rule of condensed detachment, which I'll call {CpCqp}, any theorem t which replaces CpCqp as the sole axiom has the same consequences as {CpCqp}.
In other words, where $\vdash$t in {CpCqp},
{t} |-| {CpCqp}
where |-| indicates deductive equivalence.
Not many non-trivial systems have that sort of property.
Comment: An interesting comment! I'm not sure I like 'positive paradox' either, and 'simplification' is definitely out, it's too generic. To be honest it's the rule that really needs a name. I take your point about recursion, you can keep applying the rule, however I think the word 'recursion' is a bit heavy, although I know that's a vague excuse. The word 'prefix' has possibilities, though. I'm going to go with 'PrefixedImplicationIntroduction' unless someone comes up with something better. I'm genuinely surprised the rule doesn't have a name.
Comment: @JamesSmith Here's an example of someone using the term 'positive paradox': https://math.vanderbilt.edu/schectex/papers/mingles.pdf
Comment: @JamesSmith "Simp" is used by Arthur Prior in the second appendix of his textbook "Formal Logic".  It also gets called "K" such as here: http://web.ics.purdue.edu/~dulrich/BCK-page.htm.  Prior had a paper named "Logicians at Play; or Syll, Simp, and Hilbert."
Comment: Also, given that an alternative to 'implication introduction' is 'conditional proof' you could have 'unconditional proof' or, that word again, 'vacuous proof'.
Comment: The use of 'simplication' is out because it's used for conjunction elimination. And 'positive paradox' I can't find anywhere, although I like it.
Comment: Ah, brilliant! Cheers Doug. In that case I'll add it as the main label for the result.
Comment: Cool, I've adjusted [Positive paradox](https://github.com/jecs-imperial/Propositional-Logic/blob/master/Minimal/Derived%20rules/Positive%20paradox.fls) accordingly.
Comment: Cheers, Doug. I'm a fan of Arthur Prior so I've put Simp back. I once wrote a paper on temporal logic in a week with no previous knowledge of the subject in order to demonstrate to some idiots at Imperial that I wasn't thick. There is a quote from Prior at the top, which I thought fitting at the time. https://arxiv.org/pdf/1303.7459v4.pdf


How to generate different image styles at upload?

Question: I'm using Drupal 8.
Is there a hook for generating image from image style or better one module?
For example. I upload an image. When image is displayed for the first time, Drupal generates image style. Can I hook to that event somehow, at best when uploading the image?
Comment: I upload the image when i create one content with upload function. No derivatives are being created by default, only the default image in full size is available on filesystem. As i have a decoupled installation i need to have the image derivates available when image is added to node. Nothing specially in my installation, i can reproduce this with latest drupal 8 version fresh installed.
Answer: Ah yes, you are right.

Image derivatives will be automatically generated once requested and not-yet existing.

That's a quote from my own answer to a similar question. I guess you could try to hook into the upload somehow like Image Style Warmer does it, or simply try that module out yourself.

Synopsis
The Image Style Warmer module provides options to create image styles
  during upload or via queue worker. So configured image derivates
  already exists when they are requested.
Answer: You don't need to write code to generate image styles in Drupal.
This functionality is provided by the Image module in Drupal core.
From the documentation:

To create or change image styles:

Choose Administration > Configuration > Media > Image styles.
Add or select a style to edit its settings.
When configuring styles you can add effects: crop, scale, resize, rotate, and desaturate (other contributed modules provide additional
  effects). For example, by combining effects as crop, scale, and
  desaturate, you can create square, grayscale thumbnails.
Save the settings.

To start using the defined styles they must be assigned to the display
  of a content type.
For each display (e.g. "default" or "teaser"), a different style can
  chosen.
To assign a style:

Choose Administer > Structure > Content types > Manage display.
Choose the display (e.g. "default" or "teaser") via the tab buttons.
Click the gear wheel button to choose an image style; you are also
  able to link the image to the node.
Save the settings.
Check the content that contains the image style in question to see
  your new styles at work.

By design, properly configured image styles should be generated when the image is first uploaded, not when it is displayed for the first time. (NOTE: I was wrong about this. See @leymannx comments below and answer above!)
Additionally, whenever you change an image style configuration, the module automatically refreshes all created images by flushing the imagecache.
Also from the documentation:

The Drupal 6 imagecache module was introduced into the Drupal 7 core. Since then, the option to 'flush' the image presets has changed. In the Drupal 7 UI, this 'Flush' image styles preset option was removed, but now, just resaving your image style will flush the image cache.

If you need to flush all image styles (eg. after adjusting the JPEG quality), you can execute drush image-flush, use the Image style flush module, or run the following code (for instance using Devel):
// Flush all image styles in Drupal 7.
foreach (image_styles() as $style) {
  image_style_flush($style);
}

Comment: I added a note describing how to regenerate image styles with code. Perhaps you can edit your question to explain what you are trying to accomplish.
Comment: Sorry @hotwebmatter, I maybe have set a red herring in my original comment under the question. Image derivatives are really only being created when requested, not during upload. As this would result in a lot of unused files otherwise (imagine your site as 100 image styles, but this one image will only be displayed in one style, would be 99 unused files then).
Comment: I want to have the image style being generated when i add the image, not when it get loaded first time (the node). The documentation applies only for this case, but not how to create image styles at content creation (uploading image).
Comment: @John I updated my answer again to indicate that the desired behavior (image styles generated on upload, *not* when the node is first viewed) is *already* the default in Drupal. If you are certain that your situation is different, please provide more details. Otherwise, be sure to accept my answer by clicking the green check mark, to prevent it from beiing forever marked as "unanswered" and cluttering up the forum.
Comment: This is exactly which i can´t confirm. With default new drupal 8 installation with latest version it don´t create image styles when node is created, but only when it is viewed. I use a decoupled installation where i never open open any nodes via HTTP. When image is uploaded to the node nothing is being generated on the filesystem except the initial image but not any image styles which i have created.


What's the trampoline sound in some solo piano recordings?

Question: I listen to classical music while I study. I've noticed a couple songs that have a kind of springing, trampoline-like sound in them. What is that? Is it an artifact of a specific play style or some kind of percussive element?
Examples with Spotify Links:

"Prends soin de toi" by artist Falerina Paquet
"Rana ura, zlata ura" by artist Valbona Ceferin
Comment: These are copyrighted and I couldn't find any samples on YouTube, etc. Open to suggestions, though.
Comment: https://www.youtube.com/watch?v=Xryt-8XeLQU , its a different track, but it has the same noises as the others
Answer: To me it sounds like the sustain pedal of the piano being pressed (the sounds usually appears at the beginning of a new chord). If that is the case, it is strangely loud. Because it is in recordings from both artists, its either an intentional choice to keep them (maybe even amplify them?) or maybe both recorded in the same studio with the same old piano.
On another note, and I might be wrong here / or overly pedantic, but these are often preferred to be called pieces (not songs). Also these are borderline classical music, maybe modern minimalism, but they follow rather the structure of pop compositions (chord progressions and repetition). Happy if anybody wants to correct me on this.
Comment: IMO not pedantic. Songs are sung... by a voice.
Comment: Could be an artistic choice, as in some recordings of cello or guitar for instance, the left hand action is very audible.
Your hypothesis about the same piano in the same studio seems legit to me, as both have IMO the same sound


scilab is not showing any graphical plot

Question: I am new to scilab. I am trying to do a simple 2d plot but unfortunately it is not plotting anything.
When I open scilab and run plot, it shows a window named 'graphic window number 0' but shows no plot.
when I try to run scilab-bin, it shows following error:
scilab-bin: error while loading shared libraries: libjava.so: cannot open shared object file: No such file or directory
I used sudo apt install scilab.
Should I install more packages?
Comment: Please give the output (on the terminal) when Scilab is launched and when the plot2d command is issued.
Comment: with scilab-cli:
Undefined variable: plot
with scilab-adv-cli:
a graphic window appears and no plot. The error is too long, cant paste here.
Comment: Include the error message (at least the first 10 lines) in your  question above, between two ~~~~ lines
Answer:
I used sudo apt install scilab.

If you installed Scilab from (Ubuntu or derivatives) repository it will probably not work. The package is somehow broken (since I can think).
Quick and dirty: Go to scilab.org, download the tar.gz, decompress it and run the script called "scilab" in the bin-directory.
Answer: For me the graphical plot utility doesn't work, either I install Scilab from Flathub or decompress the tar.gz file they have on the official scilab.org site (and follow the instructions).
The only thing I've found is that you can export the graph in pdf even if you can't see it in the graphical window and then, open the exported file with the pdf viewer as usual. This is enough for my necessities atm.
Hope somebody finds this comment useful.
Answer: LIBGL_ALWAYS_SOFTWARE=1 ./bin/scilab
credits: https://askubuntu.com/questions/1333764/scilab-graphic-window-appear-but-it-is-blank-there-is-no-plot-ubuntu-20-04-lts
Comment: Welcome to Stack Overflow! Please read [answer] and [edit] your answer to contain an explanation as to why this code would actually solve the problem at hand. Always remember that you're not only solving the problem, but are also educating the OP and any future readers of this post.
Answer: _JAVA_OPTIONS="-Djogl.disable.openglcore=false"  scilab &
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).


Pooled logistic regression with irregular intervals

Question: I'm planning an analysis that will require pooled logistic regression to analyze a longitudinal dataset with time-dependent explanatory variables (see here - D'Agostino, et al. 1990). However, the standard analysis uses regularly spaced intervals. My study has followup points at 0, 2, 6, 12, 24 months. What is a good way to model the effects of time given this structure of unevenly spaced intervals? Would a restricted cubic splines for the effects of time address the irregularity issue?
Thanks!
Answer: You can model the time trend in the log odds of the probability that $Y=1$ using a spline function (and using other methods).  I do not believe that the time points need to be equally spaced, but rather that they be discrete and have roughly the same schedule for every subject.


Joining two text fields in a Computed Text field

Question: I have two fields whose type is List (text). In the allowed values list, i want to join them. I used the following code.
<code>$value = $entity->field_a->value . $entity->field_b->value;
$value = $fields['field_a'][0]['value'] . $fields['field_b'][0]['value'];
</code>
I can get the result fine, but the value returns the keys only. How can I get the labels?
Answer: To get the allowed values set in your field settings you need to call as follows
<code>$entity->field_my_field->getSetting('allowed_values')[$entity->field_my_field->value];
</code>
your code should look like this
<code>$value = $entity->field_a->getSetting('allowed_values')[$entity->field_a->value] . $entity->field_b->getSetting('allowed_values')[$entity->field_b->value];
</code>
Comment: This worked great thanks.
Answer: Although the question is not very clear to me, but I guess the following could help you,
<code>$key1 = $entity->field_a->value;
$key2 = $entity->field_b->value;
$value1= $fields['field_a']["#options"][$key1];
$value2= $fields['field_b']["#options"][$key2];
$value = $value1 . $value2;
</code>
Comment: This returns an empty value.
Comment: lets say I have a list text field with allowed values as key|label:

US|United States
UK|United Kingdom
AU|Australia

Now i have a saved record in the database with selected value United States
when using $value = $entity->field_a->value;

this will return US and not United States
I want to get the label United States
Comment: @Al-JasimMohammed, as I mentioned in the solutions  the first parameter is key, then it means "UK" is a key.
you need to get the label.

how can it be possible, you should get it from the form element.
tell me where do you want to do this? on form submit?

the solution is valid, but you need to just adjust the place you want to get it.
if it's form and form submit, then we need to get it from `$from` instead of `$fields`.
please give more details
Comment: I want to get it in a computed list text field, that I added from (admin/structure/types/manage/article/fields)
Then I want to display it in a node page
Comment: @Al-JasimMohammed you don't do it only in node view template without engaging the couptation?
is there any special reason for that?
Also you can do it in the node submit.


DataGridTemplateColumn BeginEdit on selection

Question: I subclassed DataGridTemplateColumn to define a custom column type. I use the following code to initialize the edit mode.
    protected override object PrepareCellForEdit(
        FrameworkElement editingElement, RoutedEventArgs editingEventArgs)
    {
        editingElement.MoveFocus(
            new TraversalRequest(FocusNavigationDirection.First));
        return base.PrepareCellForEdit(editingElement, editingEventArgs);
    }

    private void MyTextControlGotFocus(object sender, RoutedEventArgs e)
    {
        var control = sender as MyTextControl;
        if (control != null)
        {
            control.SelectAll();
        }
    }

The problem is that PrepareCellForEdit gets only called after I 1) clicked into the cell and 2) clicked again to enter edit mode.
With DataGridTextColumn I can select the cell and type in some text to immediately enter edit mode without clicking again. I want the same behaviour for my column type.
The question is, how can I handle a KeyDown event on the cell to call DataGridOwner.BeginEdit in my DataGridTemplateColumn class. I tried to add a PreviewKeyDown handler to the control inside the CellTemplate but the event does not get fired.
Answer: I finally came up with this soultion:
<DataGrid ItemsSource="{Binding Persons}">
  <DataGrid.Columns>
    <DataGridTemplateColumn Header="C1">
      <DataGridTemplateColumn.CellTemplate>
        <DataTemplate>
          <TextBlock Text="{Binding Name}"/>
        </DataTemplate>
      </DataGridTemplateColumn.CellTemplate>
      <DataGridTemplateColumn.CellEditingTemplate>
        <DataTemplate>
          <TextBox Text="{Binding Name}" />
        </DataTemplate>
      </DataGridTemplateColumn.CellEditingTemplate>
    </DataGridTemplateColumn>
  </DataGrid.Columns>
  <DataGrid.CellStyle>

    </Style>
  </DataGrid.CellStyle>
</DataGrid>

The important part is Focusable=False (don't know why). Otherwise the IsSelected trigger works only for the first selection.


Static to fixed menu bar

Question: I am working on a website for a school project. I have a menu bar that is floating at the top of my screen, but i want things to look nice and fancy.
I did some research on how to create a menu bar that looks like this
https://nl.malwarebytes.org/mwb-download/
Can someone help me?
Comment: http://stackoverflow.com/questions/20630848/how-can-i-make-a-fix-positioned-menu-bar  here is the same quetion
Comment: Rather than simply asking for help ambiguously, if you provide us with your current code, or demonstrate what you've tried and what isn't working about it, I'd be happy to point you in the right direction, and provide some assistance with a situation specific to you.
Have a look at http://stackoverflow.com/help/how-to-ask
Comment: Sorry, first time poster :) I have managed to get a sticky navbar now :)
Answer: $(window).on('scroll', function(event) {
    if($(this).scrollTop()>50 && $(this).innerWidth()>480) {
        $('header').addClass('sticky');
    }
    else {
        $('header').removeClass('sticky');
    }
});

use this in java script then just style your nav for .sticky class
Comment: do not forget add transition on height for header
Comment: It's important to note here that the code is written in jQuery; you will require jQuery libraries for this to work.
Answer: It is important to note that the header on the example site does not transition from 'static' to 'fixed' position. It is always positioned 'fixed'.
To recreate the effects of the provided link, you need to be somewhat savvy in your choices on what gets transitioned, types of transitioning and what properties are changing.
One way to accomplish this is using jQuery to create a condition when the user has scrolled beyond the top of the window.
For instance:
HTML:

One

Two

Three

CSS:
.menu{
  position:fixed;
  width:100%;
  height:100px;
  background-color:transparent;
  -webkit-transition:  all 200ms ease;
  transition: all 200ms ease;
}

.content{
  padding:100px 60px 60px 60px; /*accomodates the fixed position header space*/

  -webkit-transition:  all 200ms ease;
  transition: all 200ms ease;
}

/* Styles for the menu after scrollTop >= 1 */
.menu.scrolled{
  height:60px;
  background-color:black;
  color:white;
}

.menu.scrolled + .content{
  padding-top:60px;
}

JS:
$(function() {
  $(window).scroll(function() {
    var scrolled = $(window).scrollTop();
    if (scrolled >= 1) {
      $('.menu').addClass("scrolled");
    } else {
      $('.menu').removeClass("scrolled");
    }
  });
});

A version containg more stylistic CSS and additional CSS transitions can be seen here:
http://codepen.io/eoghanTadhg/pen/PNRNOv
Comment: Managed to do it like thi

Static to fixed menu bar


Add a custom attribute to the main div of a node only on teaser view mode

Question: I want to add a custom attribute to the main div of a node only on teaser view mode.
e-g

I need to add tarbot attribute to the main div of a node to apply a JS function. I want to affect only teaser view mode with my Function, not Full content display.
Comment: You should be able to add custom attributes via `#attributes` key in the build array before it is sent for rendering.
Comment: @Kevin How I can. could you provide me a format or context?
Comment: Is it custom field or any entity field?
Answer: We can use preprocess function to change OR add an attribute to our node main div.
function MYMODULE_preprocess_node(&$variables){

  // Array of aatributes which we want to pass to the node div.
  $attr_array = ['tarbot' => 'value'];

  // Condition for teaser format.
  if ($variables['teaser'] == TRUE) {

    $variables['attributes_array'] = $attr_array;
  }
}

by using this function we will get our node div in this form

Answer: In {THEMENAME}.theme file(d8) or template.php file(d7) under your theme, you can use template_preprocess_node function and add the extra attributes to node by checking the view mode.
Thanks


Reading a RSTS/E V6 9-Track Tape Under VMS 7.3

Question: I am trying to recover some backups from two old 9-track tapes.  They were created ~1980 under RSTS/E version 6.
I have VMS 7.3 on a MicroVAX II and a functional TU80 tape drive.  Both systems are fully functional so it is not a "Jurassic computing" hardware issue.
I have tried several strategies including BACKUP and EXCHANGE, but is is "no go" so far.  The tape density is unknown, but likely 800 or 1600 given the age of the tapes.
Any suggestions would be appreciated.
Thanks in advance,
Kevin
Comment: I'd start with `directory` and `copy`. Mount the tape with `/override=identification` and list out the directory. (Note that a tape can have multiple files with the same name.) Tip: You can use `set magtape/rewind` to rewind the tape. IIRC, `show device/full` will display the density.
Comment: From the `retro-computing` tag: "Retro computing refers to the use of outdated hardware platforms (such as the Commodore 64 or BBC Micro), **usually for recreational purposes**." _(Emphasis mine.)_ Sigh.
Comment: Thanks for the reply.  I tried this approach first.  The tape has no recognized structure under VMS.  It will $mount/over=id just fine but $dir fails.  $sh dev mua0:/full after $mount shows the system default density, even if I $mount/density=800 first.
Comment: Thanks for the retro-computing info, this is indeed a hobby!  I have ~20 old DEC systems in the basement.  This particular task is to help a friend restore his old grad school tapes!
Comment: The spec's show that the TU-80 only supports 1600bpi. (It's been a long time since I had one. It was a leap down from the TU-78 I started with.)
Answer: Might need RSX and FLX ?  (File-explorer?)
If the mount/over=if worked, then it may have been ansi formatted no?
Could it be a nameless file that you can grab with a straight copy?
I may well be all wrong on this.  $COPY MT: test.dat
$ Mount/foreign and DUMP to see what's out there with headers and tape-marks and so on?
There are free-ware tape-copy tools out there that can get raw block on disks for analysis.
And I believe the latest LD has a virtual magtape driver as well allowing one to interpret a disk file as tape and visa versa?
Good luck!
Hein
Comment: Thanks Hein!  I tried the DUMP utility but was unable to get it to work with the target tapes.  I get a parity error on both of the tapes, so I now think that they may have been recorded at higher density than my old TU80 drive can read.  As HABO noted, the TU80 only supports 1600 BPI.
Comment: And I might add that I tested DUMP on some other known-good tapes, and I was able to read them just fine.  I think that density is the culprit.  Thanks all for the suggestions!


Not using templates Symfony 2

Question: I am having a hard time finding anything on the topic. What if I wanted to use a controller to simply render a static page such as about.php? When I try the following I get a template not found error.
public function aboutAction()
{
    return $this->render('FooBundle:Default:about.php');
}

Answer: You can render the page without creating a controller:
foo_about:
    path: /about
    defaults:
        _controller: FrameworkBundle:Template:template
        template: 'FooBundle:Default:about.html.php'

The about.html.php must be located under Resources/views/Default/ directory of the FooBundle.
See How to render a Template without a custom Controller
Comment: Thanks for the reply. This doesn't really answer the question, I want to use a controller to decide what page to render and prepare a response. I can't exactly `prepare` a response if I'm not using a controller.
Comment: Good, but you didn't mentioned it. You said static page, and for me a static page must not prepared. But take notice to the last sentence, where template files must be located.

Not using templates Symfony 2