Install Ubuntu truncate command to Mac OS X 10.7

Question: Is there any way to install Ubuntu's truncate command to Mac OS X 10.7 in order to use the command in Terminal shell?
Comment: The question is posed as 'how to install truncate', but a workable answer might be `:> file` which works equally well on Ubuntu and macOS. (see alternate answer below)
Comment: What are you trying to do?
Comment: I have a shell script that makes sure of truncate function, which I want to try it in local Mac.
Answer: Update:
As of March 2015 truncate is no longer available in Homebrew as a standalone formula.
As truncate is part of GNU Coreutils you should install it on OS X with the following command:
> brew install coreutils
After installation truncate will be available under the name gtruncate. Note that all programs from Coreutils will be available with the prefix g.
Comment: This should be the accepted Answer.
Comment: Looks like `truncate` is available in Homebrew again as its own formula.
Comment: ...and it's gone again.
Answer: This shouldn't really be in Server Fault, probably Super User or Ask Different, but you can install a lot of binaries by using homebrew, found here:
Link
Firstly you'll need to install command line tools for Xcode, which will probably require you to purchase it if you havn't already.
Once you've done this, run:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

To install homebrew, then when it's installed, run:
brew install truncate

Comment: Also would appreciate an upvote as, regardless of if it worked for you, I did provide a valid answer to your question :)
Comment: I found that truncate in homebrew is different than Ubuntu version. It does not have option flag.
Comment: There's also Fink: http://www.finkproject.org/ and Macports: http://www.macports.org/ you could try, before doing so run brew uninstall truncate, though.
Comment: Now, using Yosemite the following solved for me (also using home brew: http://www.magiclantern.fm/forum/index.php?topic=6407.0
Answer: When truncate is not available you can often use dd instead. Eg.
Make a file full of zeros:
dd if=/dev/zero of=/tmp/youroutputfile bs=1m count=50

Truncate an existing file:
dd if=yourinputfile of=/tmp/youroutputfile bs=1k count=1

Note: the OS X dd uses lower case unit letters where GNU dd uses upper case.
Comment: Not as far as I know. You'd have to do the target size calculation yourself by reading the file size first and subtracting 100 bytes.
Comment: `dd` can almost achieve the goal. Can `dd` truncate from the end of file? As I read from the man file of `dd`, I know that it can only be read from file beginning. How about I delete last 100 bytes from the file ?
Comment: I like this answer as it doesn't require me to install anything new.
Answer: One point of consideration is if the truncated file is the same file or a new file with the same name. Of importance for log files that are still being written-to, or being tailed.
:> file
Works on both *nix and macOS to truncate a file in-place.


Is [ConcurrencyCheck] decoration non-functional?

Question: I am trying to implement Optimistic Concurrency checking in a Core project. I display the record to be edited, alter the data in the back end (Sql Server Management Studio), save the data from the web page but no Concurrency exception occurs despite the decoration being on every single field in the Model (except the key).
I read this article:
https://entityframeworkcore.com/knowledge-base/47392764/ef-core-dbupdateconcurrencyexception-does-not-work-as-expected
which appears to explain that the decoration'[ConcurrencyCheck]' does absolutely nothing! It appears to recommend building duplicate, unmapped fields into the model if we use the EF context calls .Update and .SaveChanges (it appears the concurrency comparison happens when .Update retrieves a version of the data the user has never seen for comparison).
This article:
https://learn.microsoft.com/en-us/ef/core/saving/concurrency
has an example using RawSQL.
Now, I am OK using RowVersion. I might have used it first if the [ConcurrencyCheck] documentation had included the statement "Only works with RawSql".
Am I reading this wrong?
Answer: I was reading it wrong enough.  I just automatically used SaveChangeAsync without the await and
did not consider it would return before the update was completed. I changed it to .SaveChanges and it caught the exception.


Table of Intervals, "Diminished" column

Question: I found the following table of Intervals in the Harvard Dictionary of Music (re-created by me in MS Word):

(Correction to the above: It is "Harvard Dictionary", not "Oxford Dictionary".  Oops!)
I am wondering why they use the base note of "c" for all the columns, except for the "Diminished" column.  Is it simply to avoid the double-flat?  I'd rather see the same base note for all columns.  But my level of music theory knowledge is not very high, and sometimes things that initially make sense to me turn out to be wrong.  Is there anything "wrong" with the following modification to the chart?

Would someone with a great deal of music theory knowledge find the original Oxford one "better" for some reason?
Comment: Please don't post text as an image; it can't be searched, copy/pasted into answers, or read by screen-readers for the visually impaired.
Comment: @YourUncleBob:  Dude!  It took me ages to make those tables in MS Word!  Maybe it would be possible to make them with the formatting available in a stackexchange "question", but I kind of doubt it!
Comment: It would be more *logical* to keep Cnat. as the base note for all intervals, as you have. There is no real difference between making the base note higher by a semitone than making the higher note lower by a semitone - provided the letter names stay the same.
Answer: There's nothing wrong with both what's in the first table and what's in your amended version.
The "quality" of a diminished interval simply suggest how many semitones there are between the two notes, ie one semitone smaller than that of a minor interval.
Take the interval of the diminished third for instance, the example given to you is C# to Eb which spans 2 semitones apart. In your modified version, ie C to Ebb, these 2 notes are also 2 semitones apart. In fact, the interval between Cx to E would give you also a diminished third, since, again, it's 2 semitones apart AND 3 letter names apart. (This last part is important - why we name these 3 intervals as diminished 3rd and not major 2nd, even though a major 2nd interval is also 2 semitones apart.)
——————-
EDIT: The reason why the example given to you was so, was to illustrate the relationship of how the intervals are inflected from a diatonic interval between notes in the major scale.
Eg.

Major 3rd: C —> E (4 semitones)
Minor 3rd: C —> Eb (3 semitones, C remains unchanged, E lowered by a semitone making the interval smaller)
Diminished 3rd: C# —> Eb (2 semitones, Eb remains unchanged, C raised by a semitone making the interval smaller)

You can also choose to illustrate the intervals in this manner:

Major 3rd: C —> E (4 semitones)
Minor 3rd: C#—> E (3 semitones, E remains unchanged, C raised by a semitone making the interval smaller)
Diminished 3rd: Cx —> E (2 semitones, E remains unchanged, C# raised again by a semitone making the interval smaller)

Also, take note that even though C - Ebb is enharmonically equivalent (ie. sounds exactly the same as) to C - D, the intervals are treated differently because of how the notes are spelt:

C -> Ebb: "C" and "E" are 3 letters apart, hence the interval is a 3rd, and since E is inflected by a double flat, the interval is diminished 3rd.
C -> D: "C" and "D" are 2 letters apart, hence the interval is a 2nd, and since the D is not inflected at all (a natural), the interval is a major 2nd.

Both the examples listed in the Oxford version and yours essentially try to illustrate the same thing: showing how intervals are inflected one semitone at a time. I personally don't exactly have a preference for either, but my guess is the first one is more likely to be used in pedagogy since the double-flat and double-sharp are usually introduced much later in theory teaching.
Hope that helps. :)
Answer: The C# examples are more likely to occur "in the wild" -- that is, in actual compositions.  That is probably why C# were used instead of C.

Table of Intervals, "Diminished" column


Enabling RDP in all workgroup computers

Question: I need to activate RDP on all our Windows XP Computers who are currently in workgroup mode.. anyone knows a script or a software that can scan / activate the RDP Service on the machines?
I've already have a script that does part of it.. the main problem is the running of the script on all computers...
I have administrator access to all of them... but running one by one is a bit of a trouble :|
Answer: What's the script you currently have? This sounds like a job for PSExec (Sysinternals). The main regkey you need is:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\fDenyTSConnection
Which should be changed to 0. Though that depends on if it was explicitly disabled originally, and how.
Personally, I'd make a script for PSExec which copies a .reg fragment over and then logs the change to a text file centrally on a server.
You could also use something like "Reg.exe" which I think is on XP, but I don't have a station to hand to test.
Serious question - why are you using computers in a Workgroup rather than a domain?
Comment: The workgroup is being remodeled to a domain but in various sites it's not implemented yet.. I have that reg key as a script yes.. what I really wanted was some kind of software that ran in a range of ips and executed that command!
Comment: Then look at PSExec or reg.exe as I explained.
Comment: Dan but psexec only runs in a pc at a time that will gonna make me run it on every pc on the network individually
Comment: It will only execute on one PC at a time, yes, but the whole point is that it's remotely executed. I.e., you create a text file with all of your machines and create a batch script to go down them and run your command on each PC.

There are lots of examples on Google for this.

Enabling RDP in all workgroup computers


R - SVM (radial) regression using tune() (e1071) - How to uncover influential features?

Question: I have some questions regarding SVM and regression.

1) How to interpret SVM (regression) results.
2) How to make a proper plot (containing decent information)

e1071's tune() is used to uncover the best cost (C) and gamma (y) parameters. The model is fitted against a training set (train2).
svm.radial = tune( svm , dep_sev_fu ~ . , data = train2 , kernel = "radial" , type = "eps-regression" , ranges = list( cost = c(0.001 , 0.01 , 0.1 , 1 , 5 , 10 , 50 ) , gamma = c( .0001 , .001 , .01 , .1 , 1 , 5 , 10 ) ) )
Optimal model is fitted against test set:
svm.tuned <- svm( dep_sev_fu ~ . , data = test , kernel = "radial" , type = "eps-regression" , ranges = list( cost = 1 , gamma = .01 , epsilon = .1 ) )
In order to extract (significant) regression weights, there's a function called 'rfe' within caret that applies backward selection. But it is unclear how to specify a model when using SVR.
I looked at a kernlab example, where they mention the W vector. After running w = t(svm.tuned$coefs) %*% svm.tuned$SV a vector of coefficients emerge. But how do I know if they are significant weights?

Note: Do I need to predict() of some sort? Not really sure how validation should work at this step

Bonus
Expanding on the validation approach, how can I fit using 10-fold CV?
* Example of the first 10 cases & 8 variables *
disTypecomorbid.disorder disTypedepressive.disorder Sexemale Age aedu IDS BAI FQ
                        1                          0        0  50   10   6  10  4
                        0                          1        0  35   11   7   5  2
                        0                          0        0  51   15   4   3 14
                        0                          1        0  43   15  11   7  3
                        0                          0        0  38   10   7   8 15
                        1                          0        0  38   10  15  15 32
                        0                          0        0  45    9  12  12  2
                        0                          0        0  57    9   9  14  4
                        1                          0        0  43   10  12   3  0
                        1                          0        1  49   11  14  11  3
Answer: You can't really interpret SVM models unless a linear kernel is used (you have the RBF in your code above). The matrix multiplication is only for the linear case too (I believe).
There is an excessive amount of documentation on rfe at the package website. Look there for syntax.


Criação de Trigger Sql Server

Question: Estou criando um trigger para criar uma cópia dos dados alterados na tabela.
Porém apresenta um erro.
USE [BANCO1]
GO
/****** Object:  Trigger [dbo].[BANCO1]    Script Date: 08/07/2015 14:13:34 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
ALTER TRIGGER [dbo].[BANCO1]
  ON [dbo].[TABLE1]
  AFTER  UPDATE
AS

  BEGIN
    DECLARE
    @NOSSONRO   VARCHAR(15)
    SELECT @NOSSONRO = E1_NUMBCO FROM UPDATED
    SET NOCOUNT ON;

    UPDATE TABLE1 SET E1_NOSNRO = @NOSSONRO  WHERE  E1_NOSNRO='' AND E1_NUMBCO=@NOSSONRO

  END

Answer: A forma como o SQL Server manipula os eventos que acionam gatilhos (trigger) pode se transformar em verdadeira armadilha para quem programa o procedimento de gatilho e desconhece os detalhes específicos do SQL Server, pois enquanto vários gerenciadores de banco de dados acionam o gatilho uma vez para cada linha (for each row), no SQL Server o gatilho é acionado uma vez para cada instrução DML (DELETE, INSERT, UPDATE).
-- código #1
ALTER TRIGGER [dbo].[BANCO1]
  ON [dbo].[TABLE1]
  AFTER  UPDATE as
begin
set nocount on;

UPDATE T
  set E1_NOSNRO = I.E1_NOSNRO
  from TABLE1 as T
       inner join INSERTED as I on I.E1_NUMBCO = T.E1_NUMBCO
  where T.E1_NOSNRO = '';

end;
go

Sugestão de leitura: Armadilhas na programação de trigger.
Answer: Entre o Begin e o End da sua trigger utilize o seguinte código:
SET NOCOUNT ON;

DECLARE @NOSSONRO AS VARCHAR(15);

/*Código abaixo obtém o novo E1_NUMBCO, ou seja o valor atualizado.
Se você quiser pegar o valor antigo do E1_NUMBCO utilize DELETED no lugar do INSERTED*/

SELECT @NOSSONRO = E1_NUMBCO FROM INSERTED;

UPDATE TABLE1 SET E1_NOSNRO = @NOSSONRO  WHERE  E1_NOSNRO='' AND E1_NUMBCO=@NOSSONRO

No Sql Server não temos o updated. O updated é tratado como uma exclusão seguido de uma inclusão.

Criação de Trigger Sql Server


There cannot be more than three primes of the form $n^{n^2}-k$ for the same $k$?

Question: I was searching for primes of the form $n^{n^2}-k$ on PARI/GP and noticed that primes of this form for same $k$ are quite rare. The probability of finding a prime of this form is $\frac {1}{n^2 \log (n)}$, which makes me believe that the number of primes for same $k$ of this form would be finite.
I also noticed that the maximum number of primes of the form $n^{n^2}-k$ for the same $k$ seems to be three.
I searched for a range of $k \le 3 \times 10^5$ and $n \le 50$ on PARI/GP and didn't find more than three primes for same $k$. Here are the some values of $k$, for $k \le 10^4$ I found on PARI/GP, that have three primes of the form $n^{n^2}-k$. The first column displays the values of $k$ and the second column shows the values of $n$ for which $n^{n^2}-k$ is prime.
<code>1409 [4, 6, 8]
3083 [4, 6, 8]
5496 [5, 7, 13]
7427 [4, 6, 8]
8902 [3, 9, 15]
</code>
Questions:

$(1)$ Can there not be more than three primes of the form $n^{n^2}-k$ for the same $k$?
$(2)$ If there can be more than three, then is there a maximum number of primes of the form $n^{n^2}-k$ for the same $k$?

Edit 1: Following the same process, I believe that the maximum number of primes of the form $n^{n^2}+k$, for same $k$ is four.
Edit 2: Peter showed provided an example with four primes for the same $k$ which solves my question $1$. So now I only look for answers of question 2.
Comment: Mod 5 destroys a lot.
Comment: if n is even $k\equiv 1 \bmod p$ forces n congruent to 0 mod p, if n is odd k is congruent to n mod p. These additive inverse on the second form. at least in a few cases.
Comment: If $k$ is odd and $n$ is even, then $k$ might be prime itself, so none of $n$ divides $k$.
Comment: I think, $5$ solutions for the same $k$ is challenging, but the bunyakovsky conjecture could imply that arbitary many solutions are possible (at least in the +k-version), but I have to work it out first.
Comment: mod 3 Actually eliminates more than 55% and non overlapping with mod 5 would yeild about 8.4% of cases surviving mod 15.
Answer: For $$k=1996199$$ the numbers $$4^{4^2}-k$$ $$6^{6^2}-k$$ $$8^{8^2}-k$$ $$10^{10^2}-k$$ are all prime
Comment: All except n=10, get destroyed on the +k side as k is -1 mod 5. n=10 gets destroyed by mod 3 for the plus side.
Comment: Nice work Peter! Though I do believe that that a maximum number of primes for same $k$ does exist (could be 5 till now).
Comment: if not for overlap you'd only have 19 scenarios left mod 15 @Mathphile
Comment: $k=33213007$ gives (at least) $5$ solutions in the "+"-version
Comment: $k=247342073$ gives (at least) $5$ solutions in the "-" -version
Comment: Those numbers were found by Enzo Creti


How to Expose service in AKS with Istio Ingress?

Question: I used to use Type: LoadBalancer to expose my service through reverse proxy. Everything worked fine without Istio.
However, error happened when I applied Istio to my cluster.
I tried to expose my services in Kubernetes with Istio Ingress, but I think I misunderstand something when routing services with Istio.
I have 2 deployments in same namespace (See picture below):
1: Application (Bus-id)
2: Reverse Proxy of application (Bus-Proxy): Translate HTTP to gRPC
https://drive.google.com/file/d/1tby9_taJb9WMHi0ssO9Os7MQAWRMga6k/view?usp=sharing
Version:

Kubernetes version (AKS with RBAC enabled):
Client Version: v1.15.0
Server Version: v1.12.8
Istio version: 1.1.3 (AKS said that they tested on 1.1.3)
Helm:
Client: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1",   GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}

I tried BookInfo example in Istio (https://istio.io/docs/examples/bookinfo/), it worked.
But, I tried Voting example in AKS (https://learn.microsoft.com/en-us/azure/aks/istio-scenario-routing), I can't access example with external load balancer's IP, it retun "timeout"
Deployment file:
1. bus-id.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bus-id
  namespace: smart-id
  labels:
    k8s-app: bus-id
spec:
  selector:
    matchLabels:
      k8s-app: bus-id
  template:
    metadata:
      name: bus-id
      labels:
        k8s-app: bus-id
    spec:
      containers:
        - name: bus-id
          image: mydockerhub/mydockerhub:bus-id
          ports:
            - containerPort: 50001
          env:

            - name: APP_NAME
              value: bus-id
---
apiVersion: v1
kind: Service
metadata:
  name: bus-id
  namespace: smart-id
  labels:
    service: bus-id
spec:
  ports:
    - name: http
      port: 50001
      targetPort: 50001
      protocol: TCP
  selector:
    k8s-app: bus-id

2. bus-proxy.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    k8s-app: bus-proxy
  name: bus-proxy
  namespace: smart-id
spec:
  selector:
    matchLabels:
      k8s-app: bus-proxy
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: bus-proxy
    spec:
      imagePullSecrets:
        - name: duynd
      containers:
        - image: mydockerhub/mydockerhub:bus-proxy
          name: bus-proxy
          ports:
            - containerPort: 40001
              name: http
          env:
            - name: APP_NAME
              value: bus-proxy
---
apiVersion: v1
kind: Service
metadata:
  name: bus-proxy
  namespace: smart-id
  labels:
    service: bus-proxy
spec:
  ports:
    - port: 8080
      targetPort: 40001
      protocol: TCP
  selector:
    k8s-app: bus-proxy

3. ingress.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: smartid-gateway
  namespace: smart-id
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: smartid
  namespace: smart-id
spec:
  hosts:
    - "*"
  gateways:
    - smart-id/smartid-gateway
  http:
    - match:
        - uri:
            prefix: /api
      route:
        - destination:
            host: bus-proxy.smart-id.svc.cluster.local
            port:
              number: 8080

I expect It works with ingress-ip:ingress-port/api/my-function (method POST). However, it returns error 500, bus-proxy's POD also prints log (I think that the request came to bus-proxy successfully, but can go through to bus-id).

I also post on issue on github (https://github.com/istio/istio/issues/16069)
Answer: First of all, if you run all the applications in AKS with the Istio, I will suggest you install the Istio following the steps that AKS provide in Install and use Istio in Azure Kubernetes Service (AKS).
Now, take a look at the example the AKS provided here and there is something you need to know:

The Istio has the proxy itself. So you need to choose which to use or
  use both proxies, but you need to make sure it supports two proxies.

And if you use the proxy of Istio, then you also need to enable the istio-injection for the namespace of your application just like the example:
kubectl label namespace voting istio-injection=enabled

This label instructs Istio to automatically inject the istio-proxies as sidecars into all of your pods in this namespace. And you should use the right gateway for your virtual service in the ingress.yaml.
Comment: Thank you for your suggestions, my problem wasn't in deployment. The problem was connection between 2 services inside, they're stuck with passing metadata.
Answer: My problem wasn't in deployment. The problem was connection between 2 services inside, they're stuck with passing metadata.
Check white metadata white list, if you're using gRPC.


Pythonic way of merging 2 lists of dicts based on common keys

Question: I have 2 lists of dictionaries:
dict1 = [{"a":1, "b":2, "c":1295}, {"a":2, "b":5, "c":6274}, {"a":3, "b":1, "c":5337}]

and
dict2 = `[{"a":1, "b":2, "d":1884}, {"a":2, "b":5, "d":2049}, {"a":3, "b":3, "d":1086}]

The first list of dicts has the keys "a", "b" and "c", while the second list has keys "a", "b" and "d".
I want to create a list of merged dicts having all 4 keys. Only the dicts with equal values for "a" and "b" need to be merged.
The expected result looks like this:
[{"a":1, "b":2, "c":1295, "d":1884}, {"a":2, "b":5, "c":6274, "d":2049}]

I am looking for a Pythonic way of doing this.
Comment: do the orders match up as in your example. Like are you trying to match `dict1[i]` with `dict2[i]` always?
Comment: That is the case, for now.
Answer: Assuming the merging-candidates from the two lists are the dicts from the same position, you can zip your lists together, employ a list comprehension and use the **-syntax idiom for merging two dicts.
>>> dicts1 = [{"a":1, "b":2, "c":1295}, {"a":2, "b":5, "c":6274}, {"a":3, "b":1, "c":5337}]
>>> dicts2 = [{"a":1, "b":2, "d":1884}, {"a":2, "b":5, "d":2049}, {"a":3, "b":3, "d":1086}]
>>>
>>> [{**d1, **d2} for d1, d2 in zip(dicts1, dicts2) if all(d1[k] == d2[k] for k in ('a', 'b'))]
[{'a': 1, 'b': 2, 'c': 1295, 'd': 1884},
 {'a': 2, 'b': 5, 'c': 6274, 'd': 2049}]

Bonus pandas solution:
>>> df1 = pd.DataFrame(dicts1)
>>> df2 = pd.DataFrame(dicts2)
>>>
>>> df1
   a  b     c
0  1  2  1295
1  2  5  6274
2  3  1  5337
>>>
>>> df2
   a  b     d
0  1  2  1884
1  2  5  2049
2  3  3  1086
>>>
>>> pd.merge(df1, df2, on=['a', 'b']).to_dict(orient='records')
[{'a': 1, 'b': 2, 'c': 1295, 'd': 1884},
 {'a': 2, 'b': 5, 'c': 6274, 'd': 2049}]


60 is 20% percent less than what number

Question: I know the answer is 75.
I need the actual equation for solving. As in 60 + .20X = X
That equation seems correct but I'm not able to show my 6th grader how to get the answer.
Comment: Time to teach your $6$-th grader a little algebra. Also, the equation that represents the problem is $60=x-.20x$, or equivalently, $60=x(1-.20)$
Answer: The correct equation is $X-0.2X=60$ or equivalenly $0.8X=60$, which evaluates to $X=\frac{60}{0.8}=\frac34=75$.
Comment: The question says $60$, not $60$ percent.
Comment: Where is the explanation here? She knows the answer
Comment: sorry, I misread 60 for 0.6; thanks to John Doe for fixing it.
Answer: If you want to avoid algebra for some reason (you say you have the equation but can't show your 6th grader how to solve it from there), then here's another way to explain it.
$60$ is $20\%$ less than some number. So $60$ is $80\%\left(=\frac45\right)$ of this number. So this number $\times \frac45$ gives you $60$. So this number must be $60\times\frac54=75$.
Answer: Working with number is what has helped me to understand things deeply from 6th grade to now. I can still remember understanding linear algebra with 2 dimension real valued vectors in my head.
Something that would have worked on the 6th grader I was: (italic is the kid answer the rest is what the teacher show / says:
$60$ is $20%$ less than what number x?
Well that number should to be bigger than 60 since it is that number minus something. Let's try $100$:
$100 - 20% 100 = 80$ it is too big.
But what do we want? I don't know
What do we want on the right side of the equation? 60
What number on the left side should give us 60? I don't know
60 is 20% less than what number x? What number on the left side should give us 60? x
Replace 100 with x and 80 with 60 in the formula. Oh Ok
I would try to teach equations with something simpler at first maybe. The idea of percentage is tricky and pretty difficult to grasp.
Answer: $$ 20 \text {%} \text { less  means } 80\text {%} $$
$$ 80 \text {% means  4 out of 5}$$
$$\text {4 out of 5 is 60}$$
$$ \text {What is 5 out of 5 ?}$$
$$ \text {It is 75} $$
Answer: What do you know about number X, remembering that $X$ = 100% of $X$?
You know that  : 60 is the result of substracting 20% of X from X, or, equivalently,
$100$% of $X$ - $20$% of $X$ = $60$,
which means that :
$80$% of  $X = 60$, or
$\frac {80} {100} \times X = 60$.
You finally get $X$ by dividing eaxh side by $\frac {80} {100}$.

60 is 20% percent less than what number


Problems setting up Grafana Agent

Question: I was trying to set up Grafana-agent.yml file for nginx but with multiple attempts no success. The setup provides we with an sample configuration and what i have to add to the configuration. This is the example configuration:
<code>integrations:
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
    metric_relabel_configs:
    - action: keep
      regex: (prometheus_target_.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
      source_labels:
      - __name__
  prometheus_remote_write:
  - basic_auth:
      password: <your_prom_pass>
      username: <your_prom_user>
    url: <your_prom_url>
logs:
  configs:
  - clients:
    - basic_auth:
        password: <your_loki_pass>
        username: <your_loki_user>
      url: <your_loki_url>
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
    - job_name: integrations/nginx
      static_configs:
      - targets:
        - localhost
        labels:
          host: <http_hostname>
          __path__: <path to json nginx access log>
metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: <your_prom_pass>
        username: <your_prom_user>
      url: <your_prom_url>
    scrape_configs:
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal
</code>
In this configuration the configuration for nginx is already there. Now what i dont understand is with what information i have to fill out the fields like metrics.config.basic_auth.password or logs.configs.scrape_configs.targets.host
i Already attempted to search for a sample configuration that documents the values better.
Comment: I can give you some help, but in order to elaborate, I would need first to if you are using Grafana Cloud or you are using you own Prometheus instance.
The  and  are the basic auth configured in Prometheus.
Answer: In order to get started, I would recommend you to login to Grafana Cloud (it's free), and on the left side menu, go to "Connections", "Add a new connection".
There, you can just click on "Linux" and you will get a sample config for metrics and logs:
Node exporter metrics sample config
Logs sample config
This should be enough to have a sample agent communicating with Prometheus and Loki and being able to query data with Prometheus.
You can also use/check this code repository (DISCLAIMER: I'm the owner) and in particular, the agent config file.

Problems setting up Grafana Agent


what is DDODBC.DAT

Question: The file /var/local/odbc/DDODBC.DAT is being created/accessed on some (not all) servers when the database is accessed via Perl dbi:ODBC.
I did a full crawl through /var/local/odbc on a server that does create the file and a server that does not, and everything is identical (md5sum) save for the existence of that file.  There is no difference in the environment and the odbc.ini and odbcinst.ini files are also the same.
I'm currently at a loss.  I'm interested because this file caused a db issue that was resolved by force-removing this file.
Answer: Answer here.

DataDirect has confirmed that the file DDODBC.DAT is (typically)
  created when there is a licensing conflict on the host. In our case,
  our license is for servers with a maximum of 16 CPUs, however we
  installed the driver on a host with 24 CPUs. The driver realized this,
  made note of it (with DDODBC.DAT), and will, eventually, switch itself
  into evaluation mode.

what is DDODBC.DAT


Malicious Code or False Positive?

Question: I ran "Sucuri SiteCheck" in one site that im working and the sitecheck reported the following code as malware:
Known javascript malware.
Details:

http://labs.sucuri.net/db/malware/mwjs-include-unescape3?v1

code:
var i,y,x="3c646976207374796c653d22706f736974696f6e3a6162736f6c7574653b6c6566743a3070783b746f703a3070783b7669736962696c6974793a68696464656e3b222069643d2264617461646976223e3c696d67207372633d2267706c75732e637322206865696768743d2230222077696474683d2230223e3c2f696d673e3c2f6469763e";y='';for(i=0;i<x.length;i+=2){y+=unescape('%'+x.substr(i,2));}document.write(y);</script>"

Someone can tell me what this code do, and it is really malicious ?
Thank you very much!
Best Regards!
Answer: It looks like Sucuri SiteCheck doesn't like the way it was obfuscated. Decoding it as hex, it is:

The question as to whether or not it is malicious depends on what gplus.cs is.
Comment: Thank you David, now i question myself why this are obfuscated. I can use the decoded code to help with the sitecheck alarm?
Comment: @kodden I haven't used sitecheck, but I suppose you could provide it with the URL of gplus.cs to let it scan that.
Comment: gplus.cs doesnt sound like an image - it is looking for something local to the script - are you certain nothing has been slipped ont your server near the page the script was on? "datadiv" sounds suspicious doesn't it!
Comment: It may just end up being a tracking pixel. Or it might be a browser exploit designed to execute arbitrary code when the file is attempted to be viewed as an image. Hard to say.
Comment: Googled gplus.cs and came up with what looks like a games company  gplus.csgames.org - goes to https://plus.google.com/s/gplus.cs/top on search which has adverts...may not be visciously malicious but makes you wonder about oogleG  Also more oogly stuff comes up with lots of .cs  https://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&source=web&cd=11&cad=rja&uact=8&ved=0ahUKEwiDsO_6sqPJAhUC1hoKHZmxBlw4ChAWCBwwAA&url=https%3A%2F%2Fgithub.com%2Fgoogleplus%2Fgplus-verifytoken-csharp%2Fblob%2Fmaster%2Fverifytoken.ashx.cs&usg=AFQjCNHRsTyP_edO11C8xOgdRECId7M3lA&bvm=bv.108194040,d.d2s

Malicious Code or False Positive?


What determines who can view an unpublished node?

Question: The Node Access documentation explains that there are three things that affect node permissions, used in this order of preference until an answer is received: an overriding bypass node access permission; hook_node_access (used on single nodes and not in listings); hook_node_grants (which compares against the node_access table).
What I can't see is how (un)published content is handled. The only default implementation of hook_node_access is node_node_access which you can see completely ignores the 'view' mode, answering only for create/delete/edit modes.
The hook_node_grants docs say explicitly that (un)published content is "not handled by the system".
But there are the checkboxes in the user permissions table for view published content and view own unpublished content etc. Where are these implemented?
Answer: It is handled by node_access:

it checks for the "view published content" - without this permission every operation is denied, published or not.
it checks the view own unpublished content permission.
it checks if hook_node_access gives either deny/allow. If not...
If no module implements hook_node_grants then there's a default allow for published nodes (and therefore a deny if not).
If modules do implement hook_node_grants then the node_access table is consulted. I found that published content receives a "default grant" in the node_access table thanks to node_access_acquire_grants, meaning anyone can view it. Unpublished content has no default grant so is therefore denied.
Answer: basically the unpublished content is visible only to those roles that have "Administer content" permission.
you can use this module but I had some problems with it and decided to build my own, lighter module.
Comment: Where is that checked? Do you mean that the `view own unpublished content` is not used anywhere?
Comment: It's not true: Unpublished nodes are also visible to their authors, when they have the "view own unpublished content" permission.
Comment: yep, @kiamlaluno you're right, sorry, my question was rhetorical. Pls. see my full answer above.
Comment: @artfulrobot I was saying that to Marius Illie. His answer is not correct for Drupal 7.
Comment: This is not correct. The permission you meant to say is "Bypass content access control". Then there's a second permission "View own unpublished content" as well.


Correct way to link to views (to solve multi-domain problem)

Question: I have one site (same codebase, same database) that uses two different domains to display two different themes, depending on the content. (This is achieved via the Domain Access module.)
I need to be able to link from one domain to a view on another domain without manually inserting the absolute URL, but I'm not sure how (or even if it's possible).*
I can use l() to link to nodes without any problems. For example, if I'm on first-domain.com and I want to link to second-domain.com/news-story, I would do the following:
<?php print l('News Story', 'node/2213'); ?>
I don't know exactly how it works, but it realises that the news story isn't available on the first domain and automatically creates the absolute link to the second domain.
Creating a link to a view doesn't seem to be so easy. I've made sure the view's Access restrictions is set to Domains (and that only the one domain is selected) but the l() function doesn't seem to look this up.
For example, if I'm on first-domain.com and want to link to a list of all the news stories (which is created by a view) on second-domain.com, doing this:
<?php print l('Latest News Stories', 'news-stories'); ?>
Results in a link like this:
first-domain.com/news-stories
When I really want: second-domain.com/news-stories
Is there any way to solve this? Even if I could look up both domains that have been set via the Domain Access module would work, but I don't know how to do that either. (I've also thought about using Path Redirect, which is better than nothing, but not quite what I'm looking for.)
* Obviously it's possible to put in an absolute URL, but this is far from ideal as the site is currently being developed locally and will eventually be moved to a staging site, and then moved again to production once ready. Remembering to change the absolute links every time would be annoying at best.
Answer: Since you obviously are capable of PHP, the best way I can think of is to write a custom wrapper for l(), something like:
function ml($text, $path, int $domain = NULL, array $options = array()) {
  switch ($domain) {
    case 1:
      return l($text, 'http://first-domain.com/'.$path, $options);
      break;
    case 2:
      return l($text, 'http://second-domain.com/'.$path, $options);
      break;
    default:
      return l($text, $path, $options);
      break;
  }
}

That way, the absolute domains only have to be changed in this one central place with each staging step (domains may even be stored in settings.php).
Comment: Thank you so much--you've just saved me a TON of time!
(Just a minor change: PHP complained about `$domain = 0`, so I changed it to `$domain = NULL`.)
Comment: You're welcome. Also, I changed the $domain parameter to default to NULL.


Prove that a normal matrix is unitary/Hermitian

Question: I'm stuck with these two questions for while. I'd appreciate your help.
<code>Given a normal matrix A, prove that:

a) It is unitary iff the magnitudes for all of its eigenvalues are 1.
b) It is Hermitian iff all of its eigenvalues are real.
</code>
Answer: Hint: A normal matrix can be diagonalized by a unitary matrix.  The question is easy for diagonal matrices.


React Native FBSDK file not found on CI

Question: I have installed FBSDKCoreKit, FBSDKLoginKit and FBSDKShareKit by downloading the SDK to ~/Documents/FacebookSDK and then dragging the frameworks in to the frameworks folder in XCode. I checked the option to copy the files if needed so they appear in the project.
I then added the correct framework search path ~/Documents/FacebookSDK
Building locally works fine, however, building on Bitrise (CI) fails with the error:

/Users/vagrant/git/node_modules/react-native-fbsdk/ios/RCTFBSDK/share/RCTFBSDKShareDialog.h:21:9:
  'FBSDKShareKit/FBSDKShareKit.h' file not found

The framework search path also contains $(PROJECT_DIR) which should find the frameworks located in appname/Frameworks/ but I cannot seem to fix this not found error.
Answer: I ran into this problem using BuddyBuild (another CI system), and solved it via the instructions linked on their documentation page here http://docs.buddybuild.com/docs/common-react-native-errors#section-error-fbsdksharekit-fbsdksharekit-h-file-not-found, in particular this section:

error: 'FBSDKShareKit/FBSDKShareKit.h' file not found
This is generally a result of an incorrectly configured repository with
regards to the location of FBSDK dependencies. If you take a look at:
https://github.com/facebook/react-native-fbsdk/blob/master/ios/RCTFBSDK.xcodeproj/project.pbxproj
You will notice that RCTFBSDK will look for dependencies in one of two
locations:

~/Documents/FacebookSDK
$(PROJECT_DIR)/../../../ios/Frameworks

The second option is the correct option for continuous integration systems
like Buddybuild. In other words, you MUST place your FBSDK
dependencies under the "ios/Frameworks" folder in order for it to work
on a continuous integration system.

The problem is that as Viktor pointed out the CI system doesn't have a reference to your FacebookSDK folder containing the required Frameworks, so you need to actually copy the Frameworks themselves into your project itself so that everything is contained in your repository. This is necessary because the RCTFBSDK library expects to find the Frameworks that it requires in precisely the folder $PROJECT_ROOT/ios/Frameworks within your project (see the BuddyBuild documentation), and if they're anywhere else it will freak out.
For clarity the steps that I took to get my build working were:

Create the folder $PROJECT_ROOT/ios/Frameworks in my project (Frameworks may not already exist)
Copy the Facebook framework files from ~/Documents/FacebookSDK into this new folder
Reference the Frameworks in Xcode per the installation instructions on the fbsdk Github page.

Hope that this helps any future readers.
Answer: See: https://discuss.bitrise.io/t/build-failing-with-file-not-found-fbsdk-in-ios-build/399/11
It seems that you have the FacebookSDK on your Mac at ~/Documents/FacebookSDK, but you don't download it there on bitrise.io, so it's simply not available for your build.
Comment: How can we download and extract to bitrise?
Comment: @WitaloBenicio the way https://stackoverflow.com/a/44321452/974381 explained it (I'd suggest this solution, to include it in your repository), or via a step like https://www.bitrise.io/integrations/steps/resource-archive which can download a ZIP file and extract it to the path you provide.
Answer: I was using ZIP Unarchive step in Bitrise with the following path: ~/Documents/FacebookSDK and I was not able to make it work.
When I changed the Framework Search Path to ${HOME}/Documents/FacebookSDK I was able to continue my build with success.
Comment: `${HOME}/Documens/FacebookSDK`, I assume you mean `Documents`? That typo might trip people up a bit.
Comment: Thanks @Bataleonand

React Native FBSDK file not found on CI


-1 error message while trying to run H2O XGboost

Question: I am running H2O 18.104.22.168 (on  hadoop cdh5.8) on a remote server, connected to my local machine using R (3.4.1) with 'h2o.init(startH2O=F, port=54321)'.
The code below works fine:
Train.hex <- as.h2o(iris)
h2o.gbm(x="Sepal.Length", y="Petal.Width", training_frame = Train.hex)

but xgboost does not work and generates the following error message:
h2o.xgboost(x="Sepal.Length", y="Petal.Width", training_frame = Train.hex)

ERROR: Unexpected HTTP Status code: 500 Server Error (url = http://localhost:54321/3/ModelBuilders/xgboost)

java.lang.ArrayIndexOutOfBoundsException
 [1] "java.lang.ArrayIndexOutOfBoundsException: -1"
 [2] "    hex.ModelBuilder.make(ModelBuilder.java:117)"
 [3] "    water.api.ModelBuildersHandler.fetch(ModelBuildersHandler.java:35)"
 [4] "    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)"
 [5] "    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)"
 [6] "    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)"
 [7] "    java.lang.reflect.Method.invoke(Method.java:498)"
 [8] "    water.api.Handler.handle(Handler.java:63)"
 [9] "    water.api.RequestServer.serve(RequestServer.java:448)"
[10] "    water.api.RequestServer.doGeneric(RequestServer.java:297)"
[11] "    water.api.RequestServer.doGet(RequestServer.java:221)"
[12] "    javax.servlet.http.HttpServlet.service(HttpServlet.java:707)"
[13] "    javax.servlet.http.HttpServlet.service(HttpServlet.java:820)"
[14] "    org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)"
[15] "    org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)"
[16] "    org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)"
[17] "    org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)"
[18] "    org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)"
[19] "    org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)"
[20] "    org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"
[21] "    org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"
[22] "    water.JettyHTTPD$LoginHandler.handle(JettyHTTPD.java:183)"
[23] "    org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"
[24] "    org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"
[25] "    org.eclipse.jetty.server.Server.handle(Server.java:366)"
[26] "    org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)"
[27] "    org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)"
[28] "    org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)"
[29] "    org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)"
[30] "    org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)"
[31] "    org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)"
[32] "    org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)"
[33] "    org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)"
[34] "    org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)"
[35] "    org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)"
[36] "    java.lang.Thread.run(Thread.java:745)"

Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = page,  :

ERROR MESSAGE:

-1

Can anyone help me to understand what is going on?
Many thanks in advance,
Kere
Comment: What does h2o.xgboost.available() return?
Comment: I read that Windows is listed as not supported yet; (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/xgboost.html)
Answer: Unfortunately, H2O Xgboost is not supported for the time being on Windows as also the official documentation of H2O states:

Limitations
This section provides a list of XGBoost limitations - some of which
  will be addressed in a future release.
...

XGBoost is not supported on Windows.

...
Answer: XGBoost is not supported on Windows.

The list of limitations include:

XGBoost is not supported on Windows.
The list of supported platforms includes:Linux or OS X

When I met same issue I open my COLAB account and I try there. It did work for me.


Why are random walks intercorrelated?

Question: I have observed that, on average, the absolute value of Pearson correlation coefficient is a constant close to 0.560.42 for any pair of independent random walks, regardless of the walk length.
Can someone explain this phenomenon?
I expected the correlations to get smaller as the walk length increases, like with any random sequence.
For my experiments I used random gaussian walks with step mean 0 and step standard deviation 1.
UPDATE:
I forgot to center the data, that's why it was 0.56 instead of 0.42.
Here is the Python script to compute the correlations:
import numpy as np
from itertools import combinations, accumulate
import random

def compute(length, count, seed, center=True):
    random.seed(seed)
    basis = []
    for _i in range(count):
        walk = np.array(list(accumulate( random.gauss(0, 1) for _j in range(length) )))
        if center:
            walk -= np.mean(walk)
        basis.append(walk / np.sqrt(np.dot(walk, walk)))
    return np.mean([ abs(np.dot(x, y)) for x, y in combinations(basis, 2) ])

print(compute(10000, 1000, 123))

Comment: Are you talking about correlations *between* random walks (across series not within one series)? If so, it's because your independent random walks are integrated but not cointegrated, which is a well-known situation where spurious correlations will appear.
Comment: If you take a first difference, you will find no correlation. The lack of stationarity is the key here.
Comment: This is not just any "random sequence": the correlations are extremely high, because each term is just one step away from the preceding one. Note, too, that the correlation coefficient you are computing is not that of the random variables involved: it's a correlation coefficient for the sequences (thought of simply as paired data), which amounts to a big formula involving various squares and differences of all the terms in the sequence.
Comment: My first thought is that as the walk gets longer it is possible to get values with a larger magnitude, and the correlation is picking up on that.
Comment: But this would work with any random sequence, if I understand you right, yet only the random walks have that constant correlation.
Comment: What are you doing to generate the `.56` number? I don't see why the absolute value of the sample Pearson correlation coefficient should converge at all. (And in my testing, it doesn't.)
Comment: Most probably, you  are using pseudo-random numbers  generated by an algorithm and started with a seed. Is that seed always the same or does it change? And you may have to specify further what you mean by the correlation: Do you collect all possible points where a random walk has been or do you have a lot of random walks and collect their endpoints?
Comment: If OP was re-seeding with the same seed, the series would be identical, not just correllated.
Answer: Your independent processes are not correlated! If $X_t$ and $Y_t$ are independent random walks:

A correlation coefficient unconditional on time does not exist. (Don't talk about $\operatorname{Corr}(X, Y)$.)
For any time $t$, $\operatorname{Corr}(X_t, Y_t)$ is indeed 0.
But sample statistics based upon time-series averages will not converge to anything! The sample correlation coefficient you calculated based upon averaging multiple observations over time is meaningless.

Intuitively, you might guess (incorrectly) that:

Independence between two processes $\{X_t\}$ and $\{Y_t\}$ implies they have zero correlation. (For two random walks, $\operatorname{Corr}(X, Y)$ doesn't exist.)
The time series, sample correlation $\hat{\rho}_{XY}$ (i.e. the correlation coefficient calculated using time-series, sample statistics such as $\hat{\mu_X} = \frac{1}{T} \sum_{\tau = 1}^T X_\tau$) will converge on the population correlation coefficient $\rho_{XY}$ as $T \rightarrow \infty$.

The problem is that neither of these statements are true for random walks! (They are true for better behaved processes.)
For non-stationary processes:

You can talk about the correlation between processes $\{X_t\}$ and $\{Y_t\}$ at any two particular points of time (eg. $\operatorname{Corr}(X_2, Y_3)$ is a perfectly sensible statement.)
But it doesn't make sense to talk about correlation between the two
series unconditional on time! $\operatorname{Corr}(X, Y)$ does not have a well-defined meaning.

The problems in the case of a random walk?

For a random walk, unconditional population moments (i.e. which don't depend on time $t$), such as $\operatorname{E}[X]$, don't exist. (In some loose sense, they are infinite.) Similarly, the unconditional correlation coefficient $\rho_{XY}$ between two independent random walks isn't zero; it in fact doesn't exist!
The assumptions of ergodic theorems don't apply and various time-series averages (eg. $\frac{1}{T} \sum_\tau X_\tau$) don't converge towards anything as $T \rightarrow \infty$.

For a stationary sequence, the time series average will eventually converge on the mean that's unconditional on time. But for a non-stationary sequence, there is no mean that's unconditional on time!

If you have various observations of two independent random walks over time (eg. $X_1$, $X_2$, etc... and $Y_1$, $Y_2$, ....) and you calculate the sample correlation coefficient, you will get a number between $-1$ and $1$. But it won't be an approximation of the population correlation coefficient (which doesn't exist).
Instead, $\hat{\rho}_{XY}(T)$ (calculated using time-series averages from $t=1$ to $t=T$) is going to basically be a random variable (taking values in $[-1, 1]$) which reflects the two particular paths the random walks took by chance (i.e. the paths defined by the draw $\omega$ drawn from sample space $\Omega$.) Speaking extremely loosely (and imprecisely):

If both $X_t$ and $Y_t$ happened to wander off in the same direction, you'll detect a spurious positive relationship.
If $X_t$ and $Y_t$ wandered off in different directions, you'll detect a spurious negative relationship.
If $X_t$ and $Y_t$ happened to wander across each other enough, you'll detect a near zero relationship.

You can Google more about this with the terms spurious regression random walk.
A random walk isn't stationary and taking averages over time $t$ won't converge on what you would get by taking iid draws $\omega$ from in sample space $\Omega$. As mentioned in the comments above, you can take first differences $\Delta x_t = x_t - x_{t-1}$ and for a random walk, that process $\{\Delta x_t\}$ is stationary.
Big picture idea:
Multiple observations over time IS NOT the same as multiple draws from a sample space!
Recall that a discrete time stochastic process $\{ X_t \}$ is a function of both time ($t \in \mathbb{N}$) and a sample space $\Omega$.
For averages over time $t$ to converge towards expectations over a sample space $\Omega$, you need stationarity and ergodicity. This is a core issue in much time-series analysis. And a random-walk isn't a stationary process.
Connection to WHuber's answer:
If you can take averages across multiple simulations (i.e. take multiple draws from $\Omega$) instead of being forced to take averages across time $t$, a number of your issues disappear.
You can of course define $\hat{\rho}_{XY}(t)$ as the sample correlation coefficient computed on $X_1\ldots X_t$ and $Y_1 \ldots Y_t$ and this will also be a stochastic process.
You can define some random variable $Z_t$ as:
$$Z_t = |\hat{\rho}_{XY}(t)|$$
For two random walks starting at $0$ with $\mathcal{N}(0,1)$ increments, it's easy to find $E[Z_{10000}]$ by simulation (i.e. taking multiple draws from $\Omega$.)
Below, I ran a simulation of 10,000 calculations of a sample Pearson correlation coefficient. Each time I:

Simulated two 10,000 length random walks (with normally distributed increments draw from $\mathcal{N}(0,1)$).
Calculated the sample correlation coefficient between them.

Below is a histogram showing the empirical distribution over the 10000 calculated correlation coefficients.

You can clearly observe that the random variable $\hat{\rho}_{XY}(10000)$ can be all over the place in the interval $[-1, 1]$. For two fixed paths of $X$ and $Y$, the sample correlation coefficient doesn't converge to anything as the length of the time series increases.
On the other hand, for a particular time (eg. $t=10,000$), the sample correlation coefficient is a random variable with a finite mean etc...
If I take the absolute value and compute the mean over all the simulations, I calculate approximately .42. I'm not sure why you want to do this or why this is at all meaningful??, but of course you can.
Code:
for i=1:10000
  X = randn(10000,2);
  Y = cumsum(X);
  z(i) = corr(Y(:,1), Y(:,2));
end;
histogram(z,20);
mean(abs(z))

Comment: @whuber Do you mean for fixed time-series length $t$, everything is finite? (yes I agree with that.) The expectation of the sample correlation is zero (yes, I agree with that). As $t$ increases though, the sample correlation though does *not* converge on a single point. For two random walk-segments of arbitrary length, the sample correlation coefficient isn't that far from a random draw from the uniform distribution on [0, 1] (see histogram).
Comment: Since the sample size obviously is not finite, your assertions about various quantities not existing are puzzling.  It's difficult to see how your symbols apply to the situation described by the OP.
Comment: Your sample size **NEVER EVER** goes to infinity! Not as long as you are drawing samples with a computer, (*only in pure math you may make such assumptions*). And what does that mean: Because you have infinitely many points it does not converge? Where did you read that?
Comment: @whuber Hopefully this version is a bit clearer. I take it the OP is asking why the sample correlation coefficient (based upon time-series averages) between two finite segments of random walks isn't zero, even for time-series of immense length.

A fundamental problem is that for a random walk, various population moments don't exist and time-series averages don't converge to anything.
Comment: Nevertheless, for fixed $n$ everything is finite.  Moreover, the expectation of the absolute sample correlation coefficient *does* converge as $n$ increases! Note, too, that the question concerns the *absolute value* of that coefficient.  Its expectation (obviously) is zero.
Comment: What might be confusing is the OP's assertion that the absolute correlation coefficient is constant: it's not; it cannot possibly be.  But it does rapidly converge to a (non-uniform) distribution and its expectation is close to what the OP observes.
Comment: So is it .42 or .56? You have .56 in your update, but the OP now says it's .42 and you also wrote .42 in a comment to whuber's answer.
Comment: @amoeba Thanks for the careful reading! My Monte-Carlo calculation was .42.
Comment: @MatthewGunn +1 for mentioning stationarity, etc. and showing that averaging the results of multiple calculations is much different from a single case where there is absolutely no convergence no matter how large $n$ we choose :)
Answer: The math needed to obtain an exact result is messy, but we can derive an exact value for the expected squared correlation coefficient relatively painlessly.  It helps explain why a value near $1/2$ keeps showing up and why increasing the length $n$ of the random walk won't change things.
There is potential for confusion about standard terms.  The absolute correlation referred to in the question, along with the statistics that make it up--variances and covariances--are formulas that one can apply to any pair of realizations of random walks.  The question concern what happens when we look at many independent realizations.  For that, we need to take expectations over the random walk process.

(Edit)
Before we proceed, I want to share some graphical insights with you.  A pair of independent random walks $(X,Y)$ is a random walk in two dimensions.  We can plot the path that steps from each $(X_t,Y_t)$ to $X_{t+1},Y_{t+1}$.  If this path tends downwards (from left to right, plotted on the usual X-Y axes) then in order to study the absolute value of the correlation, let's negate all the $Y$ values.  Plot the walks on axes sized to give the $X$ and $Y$ values equal standard deviations and superimpose the least-squares fit of $Y$ to $X$.  The slopes of these lines will be the absolute values of the correlation coefficients, lying always between $0$ and $1$.
This figure shows $15$ such walks, each of length $960$ (with standard Normal differences).  Little open circles mark their starting points.  Dark circles mark their final locations.

These slopes tend to be pretty large.  Perfectly random scatterplots of this many points would always have slopes very close to zero.  If we had to describe the patterns emerging here, we might say that most 2D random walks gradually migrate from one location to another.  (These aren't necessarily their starting and endpoint locations, however!)  About half the time, then, that migration occurs in a diagonal direction--and the slope is accordingly high.
The rest of this post sketches an analysis of this situation.

A random walk $(X_i)$ is a sequence of partial sums of $(W_1, W_2, \ldots, W_n)$ where the $W_i$ are independent identically distributed zero-mean variables. Let their common variance be $\sigma^2$.
In a realization $x = (x_1, \ldots, x_n)$ of such a walk, the "variance" would be computed as if this were any dataset:
$$\operatorname{V}(x) = \frac{1}{n}\sum (x_i-\bar x)^2.$$
A nice way to compute this value is to take half the average of all the squared differences:
$$\operatorname{V}(x) = \frac{1}{n(n-1)}\sum_{j \gt i} (x_j-x_i)^2.$$
When $x$ is viewed as the outcome of a random walk $X$ of $n$ steps, the expectation of this is
$$\mathbb{E}(\operatorname{V}(X)) =  \frac{1}{n(n-1)}\sum_{j \gt i} \mathbb{E}(X_j-X_i)^2.$$
The differences are sums of iid variables,
$$X_j - X_i = W_{i+1} + W_{i+2} + \cdots + W_j.$$
Expand the square and take expectations.  Because the $W_k$ are independent and have zero means, the expectations of all cross terms are zero.  That leaves only terms like $W_k$, whose expectation is $\sigma^2$.  Thus
$$\mathbb{E}\left((X_j - X_i)^2\right) =\mathbb{E}\left((W_{i+1} + W_{i+2} + \cdots + W_j)^2\right)= (j-i)\sigma^2.$$
It easily follows that
$$\mathbb{E}(\operatorname{V}(X)) =  \frac{1}{n(n-1)}\sum_{j \gt i} (j-i)\sigma^2 = \frac{n+1}{6}\sigma^2.$$
The covariance between two independent realizations $x$ and $y$--again in the sense of datasets, not random variables--can be computed with the same technique (but it requires more algebraic work; a quadruple sum is involved).  The result is that the expected square of the covariance is
$$\mathbb{E}(\operatorname{C}(X,Y)^2) = \frac{3n^6-2n^5-3n^2+2n}{480n^2(n-1)^2}\sigma^4.$$
Consequently the expectation of the squared correlation coefficient between $X$ and $Y$, taken out to $n$ steps, is
$$\rho^2(n) = \frac{\mathbb{E}(\operatorname{C}(X,Y)^2)}{\mathbb{E}(\operatorname{V}(X))^2} = \frac{3}{40}\frac{3n^3-2n^2+3n-2}{n^3-n} = \frac{9}{40}\left(1+O\left(\frac{1}{n}\right)\right).$$
Although this is not constant, it rapidly approaches a limiting value of $9/40$.  Its square root, approximately $0.47$, therefore approximates the expected absolute value of $\rho(n)$ (and underestimates it).

I am sure I have made computational errors, but simulations bear out the asymptotic accuracy.  In the following results showing the histograms of $\rho^2(n)$ for $1000$ simulations each, the vertical red lines show the means while the dashed blue lines show the formula's value.  Clearly it's incorrect, but asymptotically it is right.  Evidently the entire distribution of $\rho^2(n)$ is approaching a limit as $n$ increases.  Similarly, the distribution of $|\rho(n)|$ (which is the quantity of interest) will approach a limit.

This is the R code to produce the figure.
f <- function(n){
  m <- (2 - 3* n + 2* n^2 -3 * n^3)/(n - n^3) * 3/40
}
n.sim <- 1e4
par(mfrow=c(1,4))
for (n in c(3, 10, 30, 100)) {
  u <- matrix(rnorm(n*n.sim), nrow=n)
  v <- matrix(rnorm(n*n.sim), nrow=n)
  x <- apply(u, 2, cumsum)
  y <- apply(v, 2, cumsum)
  sim <- rep(NA_real_, n.sim)
  for (i in 1:n.sim)
    sim[i] <- cor(x[,i], y[,i])^2
  z <- signif(sqrt(n.sim)*(mean(sim) - f(n)) / sd(sim), 3)
  hist(sim,xlab="rho(n)^2", main=paste("n =", n), sub=paste("Z =", z))
  abline(v=mean(sim), lwd=2, col="Red")
  abline(v=f(n), col="Blue", lwd=2, lty=3)
}

Comment: An asymptotic analysis of the issues discussed here may be found in [Phillips (1986), Theorem 1e](https://ideas.repec.org/p/cwl/cwldpp/757.html).
Comment: +1 but what is the intuition about why there is this positive asymptotic value $9/40$, whereas naively one would expect that if one takes two very long random walks they should have near-zero correlation, i.e. naively one would expect the distribution of correlations to shrink to zero as $n$ grows?
Comment: My Monte-Carlo simulation based estimate of $E[\rho^2]$ for $T = 100$ is about .24 (which appears to agree with your results). I agree with your analysis here. You might be getting at how the OP come to his number (though I calculate about .42, not .56).
Comment: If you can take repeated draws from $\Omega$, there's nothing particularly special about time-series analysis. Issues (eg. ergodicity, stationarity etc...) develop when you can only observe new values of $X$ by advancing time $t$ which I assumed was what the OP was trying to get at... (but maybe not).
Comment: @amoeba First, I don't fully believe the value of $9/40$, but I know it's close to correct.  For the intuition, consider that two independent walks $X_t$ and $Y_t$ are a random walk $(X_t,Y_t)$ in two dimensions.  Take *any* random scatterplot in 2D and measure its eccentricity somehow.  It will be rare for it to be perfectly circular.  Thus, we expect the mean eccentricity to be positive. That there is a limiting distribution for random walks merely reflects the self-similar "fractal" nature of this 2D walk.
Comment: The intuition that I have is as follows: any given random walk will not fluctuate around zero, it will tend to grow as $\sqrt{n}$, this is a very well-known result. So if we have two random walks, both will typically "grow" ("fan out") as a square-root and hence might very well have large absolute correlation coefficient, no matter what $n$ is. Do you think this intuition makes sense? Regarding 9/40: do you suspect a mistake in your formula for covariance? If not, what can be a source of the error?
Comment: @amoeba I found the immediate source of the error--a typographical error in a calculation!  However, even after fixing it I was unable to obtain results close to the simulation, suggesting there is a more basic error somewhere.  Although it's a routine calculation, it's messy and being off by just $1$ in an index somewhere will ruin it.  I haven't the time to fix it, unfortunately.  I wrote this answer hastily, hoping only that it might cast some light on the meaning of the original question and suggest some ways of analyzing the situation.
Comment: @whuber +1 for trying to calculate the formula, the scatterplots and showing the relationship between least-squares and correlation coefficient :) that helped me a lot!

Why are random walks intercorrelated?


google geochart set selectected country

Question: does anyone have idea how I can mark country from click?
I need somehowe when link is clicked to select that country on map as on hover.
I just dont understand how to use setSelection functon from documentation: https://developers.google.com/chart/interactive/docs/gallery/geochart#Example
Here is my code:

<title>Geochart</title>

        Germany

        United States

        Brazil

        Canada

        France

        Select continent:

        <select id="continents">
            <option value="002">Africa</option>
            <option value="142" selected="selected">Asia</option>
            <option value="150">Europe</option>
            <option value="021">Northern America</option>
            <option value="029">Caribbean</option>
            <option value="013">Central America</option>
            <option value="005">South America</option>
            <option value="009">Oceania</option>
        </select>

Also here is jsfiddle: http://jsfiddle.net/KS7L2/
Comment: It is unclear what you want to do. Please try to explain what your desired behavior is, and what is currently happening. Clicking on a country executes whatever event you have set. When you say you want to "select" the country, what does that mean?
Answer: Here is the sample code:
     google.visualization.events.addListener(table, 'select', function () {
                Geochart.setSelection(table.getSelection());

        var selection = table.getSelection();
        if (selection.length > 0) {
            var view = new google.visualization.DataView(data);
            view.setColumns([0, {
                type: 'number',
                label: data.getColumnLabel(1),
                calc: function (dt, row) {
                    return (selection[0].row == row) ? 2 : 1;
                }
            }]);
            Geochart.draw(view, options);
        }
        else {
            Geochart.draw(data, options);
        }
    });

    Geochart.draw(data, options);

    google.visualization.events.addListener(Geochart, 'select', function () {
                table.setSelection(Geochart.getSelection());

        var selection = Geochart.getSelection();
        if (selection.length > 0) {
            var view = new google.visualization.DataView(data);
            view.setColumns([0, {
                type: 'number',
                label: data.getColumnLabel(1),
                calc: function (dt, row) {
                    return (selection[0].row == row) ? 2 : 1;
                }
            }]);
            Geochart.draw(view, options);
        }
        else {
            Geochart.draw(data, options);
        }
    });

    Geochart.draw(data, options);
}

Here is the working sample jqfaq.com.
Hope this will help you.

google geochart set selectected country


What is the number of parameters when modeling a structural break?

Question: Lets assume that we have a data set with a structural break at some point $x$.
This can be tested using the Chow test.
To build a model that takes this break into account, I stimate two linear models for the two subintervals defined by the break point. Each model has $n$ coefficients. Now I want to calculate the AIC.
For this purpose, I need to calculate the total number of parameters of the "pooled" model. My first though was that the number of parameters is $2(n+1)$. However, I suspect that this is not true, because the separate models are estimated on a subset of the data. Am I right? What is the (effective) number of parameters of the "pooled" model?
Comment: Strictly speaking the parameters estimated include the variance parameters. Since you did two regressions with different variance estimates, both of those would be included, in addition to both intercepts and both slopes
Comment: I agree. However, as argued in my response one would obtain the same estimates in a single regression with interaction coding and the assumption of homoscedastic errors over the entire sample. In many applications this difference is practically not very relevant, though. But it may be...
Answer: If the breakpoint is known/given in advance and not estimated from the data, then you have simply twice the number of regression coefficients as in the original model. It does not matter that some regression coefficients are just influenced by a subset of the observations - that is the same as for any other dummy variable or interaction.
Moreover, if the original regression model is given by $X \beta$ with
$$ \left( \begin{array}{ccc} x_{1,1} & \dots & x_{1,k} \\ & \vdots & \\ x_{n,1} & \dots & x_{n,k} \end{array} \right) \left( \begin{array}{c} \beta_1 \\ \vdots \\ \beta_k \end{array} \right) $$
then the segmented model can also be written as a single regression in interaction coding:
$$ \left( \begin{array}{cccccc} x_{1,1} & \dots & x_{1,k} & 0 & \dots & 0 \\ & \vdots & & & \vdots & \\ x_{n_A,1} & \dots & x_{n_A,k} &0 & \dots & 0 \\ 0 & \dots & 0 & x_{n_A + 1,1} & \dots & x_{n_A + 1,k} \\ & \vdots & & & \vdots & \\ 0 & \dots & 0 & x_{n_A + n_B,1} & \dots & x_{n_A + n_B,k} \end{array} \right) \left( \begin{array}{c} \beta^A_1 \\ \vdots \\ \beta^A_k \\ \beta^B_1 \\ \vdots \\ \beta^B_k \end{array} \right) $$
So there are simply twice as many parameters ($2 \cdot k$ in my notation) as in the original model. A subtle issue is whether you assume the errors to be homoscedastic across the entire sample or just within each segment.
Things are different, of course, if the breakpoint is not known in advance and has to be estimated from the data.
Comment: Yes, it is either 2 * k + 1 in the case of constant variance and 2 * (k + 1) = 2 * k + 2 if the variance changes across the segments.
Comment: I wouldn't conduct a formal test but simply use the OLS estimate which is consistent under heteroscedasticity. Inference may have to be adjusted by suitable covariance estimators, though.
Comment: Thanks. So, if the error variance is not the same for both segments, we have to include the error variance as an additional parameter [ 2(k+1) in your notation]? This is of particular interest, if I am interested in comparing a nested model with a non-nested one.
Comment: Thank you very much. Please allow me a follow up question: Does it follow that in order to determine the number of parameters of an OLS models with segmentwise interaction coding I have to test if the error variance for each segment is the same? This could be problematic if I deal with a lot of segments.


Convergence of Stochastic Gradient Descent as a function of training set size

Question: I am going through the following section of the book by (Goodfellow et al., 2016), and I don't understand it quite well.

Stochastic gradient descent has many important uses outside the
  context of deep learning. It is the main way to train large linear
  models on very large datasets. For a fixed model size, the cost per SGD
  update does not depend on the training set size $m$. In practice, we often
  use a larger model as the training set size increases, but we are not
  forced to do so. The number of updates required to reach convergence
  usually increases with training set size. However,
  as $m$ approaches infinity, the model will eventually converge to its best
  possible test error before SGD has sampled every example in the
  training set. Increasing $m$ further will not extend the amount of training
  time needed to reach the model's best possible test error. From this
  point of view, one can argue that the asymptotic cost of training a
  model with SGD is $O(1)$ as a function of $m$.
Section 5.9, p.150

"The number of updates required to reach convergence usually increases with training set size". I can't get away around this one. In the normal gradient descent, it becomes computationally expensive to calculate the gradient at each step as the number of training examples increases. But I don't understand why the number of updates increases with the training size.
"However, as $m$ approaches infinity, the model will eventually converge to its best possible test error before SGD has sampled every example in the training set. Increasing $m$ further will not extend the amount of training time needed to reach the model's best possible test error." I don't understand this as well

Can you provide some intuition/ arguments for the above two cases?
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Answer: In the first part they are talking about large-scale SGD convergence in practice and in the second part theoretical results on the convergence of SGD when the optimisation problem is convex.

"The number of updates required to reach convergence usually increases with training set size".
I found this statement confusing but as @DeltaIV kindly pointed out in the comments I think they are talking about practical considerations for a fixed model as the dataset size $m \rightarrow \infty$. I think there are two relevant phenomena:

performance tradeoffs when you try to do distributed SGD, or
performance on a real-world non-convex optimisation problem

Computational tradeoffs for distributed SGD
In a large volume and high rate data scenario, you might want to try to implement a distributed version of SGD (or more likely minibatch SGD). Unfortunately making a distributed, efficient version of SGD is difficult as you need to frequently share the parameter state $w$. In particular, you incur a large overhead cost for synchronising between computers so it incentivises you use larger minibatchs. The following figure from (Li et al., 2014) illustrates this nicely

SGD here is minibatch SGD and the other lines are algorithms they propose.
However it's also known that for convex problems minibatchs incur a computational cost which effectively slows convergence with increased minibatch size. As you increase minibatch size $m'$ towards the dataset size $m$ it becomes slower and slower until you're just doing regular batch gradient descent.  This decrease can be kinda drastic if a large mini-batch size is used.  Again this is illustrated nicely by (Li et al, 2014):

Here they are plotting the minimum objective value they found on the KDD04 dataset after using $10^7$ datapoints.
Formally, if $f$ is a strongly convex function with global optimum $w^*$, and if $w_t$ is the output of SGD at the $t^{th}$ iteration. Then you can prove that in expectation you have:
$$
\mathbb{E}[f(w_t) - f(w^*)] \leq \mathcal{O}(\frac{1}{\sqrt{t}}).
$$
Note that this doesn't depend on the dataset size (this is relevant to your next question)! However for minibatch SGD with batch size $b$ the convergence to the optimum happens at a rate of $\mathcal{O}(\frac{1}{\sqrt{bt}} + \frac{1}{bt})$. When you have a very large amount of data (Li et al., 2014) make the point that:

Since the total number of examples examined is $bt$ while there is only
  a $\sqrt{b}$ times improvement, the convergence speed degrades with
  increasing minibatch size.

You have a trade-off between the synchronisation cost for small mini-batches, and the performance penalty for increased minibatch size. If you naively parallelise (minibatch) SGD you pay some penalty and your convergence rate slows as the data size increases.
Nonconvexity of the empirical optimisation problem
This is basically the point that @dopexxx has made. It's pretty well-known that the optimisation problem that you want to solve for deep neural nets in practice is not convex. In particular, it can have the following bad properties:

local minima and saddle points
plateau regions
widely varying curvature

In general the shape you're trying to optimise over is thought to be a manifold and as far as I am aware all you can say about convergence is that you're going converge to (a noise ball around) a stationary point. It seems reasonable that the greater the variety of real world data, the more complicated this manifold will be. Because the real gradient is more complicated as $m$ increase you need more samples to approximate $\nabla f$ with SGD.

"However, as m approaches infinity, the model will eventually converge to its best possible test error before SGD has sampled every example in the training set. Increasing m further will not extend the amount of training time needed to reach the model's best possible test error."
(Goodfellow et al., 2016) state this a little more precisely in the discussion in section 8.3.1, where they state:

The most important property of SGD and the related minibatch or online gradient-based optimisation is that computation time per update does not grow with the number of training examples. This allows convergence even when the number of training examples becomes very large. For a large enough dataset, SGD may converge to within some fixed tolerance of its final test set error before it has processed the entire training dataset.

(Bottou et al., 2017) offer the following clear explanation:

On an intuitive level, SG employs information more efficiently than a
  batch method. To see this, consider a situation in which a training set, call it $S$, consists of ten
  copies of a set $S_{sub}$. A minimizer of empirical risk for the larger set S is clearly given by a minimizer
  for the smaller set $S_{sub}$, but if one were to apply a batch approach to minimize $R_n$ over $S$, then
  each iteration would be ten times more expensive than if one only had one copy of $S_{sub}$. On the
  other hand, SG performs the same computations in both scenarios, in the sense that the stochastic
  gradient computations involve choosing elements from $S_{sub}$ with the same probabilities.

where $R_n$ is the empirical risk (essentially the training loss). If you let the number of copies get arbitrarily large, then SGD will certainly converge before it's sampled every data point.
I think this agrees with my statistical intuition. You can achieve $|x^{\text{opt}} - x_k| < \epsilon$ at iteration $k$ before you necessarily sample all the points in your dataset, because your tolerance for error $\epsilon$ really determines how much data you need to look at, i.e. how many iterations of SGD you need to complete.
I found the first part of the paper by (Bottou et al., 2017) quite helpful for understanding SGD better.
References
Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." arXiv preprint arXiv:1606.04838 (2016).
https://arxiv.org/pdf/1606.04838.pdf
Li, Mu, et al. "Efficient mini-batch training for stochastic optimization." Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2014.
https://www.cs.cmu.edu/~muli/file/minibatch_sgd.pdf
Comment: +1 for citing Bottou's paper, a reference for SGD. I suggest you improve the first part of your answer - when we increase the training set size $m$, we are not adding any parameter. The gradient is over weights, not over training samples, and the number of weights is not changing in Goodfellow's example ("[..]For a fixed model size[..]"). Thus, the reason we're slowing down is not that we have more parameters to fit: the number of parameters stays the same.
Comment: Honestly, I am puzzled by that first part hence why I'm offering a couple of different interpretations. When I quoted the passage to a couple of people they took it to mean "In practice, we often use a larger model as the training set size increases (but we are not forced to do so) **thus** the number of updates required to reach convergence usually increases with training set size.". Do you think that's not a reasonable interpretation and I should eliminate that bit? I'm aware my answer is a bit verbose.
Comment: I don't think it's a reasonable interpretation: the textbook has very clearly specified that it's considering the "fixed model size" case. Whether or not that's what one would do in practice, is a different question: here we're answering the first question (fixed model size, training set size $m\to\infty$).
Comment: Yup, good point. I've tried to remove that explanation.
Answer:
In SGD you use mini-batches of fixed size, let's call it $d$. So if you have for example: $m=1024$ and $d=8$, you need 128 updates to go thru the whole training set. If you increase the training set size to $m=2048$, you will need 256 updates. This means that the number of updates increased.
Another thing is, that if there is more data, then you need more time to converge (because model has to learn more). Hopefully you should also converge to a better point.
So the authors want to say that more time in this case means more updates (as contrary to GD, where the number of updates stays the same, but they are longer).
It just says that adding data to train set will not infinitively increase your model performance. In practice it means that if your training set is very big, performance of your model is rather limited by the model itself, not by the size of your training set.
Answer: The fact that as $m\to\infty$, the number of iterations $N$ needed for a deterministic optimization algorithm to reach the same loss value increases, is quite intuitive. Remember that Goodfellow et al. argument is made

For a fixed model size[..]

This means that you're increasing $m$, but the number of weights $n_w$ stays the same. It's clear that using a model of fixed complexity, it becomes increasingly harder to fit a larger training set.
The second point, i.e., the fact that instead $N$ for SGD reaches a maximum and then plateaus, can be understood in the following way. As $m$ grows indefinitely but the model stays the same, you'll reach a point where $m$ is sufficienty larger than the model capacity (let's say, once $m$ is enough larger than the VC dimension of the model). At this point we don't even need to examine all the training set examples, before the model has reached its minimum test set error, because the model isn't able to overfit the training set anyway. Showing the model more random examples from the training set (which is what SGD does) will only make the training set error and the test set error oscillate around a minimum value, but neither of them will decrease  significantly.
Answer: What I think it means is that the number of GD iterations required does not grow linearly with $m$. Imagine you are trying to estimate some $\theta$ parameter, with only one data vector $x$: You start with a guess, then have a better estimation based on $x_1$. next, you have even better estimation, based on $x_1$ and $x_2$. The more observations you have, the closer you get to the target. As you have more and more observations, the weight of each observation is getting smaller and thus the relative impact of each additional observation shrinks. Assuming you have no outliers (wait a sec with this), you either converge to target or get to tiny fluctuations within a number of steps which is almost constant. If you do have outliers then a surge would be observed but afterwards you'll go back to converging towards target (each outlier would require an amount of additional steps to cover for). That's as far as I can go with intuition, based on both the model and my experience with it. However, I can't provide you with sources.
Answer:
A larger training set almost inevitably implies greater variability of
examples. Thus, pulling random batches during training will yield,
on average, a higher entropy (across batches, not within). And so,
the function you try to approximate gets more complex and you need
to train longer in order to converge.
Imagine, you want to solve a standard classification problem such as recognizing animals from images. Let's say you dataset comprises $10^9$ images and imagine this set of images $1cm$ high stack. The amount of discernibly different images is in the range of $10^{40}$ with the stack being $9.98 \times 10^{25} km$, which extends the diameter of the universe which is around $8.85 \times 10^{23} km$. (I saw this calculation at ICCV 2017; for details see slide no. 3 here). Now, imagine you train on this vast dataset ($m:=10^{40}$). Even if it would be computationally feasible, at some point it would not further help you to classify your animal images correctly. That means, while training you would realize that after having seen only $n<m$ samples your algorithm does not further improve and since
$n\ll m$, you can validly argue that asymptotic training cost is in $\mathcal{O}(1)$
Answer: I think for this question, you need to understand the definition of convergence. Generally, the convergence defined by the number of steps required to reach a certain threshold (e.g., the norm of gradient). SGD will go through the dataset once, as the dataset size increase, the solution will become more accurate. Once reaching a certain point, more data will not help too much because of the convergence property of SGD. There is also another issue regarding this optimization problem, SGD only cares about error bound of samples, but we actually care about the generalization capability. So two problems are not equivalent.


Can I use indexes for speeding up select queries over dateTime ranges in BaseX?

Question: I would like to ask about the way to speed up the select queries in a Basex database.
 I have for example the following xml in a database with many events(650000 approximately)
<EventList>
    <Event>
        <ID>317849</ID>
        <Type>Measurement</Type>
        <TimeStamp>2016-03-15T18:00:09.409</TimeStamp>
        <Space>BIOCAT</Space>
        <SourceID>BIOCAT.TE310A</SourceID>
        <Content>
            <Measurement>
                <value>920</value>
            </Measurement>
        </Content>
    </Event>
    <Event>
        <ID>317850</ID>
        <Type>Measurement</Type>
        <TimeStamp>2016-03-15T18:05:09.409</TimeStamp>
        <Space>BIOCAT</Space>
        <SourceID>BIOCAT.TE310A</SourceID>
        <Content>
            <Measurement>
                <value>920</value>
            </Measurement>
        </Content>
    </Event>
</EventList>

I am retrieving the events with the following code that selects with respect to the datetime of the Timestamp node
for $b in doc('mydb/my.xml')//EventList/Event
let $date_string as xs:string := xs:string($b/TimeStamp/data())
let $date as xs:dateTime := xs:dateTime($date_string)
where $date ge xs:dateTime('"+startdate+"')
  and $date le xs:dateTime('"+enddate+"')
  and $b/Type='"+EventType+"'
return $b

But it it is very slow it makes one minute to return 60 events.
There are many data in the BaseX database.
How can I speed up the request or my database?
Comment: If you'd properly indent the XML and add some reasonable line wraps to your XQuery code before posting, chances will heavily increase somebody will read your code. Furthermore, "many" is not a reasonable quantification, give the number of documents and nodes instead to provide some arguable numbers, same applies to "slow". Finally, why are you generally writing titles all lower-case, while getting it right in the question body?
Answer: BaseX currently does not have a range index for xs:dateTime, but you can use the text index for getting all events with a given event Type by moving the comparison into the XPath:
for $b in //EventList/Event[Type = 'Measurement']
let $date as xs:dateTime := xs:dateTime($b/TimeStamp)
where $date ge xs:dateTime('2016-03-15T18:00:00.000')
  and $date le xs:dateTime('2016-03-15T19:00:00.000')
return $b

In the Info View of the GUI you can see that the text index is applied:

Compiling:

rewriting descendant-or-self step(s)
applying text index for "Measurement"
pre-evaluating "2016-03-15T18:00:00.000" cast as xs:dateTime
pre-evaluating "2016-03-15T19:00:00.000" cast as xs:dateTime

Optimized Query:
for $b_0 in db:text("mydb/my.xml", "Measurement")
        /parent::*:Type/parent::*:Event[parent::*:EventList]
let $date_1 as xs:dateTime := $b_0/TimeStamp cast as xs:dateTime?
where (($date_1 ge "2016-03-15T18:00:00")
  and ($date_1 le "2016-03-15T19:00:00"))
return $b_0

Comment: I have tested the first query and it goes faster.
I didn't understand the explantation. why does it go faster?
Where is the info view of the GUI?
Sory I am a little bit confused.
Another question is if the optimized query will go faster than your first query and why.
Thank you in advance
Comment: I also get a
"invalid database name"
with the optimized query
Comment: Your original query scans over all `Event` nodes in the database and checks if they satisfy the three conditions. If an up-to-date *text index* exists this can be improved by (1) finding all text nodes in the database containing the string `"Measurement"` and then (2) going upwards in the document hierarchy to find the associated `Event`s. By reformulating the query in the way I did it, I helped the optimizer to find the second strategy. You can also use [db:text(...)](http://docs.basex.org/wiki/Database_Module#db:text) directly if you want. The Info View can be toggled by pressing `Strg + I`.
Comment: so the optimized query will be faster than the first query that you have posted?
and could you please give me a way to surpass the error from the db:text("mydb/my.xml".....?
I get "invalid database name" error.
If I write
db:text("mydb"
I get
"mydb has no text index"
Comment: No, Basex's query optimizer will rewrite the first query into the second one automatically. Do you read the XML data from a file each time or do you have it stored in a database? `db:text(...)` needs a database name as its first argument. But as I said, it should not be necessary to use it explicitly because the optimizer automatically rewrites my first query to use it.
Comment: The weird thing is that some times the request gets the response immediately and sometimes it has to wait 1.5 minute.
Is there  a possibility that the connections to the database do not close immediately?
session.close();
I use final BaseXClient session = new BaseXClient("localhost", 1984, "..","..");
Comment: That is unrelated to your original question, please create a new one for this.


RegEx in javascript. Allow only letters, comma and punctuation

Question: I try to make a RegEx for validating a form in javascript. The RegEx should only allow letters comma and punctuation. For instance I have this string:
Hi, this is a test of RegEx. Does it work

I've tried the following
/^[A-Za-z0-9,. ]{3,50}$/;

But it doesn't seems to work. Solutions?
Thanks!
EDIT:
This is my code:

    Receipt<input type="checkbox"     name="receipt" value="1"/>

<input type="button" value="Post it" onclick="checkSsn(this.form.content.value);"/>

Comment: Works for me. Please show us your actual code.
Comment: Does `this.form.content.value` actually contain anything?
Answer: You might need to escape the "." as that is a special character in regex.
/^[A-Za-z0-9,\. ]{3,50}$/;

Actually probably not.  Try using http://www.regextester.com/  - I was able to get it to work anyway.  Can you show us the full code for how you're implementing this?
Comment: . doesn't need to be escaped in character sequences ([]), I believe only - and ^ do
Answer: if you want no punctuation at the beginning of the field:
/^[a-zA-Z]+(([\'\,\.\- ][a-zA-Z ])?[a-zA-Z]*)*$/

this also allows spaces (one scenario for this is last name - De La Hoya, or O'Doul)