Running scikit learn using dask_jobqueue on a SLURM cluster

Question: I have a 4xRaspberry pi3  SLURM cluster with a shared NFS folder. 4 workers (The master is also a worker buy using only 3 of its 4 cores)
The cluster is working ok (I have run some parallel python examples on it using mpiexec).
Now, I want to try a scikit-learn example, and some tutorials I saw were using DASK-jobqueue with SLURM.
My code looks something like this:
from dask_jobqueue import SLURMCluster

cluster = SLURMCluster( job_extra=['--partition=picluster'],
                        queue='myqueue',
                        cores=4,
                        memory='1GB'
                        )

cluster.scale(4) #the number of nodes to request

print(cluster.job_script())

from dask.distributed import Client
client = Client(cluster)

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import brier_score_loss

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, GridSearchCV

import numpy as np

#load the data from file
preds_trainval_file='./Predictions_TRAIN.csv'
outc_trainval_file = './Outcome_TRAIN.csv'
preds_test_file='./Predictions_TEST.csv'
outc_test_file = './Outcome_TEST.csv'

X_trainval= np.loadtxt(preds_trainval_file, delimiter=',')
y_trainval,_,_ = np.loadtxt(outc_trainval_file, delimiter=',' , usecols=(0, 1, 2), unpack=True)
X_test = np.loadtxt(preds_test_file, delimiter=',')
y_test,_ = np.loadtxt(outc_test_file, delimiter=',' , usecols=(0, 1), unpack=True)

#setup the classifier and perform cross validation
model = LogisticRegression( penalty='elasticnet', solver='saga', warm_start=True, max_iter=10000); param_grid= { 'l1_ratio' : [0, 0.25, 0.5, 0.75,  1],  'C':[0.1, 0.25, 0.5, 0.75, 1, 1.25]}

#setup grid search on the train+val data.
kfold = KFold(n_splits=5, shuffle=True)
grid_search = GridSearchCV(model, param_grid, cv=kfold,  scoring='neg_brier_score', n_jobs=-1)

import joblib

with joblib.parallel_backend('dask'):
  grid_search.fit(X_trainval, y_trainval)

y_prob=grid_search.predict_proba(X_test)
print(brier_score_loss(y_test, y_prob[:,0], pos_label=1))

From what I understand this is a pretty standard setup to exploit scikit's builtin parallelisation.
When  I run this script I get the following:
pi@node01:/clusterfs/Python_scripts/Expert_ensemble $ python3 ensemble_tests.py
#!/usr/bin/env bash

#SBATCH -J dask-worker
#SBATCH -p myqueue
#SBATCH -n 1
#SBATCH --cpus-per-task=4
#SBATCH --mem=954M
#SBATCH -t 00:30:00
#SBATCH --partition=picluster

/usr/bin/python3 -m distributed.cli.dask_worker tcp://192.168.1.10:38817 --nthreads 1 --nprocs 4 --memory-limit 250.00MB --name dummy-name --nanny --death-timeout 60 --protocol tcp://

Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /usr/lib/python3.7/asyncio/tasks.py:596> exception=RuntimeError('Command exited with non-zero exit code.\nExit code: 1\nCommand:\nsbatch /tmp/tmpz8a3jhys.sh\nstdout:\n\nstderr:\nsbatch: error: Memory specification can not be satisfied\nsbatch: error: Batch job submission failed: Requested node configuration is not available\n\n')>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/tasks.py", line 603, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/pi/.local/lib/python3.7/site-packages/distributed/deploy/spec.py", line 71, in _
    await self.start()
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 324, in start
    out = await self._submit_job(fn)
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 307, in _submit_job
    return self._call(shlex.split(self.submit_command) + [script_filename])
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 407, in _call
    "stderr:\n{}\n".format(proc.returncode, cmd_str, out, err)
RuntimeError: Command exited with non-zero exit code.
Exit code: 1
Command:
sbatch /tmp/tmpz8a3jhys.sh
stdout:

stderr:
sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x766f41b0>>, <Task finished coro=<SpecCluster._correct_state_internal() done, defined at /home/pi/.local/lib/python3.7/site-packages/distributed/deploy/spec.py:325> exception=RuntimeError('Command exited with non-zero exit code.\nExit code: 1\nCommand:\nsbatch /tmp/tmpc0ary0k1.sh\nstdout:\n\nstderr:\nsbatch: error: Memory specification can not be satisfied\nsbatch: error: Batch job submission failed: Requested node configuration is not available\n\n')>)
Traceback (most recent call last):
  File "/home/pi/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/home/pi/.local/lib/python3.7/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/home/pi/.local/lib/python3.7/site-packages/distributed/deploy/spec.py", line 360, in _correct_state_internal
    await w  # for tornado gen.coroutine support
  File "/home/pi/.local/lib/python3.7/site-packages/distributed/deploy/spec.py", line 71, in _
    await self.start()
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 324, in start
    out = await self._submit_job(fn)
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 307, in _submit_job
    return self._call(shlex.split(self.submit_command) + [script_filename])
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 407, in _call
    "stderr:\n{}\n".format(proc.returncode, cmd_str, out, err)
RuntimeError: Command exited with non-zero exit code.
Exit code: 1
Command:
sbatch /tmp/tmpc0ary0k1.sh
stdout:

stderr:
sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

Task exception was never retrieved
future: <Task finished coro=<_wrap_awaitable() done, defined at /usr/lib/python3.7/asyncio/tasks.py:596> exception=RuntimeError('Command exited with non-zero exit code.\nExit code: 1\nCommand:\nsbatch /tmp/tmp3sezvy1f.sh\nstdout:\n\nstderr:\nsbatch: error: Memory specification can not be satisfied\nsbatch: error: Batch job submission failed: Requested node configuration is not available\n\n')>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/tasks.py", line 603, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/pi/.local/lib/python3.7/site-packages/distributed/deploy/spec.py", line 71, in _
    await self.start()
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 324, in start
    out = await self._submit_job(fn)
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 307, in _submit_job
    return self._call(shlex.split(self.submit_command) + [script_filename])
  File "/home/pi/.local/lib/python3.7/site-packages/dask_jobqueue/core.py", line 407, in _call
    "stderr:\n{}\n".format(proc.returncode, cmd_str, out, err)
RuntimeError: Command exited with non-zero exit code.
Exit code: 1
Command:
sbatch /tmp/tmp3sezvy1f.sh
stdout:

stderr:
sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

I am not sure what I am doing wrong. Whether it is in the SLURMCluster configuration or something else.
The output of sinfo is:
PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
picluster*    up   infinite      4   idle node[01-04]

And the output of scontrol show nodes is:
scontrol show nodes
NodeName=node01 Arch=armv7l CoresPerSocket=1
   CPUAlloc=0 CPUTot=3 CPULoad=0.09
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=192.168.2.10 NodeHostName=node01 Version=18.08
   OS=Linux 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021
   RealMemory=1 AllocMem=0 FreeMem=800 Sockets=3 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=picluster
   BootTime=2021-02-20T05:49:48 SlurmdStartTime=2021-02-20T05:50:03
   CfgTRES=cpu=3,mem=1M,billing=3
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

NodeName=node02 Arch=armv7l CoresPerSocket=1
   CPUAlloc=0 CPUTot=4 CPULoad=0.27
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=192.168.2.11 NodeHostName=node02 Version=18.08
   OS=Linux 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021
   RealMemory=1 AllocMem=0 FreeMem=813 Sockets=4 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=picluster
   BootTime=2021-02-20T05:49:37 SlurmdStartTime=2021-02-20T05:50:10
   CfgTRES=cpu=4,mem=1M,billing=4
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

NodeName=node03 Arch=armv7l CoresPerSocket=1
   CPUAlloc=0 CPUTot=4 CPULoad=0.24
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=192.168.2.12 NodeHostName=node03 Version=18.08
   OS=Linux 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021
   RealMemory=1 AllocMem=0 FreeMem=821 Sockets=4 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=picluster
   BootTime=2021-02-20T05:49:37 SlurmdStartTime=2021-02-20T05:50:09
   CfgTRES=cpu=4,mem=1M,billing=4
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

NodeName=node04 Arch=armv7l CoresPerSocket=1
   CPUAlloc=0 CPUTot=4 CPULoad=0.14
   AvailableFeatures=(null)
   ActiveFeatures=(null)
   Gres=(null)
   NodeAddr=192.168.2.13 NodeHostName=node04 Version=18.08
   OS=Linux 5.10.11-v7+ #1399 SMP Thu Jan 28 12:06:05 GMT 2021
   RealMemory=1 AllocMem=0 FreeMem=813 Sockets=4 Boards=1
   State=IDLE ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=picluster
   BootTime=2021-02-20T05:49:40 SlurmdStartTime=2021-02-20T05:50:08
   CfgTRES=cpu=4,mem=1M,billing=4
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

As you can see only the master node (node01) has CPUTot=3 allocated. All the other nodes have the standard 4. But I also tested with reconfiguring the cluster with all the nodes to have the same CPUTot=4 and still got the same error when running the python script. In addition, I tried to only request 500MB of memory for each node in the cluster, but I still go the same error.
Any help appreciated.
Thanks
Answer: ok so I found a solution. I am not sure what the problem was, but you can override the memory issue by overriding the memory requirement using the header_skip option. So change the line from
cluster = SLURMCluster( job_extra=['--partition=picluster'],
                        queue='myqueue',
                        cores=4,
                        memory='1GB'
                        )

to
cluster = SLURMCluster( header_skip=['--mem'],
                        queue='picluster',
                        cores=4,
                        memory='1GB'
                        )

After that, it seems to work fine. But I still don't understand what the problem is/was.
Comment: If I set the SLURMCluster class to 1GB then I get 250mb for each worker. If I set it to 4GB then I get 1GB for each worker (the maximum memory for each Pi). Both of those configurations will fail unless I override the memory setting in the header. With the override it works fine but I haven't explored what memory it allocates when overriding
Comment: Odd, what is the memory per worker that you see with this `cluster`.


What is the point of notes lower than human hearing

Question: (Scientific pitch notation)
For example:

Organs can go down to C-1

Hyperbass Flute goes down to C0

And these notes have neglegible overtones on these instruments!
Do they modulate other notes?
They certainly can not be heard, as C-1 is ~8Hz, max overtones ~10-15 Hz (On Organ)
Comment: According to Wikipedia, the only two 64' (C -1) organ stops in the world are pipes that have plenty of overtones. Also, even though flutes are some of the most pure-toned acoustic instruments, that doesn't mean they are 100% pure toned.
Comment: Human hearing is a range that can differ between individuals by several orders of magnitude. There is no one note that you could pick as being the "last useful one".
Comment: Orthogonally, one does not find many 32' acoustic organ pipes in deadened recording studios or outdoors.  The room is part of the instrument and elite organists definitely know how different instruments will 'develop' in the space when those tones are used.
Answer: We hear with our ears, but other body parts are susceptible to vibrations.
The percussionist/drummer Evelyn Glennie is profoundly deaf, and apparently uses her bare feet to detect vibrations - which we probably all feel, but are not so aware of.
There's a new device on the market which incorporates earphones and (I suspect) a speaker which attaches to one's body producing sub-hearing vibrations.
Younger humans have a much wider hearing range than older ones, so maybe us olduns get to rely on more than just our ears - sympathetic vibrations of some organs works for me...
Comment: Delightfully unclear whether "some organs" refers to musical instruments or a heart flutter.
Comment: @Paul_Pedant - or perhaps one's  F# wit..?
Answer: If you listen to hyperbass flute recordings on Youtube, its sound has clearly higher harmonics that can be heard, and the keys have strong percussive sound. It is also possible that the fundamental tone is noticeable as vibration when the instrument is played live.
This is not that uncommon with other low pitched instruments: For example guitars, bass guitars and pianos have quite weak fundamental frequencies in the lower register but most of the energy is in higher harmonics. It's also really different from normal flute that has very strong fundamental frequency.
Comment: @ToddWilcox I'm not sure which "Yamaha grand piano" was used here, but have a look at https://www.acs.psu.edu/drussell/Piano/Dynamics.html
Comment: @Tim I think it would be accurate to say that those instruments *can* have weak fundamentals in their lowest notes, depending on the construction. For example, a spinet is just not going to have strong resonance in its soundboard for the fundamentals of its lowest notes. A 9' grand would be a different story.
Comment: https://www.jjburred.com/research/pdf/burred_acoustics_piano.pdf page 30 is also interesting, even if it quotes an older paper for the actual measurements
Comment: I think yes, in live performance, it may be heard indeed. But the question still remains... does it MODULATE the sound of an orchestra, for example?? Does that unheard C#0 or B-1 etc. from the hyperbass flute or pipe organ, respectively, influence how the higher/normal flute(s), the piano, or Horn sounds ???
Comment: @Feliks_WR No, it doesn't. Look up missing fundamental effect.
Comment: Hyperbass flute is a particularly extreme example. Flutes in general are relatively _poor_ in overtones, which is why their lower registers are mellow-sounding and easily drowned out. In addition, bigger flutes require larger and larger amounts of air. I haven't heard a hyperbass flute in person, but I'd wager at least the low notes when played straight are indeed inaudible unless you have your head or a microphone right at the instrument. It's pretty much an instrument made specifically for extended playing techniques – percussive effects, extreme overblowing etc..
Comment: Where did the 'weak fundamental frequency' information come from? I don't think it's true!
Comment: @Tim from reading acoustics textbooks and looking at spectrograms.
Comment: I think you meant to tag @Tim in your comment with a link to Yamaha piano spectra
Comment: Well, both of you. I don't believe that a huge concert grand would have that strong fundamental tones either, just less inharmonicity and overall louder.
Answer: You can't hear them as pitches, but that doesn't mean that you cannot sense them at all.  You can feel the vibrations in the air.


Duration of a MIDI file by parsing it and making a stream of parsed notes and chords do not playback the same length music. What is the problem here?

Question: I parsed a MIDI file in music21 by using converter.parse function in music21.
And for extracting notes and chords with duration, I have used the code as follows:
    score = converter.parse('midiexample.mid')
    flat = score.flat
    for el in flat.notes:
        if isinstance(el, music21.note.Note):
            pitch = el.pitch
            note_name = pitch.nameWithOctave
            new_note_c = note.Note(note_name)
            new_note_c.duration = el.duration
            dur = round(float(el.duration.quarterLength), 1)
            all_elements.append((str(note_name), dur))
        elif isinstance(el, music21.chord.Chord):
            chord_pitches = []
            for p in el.pitches:
                #note_name = p.pitch
                #note_name_oct = note_name.nameWithOctave
                new_note_c = note.Note(p.nameWithOctave)
                #new_note_c.duration = el.duration
                dur = round(float(el.duration.quarterLength), 1)
                chord_pitches.append((str(new_note_c.nameWithOctave)))
            chord_pitches = ".".join([i for i in chord_pitches])
            all_elements.append((chord_pitches, dur))

def get_music_midi_filename_from_chords(input_chords):
    midi_stream = stream.Stream()

    for note_pattern, duration_pattern in input_chords:
        if('.' in note_pattern):
            notes_in_chord = note_pattern.split('.')
            chord_notes = []
            for current_note in notes_in_chord:
                new_note = note.Note(current_note)
                new_note.duration = duration.Duration(duration_pattern)
                new_note.storedInstrument = instrument.Piano()
                chord_notes.append(new_note)
            new_chord = chord.Chord(chord_notes)
            print(f'New_chord appended as {new_chord}')
        else:
            note_single = note.Note(note_pattern)
            note_single.duration = duration.Duration(duration_pattern)
            note_single.storedInstrument = instrument.Piano()
            new_chord = note_single
            print(f'Appended note {new_chord}')
        midi_stream.append(new_chord)

        #new_tempo = tempo.MetronomeMark(number=50)

        #midi_stream.append(new_tempo)

    midi_stream = midi_stream.chordify()
    timestr = time.strftime("%Y%m%d-%H%M%S")
    new_file = 'output-' + timestr + '.mid'
    return midi_stream.write('midi', fp=new_file
)

When I call the function to get music, the MIDI file I produced has 2:51 time duration whereas the song I passed only has 1:16 of time duration. What am I doing wrong in this case? I need to make sure I am able to get the notes and chords along with their duration as correct as I can get. But an addition of a minute seems fishy. Is the duration rounding the cause? I am not able to debug this error.
Also, while comparing, I got the MIDI comparison of original MIDI and my produced MIDI and I found them as follows:

Original MIDI is dark blue, my produced one is longer. The notes are somewhat similar but the duration is prolonged in my code:
Comment: Don't any notes overlap?
Comment: Perhaps what I'm asking is not correct, but are you reading the tempo of the original file at all? I would think it could be done with https://web.mit.edu/music21/doc/moduleReference/moduleStreamBase.html#music21.stream.base.Stream.metronomeMarkBoundaries
Comment: Could you separate the original MIDI and the produced MIDI in two different graphs and change the X-axis to beats or bars?
Comment: How do you mean by overlap? I get the sequence like [('A5', 0.5), ('A3', 0.5), ('E6', 2.0), ('E4', 2.0), ('E6', 1.25), ('E4', 1.25), ('D6.E6', 0.25)] with duration and pitch/sequence of pitches in that format. There can be same pitches a number of times.
Answer:
MIDI files almost always use time units based on quarter notes, not time units based on seconds.
MIDI file software uses the tempo events in a MIDI file to convert the quarter-note-based time units into seconds-based time.
If there are no tempo events in a MIDI file, a default tempo of 120 quarter notes per minute is assumed.

It looks like your code is copying notes from your input MIDI file to your output MIDI file, but not copying any tempo information.
If your input MIDI file is 342 quarter notes long and has a tempo of 270 quarter notes per minute it would be 1 minute 16 seconds long.
If the output MIDI file is 342 quarter notes long but has a default tempo of 120 quarter notes per minute it ends up as 2 minutes 51 seconds long.


Precompiling asp.net solutions for deployment

Question: I'm using aspnet_compiler.exe to precompile my application for deployment.
However, I don't think it's working, for two reasons:

I see my applications assemblies under C:\Windows\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files, even if i manually clear out this directory & restart the app.
The performance hit to JIT the application is present. This is very specifically what I'd like to avoid, even though it is 'one time'.

Here's specifically what I'm doing:

Build the solution in studio.
execute aspnet_compiler.exe -v /Foo -p c:\builddir c:\deploydir (where Foo is the vdir my app runs under, c:\builddir is where studio builds to, and c:\deploydir is the location where
I then copy c:\deploydir to the web server.
I access http://localhost/Foo
After 30 seconds, the app displays, and I can see that assemblies have been generated in Temporary ASP.NET Files.

If it's notable, I'm using .net 3.5 SP1/Studio 2008 SP1. compilation debug=false is also set in web.config.
Comment: Is this a web site, or a web application project?
Answer: Your two points are moot:

ASP.NET precompilation doesn't result in native image generation so JIT overhead will still be there. There might be a worker process creation overhead too.
ASP.NET precompilation has the option to compile everything (incl. ASPX stuff, ...) or just source code (which you can specify with the -u switch). In case of the latter, those assemblies will still be generated for ASPX files. In case of the former, what you are seeing is not assembly generation. The runtime just copies and caches the assemblies in /bin in Temporary ASP.NET Files. You can confirm this fact by comparing the assemblies byte by byte and see that they are identical.
Comment: Being faster is relative. It's faster than non-precompiled apps not for preventing JIT compilation but for not invoking the ASPX parser and C# compiler.
Comment: I'm trying to reconcile your response for #1 (which seems logical to me) with msdn.microsoft.com/en-us/library/…, which states, "Precompiling an ASP.NET Web site provides faster initial response time for users since pages do not have to be compiled the first time they are requested. This is particularly useful for large Web sites that are updated frequently."  #2 is N/A given my sample above.
Comment: Oops... http://msdn.microsoft.com/en-us/library/ms227972.aspx was the reference URL.
Comment: Interesting. So what you're saying is that I'm not incurring the hit to generate the MSIL for the site, but I am incurring the hit to go from MSIL to x86. Thanks, this makes sense.
Answer: Precompiling the website with aspnet_compiler.exe doesn't JIT compile the code - you would need to ngen the code to do that.

The Native Image Generator (Ngen.exe)
  is a tool that improves the
  performance of managed applications.
  Ngen.exe creates native images, which
  are files containing compiled
  processor-specific machine code, and
  installs them into the native image
  cache on the local computer. The
  runtime can use native images from the
  cache instead using the just-in-time
  (JIT) compiler to compile the original
  assembly.
Comment: That KB article applies to ASP.Net 1.0 and 1.1
Comment: That article only applies to ASP.NET 1.x - NGEN'd assemblies have been supported since 2.0 - please see http://msdn.microsoft.com/en-us/magazine/cc163808.aspx
Comment: As much as I would like it to, ngen apparently does not apply to asp.net.  http://support.microsoft.com/kb/331979
Comment: Thank you both. I think I may see the solution now - precompile the website for deployment, using fixed names, deploy the solution to the server, and then ngen all assemblies in bin.
Comment: JohnW: NGENing doesn't always improve performance. It might actually reduce it. Always benchmark.
Comment: @JohnW: Mehrdad is correct - NGENing the assembly only removes the need for jitting the IL.  Since the JIT tends to be more efficient than NGENing you really ought to benchmark,


Simplify ${I}- {M} { A}^T( {I}+{A}{M}{A}^T)^{-1}{ A}$

Question: I would like to simplify
$${I}- M A^T( I+AMA^T)^{-1} A,$$
where $ M$ is $(n \times n$) and $A$ is $(m \times n)$ matrices.
I think this should be  $( I+ AM A^T)^{-1}$, but not sure how to show it?
Also, do we have to assume that $M$ is invertible?
My conjecture is true for the scalar case.
Comment: Could you add some thoughts to your post that explains you proposed answer?
Comment: @amWhy From the scalar case.
Comment: Thanks for the edit.
Comment: Under the assumptions that $\|M\|<1$, $\|AMA^{T}\|<1$ and $AA^{T}=I$, you can show that your expression reduces to $(I+M)^{-1}$. This and other observations when $\|AMA^{T}\|<1$ strongly suggest the answer would be $(I+MA^{T}A)^{-1}$ in general
Answer: Let
$${I}- {M} { A}^T( {I}+{A}{M}{A}^T)^{-1}{ A}$$
You know that
$$( {I}+{A}{M}{A}^T)^{-1}(I+AMA^T)=I \\
( {I}+{A}{M}{A}^T)^{-1}AMA^T=I-( {I}+{A}{M}{A}^T)^{-1} \\
MA^T( {I}+{A}{M}{A}^T)^{-1}AMA^T=MA^T-MA^T( {I}+{A}{M}{A}^T)^{-1} \\
MA^T-MA^T( {I}+{A}{M}{A}^T)^{-1}AMA^T=MA^T( {I}+{A}{M}{A}^T)^{-1} \\
\bigl( {I}- {M} { A}^T( {I}+{A}{M}{A}^T)^{-1}{ A}\bigr)MA^T=MA^T( {I}+{A}{M}{A}^T)^{-1}
$$
Therefore, assuming that  $MA^T$ is invertible, you get
$${I}- {M} { A}^T( {I}+{A}{M}{A}^T)^{-1}{ A}=MA^T( {I}+{A}{M}{A}^T)^{-1} \left(MA^T \right)^{-1}$$
This shows that the expression is actually conjugated by $MA^T$ to what you expect.
P.S.
$$MA^T( {I}+{A}{M}{A}^T) =MA^T+MA^TAMA^T \\
( {I}+{A}{M}{A}^T)MA^T=MA^T+AMA^TMA^T$$
Therefore, under the extra assumption that $A$ commutes with $MA^T$ you get that ${I}- {M} { A}^T( {I}+{A}{M}{A}^T)^{-1}{ A}=( {I}+{A}{M}{A}^T)^{-1} $.


How to write spam filter

Question: I'm stuck in having to write a simple spam filter
I'm not really sure about how I'm going to do it.
So far I've come up with wordlist and domain filtering, which will give or remove points up to a certain threshold.
For example, if you're writing about "v1agr4" from a blacklisted domain, you'll get like 2 points for spam, but if you're writing about "v1agr4" from a hotmail.com account, you'll get only 1 "spam point".
Do you guys have any other suggestions / ressources?
This is more about learning spam filters than developing something enterprise grade
Answer: Some really good algorithm info here:
http://www.paulgraham.com/spam.html
http://www.paulgraham.com/better.html
But, seriously, why reinvent the wheel?
Just download K9: http://keir.net/k9.html
Answer: Some open source Java projects related to Bayesian Spam Filtering (that was mentioned by LFSR Consulting):

Classifier4j
jBNC
Naiban

And one extra for C++:

SpamProbe
Answer: Look into  Bayesian Spam Filtering.
I know perl has a library for it, so I'd assume java would have one too.
Answer: I've written one with all the bells and whistles.
Answer: You can delegate that to a distributed service. Akismet is a very good solution.
Answer: Writing a spam filter depends upon your demands for scalability.
If you want a scalable solution, then content-filtering is probably not the smart choice to make as it is very CPU and memory consuming, and you would instead rather choose either reputation based filtering or blacklist based filtering, which will be way more CPU friendly on your server as well as much easier to write.
I wrote a a post on my blog that explains the idea behind writing a spam filter from a programmer's point of view and covers all the options from content based filtering to black list based filtering.

How to write spam filter


Offsite backup solutions for 40TB of data/35 Million Files on single drive

Question: I've got a server with 40TB of data and 35 million files. While the server itself has raid and all that jazz, I'm concerned about what would happen should something physically destroy the server (fire, lightning, etc).
The system is designed to run everything off a single drive (reams of legacy code) so segmenting into little drives isn't an option (the resources required to rewrite the code are prohibitive).
I was wondering what cost effective offsite backup options there are. Moving them to a hosted solution is on the table, but since a lot of this is media data that needs to be frequently processed by a server farm that may introduce latency and bandwidth issues.
edit: by "single drive" I mean from the user's perspective. The data itself may be distributed in one drive so long as the software that accesses this data can treat everything as a singe drive.
Comment: Our FAQ, which you skipped, states right at the top that we don't do "Product, service, or learning material recommendations".
Comment: @Chopper3: darn
Comment: @Chopper3 This is a rather specific question. Do you have any idea where I can find the proper experts to ask? I'm looking for someone who will help with a technical answer as much as a cost factor one.
Comment: Geting a hosted/managed service provider to back up 40TB of data for you is probably not going to be cheap :-) If you can give us some of the requirements info we do have some storage guys around here who might weigh in with implementation ideas...
Answer: Like Chopper said, we don't do product and service recommendations - so everything I'm going to tell you is pretty generic.  You're going to have to go to your thinking closet (possibly your crying corner after seeing some of the price tags) and figure out how to make this happen in your environment.

If you're asking "How do I back up this server?", which is what it sounds like you're trying to figure out, start by defining your requirements:

How much data?
How often do you need to back it up?
How frequently does it change? (How much will be in each backup cycle?) etc.

Knowing that will help you figure out which solution might be most effective.  You can probably do the backup itself with many commercial (or open-source) backup tools, but picking the media and backup schedule will vary.

Lots of high-volume changes
You're probably a candidate for a SAN and SAN replication (or equivalent with a NAS).  Basically clone the whole filesystem to another site and stream the changes over a dedicated network link.
Note this isn't a real "backup" - if someone deletes an important file it disappears from both places - it just gives you Disaster Recovery.
A big initial set, but low change volume
A single base backup and incremental backups thereafter (possibly with "consolidation" or "synthetic full" backups to keep restore time under control) could work well here.

Picking the media type is also important - Tapes are traditional, and pretty big.
If the cost of tapes is prohibitive terabyte disk drives are relatively cheap too, and reliable.  Optical media is probably not an option with your 40TB data set size.
SAN replication over a dedicated link is a great solution aside from cost.
Another option is backing up to a SAN (or just a host with a ton of disk) at a your remote site, but this would also likely require a dedicated (FAST) network connection. This is especially true if your change set is large (if you have a small change set you can always do your initial backup locally and then ship the storage device off-site and make do with a slower link).

In addition to backing up make sure you take into account the restore process -- If it takes you 3 months to get your data back over a slow network uplink the backup may not be very useful...


Why does every limited succession have a convergent subsuccession?

Question: Working on some real analysis homework, I faced this question:

Let $(a_n)\ n ∈ N$ a real number succession. Prove that $(a_n)\ n ∈ N$ does not have a converging subsuccession if and only if $\lim_{n → +∞} |a_n| = + \infty $

I solved it being a sufficient condition, but I couldn't prove it being a necessary one. I came to the conclusion that any succession which has no limit but doesn't go to infinity is limited, but nothing beyond.
I asked my professor and he said that the result that every limited succession have a convergent subsuccession is a given, but didn't prove it.
How does one go about proving it?
Comment: This is the statement of a well-known theorem: https://en.wikipedia.org/wiki/Bolzano%E2%80%93Weierstrass_theorem . In addition, the correct terms are "sequence"/"subsequence" (instead of "succession"/"subsuccession") and "bounded" (instead of "limited").
Comment: are successions and sequences same?
Comment: Suppose $\lim|a_n| \neq +\infty$ then sequence $a_n$ is bounded. Then by Bolzanno weierstrass theorem, there exists a convergent subsequence which will be a contradiction.
Comment: Your professor stated the Bolzano-Weierstrass theorem. You can prove it by first using the **monotone subsequence theorem** and then use the **monotone convergence theorem**.
Comment: This would only prove that there exists a subsequence that, *in absolute value*, is convergent.
Comment: @PaoloLeonetti That's just not true. The converse to the condition at the end is "there exists some $C$ such that $-C < |a_n| < C$ for all $n$. (Alternatively, it's essentially a one-line proof to show that any sequence with a subsequence that converges in absolute value has a convergent subsequence).
Comment: @user3482749 What you wrote is false. If a nonnegative sequence does not diverge to $\infty$ it does not imply that it is bounded. Hence also your answer below is not correct. A counterexample is the sequence $(x_n)$ such that $x_n=n$ is $n$ is even and $x_n=0$ otherwise.
Comment: Apologies, I wrote that wrong, insert "it has a subsequence such that..." at the appropriate point.
Comment: ... which is what is already written in my answer.
Comment: It's not, though: you claimed that "This would only prove that there exists a subsequence that, in absolute value, is convergent." - that's simply untrue.
Comment: If you use the Bolzano--Weierstrass' theorem on the bounded subsequence $(|a_{n_k}|: k\ge 1)$, then, *at this point*, you only obtain a convergent sub-subsequence.

Otherwise you need to say that if $(|a_{n_k}|: k\ge 1)$ is bounded if and only if  $(a_{n_k}: k\ge 1)$ is bounded too. And then use the Bolzano--Weierstrass' theorem on the latter one. In both cases, there is an additonal step in the proof.
Answer: Let me rephrase this in a more standard way: you want to show that a bounded sequence in $\mathbb{R}$ has a convergent subsequence. This is a sketch of the proof (it should be a exercise to make this rigorous): say that $$|a_n|\leq R\quad \forall\:n\in\mathbb{N},$$ which means that $$a_n\in[-R,R]\quad \forall\:n\in\mathbb{N}.$$
Now, consider the half intervals $[-R,0]$ and $[0,R]$. One of them must have infinitely many terms of the sequence, right? Say that it's $[0,R]$ and consider $[0,\frac{R}{2}]$ and $[\frac{R}{2},R]$; again one of them has to have infinitely many terms of the sequence.
You can continue this process indefinitely and get a sequence of nested intervals whose length goes to $0$. Take $a$ to be the intersection of all this intervals. Then $x$ is an accumulation point of $\{a_n\}_n$, in other words a subsequence of $\{a_n\}_n$ converges to $a$.
Hope it helps.
PS: this is a classical result from real analysis called the Bolzano–Weierstrass theorem.
Answer: If $\lim\limits_{n\to\infty}|a_n| \neq \infty$, then it has a bounded subsequence: if it did not, then for any $C$, there would be some $n$ such that for all $m > n$, $|a_n| > C$, so $\lim\limits_{n\to\infty}|a_n| = \infty$, a contradiction. Choose some $C$ at which this fails.
We now proceed by "lion hunting":
Define $b_0 = -C$ and $c_0 = C$. We shall then construct sequences $(b_n)$ and $(c_n)$ such that there are always infinitely many terms of $a_n$ between them. We shall do this inductively as follows:
Let $m_n$ be the midpoint of $b_n$ and $c_n$. Then at least one of $[b_n, m_n]$ and $[m_n, c_n]$ contains infinitely many terms of $a_n$ (we know that there are infinitely many in $[b_0,c_0]$, and by our inductive hypothesis, in $[b_n, c_n]$). Choose $b_{n+1}$ and $c_{n+1}$ to be the endpoints of that interval.
Now, $(b_n)$ and $(c_n)$ are monotone and bounded. Define $b = \limsup b_n$. We shall show that $(b_n) \to b$.
For the former, note that, by definition of $b$, for any $\varepsilon > 0$, there is some $n$ such that $0 < b - b_n < \varepsilon$. But then, for any $m > n$, we have $0 < b - b_m < b - b_n < \varepsilon$. Thus, $(b_n) \to b$.
We note also that $b_n < c_n$ for all $n$, so $b$ lies in $[b_n, c_n]$ for all $n$.
Let $n_k$ be the smallest natural number such that $a_{n_k} \in [b_k, c_k]$. Then we have $|a_{n_k} - c| \leq |c_k - b_k| = 2^{k-1}C \to 0$, and so $(a_{n_k}) \to c$, and in particular, that $(a_{n_k})$ is a convergent subsequence of $(a_n)$.
Answer: $\lim_n |a_n|\neq \infty$ can be rewritten as $$\kappa:=\liminf_{n\to \infty}|a_n| \in [0,\infty).$$
Hence, there exists a subsequence $(|a_{n_k}|: k\ge 1)$ which is bounded, i.e., such that its image is contained in the compact set $[0,\kappa]$. Therefore by the Bolzano--Weierstrass theorem, there exists a sub-subsequence $(|a_{n_{k_t}}|: t\ge 1)$ which is convergent to some limit  $\ell \in [0,\kappa]$. It follows that there is an infinite set $T\subseteq \mathbf{N}$ such that $a_{n_{k_t}} \ge 0$ for all $t \in T$ or $a_{n_{k_t}} \le 0$ for all $t \in T$. Therefore $(a_{n_{k_t}}: t\in T)$ is convergent (to $\ell$ or to $-\ell$).
Comment: Oh, right, sorry.


Not able to Remove Add-on from Google Sheet

Question: A user of one of my Google sheets add-ons wants to remove it. He has followed the correct procedure to do so but it keeps appearing in his list of add-ons when he opens sheets. He tells me it has not been installed by the admin for his domain. I can't work out what if anything I can do to help him remove it. Any advice?
This is a video illustrating the issue: https://drive.google.com/file/d/1M4WYGTwLzTWcYmrFf_SGFEn8Lv8OGbj6/view
Answer: They should make sure it's not also installed in the G Suite Marketplace: https://gsuite.google.com/marketplace/app/lntol_sheet_translator/1054412249402
This is a known issue that we will be fixing with the upcoming Chrome Web Store to G Suite Marketplace migration: https://developers.google.com/gsuite/add-ons/how-tos/cws-migration


Understanding exchangeability

Question: Suppose you had information of a true success rate of passing an exam from different schools, and that the 'true' success rates are similar, can you assume exchangeability is applicable to these values, or is there not enough info to assume it?,if so, then what more info is needed to make a reasonable assumption?
Also, since the values are considered similar, what would be a logical way to estimate the success rate at a say 5-th school given you have values of the true success rate from 4 schools already, is it just a simple average? I'm not sure whether the concept of exchangeability can be applied here, and generally not getting how it can be applied to similar situations, involving a sequence of numbers.
Comment: I highly welcome questions about exchangeability. But could you please provide a clear and simple practical example. It could look something like: exam A on matter B in school C sees 90/100 students pass, etc. What exactly are you after? Comparing success rates? Between schools? What is your understanding of exchangeability? Link to the relevant wiki. It will greatly help your question.
Comment: Ah right, so if we assume independence that implies exchangeability.
Comment: More examples at https://stats.stackexchange.com/questions/353722/exchangeability-of-group-effects/353739#353739
Comment: say the success rate in passing is 50%,75%,66% and 80% amongst 4 schools within a county in England, under what circumstances would you assume exchangeability? I am familiar with De fenetti's theorem and the general definition of exchangeability. I think exchangeability applies here as the rates in schools within a county can be assumed to be similar, and the rates seem somewhat close.
Comment: Also, to predict the 5-th school, would it be reasonable to assume a beta prior, calculate the params through mean and variance of rates, or is there an alternative to this, based on intuition?
Comment: - Just trying to make sense of exchangeability in a practical setting in relation to hierarchical models. Also, not seeing how the definition of exchangeability follows from De fenetti's theorem or the other way around? The definition implies that the rates would have the same marginal distribution(I think), but what is the link between that and what De fenetti theorised? Just confused all round.Thanks
Comment: Okay. Any reasons not to assume independence?
Comment: What about in the case we can't assume independence?
Comment: Just so you know: I'm not commenting to critique. But simply to get your question into a form in which it is most likely to receive a good answer. :)
Comment: @s.g.: Just ask yourself if you have any information about the schools (other than the pass rates) distinguishing between them. Are some in very poor neighborhoods, others not? ... independence is in a way much stronger than exchangeability, so is not a good way to argue.
Answer: Exchangeability is a modeling concept, it cannot be applied to just "a sequence of numbers". Represent those numbers as realizations of some random variable and then we are playing. Your example is pass rate from (say) five different schools. Ask yourself: prior to seeing the actual numbers, do you expect more or less the same from the five schools? Or are there some distinguishing characteristics of some schools that make you have differing expectations? In the last case, you cannot assume exchangeability, at least not directly.
What kind of information could make you have different expectations for some schools? Some possibilities:

One schools is a selective science high school, the others ordinary schools.

some schools in rich neighborhoods, others in poor.

Some schools private, other public.

Most schools large, but one is small, so you expect larger variability simply from averaging over fewer students.

...

So, exchangeability is a form of symmetry: There is symmetry in the prior information that you have. This does not at all imply independence, exchangeability is much weaker than independence. When assuming exchangeability, you can model your random variables as IID, see details here.  That should answer the question in your second paragraph. Do prediction as you would do in an IID model.
Comment: One more useful thing to note here is that if the pass rate from the five schools comes to you in a random order (via SRSWOR) then the vector of values is exchangeable.
Comment: I agree with your characterisation of whether exchangeability can be assumed here.  But surely the concept *can* be applied to a sequence of numbers, so long as there is a probability measure operating on that sequence --- indeed, isn't exchangeability defined by permutation invariance of a probability measure?  (And for an infinite sequence of numbers, it is exchangeable if any finite vector within it is exchangeable.)

Understanding exchangeability


Solving a system of polynomial equations

Question: How can I solve a system of polynomial equations like this one

Maybe I'm missing a very basic trick... Can anybody suggest me an approach?
Comment: What do you get as the product of all those equations? Something symmetric!
Answer: Take the reciprocals of each and multiply by $abcdef$ to obtain the equations
$$a^2 = \frac14 abcdef \\ b^2 = \frac19 abcdef \\ \vdots$$
Multiply all original equations to get
$$(abcdef)^4 = 4\cdot 9\cdot 16 \cdots = 1$$
Now you should be able to tell the solution without using a calculator.
Answer: Take a logarithm and solve a linear system
\begin{align}
\left[\begin{matrix}
-1 &  1 &  1 &  1 &  1& 1 \\
 1 & -1 &  1 &  1 &  1& 1 \\
 1 &  1 & -1 &  1 &  1& 1 \\
 1 &  1 &  1 & -1 &  1& 1 \\
 1 &  1 &  1 &  1 & -1& 1 \\
 1 &  1 &  1 &  1 &  1&-1
\end{matrix}\right]
\left[\begin{matrix}
\ln a \\
\ln b \\
\ln c \\
\ln d \\
\ln e \\
\ln f
\end{matrix}\right]
&=\left[\begin{matrix}
\ln 4 \\
\ln 9 \\
\ln 16 \\
-\ln 4 \\
-\ln 9 \\
-\ln 16\end{matrix}
\right]
\end{align}
\begin{align}
\left[\begin{matrix}
\ln a \\
\ln b \\
\ln c \\
\ln d \\
\ln e \\
\ln f
\end{matrix}\right]
=
\tfrac18
\left[\begin{matrix}
-3 &  1 &  1 &  1 &  1& 1 \\
 1 & -3 &  1 &  1 &  1& 1 \\
 1 &  1 & -3 &  1 &  1& 1 \\
 1 &  1 &  1 & -3 &  1& 1 \\
 1 &  1 &  1 &  1 & -3& 1 \\
 1 &  1 &  1 &  1 &  1&-3
\end{matrix}\right]
{}
\left[\begin{matrix}
\ln 4 \\
\ln 9 \\
\ln 16 \\
-\ln 4 \\
-\ln 9 \\
-\ln 16
\end{matrix}\right]
\end{align}


Javascript countdown just working on Chrome

Question: Well, I'm making a game, and in my game I have a countdown script in javascript that receives the date when the building upgrade is over, and makes a countdown, and when the countdown ends, executes a script that upgrades the building to the next level.
But just works in google chrome, and in the other browsers appears like that:

Firefox:

Google Chrome

Just the Javascript:
date_default_timezone_set('europe/lisbon');
$datephp = date('Y-m-d H:i:s');

echo'

';

Complete function:
function factory_update($get){

$userid = $_SESSION['userid'];

$query00 = "SELECT * FROM factory_upgrading WHERE userid = '$userid'";
$result00 = mysql_query($query00) or die(mysql_error());

while($row00 = mysql_fetch_array($result00)){
$factory_upgrade = $row00['userid'];
}

if(!isset($factory_upgrade)){
echo "Sem melhoramentos.";
return 0;
}

$query01 = "SELECT * FROM factory_upgrading WHERE userid = '$userid'";
$result01 = mysql_query($query01) or die(mysql_error());

while($row01 = mysql_fetch_array($result01)){
$factory_level = $row01['new_level'];
$factory_date = $row01['upgraded'];
}

if ($get == "load")
{
echo '
';
echo '

';
echo '
';}

else
{

date_default_timezone_set('europe/lisbon');
$datephp = date('Y-m-d H:i:s');

echo'

';

echo '
';
echo '
Nivel: ' . $factory_level . '

Duração:

';
echo '

';
echo '
';

echo '
    ';
}
}

Comment: why are you doing the output in php cannot you not do it nromally or via a js script file?
Comment: I have to put php variables in the middle of the javascript so I think is better that way :|
Comment: What would a typical `$factory_date` look like?
Comment: Any errors in the console of firefox? Does it work in IE?
Comment: what would the typical result of $factory_date be it might be that it not formatting correctly for the browser as in it might be of a string or integer etc
Comment: have you tried to hard code that in to your js and see what the result is
Answer: The format of $factory_date is invalid according to standard JavaScript and Chrome happens to be able to parse it.
For better results, stick to these Date constructors:
new Date();
new Date(value);
new Date(dateString);
new Date(year, month [, day, hour, minute, second, millisecond]);

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date
Comment: There is probably also a simple way to get a unix epoch timestamp in PHP, you could use that.
Comment: If the variable of $factory_date = 2013-12-30 22:21:01
How can I put that in a good format for all browsers?
Comment: Read my post and the link I provided. The different valid `dateString` are explained. You could always use whatever PHP object represents a date to extract year/month/day/hour/minute/second and use the last constructor. Be aware that the month parameter is 0-based in JavaScript.
Comment: Avoid `new Date(dateString);` as it is not consistently implemented in browsers, the others are fine.


How to determine if a set is dense

Question: My question reads:
Which of the following sets are dense? (Take $p\in\mathbb{Z}$ and $q\in\mathbb{N}$)
($a$) the set of all rational numbers $\frac{p}{q}$ with $q\leq\ 10$
($b$) the set of all rational numbers $\frac{p}{q}$ with $q$ a power of 2.
I am not too sure how to go about determining if the set if dense or not. I do know that for a set to be dense there must be an element in this set that can be found between any two real numbers $a<b$.
For the first one I was looking at the fact that fractions would be a part of this set, but I am not sure if noticing this helps me. Then, for the second one, I think once I see a bit of how the first once works I will get it.
Answer: (a) is not dense, because for example in $(0,1)$ you have only a finite number of rational numbers with denominator less than $10$. So you can find an interval in the gaps, which does not contain any of the aforementioned rational numbers
(b) is dense, because you can make $1/q$ as small as you want. So if you take an interval $(a,b)$ you can find a $q=2^n$ such that $\frac{1}{q}<b-a$. So if you start in the origin and make steps of size $\frac{1}{q}$ towards the interval, you will eventually fall in the interval $(a,b)$. See also Proving that $\Bbb Q$ is dense in $\Bbb R$. The idea is the same.
Comment: I am not too sure what you mean by (a) when you say you can find an interval in the gaps that does not contain the rational numbers.
Comment: If $S=\{p/q:q \le 10\}$ is dense (in $\mathbb R$), then $S\cap (0,1)$ has less than $100$ points ($1\le p\le q\le 10$), so it is finite. If $x$ is the smallest (minimum) element of $S\cap (0,1)$, then you may take the interval $(x/3,x/2)$, which is guaranteed not to contain elements of $S$. Contradiction. So $S$ is not dense.
Comment: Then for example could the interval [1/100, 1/1000] be a good counter example for a?
Comment: yes, [1/1000, 1/100] would do.

How to determine if a set is dense


Simulation of covariates in regression

Question: I am doing a simulation study on a simple linear regression model to see how well OLS estimator performs. I am planning to use coverage probability to assess the estimation of slope parameter using 1000 replications with sample size of 100
Suppose the model is simple as $y_i=0.5+2*x_i+\epsilon_i$
I am not sure which of the following two options to choose in simulation.

simulate all 100 $x_i$ values, say from a normal distribution.
Then in each of the 1000 replications, use the same 100 $x_i$ values for generation of $y_i$.
In each of 1000 replications, first simulate 100 $x_i$ values, and based on these covariate values, generate $y_i$.

Any help is appreciated.
Comment: Without further information, I would use option 2, because otherwise your coverage probability would depend on an arbitrary sample of $x_i$'s.
Comment: can you be more precise on how you plan to use coverage probability?
Comment: Hi, Xi'an, I am planning to calculate the confidence interval based on estimate in each of the 1000 replications, and calculate the proportion of intervals that cover the true coefficient values. I am expecting that the coverage probability should be close to nominal values, such as 95%.
Comment: To use this coverage probability, I think one assumption should be that estimates in each replication are identically distributed. Then I tend to support option 1, because if x_i values change, the standard deviation of estimate of coefficient will change, and this means these estimates are not identically distributed. But I have not found other evidence supporting that option 1 is better over option 2.
Comment: @Xi'an: Thanks. But do you agree that using option 2, the obtained estimates for the coefficient cannot be treated as iid?
Comment: On **both cases**, the replicated estimates **are** iid: in option 1, they are iid conditional on the $x_i$'s and in option 2, they are iid unconditionally (or marginally).
Answer: for $i \in \{1, 2, \ldots, 100\}$ let
$$\eta_i = .5 + 2x_i$$
which is the linear predictor and then you can simulate
$$y_i \sim \mathcal{N}(\eta_i, \sigma)$$
If you do $j$ replicates, your question boils down to:
$$y_{ij} \sim \mathcal{N}(\eta_i, \sigma)$$
or
$$y_{ij} \sim \mathcal{N}(\eta_{ij}, \sigma)$$
If you are interested in drawing inference about parameters estimates for a given $\boldsymbol{\eta}$, then I think you want $y_{ij} \sim \mathcal{N}(\eta_i, \sigma)$ where you use the same values for your linear predictor every time (i.e., option 1 in your question).
A simple example
<code>set.seed(10)
x <- rnorm(100, mean = 0, sd = 1)
res1 <- sapply(1:10000, function(i) {
  coef(lm(rnorm(100, mean = .5 + 2 * x, sd = 1) ~ x))
})

set.seed(10)
res2 <- sapply(1:10000, function(i) {
  x <- rnorm(100, mean = 0, sd = 1)
  coef(lm(rnorm(100, mean = .5 + 2 * x, sd = 1) ~ x))
})

rowMeans(res1)
# (Intercept)           x
#  0.5002331   2.0010302
apply(res1, 1, sd)
# (Intercept)           x
#  0.1005374   0.1070732
rowMeans(res2)
# (Intercept)           x
#  0.4995853   1.9987682
apply(res2, 1, sd)
# (Intercept)           x
#  0.09977722  0.10169706
</code>
Comment: Thanks for your comment. To me, drawing inference about parameters for a given eta seems to be a very rare case. Can you give an example? Also, do you think I should use option 2 if I want to evaluate the performance of estimation in a more general predictor setting?
Comment: This may be all well and good.  If you are doing this to learn and don't mind reinventing the wheel more power to you.  But the world of robust regression is vastly studied.  There is a lot of literature out there that will probably answer your questions without you needing to do simulations.  Least squares estimates are known to be sensitive to departures from normality.  Issues about leverage, sensitivity to outliers, heteroscedasticity have all been thoroughly investigated and regression diagnostics were invented to notice these problems with your data.
Comment: @MichaelChernick: I agree that this simulation is not for discovering anything new about OLS and I am aware that OLS is sensitive to outliers. For just doing this simulation, your suggestion is I choose option 1? Do you agree with my interpretation about the estimates from option 1?
Comment: @Jerry You might want to focus on a given $\eta$ if you are interested in studying various residuals.  If you don't do anything differently, I am not sure it makes a big difference either way (see edited response above).
Answer: To answer your question in the comments, if you were doing a bootstrap instead of a simulation the bootstrap would suggest something like option 1.
If you want to accept the assumption that the $x$'s are fixed and known, which is an important assumption for applying OLS, then option 1 demonstrates the performance when the assumption of known $x$ is correct.  But suppose you are interested in seeing the sensitivity to the assumption of fixed $x$. Then you want to simulate with random errors generated for both $x$ and $y$.  That would not be option 1 but it would not be option 2 either.
Comment: Thanks for this explanation. I understand now when I should use option 1 and how to do a sensitivity analysis.                                                      PS: I also searched for some literature and found that option 2 is used for assessing 'the estimator's performance in a wide variety of settings'. This is stated in 1st paragraph on page 8 of the following  paper:                                                         http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1176349741

Simulation of covariates in regression


Implementing a graph convolutional layer, pixel2mesh example

Question: I'm trying to read through some python code in order to understand how to implement a Graph Convolutional Layer.
I was particularly interested in pixel2mesh, digging through the code I've found the following class:
class GraphConvolution(Layer):
    """Graph convolution layer."""
    def __init__(self, input_dim, output_dim, placeholders, dropout=False,
                 sparse_inputs=False, act=tf.nn.relu, bias=True, gcn_block_id=1,
                 featureless=False, **kwargs):
        super(GraphConvolution, self).__init__(**kwargs)

        if dropout:
            self.dropout = placeholders['dropout']
        else:
            self.dropout = 0.

        self.act = act
        if gcn_block_id == 1:
            self.support = placeholders['support1']
        elif gcn_block_id == 2:
            self.support = placeholders['support2']
        elif gcn_block_id == 3:
            self.support = placeholders['support3']

        self.sparse_inputs = sparse_inputs
        self.featureless = featureless
        self.bias = bias

        # helper variable for sparse dropout
        self.num_features_nonzero = 3#placeholders['num_features_nonzero']

        with tf.variable_scope(self.name + '_vars'):
            for i in range(len(self.support)):
                self.vars['weights_' + str(i)] = glorot([input_dim, output_dim],
                                                        name='weights_' + str(i))
            if self.bias:
                self.vars['bias'] = zeros([output_dim], name='bias')

        if self.logging:
            self._log_vars()

    def _call(self, inputs):
        x = inputs

        # dropout
        if self.sparse_inputs:
            x = sparse_dropout(x, 1-self.dropout, self.num_features_nonzero)
        else:
            x = tf.nn.dropout(x, 1-self.dropout)

        # convolve
        supports = list()
        for i in range(len(self.support)):
            if not self.featureless:
                pre_sup = dot(x, self.vars['weights_' + str(i)],
                              sparse=self.sparse_inputs)
            else:
                pre_sup = self.vars['weights_' + str(i)]
            support = dot(self.support[i], pre_sup, sparse=True)
            supports.append(support)
        output = tf.add_n(supports)

        # bias
        if self.bias:
            output += self.vars['bias']

return self.act(output)

And I think the actual convolution implementation is given by
# convolve
        supports = list()
        for i in range(len(self.support)):
            if not self.featureless:
                pre_sup = dot(x, self.vars['weights_' + str(i)],
                              sparse=self.sparse_inputs)
            else:
                pre_sup = self.vars['weights_' + str(i)]
            support = dot(self.support[i], pre_sup, sparse=True)
            supports.append(support)
output = tf.add_n(supports)

It is my understanding that in order to compute a convolution over a graph I need to compute an eigenvector analysis over the Combinatorial Laplacian (or simply Laplacian matrix) defined by the underlying graph, the reason behind this its because they would need to apply some sort of Fourier analysis, where the Fourier transform has an appropriate definition on a graph, and it depends on the Laplacian as mentioned.
Can anyone explain to me how a Graph convolutional layer is usually implemented and how the bits of code I'm showing reflect that?
Thank you
Answer: I believe that snippet of code originated from here.
Let $g \in \mathbb{R}^n$ be some filter defined on the spatial domain, and $x \in \mathbb{R}^n$ be the input signal. Let $L$ by the laplacian, which is diagonalized as $L = U\Lambda U^T$. Recall that eigenvectors of the laplacian form a basis for functions on the graph.
Then by "inspiration from convolution theorem" we have
$$g * x = U(U^Tg \odot U^Tx) = UGU^Tx$$ where $G$ is a diagonal matrix consisting of the elements in $U^Tg$.
Inspired by the fact that $L^k_{i,j}$ is non-zero only if the distance between $i$ and $j$ is less than $k$ (we want some locality property), and $L^k = U\Lambda^kU^T$, a common choice of $G$ is $\sum_{k \leq K} \theta_k \Lambda^k$, and $K = 1$ is a common approximation, leaving us with $$g * x \approx \theta L x$$
For vector-valued signal $X \in \mathbb{R}^{n\, \times\, c}$, and weight matrix $\Theta \in \mathbb{R}^{c\, \times\, c'}$ this becomes $Y = LX\Theta$.
In the code above, support $= L$, and self.vars['weights'] $ = \Theta$. You can trace how support is computed in the code -- I believe this implementation actually uses a fast chebyshev polynomial approximation to the sum of powers $\Lambda$ mentioned above.
Comment: I think for graphs the convolution is defined in terms of eigen function of the laplacian, there's no convolution theorem as far as I know on graphs.
Comment: thanks, i've edited the answer accordingly to correct
Comment: Didn't mean to be pedantic, I'm not an expert and maybe there was some other definition I'm not aware of.
Comment: Okay, I know the explanation is using graph laplacian and cheby poly. However, in the original implementation, it's nothing, but augmented normalized adjacency matrix. https://github.com/tkipf/gcn/blob/bdcc4dd9314d7794d68b4c826c46b4b53137c05c/gcn/utils.py#L132


Probability distribution of windowed cross-correlation

Question: This question is in the context of time-delay estimation. Say I have a stationary Gaussian stochastic process $g$, and I know its autocorrelation function $R_g(\tau)$. To do time-delay estimation, I'm computing a windowed cross correlation between $g$ and a delayed version of it. In other words,
$$
g_1 = g(x-D) \\
\phi(\tau) = \int_{-T/2}^{T/2} g(x) g_1(x + \tau)
$$
and I'm going to determine the delay by finding the maximum of $\phi$.
My question is, is it possible to get an expression for the probability distribution of $\phi$?
Comment: Also asked on dsp.SE
Answer: If $g_1$ and $g$ are discretized vector representations of analog signals that are each normalized, $\phi$ is the normalized cross correlation function. Then the density function for $\phi$, at fixed time $\tau$, is a transform of the noncentral Beta distribution. You can compute this density function from the relative error in approximating the vectorized representation of $g$ with $g_{1}$.
In what follows, I'll change your notation: I am going to call the digital form of $g$ vector $\mathbf{x}$ and the digital form of $g_{1}$ vector $\mathbf{w}$. I will also assume $\mathbf{w}$ is a reference, or template signal with effectively infinite SNR. I will assume $g$ is noisy. Unless $g$ is entirely noise, it will include a signal $\mathbf{u}$ immersed in noise $\mathbf{n}$. In general, $\mathbf{u}$ is partially correlated with $\mathbf{w}$, and is not necessarily an "amplitude scaled copy of $\mathbf{w}$" (e.g., the matched filtering hypothesis is somewhat erroneous). Call the deterministic correlation between the signals (I'll get to noise shortly) $\rho_{\infty}$. The $\infty$ indicates the "infinite" SNR of the deterministic portion of the signals. The total correlation coefficient (your $\phi$) is my $r$.
Using these definitions, the relative square error in approximating a data stream $\mathbf{x}$ $=$ $\mathbf{u}+\mathbf{n}$ with a waveform template $\mathbf{w }$, is the ratio of the least-squares  error $\vert\vert \mathbf{e} \vert\vert^{2}$ to measured signal energy $ \vert \vert \mathbf{x} \vert \vert^{2}$. This error may be re-written as a ratio of quadratic forms:
\begin{equation}
\begin{split}
\cfrac{\vert \vert \mathbf{e} \vert \vert^{2}}{\vert \vert \mathbf{x} \vert \vert^{2}} &= \cfrac{\vert \vert  \mathbf{x} - \hat{A} \mathbf{w }  \vert \vert^{2} }{ \vert \vert \mathbf{x} \vert \vert^{2}}
\\
&=  \cfrac{ \Vert  \mathbf{x} - \cfrac{\langle   \mathbf{x},\,  \mathbf{w} \rangle}{ \Vert \mathbf{w }  \Vert^{2}} \mathbf{w } \Vert^{2}} {\vert \vert \mathbf{x} \vert \vert^{2}}
\\
&=
1- \cfrac{\langle   \mathbf{x},\,  \mathbf{w} \rangle^{2} }{ \vert \vert \mathbf{w }  \vert \vert^{2} \vert \vert \mathbf{x }  \vert \vert^{2}}
\\
&=
1 - r^{2}\left( \mathbf{x}\right)
\end{split}
\label{eq:betaRelError}
\end{equation}
where $\hat{A}$ is the maximum likelihood estimate for template waveform amplitude. I will rewrite the right hand side of the second equality in the preceding Equation as a ratio of subspace projections:
\begin{equation}
\begin{split}
 \cfrac{ \Vert  \mathbf{x} - \cfrac{\langle   \mathbf{x},\,  \mathbf{w} \rangle}{ \Vert \mathbf{w }  \Vert^{2}} \mathbf{w } \Vert^{2}} {\vert \vert \mathbf{x} \vert \vert^{2}}
=  \cfrac{ \Vert  P_{W}^{\perp} \left( \mathbf{x} \right) \Vert ^{2}} { \Vert P_{W}^{\perp}\left( \mathbf{x} \right) \Vert ^{2} + \vert \vert P_{W} \left( \mathbf{x} \right) \vert \vert ^{2}}
%
\end{split}
\label{eq:betaSubspace}
\end{equation}
where $W$ is the subspace $\text{span}\left( \mathbf{w} \right)$, $W^{\perp}$ is the orthogonal complement to $W$, $P_{W}$ is the projector onto $W$ and $P^{\perp}_{W}$ is the projector onto $W^{\perp}$. The denominator follows from the Pythagorean identity for Hilbert Spaces.  I define two noncentrality parameters from these terms:
\begin{equation}
\begin{split}
\lambda &=  \cfrac{  \Vert P_{W}^{}\left( \mathbb{E} \left\{  \mathbf{x} \right\}  \right) \Vert ^{2}   }{\sigma^{2}} = \cfrac{\vert \vert P_{W} \left( \mathbf{u} \right) \vert \vert ^{2}}{\sigma^{2}} = \rho_{\infty}^{2}\cfrac{\Vert  \mathbf{u} \Vert ^{2}}{\sigma^{2}}
\\
\lambda^{\perp}  &= \cfrac{   \Vert P_{W}^{\perp}\left(  \mathbb{E} \left\{  \mathbf{x} \right\}   \right) \Vert ^{2}   }{\sigma^{2}} =  \cfrac{\vert \vert P_{W}^{\perp} \left( \mathbf{u} \right) \vert \vert ^{2}}{\sigma^{2}} = \left(1 - \rho_{\infty}^{2}\right)\cfrac{\Vert  \mathbf{u} \Vert ^{2}}{\sigma^{2}}
\label{eq:lambdaDefs}
\end{split}
\end{equation}
where the expected value and linear-projection operators commute. Now combine the previous three equations to rewrite $r^{2}\left(\mathbf{x} \right)$:
\begin{equation}
\begin{split}
1-\left(1 - r^{2}\left(\mathbf{x} \right)\right) &=
\frac{\Vert P_{W}^{\perp}\left( \mathbf{x} \right) \Vert ^{2} + \vert \vert P_{W} \left( \mathbf{x} \right) \vert \vert ^{2}}{\Vert P_{W}^{\perp}\left( \mathbf{x} \right) \Vert ^{2} + \vert \vert P_{W} \left( \mathbf{x} \right) \vert \vert ^{2}}
\\
&\quad -\cfrac{ \Vert  P_{W}^{\perp} \left( \mathbf{x} \right) \Vert ^{2}} { \Vert P_{W}^{\perp}\left( \mathbf{x} \right) \Vert ^{2} + \vert \vert P_{W} \left( \mathbf{x} \right) \vert \vert ^{2}}
\\
&= \cfrac{ \Vert  P_{W}\left( \mathbf{x} \right) \Vert ^{2}} { \Vert P_{W}^{\perp}\left( \mathbf{x} \right) \Vert ^{2} + \vert \vert P_{W} \left( \mathbf{x} \right) \vert \vert ^{2}}
\\
&\overset{d}{=}  \cfrac{ \chi_{1}^{2}( \lambda )} { \chi_{1}^{2}( \lambda ) +  \chi_{N_{E} - 1}^{2}( \lambda^{\perp} ) }
\end{split}
\end{equation}
where $\overset{d}{=}$ is distributional equality, $\chi_{N_{E}-1}^{2}( \lambda^{\perp} )$ is the noncentral Chi-square distribution with $N_{E} -1$ degrees of freedom and noncentrality parameter $\lambda^{\perp}$, and $\chi_{1}^{2}( \lambda )$ is the noncentral Chi-square distribution with one degree of freedom and noncentrality parameter $\lambda$. From the definition of the Beta distribution:
\begin{equation}
\begin{split}
%
r^{2}\left(\mathbf{x} \right) \sim \text{B} \left(t, \frac{1}{2}, \frac{1}{2}(N_{E}-1) ; \lambda, \lambda^{\perp} \right)
\end{split}
\end{equation}
where $\text{B}\left(t, N_{1}, N_{2}, \alpha, \beta \right)$ is the doubly noncentral Beta distribution function. It is evaluated at $t$ (with the same domain as $r^{2}$), has $N_{1}$ and $N_{2}$ degrees of freedom, and noncentrality parameters $\alpha$ and $\beta$. The scalar $N_{E}$ denotes the effective number of independent samples within $\mathbf{x}$.
Now derive the PDF $f_{R}\left(r  ; \tau \right)$ for $r$ from the density of $r^{2}$ use a variable transformation; additionally consider values $r<0$, but fix $\tau$:
\begin{equation}
\begin{split}
f_{R}\left(r  ; \tau \right) &=  \lvert r \left( \mathbf{x} \right) \rvert \left[ \text{B}\left( r^{2}\left(\mathbf{x}\right); \,\frac{1}{2}, \,\frac{1}{2}\left( N_{E}-1\right), \,\lambda,\,\lambda^{\perp}\right) + \text{B}\left( -r^{2}\left(\mathbf{x}\right); \,\frac{1}{2}, \,\frac{1}{2}\left( N_{E}-1\right),\, \lambda,\, \lambda^{\perp}\right) \right].
%\nonumber
\label{eq:rBetaDist}
\end{split}
\end{equation}
Remember, your $\phi$ is my $r$.
One minor self promotional note: there is a seismology paper in press with this derivation in it; please cite if you use (Bulletin of the Seismological Society of America, ''A Waveform Detector that Targets Template-Decorrelated Signals and Achieves its Predicted Performance: Demonstration with IMS Data (Part I)''; Vol 106, Issue 5. I'll omit my name).
Comment: Thanks for taking the time to write this answer up, particularly to typeset all the Latex
Comment: Many thanks. I will delete this comment shortly. (I suggest you delete your comment also now that it's dealt with.)


Binary version 11.1.0 does not match the expected package version 9.7.0

Question: in my react repo
export CYPRESS_INSTALL_BINARY=~/cypress-11.1.0.zip
yarn add -D firstname.lastname@example.com

later I found that does not support cypress.json
so I did
export CYPRESS_INSTALL_BINARY=~/cypress-9.7.0.zip
yarn add -D email@example.com

Now it keeps complaining:
Binary version 11.1.0 does not match the expected package version 9.7.0
Update 1:
In my react repo, ./node_modules/.bin/cypress --version
Cypress package version: 9.7.0
Cypress binary version: 11.1.0 (???????)
Electron version: 21.0.0
Bundled Node version: 16.16.0

Answer: yarn add -D email@example.com maybe is not doing a download to cache, due to the later version already present.
I always yarn remove cypress before installing another version.
Comment: Hi, ./node_modules/.bin/cypress --version -> Cypress binary version: 11.1.0. Is it a way to set it?
Comment: No it won't, nor should you attempt to patch things. Instead, remove the previous version and add the older version you desire.
Comment: Do you know where I can find "Cypress binary version: 11.1.0" It is not in the local repo and it is not in global @TesterDick
Comment: I just install it same way you have above. There's a newer version than that now.
Comment: You don't want a "retro" version anyway, stick with the up-to-date.


How to prevent user from viewing profile

Question: I go into Administration>People>Permissions>User>View user profiles and see that only admin has this permission, however while logged in as non-admin I can still see his profile when I go to  /user
Comment: do you mean you don't want a user to be able to view his or her own profile?
Answer: There'd be a few ways you could do it, the fastest might be to create a user-profile.tpl.php and do a check for the user's group.
Answer: I had success with Tab Tamer module and I was able to hide tabs while still giving the admin access to modify if needed.
Comment: I think that the issue is not just with tabs, we want to disable access to /user. does Tab Tamer really do this?


Annuity formula proof $\frac{a_{\overline{n}|}}{a_{\overline{k}|}}$

Question: I have the actuarial exam FM in 2 days and there is one more thing that I would like to understand.
I cam across a problem having to do with identities and this is the following.

A perpetuity paying $50 on the last day of each year was purchased on January 1, 1928.
On January 1, 1978, the perpetuity was exchanged for a 15-year annuity-due with semi-annual payments of amount X.
The interest rate is 6 percent, convertible monthly.
Find X.

When there are $n$ payments of $1$ with $k$ conversion periods between each payments at the end of each $n$ conversion period, I can see why the present value of those payments will be
$$\frac{a_{\overline{n}|}}{s_{\overline{k}|}}$$
I think the argument is that if a payment of $1$ is to be made in one payment period at the end, the equivalent payments that we can pretend that there is is $1 \over {s_{\overline{k}|}}$.  Thus calculating the present value is straight forward.
However, if the payments are made at the beginning I am thinking that each payment of $1$ is equivalent to $1 \over {a_{\overline{k}|}}$ but the numerator should be an annuity-due rather than an annuity-immediate which is expressed as
$$\frac{a_{\overline{n}|}}{a_{\overline{k}|}}$$
Either I am not even deriving the first formula or there is something missing in my argument... can someone help me out?
Answer: Let be $i^{(12)}=6\%$ the nominal interest convertible monthly, so that the effective annual interest rate is $i=\left(1+\frac{i^{(12)}}{12}\right)^{12}=6.17\%$.
The value of the perpetuity at the time of the exchange is
$$
50 a_{\overline{\infty}\!|i}=\frac{50}{i}=\frac{50}{0.0617}= 810.66 \tag 1
$$
The semiannual interest rate is $i^{(2)}=6.08\%$ found by
$$
\left(1+\frac{i^{(2)}}{2}\right)^{2}=\left(1+\frac{i^{(12)}}{12}\right)^{12}\Longrightarrow i^{(2)}=2\left[\left(1+\frac{i^{(12)}}{12}\right)^{6}-1\right]=6.08\%
$$
For the 15-year annuity-due with semi-annual payments we have 30 payments at the beginning of each period. The present value is
$$
X\,\ddot a_{\overline{30\,}\!|i^{(2)}}=X\,\frac{1-v^n}{1-v}=X\, \times 14.4839\tag 2
$$
where $v=\frac{1}{1+i^{(2)}}$.
Putting $(1)=(2)$, we find

$$
50 a_{\overline{\infty}\!|i}=X\,\ddot a_{\overline{30\,}\!|i^{(2)}}\Longrightarrow \boxed{X= 55.97 }
$$
Answer: There are two small errors in alexjo's answer. First, the equation for the the calculation of the effective annual interest rate should be $i = (1 +\frac{i^{(12)}}{12})^{12} - 1$. Second, you do not use $i^{(2)}$ in the 15-year annuity-due but $\frac{i^{(2)}}{2}$. The quantity $i^{(2)}$ is an annual nominal rate of interest, compounded semi-annually, which yields an effective six-month rate of $0.005$. Because the period we are dealing with is half a year, we need to use the effective six-month rate.$$810.66 = X\,\ddot a_{\overline{30\,}\!|0.005}$$ $$ X = 40.336\tag1$$
However, the multiple choice only offers less straightforward answers, all in symbolic form. We can still follow the path alexjo outlined for us, but we must be aware of an added twist while we do it $-$ do not convert the interest rate of 6%, convertible monthly. Therefore, our annuities' time periods will in the unit of months.
A payment of 50 at the end of the year is equivalent to $50/{s_{\overline{12\,}\!|0.005}}$. And so, the present value of the perpetuity paying \$50 on the last day of each year at the time of the exchange is $$\left(\frac{50}{s_{\overline{12\,}\!|0.005}}\right)\left(\frac{1}{0.005}\right).$$
Now for the exchange. There are 180 months in 15 years, which gives us $\ddot a_{\overline{180\,}\!|0.005}$ for an annuity-due, but we must account for the fact that payments are made on a semiannual basis; so we must divide by $\ddot a_{\overline{6\,}\!|0.005}$. The result is $$\frac{10000}{s_{\overline{12\,}\!|0.005}} = X\frac{\ddot a_{\overline{180\,}\!|0.005}}{\ddot a_{\overline{6\,}\!|0.005}}, $$
which, when simplified, gives $$X = \frac{10000a_{\overline{6\,}\!|0.005}}{s_{\overline{12\,}\!|0.005}a_{\overline{180\,}\!|0.005}}, $$
which agrees with (1). Remember, the dots on the annuity-due cancel.


Why is my autoscaling group instance unhealthy?

Question: I'm noticing a very strange issue with an AWS auto-scaling group.
Instances are being reported (incorrectly) as being unhealthy. The instances are then being terminated and replaced unnecessarily. This is causing problems because it is leaving the ASG with insufficient capacity to cope with the load.
To try to identify the problem, I've temporarily suspended the "Terminate" process for the ASG.
Right at the moment I have a single instance in the group reported by the ASG as being unhealthy. Logging in to the instance and testing the health check directly proves that it is in fact healthy.
Additionally, the load balancers associated with the ASG also report all instances as healthy.
My question is. How can my ASG consider an instance to be "UNHEALTHY", if the health check type for the group is set to "ELB", and the load balancers report the instance as healthy?
Is there a way I can find out when and why the ASG flagged the instances as "Unhealthy"?
This ASG is currently associated with 2 classic ELB's, and 2 ALB Target Groups. We're in the process of migrating from ELB to ALB.
As mentioned though, both ELB's, and both Target Groups report all instances as healthy.
Answer: I guess the problems come from having the instance part of multiple ELBs / ALB TGs. I suspect that any one of those ELBs / ALBs can then trigger the instance termination if it deems it unhealthy for whatever reason.
Change the health check type to EC2 until you're done with the migration.
The best practice is to have ASG bound to only a single load balancer.
Hope that helps :)
Comment: I also suspect this may have been the problem. I've detached the legacy ELB's and so far the issue hasn't happened again.


How can I tell the Data Annotations validator to also validate complex child properties?

Question: Can I automatically validate complex child objects when validating a parent object and include the results in the populated ICollection<ValidationResult>?
If I run the following code:
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;

namespace ConsoleApplication1
{
    public class Person
    {
        [Required]
        public string Name { get; set; }

        public Address Address { get; set; }
    }

    public class Address
    {
        [Required]
        public string Street { get; set; }

        [Required]
        public string City { get; set; }

        [Required]
        public string State { get; set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            Person person = new Person
            {
                Name = null,
                Address = new Address
                {
                    Street = "123 Any St",
                    City = "New York",
                    State = null
                }
            };

            var validationContext = new ValidationContext(person, null, null);
            var validationResults = new List<ValidationResult>();

            var isValid = Validator.TryValidateObject(person, validationContext, validationResults);

            Console.WriteLine(isValid);

            validationResults.ForEach(r => Console.WriteLine(r.ErrorMessage));

            Console.ReadKey(true);
        }
    }
}

I get the following output:

False
The Name field is required.

But I was expecting something similar to:

False
The Name field is required.
The State field is required.

I offered a bounty for a better child object validation solution but didn't get any takers, ideally

validating child objects to an arbitrary depth
handling multiple errors per object
correctly identifying the validation errors on the child object fields.

I'm still surprised the framework doesn't support this.
Comment: Similar question: http://stackoverflow.com/questions/3006516/support-for-nested-model-and-class-validation-with-asp-net-mvc-2-0
Answer: Issue - Model Binder Order
This is, unfortunately, the standard behavior of Validator.TryValidateObject which

does not recursively validate the property values of the object

As pointed out in Jeff Handley's article on Validating Object and Properties with the Validator, by default, the validator will validate in order:

Property-Level Attributes
Object-Level Attributes
Model-Level implementation IValidatableObject

The problem is, at each step of the way...

If any validators are invalid, Validator.ValidateObject will abort validation and return the failure(s)

Issue - Model Binder Fields
Another possible issue is that the model binder will only run validation on objects that it has decided to bind.  For example, if you don't provide inputs for fields within complex types on your model, the model binder won't need to check those properties at all because it hasn't called the constructor on those objects.  According to Brad Wilson's great article on Input Validation vs. Model Validation in ASP.NET MVC:

The reason we don't "dive" into the Address object recursively is that there was nothing in the form that bound any values inside of Address.

Solution - Validate Object at the same time as Properties
One way to solve this problem is to convert object-level validations to property level validation by adding a custom validation attribute to the property that will return with the validation result of the object itself.
Josh Carroll's article on Recursive Validation Using DataAnnotations provides an implementation of one such strategy (originally in this SO question).  If we want to validate a complex type (like Address), we can add a custom ValidateObject attribute to the property, so it is evaluated on the first step
public class Person {
  [Required]
  public String Name { get; set; }

  [Required, ValidateObject]
  public Address Address { get; set; }
}

You'll need to add the following ValidateObjectAttribute implementation:
public class ValidateObjectAttribute: ValidationAttribute {
   protected override ValidationResult IsValid(object value, ValidationContext validationContext) {
      var results = new List<ValidationResult>();
      var context = new ValidationContext(value, null, null);

      Validator.TryValidateObject(value, context, results, true);

      if (results.Count != 0) {
         var compositeResults = new CompositeValidationResult(String.Format("Validation for {0} failed!", validationContext.DisplayName));
         results.ForEach(compositeResults.AddResult);

         return compositeResults;
      }

      return ValidationResult.Success;
   }
}

public class CompositeValidationResult: ValidationResult {
   private readonly List<ValidationResult> _results = new List<ValidationResult>();

   public IEnumerable<ValidationResult> Results {
      get {
         return _results;
      }
   }

   public CompositeValidationResult(string errorMessage) : base(errorMessage) {}
   public CompositeValidationResult(string errorMessage, IEnumerable<string> memberNames) : base(errorMessage, memberNames) {}
   protected CompositeValidationResult(ValidationResult validationResult) : base(validationResult) {}

   public void AddResult(ValidationResult validationResult) {
      _results.Add(validationResult);
   }
}

Solution - Validate Model at the Same time as Properties
For objects that implement IValidatableObject, when we check the ModelState, we can also check to see if the model itself is valid before returning the list of errors.  We can add any errors we want by calling ModelState.AddModelError(field, error).  As specified in How to force MVC to Validate IValidatableObject, we can do it like this:
[HttpPost]
public ActionResult Create(Model model) {
    if (!ModelState.IsValid) {
        var errors = model.Validate(new ValidationContext(model, null, null));
        foreach (var error in errors)
            foreach (var memberName in error.MemberNames)
                ModelState.AddModelError(memberName, error.ErrorMessage);

        return View(post);
    }
}

Also, if you want a more elegant solution, you can write the code once by providing your own custom model binder implementation in Application_Start() with ModelBinderProviders.BinderProviders.Add(new CustomModelBinderProvider());.  There are good implementations here and here
Comment: Great answer. Nevertheless it really sucks that it cannot be done out of the box with like setting an option.
Comment: Could this solution be amended to work with unobtrusive javascript?
Comment: @JasonlPrice, pedantically, I'm sure it *could* be, but I'm not sure what edge cases you're having trouble with.  Can you post a test case and an [MCVE](http://stackoverflow.com/help/mcve) in a new question?
Comment: @KyleMit I have created a question at the following link http://stackoverflow.com/questions/41064842/create-a-complex-type-model-validation-attribute-with-server-and-client-validati
Answer: I also ran into this, and found this thread. Here's a first pass:
namespace Foo
{
    using System.ComponentModel.DataAnnotations;
    using System.Linq;

    ///
    /// Attribute class used to validate child properties.
    ///
    /// <remarks>
    /// See: http://stackoverflow.com/questions/2493800/how-can-i-tell-the-data-annotations-validator-to-also-validate-complex-child-pro
    /// Apparently the Data Annotations validator does not validate complex child properties.
    /// To do so, slap this attribute on a your property (probably a nested view model)
    /// whose type has validation attributes on its properties.
    /// This will validate until a nested <see cref="System.ComponentModel.DataAnnotations.ValidationAttribute" />
    /// fails. The failed validation result will be returned. In other words, it will fail one at a time.
    /// </remarks>
    public class HasNestedValidationAttribute : ValidationAttribute
    {
        ///
        /// Validates the specified value with respect to the current validation attribute.
        ///
        /// <param name="value">The value to validate.</param>
        /// <param name="validationContext">The context information about the validation operation.</param>
        /// <returns>
        /// An instance of the <see cref="T:System.ComponentModel.DataAnnotations.ValidationResult"/> class.
        /// </returns>
        protected override ValidationResult IsValid(object value, ValidationContext validationContext)
        {
            var isValid = true;
            var result = ValidationResult.Success;

            var nestedValidationProperties = value.GetType().GetProperties()
                .Where(p => IsDefined(p, typeof(ValidationAttribute)))
                .OrderBy(p => p.Name);//Not the best order, but at least known and repeatable.

            foreach (var property in nestedValidationProperties)
            {
                var validators = GetCustomAttributes(property, typeof(ValidationAttribute)) as ValidationAttribute[];

                if (validators == null || validators.Length == 0) continue;

                foreach (var validator in validators)
                {
                    var propertyValue = property.GetValue(value, null);

                    result = validator.GetValidationResult(propertyValue, new ValidationContext(value, null, null));
                    if (result == ValidationResult.Success) continue;

                    isValid = false;
                    break;
                }

                if (!isValid)
                {
                    break;
                }
            }
            return result;
        }
    }
}

Answer: You will need to make your own validator attribute (eg, [CompositeField]) that validates the child properties.
Comment: This is the approach I am trying and it works, but I am hung up on the fact that the attribute can only return 1 ValidationResult.

This is an issue if the child has multiple validation errors. I can stuff the error messages into a single string but it seems messy and wrong.


How to configure bitbucket cloud default groups for a new repo

Question: I am trying to configure bitbucket cloud so as to include only administrators on a new repository. I don't know who/how this account I am working with was setup, but when I new repo is created it has 10 groups automatically added with lots of overlap.
A 20 minute search on atlassian's website yields a lot of information about groups in general, but not how the defaults are set
Answer: A bit late
Default groups are defined within the "Create/Edit group" modal.
If you want to remove all unwanted groups you should go to your team settings and select User groups. Select the group you want to remove from defaults and then select Edit. BitBucket will show a modal with the group settings and there you should change Default repository access to None.
You can find that here


Secondary dominants/slash notation in Elton John

Question: Please see the section labelled "chorus" in the screenshot below:

My textbook labels these same chords as follows:
I V(2-4)/IV vi7 V(6-5) I(6-4) V11 V I
I have questions regarding the second and third chords (I haven't looked any further than that).
So, the second chord is the third inversion of the fifth of IV. In C, this would be a C chord with an G note in the root.  If that is the case, then where did the b-flat note come from? I understand that b-flat is the minor 7th of C, so then would this be a dominant 7 chord?  If so, then I am wondering why my textbook didn't notate it as such.
Moving on, this third chord is a vi7. A seventh from A is G, no? So, following my textbook,  I would play A in the bass and something that included the g note in my right hand. But, in the screenshot, the youtuber just plays an F major chord in his right hand.  Keep in mind that in the video (link:

 (go to 29 seconds in)) these chords all sound accurate.
Any attempt to clear up my confusion is appreciated.
Comment: The analysis in your textbook does not match the chords shown in the image and video. For example, F/A and vi7 are not the same thing. I found the first page of some sheet music for this song with Google, and there is no justification there for vi7. Mind you, vi7 could be played, I just haven't heard a version yet that does that.
Comment: The third inversion of a triad is equivalent to the root position as there are only three notes present, so the term is not used. Normally, speaking of a third inversion implies a seventh, which would be the Bb in your example. This matches the notation C/Bb. The notation V(4-2)/IV, combining Roman numerals with figured bass, represents the same thing. The third inversion seventh figuring 642 is conventionally reduced to 42.
Answer:
So, the second chord is the third inversion of the fifth of IV. In C, this would be a C chord with an G note in the root.

No, third inversion (4/2) puts the seventh as the bass note of the chord (the root note of a C chord is always C, the bass note of a chord is its lowest sounding note). C/Bb = V4/2/IV in the key of C. (4/3 of a seventh chord is second inversion, and 6/5 is first inversion).

Moving on, this third chord is a vi7.

F/A is IV6 in C. There is a discrepancy between your textbook and the video.

So, following my textbook, I would play A in the bass and something that included the g note in my right hand.

Yes, if you were following your textbook (and not the video) you would play A-C-E-G. However, if you were following the video, you would play A-C-F (or some voicing similar). You will notice that you can fudge the resolution of the C/Bb chord to the Am7 if you wanted to because Am and F are somewhat interchangeable chords. So while normally a V4/2/IV would naturally lead to a IV6, you could substitute a vi7 (Am7) if you wanted to do so. If that's what your textbook (or teacher) is telling you to do, then that's what you should do for now.
Comment: regarding the first point--V4/2/IV--where does the IV appear?  I dont see an F in the bass?


create thumbnail from video URL in C#

Question: I want to generate thumbnail from a video URL in C#. I have searched a lot to find a neat way but no success. I have used Nreco and MediaToolKit but none of them extract thumbnail image. using ffmpeg also has mumbo jumbos which didn't worked!
using NReco:
var ffMpeg = new NReco.VideoConverter.FFMpegConverter();
string thumbnailJPEGpath = "http://localhost:81882/content/hashem.jpeg";
ffMpeg.GetVideoThumbnail(videoUrl,thumbnailJPEGpath);

using ffmpeg:
try
        {
            System.Diagnostics.Process ffmpeg;

            string video;
            string thumb;

            video = Server.MapPath("~/Content/Movies/bye.mp4");
            thumb = Server.MapPath("~/Content/frame.jpg");

            ffmpeg = new System.Diagnostics.Process();

            ffmpeg.StartInfo.Arguments = " -i " + video + " -ss 00:00:07 -vframes 1 -f image2 -vcodec mjpeg " + thumb;
            ffmpeg.StartInfo.FileName = Server.MapPath("~/Content/ffmpeg.exe");
            ffmpeg.Start();
            ffmpeg.WaitForExit();
            ffmpeg.Close();
        }
        catch (Exception ex)
        {
            Response.Write("Error: " + ex.Message);
        }

to consider video files are not local and I have only a direct link to the file:
e.g.:http://phytonord.com/Film-Series/hana/26.mp4
does anyone has any solution? any sample code that works?
Comment: possible duplicate of [get thumbnail image of video file in C#](http://stackoverflow.com/questions/15702031/get-thumbnail-image-of-video-file-in-c-sharp)
Comment: tested both but not working. would you mind to test with video link provided?
Answer: Using NReco to Create Video Thumbnail:
I found my problem:
1: I have not used Server.MapPath. I had just entered relativePath.
2: it is not necessary the Video file be local it can be hosted someWhere else.
NReco just download the desired portion of video then extract the Thumbnail.
your Video file should be in localHost directory of your local server. I mean if your site is under development and video file is in your local Computer folder it WON'T Work because NReco requires byte range support in HTTP response Header file.
Unacceptable Link: "http://localhost:81882/content/AVSEQ01.mp4"
so for my local test, I have moved my video file to Local IIS directory :  C:\inetpub\wwwroot\AVSEQ01.mp4
//sample remote video file
//string videoUrl = "http://phytonord.com/Film-Series/peik%20sahar/1.mp4";

 //local video file
string localVideoFile = "http://localhost/AVSEQ01.mp4"
var ffMpeg = new NReco.VideoConverter.FFMpegConverter();
string thumbnailJPEGpath = Server.MapPath("~/Content/videoThumb.jpg");
ffMpeg.GetVideoThumbnail(videoUrl, thumbnailJPEGpath);

Comment: I thought I need to download the file to a local path first. Turns out I can simply use the url. Thank you for clarifying that.
Comment: The command your provided will create a single thumbnail image, what about creating multiple images after every second of the total video length. Currently, we are using the below command to create a sprite image. **Step1 -->** `ffmpeg -i corpfilm.mp4 -s 280x150 -vf fps=1 thumb/img%03d.jpg` **Step 2-->** `magick montage *.jpg 160x90 -geometry +0+0 overview.jpg` from command prompt. How do we achieve it through C#? Plz guide.
Answer: I can see the question is answered But not written clearly, So here is my answer:
First download the DLL
ffMpeg.GetVideoThumbnail(inputFile,outputFile, frameTime);
// Summary:
    //     Extract video frame from local video file at specified position
    //
    // Parameters:
    //   inputFile:
    //     path to local video file
    //
    //   outputFile:
    //     path to thumbnail jpeg file
    //
    //   frameTime:
    //     video position (in seconds)

this is how I did in my project
            var thumbNailMovieImage = PHYSICAL_PATH_UPLOAD_FILE + "/" + folder + "/thumb_" + filenameWithoutExtn + ".jpg";
            var ffMpeg = new FFMpegConverter();
            ffMpeg.GetVideoThumbnail(path, thumbNailMovieImage, 1);

Now run this code and check the location your thumbnail file is created
Comment: Is any way to  get return Image from actual video link?
Comment: bcoz I am using the same video converter. And I don't want to return image into Local folder..I want doing some updates from the return Image(fromvideo file/url).
Answer: Just in case, if some one looking for creating multiple thumb image extracting for sprite image creation using ffmpeg or NReco library.
Using Command line --> ffmpeg -i corpfilm.mp4 -s 280x150 -vf fps=1 thumb/img%03d.jpg
Under the hood NReco.VideoConverter executes ffmpeg with System.Diagnostics.Process API; this means that you can run any command that is possible in the command line.
For this concrete command VideoConverter's API equivalent:
var ffmpeg = new NReco.VideoConverter.FFMpegConverter();
ffmpeg.ConvertMedia(
  "corpfilm.mp4", null,
  " thumb/img%03d.jpg", null,
  new ConvertSettings() {
    CustomOutputArgs = " -s 280x150 -vf fps=1 "
  });

create thumbnail from video URL in C#


Set up updating source code on server by Using GitHub's Service Hooks

Question: I followed this article
I tried using this command ssh -T email@example.com and I received the feedback: Hi Tran! You've successfully authenticated, but GitHub does not provide shell access. which shows that it's OK with my ssh key and GitHub account.
This's my repository's Webhook image:

I think the problem is the github.php file with the content:
<?php `git pull`;?>

I tried with replace ` with ' (single quote) but it's the same. After I push, I didn't see any change on my server.
Comment: Even, I tried to use shell_exec('git pull') in php file. But I didn't see any changes.
Answer: This is a very bad idea... anyone could access your php file and update your code. But if you still want to do it that way, you need to upload a pair of ssh keys into your server (make sure they're in the right place) and also add the public key to your github account, maybe as a deployment key in your repo.
Comment: - Thanks for your idea! But please help me by comment your idea more details.

-About anyone could access my php file: this is very simple, just add a param as password. eg. http://mydomain.com/gitpull.php?p=a_pass_phase. I don't think anyone can guess my link and pass phase. ^^

Set up updating source code on server by Using GitHub’s Service Hooks


What to look for in a single 9v power supply for various brands of foot pedal

Question: I have the following foot pedals:

MXR Dynacomp
Pro-co RAT II
BOSS CE-5 (Chorus Ensemble)
Korg AX30G

... and of course would like to add more. :)
I'd like a single power supply to rule power them all (for example this Chord 9Vdc Power Bank).
I'm not necessarily after a recommendation per-sé, as it'll go out of date eventually. However, what should I be checking/looking for when trying to choose one of these adapters so I get one that will work without frying my gear.
I'm thinking adding up amps and checking centre-pin polarity etc, but I don't know for sure.
Answer: I'm powering 10 pedals with a single 1spot power supply.
You just have to make sure that you pedal do not consume in total more that power supply output (those mA).
T-rex makes some nice power supplies.
Comment: Cool. Are the pedals all the same or do they have different brands and power requirements? Are they all centre-pin positive?
Comment: They are center pin negative and they're all different.
Korg Pitchblack, CAE MC-404, VoodooLab Microvibe, Fulltone Catalyst, Xotic AC+, Ibanez BC-9, EHX Stereo Pulsar, TC Electronic NovaRepeater.
Answer: There are a wide range of companies that produce power supplies suitable for multiple effects. Core factors which should help your decision:

What is the total current output? This must be higher than the total current needs of your pedals
What is the individual current available? This must be higher than an individual pedal's needs
Can it cope with centre positive, centre negative or odd connector types?
How robust is it?
How heavy is it?