Como executar uma função javascript de um arquivo json

Question: Eu tenho um arquivo json que é gerado pelo php e ele é assim:
{
     "status":200,
     "command":"faca algo",
      "action":"function (){document.write('quer que eu faça algo?');}",
      "type":"acao"
}

como executar a função que está no arquivo?
Answer: Tecnicamente, isso não é uma função, pois JSON não comporta funções. É uma string com o código-fonte de uma função dentro. O que você pode fazer para executar é usar o eval (vale ler o conteúdo do link para entender os perigos associados com isso):
var json = '{"status":200,"command":"faca algo","action":"function (){document.write(\'quer que eu faça algo?\'); }","type":"acao"}';
var obj = JSON.parse(json);
// Guarda a função num objeto
var funcao = eval('(' + obj.action + ')');
// Executa a função
funcao();

http://jsbin.com/qonesiluyolo/1/edit
Se estiver usando jQuery par obter o JSON, o próprio jQuery já cuida do parse:
$.getJSON("meuendereco.com/json.json", function (data){
    var funcao = eval('(' + data.action + ')');
    // Executa a função
    funcao();
});

Ver também: Uso do eval() em javascript: quais os prós e contras?
Comment: não deu quado eu usei o $.getJSON("meuendereco.com/json.json", function (data){var obj = JSON.parse(data);
// Guarda a função num objeto
var funcao = eval('(' + obj.action + ')');
// Executa a função
funcao();});
Comment: Atualizei a resposta. É que nesse caso não precisa do `JSON.parse`, o jQuery já faz isso pra você.
Comment: Valeu, vaelu mesmo, ficou show.


Relation for Bessel functions

Question: I have function $$Q_{n}(z)=\frac{J_{n+1}(z)}{zJ_{n}(z)},$$where $J_{n+1}(z)$ and $J_{n}(z)$ are Bessel functions of the first kind. I need to prove
$$\frac{dQ_{n}}{dz}=\frac{1}{z}-\frac{2(n+1)}{z}Q_{n}(z)+zQ_{n}^{2}(z)$$
but I don't know where to start.
Comment: Equation $59$ here https://mathworld.wolfram.com/BesselFunctionoftheFirstKind.html ... otherwise looks pretty routine ?
Comment: Should I use quotient rule on the first function?
Comment: That's right. Differentiate using the quotient rule ... then use $59$ to deal with the derivatives of the Bessel functions. ... I have not actually done it, but it looks quite easy ?
Comment: After applying quotient rule I get two derivative terms $\frac{d}{dz}[J_{n+1}(z)]$ and $\frac{d}{dz}[zJ_{n}(z)]$ and I don't know what to do next. I've used 59 before, but when n is defined.
Comment: The other formula you will need is
\begin{eqnarray*}
z(J_{n+1}(z)+J_{n-1}(z))=2nJ_n(z).
\end{eqnarray*}
Answer: The shown identity is more or less straightforward according to your definition of $J_n(z)$. Assuming the series definition
$$ J_n(x) = \sum_{m\geq 0}\frac{x^{2m}(-1)^m}{2^n 4^m n!(n+m)!} $$
we clearly have that $J_n$ is an entire function and $2\,J_n'(z)=J_{n-1}(z)-J_{n+1}(z)$ (which is even more evident from the integral definition $J_n(z)=\frac{1}{\pi}\int_{0}^{\pi}\cos(z\sin\theta - n\theta)\,d\theta$ and the cosine addition formulas), so
$$ Q_n'(z) = -\frac{J_{n+1}(z)}{z^2 J_n(z)}+\frac{J_{n+1}'(z)}{z J_n(z)}-\frac{J_{n+1}(z)J_n'(z)}{z J_n(z)^2} $$
equals
$$ -\frac{J_{n+1}(z)}{z^2 J_n(z)}+\frac{J_n(z)-J_{n+2}(z)}{2z J_n(z)}-\frac{J_{n+1}(z)(J_{n-1}(z)-J_{n+1}(z))}{2z J_n(z)^2} $$
while
$$ \frac{1}{z}-\frac{2(n+1)}{z}Q_n(z)+zQ_n^2(z)=\frac{1}{z}-\frac{2(n+1)J_{n+1}(z)}{z^2 J_n(z)}+\frac{J_{n+1}(z)^2}{z J_n(z)^2} $$
so the given identity is equivalent to
$$ -2J_{n+1}(z)J_n(z)+zJ_n(z)(J_n(z)-J_{n+2}(z))-z J_{n+1}(z)(J_{n-1}(z)-J_{n+1}(z))\\= 2z J_n(z)^2-4(n+1)J_n(z) J_{n+1}(z)+2z J_{n+1}(z)^2$$
which is a consequence of $\frac{2n}{x}J_n(x) = J_{n-1}(x)+J_{n+1}(x)$.
This is usually done "in the opposite way", i.e. by proving that the solutions of certain differential equations of the Riccati type are given by ratios of adjacent Bessel functions.
Comment: @DonaldSplutterwit: of course, now fixed.

Relation for Bessel functions


Discord.NET - Framework version issue

Question: I am trying to write a bot for the Discord chat program in Visual C#. However, when I try to install Discord.NET from the NuGet package manager, it says I need .NET framework version 4.5, which is the one I have. I tried restarting my computer but it still doesn't work.
Thanks,
Matt
I am using visual studio 2010 ultimate.
Comment: when you say that you have .NET 4.5, do you mean that you have it installed on the machine? or do you mean that you set in project properties that 4.5 should be used for this project?
Comment: Installed on machine, I think that might have been the error. Ive replaced vs 2010 with vs 2012 now, and ill set that properties thing so it should be alright
Comment: Is your project created with .NET 4.5 framework?
Comment: I never heard of [DISCORD](https://en.wikipedia.org/wiki/Discord_(software)) but it seems like $20M was put into the development. VOIP and instant text-messaging for gamers.
Answer: In your Visual Studio, you can right-click on the project, select properties, and in the opened dialog you will be able to set target version of .NET framework

Discord.NET - Framework version issue


How can i resize, drag and drop images in react?

Question: I was looking at websites like https://milanote.com/ and I am trying to implement image manipulation (functions such as dragging, resizing, cropping), and I noticed that they tend not to use HTML canvas. I've looked into react libraries that support drag and drop, but I have not found one supports image rescaling.
Any suggestions or idea to the best approach to this?
Answer: react-grid-layout is a complex grid layout system. You can set the image as a background of a grid element. There are a lot of possibilites.


Comparing date string in python

Question: I have two dates I am comparing.
a = datetime.datetime.strptime('2019-03-09','12:09 AM')
b = datetime.datetime.strptime('2019-03-09','11:56 PM')
a = 2019-03-09 12:09
b = 2019-03-09 11:56
if a > b:
    print('yes')

What's happening is a is greater than b.  But it shouldn't be. It seems like its cutting off the am and pm.
Answer: You aren't using that strptime function correctly, it should be like this:
a = datetime.datetime.strptime('2019-03-09 12:09 AM','%Y-%m-%d %I:%M %p')
b = datetime.datetime.strptime('2019-03-09 11:56 PM','%Y-%m-%d %I:%M %p')

if a > b:
    print('yes')

And then, you're overwriting a and b anyway.
Here is the doc for how to use strptime
Comment: I just spotted my error.  Thanks I will upvote and accept your answer.

Comparing date string in python


How to use multiprocessing queue in Python?

Question: I'm having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let's call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such.
My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues)
Comment: pass the Queues to each process class as a parameter when you instantiate them in the parent process.
Answer: Short Summary
As of CY2023, the technique described in this answer is quite out of date. These days, use concurrent.futures.ProcessPoolExecutor() instead of multiprocessing, below...
This answer describes the benefits and shortcomings of using concurrent.futures.ProcessPoolExecutor(). FYI, multiple python processes are sometimes used instead of threading to get the most benefit from concurrency. That said, python threading works pretty well as long as there is sufficient CPU activity to avoid the GIL (activity such as sending / receiving network traffic).
Original Answer

My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues)

This is a simple example of a reader and writer sharing a single queue... The writer sends a bunch of integers to the reader; when the writer runs out of numbers, it sends 'DONE', which lets the reader know to break out of the read loop.
You can spawn as many reader processes as you like...
from multiprocessing import Process, Queue
import time
import sys

def reader_proc(queue):
    """Read from the queue; this spawns as a separate Process"""
    while True:
        msg = queue.get()  # Read from the queue and do nothing
        if msg == "DONE":
            break

def writer(count, num_of_reader_procs, queue):
    """Write integers into the queue.  A reader_proc() will read them from the queue"""
    for ii in range(0, count):
        queue.put(ii)  # Put 'count' numbers into queue

    ### Tell all readers to stop...
    for ii in range(0, num_of_reader_procs):
        queue.put("DONE")

def start_reader_procs(qq, num_of_reader_procs):
    """Start the reader processes and return all in a list to the caller"""
    all_reader_procs = list()
    for ii in range(0, num_of_reader_procs):
        ### reader_p() reads from qq as a separate process...
        ###    you can spawn as many reader_p() as you like
        ###    however, there is usually a point of diminishing returns
        reader_p = Process(target=reader_proc, args=((qq),))
        reader_p.daemon = True
        reader_p.start()  # Launch reader_p() as another proc

        all_reader_procs.append(reader_p)

    return all_reader_procs

if __name__ == "__main__":
    num_of_reader_procs = 2
    qq = Queue()  # writer() writes to qq from _this_ process
    for count in [10**4, 10**5, 10**6]:
        assert 0 < num_of_reader_procs < 4
        all_reader_procs = start_reader_procs(qq, num_of_reader_procs)

        writer(count, len(all_reader_procs), qq)  # Queue stuff to all reader_p()
        print("All reader processes are pulling numbers from the queue...")

        _start = time.time()
        for idx, a_reader_proc in enumerate(all_reader_procs):
            print("    Waiting for reader_p.join() index %s" % idx)
            a_reader_proc.join()  # Wait for a_reader_proc() to finish

            print("        reader_p() idx:%s is done" % idx)

        print(
            "Sending {0} integers through Queue() took {1} seconds".format(
                count, (time.time() - _start)
            )
        )
        print("")

Comment: Great example. Just as an additional bit of information to address the OP's confusion... This example shows that a shared queue needs to originate from the master process, which is then passed to all of its subprocesses. In order for two completely unrelated processes to share data, they must communicate over some central or associated network device (sockets for example). Something has to coordinate the information.
Comment: Qsize can go to zero if the queue readers exceed the rate of the queue writer
Comment: nice example.. i'm also new to this topic.. if i have multiple processes running the same target function (with different arguments), how to make sure that they dont clash while putting the data into the queue.. is lock necessary?
Comment: @bharat_iyengar From the multiprocessing module documentation, it says that Queue is implemented using a few locks/semaphores. So when you use the get() and put(object) Queue methods, the queue will block if some other process/thread is trying getting or putting something on the queue. So you don't have to worry about manually locking it.
Comment: Explicit stop conditions are better than implicit stop conditions
Comment: @WYSIWYG it sounds like `multiprocessing.Pool().map` may be easier for your application. It works just like normal `map`, running one function for each argument and putting the results in an array in the correct order.
Comment: Can I share the queue between several machines in local network?
Comment: Yes as one example, Celery has a python library to do it
Comment: @MikePennington why not check if queue is empty? Is sending"DONE" a more proper way to check if queue is empty?
Comment: Instead of using `queue.put('DONE')` and checking if `msg == 'DONE'` you can drop the former and swap out the latter by checking the size of the queue, e.g. `queue.qsize() == 0`
Comment: I guess it is better to try getting from queue with some tolerance in terms of retrying and/or timeout, When this fails, assume the queue is empty.
Comment: Just a warning:  the "DONE" sentinel isn't guaranteed to be the last thing put into the queue if you have multiple writers. According to the docs: "If multiple processes are enqueuing objects, it is possible for the objects to be received at the other end out-of-order."
Comment: Re: `Just a warning: the "DONE" sentinel isn't guaranteed to be the last thing put into the queue`.  Keeping it brief... A) I've been using this technique off and on since 2012... never had a problem like you're describing.  B) These days we should all be using [`concurrent.futures.ProcessPoolExecutor()`](https://docs.python.org/3/library/concurrent.futures.html) instead of this answer.  `ProcessPoolExecutor()` does not need an explicit "STOP" message to terminate the task loop.
Answer: Here's a dead simple usage of multiprocessing.Queue and multiprocessing.Process that allows callers to send an "event" plus arguments to a separate process that dispatches the event to a "do_" method on the process. (Python 3.4+)
import multiprocessing as mp
import collections

Msg = collections.namedtuple('Msg', ['event', 'args'])

class BaseProcess(mp.Process):
    """A process backed by an internal queue for simple one-way message passing.
    """
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.queue = mp.Queue()

    def send(self, event, *args):
        """Puts the event and args as a `Msg` on the queue
        """
        msg = Msg(event, args)
        self.queue.put(msg)

    def dispatch(self, msg):
        event, args = msg

        handler = getattr(self, "do_%s" % event, None)
        if not handler:
            raise NotImplementedError("Process has no handler for [%s]" % event)

        handler(*args)

    def run(self):
        while True:
            msg = self.queue.get()
            self.dispatch(msg)

Usage:
class MyProcess(BaseProcess):
    def do_helloworld(self, arg1, arg2):
        print(arg1, arg2)

if __name__ == "__main__":
    process = MyProcess()
    process.start()
    process.send('helloworld', 'hello', 'world')

The send happens in the parent process, the do_* happens in the child process.
I left out any exception handling that would obviously interrupt the run loop and exit the child process. You can also customize it by overriding run to control blocking or whatever else.
This is really only useful in situations where you have a single worker process, but I think it's a relevant answer to this question to demonstrate a common scenario with a little more object-orientation.
Comment: Outstanding answer! Thank you. +50 :)
Answer: I had a look at multiple answers across stack overflow and the web while trying to set-up a way of doing multiprocessing using queues for passing around large pandas dataframes. It seemed to me that every answer was re-iterating the same kind of solutions without any consideration of the multitude of edge cases one will definitely come across when setting up calculations like these. The problem is that there is many things at play at the same time. The number of tasks, the number of workers, the duration of each task and possible exceptions during task execution. All of these make synchronization tricky and most answers do not address how you can go about it. So this is my take after fiddling around for a few hours, hopefully this will be generic enough for most people to find it useful.
Some thoughts before any coding examples. Since queue.Empty or queue.qsize() or any other similar method is unreliable for flow control, any code of the like
while True:
    try:
        task = pending_queue.get_nowait()
    except queue.Empty:
        break

is bogus. This will kill the worker even if milliseconds later another task turns up in the queue. The worker will not recover and after a while ALL the workers will disappear as they randomly find the queue momentarily empty. The end result will be that the main multiprocessing function (the one with the join() on the processes) will return without all the tasks having completed. Nice. Good luck debugging through that if you have thousands of tasks and a few are missing.
The other issue is the use of sentinel values. Many people have suggested adding a sentinel value in the queue to flag the end of the queue. But to flag it to whom exactly? If there is N workers, assuming N is the number of cores available give or take, then a single sentinel value will only flag the end of the queue to one worker. All the other workers will sit waiting for more work when there is none left. Typical examples I've seen are
while True:
    task = pending_queue.get()
    if task == SOME_SENTINEL_VALUE:
        break

One worker will get the sentinel value while the rest will wait indefinitely. No post I came across mentioned that you need to submit the sentinel value to the queue AT LEAST as many times as you have workers so that ALL of them get it.
The other issue is the handling of exceptions during task execution. Again these should be caught and managed. Moreover, if you have a completed_tasks queue you should independently count in a deterministic way how many items are in the queue before you decide that the job is done. Again relying on queue sizes is bound to fail and returns unexpected results.
In the example below, the par_proc() function will receive a list of tasks including the functions with which these tasks should be executed alongside any named arguments and values.
import multiprocessing as mp
import dill as pickle
import queue
import time
import psutil

SENTINEL = None

def do_work(tasks_pending, tasks_completed):
    # Get the current worker's name
    worker_name = mp.current_process().name

    while True:
        try:
            task = tasks_pending.get_nowait()
        except queue.Empty:
            print(worker_name + ' found an empty queue. Sleeping for a while before checking again...')
            time.sleep(0.01)
        else:
            try:
                if task == SENTINEL:
                    print(worker_name + ' no more work left to be done. Exiting...')
                    break

                print(worker_name + ' received some work... ')
                time_start = time.perf_counter()
                work_func = pickle.loads(task['func'])
                result = work_func(**task['task'])
                tasks_completed.put({work_func.__name__: result})
                time_end = time.perf_counter() - time_start
                print(worker_name + ' done in {} seconds'.format(round(time_end, 5)))
            except Exception as e:
                print(worker_name + ' task failed. ' + str(e))
                tasks_completed.put({work_func.__name__: None})

def par_proc(job_list, num_cpus=None):

    # Get the number of cores
    if not num_cpus:
        num_cpus = psutil.cpu_count(logical=False)

    print('* Parallel processing')
    print('* Running on {} cores'.format(num_cpus))

    # Set-up the queues for sending and receiving data to/from the workers
    tasks_pending = mp.Queue()
    tasks_completed = mp.Queue()

    # Gather processes and results here
    processes = []
    results = []

    # Count tasks
    num_tasks = 0

    # Add the tasks to the queue
    for job in job_list:
        for task in job['tasks']:
            expanded_job = {}
            num_tasks = num_tasks + 1
            expanded_job.update({'func': pickle.dumps(job['func'])})
            expanded_job.update({'task': task})
            tasks_pending.put(expanded_job)

    # Use as many workers as there are cores (usually chokes the system so better use less)
    num_workers = num_cpus

    # We need as many sentinels as there are worker processes so that ALL processes exit when there is no more
    # work left to be done.
    for c in range(num_workers):
        tasks_pending.put(SENTINEL)

    print('* Number of tasks: {}'.format(num_tasks))

    # Set-up and start the workers
    for c in range(num_workers):
        p = mp.Process(target=do_work, args=(tasks_pending, tasks_completed))
        p.name = 'worker' + str(c)
        processes.append(p)
        p.start()

    # Gather the results
    completed_tasks_counter = 0
    while completed_tasks_counter < num_tasks:
        results.append(tasks_completed.get())
        completed_tasks_counter = completed_tasks_counter + 1

    for p in processes:
        p.join()

    return results

And here is a test to run the above code against
def test_parallel_processing():
    def heavy_duty1(arg1, arg2, arg3):
        return arg1 + arg2 + arg3

    def heavy_duty2(arg1, arg2, arg3):
        return arg1 * arg2 * arg3

    task_list = [
        {'func': heavy_duty1, 'tasks': [{'arg1': 1, 'arg2': 2, 'arg3': 3}, {'arg1': 1, 'arg2': 3, 'arg3': 5}]},
        {'func': heavy_duty2, 'tasks': [{'arg1': 1, 'arg2': 2, 'arg3': 3}, {'arg1': 1, 'arg2': 3, 'arg3': 5}]},
    ]

    results = par_proc(task_list)

    job1 = sum([y for x in results if 'heavy_duty1' in x.keys() for y in list(x.values())])
    job2 = sum([y for x in results if 'heavy_duty2' in x.keys() for y in list(x.values())])

    assert job1 == 15
    assert job2 == 21

plus another one with some exceptions
def test_parallel_processing_exceptions():
    def heavy_duty1_raises(arg1, arg2, arg3):
        raise ValueError('Exception raised')
        return arg1 + arg2 + arg3

    def heavy_duty2(arg1, arg2, arg3):
        return arg1 * arg2 * arg3

    task_list = [
        {'func': heavy_duty1_raises, 'tasks': [{'arg1': 1, 'arg2': 2, 'arg3': 3}, {'arg1': 1, 'arg2': 3, 'arg3': 5}]},
        {'func': heavy_duty2, 'tasks': [{'arg1': 1, 'arg2': 2, 'arg3': 3}, {'arg1': 1, 'arg2': 3, 'arg3': 5}]},
    ]

    results = par_proc(task_list)

    job1 = sum([y for x in results if 'heavy_duty1' in x.keys() for y in list(x.values())])
    job2 = sum([y for x in results if 'heavy_duty2' in x.keys() for y in list(x.values())])

    assert not job1
    assert job2 == 21

Hope that is helpful.
Answer: in "from queue import Queue" there is no module called queue, instead multiprocessing should be used. Therefore, it should look like "from multiprocessing import Queue"
Comment: While years late, using `multiprocessing.Queue` is correct. The normal `Queue.Queue` is used for python *threads*. When you try to use `Queue.Queue` with multiprocessing, copies of the Queue object will be created in each child process and the child processes will never be updated. Basically, `Queue.Queue` works by using a global shared object, and `multiprocessing.Queue` works using IPC. See: https://stackoverflow.com/questions/925100/python-queue-multiprocessing-queue-how-they-behave
Answer: Just made a simple and general example for demonstrating passing a message over a Queue between 2 standalone programs. It doesn't directly answer the OP's question but should be clear enough indicating the concept.
Server:
multiprocessing-queue-manager-server.py
import asyncio
import concurrent.futures
import multiprocessing
import multiprocessing.managers
import queue
import sys
import threading
from typing import Any, AnyStr, Dict, Union

class QueueManager(multiprocessing.managers.BaseManager):

    def get_queue(self, ident: Union[AnyStr, int, type(None)] = None) -> multiprocessing.Queue:
        pass

def get_queue(ident: Union[AnyStr, int, type(None)] = None) -> multiprocessing.Queue:
    global q

    if not ident in q:
        q[ident] = multiprocessing.Queue()

    return q[ident]

q: Dict[Union[AnyStr, int, type(None)], multiprocessing.Queue] = dict()
delattr(QueueManager, 'get_queue')

def init_queue_manager_server():
    if not hasattr(QueueManager, 'get_queue'):
        QueueManager.register('get_queue', get_queue)

def serve(no: int, term_ev: threading.Event):
    manager: QueueManager
    with QueueManager(authkey=QueueManager.__name__.encode()) as manager:
        print(f"Server address {no}: {manager.address}")

        while not term_ev.is_set():
            try:
                item: Any = manager.get_queue().get(timeout=0.1)
                print(f"Client {no}: {item} from {manager.address}")
            except queue.Empty:
                continue

async def main(n: int):
    init_queue_manager_server()
    term_ev: threading.Event = threading.Event()
    executor: concurrent.futures.ThreadPoolExecutor = concurrent.futures.ThreadPoolExecutor()

    i: int
    for i in range(n):
        asyncio.ensure_future(asyncio.get_running_loop().run_in_executor(executor, serve, i, term_ev))

    # Gracefully shut down
    try:
        await asyncio.get_running_loop().create_future()
    except asyncio.CancelledError:
        term_ev.set()
        executor.shutdown()
        raise

if __name__ == '__main__':
    asyncio.run(main(int(sys.argv[1])))

Client:
multiprocessing-queue-manager-client.py
import multiprocessing
import multiprocessing.managers
import os
import sys
from typing import AnyStr, Union

class QueueManager(multiprocessing.managers.BaseManager):

    def get_queue(self, ident: Union[AnyStr, int, type(None)] = None) -> multiprocessing.Queue:
        pass

delattr(QueueManager, 'get_queue')

def init_queue_manager_client():
    if not hasattr(QueueManager, 'get_queue'):
        QueueManager.register('get_queue')

def main():
    init_queue_manager_client()

    manager: QueueManager = QueueManager(sys.argv[1], authkey=QueueManager.__name__.encode())
    manager.connect()

    message = f"A message from {os.getpid()}"
    print(f"Message to send: {message}")
    manager.get_queue().put(message)

if __name__ == '__main__':
    main()

Usage
Server:
$ python3 multiprocessing-queue-manager-server.py N

N is a integer indicating how many servers should be created. Copy one of the <server-address-N> output by the server and make it the first argument of each multiprocessing-queue-manager-client.py.
Client:
python3 multiprocessing-queue-manager-client.py <server-address-1>

Result
Server:
Client 1: <item> from <server-address-1>

Gist: https://gist.github.com/89062d639e40110c61c2f88018a8b0e5

UPD: Created a package here.
Server:
import ipcq

with ipcq.QueueManagerServer(address=ipcq.Address.AUTO, authkey=ipcq.AuthKey.AUTO) as server:
    server.get_queue().get()

Client:
import ipcq

client = ipcq.QueueManagerClient(address=ipcq.Address.AUTO, authkey=ipcq.AuthKey.AUTO)
client.get_queue().put('a message')

Comment: Getting this error type object 'Address' has no attribute 'DEFAULT'
Comment: It's renamed to 'AUTO', just updated the answer. Thanks.
Answer: We implemented two versions of this, one a simple multi thread pool that can execute many types of callables, making our lives much easier and the second version that uses processes, which is less flexible in terms of callables and requires and extra call to dill.
Setting frozen_pool to true will freeze execution until finish_pool_queue is called in either class.
Thread Version:
'''
Created on Nov 4, 2019

@author: Kevin
'''
from threading import Lock, Thread
from Queue import Queue
import traceback
from helium.loaders.loader_retailers import print_info
from time import sleep
import signal
import os

class ThreadPool(object):
    def __init__(self, queue_threads, *args, **kwargs):
        self.frozen_pool = kwargs.get('frozen_pool', False)
        self.print_queue = kwargs.get('print_queue', True)
        self.pool_results = []
        self.lock = Lock()
        self.queue_threads = queue_threads
        self.queue = Queue()
        self.threads = []

        for i in range(self.queue_threads):
            t = Thread(target=self.make_pool_call)
            t.daemon = True
            t.start()
            self.threads.append(t)

    def make_pool_call(self):
        while True:
            if self.frozen_pool:
                #print '--> Queue is frozen'
                sleep(1)
                continue

            item = self.queue.get()
            if item is None:
                break

            call = item.get('call', None)
            args = item.get('args', [])
            kwargs = item.get('kwargs', {})
            keep_results = item.get('keep_results', False)

            try:
                result = call(*args, **kwargs)

                if keep_results:
                    self.lock.acquire()
                    self.pool_results.append((item, result))
                    self.lock.release()

            except Exception as e:
                self.lock.acquire()
                print e
                traceback.print_exc()
                self.lock.release()
                os.kill(os.getpid(), signal.SIGUSR1)

            self.queue.task_done()

    def finish_pool_queue(self):
        self.frozen_pool = False

        while self.queue.unfinished_tasks > 0:
            if self.print_queue:
                print_info('--> Thread pool... %s' % self.queue.unfinished_tasks)
            sleep(5)

        self.queue.join()

        for i in range(self.queue_threads):
            self.queue.put(None)

        for t in self.threads:
            t.join()

        del self.threads[:]

    def get_pool_results(self):
        return self.pool_results

    def clear_pool_results(self):
        del self.pool_results[:]

Process Version:
  '''
Created on Nov 4, 2019

@author: Kevin
'''
import traceback
from helium.loaders.loader_retailers import print_info
from time import sleep
import signal
import os
from multiprocessing import Queue, Process, Value, Array, JoinableQueue, Lock,\
    RawArray, Manager
from dill import dill
import ctypes
from helium.misc.utils import ignore_exception
from mem_top import mem_top
import gc

class ProcessPool(object):
    def __init__(self, queue_processes, *args, **kwargs):
        self.frozen_pool = Value(ctypes.c_bool, kwargs.get('frozen_pool', False))
        self.print_queue = kwargs.get('print_queue', True)
        self.manager = Manager()
        self.pool_results = self.manager.list()
        self.queue_processes = queue_processes
        self.queue = JoinableQueue()
        self.processes = []

        for i in range(self.queue_processes):
            p = Process(target=self.make_pool_call)
            p.start()
            self.processes.append(p)

        print 'Processes', self.queue_processes

    def make_pool_call(self):
        while True:
            if self.frozen_pool.value:
                sleep(1)
                continue

            item_pickled = self.queue.get()

            if item_pickled is None:
                #print '--> Ending'
                self.queue.task_done()
                break

            item = dill.loads(item_pickled)

            call = item.get('call', None)
            args = item.get('args', [])
            kwargs = item.get('kwargs', {})
            keep_results = item.get('keep_results', False)

            try:
                result = call(*args, **kwargs)

                if keep_results:
                    self.pool_results.append(dill.dumps((item, result)))
                else:
                    del call, args, kwargs, keep_results, item, result

            except Exception as e:
                print e
                traceback.print_exc()
                os.kill(os.getpid(), signal.SIGUSR1)

            self.queue.task_done()

    def finish_pool_queue(self, callable=None):
        self.frozen_pool.value = False

        while self.queue._unfinished_tasks.get_value() > 0:
            if self.print_queue:
                print_info('--> Process pool... %s' % (self.queue._unfinished_tasks.get_value()))

            if callable:
                callable()

            sleep(5)

        for i in range(self.queue_processes):
            self.queue.put(None)

        self.queue.join()
        self.queue.close()

        for p in self.processes:
            with ignore_exception: p.join(10)
            with ignore_exception: p.terminate()

        with ignore_exception: del self.processes[:]

    def get_pool_results(self):
        return self.pool_results

    def clear_pool_results(self):
        del self.pool_results[:]

def test(eg):
        print 'EG', eg

Call with either:
tp = ThreadPool(queue_threads=2)
tp.queue.put({'call': test, 'args': [random.randint(0, 100)]})
tp.finish_pool_queue()

or
pp = ProcessPool(queue_processes=2)
pp.queue.put(dill.dumps({'call': test, 'args': [random.randint(0, 100)]}))
pp.queue.put(dill.dumps({'call': test, 'args': [random.randint(0, 100)]}))
pp.finish_pool_queue()

Answer: A multi-producers and multi-consumers example, verified. It should be easy to modify it to cover other cases, single/multi producers, single/multi consumers.
from multiprocessing import Process, JoinableQueue
import time
import os

q = JoinableQueue()

def producer():
    for item in range(30):
        time.sleep(2)
        q.put(item)
    pid = os.getpid()
    print(f'producer {pid} done')

def worker():
    while True:
        item = q.get()
        pid = os.getpid()
        print(f'pid {pid} Working on {item}')
        print(f'pid {pid} Finished {item}')
        q.task_done()

for i in range(5):
    p = Process(target=worker, daemon=True).start()

# send thirty task requests to the worker
producers = []
for i in range(2):
    p = Process(target=producer)
    producers.append(p)
    p.start()

# make sure producers done
for p in producers:
    p.join()

# block until all workers are done
q.join()
print('All work completed')

Explanation:

Two producers and five consumers in this example.
JoinableQueue is used to make sure all elements stored in queue will be processed. 'task_done'  is for worker to notify an element is done. 'q.join()' will wait for all elements marked as done.
With #2, there is no need to join wait for every worker.
But it is important to join wait for every producer to store element into queue. Otherwise, program exit immediately.
Comment: If I can afford max of 20 processes and I start with 15 producer and 5 workers, how can I move producers to workers when all the tasks are put in the queue ?

i.e. In this case of 2 producer and 5 workers , once producer has produce all the tasks and added them to the queue, does those 2 process works as worker ? as in total 7 workers ?
Comment: If the consumers go too fast and make the queue empty, they will all complete before the producers have finished enqueuing all the tasks.  This is why the accepted answer is better because it uses a sentinel.
Comment: Also you should invoke the `.start()` for each Process object in a separate loop after first creating your processes or else they will run sequentially.


How to default checkboxes to checked while maintaining custom return value

Question: I'm working on a migration project. I have a form in a custom module that creates a series of checkboxes for URLs to scrape. The form logic looks like this:
foreach ( \Drupal\scrape\Api\Scraper::loadUrls() as $index => $url ) {
  $form['url_' . $index] = [
    '#type' => 'checkbox',
    '#title' => $url . ' — ' . \Drupal::l('(link)', Url::fromUri($url)),
    '#return_value' => $url,
  ];
}

This code does what I want it to do, but all the checkboxes default to 'unchecked'.
'url_0' => integer0
'url_1' => string(27) "https://myurl1.com"
'url_2' => integer0

I want these checkboxes to default to checked. I couldn't find documentation on how to do it in Drupal 8, so I tried several things, and finally hit upon '#checked' => TRUE, (or any value that evaluates truthily). Now the checkboxes are defaulting to checked.
However, even though I have #return_value specified, the value returned in the form submission is now TRUE (from kint):
'url_0' => boolTRUE
'url_1' => boolTRUE
'url_2' => boolTRUE

Since any truthy value defaults the checkbox to checked, I tried '#value' =>  $url, but then upon submission, the values don't indicate their checked status:
'url_0' => string(86) "https://myurl0.com"
'url_1' => string(27) "https://myurl1.com"
'url_2' => string(29) "https://myurl2.com"

I guess my final option is to have a separate, hidden value for each URL, and map checkbox values to see if the user wants that URL included.
Is there a way I can default my checkboxes to TRUE, while maintaining their #return_value?
Answer: Sorry for such a late response but it is possible, I ran into this same issue using 8.5.3 the below configuration worked for me.  You just have to use both together, i successfully passed numbers, text, bool (becomes 0 or 1), and arrays through #return_value.  The class was just for formatting, you can leave that out if you're not using it.
'checkbox_1' => [
    '#type' => 'checkbox',
    '#title' => t('Checkbox 1'),
    '#default_value' => 1,         // Default to checked
    '#return_value' => "checked",  // Return the value if checked
    '#attributes' => ['class'=>['check_group']],
],
'checkbox_2' => [
    '#type' => 'checkbox',
    '#title' => t('Checkbox 2'),
    '#default_value' => 0,         // Default to unchecked
    '#return_value' => "checked",  // Return the value if checked
    '#attributes' => ['class'=>['check_group']],
],

Output will look like this if you dpm($form_state->getValues()) in your submit or validate function.
Array
(
    [checkbox_1] => checked
    [checkbox_2] => 0
)

Answer: You should use the #default_value property.
E.g:
 $form['demo'] = [
    '#type' => 'checkbox',
    '#title' => $this->t('Demo'),
    '#default_value' => TRUE,
 ];

Answer: I guess you can't. I just created hidden fields, and mapped them in the submit.
This is what I basically did:
public function submitForm(array &$form, FormStateInterface $form_state) {

  $values = $form_state->getValues();

  $i = 0;

  while ( TRUE ) {
    if ( ! isset($values['checkboxes'][$i] ) {
      break;
    }
  }
}

That's not verbatim but basically what I did. Simple looping and checking for the checkbox value.
Comment: can you share sample?
Comment: @dresh see edited question. Not a complicated solution.


Subadditivity question

Question: I learned Fekete's subadditivity lemma for sequences, so I would like to know some  interesting examples of arrays that are subadditive?
Any hint is welcome, thanks in advance.
Comment: @chris for sequences, not for "arrays".
Comment: Why didn't you give some more flesh to the bones, as asked for in @s.harp 's first comment? As it stands, your post is fairly vague...
Comment: What is an array and what does it mean for an array to be sub-additive?
Comment: @s.harp Definitions and the lemma are at https://en.wikipedia.org/wiki/Subadditivity
Comment: @alans   Did you search the internet? Do you want examples of sequences of real numbers (such as $a_n = \frac{1}{n}$), or not limited to (e.g. sequences in probability theory)?
Answer: There is, IMHO, a very elegant application of Fekete's Lemma in fixed point theory. It allows to generalize the Banach fixed point theorem with assumption on the map which is invariant of the metric up to equivalence.

All the details of the following discussion can be found in the Section 3.2. of
Khamsi, M. A., & Kirk, W. A. (2011). An introduction to metric spaces and fixed point theory (Vol. 53). John Wiley & Sons.
For a general discussion on subadditive functions ("generalizations" of Fekete's Lemma) I recommend chapter VII of Hille, E., & Phillips, R. S. (1996). Functional analysis and semi-groups (Vol. 31). American Mathematical Soc..

Let $(M,d)$ be a complete metric space and $f\colon M \to M$. $x\in M$ is a fixed point of $f$ if $f(x)=x$. Define the smallest Lipschitz constant of $f$ with respect to $d$ as
$$Lip(f,d) = \inf\{\alpha \mid d(f(x),f(y))\leq \alpha \,d(x,y),\forall x,y\in M\}.$$
Banach fixed point theorem says that

Theorem 1. If $Lip(f,d)<1$, then $f$ has a unique fixed point in $M$.

Now, for every $k\geq 1$, let $f^k = f\circ \ldots \circ f$ be the $k$-the composition of $f$ with itself.
The following result generalizes the Banach fixed point theorem stated above.

Theorem 2. If $\displaystyle\inf_{k\geq 1}Lip(f^k,d)^{1/k}<1$, then $f$ has a unique fixed point in $M$.

Fekete's Lemma help a lot to understand the real improvement offered by Theorem 2 over Theorem 1.
Consider the sequence
$$a_k = \ln(Lip(f^k,d))\qquad \forall k\geq 1.$$
Then, $(a_k)_k$ is subadditive since, by the property of Lipschitz constants, for $k,m\geq 1$, we have
\begin{align*}
a_{k+m} &= \ln(Lip(f^{k+m},d))\\
&\leq\ln(Lip(f^k,d)Lip(f^m,d))\\
&\leq \ln(Lip(f^k,d))+\ln(Lip(f^m,d))=a_k+a_m.
\end{align*}
Fekete's Lemma now implies that
$$\inf_{k\geq 1}\frac{a_k}{k} = \lim_{k\to \infty }\frac{a_k}{k}.$$
Since the logarithm is continuous, bijective and monotonic, we have
$$\inf_{k\geq 1}\frac{a_k}{k}= \inf_{k\geq 1}\frac{\ln(Lip(f^k,d))}{k}=\ln\Big(\inf_{k\geq 1}Lip(f^k,d)^{1/k}\Big).
$$
With a similar observation for $\lim_{k\to \infty }\frac{a_k}{k}$, we conclude that
$$\inf_{k\geq 1}Lip(f^k,d)^{1/k}=\lim_{k\to \infty } Lip(f^k,d)^{1/k}$$
Suppose now that $d'$ is another metric on $M$ for which there exists $0<\alpha,\beta$ such that
$$ \alpha\, d(x,y) \leq  d'(x,y) \leq \beta \, d(x,y)\qquad \forall x,y\in M.\tag{$*$}$$
We show using Fekete's Lemma that for every metric $d'$ as above, it holds $$\displaystyle\inf_{k\geq 1 }Lip(f^k,d)^{1/k}=\inf_{k\geq 1 }Lip(f^k,d')^{1/k}.$$
This is a quite fundamental observation as it implies that the assumption $\displaystyle\inf_{k\geq 1}Lip(f^k,d)^{1/k}<1$ in Theorem 2 only depends on $d$ up to equivalence. This is a good thing since the fact of having a fixed point only depends on $f$.
It holds
$$\frac{\alpha}{\beta}Lip(f^k,d)\leq Lip(f^k,d') \leq \frac{\beta}{\alpha}Lip(f^k,d)\qquad \forall k \geq 1,$$
and thus
$$\lim_{k\to \infty }\frac{\ln(Lip(f^k,d))}{k}=\lim_{k\to \infty }\frac{\ln(Lip(f^k,d'))}{k}.$$
With Fekete's Lemma we conclude that
$$\inf_{k\geq 1 }Lip(f^k,d)^{1/k}=\inf_{k\geq 1 }Lip(f^k,d')^{1/k}$$
for every metric $d'$ satisfying $(*)$.
Answer: Take any subadditive function $f:[0,\infty)\rightarrow[0,\infty)$ and define $a_n:=f(n)$.
If by "array" you mean a double indexed sequence for which
$$a_{k+m,l+n}\leq a_{k,l}+a_{m,n}$$
You can always take two subadditive sequences $a_n,b_m$ and define $c_{n,m}=a_n+b_m$ or $c_{n,m}=a_{n+m}$. More generally, you can take any subadditive function $f:[0,\infty)^2\rightarrow[0,\infty)$ and define $a_{m,n}:=f(m,n)$. For example, any supremum of linear functions is sub-additive so
$$a_{m,n}=\sup_{(\alpha,\beta)\in A}\{\alpha m+\beta n\}$$
where $A$ is some subset of $\mathbb{R}^2$.

Subadditivity question


Dell R710 RAM upgrade

Question: I'm upgrading the RAM in my Dell PowerEdge R710, and I think I have a solid enough understanding of ranking and how that should effect my purchase.  Using the guide below -
http://www.dell.com/downloads/emea/products/pedge/en/installing_upgrading_ddr3_memory_11g.pdf

could someone give me a sanity check?

It looks like there is no way to fill the server with dual ranked DIMMs.  So I couldn't buy a bunch of 4GB dual ranked DIMMS for this server, because I would hit the max of 8 ranks on whatever channel that is (bank A channel 1 and bank B channel 1 are the same channel, right?  so - 2 x 6 = 12, which is greater than the 8 max).
Also, can anyone give me a valid scenario where someone would want to buy a 16GB quad ranked DIMM?  I'm confused as to why those would even be sold... unless it's for single CPU configurations, where only one bank would be used and you'd have some ranks to burn through?
Any additional detail to aid in my understanding would be appreciated!  Thanks.
Comment: The Dell support will help you. Or better: the sales department helps you out. That's their job and they should guarantee that everything they tell you is going to work.
Answer: I solved this by following my best understanding and limiting the number of ranks being used.  I had no problem with the RAM I purchased, after changing how it was laid out on the motherboard a few times.

Dell R710 RAM upgrade


variance of theoretical mean of y at given value of x in regression

Question: Could someone tell me where I might find information on deriving the standard error used in confidence and prediction intervals of y at a given value of x on a regression line. I can't find anything on the internet. Thanks for any help.
Comment: If it's Simple Linear Regression, then the expression can be derived easily by expanding the variance term into variance of both the parameters and the covariance between them.Also in S.E for Prediction Intervals,sigma square term is added. You may refer to this problem [link] (https://www.cheenta.com/isi-mstat-psb-2013-problem-4-linear-regression/)
Comment: Thanks a ton, Napoleon. I was able to derive it like you said.
Answer: In the regression model, knowing that Y is Normally distributed,  one can use properties of the multivariate normal distribution to provide an answer.
I would recommend you look at Wikipedia's comment, especially for example, in the case of the bivariate normal distribution, and in particular, the conditional distribution of Y given X.
There is also an associated expression for the variance of the conditional distribution, which answers your question. See also, summary presented on  Page 4 of this educational reference.
Note, for a prediction interval, one adds an added sigma square term.
Comment: Thanks, AJKOER. To follow up on your last sentence, since we add the variances of the mean of y and the error of y, am I correct that they are assumed independent, since there is no covariance term in the formula?


gunzip "No space left on device"

Question: I have a gzipped file that Ubuntu reports is 2,119,952,848 bytes in size.  When I try to gunzip on a mount with 12GB free space, I get the error "No space left on device".
Here's where it gets really strange... the file is a compressed mysqldump so should be highly compressible.  However, gzip reports a large negative compression ratio:
ubuntu:/mnt/tmp$ gzip -l my.sql.gz
         compressed        uncompressed  ratio uncompressed_name
         2119952848          1471106218 -44.1% my.sql

Any idea what might be going on with this?
Comment: Can you give the gzip version? "gzip --version"
and your architecture (32 or 64 bits) ?
Answer: May be the uncompressed counter is only 32 bit and so it reset to this wrong value
Answer: Try to verify size of uncompressed data:
gzip -dc my.sql.gz | wc

If it will be smaller than your free disk size - check limits/quota.
Answer: Could be:

The file system /mnt/tmp only supports file sizes up to 2GB.
/mnt/tmp really is running out of space (gunzip -c FILE | wc -c will show you the uncompressed size)
Answer: could be inodes!  df -i
Answer:
What about the root "reserved" area on the disk? Depending on how this is accounted for in df, some of your empty space might not be available to you and reserved to the root user.
Another issue is that the compressed file is not released and deleted until the uncompression is complete.  Your partition might not be able to hold both of these with the amount of space you have available.

gunzip "No space left on device"


Classification test set accuracy lower than validation

Question: My validation set is having high accuracy for classification. But low on test data as data is not balanced. Please suggest here.
What could be the reason for getting high accuracy on the validation set but low accuracy on the test set?
Answer: It sounds like you are overfitting on the test set, but it is impossible to say for sure, as your question only barely describes what you are doing. What kind of problem do you have? Multi-label, multi-class? Or something more simple? How do you measure accuracy? How many classes? Which framework do you use?
Take a good look at your data and its distributions. If they are unbalanced, balance them before training. There is no point in doing vanilla training of a network on two classes, if the distribution is 1:99. You must address this issue, either via re-balancing your dataset (e.g., by oversampling the underrepresented class) or by suitably modifying your loss function.
Assuming the issue is not balancing but overfitting, use data augmentation and regularization. Some possible things to try are to reduce the model capacity (fewer parameters) and to add something like dropout and/or weight decay.


Determining if a set of vectors is linearly independent

Question:
Assume that $v$ and $w$ are linearly independent vectors in $\Bbb R^3$. Determine which of the following sets of vectors is linearly independent.
a) $\left\{v\times w,0,v\right\}$
b) $\left\{v,v\times w,v\times \left(v\times w\right)\right\}$
c) $\left\{v,v+w,v+2w\right\}$
d) $\left\{v+w,v-w\right\}$
e) $\left\{v\times w,2v,\:w\times v\right\}$
f) $\left\{v\times w,\begin{pmatrix}1\\ 2\\ 3\end{pmatrix},v,w\right\}$

I am not entirely sure how to approach this problem. My intuition is telling me that $a$ is dependent because of the zero vector, $c$ is dependent because one row is a multiple of a another row, and I thought $f$ was dependent because $v$ and $w$ are vectors in $\Bbb R^3$ whereas that set has $4$ vectors.
But I know that I am supposed to find whether or not there is a non-trivial linear combination of vectors that equals zero. But I am not sure how to do that in this case as no numbers are have been provided.
Any help would be highly appreciated!
Comment: So many questions! What did you try so far? a) is really for you!
Comment: You're right about (a) and (f), and while your conclusion is correct about (c), I don't see how "one row is a multiple of another".
Comment: That statement, if you replace "row" with "vector", does apply to another option here, namely (e).
Comment: I edited out the bold face.  Use it to highlight one key word, or maybe one sentence, or a header.  Do not write half your post in bold-face.
Comment: @likey Please remember that you can choose an answer among the given if the OP is solved, more details here https://meta.stackexchange.com/questions/5234/how-does-accepting-an-answer-work
Answer: HINT
Note that
a) is not since it contains $\vec 0$
b) $\vec v \times (\vec v \times \vec w)=\vec 0$
c) $\vec u_2-\vec u_1=\vec v+\vec w-\vec v=\vec w$ and $\vec u_3-\vec u_2=\vec w$
d) prove that $a\vec u_1+b\vec u_2=0 \iff a=b=0$
e) $\vec v \times \vec w=-\vec w \times \vec v$
f) #vectors > dimension


Is this infinite product for zeta(2) trivial?

Question: I have crafted an infinite product for zeta(2) shown here. Euler's prime product is the only one I'm aware of. In checking Math World, I don't see any products. Is that because they are trivial?
Comment: On the page linked above, we find some interesting patterns. The mathematics may be trivial, but the patterns might not be trivial.
Answer: Your product can be easily rewritten as $\dfrac B{A^2}$, where $A=\displaystyle\prod_{n=1}^\infty\bigg(1-\dfrac{x^2}{n^2}\bigg)$ with $x=\dfrac16$, and
$B=\displaystyle\prod_{n=2}^\infty\frac{n^3+1}{n^3-1}$, which, believe it or not, is telescopic, and yields a rational number as result.
$(B$ has been evaluated on this site several times$)$. So the answer to your question is yes.
Comment: Is $A$ evaluated using the infinite product for $\sin$ or something similar?
Comment: I found $B$ on this site. $A$ I found several years ago.
Comment: @AkivaWeinbergercolumbus: Yes.


Could anyone clarify why this Boolean expression AB'+AB'AC' = AB'?

Question: Could anyone clarify why this Boolean expression AB'+AB'AC' = AB'? I did not understand what happened to the C'
Answer: $$AB'+AB'AC'=AB'(1+AC')=AB'1=AB'$$
Answer: If $AB'$ is true, the $+$ will not change that.
If $AB'$ is false, then $(AB')(AC')$ is also false, no matter what $C'$ is. Then the entire expression is false, which is equal to $AB'$
Comment: Thank's Andrei!!!!
Comment: No Problem. If you find the answer useful, please mark it as accepted


Show that $f\not\in L^1(\mathbb{R})$ but $f\in L^2(\mathbb{R})$.

Question: Let $f(x)=\min\{1,1/|x|\}$. Show that $f\not\in L^1(\mathbb{R})$ but $f\in L^2(\mathbb{R})$.
I think I want $||f||_1$ to not be finite and $||f||_2<\infty$, but I'm confused as to how to compute those from a minimum function.  Would I just take $f(x)=\frac{1}{|x|}$ and compute as normal?  Or is there something else I'm missing?
Any help would be greatly appreciated.
Comment: It would be better to have your function written in piecewise form, like $f(x)=\begin{cases} f_1(x) & x \in A_1 \\ f_2(x) & x \in A_2 \end{cases}$ and so on, because then $\| f \|_1 = \int_{A_1} |f_1| + \int_{A_2} |f_2| + \dots$ and similar for $\| f \|_2$. Can you do that for your function? Hint: there will be three pieces.
Answer: Set $f(x)=\min\{1,1/|x|\}$.
We may write
$$
\begin{align}
\int_\mathbb{R}|f(x)|dx&=\int_{|x|\leq 1}|f(x)|dx+\int_{|x|> 1}|f(x)|dx\\\\
&=\int_{|x|\leq 1}1\:dx+\int_{|x|> 1}\frac1{|x|}dx\\\\
\end{align}
$$ and the latter integral is divergent, thus $f\not\in L^1(\mathbb{R})$.
Similarly$$
\begin{align}
\int_\mathbb{R}|f(x)|^2dx&=\int_{|x|\leq 1}|f(x)|^2dx+\int_{|x|> 1}|f(x)|^2dx\\\\
&=\int_{|x|\leq 1}1\:dx+\int_{|x|> 1}\frac1{|x|^2}dx\\\\
\end{align}
$$ and both integrals are convergent, thus $f\in L^2(\mathbb{R})$.


Using boost::gil::color_converted_view with boost::gil::for_each_pixel

Question: I realized that writing to gil::color_converted_view doesn't affect the underlying view's data. I wonder if that's correct?
For example, let's say that I want to write a program that will take the value of the red channel and set the blue channel's value to half of that. Here's my failed attempt:
template <typename SrcView>
void half_red_to_blue(SrcView & view)
{
    // Since SrcView might be RGB or BGR or some other types,
    // I decided to use a color_converted_view to ensure that I'm
    // accessing the correct channels
    typedef gil::color_converted_view_type<SrcView, gil::rgb8_pixel_t>::type MyView;
    MyView my_view = gil::color_converted_view<gil::rgb8_pixel_t>(view):
    struct my_lambda
    {
        void operator()(gil::rgb8_pixel_t & p)
        {
            p[2] = p[0] / 2;
        }
    };
    gil::for_each_pixel(my_view, my_lambda());
}

However, it only works when SrcView is actually gil::rgb8_view_t. If I call, e.g. half_red_to_blue<gil::bgr8_view_t>(view), the view is not changed at all! I inspected a little in the debugger and it seems the write operation is writing to some kind of proxy location instead of the original pixels.
Any ideas? Thanks in advance!
Answer: This is valid behaviour in Boost.GIL because the colour components of pixel are touched only upon access of pixel. You can modify my_lambda::operator() to use get_color to trigger colour component access.


Does kernel regression preserve monotonicity?

Question: Consider the Kernel regression estimator:
$$\hat{y}(x)=\frac{\sum_{i=1}^n{K(x-x_i)y_i}}{\sum_{i=1}^n{K(x-x_i)}},$$
where $x,x_1,\dots,x_n\in\mathbb{R}^d$, $y_1,\dots,y_n\in\mathbb{R}$, where $K:\mathbb{R}^d\rightarrow(0,\infty)$ is a strictly positive valued, differentiable kernel function, with a unique maximum at $0$.
Suppose further that for all $i,j\in\{1,\dots,n\}$, if $x_i\le x_j$ then $y_i \le y_j$.
Is it the case that for all $x\in\mathbb{R}^d$:
$$\frac{\partial\hat{y}(x)}{\partial x} \ge 0?$$
It seems obvious in the $d=1$ case, but even there I haven't been able to prove it. It's unclear to me if it holds for $d>1$. If it only holds under additional assumptions on $K$ I'd be interested in them.

Notation:
$\frac{\partial \hat{y}(x)}{\partial x}$ is the column vector of partial derivatives of $\hat{y}(x)$, i.e. the (transposed) Jacobian.
For vectors $a=[a_1,\dots,a_d]^\top$ and $b=[b_1,\dots,b_d]^\top$, $a\le b$ if and only if $a_i \le b_i$ for all $i\in\{1,\dots,d\}$.
Comment: Is that a directional derivative or what?  $d/dx$ with $x\in \mathbb{R}^d$ and $>0$ like a scalar all seem to all be conflicting symbols to me?
Comment: What do you mean by "mode"?
Comment: @jvc I meant maximum. (Kernel functions are often scaled PDFs, hence the alternate terminology.) Changed now!
Comment: This certainly depends on the kernel. Say, if the kernel is very narrow, then $\hat y(x)$ is close to $\sum_{i=1}^n y_i\mathbf{1}_{x=x_i}$, which is far from being monotone.
Comment: My intuition tells me that if it is symmetric and at the origin it has the maximum value monotonic property in the sense you wrote will be preserved.
Comment: I think you better copy the answer here and mark it as a Wiki. Of course add reference and credit to the original.
Comment: @Renard It was intended to be a vector of partial derivatives, i.e. a Jacobian. I've used less lazy notation now!
Comment: I am assuming the kernel is strictly positive, which I thought would be sufficient to avoid the problem you point to.
Comment: This question has now been answered fairly comprehensively on MO here: https://mathoverflow.net/questions/358127/does-kernel-regression-preserve-monotonicity
Answer: The below is taken from Iosif Pinelis's answer here: https://mathoverflow.net/a/358137/96912

The result is false in general even for $d=1$. E.g., let
$$K=f_r+f_s,$$
where $f_t$ is the density of $N(0,t^2)$. Then for $x_i=y_i=i$ ($\forall i=1,\dots,n$) and
$$(n,r,s,x_*)=\Big(3,\frac{427}{215},\frac{1}{1547},\frac{472}{473}\Big)$$
we have
$$\hat y'(x_*)=-527.1\ldots<0.$$
Here is the graph $\{(x,\hat y(x))\colon\frac{471}{473}\le x\le\frac{475}{473}\}$:

We see a very narrow dip.

However, $\hat y'\ge0$ if $K$ is log concave. Indeed, letting
$$k_i:=K(x-x_i)\quad\text{and}\quad k'_i:=K'(x-x_i),$$
we have
$$
\begin{aligned}
2\Big(\sum_{i=1}^n k_i\Big)^2\hat y'(x)
&=\sum_{i,j=1}^n(k'_i y_i k_j-k_i y_i k'_j+k'_j y_j k_i-k_j y_j k'_i) \\
&=\sum_{i,j=1}^n(y_i-y_j)\Big(\frac{k'_i}{k_i}-\frac{k'_j}{k_j}\Big)k_ik_j\ge0,
\end{aligned}\tag{1}
$$
because $y_i$ is increasing in $i$ and
$$\frac{k'_i}{k_i}=(\ln K(x-x_i))' \tag{2}$$
is increasing in $i$; the latter holds because $x_i$ is increasing in $i$ and $(\ln K)'$ is decreasing (since $K$ is log concave). (In particular, any normal density is log concave.)

In the case $d>1$, the desired monotonicity fails to hold in general even when $K$ is log concave. E.g., let
$$K(u_1,\dots,u_d):=\exp\{u_1u_2-u_1^2-\cdots-u_d^2\},$$
$n=2$, $x_1=(0,\dots,0)$, $x_2=(0,1,0,\dots,0)$, $y_1=0$, and $y_2=1$.
Then $(\partial_1\ln K)(u_1,\dots,u_d)=u_2-2u_1$, where $\partial_1$ denotes the partial derivative with respect to the first coordinate. On the other hand (cf. (1) and (2)), $(\partial_1\hat y)(0,\dots,0)$ equals $l'_2-l'_1$ in sign, where
$$l'_1:=(\partial_1\ln K)(0,\dots,0)=0$$
and
$$l'_2:=(\partial_1\ln K)(0,-1,0,\dots,0)=-1<0=l'_1.$$
So, $(\partial_1\hat y)(0,\dots,0)<0$, as claimed.


How to make a query for Big Brother nomination?

Question: I have a table with three columns:

Participant.
First vote participant (two points).
Second vote participant (one point).

CREATE TABLE `voting` (
  `participant` varchar(11) NOT NULL,
  `firstVote` varchar(11) NOT NULL,
  `secondVote` varchar(11) NOT NULL
)

I am stuck trying to write a query that returns two columns: the participant and the number of points they received.
Example:

participant
firstVote
secondVote

Mary
Jane
Holland

Jane
Mary
Holland

Donatella
Mary
Jane

Holland
Mary
Jane

Query:

participant
votesReceived

Mary
6

Jane
4

Holland
2

Donatella
0
Answer: Use a self LEFT join and conditional aggregation:
SELECT v1.participant,
       COALESCE(
         SUM(
           2 * (v1.participant = v2.firstVote) +
           (v1.participant = v2.secondVote)
         ),
         0
       ) AS votesReceived
FROM voting v1 LEFT JOIN voting v2
ON v1.participant IN (v2.firstVote, v2.secondVote)
GROUP BY v1.participant
ORDER BY votesReceived DESC;

Or:
SELECT v1.participant,
       SUM(
         CASE v1.participant
           WHEN v2.firstVote THEN 2
           WHEN v2.secondVote THEN 1
           ELSE 0
         END
       ) AS votesReceived
FROM voting v1 LEFT JOIN voting v2
ON v1.participant IN (v2.firstVote, v2.secondVote)
GROUP BY v1.participant
ORDER BY votesReceived DESC;

See the demo.


Statistical test to verify when two similar time series start to diverge

Question: As from title I would like to know if exist a statistical test that can help me to identify a significant divergence between two similar time series.
Specifically, looking the figure below, I would like to detect that the series start to diverge at time t1, ie when the difference between them starts to be significant.
Moreover, I would also detect when the difference between the series returns to be not significant.
Is there any useful statistical test to do this?
Answer: There are a few ways that come to mind. The first is to take the difference between the two series and create a "new series" . Analyze that series and empirically identify Pulses,Level Shifts/ Local Time Trends and a possible ARIMA component. The results will/might suggest any identifiable divergence. A second approach is to build a common ARIMA model for both time series and use the CHOW TEST to test for statistically significant parameters.
Answer: Another approach that might work is to consider algorithms for change detection.
A first idea is to apply a change detection method like CUSUM on both series and compare the change points. In your example it is very likely that the red series will yield a change point at t1 whereas the yellow one would not. Interestingly, both red and yellow would probably both yield a change point at the first bump of the curve (depending on the sensibility of the CUSUM parameters) but you really don't mind as they behave similarly.
Answer: Some options you may wish to consider:

If you are looking for identifying a significant difference, a Statistical Process Control (SPC) chart using Western Electric rules might also help you identify that it is occurring.  As @IrishStat has suggested, graphing the difference between the two time series is the best start.  Then applying SPC rules based on analysis of a stable period of the two time series is good.

https://en.wikipedia.org/wiki/Western_Electric_rules

A more detailed pragmatic approach is chronostatistics that is gathering wide acceptance in the mining industry for identifying change and the specific characteristics of noise in time series data.  As you can imagine, in an environment where you are interested in 0.001% of the material, uncertainty in the sampling and variability of the process must be understood to know if you have a difference in two time series.

As a mine process engineer, I am used to dealing with time series data that is a lot more noisy than this and chronostatistics (proponents include Pierre Gy and Francis Pitard) allows identification of the errors introduced by the data sampling technique and other aspects of data gathering.  More accessible papers (i.e. easier for non-professional statisticians) have been written by Tim Napier-Munn who has a very application-based approach to assessing time series data.
I am not aware of any open source papers but both of these authors have published through Elsevier.


Impulse response: does it represent a cabinet only or the whole chain used to record it?

Question: I'm trying to grasp a concept of an impulse responses and getting a little bit confused. Most commercially available ones are meant to represent a cabinet with some microphones, but not the amp. And this is really what makes me confused: in order to make cabinet to make sound, you obviously need an amp.
On the other hand, I'm pretty sure you can find impulse responses intended to represent the whole chain. For example, in this video Ola makes one.
The concept of "snapshot of the whole chain" makes much more sense to me, because as I wrote above, you need an amp to make cabinet to make sound.
So, can anyone please clarify this to me? Also I'm not quite sure how I am intended to use IR - for example, after all the effect pedals and my preamp?
Comment: Since it is not possible to prevent the sonic characteristic of the amp (or mic preamp) from being part of a captured IR, the "solution" for getting "only the sound of the cabinet" is to make sure all of the other components have the most neutral sonic characteristics possible. In practice this is feasible for IRs of guitar cabinets because the amount of color added by a guitar speaker cabinet is much greater than the amount of color added by available neutral-sounding amplifiers. The qualities of the amp are rendered inaudible by the qualities of the cabinet.
Answer: The impulse response can represent any linear time invariant parts of the signal path. In practice this means frequency response and delay, so it is good representing room response and equalisation, and cabinet and microphone response as long as they are not driven to distortion. On the other hand, it is not suitable at all for representing distortion and modulation effects.
Answer: An Impulse Response is anything you want it to be.
It's a convolution 'map' of an output for a given measured input. It can be anything from a simple mono impulse of a telephone or portable radio, an amp alone or amp & speakers with a specific mic in a specific room in stereo, right up to a 10s reverb in a warehouse… in 5.1 surround.
Google 'impulse reponse generator' for many ways to make them yourself. I've never tested any of those results, as I have the one supplied with Altiverb
Comment: It's the 'difference' between the input & output. How you record it depends on the result you need. Recording a guitar amp alone, you'd use direct input & output, no mic at all, of course. To record just a cab you'd need a truly clean power amp input to not affect your signal, & appropriately-placed mic[s] to record the result. To get just the cab without the room it's in, you'd need a very dead room. To get the full volume gamut as well as frequency spectrum you need several IRs which you then interpolate so the result changes with volume too.
Comment: -1, there are lots of things that an impulse response cannot capture.  Including most guitar pedals (distortion, tremolo, autoharmonizer, compressor, etc etc)
Comment: Did I understand your answer correctly - if it is a map of an input to some system to its output, than in order to create a response of a cabinet you need a generator capable of generating a signal suitable for the cabinet, and a mic(s)? And that is the way to exclude an amp from an equation.
Comment: @AlexeyMalev I'd suggest asking a separate question about how to capture an IR.
Comment: @Edward - the very first convolution process I ever heard was in the mid 90s when someone was demoing a prototype of this type of modelling in my office. It was a guitar amp - we had to go out to lunch while it processed, but it did it. I honestly don't know the difference between that and an 'IR' mathematically. [It turned out the guy's idea wasn't patentable, so we 'borrowed it' - the rest is history :\
Comment: @Tetsujin cool story, but it doesn't make the answer any less wrong.
Comment: @ojs - so… explain to me the difference. Both are convolution processes, as far as I can see.
Comment: @Tetsujin convolution by impulse response is convolution by impulse response, and I don't really understand what the "both" would be. On the other hand, the "anything you want it to be" part is totally wrong unless all you ever want is cabinet, room and microphone. The distortion, tremolo, etc are nonlinear and plain IR won't capture them.
Comment: @ojs - well… thanks for not explaining that at all.
Comment: @Tetsujin what do you need to have explained?


Prove that $(S, d)$ is a metric space, i.e. $d$ is a distance function on $S$ .

Question: Let $S$ be the set of nonempty compact subsets of $\mathbb{R}^{2}$.
For any $r > 0$ and $K ∈ S$ , we define the $r -neighborhood$ of $K$ to be
$$B_{r}(K) := \{x ∈ \mathbb{R}^2: d(x, a) < r\; for \; some \;a ∈ K\}=\bigcup_{a\in K}B_{r}(a)$$
For $K_{1}$, $K_{2}\in S$, we define
$$d(K_{1},K_{2}):=\inf \{ r>0:K_{1} \subset B_{r}(K_{2}) \; and \; K_{2} \subset B_{r}(K_{1}) \}$$
Prove that $(S,d)$ is a metric space.
As we always do, I proved with the definition of a Metric Space.

Def.1: Consider $d(K_{a},K_{a})$ & $d(K_{a},K_{b})$ for all $K_{a} \neq K_{b}$

$$d(K_{a},K_{a})$$ $$= \inf \{r>0: K_{a} \subset B_{r}(K_{a}) \}$$ $$= \inf \{ r:r>0 \}=0$$ $$\Rightarrow d(K_{a},K_{a})=0 \; (\forall K_{a} \in S)$$
And, $$d(K_{a},K_{b})$$ $$= \inf \{r>0: K_{a} \subset B_{r}(K_{b}) \;and\;  K_{b} \subset B_{r}(K_{a}) \}$$
Since $K_{a} \neq K_{b}$, $\exists x \in K_{a}$, but $x \notin K_{b}$.
Then, $\exists y \in K_{b}$ s.t $d_{\mathbb{R}^2}(x,y)$ is least possible.
Then, $\inf \{r>0: K_{a} \subset B_{r}(K_{b}) \;and\;  K_{b} \subset B_{r}(K_{a}) \} \geq d_{\mathbb{R}^2}(x,y)>0$
Thus, $d(K_{a},K_{b})>0$
Hence, Def.1. satisfies for $(S,d)$

Def.2.: Consider $d(K_{a},K_{b})$ $\; \forall K_{a},K_{b} \in S$

$$d(K_{a},K_{b}) = \inf \{r>0: K_{a} \subset B_{r}(K_{b}) \;and\;  K_{b} \subset B_{r}(K_{a}) \}$$
If $K_{a}=K_{b}$, then $d(K_{a},K_{b})=0=d(K_{b},K_{a})$
If $K_{a} \neq K_{b}$, then $$d(K_{a},K_{b})$$ $$= \inf \{r>0: K_{a} \subset B_{r}(K_{b}) \;and\;  K_{b} \subset B_{r}(K_{a}) \}$$ $$ = \inf \{r>0: K_{b} \subset B_{r}(K_{a}) \;and\;  K_{a} \subset B_{r}(K_{b}) \}$$ $$=d(K_{b},K_{a})$$
Hence, Def.2. satisfies for $(S,d)$

Def.3.: Consider $d(K_{a},K_{c})$ $\; \forall K_{a},K_{c} \in S$

If $K_{a}=K_{c}$, then $d(K_{a},K_{c})=0\leq d(K_{a},K_{b}) + d(K_{b},K_{c})$ $\; \forall K_{b} \in S$ $\; \Rightarrow d(K_{a},K_{c}) \leq d(K_{a},K_{b}) + d(K_{b},K_{c})$
If $K_{a} \neq K_{c}$, then $\exists x \in K_{a}$, but $x \notin K_{c}$.
Also, $\exists y \in K_{c}$ s.t $d_{\mathbb{R}^2}(x,y)$ is greatest possible.
Let $z \in K_{b}$ s.t $d_{\mathbb{R}^2}(x,z),d_{\mathbb{R}^2}(z,y)$ are less than or equal to $d(K_{a},K_{b}),d(K_{b},K_{c})$
Note that, $\inf \{r>0: K_{a} \subset B_{r}(K_{c}) \;and\;  K_{c} \subset B_{r}(K_{a}) \} \leq d_{\mathbb{R}^2}(x,y)$ $\Leftrightarrow d(K_{a},K_{c})\leq d_{\mathbb{R}^2}(x,y)$
Also, by Triangle Inequality, $d_{\mathbb{R}^2}(x,y) \leq d_{\mathbb{R}^2}(x,z) + d_{\mathbb{R}^2}(z,y)$ $$\Rightarrow d(K_{a},K_{c}) \leq d_{\mathbb{R}^2}(x,y) \leq d_{\mathbb{R}^2}(x,z) + d_{\mathbb{R}^2}(z,y) \leq d(K_{a},K_{b}) + d(K_{b},K_{c})$$
Hence, Def.3. satisfies for $(S,d)$
Consequently, $(S,d)$ is a metric space. $$\blacksquare$$
Is it Okay to prove in this way? Is it clear enough?
Answer: There are some specific points that need attention:

Since $K_{a} \neq K_{b}$, $\exists x \in K_{a}$, but $x \notin K_{b}$.

This is not true when $K_a$ is a subset of $K_b$, and if $K_a$ is a strict subset of $K_b$ then $K_a \neq K_b$ holds as well. But, since $K_a \neq K_b$, then one of the sets must not be a subset of the other; either $K_a \not\subseteq K_b$ or $K_b \not\subseteq K_a$ holds. So, you can assume $x \in K_a \setminus K_b$ exists, or $x \in K_b \setminus K_a$ exists.
Of course, if you prove symmetry of $d$ first, then you can assume without loss of generality that $x \in K_a \setminus K_b$ exists, as the roles of $K_a$ and $K_b$ are now symmetric.
You also repeat this error when it comes to proving triangle inequality, but again, the roles of $K_a$ and $K_c$ are symmetric if you already have symmetry of $d$ proven.

Then, $\exists y \in K_{b}$ s.t $d_{\mathbb{R}^2}(x,y)$ is least possible.

This is very much true, but it might be worth citing a theorem to back this up, or at least mentioning that $d_{\Bbb{R}^2}(x, \cdot\,)$ is a continuous function, and thus achieves a minimum on the compact set $K_b$.

Def.2.: Consider $d(K_{a},K_{b})$ $\; \forall K_{a},K_{b} \in S$
$$d(K_{a},K_{b}) = \inf \{r>0: K_{a} \subset B_{r}(K_{b}) \;and\;  K_{b} \subset B_{r}(K_{a}) \}$$
If $K_{a}=K_{b}$, then $d(K_{a},K_{b})=0=d(K_{b},K_{a})$

This is fine, but you shouldn't need to treat the $K_a = K_b$ case separately. The definition of $d$ makes it pretty clear that the roles of $K_1$ and $K_2$ are interchangeable, and indeed the argument following does not use the assumption that $K_a \neq K_b$.

Let $z \in K_{b}$ s.t $d_{\mathbb{R}^2}(x,z),d_{\mathbb{R}^2}(z,y)$ are less than or equal to $d(K_{a},K_{b}),d(K_{b},K_{c})$

Such a $z$ may not exist. Let $K_a, K_b, K_c$ be closed unit balls centred at $(-1, 0), (0, 0), (0, 1)$ respectively. If we take $x \in K_a$ to be $(-2, 0)$, then by your construction, we must take $y$ to be the furthest point in $K_c$ from $x$, which is $(2, 0)$. We can then compute $d(K_a, K_b) = d(K_b, K_c) = 1$, and so we want $z$ such that
\begin{align*}
d_{\Bbb{R}^2}(z, x) &\le d(K_a, K_b) = 1 \\
d_{\Bbb{R}^2}(z, y) &\le d(K_b, K_c) = 1.
\end{align*}
Given $d_{\Bbb{R}^2}(x, y) = 4$, this would violate triangle inequality on $\Bbb{R}^2$.
This would seem to be a fatal error to your approach with the triangle inequality. You will need to come up with another approach.
P.S. This metric has a name: the Hausdorff metric.


Stop spool printing the sql query used (Oracle)

Question: I have this running in SQLplus
set feedback off
set pagesize 0
spool TypeDrop.sql
select distinct 'drop type '||object_name|| ';' from user_objects where object_type='TYPE';
spool off

It prints out to TypeDrop.sql:
SQL> select distinct 'drop type '||object_name||';' from user_objects where object_type='TYPE';
drop type ADDRESS_OBJTYP;
drop type PERSON_OBJTYP;
SQL> spool off

How do i get it to just output the drop statements? thanks
Answer: Use sqlplus -s. The -s flag means silent
Answer: Running from within a script, set termout off works. It doesnt if you are just typing into the terminal.
create myScript.sql:
set feedback off
set pagesize 0
set termout off
spool TypeDrop.sql
select distinct 'drop type '||object_name|| ';' from user_objects where object_type='TYPE';
spool off

and from your sqlplus prompt:
SQL> @myScript

will do the trick.
Answer: try
set echo off


Passar variável do php pelo javascript e depois pegar no php via post

Question: Bom, tenho uma variável PHP uma uma página index.php. E ao clicar, carrego outra parte da página dentro de uma div com javascript.
Ao carregar a página, a variável da index não é reconhecida dentro da pagina da div (que também é .php).
<?php
$variavel = 1234;
?>

Registro de peso

Comment: Pode passar para uma variável `Javascript` assim: `var x = '';`
Answer: bem simples imprima sua variavel dentro do script passando como parameto $_GET
veja o exemplo abaixo

...
Registro de peso

Agora dentro da sua pagina
pesoaluno.php
você receberá a variavel da seguinte forma
<?php
   echo $_GET['pegaAqui'];
?>

Comment: Deu certo. Muito obrigado!


Variance of a sum of identically distributed random variables that are not independent

Question: I am "new" to probability/statistics and I was hoping someone could verify that this is correct. Let $Y_1,\ldots,Y_n$ be random variables that follow a common distribution with mean $\mu$ and variance $\sigma^2$. However, I am not assuming that the variables are independent. I want to compute the expected value and variance of $X=Y_1+\cdots+Y_n$. Then,
$$\mathbb{E}(X) = \mathbb{E}\left(\sum_{i=1}^{n} Y_i \right) =  \sum_{i=1}^{n} \mathbb{E}\left( Y_i \right) = n \cdot \mu.$$
Moreover,
\begin{eqnarray*}
\operatorname{Var}(X) &=& \operatorname{Var}\left(\sum_{i=1}^{n} Y_i \right) \\
&=&  \sum_{i=1}^{n} \operatorname{Var}\left( Y_i \right) + 2\cdot\left(\sum_{1\leq i<j\leq n} \operatorname{Cov}(Y_i,Y_j) \right) \\
&=&  \sum_{i=1}^{n} \operatorname{Var}\left( Y_i \right) + 2\cdot\left(\frac{ n(n -1 )}{2}\right) \cdot C \\
&=& n \cdot \sigma^2 + n(n -1)\cdot C\\
&=& n\cdot ( \sigma^2 + (n-1)\cdot C),
\end{eqnarray*}
where we have used the properties of the variance, and the fact that for any $i\neq j$, the variables $Y_i,Y_j$ follow the same distribution,  and so $\operatorname{Cov}(Y_i,Y_j)=C$ for all $i\neq j$, for some constant $C$.
Question 1: Is this right? I have an uneasy feeling about assuming that all the covariances equal a constant C. Is there some additional hypothesis on the original random variables $Y_1,\ldots,Y_n$ that one needs to assume so that the covariance   $\operatorname{Cov}(Y_i,Y_j)=C$ independent of the chosen $i\neq j$?
Question 2: Also, is $C=\operatorname{Cov}(Y_i,Y_j)=\operatorname{Cov}(Y_i,Y_i)=\sigma^2$?
Thanks!
Comment: Your second question asks whether all the variables are perfectly correlated.  What do you think?
Comment: Your question asks whether it is possible to have Cov$(Y_i, Y_j)$ to have the same value $C$ for all $i \neq j$ where the $Y_i$ are identically distributed with variance $\sigma^2$. The answer is Yes, and the answers to [this question](http://stats.stackexchange.com/a/72798/6633) show that it must be the case that $C \in \left[-\frac{\sigma^2}{n-1}, \sigma^2\right]$. Also, identical distribution (or same value for mean) are not required; same value for variance is all that is needed.
Comment: [This answer][1] might help you understand the steps. In fact, it first proves your form, and then extend it further to another form suitable for the question.

  [1]:https://stats.stackexchange.com/a/388665/240395
Answer: Let $n = 3$. $Cov(Y_1,Y_2), Cov((Y_1,Y_3), Cov(Y_2,Y_3)$ can all be the same or different. They can take any values, so long as the 3 by 3 covariance matrix is positive semi-definite.
A very common case, much studied on this board, would be a 3 dimensional Normal, which in your case of identically distributed components would have equal elements on the diagonal.  The covariances between the components (and in this case, therefore also the correlations between the components) can be the same or different, as long as the covariance matrix is positive semi-definite.
As @whuber has hinted in a comment, while the scenario in your question 2 would be possible, and corresponds to perfectly correlated variables, which corresponds to a covariance matrix in which all entries are equal (and therefore the covariance matrix is singular), it certainly need not be the case.
Answer: Question 1
The additional assumption you need so that the covariances become equal is called `Exchangeability'. The definition of exchangeability is as follows:

Let $π$ denote an arbitrary permutation on n elements (that is, a function that maps $\{1, . . . , n\}$ onto itself). The finite sequence $X_1, . . . , X_n$ is said to be exchangeable if the joint distribution of the permuted random vector $(X_{π(1)}, . . . , X_{π(n)})$ is the same no matter which π is chosen. The infinite sequence X1, X2, . . . is said to be exchangeable if any finite subsequence is exchangeable.

Under exchangeability, the covariance between $X_i$ and $X_j$ is always the same, say $C$, when i $\neq$ j. Under the assumption of exchangeability, your computations are correct.
Question 2
Under exchangeability, this is not true. You will need a completely different assumption to make the covariance equal to the variance.
Check out this fragment of a book. They explain identically distributed by not identical with an example very similar to yours (instead of sum of random variables, they work with with the mean): http://sites.stat.psu.edu/~dhunter/asymp/lectures/p51to58.pdf


Deprecated version of DFP (v201508)

Question: We are getting emails from google about the latest version of Double-click for Publishers' (dpf) API.
Currently our site appears to still be using a deprecated version (v201508) and need to update to v201608 by November 30.
https://developers.google.com/doubleclick-publishers/docs/deprecation
Is this dfp api change is going to affect the dfp module?
Answer: As far as I can tell the DFP module doesn't load API version specific files and so unless you are using additional custom JS or loading the DFP JS files directly from your site (instead of pulling them from Google), there is no reason the API change should effect the module itself.

Deprecated version of DFP (v201508)


How does changing the training/test ratio, affect the RSE ans R2 metrics?

Question: If I change the training/test ratio from 90/10 to say 80/20 or 70/30, how does that affect the RSE and R2 metrics? I see changes in the results of the RSE and R2 when I implement the model, but I cannot grasp the underlying reasoning behind those.
For a 90/10 split (Linear regression model on the Boston Housing Dataset) I had the following results:
RSE for training = 0.02577413485637635 |||
RSE for testing = 0.0782435232487672
R2 for training = 0.7612428866092609 |||
R2 for testing = 0.7754233967070088.
For a 70/30 split I had the following:
RSE for training= 0.031168053531247658 |||
RSE for testing= 0.03775631124698821 |||
R2 for training= 0.7437862095331289 |||
R2 for testing= 0.8063900607780959
and for a 50/50 split I had these:
RSE for training= 0.038324514690667257 |||
RSE for testing= 0.03136868881928254 |||
R2 for training= 0.7353321015108675
R2 for testing= 0.7832763784168615.
What do those differences show overall?
Answer: Boston housing dataset is a small one (506 samples this link says), so you can certainly have variability. Pick a decent split and repeat your experiments several times. In the end, report the mean and std of the performance metric.


How list and filter stacks with boto3 cloudformation resource?

Question: I want to get a list of stacks based on a give set of states. I am aware of a solution using the boto3 client for cloudformation, e.g. from the documentation:
response = client.list_stacks(
    NextToken='string',
    StackStatusFilter=[
        'CREATE_IN_PROGRESS'|'CREATE_FAILED'|'CREATE_COMPLETE'|
        'ROLLBACK_IN_PROGRESS'|'ROLLBACK_FAILED'|'ROLLBACK_COMPLETE'|
        'DELETE_IN_PROGRESS'|'DELETE_FAILED'|'DELETE_COMPLETE'|
        'UPDATE_IN_PROGRESS'|'UPDATE_COMPLETE_CLEANUP_IN_PROGRESS'|
        'UPDATE_COMPLETE'|'UPDATE_ROLLBACK_IN_PROGRESS'|'UPDATE_ROLLBACK_FAILED'|
        'UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS'|'UPDATE_ROLLBACK_COMPLETE'|
        'REVIEW_IN_PROGRESS',
    ]
)

(side note: I think the | should be ,, but anyway)
However, I understand that the "recommended" way is using boto3 resources to access the AWS services. Unfortunately, the boto3 cloudformation resource has not method to list and filter stacks (see the documentation).
I would like to avoid looping over all stacks and check each retrieved stack object for its status.
The only way I found so far, is the work around by accessing the client of the resource, though it feels a bit hacky and there is no way to filter for instance on the stackname.
cf = boto3.Session().resource('cloudformation')
cf.meta.client.list_stacks(StackStatusFilter=['ROLLBACK_COMPLETE'])

Question: Any idea how I could get something like list_stacks(StackStatusFilter=..) for cf, which is a cloudformation.ServiceResource object that allows to filter on the status and the stackname?
Comment: For an environment with a significant number of stacks, probably using the boto3 client is the way to go, so that the filter is applied AWS side instead of you filtering it client side.  I liked this solution: https://stackoverflow.com/questions/39228551/listing-more-than-100-stacks-using-boto3
Answer: Interesting; that's an unfortunate asymmetry in the boto API. That said, I would stick with the resource and use a list comprehension vs. going through the meta attribute:
statuses = ['ROLLBACK_COMPLETE', 'CREATE_COMPLETE', 'UPDATE_COMPLETE']
cfn = boto3.resource('cloudformation')
stacks = [stack for stack in cfn.stacks.all() if stack.stack_status in statuses]

By dropping back to the meta.client, you get a dict response rather than Resource objects. If that's ok, just use cloudformation client instead of the resource. The comprehension will return Stack resource objects instead of dicts
Ultimately, either works and it's a matter of preference; just avoid mixing paradigms if you can.
Comment: Thanks @bimsapi that's basically my current solution. However, like this I have to iterate over the full set of stacks.
Comment: Understood. Without API support, that's the only option, and the overhead is modest. I think the more important issue is deciding on resource vs. client and sticking with it in your code for consistency.
Comment: cf.staks.all() only lists "active" stacks, so it doesn't work for statuses like DELETE_COMPLETE.  In other words, you will need to use client.list_stacks to query for those statuses instead.

stacks1 = cf.meta.client.list_stacks(StackStatusFilter=['DELETE_COMPLETE'])
stacks2 = [stack for stack in cf.stacks.all() if stack.stack_status in ['DELETE_COMPLETE']]

stacks1 get you list of deleted stacks while stacks2 will always give you an empty list.
Answer: This is an old post, but I thought this detailed answer would be helpful to others.
import boto3
COMMA = ','
GOOD_STATES = ('CREATE_COMPLETE,UPDATE_COMPLETE,UPDATE_ROLLBACK_COMPLETE').split(COMMA)
BUSY_STATES = ('CREATE_IN_PROGRESS,ROLLBACK_IN_PROGRESS,DELETE_IN_PROGRESS,UPDATE_IN_PROGRESS,UPDATE_COMPLETE_CLEANUP_IN_PROGRESS,UPDATE_ROLLBACK_IN_PROGRESS,UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS,REVIEW_IN_PROGRESS').split(COMMA)
BAD_STATES  = ('CREATE_FAILED,ROLLBACK_FAILED,DELETE_FAILED,UPDATE_ROLLBACK_FAILED,DELETE_COMPLETE,ROLLBACK_COMPLETE').split(COMMA)

cfn_rs = boto3.resource('cloudformation')

#use one of the following that serves your purpose

good_stacks = [stack for stack in cfn_rs.stacks.all() if stack.stack_status in GOOD_STATES]
good_stack_names = [stack.name for stack in cfn_rs.stacks.all() if stack.stack_status in GOOD_STATES]

busy_stacks = [stack for stack in cfn_rs.stacks.all() if stack.stack_status in BUSY_STATES]
busy_stack_names = [stack.name for stack in cfn_rs.stacks.all() if stack.stack_status in BUSY_STATES]

bad_stacks = [stack for stack in cfn_rs.stacks.all() if stack.stack_status in BAD_STATES]
bad_stack_names = [stack.name for stack in cfn_rs.stacks.all() if stack.stack_status in BAD_STATES]

my_stacks = [stack for stack in cfn_rs.stacks.all() if stack.stack_status not in BAD_STATES]
my_stack_names = [stack.name for stack in cfn_rs.stacks.all() if stack.stack_status not in BAD_STATES]

Please note the difference between the following two:

ROLLBACK_COMPLETE: Stack creation process started and rolled back after some resources were created due to some failures. Usually, you need to delete this stack before creating a stack with the same name.
UPDATE_ROLLBACK_COMPLETE: Update process started on an existing stack but rolled back after modifying some resources due to some failures.


How to get the frame from the vlc media player

Question: I'm currently using vlc library for python (python-vlc) to get the video stream from a sdp stream describe with a sdp file. I'm currenctly using this code :
import vlc
import time

instance = vlc.Instance()
player = instance.media_player_new()
media = instance.media_new("./bebop.sdp")
player.set_media(media)

player.play()
time.sleep(10)

Which works well to display the video. But I don't want to just display it, I want to use each frame from the video to make some image processing on it and then to display the modified frames.
I've read almost all the documentation and diverse post on forums but I can't find a way to do this.
I use vlc to get the video stream because open cv can't open it because of some ffmpeg error.
Answer: Get the frames per second from the video or default to 25 as a reasonable guess, then step through the paused video, using the frames per second as your jump forward each time.
import vlc
import time

instance = vlc.Instance('--no-xlib --quiet')
player = instance.media_player_new()
media = instance.media_new("./bebop.sdp"")
player.set_media(media)
player.play()
mfps = int(1000 / (player.get_fps() or 25))
player.set_time(30000) # start at 30 seconds
player.pause()
t = player.get_time()
for iter in range(30):
    t += mfps
    player.set_time(t)
    if player.get_state() == 3:
        player.pause()
    time.sleep(0.5)

Here, I start at the 30 second mark and then advance just 30 times.
You will probably want to keep advancing until player.get_state() == 6 indicating that the video has finished. [state 3 = playing]
Answer: # -*- coding: utf-8 -*-

import vlc
import time

def mspf(mp):
    """Milliseconds per frame"""
    return int(1000 // (mp.get_fps() or 25))

if __name__ == "__main__":
    instance = vlc.Instance()
    player = instance.media_player_new()
    media = instance.media_new("1.m4v")
    player.set_media(media)

    player.play()

    """Play 800th frames"""
    new_time = 800 * mspf(player)
    player.set_time(new_time)

    time.sleep(10)

Comment: This code just play the video at a specific time no ? I don't how it can help me to use the specific frame.
Answer: You need to use a bunch of libvlc calls, not sure what those are with your python wrapper but they should be exposed on the mediaplayer object.
https://www.videolan.org/developers/vlc/doc/doxygen/html/group__libvlc__media__player.html#gae46128c21d0d0151aca3ba017d1d6b35
https://www.videolan.org/developers/vlc/doc/doxygen/html/group__libvlc__media__player.html#ga2189c9deefc2154ddc5d5f1871b0a106
https://www.videolan.org/developers/vlc/doc/doxygen/html/group__libvlc__media__player.html#ga193de3e82f6de3f9d8f5ccd8b5f2aa98
https://www.videolan.org/developers/vlc/doc/doxygen/html/group__libvlc__media__player.html#ga024c0f0bba9d3b32a881c40f3cb479bb
You need to setup and use these callbacks. C# example using those APIs: https://github.com/ZeBobo5/Vlc.DotNet/blob/develop/src/Vlc.DotNet.Wpf/VlcVideoSourceProvider.cs


Mixed SLURM Cluster using Centos7 and Debian8

Question: I am using SLURM on a Centos7 cluster.
Is it possible to get SLURM working with various OSes (for example Centos7 and Debian8) on the same cluster?
Answer: Yes Slurm will run, but beware that programs compiled on one OS will not necessarily function on the other OS.
Comment: Yes that was actually the reason why i planned to do it. But then i found a way to install those programs at Centos. Thanks anyway!


Use custom HTML file for page created by Views?

Question: I use Views to create new content and a new page for the content, but how am I supposed to customize the HTML layout of the page? I can style elements of the page via CSS, but what if I wanted to have my own CSS3 (animated sprite) elements? For example, most CSS3 sprites require you to have divs contained in a specific manner, but those divs are not present in a Views created page and I see no viable way to manually add them? How would I go about doing this?
Answer: When you edit the View, under the Advanced settings on the right-hand side you should see "Theme information". If you click that it will give you the content of the theme templates for the View, which you can then override by copying into your theme, in a file with a corresponding name.
Comment: Don't modify the files that Views provides directly. Instead, copy them into your theme, rename them accordingly, and modify your version of them. That overrides the originals instead of modifying them directly. You could still put some awful markup in there, and no one can stop you, but you're not hacking Views or core or anything.
Comment: Thank you! Just to clarify though, is this considered 'hacking?' i.e. Will I mess things up just by hard-coding markup in these files? Similar to how you screw over Drupal by modifying core files.