Append to csv or create a new sheet in same csv

Question: The Script belows involves two parts. Fetching the security groups of account A and account B.
When It writes to csv, it first writes the security groups of account A and then overwrites the same csv with security groups
of account B. So when I download the file, it contains only the results from account B.
I need a solution either to append the security groups of B to the same sheet, Or to create a new sheet for account B in the same CSV. Please guide me how to achieve it.
Thanks for reading.
import boto3
import csv
import io
from io import BytesIO
from email.mime.text import MIMEText
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart

a_dev_client = boto3.client('ec2', region_name='eu-west-1')

def lambda_handler(event, context):

    sts = boto3.client('sts')
    b_dev = sts.assume_role(
        RoleArn='arn:aws:iam::2222222222222:role/Unused_Security_Groups',
        RoleSessionName='test'
    )

    b_dev_client = boto3.resource(
        'ec2',
        region_name='eu-west-1',
        aws_access_key_id=b_dev["Credentials"]["AccessKeyId"],
        aws_secret_access_key=b_dev["Credentials"]["SecretAccessKey"],
        aws_session_token=b_dev["Credentials"]["SessionToken"]
    )

    def a_dev():
        Account_Name = 'A'
        regions = ['eu-west-1','eu-west-2','us-east-2','us-west-1','us-west-2','us-east-1','ap-northeast-2','ap-southeast-1','ap-southeast-2','ap-northeast-1','ca-central-1','eu-west-3']
        csvio = io.BytesIO()
        writer = csv.writer(csvio)
        writer.writerow([
            'Account Name',
            'Region',
            'SecurityGroup-Id'
        ])
        for region in regions:
            ec2 = boto3.resource('ec2', region_name=region)
            sgs = list(ec2.security_groups.all())
            insts = list(ec2.instances.all())
            all_sgs = set([sg.group_id for sg in sgs])
            all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
            inst.security_groups])
            unused_sgs = all_sgs - all_inst_sgs
            for elem in unused_sgs:
                writer.writerow([
                    Account_Name,
                    region,
                    elem
                    ])
        s3 = boto3.client('s3')
        s3.put_object(Body=csvio.getvalue(), ContentType='application/vnd.ms-excel', Bucket='######', Key='Unused_Security_Groups.csv', ACL='public-read')

        csvio.close()
        s3.get_object(Bucket='#######', Key='Unused_Security_Groups.csv')

    a_dev()

    def b_dev():
        Account_Name = 'B'
        #regions = ['eu-west-1','eu-west-2','us-east-2','us-west-1','us-west-2','us-east-1','ap-northeast-2','ap-southeast-1','ap-southeast-2','ap-northeast-1','ca-central-1','eu-west-3']
        regions = ['eu-west-1']

        csvio = io.BytesIO()
        writer = csv.writer(csvio)
        writer.writerow([
            'Account Name',
            'Region',
            'SecurityGroup-Id'
        ])
        for region in regions:
            #ec2 = boto3.resource('ec2', region_name=region)
            sgs = list(b_dev_client.security_groups.all())
            insts = list(b_dev_client.instances.all())
            all_sgs = set([sg.group_id for sg in sgs])
            all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
            inst.security_groups])
            unused_sgs = all_sgs - all_inst_sgs
            for elem in unused_sgs:
                writer.writerow([
                    Account_Name,
                    region,
                    elem
                    ])
        s3 = boto3.client('s3')
        s3.put_object(Body=csvio.getvalue(), ContentType='application/vnd.ms-excel', Bucket='#######', Key='Unused_Security_Groups.csv', ACL='public-read')

        csvio.close()
        s3.get_object(Bucket='#######', Key='Unused_Security_Groups.csv')

    b_dev()

Update:
After I try to append the csv, I am getting "Read-only File System" Error.
    def b_dev():
        Account_Name = '2'
        regions = ['us-west-1']

        csvio = io.BytesIO()
        writer = csv.writer(csvio)
        fields=['Account Name','Region','SecurityGroup-Id']
        with open('Unused_Security_Groups.csv', 'a') as f:
            writer = csv.writer(f)
            writer.writerow(fields)
            for region in regions:
                #ec2 = boto3.resource('ec2', region_name=region)
                sgs = list(bi_dev_client.security_groups.all())
                insts = list(bi_dev_client.instances.all())
                all_sgs = set([sg.group_id for sg in sgs])
                all_inst_sgs = set([sg['GroupId'] for inst in insts for sg in
                inst.security_groups])
                unused_sgs = all_sgs - all_inst_sgs
                for elem in unused_sgs:
                    writer.writerow([
                        Account_Name,
                        region,
                        elem
                        ])
            s3 = boto3.client('s3')
            s3.put_object(Body=csvio.getvalue(), ContentType='application/vnd.ms-excel', Bucket='######', Key='Unused_Security_Groups.csv', ACL='public-read')

            csvio.close()
            s3.get_object(Bucket='######', Key='Unused_Security_Groups.csv')

Answer: First add a new sheet to a csv file is plain non sense. A CSV file is not an Excel file. It is a sequential text file this is automatically processed by Excel as a spreadsheet. But it has no support for multiple sheets.
But it is easy to append to a csv file: just open the file in append mode ("a") instead of overwrite mode ("w").
Comment: Thanks for the answer.. When I trying to append I am getting an error "Read Only file system. Can you please guide on this..
Comment: @itechworkerz: Of course you must have write permission on the file... That is not a programming problem, and I cannot help you with that.
Comment: The File in the s3 has the write permission, because the first function(Account A) writes to the file.
I assume the problem in the code, as its not writing(append) to the CSV. The problem I feel is with the code that I updated. Please check and guide.
Thank You
Comment: `    with open('Unused_Security_Groups.csv', 'a') as f:
            writer = csv.writer(f)
            writer.writerow(fields)
`
The problem is here.


Loading services from other .dll and run them isolated

Question: I'd like to run several services from different .dll's in a isolated way. Basically, all services are derived from RoleEntryPoint , and I want to load each one in a separated AppDomain and run it there in a different thread.
So far, I can locate the service and get its type:
        String pathToDll = @"C:\....\bin\Debug\ChildWorkerRole.dll";
        Assembly assembly = Assembly.LoadFrom(pathToDll);
        Type serviceType = assembly.GetTypes().SingleOrDefault(t => t.BaseType == typeof(RoleEntryPoint));

And also run it in the current AppDomain and Thread:
        RoleEntryPoint myRole2 = (RoleEntryPoint)Activator.CreateInstance(serviceType);
        if (myRole2.OnStart())
            myRole2.Run();

But when I try to run it in a separate in different AppDomain I get an exception:
        AppDomain domain = AppDomain.CreateDomain("MyNewDomain");
        RoleEntryPoint myRole = (RoleEntryPoint)domain.CreateInstanceFromAndUnwrap(pathToDll, serviceType.FullName);
        if (myRole.OnStart())
            myRole.Run();

This exception:
System.Runtime.Serialization.SerializationException was unhandled
  Message=Type 'ChildWorkerRole.WorkerRole' in assembly 'ChildWorkerRole, Version=188.8.131.52, Culture=neutral, PublicKeyToken=null' is not marked as serializable.
  Source=mscorlib
  StackTrace:
       at System.AppDomain.CreateInstanceFromAndUnwrap(String assemblyName, String typeName)
.......

The funny thing is that ChildWorkerRole is actually marked with the SerializableAttribute ... but may be because RoleEntryPoint is not, it cannot be done.
Any idea or workaround?
thanks!
Answer: In order to work with a type in a separate AppDomain, it, and all of the types you directly use, need to be either marked as [Serializable], or you need to derive them from MarshalByRefObject.
If this is not possible, the only real option is to make a proxy class that you can use which does follow the above criteria, and allow it to manage your types internally.


In YQL is there an option to only limit the first item of each feed?

Question: In an YQL query how do I return only the first item of each feed ( example with 2 feeds but i will have more )
select channel.title,channel.link,channel.item.title,channel.item.link
from xml where url in(
  'http://code.flickr.com/blog/feed/rss/',
  'http://www.quirksmode.org/blog/index.xml'
)

I know of the tail option but that is set over the final result, how can i do this per feed
thanks in advance
Answer: You could make multiple queries using the yql.query.multi table like:
select *
from yql.query.multi where queries in (
    "select channel.title,channel.link,channel.item.title,channel.item.link from xml where url='http://code.flickr.com/blog/feed/rss/' limit 1",
    "select channel.title,channel.link,channel.item.title,channel.item.link from xml where url='http://www.quirksmode.org/blog/index.xml' limit 1"
);

Or, you could just filter your original query such that only one result from each feed is returned:
select channel.title,channel.link,channel.item.title,channel.item.link
from xml where url in(
    'http://code.flickr.com/blog/feed/rss/',
    'http://www.quirksmode.org/blog/index.xml'
 ) | unique(field="channel.link")

Answer: By adding LIMIT 1 at the end
select channel.title,channel.link,channel.item.title,channel.item.link
from xml where url in(
  'http://code.flickr.com/blog/feed/rss/',
  'http://www.quirksmode.org/blog/index.xml'
) LIMIT 1

Comment: No, LIMIT 1 limits the final resultset, I want to have the first item of each feed, now it just returns the first item of the combined feeds.


Guitar making a weird sound

Question: In short I've got an electric guitar that makes a buzzing sound when I'm not touching metal, I know it sounds like the thousandth time this has been asked but bare with me, it doesn't go away ONLY by touching the metal of the guitar, if I'm touching the body of the guitar (so no metal just wood) and then I touch a piece of metal it will go away. For example if I sit it on my lap and touch my MacBook it will go away, again I'm not touching any metal on the guitar BUT there is an exception.
If I plug my computer in to charge and then touch it the buzzing gets significantly louder, the same thing happens if I just touch my phone in general (doesn't have to be charging and the buzzing gets louder the closer my guitar is to my phone) and the whole time I'm not touching the metal of my guitar, if I do while touching either my phone or my charging MacBook it gets slightly quieter but doesn't go away.
I am very curious as to if this is just a faulty ground that gets louder when I touch my phone for whatever reason or if its something else. Again I am stressing I am NOT touching the metal of the guitar and it gets louder and can CAN either touch the metal of the guitar OR touch something else metal with the guitar just resting on me for it to go away.
For information I am using a Spider V 20 amp and a Yamaha guitar (I can't remember the exact model)
Comment: This really is a duplicate of  [Noise while not touching guitar strings or metal parts](https://music.stackexchange.com/questions/7491/noise-while-not-touching-guitar-strings-or-metal-parts) but with one additional factor. When your computer is plugged in you gain addition noise from the switch-mode power supply that these things use. There is no fix for that other than unplug it or see if you can find a proper, heavyweight core-wound transformer, fully earthed, with similar output spec. [This, btw, is not an actual earthing issue, per se]
Comment: @Tetsujin The linked question says *"when I touch the strings or touch any metal parts (strings, bridge or the metal portion near the output jack), the noise goes away completely"*. This question says "it **doesn't go** away ONLY by touching the metal of the guitar".
Answer: I was about to chalk it up to your guitar electronics, which would've been sad but a cheap fix, but the amp might very well be the problem. Even for a beginner, Line 6 amps and especially the Spider series, are some of the absolute worst amplifiers money can buy. Not only does it sound like a box of drunk wasps, but they often have engineering issues that aren't worth the repair price. Have you tried playing with a friend's amp on your guitar or their guitar in your amp?
Comment: Welcome to the Music SE.  Your advice is helpful, but it could have been conveyed without belittling the OP's choice in gear.
Comment: Wasn't meant to sound belittling, I absolutely sympathize with not being able to afford decent gear. I used a Chinese strat knockoff for my first three years of guitar and after eight, I'm still stuck with my crappy 20-watt digital. That's why I suggested trying combos of theirs and others' gear to eliminate the issue. It's very likely just a cheap amp, but I'm willing to recognise it may not be.
Answer:
It doesn't go away ONLY by touching the metal of the guitar

It may be that there's an earthing problem in your instrument (or cable, or amp). Normally, noise that's picked by your body and retransmitted to the pickups is reduced significantly when you touch the strings or bridge of your guitar, because that grounds you. See https://hazeguitars.com/blog/electric-guitar-string-ground.
See if the problem goes away when you touch the 'shield' of the cable that plugs into the guitar (while it's plugged into the amp). If it does, then that part of the chain is all fine, and there's probably an issue with the earth aspect of the wiring in the guitar.

Guitar making a weird sound


python dictionary manipulation not working

Question: I am doing data processing to filter and sort and etc...
for stock in context.timeSeriesData:
    try:
        if context.timeSeriesData.iloc[1,:][stock][2] > 0 and context.timeSeriesData.iloc[0,:][stock][2] > 0:
            if context.timeSeriesData.iloc[1,:][stock][2] > context.timeSeriesData.iloc[0,:][stock][2]:
                context.tradingList[stock] = []
                increment = context.timeSeriesData.iloc[1,:][stock][2] / context.timeSeriesData.iloc[0,:][stock][2]
                context.tradingList[stock].append(increment)
    except:
        pass

try:
    sorted_tradingList = sorted(context.tradingList.items(), key = lambda x: x[1], reverse = True)
except:
    pass

try:
    sorted_tradingList.clear()
except:
    pass

for x,stock in sorted_tradingList:
    if len(context.tradingBook) < (1.25/context.weightLong):
        if stock in data and 'price' in data[stock]:
            context.tradingBook[stock] = []
            context.tradingBook[stock].append(0.0)
            order_target_percent(stock, context.weightLong)

for k, v in context.tradingBook.iteritems():
    v[0] +=1

timeToSell = {k:v for (k,v) in context.tradingBook.iteritems() if v[0] == 10}

for stock in timeToSell:
    if stock in data and 'price' in data[stock]:
        TargetHedge = (len(context.tradingBook) - 1) * context.weightShort
        order_target_percent(stock, 0.0)
        del context.tradingBook[stock]

if len(timeToSell)>0:
    timeToSell.clear()

TargetHedge = (len(context.tradingBook)) * context.weightShort
order_target_percent(sid(8554), TargetHedge)

In the first loop context.timeSeriesData will generate many sets of company with three data elements.  For example, the first two is shown below.  Lets look at AAPL, 311 is an industry code (which I don't care to use it now), the second element 18024000000 is net income, and 3.08 is the earning per share (which I want to manipulate)
DataFrame:
          Equity(24 [AAPL])
1  [311.0, 18024000000.0, 3.08]
2  [311.0, 18024000000.0, 3.08]

In the data above first line indicates data received prior day and the second line indicates the data just arrived.  So in the future if the DataFrame: changes to as shown below. The the computer should put that into context.tradingBook
DataFrame:
          Equity(24 [AAPL])
1  [311.0, 18024000000.0, 3.08]
2  [311.0, 19033000000.0, 3.79]

so the purpose is that when a new data is received, and the earning per share improves it will be stored into context.tradingList.  Then it shorts by the highest "increment"
sorted_tradingList = sorted(context.tradingList.items(), key = lambda x: x[1], reverse = True)

Then the code
for k, v in context.tradingBook.iteritems():
    v[0] +=1

will add one day each time this whole block of code is processed (once a day).  As you can see it starts with 0 day, context.tradingBook[stock].append(0.0), but each day it will add one day until 10 days have passed, at which point, I will try to sell them.
order_target_percent(stock, context.weightLong) is a code to buy or sell stocks which is unique to my system so don't worry about it.  My question is...  why it is not adding days.  I am having issue with
for k, v in context.tradingBook.iteritems():
    v[0] +=1

It is currently not adding days.  What am I doing wrong.  Also I have another question, if you are interested in following along this far.  But I will let you know once we get the first step down.  Thank you so much.
Comment: Your  bare except is not a good idea, you should catch specific errors or at least  print/log any execpetions
Comment: @UjaeKang exceptions, by definition, are things that don't happen normally.
Comment: If you want to change a value in the dict to an empty set use `your_dict[key] = set()`, once you  don't have any other references to objects, when you rebind a name in python `my_whatever = whatever` the old data will be gc'd.  Not sure if you mean dict or set as `{}` would create a dictionary, you need `set()` when creating an empty set
Comment: Anyway, since you don't handle the exceptions you'd better not catch them. For example the second `try`-`catch` if catched you will end up without `sorted_tradingList` defined which means that subsequent code couldn't succeed anyway.
Comment: I know.  But when I run through the debug mode, nothing appears malfunctioning.
Comment: But codes do work as expected when I run through the debug mode.  Problem happens when it gets to the     [for k, v in context.tradingBook.iteritems():
        v[0] +=1]   line
Comment: @UjaeKang, what is the problem that happens when you have don't catch any exceptions?
Comment: Padraic,  I updated the code and I am realizing that your were right, it is not processing [sorted_tradingList.clear()]  error message is [AttributeError: 'list' object has no attribute 'clear'].  I wasn't aware of it, because of no exception comment.
Comment: Yes, lists don't have a clear method, if you want to delete the list use `del sorted_tradingList`, if you want to create an empty list use `sorted_tradingList = []`, but if you do that then your next loop will have nothing to iterate over as `sorted_tradingList` will be empty
Comment: Padraic, would you please offer me a solution, how I should do to clean out the data element of dictionary so that it would be back to empty set like when I first declared it?  [context.tradingList = {}]
Comment: I think I got to the problem.  I am a bit slow.  I will give you an update.  That was where the problem was anyways...
Answer: The problem was at
try:
    sorted_tradingList.clear()
except:
    pass

It was not working and I wasn't aware of it.
Comment: lesson: remove all bare `except`


Bash. Para que serve export?

Question: Observando os scripts Bash, percebi que as variáveis podem ser criadas das seguintes maneiras:
script1.sh:
variable1=Ola
export variable2="Ola de novo"

Procure saber mais sobre a diferença dessas duas maneiras de declarar as variáveis. E pelo que eu entendi export serviria para fazer variáveis globais.
Mas, rodando o script e tentando imprimir a variável global variable2, recebo o mesmo resultado que a variável local do meu script `` variable1````
$ ./script1.sh
$ echo $variable1
$
$ echo $variable2
$

Usando o comando source ou usando . E executando o mesmo procedimento acima, obtém como resultado a impressão das duas variáveis.
$ source script1.sh
$ echo $variable1
Ola
$ echo $variable2
Ola de novo

Neste ponto entendi que o objetivo de export não é declarar variáveis globais.
Qual é o objetivo do comando export?
Comment: Essa pergunta pode te ajuda: https://stackoverflow.com/questions/1158091/defining-a-variable-with-or-without-export
Answer: Quando você define uma variável usando export, essa variável é copiada para os filhos do shell quando ele faz fork. Sem o export, a variável não é copiada para os filhos.
Quando você chama ./script1.sh, o shell faz um fork e o script é executado no processo filho, e não no shell onde você chamou o comando. Portanto, a definição das variáveis é feita no processo filho, e não no shell onde você está logado. Por isso você não consegue dar echo nas variáveis no processo pai. O processo filho não mexe no environment do processo pai.
O programa source é um programa que é built-in no próprio shell, e serve para executar um script no próprio shell, sem fazer fork. Por isso que quando você chama source script1.sh, as variáveis aparecem no shell onde você está logado.

Bash. Para que serve export?


Is range of A equal to range of AB?

Question: I have a simple question.
Actually I just tried to solve the question 'Is range of $A$ equal to range of $AA^TA$'.
But it looks like much general question to ask 'Is range of $A$ equal to range of $AB$'.
In my first impression, I think range of $A$ is same as range of $AB$ because no matter which vectors come after, matrix $A$ would linearly transform it to the column space of $A$.
But, it looks wrong.
Can you help me to understand it?
And how can I prove range of $A$ is equal to $AA^TA$?
I am studying power iteration in randomized SVD and it said they are same but I cannot get it.
https://dl.acm.org/doi/pdf/10.1145/3004053?download=true (in page 4)
I guess something like '$Null(A)$ is equal to $Null(A^TA)$'would be helpful, but hard to apply it.
Thank you very much.
Comment: Consider the range of $AB$ if $B=0$.
Answer: In general, you can say that
$$Range(AB) \subseteq Range(A)$$
because if $v\in Range(AB)$, then
$$ABx=v\implies A(Bx) = v\implies v\in Range(A).$$
The opposite is generally false, since, as Arthur suggested, if $B=0$, then $AB=0$ that has usually a very different range from the one of $A$.

For your case, here's a simple explanation using SVD. If $A=U\Sigma V$ is the SVD, then (assuming  $A$ real) $AA^TA = U\Sigma^3 V$. From the outer expansion
$$
A =\sum_i \sigma_i u_iv_i^T
$$
you can see that the range of $A$ is the span of $u_1,\dots,u_r$, where $r$ is the rank of $A$, or equivalently the number of non-zero singular values.
Since
$$
AA^TA = U\Sigma^3 V = \sum_i \sigma_i^3 u_iv_i^T
$$
you see that $AA^TA$ has the same rank of $A$, and the range of $AA^TA$ is still the span of $u_1,\dots,u_r$.
Answer: As to the general question, $Range(AB)\subseteq Range(A)$, but $Range(AB)=Range(A)$ if $B$ is square and full rank:
$$Range(A)=Range(ABB^{-1})\subseteq Range(AB)\subseteq Range(A)$$
Comment: @twosigma: Right! Edited. Thanks.

Is range of A equal to range of AB?


Music multiple simultaneous tones, python

Question: I am working on an interesting project relating solving mathematical problems and music.  It is easy to generate a specific note (or tone):
ton = amplitude * sin(2pi * frequency * time / samplerate)

I'm working in python, and have code to do this much from http://code.google.com/p/scitools/.
def note(frequency, length, amplitude=1, sample_rate=44100):
    time_points = numpy.linspace(0, length, length * sample_rate)
    return numpy.sin(2 * numpy.pi * frequency * time_points) * amplitude

Of course, in real music, there are generally multiple tones being played during the same time step.  I tried to do this by generating then summing two tones, i.e.:

twotone = note(440, 2)+note(261.63, 2)

but this just gives crap.  How would I mathematically encode more than one simultaneous tone?
Answer: you should half the individual amplitudes when using two tones, or the total amplitude potentially doubles.  If you use more than two, you should mix in a ratio that reflects their relative volume, with a total amplitude of one.


Why A.class.getClassLoader() does not return the loader used by loader.loadClass("A")

Question: In this test case we are loading a class A through a specific class loader instance, i.e. GwClassLoader. However, the A.class.getClassLoader() does not retrieve that class loader? This is the test case:
class A{
  static {
    ClassLoader loader = A.class.getClassLoader();
    System.out.println("A.<cinit>: " + loader);
  }
}

class ClassLoaderGw extends ClassLoader { }

public class App {
  public static void main(String [] args)  throws ClassNotFoundException,  InstantiationException,  IllegalAccessException {
    ClassLoaderGw loader = new ClassLoaderGw();
    System.out.println("Main: " + loader);
    Class<?> klass = loader.loadClass("A");
    Object obj = klass.newInstance();
  }
}

And, this is the output of running the class App:
Main: ClassLoaderGw@6d06d69c
A.<cinit>: sun.misc.Launcher$AppClassLoader@73d16e93

How can I get the instance of GwClassLoader inside the static constructor of the class A?
Answer: The ClassLoaderGw simply extends ClassLoader without overriding any methods.  Therefore, it will follow the strategy of the standard loadClass method (javadoc) ... which is to delegate to the parent classloader first.
In this case, your A class is on the parent classloader's classpath, so that is what will load it.
The final piece of the puzzle is that getClassLoader() returns the classloader that actually loaded the class, not the classloader that you asked to do the loading.

How can I get the instance of GwClassLoader inside the static constructor of the class A?

Basically, you can't ... unless GwClassloader was responsible for loading the class A.
Could you make it work?  Well ... if you used a different class loading strategy you could.  Maybe.  If GwClassloader was able to find the bytecodes for A and call defineClass itself, then it would be the classloader for A.
However, you need to worry about the possibility that class A could also be loaded by the parent classloader.  If that happens you could have two distinct A types in existence at the same time.  This can lead to class cast exceptions in unexpected places.
Comment: Solved. By invoking `defineClass`, then it would be the `classloader` for intended class. Here it is the new implementation for [`ClassLoaderGw`](https://gist.github.com/fmcarvalho/1fecd5310f74c1153154886202a1e133#file-classloadergw-java)
Answer: This is described in the Javadoc:

Loads the class with the specified binary name. The default implementation of this method searches for classes in the following order:

Invoke findLoadedClass(String) to check if the class has already been loaded.
Invoke the loadClass method on the parent class loader. If the parent is null the class loader built-in to the virtual machine is used, instead.
Invoke the findClass(String) method to find the class.

Your ClassLoader has the sun.misc.Launcher$AppClassLoader1 as a parent, because as described in the Javadoc of the default constructor (which is implicitly invoked in the ClassLoaderGw class, because there is no explicit constructor):

Creates a new class loader using the ClassLoader returned by the method getSystemClassLoader() as the parent class loader.


Barracuda Bouncing Emails

Question: I'm not too familiar with all the intricacies of email servers and am curious what you guys think this looks like.
Is the receiving blocking it or is this something on our end?
The receiving company has stated that they have whitelisted our IP and they do not route to any websitewelcome.com nameservers. (We use Bluehost for our email)
Reporting-MTA: dns; gateway31.websitewelcome.com
X-Postfix-Queue-ID: 9B5CB2C9D77
X-Postfix-Sender: rfc822; email@example.com
Arrival-Date: Mon,  5 Apr 2021 12:09:51 -0500 (CDT)

Final-Recipient: rfc822; email@example.com
Original-Recipient: rfc822;email@example.com
Action: failed
Status: 5.0.0
Remote-MTA: dns; xxxxxx.ess.barracudanetworks.com
Diagnostic-Code: smtp; 550 permanent failure for one or more recipients
    (email@example.com:550 Sender IP reverse lookup rejected)

Thanks
Answer: 550 Sender IP reverse lookup rejected

You need to have your ISP create a PTR record for your mail server, for the IP you are sending mail from.
Answer: The Reporting-MTA has tried to send a message to the Remote-MTA which has rejected the message with an error described in Diagnostic-Code.
The error 550 Sender IP reverse lookup rejected is most likely related to SMTP Reverse DNS Resolution which is further related to a Reverse DNS Mismatch test.

Barracuda Bouncing Emails


Trying to get autocomplete list to appear on top of chart

Question: I initially built this page using an autocomplete control. If you enter a 'G' into the text field, the list of tickers will appear on top of the chart.
http://www.pikefin.com/phptest/charts/allassets/linechart.php?e=1&group=1&title=US%20Equities%20Indivdual%20Line%20Charts
Then I switched the chart type and now the list of tickers is underneath the chart:
http://www.testpikefin.com/phpdev/charts/allassets/linechart.php?e=1&group=1&title=US%20Equities%20Indivdual%20Line%20Charts
Answer: This is because your Chart Flash object. Flash object appears above all other objects on the page.
Here are some solutions for your problem:
http://kb2.adobe.com/cps/155/tn_15523.html
http://kb2.adobe.com/cps/142/tn_14201.html
Hope this helps ^^
Answer: Change the wmode to "opaque"
I tried this in the chrome console and worked
<code>$("embed").attr("wmode", "opaque").wrap("span");
</code>
I had to wrap in a span to force the repaint


Proving that $\sin(n)$ is not uniformly distributed modulo $1$

Question: How do you prove that $\sin(n)$ with $n=0,1,2...$ is not uniformly distributed mod 1?
(This is an exercise in Uniform Distribution of Sequences by Kuipers and Niederreiter.)
Answer: You know that the sequence of powers of $e^{i}$ are equidistributed in the unit circle of $\mathbb C$. The sequence you have is obtained from this one by projecting to the imaginary axis. Can you see what happens?
Comment: I'm sorry, I am new to these things now, and I don't understand a word of your answer (at least from my knowledge of first two sections of first chapter of [Uniform Distribution of Sequences by Kuipers and Niederreiter](https://web.maths.unsw.edu.au/~josefdick/preprints/KuipersNied_book.pdf) after which this exercise appears). I guess, the link you provided to equidistribution theorem simply says that $(n\theta)$ is ud mod 1 for irrational $\theta$. But, I have no idea of how we can "project to the imaginary axis". Can you please make your answer a little more elaborate and better accessible?
Comment: Beautiful idea.


How to pass struct as argument

Question: I am writing a piece of C code, and I am supposed to use a static library, containing one function. I have a header file, containing a typedef struct, which I am supposed to use as input argument.
My understanding of the function is, that I create a structure of the data type defined in header file and give it as pointer to the function and the function will fill the structure. The other two arguments are data and strlen(data).
My question is. How do I create the structure in my .c file and how do I get the data from it?
Here is a my .h file
#include <stdint.h>
#include <stddef.h>
typedef struct
{
  uint8_t arrayA[2];
  uint8_t arrayB[16];
} structDefinition;

void getData(structDefinition* myStructure, void const* data, size_t dataSize);

And this is my .c file
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include "myHeader.h"

structDefinition myStructure;

int main()
{
    char data[5] = {'123AF'};
    size_t dataSize = 5;

    getData(*myStructure, *data, dataSize);
    printf(mySignature.arrayA);
    printf(mySignature.arrayB);
    return 0;
}

Comment: How is the function defined?
Comment: I would suggest reading the chapter about pointers in your beginner-level C programming book. And the chapter about arrays. And the chapter about strings.
Answer: You need to pass the address of the structure for your function.
myStructure is a object of type struct Definition
Now pass
getData(&myStructure,data,dataSize);

data is an array of type char and it can be passed to a char *
Please check character array initialization
char data[6] = "123AF";/* char array comes with a nul terminator */

and length of the char array can be got from
size_t dataSize = strlen(data);

If you want to know the strlen() then 3rd parameter is not required.
Because you can get it from the string passed.
Comment: in C you can do `char data[5] = "123AF";` which gets no terminator, and use `sizeof data` instead of strlen
Comment: You don't actually need to pass the address of the structure, but in general it is advised to do so. BTW. it's good style to use definitions or, what I prefer, enums for constant sized data objects: `enum datasizes { DATASIZE=5, ASIZE=2, BSIZE=16 };`
Comment: @M.M What OP is trying to do is wrong. Hardcoding string length as well as initializing and also in C a string should be nul terminated for it be a valid string. I hope this is what OP wants
Comment: a constant sized data array is not necessarily a string, it's just a blob of memory with a constant amount of bytes in it, even if it is initialized as a string.
Comment: @ikrabbe If in the called API .If OP uses `data` as a normal string won't it lead to UB. Shouldn't the array be nul terminated?
Comment: @Gopi actually we don't know how op *uses* `data`, but only how he defines his array. BTW. the function name `getData` suggests that the data is only read from `myStructure` but we all can read from the context that it should better be called `setData`, as the data in `myStructure` seems to get modified by that function.
Answer: This definition
char data[5] = {'123AF'};

is wrong. '123AF' is a character literal.. Its value is implementation defined. It seems you mean
char data[5] = { "123AF" };

where "123AF" is a string literal.
The function call
getData(*myStructure, *data, dataSize);

is also wrong. It must look like
getData( &myStructure, data, dataSize );

These output statements can be also wrong because it seems the arrays can contain characters without terminating zeroes
printf(mySignature.arrayA);
printf(mySignature.arrayB);

To output correctly these arrays you need to look through the definition of the function getData to determine how the arrays are assigned.
Comment: If in the called API .If OP uses `data` as a normal string won't it lead to UB. Shouldn't the array be nul terminated?
Comment: @Gopi In this case what is the purpose of the third parameter?:)
Comment: Yeah agree !! Just wanted to clarify .. But still we are not clear what OP is actually trying to do.. Only concern is in the called API the passed `const char *` shouldn't be used as a string
Comment: @Gopi Yes it is not clear how the function is defined.:)
Comment: `char data[5] = { "123AF" };` is also wrong. Should be `char data[5+1] = "123AF"`.
Comment: @Lundin You are wrong . It is a valid C initialization of a character array.
Comment: It is (unfortunately) valid C but most likely not the intent. If you wanted a raw character array instead of a string, you wouldn't use the `" "` initializer syntax.
Comment: @Lundin It is unknown how the function is defined. So it is difficult to say how the array should be initialized. Taking into account the third parameter of the function I assume that the second parameter is not necessary a string.
Comment: Given that the OP is a complete beginner, I think it is rather safe to assume that they want a string. At any rate, it can't be wrong to declare room for the null terminator, one extra byte of used RAM won't matter.

How to pass struct as argument


Simplify httpd.conf file

Question: Any suggestions on how to simplify my httpd.conf file?  I've stripped out everything but the relevant portions:
Listen <ip-addr-1>:80
Listen <ip-addr-1>:443

<VirtualHost <ip-addr-1>:80>
ServerName myservername.com

<Proxy balancer://mycluster>
BalancerMember http://172.22.22.20
BalancerMember http://172.22.22.21
</Proxy>

ProxyPass /app1 balancer://mycluster/app1
ProxyPassReverse /app1 balancer://mycluster/app1
</VirtualHost>

# Repeat for 443
<VirtualHost <ip-addr-1>:443>
ServerName myservername.com

SSLEngine On
SSLCertificateFile      conf/ssl/mycert.crt
SSLCertificateKeyFile   conf/ssl/mycert.key
SSLCACertificateFile    conf/ssl/intermediate.crt

<Proxy balancer://mycluster>
BalancerMember http://172.22.22.20
BalancerMember http://172.22.22.21
</Proxy>

ProxyPass /app1 balancer://mycluster/app1
ProxyPassReverse /app1 balancer://mycluster/app1
</VirtualHost>

Is there anyway to not repeat everything for port 80 and 443, but turn on ssl for 443?
Answer: You could use an include if you're concerned about the editability - just move the repeated part to a separate file, say balancer.cfg and then Include balancer.cfg.
If your aim is to have a single Virtual Host definition I reckon that's not possible in this case. If you had multiple ports speaking HTTP you could just use VirtualHost 10.1.2.3:* and it would match every port, or use a list like VirtualHost  10.1.2.3:80 10.1.2.3:81 and so on. But in this case I don't see a way to serve both HTTP and HTTPS using the same definition - only one can match, it would seem, so the one being matched should contain the complete configuration.

Apache selects the best match only on
  the basis of the IP address (or
  wildcard) and port number. If there
  are multiple identical best matches,
  the first VirtualHost appearing in the
  configuration file will be selected.

http://httpd.apache.org/docs/current/vhosts/details.html
Comment: Thank you.  Includes should work pretty well for what I need.
Answer: I think I can do a touch better:
Listen 80
Listen 443

ServerName myservername.com
ProxyPass /app1 balancer://mycluster/app1
ProxyPassReverse /app1 balancer://mycluster/app1

SSLCertificateFile      conf/ssl/mycert.crt
SSLCertificateKeyFile   conf/ssl/mycert.key
SSLCACertificateFile    conf/ssl/intermediate.crt

<Proxy balancer://mycluster>
    BalancerMember http://172.22.22.20
    BalancerMember http://172.22.22.21
</Proxy>

<VirtualHost *:80>
</VirtualHost>

# Repeat for 443
<VirtualHost *:443>
    SSLEngine On
</VirtualHost>

I confess that I don't know whether the VirtualHost directive for *:80 can be entirely omitted, but based on the Apache docs, this much should work.

Simplify httpd.conf file


Quero exemplo do por quê substituir ' por '' e \ por \\ para parâmetros de um query é perigoso

Question: Eu vejo muitos posts sobre este assunto, porém não vejo nenhum exemplo que tem cabimento para este meu caso.
function sanitize($value)
{
    $val = str_replace("'", "''", $value);
    $val = str_replace("\\", "\\\\", $val);

    return $val;
}

A query ficaria mysqli_query($conn,"insert into tabela (Nome) values ('".sanitize($ipt)."')")
Considerando que até mesmo int entrará neste procedimento acima e considerando que todas as querys serão escritas EXATAMENTE desta maneira, com parâmetros demarcados com '(aspas simples), qual input seria capaz de quebrar a segurança e causar um injection?
Comment: Se existe erro ou não eu não posso afirmar, mas chutaria que talvez exista. A melhor alternativa seria usar o `mysqli_real_escape_string($ipt)`. Você pode usar `prepared statements` (isso é, usar o `mysqli_prepare ()` e depois o `mysqli_bind_param()`) essa solução pode ser mais difícil de ser aplicada, exigindo mais alterações. Acredito que isso será melhor do que tentar "reinventar" a roda.
Answer: No seu exemplo a inserção realmente está escapando os caracteres a fim de evitar injection. Ok.
O grande problema está mesmo na forma que você irá fazer:

Considerando que até mesmo int entrará neste procedimento acima e
  considerando que todas as querys serão escritas EXATAMENTE desta
  maneira [...]

De que forma você irá garantir isso com extrema certeza? Se você esquecer o sanitize em uma única query, como você irá descobrir isso? Basta um único lugar e seu site estará vulnerável.
Se algo tem chance de dar errado, isso VAI dar errado... Então pra que dar chance ao azar?
Se a linguagem oferece formas de evitar esse tipo de problemas de forma nativa, qual o problema de usar essa solução?
Temos os prepared statments no mysqli, no PDO, onde o sanitize da query é feita de forma assertiva e transparente que chega a ser incoerente não utilizá-las caso conheça da sua existência.
A manutenção do código também fica mais simples. Veja um exemplo:
$connection = new PDO('mysql:dbname=test;host=127.0.0.1', 'db', 'pass');

$stment = $connection->prepare('INSERT INTO user VALUES (:id, :name, :idade);');
$stment->bindParam(':id', $id, PDO::PARAM_INT);
$stment->bindParam(':name', $name, PDO::PARAM_STR, 25);
$stment->bindParam(':idade', $idade, PDO::PARAM_INT);

$id = 1;
$name = 'Joao';
$idade = 18;

$stment->execute();

O código acima não está mais claro que o de baixo?
$connection = new PDO('mysql:dbname=test;host=127.0.0.1', 'db', 'pass');

$id = 1;
$name = 'Joao';
$idade = 18;

$connection->query("INSERT INTO user VALUES ('" . sanitize($id) . "', '" . sanitize($name) "','" . sanitize($idade) . "');");


Is it possible to combine Regex with several characters being replaced?

Question: So I'm currently learning jQuery and I learned about regex recently and decided to put it into play.  What I've done works perfectly fine, but I have a feeling I'm going the long way about things.
I've looked online and I'm certain you can combine regex, but it looks really confusing and I really don't understand it!
Here's what I have:
var trackingPlus = contentURL.text().replace(/ /g, '+')
                                    .replace(/&/g, 'And')
                                    .replace(/%/g, 'PC')
                                    .replace(/!/g, '')
                                    .replace(/£/g, '')
                                    .replace(/,/g, '')
                                    .replace(/\./g, '')
                                    .replace(/\+>/g, '');

Is there any way to combine this into one "thing"?
Or if not, maybe the last 5 as they're all being replaced with nothing?
Any help, or explanation would be really appreciated!
Thanks!!
Answer: You can use a character class here. Simply put, a character class is a way of denoting a set of characters in such a way that one character of the set is matched.
A character class matches only a single character, so [!£,.+] would match one character from the list !, £, ,, .,+. It's important to note that the usual regex meta-characters (*, + etc.) are normal characters inside a character class and do not require additional escaping.
var trackingPlus = contentURL.text().replace(/ /g, '+')
                                    .replace(/&/g, 'And')
                                    .replace(/%/g, 'PC')
                                    .replace('/[!£,.+]/g', '');

Comment: You would'nt even need to use regular expressions for the first three replacements. :)
Comment: @YMMD. But... but... it looks so nice when stacked together! :D
Comment: That's what I thought too haha!  Thanks for your help! :)


What problem does oversampling, undersampling, and SMOTE solve?

Question: In a recent, well recieved, question, Tim asks when is unbalanced data really a problem in Machine Learning?  The premise of the question is that there is a lot of machine learning literature discussing class balance and the problem of imbalanced classes.  The idea is that datasets with an imbalance between the positive and negative class cause problems for some machine learning classification (I'm including probabilistic models here) algorithms, and methods should be sought to "balance" the dataset, restoring the perfect 50/50 split between positive and negative classes.
The general sense of the upvoted answers is that "it's not, at least if you are thoughtful in your modeling".  M. Henry L., in an up-voted comment to an accepted answer, states

[...] there isn't a low level problem with using unbalanced data. In my experience, the advice to "avoid unbalanced data" is either algorithm-specific, or inherited wisdom. I agree with AdamO that in general, unbalanced data poses no conceptual problem to a well-specified model.

AdamO argues that the "problem" with class balance is really one of class rarity

Therefore, at least in regression (but I suspect in all circumstances), the only problem with imbalanced data is that you effectively have small sample size. If any method is suitable for the number of people in the rarer class, there should be no issue if their proportion membership is imbalanced.

If this is the true issue at hand, it leaves an open question: what is the purpose of all the resampling methods intended to balance the dataset: oversampling, undersampling, SMOTE, etc?  Clearly they don't address the problem of implicitly having a small sample size, you can't create information out of nothing!
Comment: @MichaelChernick I'm not talking about the bootstrap, that's what Glen was commenting about.  I'm speaking of "class balancing" approaches like over and under sampling so that the positive to negative class ase equally represented in a data set.
Comment: That's exactly what I'd have changed it to... thanks. It doesn't cover the entire scope of your question but a title doesn't have to-- it does clearly get at what kind of thing you're asking about.
Comment: There are certainly situations where bootstrap and subsampling methods that are useful and sometimes better than other nonparametric methods.  Books on the bootstrap and subsampling cover this. There are discussions on this site that discuss this including superiority of the bootstrap over leave-one-out in discriminant analysis even in relatively small samples.  There are certainly some situations where the bootstrap fails and those are mentioned in my book as well as other.
Comment: Do you include subsampling?  Are you referring to unequal sample size only? How general a statement are you making?
Comment: @MichaelChernick I added some clarifying remarks in the first and last paragraphs, I hope that helps.
Answer: The problem that these methods are trying to solve is to increase the impact of minority class on cost function. This is because algos trying to fit well the whole dataset and then adapt to majority. Other approach would be to use class weights, and this aporoach in most cases gives better results, since there is no information loss by undersampling or performance loss and introduction of noise by oversampling.
Comment: i don't understand your answer.
Comment: I meant that performance of classifier is evaluated on the whole dataset (average error on both positive and negative examples), where error for each example is equally weighted. Thus algorithm (e.g. Logistic regression) adapts its hypothesis function to examples that will maximize error reduction. In this case to majority class , where minority (negative class) is practically disregarded because it doesn't have high influence on error on the whole dataset. This is why oversampling, under sampling or class weighting allow better adoption of algorithm to minority class.
Answer: Some sampling techniques are to adjust for bias (if the population rate is known and different), but I agree with the notion that the unbalanced class is not the problem itself. One major reason comes down to processing performance. If our targeted class, for example, is an extreme rare case at 1:100000, our modeling dataset would be massive and computation would be difficult. Sampling, no matter what the strategy, is always throwing away some data in order to reduce the total dataset size. I suppose the difference among all the different sampling strategies, is just cleverness around which data do we throw away without sacrificing a loss in predictive possibilities.
Comment: This is all true, but many of the methods I'm discussing create *larger* datasets.  Upsampling creates more instances of the positive class by copying existing datapoints, and SMOTE creates "synthetic" data points of the rare class.  These certainly wouldn't help with computing efficiency.
Comment: [It seems like these don't *adjust for bias*, they actively *introduce bias*, and they *induce* the loss in predictive possibilities you are concerned about.](https://stats.stackexchange.com/q/357466/1352)
Answer: I will give you a more extreme example. Consider the case where you have a dataset with 99 data points labeled as positive and only one labeled as negative. During training, your model will realize that if it classifies everything as positive, it will end up getting away with it. One way of fixing this is to oversample the underrepresented class and another is to undersample the overrepresented class. For example, in a dataset of 70 positive and 30 negative labels, I might sample the negative labels with replacement and positive ones without replacement which will result in my model encountering more negative labels during training. This way, if my model tries to classify everything as positive, it will incurr greater loss than it would have otherwise.
One more approach that does not pertain to sampling is to adjust the cost function to give higher weights to the data points with the minority label. For example, if you are using NLL loss in a dataset where 1's are overrepresented compared to 0's among labels, you could adjust your loss function to be:
$L(\tilde{x_i}, y_i) = -\alpha(y_i)\ln(\tilde{x_i}) - \beta(1 - y_i) \ln(1 - \tilde{x_i})$
where $\beta > \alpha$. The magnitude of the diference  $\beta - \alpha$ depends on the extent of overrepresentation/underrepresentation.
Answer: I'm going to disagree with the premise that unbalanced data isn't a problem in machine learning.  Perhaps less so in regression, but it certainly is in classification.
Imbalanced Data is relevant in Machine Learning applications because of decreased performance of algorithms (the research I am thinking of is specifically on classifiers) in the setting of class imbalance.
Take a simple binary classification problem with 25:1 ratio of training examples of class A' vs. 'class B'.  Research has shown that accuracy pertaining to the classification of class B takes a hit simply because of the decreased ratio of training data.  Makes sense, as the less # of training examples you have, the poorer your classifier will train on that data.  As one of the commenters stated, you can't make something out of nothing.
From the papers I've seen, in multiclass classification problems, it seems you need to get to a 10:1 ratio to start having a significant impact on accuracy of the minority class.  Perhaps folks who read different literature than I've seen have different opinions.
So, the proposed solutions are: Oversampling the minority class, Undersampling the majority class, or using SMOTE on the minority class.  Yes, you can't really create data out of nowhere (SMOTE sort-of does, but not exactly) unless you're getting into synthetic data creation for the minority class (no simple method).  Other techniques like MixUp and the like potentially fall into this concept, but I think that they are more regularizers than class imbalance solutions.  In the papers I have read, Oversampling > SMOTE > Undersampling.
Regardless of your technique, you are altering the relationship between majority and minority classes which may affect incidence.  In other words, if you are creating a classifier to detect super-rare brain disease X which has an incidence of 1 in 100,000 and your classifier is at 1:1, you might be more sensitive and less specific with a larger number of false positives.  If it is important that you detect those cases and arbiter later, you're ok.  If not, you wasted a lot of other people's time and money.  This problem eventually will need to be dealt with.
So to answer the question:
tl/dr: Class-balancing operations like Over/Undersampling and SMOTE (and synthetic data) exist to improve machine learning algorithm (classifier) performance by resolving the inherent performance hit in an algorithm caused by the imbalance itself.
Comment: @Michal I suspect the issue is "Research has shown that accuracy pertaining to the classification of class B takes a hit simply because of the decreased ratio of training data."  That is not a problem, that is exactly how it should be.  The Bayes optimal decision boundary depends on the prior probability of the classes.
Comment: Why did (or might have) this answer received a downvote? What are the specific problems in this answer?
Answer: There are many technqiues for oversampling and undersampling to overcome the sparsity of minority in imbalanced data anv vice versa.... Yet most of them have consequence on behavior of your model (roughly speaking variance). I personally use an self-devised technique where oversampling and undersampling are done simoultanously. Spicing this Combined Sampling with Adapative Synthetic Sampling (ADASYN), I call it C-ADASYN. You may think of hair-transplant where there you oversample in sparse area and unersample in dense area, to keep the behaviour fair and add synthetic sample if needed to augment the population.


Applying a transformation to NV12 like for I420

Question: I have an injective function that moves around some pixels in an image:
pixel (x, y) ===func===> pixel (X, Y)
X = funcX(x, y)
Y = funcY(y, x)

I want to use this function to transform the whole image in RGB, I420 and NV12 mode.
* RGB *: If the image is in RGB mode, it's pretty obvious:
strideR = strideG = strideB = width;

//Temporary table for the destination
for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        toR[i][j] = j * strideR + i;
        toG[i][j] = j * strideG + i;
        toB[i][j] = j * strideB + i;
    }

//Temporary table for the source
for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        fromR[i][j] = funcY(i, j) * strideR + funcX(i, j);
        fromG[i][j] = funcY(i, j) * strideG + funcX(i, j);
        fromB[i][j] = funcY(i, j) * strideB + funcX(i, j);
    }

for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        destR[ toR[i][j] ] = srcR[ fromR[i][j] ];
        destG[ toG[i][j] ] = srcG[ fromG[i][j] ];
        destb[ toB[i][j] ] = srcB[ fromB[i][j] ];
    }

* I420 *: If the image is in I420 mode (YYYYYYYY UU VV), the following is working:
strideY = width;
strideU = strideV = width / 2;

//Temporary table for the destination
for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        toY[i][j] = j * strideY + i;
        toU[i][j] = j / 2 * strideU + i / 2;
        toV[i][j] = j / 2 * strideV + i / 2;
    }

//Temporary table for the source
for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        fromY[i][j] = funcY(i, j) * strideY + funcX(i, j);
        fromU[i][j] = funcY(i, j) / 2 * strideU + funcX(i, j) / 2;
        fromV[i][j] = funcY(i, j) / 2 * strideV + funcX(i, j) / 2;
    }

    for (j = 0; j < height; j++)
        for (i = 0; i < width; i++) {
            destY[ toY[i][j] ] = srcY[ fromY[i][j] ];
            if ((i % 2 == 0) && (j % 2 == 0)) {
                destU[ toU[i][j] ] = srcU[ fromU[i][j] ];
                destV[ toV[i][j] ] = srcV[ fromV[i][j] ];
            }
        }

* NV12 *: If the image is in NV12 mode (YYYYYYYY UVUV), the following is NOT working:
strideY = strideUV = width;

//Temporary table for the destination
for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        toY[i][j] = j * strideY + i;
        toUV[i][j] = j / 2 * strideUV + i;
    }

//Temporary table for the source
for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        fromY[i][j] = funcY(i, j) * strideY + funcX(i, j);
        fromUV[i][j] = funcY(i, j) / 2 * strideUV + funcX(i, j);
    }

for (j = 0; j < height; j++)
    for (i = 0; i < width; i++) {
        destY[ toY[i][j] ] = srcY[ fromY[i][j] ];
        if ((i % 2 == 0) && (j % 2 == 0)) {
            destUV[ toUV[i][j] ] = srcUV[ fromUV[i][j] ];
            destUV[ toUV[i][j] + 1 ] = srcUV[ fromUV[i][j] + 1 ];
        }
    }

I got the image but with wrong colors. The black and white portion (aka the Y portion) is correct but the color portion (aka the UV portion) is altered. What am I doing wrong?
Answer: Found the problem! Solution is:
fromUV[i][j] = funcY(i, j) / 2 * strideUV + ((int)(funcX(i, j) / 2)) * 2;

I needed to floor X/2 to get the start of the UV byte.
Comment: Hi, we have an issue to convert I420 to RGB, can you help?


jQuery height/outerheight broken when using img tags in Webkit?

Question: I think I found a bug in webkit: FF doesn't have a problem reporting the height of this image, but webkit will only show the height here at 0. If there a fix for this other than specifying the height in CSS? I have a few images with the same width, but different heights.
<code>$(document).ready(function() {

   var rowTitleWidth = $('.art').height();
   alert(rowTitleWidth);
})

<img class="art" src="images/channel_art/channel_hd.png" />
</code>
Answer: works fine for me. You might want to take a look into jquery images loaded plugin though, it may be an issue with ready() being fired before the image has downloaded.
Comment: Ah, I get it. You were kinda right. The fix is to use .load first, then run the rest as a function of .load.
Comment: Great! Glad you could figure it out. Dont forget to check as answered :)


Preciso condicionar para trazer no group by apenas os dados que não possuem uma condição

Question: tenho a seguinte query:
SELECT id_grupo, id_analista
FROM tb_grupo_analista
GROUP BY id_grupo, id_analista;

existe uma terceira coluna chamada tp_controle, onde preciso condicionar a query de cima para me trazer o agrupado somente onde nesse agrupado não existe o tp_controle = 2
Comment: não entendi o que quer dizer com "trazer no group by".. pode explicar melhor?
`group by` é usado para agrupamento, ou seja, quando você usa alguma função para juntar/somar/contar..
Answer: Quando você utiliza o comando SELECT, pode usar a condição WHERE para condicionar o que você quer trazer. Tenta assim:
select id_grupo,id_analista from tb_grupo_analista where tp_controle != 2 group by id_grupo,id_analista;

EDIT:
para agrupar com a condição, de acordo com o comentário aqui na resposta, você pode tentar esse caminho:
select id_grupo,id_analista from tb_grupo_analista where tp_controle = 2
union
select id_grupo,id_analista from tb_grupo_analista where tp_controle != 2 group by id_grupo,id_analista;

Comment: Mas eu preciso agrupar apenas os que não tenha esse tp_controle =  2 saca?
Comment: entendi. porem, quando numa seleção vc pede pra agrupar, vc não tem opção de agrupar de maneiras diferentes os elementos na coluna que vc escolheu. pensando aqui, acredito que dê pra fazer um SELECT primeiramente sem agrupamento, com tp_controle = 2, e dar um UNION com outro SELECT, com agrupamento, onde tp_controle != 2. vou atualizar a resposta com o comando pra vc testar


Variable already defined in .obj; What is going on here?

Question: head.h

#pragma once

namespace foo
{
    int bar;

    int funct1();
}

head.cpp
#include "head.h"

int foo::funct1()
{
    return bar;
}

main.cpp
#include <iostream>

#include "head.h"

int main()
{
    foo::bar = 1;
    std::cout << foo::funct1() << std::endl;
    return 0;
}

Error   LNK2005 "int foo::bar" (?bar@foo@@3HA) already defined in head.obj
I don't understand what is going on. I tried looking for the answer but everyone's questions are so specific to their code and don't even look close to the problem that I am having.
I am not including .cpp files into main. I am not redefining anything. I am literally just assigning 1 to the variable then returning it with a function in the same namespace. How is it being defined multiple times?
Comment: Possible duplicate of [Variable definition in header files](https://stackoverflow.com/questions/2216765/variable-definition-in-header-files)
Comment: *How is it being defined multiple times?* -- You must have not realized what `#pragma once` is supposed to do.  It doesn't mean "this is only declared once in my entire program".
Comment: Possible duplicate of [Variable definition in header files](https://stackoverflow.com/questions/2216765/variable-definition-in-header-files)
Answer: The header head.h is included in two compilation units head.cpp and main.cpp. So the variable bar is defined twice. You could declare the variable without its definition the following way
#pragma once

namespace foo
{
    extern int bar;

    int funct1();
}

and then define it in some cpp module.
Comment: @JwGaming Types and objects are two different things.
Comment: Thank you. I never actually used this before on a variable. I couldn't understand how it was a problem since it is not a problem with classes or structs.
Comment: @JwGaming *it is not a problem with classes or structs.* -- It certainly is a problem with classes and structs if you had done the same thing.  `struct foo {}; foo f;`  If you had that in a header, you would have gotten the same error.
Answer: This foo namespace-level bar declaration:
namespace foo
{
    int bar;
}

is actually a definition.
To make it a declaration, mark the bar as extern in head.h:
namespace foo
{
    extern int bar;
}

Then define it in head.cpp:
int foo::bar = 0;

Answer: head.h is included in both main.cpp and head.cpp.
So the variable is defined twice.
Possible Solution: make it inline. The "extern" solutions are also good, although older in approach.
namespace foo
{
    inline int bar;
}

Comment: This gave me an error: Requires stdC++17
Only allowed on functions.
Comment: @JwGaming If you are using an old standard & toolchain, please state that in the question. It's 2019 now!
Comment: @Lightness Races in Orbit I am using the most up-to-date version of visual studio 2019...
Comment: Then flick it into C++17 mode!
Answer:
How is it being defined multiple times?

It is defined once in head.cpp and once in main.cpp. That is a total of two times. This violates the one definition rule, which states that there may only be one definition for every variable.

int bar;

This is a definition of a variable. You've included it into two translation units.
A variable can be declared without definition in an extern declaration:
extern int bar;

Replace the definition with such declaration, and put the definition into exactly one translation unit.
Answer:
I am not redefining anything. I am literally just assigning 1 to the variable

You're redefining the variable!
head.cpp has one via #include "head.h", and main.cpp has one via #include "head.h".
You need to merely declare it (unusual but not too strange) in the header:
extern int bar;

…then define it in one translation unit. This is just like what you do with static class members (albeit with slightly different syntax).
Since C++17, you may do this by instead plopping the inline keyword on to your definition.
Alternatively, avoid mutable globals…
Comment: I had no idea there was a difference. I assumed it /was/ a declaration and that it became defined the moment it was given a value and not before. Only that the compiler needed a way to link all the uses together as it would not be used before the first time it was given a value.
Comment: @JwGaming It's an old wives' tale that the first assignment to an object is a "definition" or "initialisation". It's not; it's an assignment to an object that was already defined and constructed (and potentially default-constructed) previously. Most variable declarations are also definitions; we can use `extern` to suppress the definition part.
Answer: Do carefully note that foo is not a class, but a namespace. When you declare a free variable in the header file:
int bar;
And then #include this header file multiple times (into different CPP files, causing multiple translation unit compilations), you'd get a linker error. Everyone knows it.
And, you'd add extern attribute at the declaration, and define the variable elsewhere in one of the CPP/C file.
Putting a global variable into a namespace is nothing but giving the global variable a different name. Think if foo::bar as foo__NS__bar, and hence you must add extern in the header and define foo::bar at some locatoin.
Note that this is different from a non-static member variable of a class. A class variable doesn't need to be defined/declared this way since the class is a type. Each instantiated variable of class-type would have a separate copy of the variable.
Further to that, when you add a static variable in the class, you are giving that logically global variable another name. Thence, you must have that static-variable defined one of the CPP file.


Bayesian Inference Notation Confusion

Question: In Bayesian Inference the following notation is quite common:
$P(H|D) = \frac{P(D|H)P(H)}{P(D)}$
where $D$ is data and $H$ is hypothesis. Moreover $P(D)$ is represented as total probability.
$P(D) = \sum P(D|H)P(H)dH$
However in my opinion notation seems to be quite sloppy. It is confusing on two fronts:
1) In order for this to work H should be a partitioning of sample space of $D$ by definition of total probability. But if you look at this point of view, let's say $D\sim N(m,v)$ then its sample space is $-\infty$ to $\infty$. I don't see why a hypothesis should partition this sample space, and what is that partition exactly? Let's say that my hypothesis is $D \sim N(m',v)$. In reality there should be a selector function behind the scenes, but I cannot really put my finger on it.
2) I think that distribution of $D$ is independent of what I select as a hypothesis, because my selection doesn't have the power to change reality. Therefore $P(D|H) = P(D)$. And inference cannot be done.
My interpretation is that in reality the notation is trying to partition a sample space of hypotheses. But how that should be properly written in a rigorous notation I don't know. Or my interpretation is wrong.
Any help appreciated.
Answer: Starting with your comment in part 2: $P(D|H)$ is not to be read as "the probability of getting this data given that I select hypothesis H".  It is saying "probability of getting this given that hypothesis H is true".  Clearly therefore $P(D|H) + P(D|¬H) = P(D)$
In other words, the partition can always in principle be between H is true and H is not true.  By the law of the excluded middle, this must cover everything.
However, you are right to observe that if you are summing up different models that could explain the data, then there is no guarantee (or even reason to believe) that you have got "everything" covered (i.e. in practice it is usually impossible to know $P(D|¬H)$).  That is why typically Bayesian analysis doesn't try to talk about the absolute probability of the model being correct given the data but only the relative probabilities of competing models.
So although I can in theory I can state:
$P(H_1 | D) = \frac{P(D|H_1) P(H_1)}{P(D)}$
$P(H_2 | D) = \frac{P(D|H_2) P(H_2)}{P(D)}$
When I do my analsis there is an implicit conditioning variable that is $M =$ "One of my models is correct".  Perhaps being strict one should write:
$P(H_1 | D, M) = \frac{P(D|H_1) P(H_1|M)}{P(D|M)}$
$P(H_2 | D, M) = \frac{P(D|H_2) P(H_2|M)}{P(D|M)}$
Noting that $P(H) = P(H,M)$ since if H is true then M is true.
Now, however, notice that defined like this my priors must logically add up to 1.  Often people don't have priors that add up to one.  We could incorporate this by saying $P(H_1) = P(H_1|M)P(M)$.  Writing our priors like this would then give us:
$P(H_1 | D, M) = \frac{P(D|H_1) P(H_1)}{P(D|M)P(M)}$
$P(H_2 | D, M) = \frac{P(D|H_2) P(H_2)}{P(D|M)P(M)}$
Of course none of this makes the slightest difference, because we are only going to compare the models, so we calculate:
$\frac{P(H_1 | D, M)}{P(H_2 | D, M)} = \frac{P(D|H_1,M) P(H_1)}{P(D|M)P(M)} \cdot \frac{P(D|M)P(M)}{P(D|H_2,M) P(H_2)} = \frac{P(D|H_1) P(H_1)} {P(D|H_2) P(H_2)}$
And all that intricacy cancels out and we are back at the original expressions.
Is it just laziness not writing the extra conditional variables that exist?  Not really no.  To me this is just another facet of the conditionality principle.

Edit
To take your example, suppose that $H_1$ is that $D \sim N(m,v)$ and $H_2$ is that $D \sim N(m', v)$ then clearly these two do not partition the space fully.  Since we cannot evaluate $P(D|¬H_1 \cup H_2)$ we cannot work out the true $P(D)$ and so cannot ever get the absolute probability that either model is correct.
What we can do though is establish the ratio of those probabilities, since the unpartitioned part of the space cancels out (see above).  This could then tell us that say, that $P(H_1|D)$ is 10 times more likely than $P(H_2|D)$.
In order to derive this, we had to write down expressions involving $P(D)$, even though we knew we'd never actually be able to calculate them.
Comment: "Clearly therefore P(D|H)+P(D|¬H)=P(D)". I don't agree with this statement. If my hypothesis is D~N(m',v), then probability of data given my hypothesis is true is P(D|H) = N(m',v). In that case what is P(D|not H)? 1-N(m',v)? How is that going to add up to N(m,v) which is real P(D).
Comment: That is my point, it is impossible to know "not H".  But you can still write down an expression for it.  $P(D)$ cannot usually be calculated, unless you happen to have a situation where "H" and "not H" are known and perfectly partition the universe.  If your hypothesis is $D\sim N(m,v)$ then you clearly can't define the probability of the data NOT coming from the model - it could be anything!
Comment: Bayesian inference is all about chosing the most likely model, given the data.  Not about evaluating the absolute probability that the model is correct.  Remember "all models are wrong; some are useful".  In reality $P(H)=0$, but what we really mean is not "H is true" but "H is close enough".
Comment: @CagdasOzgenc I've added a bit based on your example that might make things a little clearer?
Answer: The easiest way to imagine this is to think of the sample space as the Cartesian product $\Omega = \mathcal{D} \times \mathcal{H}$, with measure $\mathbb{P}(D, H) := \mathbb{P}(H) \mathbb{P}(D|H)$. In this way, each $D$ or $H$ correspond to subsets of pairs in $\Omega$ - both $\mathcal{D}$ and $\mathcal{H}$ partition the sample space differently. In this simple scheme,

$\mathbb{P}(H)$ is the prior measure over the members of the partition $\mathcal{H}$,
$\mathbb{P}(D|H)$ is measure of $D$ relative to the subset $H$.

Note that a particular observation $D \in \mathcal{D}$ can be regarded as a subset $\{(D, H): H \in \mathcal{H} \}$ containing all the pairs consistent with $D$. This way of reasoning can easily be generalized to multiple observations. With this in mind, your answers are:
1) No, your hypotheses do not partition the set of observations.
2) Yes, the distribution of $D$ is independent of your model about the world represented by the sample space (, the $\sigma$-algebra) and the probability measure. Bayesian probability theory is just a model of how to systematically update your beliefs given your prior assumptions plus data.
Comment: Thank you Pedro. Could we make a concrete toy example? Let's say in reality D is distributed with N(m,v) and I have two hypothesis N(m,v) and N(m',v) with equal probability. What will be omega?
Comment: In this case you have two hypotheses: $H_1 = m_1$ and $H_2 = m_2$ (note that $v$ is irrelevant). Your data space is $\mathcal{D} = \mathbb{R}$. Hence, the sample space is $\Omega = \mathcal{D} \times \mathcal{H} = \{ (x,m) : x \in \mathbb{R}, m \in \{m_1, m_2\} \}$.
Comment: Thanks. That made great sense now. In this case I think it is ok to say that choice of an H is partitioning omega, isn't it? The odd situation is that the total probability has nothing to do with real D.
Comment: Yes, it is OK to say that a choice of an $H$ is partitioning the sample space. But no, it doesn't have anything to do with the real $D$, but that's ok, since you **don't know** the real $D$. What matters is that there should be at least one of the hypotheses that matches the real $D$.
Comment: One follow up question. If I have sequential observations, can I make a sequential update in the following manner: calculate P(H|D) and plug it back to the place of P(H) for next observation. Or do I have to make a bigger cartesian space as I observe more data and calculate P(DxD...|H)?
Comment: Both ways you mentioned are equivalent ;-)
Comment: I don't see that as easily as you do. Could you elaborate on that? Thank you.
Comment: let us [continue this discussion in chat](http://chat.stackexchange.com/rooms/7596/discussion-between-pedro-a-ortega-and-cagdas-ozgenc)

Bayesian Inference Notation Confusion


Dúvida funcionamento do código JS

Question: tudo bom?
Estou iniciando o aprendizado em JS e estou fazendo uma formula que não está dando certo. É bem simples, gostaria de clicar em um botão, esse botão não atualizar a página (fiz a função event.preventDefault()) e simplesmente aparecer um alert na tela. O código já possui algumas linhas devido ao calculo de IMC que criei, vou postar abaixo.
Já busquei a solução 500 vezes e não consigo enxergar o erro. Acredito que o JS não está conseguindo ler a função botao2 criada pois, ao inspecionar, a mesma aparece indefinida.
Vejam abaixo:
var pacientes = document.querySelectorAll(".paciente");

for (var i = 0; i <= pacientes.length; i++){

function totalImc (a, b){
    return a / (b * b);
}

var paciente = pacientes[i];

var peso = paciente.querySelector(".info-peso").textContent;
var altura = paciente.querySelector(".info-altura").textContent;
var valorImc = paciente.querySelector(".info-imc");

var pesoValido = true;
var alturaValida = true;

if (peso < 0 || peso > 300){
    pesoValido = false;
    valorImc.textContent = "Peso Inválido!";
    paciente.classList.add("erro");
}

if (altura < 0 || altura > 2.30){
    alturaValida = false;
    valorImc.textContent = "Peso Inválido!";
    paciente.classList.add("erro");
}

if (alturaValida && pesoValido){
    var imc = totalImc(peso, altura);
    valorImc.textContent = imc.toFixed(0) +"%";
}

}

var botao2 = document.querySelector("#adicionar-paciente");
botao2.addEventListener("click", function(event){
    event.preventDefault();
    alert("Fui clicado!");
})

Código HTML (somente o body):

Aparecida Nutrição

Meus pacientes

                        Nome
                        Peso(kg)
                        Altura(m)
                        Gordura Corporal(%)
                        IMC

                        Paulo
                        100
                        2.00
                        10
                        0

                        João
                        80
                        1.72
                        40
                        0

                        Erica
                        54
                        1.64
                        14
                        0

                        Douglas
                        85
                        1.73
                        24
                        0

                        Tatiana
                        46
                        1.55
                        19
                        0

Adicionar novo paciente

    <form id="form-adiciona">

            Nome:
            <input id="nome" name="nome" type="text" placeholder="digite o nome do seu paciente" class="campo">

            Peso:
            <input id="peso" name="peso" type="text" placeholder="digite o peso do seu paciente" class="campo campo-medio">

            Altura:
            <input id="altura" name="altura" type="text" placeholder="digite a altura do seu paciente" class="campo campo-medio">

            % de Gordura:
            <input id="gordura" name="gordura" type="text" placeholder="digite a porcentagem de gordura do seu paciente" class="campo campo-medio">

        <button id="adicionar-paciente" class="botao bto-principal">Adicionar</button>
    </form>

Alguém poderia me dar um help, por favor?
Obrigado,
Answer: basta deixar seu for dessa forma:
for (var i = 0; i < pacientes.length; i++)

O atributo length retorna o numero de itens dentro do seu array, porem a contagem dos índices começa no 0, de modo que a ultima execução do seu for não encontrava índice nenhum.
Se você der um console.log(paciente) antes de fazer essa alteração, vai perceber que o ultimo retorno é undefined, pois não mais índices no array pacientes para ele percorrer, dessa forma ele retornava um erro, de modo que sua variável botão2 nunca chegava a ser definida.
Comment: Obrigado Gabriel. Definitivamente um erro muito básico mas não estava enxergando isso.
Agradeço muito pela ajuda.

Dúvida funcionamento do código JS


What ordes does the IBActions fire in when multiple IBActions supplied for same event

Question: As per my header, I am building a view with some UIButtons in it.
I was thinking about hooking up 2 actions to one UIButton event, since that button needs to do two things, but I need to do them in a certain order.
No matter how I try, I can't seem to change the order in which the Actions fire. Is there a way for me to decide this, or is it random?
When I right-click the UIButton in Interface Builder, I see both Actions added to the same event, but no matter in which order they appear does the firing order change.
Hoping some of you out there can help me.
Answer: UIButtons are subclasses of UIControl, which use the target-action pattern to communicate events. The programatic interface for adding targets is exposed via the addTarget:action:forControlEvents: method. Hooking up IBActions in interface builder is just a visual way of using this same interface; the bundle loader will call that method on the unarchived button when loading the .xib file that contains it.
The key point is that the target argument in that method is added to an NSSet (or mutable subclass) internally by the UIControl. NSSets are, by definition, unordered. This means that when the button needs to enumerate its set of targets to dispatch events, the order of enumeration is not well defined.
Answer: You will not get answer to this question in any book or even googling.
I have tried that personally and it work in alphabetical order according to IBActions Names. Strange but its a fact!!
Tested Example Below :
helloworld.h
-(IBAction)a:(id)sender;
-(IBAction)b:(id)sender;
-(IBAction)c:(id)sender;
-(IBAction)d:(id)sender;
-(IBAction)e:(id)sender;
-(IBAction)f:(id)sender;

Action)g:(id)sender;
**HelloWorld.m file**

-(IBAction)a:(id)sender
{
    NSLog(@"I clicked a");
}
-(IBAction)f:(id)sender
{
    NSLog(@"I clicked f");
}
-(IBAction)b:(id)sender
{
    NSLog(@"I clicked b");
}
-(IBAction)c:(id)sender
{
    NSLog(@"I clicked c");
}
-(IBAction)d:(id)sender
{
    NSLog(@"I clicked d");
}
-(IBAction)e:(id)sender
{
    NSLog(@"I clicked e");
}

Now link these actions to your Button in [Touch Up Inside] event in any order. You will find the same output as shown below :
Output
2012-03-19 15:35:34.548  I clicked a
2012-03-19 15:35:34.554  I clicked b
2012-03-19 15:35:34.564  I clicked c
2012-03-19 15:35:34.566  I clicked d
2012-03-19 15:35:34.568  I clicked e
2012-03-19 15:35:34.572  I clicked f

Cheers!!
Happy Coding!!
Comment: I'm curious to know if this is the order you added them to the control
Comment: Seems reasonable. This has been causing an issue in my app for a while now.
Answer: Since there's no clear way to indicate the order in which the actions should be sent, you should not rely on the order. Instead, create a separate action that calls the two actions that you care about in the desired order:
- (IBAction)doFooAndBar:(id)sender
{
    [self foo:sender];
    [self bar:sender];
}


Probability that a given number falls between the minimum and the maximum of a sample

Question: Let $X$ be a real random variable with absolutely continuous cumulative distribution function $F$. Let $x_{(1)}, ..., x_{(n)}$ be a i.i.d. ordered sample of size $n$ of $X$:
$$
x_{(1)} \leq x_{(2)} \leq ... \leq x_{(n)}.
$$
I am interested by the interval $[x_{(1)}, x_{(n)}]$ defined by the sample minimum and maximum. Let $x \in \mathbb{R}$ be a given arbitrary, constant, number. I want to compute the probability that $x$ falls in the random interval between the minimum and the maximum:
$$
\mathbb{P}\left(X_{(1)} \leq x \leq X_{(n)} \right).
$$
From this post, I can see that:
$$
\mathbb{P}(X_{(1)} \leq x, X_{(n)} \leq y)
= [F(y)]^n - [F(y) - F(x)]^n
$$
for any $x < y$. But this does not finish the proof.
Comment: For $x\le y$ use $\mathbb P(X_{(1)}\ge x,X_{(n)}\le y) = \mathbb{P}(X_{(n)}\le y) - \mathbb P(X_{(1)}\le x,X_{(n)}\le y) + \mathbb P(X_{(1)}=x,X_{(n)}\le y).$
Comment: Then the probability depends critically on how $x$ is selected; off-hand, I'd say the probability is zero.  Consider how "big" the real line is compared to any closed interval and you will see why.
Comment: Is the new number $x \in \mathbb{R}$ following the same distribution as the elements of the sample of size $n$?
Comment: The number $x$ is an arbitrary real number, not a realization of $X$.
Answer: The interval $[x_{(1)},x_{(n)}]$ will not contain $x$ if either all $x_{(i)}$ are below $x$ or all of them are above $x$. Since those are mutually exclusive events the probability of the complement is just
$$ P(X_{(1)} \le x \le X_{(n)}) = 1 - (F(x))^n - (1-F(x))^n$$
Comment: That sounds very likely
Comment: It is not the case that the probability all are less than $x$ equals $F(x)^n$!  You neglect the possibility of non-continuous distributions.
Comment: You are correct but the question specifically states that the distribution is absolutely continuous
Comment: Yes--but it did not originally.  That is what made it a new question.  I'm confident this question (and answer) has appeared several times here on CV.
Answer: The probability is
$$
\mathbb{P} \left(X_{(1)} \leq x \leq X_{(n)}\right)
= 1 - [1 - F(x)]^n - [F(x)]^n.  \qquad (1)
$$
More generally, given any $x$ and $y$ such that $x \leq y$, we have
$$
\mathbb{P} \left(X_{(1)} \leq x \; \textrm{ and } \; X_{(n)} > y\right)
= 1 - [1 - F(x)]^n - [F(y)]^n + [F(y) - F(x)]^n.
$$
In order to get the previous result, we just have to plug $x = y$ into the equation (1).
To see the more general result, we use the fact that :
$$
\begin{aligned}
& \mathbb{P} \left(X_{(1)} \leq x \; \textrm{ and } \;  X_{(n)} > y\right) \\
& = \mathbb{P} \left(X_{(1)} \leq x \right)
- \mathbb{P} \left(X_{(1)} \leq x \; \textrm{ and } \;  X_{(n)} \leq y\right). \qquad (2)
\end{aligned}
$$
Moreover, the cumulative distribution function of the smallest observation is:
$$
\mathbb{P} \left(X_{(1)} \leq x \right)
= 1 - [1 - F(x)]^n.
$$
Finally, we substitute the equation
$$
\mathbb{P} \left(X_{(1)} \leq x \; \textrm{ and } \; X_{(n)} \leq y\right)
= [F(y)]^n - [F(y) - F(x)]^n
$$
in the equation (2) and get the result.
Comment: This answer assumes $F$ has a continuous distribution, whereas the formula you originally quoted in the question is for *arbitrary* distributions.
Comment: This is true : I has this hypothesis in mind, but did not write it out initially and this is why I clarified the hypotheses afterward. The answer by @h-delaney is simpler and right to the point and this is why it is the best answer. But my proof is based on a more general result, which is interesting on its on right.


How to group by with sum transformation

Question: I want to transform dataframe so all ltv in identical (date, country_code, media_source ) would sum up
Here is dataframe that I have:
date        country_code    media_source    ltv      cost
2020-07-01  AE               Facebook Ads   22.5    59.30
2020-07-01  AE               Facebook Ads   0.00000 59.30
2020-07-01  AE               Facebook Ads   11.2    59.30
2020-07-01  AT               Facebook Ads   11.2    7.53
2020-07-01  AU               Facebook Ads   11.2    73.72
2020-07-01  AU               Facebook Ads   0.00000 73.72
2020-07-01  AU               Facebook Ads   56.3    73.72
2020-07-01  AU               Facebook Ads   11.2    73.72
2020-07-01  AU               Facebook Ads   0.00000 73.72
2020-07-01  AU               Facebook Ads   22.5    73.72
2020-07-01  BE               Facebook Ads   33.7    13.26
2020-07-01  BM               Facebook Ads   56.3    0.92
2020-07-01  BN               Facebook Ads   11.2    1.22
2020-07-01  BR               Facebook Ads   11.2    249.99
2020-07-01  BR               Facebook Ads   0.00000 249.99
2020-07-01  BR               Facebook Ads   11.2    249.99
2020-07-01  BR               Facebook Ads   11.2    249.99
2020-07-01  BR               Facebook Ads   0.00000 249.99
2020-07-01  BR               Facebook Ads   11.2    249.99

and here is expected dataframe:
date       country_code media_source      ltv(sum)   cost
2020-07-01    AE        Facebook Ads        33.7    59.30
2020-07-01    AT        Facebook Ads        11.2    7.53
2020-07-01    AU        Facebook Ads        101.2   73.72
2020-07-01    BE        Facebook Ads        33.7    0.92
2020-07-01    BN        Facebook Ads        11.2    1.22
2020-07-01    BR        Facebook Ads        44.8    249.99

How could i transform it into this dataframe
Answer: You can try
out = (df.groupby(['date', 'country_code', 'media_source'], as_index=False)
       .agg(**{'ltv(sum)': ('ltv', 'sum'), 'cost': ('cost', 'first')}))

print(out)

         date country_code  media_source  ltv(sum)    cost
0  2020-07-01           AE  Facebook Ads      33.7   59.30
1  2020-07-01           AT  Facebook Ads      11.2    7.53
2  2020-07-01           AU  Facebook Ads     101.2   73.72
3  2020-07-01           BE  Facebook Ads      33.7   13.26
4  2020-07-01           BM  Facebook Ads      56.3    0.92
5  2020-07-01           BN  Facebook Ads      11.2    1.22
6  2020-07-01           BR  Facebook Ads      44.8  249.99

How to group by with sum transformation


When to calc std dev? Before or after?

Question: I have a sample which is analysed 5 times. I get an average of these 5 results.
I repeat this test in triplicate to give me three answers which I then find an average of.
My question: should I use the standard deviation of all 15, or just the standard deviation of the final 3 results?
Thanks
Comment: Use *for what purpose?*  What do you want this SD to represent or measure?
Comment: Insightful question.
Answer: You could find the standard deviation of
each of the three runs of five. Then "pool"
the three variances. There is some chance
that the three runs will differ slightly, so
this pooled result may differ somewhat
from the standard deviation of all 15 observations. I would use the pooled SD, rather than the standard deviation of all 15 observations.
[The idea of the pooled variance is similar to the
pooled variance in a pooled 2-sample t test (but
generalized to three groups). It is also the
same idea as computing the mean square for residuals in a one-factor ANOVA with three levels of the factor. See end note.]
Computations, using R mainly as a calculator:
set.seed(2022)
x1 = rnorm(5, 50, 7)
x2 = rnorm(5, 52, 7)
x3 = rnorm(5, 48, 7)

Stripchart of the three runs:
stripchart(list(x1,x2,x3), ylim=c(.5,3.5), pch=19)

s1 = sd(x1);  s2 = sd(x2);  s3 = sd(x3)
s1;  s2;  s3
[1] 6.503417
[1] 10.37375
[1] 4.973423

# pooling
v.p = (4*s1^2 + 4*s2^2 + 4*s3^2)/12;  v.p
[1] 58.21469
sd.p = sqrt(v.p);  sd.p
[1] 7.629855

# all 15
sd.all = sd(c(x1,x2,x3));  sd.all
[1] 7.143844

Note: Here is an ANOVA table for the three groups,
showing the mean square for residuals, which agrees with v.p (except for rounding).
x = c(x1, x2, x3)
g = as.factor(rep(1:3, each=5))
anova(lm(x~g))

Analysis of Variance Table

Response: x
          Df Sum Sq Mean Sq F value Pr(>F)
g          2  15.91   7.953  0.1366 0.8736
Residuals 12 698.58  58.215

The large P-value $0.8736 > 0.05 = 5\%$
indicates that that no significant differences among the three group sample means is detected.
I suppose one purpose of your experiment may be to estimate the population standard deviation $\sigma.$ You don't have much information in your 15 observations.
However, in case you do want to use the pooled variance
$V_p  = 58.21469$ to get a 95% CI for
the population standard deviation $\sigma,$
you can use the relationship
$\frac{12V_p}{\sigma^2} \sim
\mathsf{Chisq}(\nu = 3(5-1)=12),$ which has
$P\left(4.404 \le \frac{12V_p}{\sigma^2} \le 23.34\right) = 0.95,$ to get
the 95% CI $(5.47,\, 12.59)$ for $\sigma.$
qchisq(c(.025,.975), 12)
[1]  4.403789 23.336664
sqrt(12*v.p/qchisq(c(.975,.025), 12))
[1]  5.471262 12.594870

Comment: Please see my comment to the question.  In particular, where does the question ask about testing?
Comment: Nowhere, But the pooled variances used in testing seem a natural response.

When to calc std dev? Before or after?


SSH: Port Forwarding, Firewalls, & Plesk

Question: I edited my SSH configuration to accept connections on Port 213, as it was one of the few ports that my work firewall allows through.
I then restarted sshd and everything was going well.
I tested the ssh server locally, and checked the sshd service was listening on port 213; however, I still cannot get it to work outside of localhost.
PuTTY gives a connection refused message, and some of the sites that allow check of ports I tried said the port was closed.
To me, this is either firewall or port forwarding. But I've already added inbound and outbound exceptions for it.
Is this a problem with my server host, or is there something I've missed?
My full SSH config file, as requested:
#   $OpenBSD: sshd_config,v 1.73 2005/12/06 22:38:28 reyk Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options change a
# default value.

Port 22
Port 213
#Protocol 2,1
Protocol 2
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 768

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin yes
#StrictModes yes
#MaxAuthTries 6

#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile .ssh/authorized_keys

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication yes

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication mechanism.
# Depending on your PAM configuration, this may bypass the setting of
# PasswordAuthentication, PermitEmptyPasswords, and
# "PermitRootLogin without-password". If you just want the PAM account and
# session checks to run without PAM authentication, then enable this but set
# ChallengeResponseAuthentication=no
#UsePAM no
UsePAM yes

# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#ShowPatchLevel no
#UseDNS yes
#PidFile /var/run/sshd.pid
#MaxStartups 10
#PermitTunnel no
#ChrootDirectory none

# no default banner path
#Banner /some/path

# override default of no subsystems
Subsystem   sftp    /usr/libexec/openssh/sftp-server

Comment: How could we possibly know? It's _your_ server; go take a look.
Comment: (1) ***Why aren't you having your admin open standard SSH ports for you, or port-forward something from an external IP to your machine's SSH port?*** -- This question smells of "I don't like my work policy on network connectivity so I'm trying to get around it." -- that's [not kosher around these parts](http://serverfault.com/faq).  (2) Have you verified that sshd is listening on an IP that's accessible from outside your firewall?
Comment: Sounds to me like you bound ssh to listen to the local interface instead of all interfaces... or your company's network systems killed you dead.
Comment: I've asked. They won't.
Comment: @JeffFerland I can still connect via Port 22 though?
Comment: Why don't you post the configuration file? Then we don't have to guess what you wrote.
Answer: You could verify if it is listening on those ports on a public interface with a simple command.
netstat -tunelp | grep -i ssh

Obviously, you'd also need to make sure that the remote machine has the outbound port 213 open and that the server in question has inbound 213 open. Nmap from a remote machine would show this
nmap -v -sT myremotehost.com -p 213

This isn't going to be an exact science considering the remote host could be blocked outbound and that the server could be blocking inbound. So disabling the firewall and using process of elimination would be easy enough.
Save the existing iptables rules
iptables-save > /root/iptables.rules

Flush the existing rules
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

Then run your tests.
Then to restore the old rules ...
iptables-restore < /root/iptables.rules

Just note. I'm not condoning disabling a firewall on a production machine.
Comment: http://weblogs.amtex.nl/index.php/2007/12/30/openvz_server_crash_centos5_when_iptable?blog=2
Comment: Well it's definitely listening on a public interface. It's annoying, because my home computer is Windows, and I don't have any of the cables for my raspberry pi so I keep having to download Windows ports of tools.
Comment: OK, so I think it crashed when I disabled the firewall. I can't SSH, or FTP, HTTP


Instagram Gem and Faraday

Question: I'm having an issues with the instagram gem, I don't know where these issues came from, but everything was working fine until I performed a bundle update. Here's the issue:
If I use gem 'instagram' in my gemfile, I get the error:
undefined method `configure' for Instagram:Module

I then read online and found a solution to this issue is to add the github path like gem "instagram", :git => 'git://github.com/Instagram/instagram-ruby-gem.git' but then I get the error:
Bundler could not find compatible versions for gem "faraday":
  In Gemfile:
    instagram (>= 0) ruby depends on
      faraday (< 0.9, >= 0.7) ruby

    omniauth-facebook (>= 0) ruby depends on
      faraday (0.9.0)

I tried manually editing the gem to use faraday 0.9.0, but it causes additional problems. Any ideas how to fix this issue and get the instagram gem working again? (it was working before, no idea what messed it up!)
Comment: remove the **Gemfile.lock** and then issue `bundle install`, and show the errors if any.
Comment: tried that, the errors i got are listed above. solved the problem by explicitly stating that rails should use faraday ~> 0.8.9
Answer: Solved by putting gem "faraday", "~> 0.8.9" in my gem file, forcing it to use the old version
Comment: I'm trying to do this but Twitter gem requires faraday > 0.9. Can't win.
Comment: Having the same issue as @Dex
Comment: bundle update instead of bundle install did it for me! ;)

Instagram Gem and Faraday