Push out Commands to all Domain PCs

Question: I'm looking for a way to push out commands to all workstations. The scenario is as follows:
I often go to environments that I am not familiar with to audit the network. Part of that is a network scan, but to use our specific tools we need to configure a couple of things on every workstation (enable wmi access, enable file and printer sharing, etc.). We have a batch file we can run on every computer, but this solution does not scale well as you can imagine. I've included the commands we run below.
Ideally, there would be a way to push out the batch file to run one time on all computers connected to the domain. Alternatively, we could create a new batch file that creates GPO that does the same things, but this is something that I have not done before.
Any help is really appreciated!
rem Allow the device to be pingable through Windows Firewall
netsh firewall set icmpsetting type=ALL mode=enable
netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow
netsh advfirewall firewall add rule name="ICMP Allow incoming V6 echo request" protocol=icmpv6:8,any dir=in action=allow

rem Turn on File and Printer Sharing
netsh advfirewall firewall set rule group="File and Printer Sharing" new enable=Yes

rem Allow WMI access through Windows Firewall
netsh firewall set service type=remoteadmin mode=enable
netsh advfirewall firewall set rule group="windows management instrumentation (wmi)" new enable=yes

rem Add user account
net user [REDACTED] /add
net localgroup Administrators [REDACTED] /add

Rem Set WMI Permissions
sc sdset SCMANAGER D:(A;;CCLCRPRC;;;AU)(A;;CCLCRPWPRC;;;SY)(A;;KA;;;BA)S:(AU;FA;KA;;;WD)(AU;OIIOFA;GA;;;WD)

ECHO End of script
PAUSE

Answer: Firewall settings and local users and groups can be directly crontrolled by GPOs, you don't need to run a batch file in order to set them.
Relevant settings:
Comment: Hi and thank you for your reply. I am aware of setting GPO's to do this, but for what we are after this is not an adequate method. What we really want is to push out the commands included to all domain computers one command a single time. Even if someone could provide a script to create those GPOs that might be better.
Comment: Please note that if Windows Firewall is blocking communications, you won't be able to use *any* tool to reach domain computers; in this case, you *will* need a GPO to configure Windows Firewall and allow you remote access.
Comment: You will need to use a GPO to enable some traffic in Windows Firewall anyway; you might as well do all your Windows Firewall configuration from there.
Answer: Additional answer to address your comment.
GPOs don't provide any way to execute something immediately; the next best thing is a startup script, but that would only be executed at the next reboot of the machines.
If you really need to immediately execute a command on all domain computers, you can use PowerShell Remoting to run your command against all computer objects returned from an Active Directory query, like this:
$Computers = Get-ADComputer -Filter {Enabled -eq $True} -searchbase "OU=myou,DC=domain,DC=tld"

foreach ($Computer in $Computers) {

    Invoke-Command -ComputerName $_.Name -ScriptBlock {

        # Your script here
    }
}

Of course, you can add additional filters to your LDAP search and/or change the search base; it would be also wise to test if the computers are actually running and reachable, otherwise you will have to wait for Invoke-Command to timeout on each unreachable computer.
Comment: Hi and thank you for taking the time to reply. I'm not super familiar with powershell, so excuse any questions that seem elementary. Is there a way to enable PS remoting remotely? Would I know if the script failed on any computer? Does it run simultaneously on domain computers or one at a time?

Thank you in advance!
Comment: The script as shown is a simple foreach loop, it gets a list of computers from AD and then runs the command against each of them; please note that PowerShell runs *on your computer*, only the script block is executed on the remote computer; all of the output will be shown on your console.
Comment: This example is sequential; there are ways to do this in parallel, but they are a lot more complex.
Comment: PowerShell Remoting is based on WinRM, which is normally enabled by default on server systems but not on client ones; it can be enabled by Group Policy, there are lots of tutorials around.

Push out Commands to all Domain PCs


express js: is there a way to map all the rest of url as one parameter?

Question: The code segment is:
  app.get('/api/photo/:S3ObjectKey', photo.get);

And photo.get():
  const S3ObjectKey = req.params.S3ObjectKey;
  if (!S3ObjectKey) {
    console.log("No S3ObjectKey specified");
  }
  console.log("S3ObjectKey: ", S3ObjectKey);

  const canShow = true;
  if (canShow) {
    const bucket_name = process.env.AWS_S3_BUCKET_NAME;
    const data = await S3.getFile(bucket_name, S3ObjectKey);
    data.Body.pipe(res);
  }

Is there a way to map all the rest of url as one parameter, such that:
a GET request to https://mylovelydomain/api/photo/bucketMyPhoto/2020/07/12/home/table.jpeg would hit the api endpoint and S3ObjectKey has the value bucketMyPhoto/2020/07/12/home/table.jpeg?
Answer: Express uses path-to-regex:

Custom Matching Parameters
Parameters can have a custom regexp, which overrides the default match ([^/]+).

On your routing path you can use * to get everything overwriting that default match (note that this code also reatcs to empty requests):
    app.get("/api/photos/:S3ObjectKey(*)", /*...*/)

Answer: As per express documentation, route path supports regular expression, so should be able to do it. Quoting the example from the documentation.


Backing up logs

Question: I've been looking at my site logs recently. I have the limit of logs set to 10,000. The problem is, due to the amount of traffic we get, that number will only represent, at best, half a day of logs. Most of these are ether from access denied (I'm guessing spam bots trying to create nodes) or page not founds. The rest are items from custom modules I've built that I use to monitor how those modules are functioning (Like a notification that a queue is being run, or nodes are being created, etc.).
I've toyed with the idea of bumping the log messages to keep up to 100,000 or 1,000,000, but I'm not sure if this will cause performance issues or if it will drastically increase the size of my db.
Ultimately though, that would only really be a band-aid at best. As it would be nice to be able to keep a longer track of error messages, say weeks or months, so that I can further analyze what my site is doing, what areas need to be improved, etc. Is there any way to create a back up of log files so I don't loose them forever once they hit that ceiling?
Answer: I believe it's a best practice to not enable the core Logging module for Production Drupal websites (eg, outside of your development environment) and to setup server (host) based logging -- or offload the logging to a remote service.
See the Syslog documentation page. Which is what I mean by "host based logging".

The Syslog module logs events by sending messages to the logging
  facility of your web server's operating system. Syslog is an operating
  system administrative logging tool that provides valuable information
  for use in system management and security auditing. Most suited to
  medium and large sites, Syslog provides filtering tools that allow
  messages to be routed by type and severity.
It is not suitable for shared hosting environments.
It might not be as user friendly as Database Logging but will allow
  you to see logs and troubleshoot if your site is not accessible.
  Because the Database logging module writes logs to the database it can
  slow down the website. By using Syslog you can improve the performance
  of the site.

An example of a remote log is Redis Logging, there are numerous other 3rd party logging modules.
EDIT
A common practice for webservers is to Rotate Logfiles which builds an archive of compressed logfiles after your logfile has accrued to a certain size or after X days. By using Syslog you can configure your webserver to then rotate logfiles.
Comment: Definitely a good idea to keep logging out of the db if at all possible - a little while ago on a high-traffic site I maintain a remote filesystem went AWOL for a while which resulted in a ton of 404 errors being logged for every request. Sheer weight of traffic combined with the extra db load from the watchdog made the whole site basically unresponsive until the dblog module was disabled manually. These things don't happen often, but when they do it's doubly annoying if you could've prevented it!
Comment: Very good point, I'll look into getting this set up. Thanks for the help guys!
Comment: You can use a service like Splunk to offload your logs for later use.

Backing up logs


How to enable CONFIG_PREEMPT option in Linux kernel?

Question: I am a new bee in Linux kernel programming, trying to work with an old kernel Linux 2.6.32 on x86_64. I want to enable the CONFIG_PREEMPT option in it but can not find information about how can I do it. I can compile a new kernel with my preferred options, but do not know what I need to do in this case. So can anyone please tell me
How can I enable CONFIG_PREEMPT option? Do I need to recompile the kernel again with new menuconfig? In that case which option is responsible for CONFIG_PREEMPT?
I think it's trivial for anyone who worked with this before, so please give me your valuable help!
Answer: CONFIG_PREEMPT in a main part of the kernel, you can not modularize it. So if you don't have already a PREEMPT kernel, you need to compile it from sources. The option is selected by its name 'PREEMPT' or edit file manually to activate it.
While in menuconfig, press the / key, and search for PREEMPT.  This will show you where the option is and what it will be called in the menu.
Check if it is not available in your preferred distribution.


Abrir tela dentro do mainactivity

Question: Estou fazendo uma calculadora de mecânica, porem queria fazer varias opções de calculo no navigation drawer e quando clicar a calculadora especifica carregasse na tela (a intenção é fazer uma activity para cada tipo de calculo), estou usando o código para abrir a activity de um calculo quando se clica na primeira opção do navigation drawer
public boolean onNavigationItemSelected(MenuItem item) {
    // Handle navigation view item clicks here.
    int id = item.getItemId();

    if (id == R.id.calcRPM) {
        startActivity(new Intent(getBaseContext(),calculorpm.class));
    } else if (id == R.id.nav_gallery) {

Mas como podem ver nas imagens ele abre uma nova tela, sem o drawer e o menu lá em cima. como poderia fazer isso carregar como parte do app "principal"
Comment: Pesquise sobre fragments, voce vai ter apenas uma atividade e varios fragments que trocam o conteudo interno da atividade. Sua pergunta esta muito ampla pra oferecermos um codigo de resposta..
Comment: Vou da uma pesquisada, valeu
Comment: Eu não usaria fragmento porque limita muito a aplicação.
Você pode fazer uma BaseActivity, as outras atividades que precisem do navigation drawer devem estender ela.
No inicio da um pouco de trabalho, mas depois tudo fica mais fácil.
Da uma olhada nesse exemplo: http://mateoj.com/2015/06/21/adding-toolbar-and-navigation-drawer-all-activities-android/ Estava com a mesma dificuldade que você e foi ai que achei o caminho.
Answer: Para fazer isso, você não deve iniciar uma nova activity, quando clicar em um item no  navigation drawer. O que você deve fazer é trabalhar com fragmentos na MainActivity e substituir esse fragmento quando um item for selecionado ao invés de iniciar uma nova activity.
Na sua activity_main.xml

<LinearLayout
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">

    <include
        android:id="@+id/toolbar"
        layout="@layout/tool_bar" />

    <FrameLayout
        android:id="@+id/frame"
        android:layout_width="match_parent"
        android:layout_height="match_parent">

    </FrameLayout>

</LinearLayout>

<android.support.design.widget.NavigationView
    android:id="@+id/navigation_view"
    android:layout_width="wrap_content"
    android:layout_height="match_parent"
    android:layout_gravity="start"
    android:fitsSystemWindows="true"
    app:theme="@style/NavigationDrawerStyle"
    app:menu="@menu/drawer">

</android.support.design.widget.NavigationView>

Na sua MainActivity
    navigationView.setNavigationItemSelectedListener(new NavigationView.OnNavigationItemSelectedListener() {

        // This method will trigger on item Click of navigation menu
        @Override
        public boolean onNavigationItemSelected(MenuItem menuItem) {

            //Checking if the item is in checked state or not, if not make it in checked state
            if (menuItem.isChecked()) menuItem.setChecked(false);
            else menuItem.setChecked(true);

            //Closing drawer on item click
            drawerLayout.closeDrawers();
            FragmentTransaction fragmentTransaction = getSupportFragmentManager().beginTransaction();

            //Check to see which item was being clicked and perform appropriate action
            switch (menuItem.getItemId()) {

                //Replacing the main content with right fragment;
                case R.id.start:
                    if (startfragment == null) {
                        startfragment = new StartFragment();
                    }
                    fragmentTransaction.replace(R.id.frame, startfragment);
                    fragmentTransaction.commit();
                    return true;

                case R.id.second:
                    if (secondfragment == null) {
                        secondfragment = new Secondfragment();
                    }
                    fragmentTransaction.replace(R.id.frame, secondfragment);
                    fragmentTransaction.commit();
                    return true;

                default:
                    if (startfragment == null) {
                        startfragment = new StartFragment();
                    }
                    fragmentTransaction.replace(R.id.frame, startfragment);
                    fragmentTransaction.commit();
                    return true;

            }
         }
    });

E o modelo de um fragmento
public class StartFragment extends Fragment {
    //Ui Elements
//Atributtes
private View view;

@Nullable
@Override
public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) {

    view = inflater.inflate(R.layout.fragment_start, container, false);
    return view;
}
@Override
public void onActivityCreated(Bundle savedInstanceState) {
    super.onActivityCreated(savedInstanceState);

}}

Comment: Você vai precisar troca de Activity apenas se você for utilizar um Collapsing Toolbar, algo do gênero. Caso não, faça como pede os comentários a cima

Se for usar a troca de activitys, terá que usar drawer em todas as activity.
Comment: Criei um novo projeto e criei 2 fragmentos ( um com um textview escrito calculo rpm ), entendi a parte de colocar ele no menu, quando eu clico ele abre o fragmento, porem dessa forma: http://imgur.com/a/yE9TF com o conteudo do fragmento sobre o app bar e sobre o resto do app, como deixar ele parte do app ?
Comment: Consegui, foi só alterar o frame do main activity pro content main

Abrir tela dentro do mainactivity


connecting to Exchange works on mac, not on windows

Question: OK, so here's the scenario:
We have an environment with Exchange 2013 running on Windows Server 2012 (including the DC). Current users can access their emails through outlook and OWA.
However, whenever I try to configure a second Exchange account on a computer on the internal network, I get an error (it's quite random, one says there's a problem with the OST file, the other says it couldn't load the information store). When I try to configure an Exchange account on my mac (NOT on the internal network) it works perfectly with mail.app. When I try to do the same for my Windows PC (remote as well) it can't check the username...
All laptops on the internal network are connected through a switch to the server. No firewall or router is present between them AFAIK, so I think we can rule out a networking issue. Anyone have a clue what might be going on here?
Comment: When you say second Exchange account, do you mean for the same Windows user profile, or a different user on the same machine?
Comment: Same Windows profile indeed. The new account is for a new employee, the old account needs to be active as well...
Comment: That doesn't make sense. Are you saying John Doe has an email account setup in Outlook, and you are trying to add it a second time? I think you need to clarify the steps you are taking.
Comment: John Doe has an email account, and I'm adding one for Jane Doe (who replaces him, while customers might still send mails to John Doe because not all of them were notified - hence the fact that there should be 2 mailboxes in Outlook)
Comment: Are you logged into the system as John, and trying to add Jane in Outlook?
Comment: Yes I am. But then again, that doesn't matter much, since the organisation's devices (apart from servers) haven't been added into Active Directory yet. Yeah, long way to go...
Answer: Not saying DanBig's answer isn't valid...but:

In this particular scenario, why not simply set forwarding on the mailbox of the employee that is leaving (in the EAC): http://technet.microsoft.com/en-us/library/dd351134%28v=exchg.150%29.aspx and possibly set an OOF auto-reply as well stating that "I'm no longer with the company, direct all email to email@example.com now."
If you really want "new employee" to have access to "old employee" mailbox, again it's easier in the EAC to simply give "new employee" full access permissions http://technet.microsoft.com/en-us/library/bb124097%28v=exchg.150%29.aspx and then in Outlook it will automatically show the additional mailbox on the left window pane (after a restart of Outlook)
Comment: I agree with the EMC persmissions. I didn't include those because they seemed like it may be out of comfort zone.
Comment: Considering it is both a DC and an Ex2013 box in one, you may be right. :)
Answer: One option is to setup delegate access to accomplish this.  Look at the following link, but i will post the pertinent info. Replace User A (Jane) and User B (John) appropriately. In the end, the user that actually works there, should be logged into windows, not the other way around.
http://support.microsoft.com/kb/291626/en-gb
Microsoft Office Outlook 2010

Start Outlook with a profile that is configured for the Exchange Server mailbox for account B. You may have to log on to the network as the user of account B for proper validation.
On the Files menu, click Info.
Click Account Settings, and then click Delegate Access.
On the Delegates tab, click Add.
Type or select the name for the user of account A, click Add, and then click OK.
In all lists, click to select Editor (can read, create, and modify items) in the Delegate Permissions dialog box.
Click OK two times.
If the Folder list is not visible, click Folder List on the View menu.
Right-click Mailbox - user name, and then click Properties for 'Mailbox - user name on the shortcut menu.
On the Permissions tab, click Add.
Type or select the name for the user of account A, click Add, and then click OK.
In the Name box, click the newly added entry for account A.
In the Permission Level box, click Owner, and then click OK.
Repeat steps 8 through 12 for all the other folders in the mailbox.
On the File menu, click Exit and Log Off.
Restart Windows, and then log on as the user of account A.
Start Outlook with a profile that is configured for the Exchange Server mailbox for account A.
On the Files menu, click Info.
Click Account Settings, and then click Account Settings.
On the E-mail tab, click the name of the account, and then click the Change icon.
Click the More Settings button.
Click the Advanced tab.
Click Add to add an account to the Open these additional mailboxes: list.
Type the name for the user of account B, and then click OK three times.
Click Next, click Finish, and then click Close.

The mailbox for account B appears in your Folder List. The user for account A can send messages and meeting requests on behalf of the user of account B by entering the name of the user for account B in the From Field. These messages contain the names of the users for both account A and account B.
Comment: Thanks for the info! Though I should mention, with one out of the two laptops I removed the "old" account and I still got errors... I'll see how this approach works.


Dell R410 and X5690 wont boot

Question: I have an R410 that we opened up to upgrade from 2 x E5520 to 2 x X5690.
However after the upgrade network didnt work and we could not even get a monitor output. We tried swapping CPU positions and eventually pulled the processor out.
We since discovered we were running 1.2 bios and dell is up to 1.12. I couldnt get release notes between 1.2 and 1.9 so I am not sure if there was a fix to make the X5690 compatible.
Is there a problem with going from E series to X series on R410? Do I just need bios upgrade?
How do I tell if the CPU is faulty? they were used CPU's.
Thanks in advance.
Comment: That was a pretty big shift from a lower-end Nehalem CPU to a high end Westmere CPU. I would have checked compatibility prior to purchasing. Tough lesson, but the CPUs aren't compatible with your platform.
Answer: The X5690 has a TDP of 130 W which means it is unsupported with this server. See ftp://ftp.dell.com/Manuals/all-products/esuprt_ser_stor_net/esuprt_poweredge/poweredge-r410_User%27s%20Guide6_en-us.pdf
That is something you could and should have learned before buying the CPUs.
Comment: Thank you for your help and the link. We made an assumption. All is not lost, we will put those processors into an R710 instead. So it looks from the data that we could purchase the X5675 which is compatible with the R410 with idrac upgrade (done) and bios upgrade. Were on 1.2 and they have a 1.12 bios upgrade available

Dell R410 and X5690 wont boot


Is a retry necessary here in a timeout?

Question: I have the following code:
task :wait_for_vault_ssl_up do
  Timeout::timeout(15) do
    begin
      TCPSocket.new('localhost', 8200).close
      puts "Vault ssl up and ready!"
      true
    rescue Errno::ECONNREFUSED, Errno::EHOSTUNREACH, SocketError
      puts "Vault ssl not ready... retrying"
      retry
      false
    end
  end
  rescue Timeout::Error
    false
end

In this, i set a timeout of 15 seconds and try to connect to the localhost:8200 my question is, if its not up, i currently do a retry in the rescue. is that necessary or will it automatically continue trying to connect for 15 seconds?
Comment: `TCPSocket.new` has a `connect_timeout` option that will raise `Errno::ETIMEDOUT`. Don't use `Timeout::timeout` in a situation like this.
Comment: @Stefan how should i handle waiting for a port to be open? can you provide a solution? just retrying on the tcp connect for a given time?
Comment: Write a loop that keeps trying to connect. You probably want to [`sleep`](https://ruby-doc.org/core-3.0.1/Kernel.html#sleep-method) some time after a failed attempt.
Comment: BTW, `Timeout::timeout(15)` will _interrupt_ the given block after 15 seconds, no matter what. It's used to forcefully terminate long-running operations, like you would `kill -9` an unresponsive process. You usually don't want to use it.
Answer: The use of retry will retry the execution of the whole block being rescued. So in this case, retry will start execution over again at your call to TCPSocket.new. This will happen immediately after the exception is rescued and there will not be a delay.
As Stefan has mentioned in the comments, you should use Kernel#sleep before calling retry. This will continue retrying until interrupted, so if you found yourself in a situation where the SSL connection will never be ready, the code will loop forever. You can either keep track of a retry count, or continue utilizing Timeout::timeout, using a longer timeout length, at which point the loop should give up entirely.
I think the clearer and simpler solution would be to use a retry counter, and give up after a predefined number of retries.

Is a retry necessary here in a timeout?


Understanding the measurement error in Levenberg-Marquardt algorithm for non-linear least square

Question: I have been reading this pdf on LM algorithm and I have been having trouble understanding a few parts of it. For example, in the Introduction:

where $σ_{y_i}$ is the measurement error for measurement $y(ti)$

What exactly is meant by this measurement error and why does it have the symbol of the standard deviation? Is there a formula for this measurement error for $y_i$ ?

Typically the weighting matrix $W$ is diagonal with $W_{ii} = 1/σ^2_{y_i}$ , More formally, $W$ can be set to the inverse of the measurement error covariance matrix

How do we visualize the covariance matrix of the error? And how is a diagonal matrix produced by taking inverse of this covariance matrix? And what is the role of $σ_{y_i}$ in this matrix?
Answer: There is an association between the error of the measurement and the width of the distribution of measurements. I don't know of an official source where it is written down to reference it. In every natural science and engineering this implicitly used like this.
It is assumed that the measurements have a fixed and homoskedastic distribution, so a single value is sufficient for the variation. In the case of a Gaussian distribution, it is obvious to use the standard deviation. But for every other (well behaved) case, you can use principally use a transformation; but the shorthand is also just the standard deviation $\sigma$. So it's the usual formulation for the second central moment of the distribution:
$$\sigma_i^2 = \int (y_i- \hat{y_i})^2y_i() dy_i$$
(for a distribution $y_i()$ and its central value $\hat{y_i}$)
For the weighting matrix, the general case is the inverse covariance matrix. For "visualization", you can start with the most simple case: if you chose a diagonal matrix, the $\chi^2$ is simply the squared sum of deviations divided by the variance -- the inverse elements of the covariance matrix. You can think of these inverse variances $\frac{1}{\sigma^2}$ as weights.
A non diagonal covariance matrix simply means that the measurements are not independent from each other, but have a mixed expression. You might know the expression and could diagonalize it, but you don't need to. The matrix multiplication expression of the $\chi^2$ formula entirely takes it into account. You can refer to a standard book of applied statistics, depending on your field.
The elements of the covariance matrix are computed pairwise:
$$cov(y_i, y_j) = \sigma_{i,j} = \int (y_i-\hat{y_i})(y_j-\hat{y_j}) dy_idy_j $$
First of all you can see that the diagonal elements are the variances, $cov{i,i} = \sigma_i^2$. And the off-diagonal elements are values that describe the kind of average deviation from the mean value: a positive number means that if you go above the mean value in one variable, the other variable will be distributed around a higher value than its central value. If the covariance value is smaller than zero, you expect values smaller than the central value in the second variable, if you still chose higher values than the central value in the first.
For a plot of this, simply look at the first image in the wikipedia entry on covariance.
Comment: How I'm trying to imagine it is instead of thinking of all the $y_i$ 's as values of a random variable $y$ , in the general case, each $y_i$ is an independent random variable with its own distribution and standard deviation. Does it make sense?
Comment: Yes, exactly. Think of the case where you might measure something like the polar coordinates. You might have different devices for the angle and the distance, with different associated errors. And you might want to extract the best parameters in cartesian coordinates -- and directly you could derive the covariance matrix by analytical means.
Comment: Another thing. The covariance matrix in the general case seems to be diagonal as well, since we are taking $W$ as the inverse of the cov matrix, whose diagonal entries are the ${\sigma_i}^2$. If so, then are we assuming the $cov(y_i, y_j) = 0$ for all $i \neq j$ ?
Comment: In the general case, the covariance matrix isn't diagonal. But if you design your measurement carefully, you can build this in. In the case of a diagonal covariance matrix, all non-diagonal elements are zero. To invert the matrix, it doesn't have to be diagonal; that just makes it rather simple and in case of a numerical expression (e.g. automation with computers) very fast and accurate.


Initializing gradient boosting with the sample mean

Question:
For gradient boosting in the regression setting, the final vector of fitted values is
$$F_M(x) = \bar{y} + \rho_1h_1(x) + \ldots, + \rho_M h_M(x)$$
Suppose I have a new data set $x_{new}$ that I'd like to make predictions on. Are my predicted values the following:
$$F_M(x_{new}) = \bar{y} + \rho_1h_1(x_{new}) + \ldots, + \rho_M h_M(x_{new})$$
My question is, am I also adding the sample mean of my responses in my training data, $\bar{y}$ to the final set of predictions on this new data set?
Comment: Yes, we do. Notice that the mean if simply $F_0(x)$.
Answer: Yes. We take the mean as the "best first guess" and then keep adding terms to try and correct the errors of the model so far.
When we use this model to make predictions, we don't just need the error correction terms, but we also need the same initial guess. So we use the $\bar{y}$ from the training set.


Using Parzen Window approach

Question: When is preferable using Parzen Windows approach, so a nonparametric approach, instead to a parametric approach?
Answer: It's under kernel density estimation (KDE). It's non-parametric so you don't have to assume a form. Otherwise, you'd need to assume some form of distribution, (e.g. exponential, normal, mixture of normals etc) and estimate the parameters. With Parzen, you could fit a distribution without making any initial assumptions.

Using Parzen Window approach


Finding the position of some substrings in a string

Question: After a bit of hoogle-ing I came up with this:
import Data.List
substrPos xs str = map (\x -> findIndex (isPrefixOf x) $ tails str) xs

That should give me a list of indices that represent the first appearance in str of every string in xs.
E.g.:
Prelude Data.List> substrPos ["Foo", "Bar"] "FooBar"
[Just 0,Just 3]

The question is how can I write it more idiomatic, maybe I can lose the lambda?
Maybe somehow (function composition?) remove the parens enclosing isPrefixOf x?
Maybe write it in point free style?
How should I go about thinking of these things?
PS: The only modification I could come up with (after first writing the function) is the removal of parens from tails xs by function application.
Comment: You may want to see if this would get be more appropriate and get more attention at [codereview.se].
Comment: Thanks, I will if needed.
Comment: Although you could write it several other ways (in addition to point free, you could e.g. go the other way and use a list comprehension), I think your code is plenty idiomatic enough. Although if your argument order were the other way around, I'd have recommended losing the explicit `xs`.
Answer: You can always install the pointfree tool and check it for yourself and try to understand the resultant code.
Pointfree tool gives me the following result:
map (($ tails str) . findIndex . isPrefixOf) xs

Although you should note that pointfree style is not always idiomatic. Some people consider it as unreadable and pointless.
Comment: Thanks for pointing out that tool, I think I understood its result. Indeed the result may be much harder to read, but I'm doing this for my own exercise with haskell.


String replace regex character classes using regex

Question: This string has regex character classes which need to be removed. As well as reduce multiple spaces to single space.
I can chain replace() but thought to ask if one can suggest a one regex code to do the whole job at one go. How can it be done? Thanks

"\n\t\t\t   \n\n\t           \n\t           \t\tFood      and drinks \n   \t\n"

This is needed:

"Food and drinks"

var newStr = oldStr.replace(/[\t\n ]+/g, '');  //<-- failed to do the job

Comment: Don't forget that replacing the spaces will replace _all_ the spaces.
Comment: I wouldn't do it in one replace, although I think it is possible. Its much more clear to use one regex for removing chars, one for retaining single spaces, and one for trimming.
Answer: I'd recommend this pattern (assuming you want to keep \ns or \ts in your main string):
/^[\t\n ]+|[\t\n ]+$/g

If you don't want to keep them, you can use something like this:
/^[\t\n ]+|[\t\n]*|[\t\n ]+$/g

Comment: I tried the last pattern since I don't want to keep them, but it leaves a new line at the end. Try it in the console and you should see.
Comment: Trying your first removed the leading but left the trailing spaces, because it needs the global option to get both: `oldStr.replace(/^[\t\n ]+|[\t\n ]+$/g, '');`
Answer: You want to remove all leading and trailing whitespace (space, tab, newline) but leave the spaces in the internal string. You can use the whitespace character class \s as shorthand, and match either the start or the end of the string.
var oldStr = "\n\t\t\t \n\n\t \n\t \t\tFood and drinks \n \t\n";

// ^\s+ => match one or more whitespace characters at the start of the string
// \s+$ => match one or more whitespace characters at the end of the string
// | => match either of these subpatterns
// /g => global i.e every match (at the start *and* at the end)

var newStr = oldStr.replace(/^\s+|\s$/g/, '');

If you also want to reduce the internal spaces to a single space, I would recommend using two regexes and chaining them:
var oldStr = "\n\t\t\t \n\n\t \n\t \t\tFood   and      drinks \n \t\n";
var newStr = oldStr.replace(/^\s+|\s+$/g, '').replace(/\s+/g, ' ');

After the first .replace() all of the leading and trailing whitespace is removed, leaving only the internal spaces. Replace runs of one or more space/tab/newline with a single space.
One other way to go could be to reduce all runs of whitespace to a single space, then trim the one remaining leading and trailing space:
var oldStr = "\n\t\t\t \n\n\t \n\t \t\tFood   and      drinks \n \t\n";

var newStr = oldStr.replace(/\s+/g, ' ').trim();
// or reversed
var newStr = oldStr.trim().replace(/\s+/g, ' ');

.trim() doesn't exist prior to ES5.1 (ECMA-262) but the polyfill is essentially .replace(/^\s+|\s+$/g, '') (with a couple of other characters added) anyway.


Big difference in p-value from permutation test

Question: I have roughly 60 values of discrete temperature satisfaction votes ranging from 0 to 4. In the figure below I have grouped the votes depending on the daily number of hours the individual occupants are sitting at their desks. Now I want to make a permutation test, to show if there is any significant difference on the mean vote of occupants having up to 7.5 work hours per day at their workplace, and occupants having more than 7.5 work hours per day.

I have used the following code to carry out the one-tailed permutation test:
g1 <- subset(df, deskhours <= 7.5)$temperatureSatisfaction  # length = 13
g2 <- subset(df, deskhours > 7.5)$temperatureSatisfaction  # length = 43

k <- 100000

perms <- replicate(k, sample(c(g1, g2)))
my.mean.difs <- apply(perms[1:13, ], 2, mean) - apply(perms[14:56, ], 2, mean)
sum(my.mean.difs <= (mean(g1) - mean(g2))) / k

Actually I did not expect the use of <= or < (in the last line of the code) to make any significant difference on the p-value. However, I found that the use of <= gave me a p-value of 0.084, and the use of < gave me a p-value of 0.046. So, almost 4 % of the differences in means in the my.mean.diffs vector, is exactly equal to the actual difference between the two groups (mean(g1) - mean(g2)).
My logic says that it is very unlikely that the difference in means from the shuffled data, will hit exactly the observed difference in means. Am I wrong here? And which is the most correct sign to use: <= or <? With my significance level of 0.05, the conclusion is absolutely dependent on this.
Answer:
My logic says that it is very unlikely that the difference in means from the shuffled data, will hit exactly the observed difference in means. Am I wrong here?

Yes, you're wrong - in that the observed difference may not be at all unlikely. The distribution of your response is discrete, so the distribution of the difference in means will be discrete, which means that you'll expect a number of other arrangements to produce the same mean difference.

And which is the most correct sign to use: <= or <?

The definition of p-value is "the probability of a result at least as extreme as the one observed". Which is to say, when you work out the p-value, you include values equal to the observed one.
--
Why do you dichotomize the work hours?
Comment: The data comes from a questionnaire, where the occupants select a number representing the daily number of work hours. But I see the point in making the scale continuous.
Comment: Ok, It is more clear to me now. I thought the difference in means was continuer, due to the fact that it could take a value like -0.5527728--which is the difference in the observed means. But it makes sense that the difference in means not can be any value, since the original data are continuos. Thanks.
Comment: It's easier if you realize that (given the total of both samples is the same in every permutation), the difference in means is a linear function of the sum of the first sample. The sum of the first sample is the sum of 13 numbers from 4, 5, ... up to 9. It will either be an integer, or an integer $+\frac{1}{2}$. You might get a value like '92' or '89.5' for the sum of the values in the first sample ... and there's lots of ways to get a value like those. The distribution of the difference in means is identical, but scaled and shifted.
Comment: It not entirely correct. First, remember that it is the mean of the temperature satisfaction (values on y-axis) I use to calculate the difference in means. Not the number of hours at the work desk. Second, since the samples has the length of 13 and 43. The mean of the two groups will therefore be able to take a value different from an integer or an integer + 0.5. However, the number of possible means, is finite and discrete, because the distribution the means are coming from is discrete.
Comment: Sorry, you're right about me looking at the wrong variable but temp satisfaction is even easier - it's pure integer. The rest of the argument you appear to have misunderstood, so let me repeat it in terms of the correct $y$. Given the 13+43 observations, the **sum** of observations across both resamples is fixed. It immediately follows that the difference in mean temperature satisfaction is a linear function of the **sum** of temperature satisfaction in the first sample. The **sum** of temperature satisfaction in the first sample is integer. Therefore the difference in means must be discrete.


How to detect that the mapbox viewport have been moved significantly to "search within this area"?

Question: I am using "react-map-gl", nothing of the sort was available out of the box.
What is the most preferred method for detecting this scenario?
Answer: This seems like a good place for the useEffect hook.
viewport state changes whenever a user pans or zooms, so if you place these three properties in the useEffect dependency array, whatever logic within useEffect body will be invoked whenever the viewport state updates.
Example
useEffect(() => {

  // implement your search logic here

}, [viewport.latitude, viewport.longitude, viewport.zoom]);


General hints regarding the use of the binomial distribution in conditional probabilty problems

Question: This is from an online course textbook (Harvard Statistics 110: see #45, p. 28 of  pdf). It's not a homework question, as the answer is provided in the text. I just really want to understand why the solution is that way.

A new treatment for a disease is being tested, to see whether it is better than the standard treatment. The existing treatment is effective on $50\%$ of patients. It is believed initially that there is a $2/3$ chance that the new treatment is effective on $60\%$ of patients, and a $1/3$ chance that the new treatment is effective on $50\%$ of patients. In a pilot study, the new treatment is given to $20$ random patients, and is effective for $15$ of them.
(a) Given this information, what is the probability that the new treatment is better than the standard treatment?
(b) A second study is done later, giving the new treatment to $20$ new random patients. Given the results of the first study, what is the PMF for how many of the new patients the new treatment is effective on? (Letting $p$ be the answer to (a), your answer can be left in terms of $p$.)

The answer to (b):
   Let $Y$ be how many of the new patients the new treatment is effective for and $p=P(B|X=15)$ be the answer from (a).  Then for $k\in \{0,1,\ldots,20\}$,
   \begin{align}
P(Y=k|X=15) &= P(Y=k|X=15,B)P(B|X=15)  \\
            &\quad + P(Y=k|X=15,B^c)P(B^c|X=15)  \\[5pt]
            &= P(Y=k|B)P(B|X=15) + P(Y=k|B^c)P(B^c|X=15)  \\[5pt]
            &= {20\choose k} (0.6)^k(0.4)^{20-k}p + {20\choose k}(0.5)^{20}(1-p).
\end{align}

Questions:

Before I looked at the answer, I didn't think that the answer to (a) would be found using a binomial distribution. But after I read the answer I was convinced by it. So, is there a kind of "general hint" when to use a binomial distribution?
I don't understand why the answer to (b) is that way. Specifically, why is $$P(Y=k|X=15, B)=P(Y=k|B)$$ and the analogous one conditioning on $B^c$?
This equality is implied from the second equality of (b)'s answer.
$$P(Y=k|\underbrace{X=15, B}_{\text{intersection removed}})P(B|X=15)+P(Y=k|\underbrace{X=15, B^c}_{\text{the same here}})P(B^c|X=15)$$
$$=P(Y=k|B)\ \ \ \ \ \  P(B|X=15)\ \ \ \ \ + \ \ \ \ \ \ \ \ P(Y=k|B^c) \ \ \ \ \  P(B^c|X=15)$$
Answer:
Specifically, why is $$P(Y=k|X=15, B)=P(Y=k|B)$$

The expression P(Y=k|X=15,B) = P(Y=k|B) is allowed because
$$P(Y=k|X=i,B) = {{20}\choose {k}} 0.6^k0.4^{20-k} $$
independent from $X=i$, so it can be left out
This logic is indeed not generally true, ie $P(a|b,c)$ may be different from $P(a|c)$, and is just true for this specific case where the experiments are independent (new random patients) and it is assumed that the result have no correlation other than that they are both defined by the condition $B$ or $B^c$.
Comparable expression with a dice roll for any $k$ and $l$: $$P(\text{next roll} =k \, | \,\text{ last roll} = l, \text{ dice = fair })=\frac{1}{6}$$
when we assume the rolls are independent. For that dice roll you are allowed to write $P(k|l,fair) = P(k|fair) = 1/6$

Before I looked at the answer, I didn't think that the answer to (a) would be found using a binomial distribution. But after I read the answer I was convinced by it. So, is there a kind of "general hint" when to use a binomial distribution?

Effectively you have a mixture distribution of two binomial distributions
$$ p \underbrace{ {{20} \choose {k}} 0.6^k0.4^{20-k}}_{\text{Binom$(20,0.6)$}} + (1-p) \underbrace{ {{20} \choose {k}} 0.5^k0.5^{20-k}}_{\text{Binom$(20,0.5)$}} $$
being equivalent to a probability $p$ that the medicine works on 60% of the people and a probability $1-p$ that the medicine works on 50% of the people.
This logic may not seem so correct. How could one, based on question 'a', believe that you have the medicine working on only either 50% or 60%? That is a false dichotomy.
However, note that this is created due to the weird prior believe that it
is either working on 50% or 60% (probabilities 2/3 vs 1/3) and
nothing else (so the posterior distribution will stick to this weird
believe and only change these ratios 2/3 and 1/3 into $p$ and $1-p$).
It is not a realistic question, what one would get in practice. But it illustrates a more complicated principle (see next point).
Eventually (a better alternative to the question problem statement which may occur later in your course), one could use a beta-binomial distribution in which every probability has an assigned value (not just 2/3 60% and 1/3 50%, or $p$ 60% and $1-p$ 50%). Then the posterior distribution for $f$, the fraction of people on which the medicine works (which could be modeled as a beta distribution), is a spectrum instead of just two values. This spectrum of probabilities for different $f$ is then used to write out the beta-binomial distribution.
note that the letter $p$ is often a parameter for the binomial distribution, the probability that the medicine works. Here it is used differently for the probabilities $p$ and $1-p$ that the medicine works with 60% or 50%. This might be confusing.
Comment: Many thanks for your answer :); it's detailed and convincing. I hope the following illustrate what I meant by my expression. In the answer to (b)[the image]:
$$P(Y=k|\underbrace{X=15, B}_{\text{intersection taken off}})P(B|X=15)+P(Y=k|\underbrace{X=15, B^c}_{\text{the same here}})P(B^c|X=15)$$ $$=P(Y=k|B)\ \ \ \ \ \  P(B|X=15)\ \ \ \ \ + \ \ \ \ \ \ \ \ P(Y=k|B^c) \ \ \ \ \  P(B^c|X=15)$$
Comment: @AbduMagdy the probability is $$P(Y=k|X=i,B) = {{20}\choose {k}} 0.6^k0.4^{20-k} $$ **independent from *i***. So you can leave that part out (and similarly for the other expression with $B^c$). ----- Was this (the step in which X=15 gets dropped from the condition) the heart of the matter? (should I change my answer to focus on this point?)  Or was it about the entire derivation?
Answer: In this response I will limit myself to the general question of when to use the binomial.
The binomial arises naturally as the count of successes in a sequence of independent Bernoulli trials with constant success probability $p$.
It's useful to keep the conditions in mind (both for Bernoulli trials and the independent and constant $p$ of the sequence of them) and see whether that might be more-or-less plausible. [Often the model is a reasonable approximation when the conditions don't quite hold.]
For example if you have a population with some fraction having some attribute and you sample it at random with replacement, the count of the number having the attribute in your sample would be an obvious candidate for using a binomial. If you instead sample without replacement but the population is very very large compared to your sample (making the effect on $p$ at trial $i$ and the dependence negligible) then you could use the binomial as an approximation.
Comment: Thanks so much for you answer :). The hardship really arises (at least from my viewpoint) when you try to apply the characteristics you've provided on the experiment in hand. For instance, the treatment experiment in the OP is so convoluted and confusing that I couldn't specify (recognize or identify) the Bernoulli Trials and $p$ that I can build on my model. However, I think it needs more experience.


How to import multiple terms from XML into a node?

Question: I am trying to import all the Destinations as taxonomy terms using following XML. In my situation, the list is not a comma separated list. It is as follows:
<product>
    <category>Ship</category>
    <id>000032</id>
    <Description>The description comes here</Description>
    <list>
        <Destination>
            <DestinationName>Dest 1</DestinationName>
            <DestinationCountryISOAlpha3>US</DestinationCountryISOAlpha3>
        </Destination>
        <Destination>
            <DestinationName>Dest 2</DestinationName>
            <DestinationCountryISOAlpha3/>
        </Destination>
    <list>
</product>

I have a taxonomy called destinations and the list is already imported. Now how can I import Destinations list of a product as a term reference for the product node. I am using feed module, Feeds XPath Parser, and Feeds Tamper modules?
Answer: I finally found a way to achieve this goal at least for my use case. Publish my solution here for anyone to use if required.
My Settings for XPath XML parser :
Context -> //product/list/Destination
Product id -> ../../id
List of ports -> DestinationName # This is a term reference field which well have multiple terms per product

Then mapped all required fields as normal.
I created a small module with hook_feeds_after_parse() hook (I will use "My module").
My module has one .info file and the .module file.
function my_module_feeds_after_parse(FeedsSource $source, FeedsParserResult $result) {
    $current_id = -1;
    $cuurent_destination_list_final = '';
    foreach($result->items as $key => $row) {
        if($current_id !== $row['xpathparser:0']){
            $cuurent_destination_list_final = '';
            $current_id = $row['xpathparser:0'];
        }
        if($cuurent_destination_list_final === ''){
            $cuurent_destination_list_final = $row['xpathparser:1'];
        }else{
            $cuurent_destination_list_final = $cuurent_destination_list_final . ',' . $row['xpathparser:1'];
        }
        $result->items[$key]['destination_list_final'] = $cuurent_destination_list_final;
    }
}

What this module does is it creates a comma separated list of destinations.
Once we set $result->items[$key]['destination_list_final'] (This builds a comma separated list of terms) using this module, we can refer destination_list_final from Feeds Tamper (Refer this).
Install and enable Feeds Tamper PHP.
Add 3 pluggings to xpathparser:1 -> List of Destinations

Execute php code (This come from above installed module) -> return     $item['destination_list_final'];
Find replace - Use to remove double quotes (")
Explode - Use to explode the comma separated list we created from the custom module.

That's it. It worked for me.


Logarithmic Regression by hand

Question: I'm trying to write some code to do a regression on data weight (x) and time (y). As best as I can tell, the model should be y = b1 + b2ln(x), but I don't know how you can do this by hand (I know how to in R...). I know how to do a simple linear regression by hand. Appreciate it.
Comment: calculate logs: x1 = ln(x). Regress y on x1. Done. BTW your model is wrong unless your errors are all 0 (in which case you can just compute it from any two distinct points). That is, you either need to write $E(y)=$ or you need to add $+e$ on the end
Comment: can it be done using the least squares approach?
Comment: That's what I'm saying up there, yes. How are you doing it in R if not by least squares?
Comment: I guess I'm not exactly sure lol. So normally you would calculate SXX = sum(x-xbar)^2; SXY = sum((x-xbar)(y-ybar)); SYY = sum(y-ybar)^2. Then B1 = SXY/SXX. How would this differ given the log term?
Comment: The *entire* thing is in my first comment. Try reading it again, please. It has two steps. Tell me what you don't understand about step 1 (Calculate the logs, calling the result $x_1$). If you understand what that means, we can move to step 2.
Comment: Am I just taking all of my x data points and applying ln to them? Giving me a new set of data points?
Comment: Yes. That's it exactly.
Comment: Ok, then from there I just proceed like normal?
Comment: Yes, treating the new variable as if it were $x$. I'll post an example for you to follow.
Comment: Example posted (I round off the numbers a bit when I take logs to save you some calculation).
Answer: You say that you'd calculate the slope as follows:

So normally you would calculate
$$S_{XX} = \sum_i (x_i-\bar x)^2\\
S_{XY} = \sum_i ((x_i-\bar x)(y_i-\bar y))\\
S_{YY} = \sum(y_i-\bar y)^2$$
Then $b_2 = S_{XY}/S_{XX}$.

So imagine you have a set of x-values and y-values:
<code>    y       x
1 2.3 0.36772
2 5.3 1.64873
3 6.5 7.38910
</code>
Step 1:
calculate a new $x$, $x_1 = \ln(x)$
<code>    y   x1
1 2.3 -1.0
2 5.3  0.5
3 6.5  2.0
</code>
Now regress $y$ on this new $x_1$ as usual
<code> SXX = 4.50; SXY= 6.30; SYY = 9.36

 b2 = SXY/SXX = 1.4

 b1 = mean(y) - b2 . mean(x) = 4.7 - 1.4 . 0.5 = 4
</code>

Logarithmic Regression by hand


Cumulant of sum of correlated random variables?

Question: Let $X,Y$ be two random variables. We denote by $[X^k]$ and $[Y^k]$ the $k$'th order cumulants of $X$ and $Y$, respectively. I'm interested in computing the $k$'th order cumulant of $Z = X+Y$.
If $X,Y$ were independent, then a well-known property implies that $[Z^k] = [X^k] + [Y^k]$.
Now suppose $X,Y$ are not independent. Can we write an expression for the cumulants of $Z$, from the cumulants of $X$, $Y$, plus additional terms related to the dependence between $X$ and $Y$ (e.g., the cumulants of products $XY$?
Update: wolfie's post almost answers this question. This expansion feels like something that should be known in the literature. So I am adding the reference tag here, in case anyone can suggest relevant papers.
Comment: Also asked at https://math.stackexchange.com/q/3626887/321264.
Answer: There might be a number of ways to tackle this. To get a feel for the problem, my first thought was which tool / function could be used to check it out. In the mathStatica package for Mathematica, there is a general function that can find cumulants of power sums $s_{r,t}=\sum _{i=1}^n X_i^r Y_i^t$. For example, $s_{1,0} = \sum _{i=1}^n X_i$ and  $s_{0,1} =\sum _{i=1}^n Y_i$.
Then, in a bivariate dependent world, $Z = X+Y$ can be written as $s_{1,0} + s_{0,1}$ where we are considering the very simple case of $n = 1$.
Then, the problem at hand is to express the $r^\text{th}$ cumulant of Z, in terms of cumulants of the $X$ and $Y$. This can be done using the CumulantMomentToCumulant function.
Here, for example, is the 3rd cumulant of Z expressed in terms of the bivariate cumulants of $X$ and $Y$:

where $\kappa _{r,s}$ denotes the various bivariate cumulants.
Here are the first 8 cumulants of $Z = X+Y$:

The solution is immediately identifiable by induction as Pascal's Triangle / Binomial Theorem at play.
Comment: This is very cool. I did not expect to get a binomial rule here. I tried to prove it but it gets very messy and I did not manage to get it done.


Constraints on the Coefficients of a Seasonal ARIMA Model (Possible Software Bug ITSM)

Question: I am attempting to fit a seasonal ARIMA models using ITSM software. The following is the model.
ARIMA$(1,1,0)\times(1,1,0)_{12}$: $\phi(B) \Phi(B^{12}) = (1-\phi B)(1-\Phi B^{12})=1-\Phi B^{12}-\phi_{1}B+\phi_{1} \Phi B^{13}$
Unfortunately, the output from ITSM has a negative coefficient for the 13th term, despite specifying in the software that the 13th term is multiplicative.
Method: Maximum Likelihood
ARMA Model:
X(t) = - .3485 X(t-1) + .0000 X(t-2) + .0000 X(t-3) + .0000 X(t-4)
       + .0000 X(t-5) + .0000 X(t-6) + .0000 X(t-7) + .0000 X(t-8)
       + .0000 X(t-9) + .0000 X(t-10) + .0000 X(t-11) - .4565 X(t-12)
       - .1591 X(t-13)
     + Z(t)

WN Variance = .001421

AR Coefficients
      -.348543       .000000       .000000       .000000
       .000000       .000000       .000000       .000000
       .000000       .000000       .000000      -.456469
      -.159099

Standard Error of AR Coefficients
       .085875       .000000       .000000       .000000
       .000000       .000000       .000000       .000000
       .000000       .000000       .000000       .082635
       .000000

(Residual SS)/N = .00142069

AICC = -.433387E+03
BIC  = -.434130E+03
FPE  = .001469

Any chance that this output is valid in terms of the AICC and the BIC? When I ran an ARIMA$(0,1,1)\times(0,1,1)_{12}$, all of the coefficients where correct in terms of their signs, and the AICC and the BIC was not all that different from this, AICC = -.440119E+03,
BIC  = -.444948E+03 .
Here is the data:
112 118 132 129 121 135 148 148 136 119 104 118 115 126 141 135 125 149 170 170 158 133 114 140 145 150 178 163 172 178 199 199 184 162 146 166 171 180 193 181 183 218 230 242 209 191 172 194 196 196 236 235 229 243 264 272 237 211 180 201 204 188 235 227 234 264 302 293 259 229 203 229 242 233 267 269 270 315 364 347 312 274 237 278 284 277 317 313 318 374 413 405 355 306 271 306 315 301 356 348 355 422 465 467 404 347 305 336 340 318 362 348 363 435 491 505 404 359 310 337 360 342 406 396 420 472 548 559 463 407 362 405

Note: I took the log of the data, and used Yule-Walker to get the initial parameter estimates.
Thanks!
Comment: Can you please confirm whether the output that you show is what you get when the restriction on the coefficient for $B^{13}$ is selected or is it the output from an AR model of order 13 with zeros for some of the coefficients?
Comment: Can you post the data?
Comment: In order to specify a seasonal ARIMA in ITSM, one must first get preliminary estimates of the coefficients of a standard ARMA model (in this case an AR(13) model). After this, you can set $\phi_{2},..,\phi_{11}$ to zero prior to finding the MLE.  The problem is $\phi_{13}$, is actually $ \phi_{1} B^{1} \phi \Phi B^{12} $, so ITSM gives you an option to specify this constraint. I am not sure if the sign of this coefficient being wrong in the output matters.
Comment: @javlacalle it is a restriction on $B^{13}$, with parameters 2-11 specified as 0.
Answer: I don't have the software ITSM but I could do some inspection upon the data and output that you posted. I fitted the model with different possible mistakes to see which one gives results closer to your output. I used a simplified version of the conditional sum of squares to fit this particular model.
My conclusion is that the code of ITSM that makes the computations is probably correct. The error arises most likely when reporting the results and printing the output. Below I give some details based on some examples that I run in the R software.

I fitted the ARIMA(1,1,0)(1,1,0) model written in expanded form $1 - \phi_{1}B - \Phi B^{12} + \phi_{1} \Phi B^{13}$ by minimizing the squared sum of residuals of the following equation with two parameters ($\phi_1$ and $\Phi$):
$$
y_t = \phi_1 y_{t-1} + \Phi y_{t-12} - \phi_1\Phi y_{t-13} + \hbox{error}_t \,.
$$
y <- log(window(AirPassengers, end = c(1959, 12)))
# compute residual sum of squares for ARIMA(1,1,0)(1,1,0) with monthly data
# and minimize using BFGS
# this approach and code is intended just for testing purposes
ssq <- function(coefs, y)
{
  n <- length(y) - 13
  z <- diff(diff(y), 12)
  e <- rep(NA, n-13)
  for (i in seq.int(14, length(z)))
    e[i-13] <- z[i] - coefs[1]*z[i-1] - coefs[2]*z[i-12] + prod(coefs)*z[i-13]
  sum(e^2)
}
# minimize the sum of the squared residuals
res <- optim(ssq, par=c(0,0), method="BFGS", y=y)
# estimated AR coefficients (phi1 and Phi, respectively)
res$par
# [1] -0.3942546 -0.4516274
res$par[1] * res$par[2]
# [1] 0.1780562

The fitted model is therefore (in the expanded form and moving the terms to the right-hand-side):
$$
y_t = 0.3942546 y_{t-1} + 0.4516274 y_{t-12} - 0.1780562 y_{t-13} +  \hbox{error}_t \,.
$$
As the OP said, the sign of the AR coeffcient related to $B^{13}$ should be the opposite of those for the other coefficients and this is what we get here.
Now, I make intentionally a mistake in the sign of the $B^{13}$ coefficient by typing - prod(coefs)*z[i-13] instead of + prod(coefs)*z[i-13]. Then the following is obtained:
ssq.bug <- function(coefs, y)
{
  n <- length(y) - 13
  z <- diff(diff(y), 12)
  e <- rep(NA, n-13)
  for (i in seq.int(14, length(z)))
    e[i-13] <- z[i] - coefs[1]*z[i-1] - coefs[2]*z[i-12] - prod(coefs)*z[i-13]
  sum(e^2)
}
res.bug <- optim(ssq.bug, par=c(0,0), method="BFGS", y=y)
res.bug$par
# -0.2459668 -0.3511647
res.bug$par[1] * res.bug$par[2]
# [1] 0.08637484

The results are far from the values reported by the OP. The results obtained before are much closer to the output returned by ITSM (except for the sign related to $B^{13}$). Thus, I would say there is no bug in the code that makes the computations.
We could also suspect that the restriction was actually not enforced, but that cannot be the case because .1591 is exactly equal to the product .3485*.4565, so I don't think it is the case either.

It's interesting to see that if the restriction for $B^{13}$ is not enforced and the AR(13) is fitted (with zeros for the AR coefficients of order 2 to 11), then the estimated parameters are very close those obtained above for the restricted model (this may suggest that, given a model of order 13, the multiplicative seasonal ARIMA model is appropriate for the data).
The unrestricted model contains three coefficients, $\phi_1$, $\Phi$ and $\phi_{13}$. The optimization algorithm searches for values of theses coefficients that minimize the sum of squared residuals:
$$
y_t = \phi_1 y_{t-1} + \Phi y_{t-12} - \phi_{13} y_{t-13} + \hbox{error}_t \,.
$$
ssq.ur <- function(coefs, y)
{
  n <- length(y) - 13
  z <- diff(diff(y), 12)
  e <- rep(NA, n-13)
  for (i in seq.int(14, length(z)))
    e[i-13] <- z[i] - coefs[1]*z[i-1] - coefs[2]*z[i-12] + coefs[3]*z[i-13]
  sum(e^2)
}
res.ur <- optim(ssq.ur, par=c(0,0,0), method="BFGS", y = y)
res.ur$par
# [1] -0.3951237 -0.4539019  0.1493490
res.ur$par[1] * res.ur$par[2]
# [1] 0.1793474

As we would expect, the parameters estimated in the unrestricted model do not satisfy the restriction exactly, but the last coefficient turns out to be close to the product of the two others. If we ran a test, the restriction would probably not be rejected.
Comment: @k6adams I have edited the answer adding the equations of the restricted and unrestricted models, which may be easier to read than the R code.
Comment: Thanks for help, I am going to need a bit of time to digest your answer. What did you mean by, "As we would expect, the parameters estimated in the unrestricted model do not satisfy the restriction exactly"? It looks like in your third model R did impose the restrictions correctly, am I wrong?
Comment: I should point out that the last 12 observations of the original "airpass" dataset were intetionally truncated for the purpose of forcast comparrision. Just thought I would mention this incase you used the full dataset.
Comment: I noticed it in your data, I did not include the last year.
Comment: The last model (`res.ur`) contains three coefficients to be estimated, related to the lags or order 1, 12 and 13. The coefficient related to lag 13 is therefore allowed to take any value during the process of minimizing the sum of squares in `optim`.
Comment: Even if the restriction is appropriate, it would be quite a coincidence to find that the product of the estimates for the first two coefficients happens to be equal to the estimate of the third coefficient.
In your example we have exactly `.1591 = .3485*.4565`, therefore it seems that the restriction was imposed when fitting the model. Nevertheless, it's interesting to see that even if the restriction is not imposed when fitting the model, the result is not far from this restriction.
Comment: Opposed to the third model (`res.ur`), only two coefficients are estimated in the first model (`res`),
related to lag 1 and 12. The coefficient for lag 13 is not estimated, it is instead restricted to take the value of the product of the other two coefficients `prod(coefs)`.


How do I force the C# compiler to throw an exception when any math operation produces 'NaN' in .net 4?

Question: I have a long complicated source code and I need to find the exact location at which a variable value set to nan. So I need the compiler to throw an exception at that point.
This question has been asked before. I found the following answer as a good answer. This code works well in .net 3.5.But when I use .net 4, this solution doesn't work correctly. Even when I Enable "break when an exception is thrown" in the debugger, the exception can't be located in the code. Any idea?
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Runtime.InteropServices;

namespace ConsoleApplication2
{
  class Program
  {
    [System.Runtime.InteropServices.DllImport("msvcrt.dll")]
    public static extern uint _control87(uint a, uint b);

    [System.Runtime.InteropServices.DllImport("msvcrt.dll")]
    public static extern uint _clearfp();

    static void Main(string[] args)
    {
      float zero = 0.0f - args.Length; // Want 0.0f. Fool compiler...
      System.Console.WriteLine("zero = " + zero.ToString());

      // A NaN which does not throw exception
      float firstNaN = zero / 0.0f;
      System.Console.WriteLine("firstNaN= " + firstNaN.ToString());

      // Now turn on floating-point exceptions
      uint empty = 0;
      uint cw = _control87(empty, empty); // Debugger halts on this one and complains about false signature, but continue works.
      System.Console.WriteLine(cw.ToString());
      uint MCW_EM = 0x0008001f; // From float.h
      uint _EM_INVALID = 0x00000010; // From float.h (invalid corresponds to NaN
      // See http://www.fortran-2000.com/ArnaudRecipes/CompilerTricks.html#x86_FP

      cw &= ~(_EM_INVALID);
      _clearfp(); // Clear floating point error word.
      _control87(cw, MCW_EM); // Debugger halts on this one and complains about false signature, but continue works.
      System.Console.WriteLine(cw.ToString());

      // A NaN which does throw exception
      float secondNaN = 0;
      try
      {
        // Put as much code here as you like.
        // Enable "break when an exception is thrown" in the debugger
        // for system exceptions to get to the line where it is thrown
        // before catching it below.
        secondNaN = zero / 0.0f;
      }
      catch (System.Exception ex)
      {
        _clearfp(); // Clear floating point error word.
      }

      System.Console.WriteLine("secondNaN= " + secondNaN.ToString());
    }
  }
}

Comment: What is your question... compile time errors or swallowed runtime ones?
Comment: @SimonWhitehead I'll say he wants to catch signaling NaN as an exception.
Comment: If you want to catch *all* floating point exceptions, change `uint _EM_INVALID = 0x0000001F;` . See the documentation on the `_controlfp` command.  For example, to catch `Math.Log(0.0)` .
Answer: Found the problem.   Modify the calls to use _controlfp instead of _control87, and then change the DllImport to:
[System.Runtime.InteropServices.DllImport("msvcrt.dll", CallingConvention = System.Runtime.InteropServices.CallingConvention.Cdecl)]
public static extern uint _controlfp(uint a, uint b);

[System.Runtime.InteropServices.DllImport("msvcrt.dll", CallingConvention = System.Runtime.InteropServices.CallingConvention.Cdecl)]
public static extern uint _clearfp();

You have to explicitly tell .NET that your methods must be called with the Cdecl convention. Then it works. Next time please write the full exception... The full message should be:

A call to PInvoke function '_controlfp' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

Or you can cheat :-)
struct DoubleNoNan
{
    public readonly double Value;

    public DoubleNoNan(double value)
    {
        if (Double.IsNaN(value))
        {
            throw new Exception("NaN");
        }

        this.Value = value;
    }

    public static implicit operator double(DoubleNoNan value)
    {
        return value.Value;
    }

    public static implicit operator DoubleNoNan(double value)
    {
        return new DoubleNoNan(value);
    }

    public override bool Equals(object obj)
    {
        return this.Value.Equals(obj);
    }

    public override int GetHashCode()
    {
        return this.Value.GetHashCode();
    }

    public override string ToString()
    {
        return this.Value.ToString();
    }
}

and then:
DoubleNoNan dn = 0.0;
dn = / 0.0;

For the float, replace the word Double with Single and the word double with float. Operators will be provided by the implicit casts.
Clearly it has its limits (it can't be used in ref/out expressions, and many others)
Your code seems to work correctly... I've checked it at 32 and 64 bits
Comment: Thank you for your answers. Your first solution solves half of the problem but I still see an stack overflow exception. when code reaches to where I hope the nan exception occurs, I see : "An unhandled exception of type 'System.StackOverflowException' occurred in mscorlib.dll" .
Comment: @Soraya Then there is a StackOverflow :-) Look at the call stack (Debug->Windows->Call Stack)
Comment: unfortunately call stack can't help me. "[External Code]" is the content of my call stack :D
Comment: @Soraya Only "External Code"? You can't scroll down or up?
Comment: yes. nothing more than "External Code". Did the code runs correctly on your computer after replacing the dlls with your new code and the exact place of nan exception was located?
Comment: @Soraya I put a breakpoint on the `_clearfp()` and it worked (and stopped there). Tested on 32 and 64 bits in the debugger


Which prime factors can a number of the form $1212...2121$ have?

Question: The number $1212...2121$ with $n$ ones can be represented by $$z_n\ := \frac{2\times 100^n-35}{165}$$ or, substituting $m=2n-1$, by $$z_n\ :=\ \frac{4\times 10^m-7}{33}$$
It is easy to show that $2,3,5,7$ cannot be a factor of such a number.
$11$ can be a factor, which can be shown by $11|121$. Obviously, greater
prime factors $p$ must fulfil the equation $$4\times 10^m\equiv 7\ (\ mod\ p\ )$$
with some odd number $m$ or equivalent the equation $$2\times 100^n\equiv 35\ (\ mod\ p\ )$$

Is there an easy criterion, for which primes $p$, there is an $n$ (or an odd $m$), such that the equation holds ?

A necessary condition that $p\ge13$ occurs, is that $70$ is a quadratic residue
modulo $p$. If $70$ is a quadratic residue modulo $p$ and $ord_{100}(p)=p-1$
additionally holds, $p$ will occur infinite many often.
Comment: What might help : $$z_n-1=\frac{2\times 100^n-200}{165}=\frac{200(100^{n-1}-1)}{165}=\frac{120(100^{n-1}-1)}{99}$$
Comment: Or may be you meant that $10$ should be primitive! That has the well known reinterpretation that the period of decimals in the expansion of $1/p$ has length $p-1$. Anyway, barring a mistake in my calculation/programming $70$ is a QR modulo a prime $p$, iff $p$ is congruent to one of $$\{1, 3, 9, 11, 17, 23, 27, 31, 33, 37, 51, 53, 61, 69, 73, 81, 83, 93, 97, 99, 101, 111, 121, 127\}$$ or
one of $$\{153, 159, 169, 179, 181, 183, 187, 197,
199, 207, 211, 219, 227, 229, 243, 247\}$$ or one of $$\{249, 253, 257, 263, 269, 271,
277, 279\}$$ modulo $280$ - 48 residue classes out of $\phi(280)=96$.
Comment: Since $100^n-1$ has "many" prime factors, the numbers $z_n$ tend to be prime, or at least to be coprime to, lets say $20!$. $z_n$ is prime for $n=4,6,22,70,314,700,799,990$ and PRP for $n=3905,7030$. Note that the number $314$, related to $\pi$, is member of the (probably) infinite sequence of $n's$, such that $z_n$ is prime.
Comment: In addition to the necessary condition that $70$ is a QR mod $p$ won't it suffice that $\operatorname{ord}_{\Bbb{Z}_p^*}(100)=(p-1)/2$? After all, $100$ is a QR itself, and cannot really be a primitive root, but generating all the QRs suffices. Not sure how useful that (sufficient) order criterion really is, because it comes uncomfortably close to the problematics of the open Artin's conjecture.
Answer: Not at first glance,
$$z_2=121=11^2$$
$$z_3=12121=17 \cdot 23 \cdot 31$$
$$z_4=1212121=1212121$$
$$z_5=121212121=83 \cdot 577 \cdot 2531$$
Then the next prime is $z_6$. I didn't find any other $z_n$ that are prime with $ n \lt 16$.
Organizing the primes found so far, up to $n=16$, we have,
$11$,$17$,$23$,$31$,$83$,$263$...
The Case of $13$
For some reason, $13$ was the first prime missing from the above list. Using the divisibility rule for $13$ given here. I guess one could prove that, $z_n \mod 13$ can have values of $[1,4,5]$ that repeat in that order and nothing else.
Conclusion
As mentioned in the Wikipedia article, the general test for divisibility of a number is not explicitly known. Since the sequence of numbers you present is no different from a collection of random numbers, with respect to prime factorization, there is no reason to expect a closed form or "easy" solution to the problem.
Comment: Indeed $13$ will never show up.
Comment: The numbers are far away from being random, even with respect to factorization (See my conditions for $p$ occuring as a factor).
Comment: Zach, I don't really agree with your conclusion. IMHO this question should be seen contrasting the well known fact that every prime $p$ other than $2$ or $5$ occurs as a factor of an integer with decimal expansion consisting of a string of $1$s.
Comment: @JyrkiLahtonen "Is there an easy criterion, for which primes $p$ , there is an $n$  (or an odd $m$ ), such that the equation holds?" I think you've misconstrued my statement a bit. My intent is to point out that there is *no* easy method to find all the primes that satisfy the equation. In fact, we don't even know all the primes, so the question is arguably ill-posed!


How to code in Perl CGI to include action with https, not http

Question: I have a server behind a load balancer. The LB translates out of https into http so the web server only sees http.
I have a perl script that uses CGI standard to generate an input form using
start_form( -action=>url())

Right now, the url in the HTML points to the http version of the url, not the https. What is the cleanest way to get it to always use https (if https is being used by the user)?
I've tried "-rewrite" as both 0 and 1 and that didn't make the difference.
Comment: Please see [my blog post](http://blogs.perl.org/users/grinnz/2018/11/modern-perl-cgi.html) and the responses for better options for creating CGI scripts than CGI.pm, if it is a case where you have such options. But to begin with, your load balancer should ideally be passing the header `X-Forwarded-Proto: https` for any of these solutions to use.
Answer: At least according to the documentation and the code of CGI.pm, url() and url( full => 1) should both return https:// if your CGI script was served from an https:// URL. If the load balancer strips https and does not provide the appropriate headers, you can set $ENV{HTTPS} to ON to fake that in your script yourself:
#!perl
use strict;
use warnings;

use CGI qw(url);

$ENV{HTTPS} = 'ON';
print url();

See also
CGI
Comment: I hope that it does not automatically use such headers, as this would be a possible vulnerability if the client can tell the code it's using a different protocol when there's no proxy setting the header! This is why you have to enable it with `MOJO_REVERSE_PROXY=1` or the hypnotoad `proxy` option in Mojolicious.


Advantage & disadvantage of PCA vs kernel PCA

Question: PCA is used for dimensional reduction. I learned today that PCA cannot be used for nonlinear data. When nonlinear, you have to use kernel PCA (KPCA).
It seems that since KPCA is more applicable to more variety of data, it seems like a superior method, other than its longer computation time. Is KPCA truly superior to PCA, or are there any practical disadvantages of using KPCA when data is linear?
It seems that if computational efficiency isn't that big of a concern, I can just almost always naively use KPCA.
Comment: Kernel PCA is much slower because you have to compute a kernel for each data point. For big $n$ that can take a bit.
Answer: Kernel PCA (kPCA) actually includes regular PCA as a special case--they're equivalent if the linear kernel is used. But, they have different properties in general. Here are some points of comparison:

Linear vs. nonlinear structure. kPCA can capture nonlinear structure in the data (if using a nonlinear kernel), whereas PCA cannot. This can be a big deal; if this capability is needed, PCA simply isn't an option and kPCA (or one of many other nonlinear dimensionality reduction techniques) must be used. If the data truly live on a linear manifold, kPCA can do no better than PCA. In fact, using a nonlinear kernel may give worse performance due to overfitting.
Interpretability. PCA provides a set of weights that define a linear mapping from the input space to the low-dimensional embedding space. These weights can provide useful information about the structure of the data. No such weights exist for kPCA with nonlinear kernels because the mapping is nonlinear. It's also nonparametric.
Inverse mapping. PCA provides an inverse mapping from the low-dimensional space back to the input space. So, input points can be approximately reconstructed from their low-dimensional images. kPCA doesn't inherently provide an inverse mapping, although it's possible to estimate one using additional methods (at the cost of extra complexity and computational resources).
Hyperparameters. Both methods require choosing the number of dimensions. kPCA also requires choosing the kernel function and any associated parameters (e.g. the bandwidth of an RBF kernel, or degree of a polynomial kernel). This choice (and the method/criterion employed to make it) is problem dependent. Typically, one needs to re-fit kPCA multiple times to compare different kernel/parameter choices.
Computational cost. PCA generally has lower memory and runtime requirements than kPCA, and can be scaled to massive datasets. Various strategies exist for scaling up kPCA, but this requires making approximations (e.g. see the Nystroem approximation).

As usual, the right tool for the job depends on the problem.


How to prove independence is not a transitive relation?

Question: Suppose $A$ and $B$ are independent, and $B$ and $C$ are independent.
Is $B$ independent of $A \cap C$?  Is $B$ independent of $A \cup C$? If so, prove. If not so, give a counterexample.
Could someone give some hints on how to prove them? I am quite confused about them.
Comment: The title seems to be misleadingly addressing people to answer to a different problem ("does $A$ independent on $B$ and $B$ independent on $C$ implies $A$ idependent on $C$?")
Answer: The probability of rain here (in Boston) is pretty close to independent of the probability of rain in north London and the rain in south London. But those two events are probably correlated.
Edit in response to comments. True enough - I answered the question in the title, not in the post. Fortunately, there are two correct answers. Here's another that may be useful.
Flip two coins. $A$ is first coin heads, $C$ is second coin heads and $B$ is that the coins agree. Then each event is independent of each of the other two and each has probability $1/2$.  $A \cap C$ say both coins are heads, so $B$ is true - hence not independent of $A \cap C$ . $A \cup C$ says there's at least one head, so two tails can't happen and the probability of $B$ is just $1/3$, so $B$ is not independent of $A \cup C$ .
In fact, in this example you can take any one of the three events and think of it as the middle one that the other two are independent of.
Comment: @MiloBrandt: Good point!  Though if the OP really doesn't mean to ask the question in the title, s/he should probably change the title.
Comment: This answer does not address the question at all. For instance, the question asks about the independence of three events, but there are not even three here.
Comment: @MiloBrandt: rain in boston, rain in north London, rain in south London -- three events.  Two pairs of them are independent, one pair is not.
Comment: @HughThomas Ah, sorry; I had misread. That's still not what the question asks though. It's not "If A is indepedent of B and B is independent of C, is A independent of C?" which seems addressed here. That's what the title has, but the body of the post says something else. For what is asked, you'd be wanting to know whether rain in Boston is independent of it raining in either/both South/North London.
Answer: Let $\Omega=\{1,2,3,4,5,6\}$ with $\mathbb P(E)=\frac{|E|}6$ for $E\subset\Omega$. Let $A=\{1,2,3\}$, $B=\{1,4\}$, and $C=\{1,5,6\}$. Then $$\mathbb P(A\cap B) = \mathbb P(A)\mathbb P(B) = \mathbb P(B\cap C) =\mathbb P(B)\mathbb P(C) = \frac16,$$ so $A$ and $B$ are independent and $B$ and $C$ are independent. But $$\mathbb P(B\cap(A\cap C)) = \frac16\ne \frac1{18}=\mathbb P(B)\mathbb P(A\cap C) $$ and $$\mathbb P(B\cap(A\cup C))=\frac16\ne \frac5{18}=\mathbb P(B)\mathbb P(A\cup C), $$ so $B$ is not independent of $A\cap C$ or $A\cup C$.
Comment: This case is great! Thanks!
Comment: @Aky Indeed, I fixed the typo.
Answer: Here is a visual counterexample:
Answer: I flip two coins. Let $A$ be the event that the first coin lands heads up, and let $B$ be the event the second coin lands heads up. Let $C$ be the same event as $A$!
$A$ and $B$ are independent, and $B$ and $C$ are independent, but $A$ and $C$ are as far from independent as it gets.
Comment: This is a good and an easy-to-remember example which I want to internalize for future reference; thank you for sharing.
Comment: That's an interesting case. I've never thought about that A could be the same case as C. Thanks~~~


CSS animations transitioning from behind other elements broken in Firefox

Question: Here's a simple example of what I mean.
HTML

CSS
.main-container {
    width: 100%;
    height: 200px;
    margin: 250px 0 0 0;
    position: relative;
    background-color: yellow;
}

.headtest {
    font-family: 'quicksand', helvetica;
    background-color: #a2aba2;
    width: 100%;
    height: 200px;
    position: absolute;
    top: 0;
    right: 0;
}

.ht-tx1 {
    width: 100%;
    height: 20px;
     text-align: center;
     background-color: #000;
     animation: test-ani1 2s forwards;
     position: absolute;
     top: 0;
     right: 0;
}

.ht-tx2 {
    width: 100%;
    height: 20px;
    text-align: center;
    background-color: #000;
    animation: test-ani2 2s forwards;
    position: absolute;
    bottom: 0;
    right: 0;
}

@keyframes test-ani1 {

    100% {
        transform: translateY(-20px);
    }
}

@keyframes test-ani2 {

    100% {
        transform: translateY(20px);
    }
}

-
If you view in Chrome, both black bars slide out perfectly. The one transitioning from behind, and the one in front.
If you view it in Firefox, the bar transitioning from behind is broken. It sometimes works, but mostly it ignores the slide animation and just appears at the end of the animation duration.
I've re-created this a number of times and it seems that items that transition from behind another element are broken in firefox.
I've tried using -moz- which doesn't work. IS there anything else you can think of?
I've tried it without the absolute positioning, with z-indexs. and nothing seems to work.
EDIT ----
I appreciate work-around ideas, but I'd really like to know the route cause of this if anyone knows?
Thanks very much.
Comment: It is a fact that FF does that, so is my answer to your satisfaction or you need something more?
Comment: Yours is a fix, but I'd like to understand the route cause it possible. I've updated my initial post to reflect this.
Comment: I google it in so many ways and came up empty, so most likely a bug. At the end of they day you hopefully have better use of a workaround than an official bug report, assuming you need to create the effect of course.
Answer: It seems Firefox is inconsistent when animate the transform property, and I can't say why (yet), most likely a bug though.
Here is 2 workarounds to achieve the same effect

.main-container {
    width: 100%;
    height: 200px;
    margin: 50px 0 0 0;
    position: relative;
    background-color: yellow;
}

.headtest {
    font-family: 'quicksand', helvetica;
    background-color: #a2aba2;
    width: 100%;
    height: 200px;
    position: absolute;
    top: 0;
    right: 0;
}

.ht-tx1 {
    width: 100%;
    height: 20px;
     text-align: center;
     background-color: #000;
     animation: test-ani1 2s forwards;
     position: absolute;
     top: 0;
     right: 0;
}

.ht-tx2 {
    width: 100%;
    height: 20px;
    text-align: center;
    background-color: #000;
    animation: test-ani2 2s forwards;
    position: absolute;
    bottom: 0;
    right: 0;
}

@keyframes test-ani1 {
    0% {
        transform: translateY(-1px);
    }
    0.1% {
        transform: translateY(0px);
    }
    100% {
        transform: translateY(-20px);
    }
}

@keyframes test-ani2 {
    100% {
        transform: translateY(20px);
    }
}

.main-container {
  width: 100%;
  height: 200px;
  margin: 50px 0 0 0;
  position: relative;
  background-color: yellow;
}

.headtest {
  font-family: 'quicksand', helvetica;
  background-color: #a2aba2;
  width: 100%;
  height: 200px;
  position: absolute;
  top: 0;
  right: 0;
}

.ht-tx1 {
  width: 100%;
  height: 20px;
  text-align: center;
  background-color: #000;
  animation: test-ani1 2s forwards;
  position: absolute;
  top: 0;
  right: 0;
}

.ht-tx2 {
  width: 100%;
  height: 0px;
  text-align: center;
  background-color: #000;
  animation: test-ani2 2s forwards;
  position: absolute;
  bottom: 0;
  right: 0;
}

@keyframes test-ani1 {
  100% {
    top: -20px;
  }
}

@keyframes test-ani2 {
  100% {
    height: 20px;
    bottom: -20px;
  }
}

Answer: The solution relies on the z-index property of your elements: if you don't specify it the elements lay out one on top of the others, following the flow of the HTML document, when their "position" is set to "absolute". So "ht-txt1" is underneath "headtest" and "ht-tx2" is on top of "headtest".
To correct this "ht-tx1" and "ht-tx2" should take a "z-index" value of -1, so they are hidden underneath "headtest".
As for FF compatibility you need to prefix your "transform" effect with -moz-, check http://caniuse.com/#feat=transforms2d for more details.
Here's the CSS style code:
    .main-container {
        width: 100%;
        height: 200px;
        margin: 250px 0 0 0;
        position: relative;
        background-color: yellow;
    }

    .headtest {
        font-family: 'quicksand', helvetica;
        background-color: #a2aba2;
        width: 100%;
        height: 200px;
        position: absolute;
        top: 0;
        right: 0;
    }

    .ht-tx1 {
        width: 100%;
        height: 20px;
         text-align: center;
         background-color: #000;
         animation: test-ani1 2s forwards;
         position: absolute;
         top: 0;
         right: 0;
         z-index: -1;
    }

    .ht-tx2 {
        width: 100%;
        height: 20px;
        text-align: center;
        background-color: #000;
        animation: test-ani2 2s forwards;
        position: absolute;
        bottom: 0;
        right: 0;
        z-index: -1;
    }

    @keyframes test-ani1 {

        100% {
            -ms-transform: translateY(-20px);
            -webkit-transform: translateY(-20px);
            -moz-transform: translateY(-20px);
            transform: translateY(-20px);
        }
    }

    @keyframes test-ani2 {

        100% {
            -ms-transform: translateY(20px);
            -webkit-transform: translateY(20px);
            -moz-transform: translateY(20px);
            transform: translateY(20px);
        }
    }

Comment: Did you test this in FF? ... I did and it still has the animation issue. Furthermore, FF dropped the `moz` prefix a few years ago.
Comment: You're right for the -moz- prefix part. Tested it straight on FF, not through jsfiddle and worked fine. Under jsfiddle the issue is not solved... Weird...Using FF 44.0.2
Comment: After looking at this again, I got the same behavior as @Zakalwe mentionned, it sometimes works. Looks like the browser processes the hidden element events in a lower priority than the visible ones. So your solution is a welcome workaround.


Ajax based commenting system

Question: In Drupal 7 I'm facing one problem, I'm going to implement a comment system in that way that it should work as chatting systems. I've a content type Article, at the end there is a comment box where users can leave comments.
If I add a comment to the article it should be showed in every user's browser if they are viewing it, and at the same time they can debate/talk about that article without refreshing the page.
It will be real time commenting.
Answer: We have solved this issue with two modules

Ajax Comments

Provides ajax comments to Drupal sites (commenting like a social networking sites: Facebook, Google+, vk.com etc).

Block Refresh

Block Refresh allows an administrator to configure a block to refresh it's content via 3 different methods: automatically via a timer (eg every 30 seconds) , manually via a 'refresh' link, and once on page load.

You can view the real sample at http://beinghuman.org/.
Answer: The Ajax Comments module AJAXifies comments - works well and no need to refresh page.
Ajax Comments

Provides ajax comments to Drupal sites (commenting like a social networking sites: Facebook, Google+, vk.com etc).
Answer: There is a patch in ajax comments module to allow for time-based refreshing, but it is not yet committed:
https://drupal.org/node/401288
That would allow the usage of just one module alone to achieve the desired functionality.
I wander how you embedded the comments in the block, as I tried both commentsblock and node_comment_block but then they both didn't refresh using block_refresh.

Ajax based commenting system