How to validate models beyond checking for overfitting

Question: I have an unusual problem, which is that my model is performing too well and I am struggling to trust it.
The data is a table of "snapshots" about moments in games of chess. For example, a game that lasted 20 minutes might have 4 observations in the data, corresponding to the state of play at 1, 6, 11 and 16 minutes into the game. Features are things like how many moves my player has made, how many pawns he's lost, whether he's lost his queen and so on. The target feature on each observation is whether or not this player eventually won the game, and that's what I want to predict.
I trained a model (a RandomForestClassfier) on a bunch of data. The outcome was a 0.929 Accuracy and 0.983 AUC score; that score is so high that I struggle to trust it, so i want to validate it as securely as possible.
Strangely, the model doesn't seem to be significantly overfitted:
rfc = RandomForestClassifier(n_jobs=6, max_depth=15)

x_train, x_test, y_train, y_test = train_test_split(
    df_full.drop(
        'Status',
        axis=1
    ),
    df_full.Status
)

rfc.fit(x_train, y_train)

print('Train Score: ',
      rfc.score(x_train, y_train),
      'Test Score: ',
      rfc.score(x_test, y_test),
      'Test AUC Score: ',
      roc_auc_score(y_test, rfc.predict_proba(x_test)[:,1])
     )

Train Score:  0.9350048879188833 Test Score:  0.929179472533281 Test AUC Score:  0.9832906048250383

It's performing slightly better on the training set (750k observations) compared to the testing set (250k), but only by about 0.005 accuracy score. I've also performed cross validation using StratifiedKFold(shuffle=True, n_splits=5) and gotten similar scores. Now the character of some features in the dataset are changing over time. For example for newer observations the "GameLength" feature is trending firmly downwards and the player is getting better and better, winning more often:

So I decided to test the possibility that I'm overfitting to the earlier observations some how by splitting a random sample out of the dataframe, then scoring the model against observations from the sample with CreatedDate < 2018-01-01 and against observations from the sample with CreatedDate >= 2018-01-01. This does seem to bear out that hypothesis:
df_s = df.sample(frac=0.25)
df = df.drop(df_s.index)
df_s_1 = df_s.loc[df_s.CreatedDate < '2018-01-01']
df_s_2 = df_s.loc[df_s.CreatedDate >= '2018-01-01']

rfc = RandomForestClassifier(n_jobs=6, max_depth=15)

x_train, x_test, y_train, y_test = train_test_split(
    df.drop(
        'Status',
        axis=1
    ),
    df.Status
)

rfc.fit(x_train, y_train)

print('Full Train Score: ',
      rfc.score(x_train, y_train),
      'Full Test Score: ',
      rfc.score(x_test, y_test),
      'Holdout Pre 2018 Score: ',
      rfc.score(df_s_1.drop('Status', axis=1), df_s_1.Status),
      'Holdout Post 2018 Score: ',
      rfc.score(df_s_2.drop('Status', axis=1), df_s_2.Status)
     )

Full Train Score:  0.9285909152912291
Full Test Score:  0.9207593281991532
Holdout Pre 2018 Score:  0.9304484856471416
Holdout Post 2018 Score:  0.889780437600061

We can see that the Post-2018 Score is significantly lower than the Pre-2018 score. This problem is not specific to the holdout data however. If I compare the prediction accuracy against the main set, they follow broadly equivalent patterns:

Which would seem to imply the model is reasonably well fit and that it simply predicts new observations less well; presumably because they're drowned out by the much larger number of earlier observations and their different characteristics. With that being the case, I retrained the model on just those observations later than 2018-01-01 to the following results:
df3 = df.loc[df.CreatedDate >= '2018-01-01'].drop(['score', 'Correct_Prediction'], axis=1)

x3_train, x3_test, y3_train, y3_test = train_test_split(df3.drop(['CreatedDate', 'Status', 'LifeBin'], axis=1), df3.Status)

rfc3 = RandomForestClassifier(max_depth=15, n_jobs=6)

rfc3.fit(x3_train, y3_train)

print('Train Score: ',
      rfc3.score(x3_train, y3_train),
      'Test Score: ',
      rfc3.score(x3_test, y3_test)
     )

Train Score:  0.9588592284513856 Test Test Score:  0.9382511713179874

There's now much less of a trend, and the model is back to its previous good level of performance. I'm out of ideas to poke holes in that score now, so my question is is there anything else I can do to test the validity of that score?
Answer: Could you please explain your model in more detail. The Python code obscures what's happening, since an X matrix and Y matrix tell us little about the actual model. What are your X's for each sample, and what is your Y -- at what point in time?
Is each observation a point in time and you can have multiple observations from a single game? In that case, the observations are not independent. (Not to mention that if you're not careful, you will be using information "from the future" in your model, which will cause it to perform unrealistically well.)
Is your model, in the end, identifying individual players or competitions and therefore simply predicting that good players (or players that play in certain venues) are more likely to win? Machine Learning will pretty much always find a signal: the question is what is the signal?
As an example of what I mean, I once heard a talk by an incredible data scientist who won a competition to predict the severity of cancer for patients based on MRI results. She looked back at the data after the fact and figured out that what her model had really done was to identify which cancer center the patient had gone to for their MRI -- which MRI machine they had used, actually -- and since different cancer centers tend to see patients with different levels of severity, that was what her model was predicting.
Comment: see my response to your comment but yes the observations are points in time, there are multiple observations from single games. I'd like ideally to retain that characteristic, as obviously performance in the game so far is likely to help.
Comment: Apologies sent the previous comment early; forgot to not press enter on Stack Overflow. I exclude information about the other players, except for features on how often my player has beaten this particular opponent in the past. Perhaps I'm misunderstanding; does the lack of independence of the data points invalidate what I'm attempting here entirely? The score for entirely new observations in the test set (I.E. for games which **do not** have an observation in the training set) is 0.85, which did not seem wholly unreasonable
Comment: I think you lead me down the right path here. Your comment about the independence of the observations reminded me I should be using `GroupKFold()` to take the slices for cross_validation, which I wasn't doing. When I do that the score drops significantly.


branch de github no git

Question: Eu fiz um fork de um projeto que estou fazendo em grupo, no Github pelas abas de branch eu fiz 3 branch novas, no meu repositório local no sistema operacional Windows 10 eu digito o comando git branch para visualizar os ramos e os ramos do Github não estão no repositório local, então fiz o comando git pull origin main para atualizar o repositório local e ele parece estar atualizado mas do mesmo modo não conseguir acessar as branch que existem no github no git, não ficou claro o que devo fazer preciso criar as branch pelo git no repositório local? eu desejo ter as mesmas branch do Github no git. Desde já agradeço pelas respostas.
Comment: Supondo que você tenha uma branch `feature-a` no repositório remoto e tentar ir para ela no seu repositório local com `git checkout feature-a`, você não consegue?
Comment: Não consigo Rafael, não consigo entender porque isso parece que as branchs que criei só existem no github, inclusive eu nomeei elas pelo tipo de desenvolvimento que vou fazer, um banco-de-dados, outra com php e a ultima com bootstrap, e reparei que você deu um nome de recurso, aproveitando isso devo fazer uma branch para um recurso ao invés de todo um desenvolvimento de diversos recursos em uma linguagem?
Comment: Estranho, a branch remota geralmente não aparece mesmo com `git branch` mas você pode dar um `git checkout` após o `git pull`. Sobre o outro assunto, o nome e processo para criação de branch é baseado em opinião e varia de acordo com o time que está trabalhando no projeto, com o tempo você pode aperfeiçoar isso para suas necessidades caso perceba que a ideia inicial não ficou legal, não pense demais sobre isso - você pode mudar de ideia a qualquer momento.
Comment: * branch            main       -> FETCH_HEAD
Already up to date. ele aparece isso quando faço o comando git pull origin main
Comment: Estou usando o comando git branch para tentar ver as branchs não sei se esse comando é o correto para esse propósito.
Answer: Se continuar tendo esse erro tente criar a branch local e aí vc sobe pro github.
git checkout -b nome-da-sua-branch

Com esse comando você cria e já dá checkout.
git push --set-upstream origin nome-da-sua-branch

E com esse comando vc sobe a branch pro github.
Comment: Sua resposta pode ser melhorada adicionando informações fundamentais. Basta [edit] para adicionar mais detalhes, como citações ou documentação para que outras pessoas possam confirmar que sua resposta está correta. Você pode encontrar mais informações sobre como escrever boas respostas [na Central de Ajuda](/help/how-to-answer).

branch de github no git


VB.net how to wait for user to select an Excel cell

Question: I have created a VSTO project in Visual Studio with a Ribbon and a Form. There is a button on this form which I want when a user clicks on it, it waits for user to select a single cell from the workbook which this add-in is used in it.
I am currently at designing stage and have not much code written. Any key ideas to achieving this? Should I use Application.SendKeys method? Any other/better ideas?
Thanks
My Ribbon Code so far:
Imports Microsoft.Office.Tools.Ribbon

Public Class Ribbon1

    Private Sub Ribbon1_Load(ByVal sender As System.Object, ByVal e
    As RibbonUIEventArgs) Handles MyBase.Load

    End Sub

    Private Sub Button1_Click(sender As Object, e As
        RibbonControlEventArgs) Handles Button1.Click
        Dim f As New Form1
        f.Show()
    End Sub

End Class

My Form Code so far:
Public Class Form1

    'Variables for my "Custom Vlookup Function"
    Dim RHeader As Excel.Range 'Reference Header
    Dim SRange As Excel.Range 'Seleceted Range
    Dim ColNo As Integer 'Column number ahead

    Dim xlApp As Excel.Application = Globals.ThisAddIn.Application
    Dim xlWB As Excel.Workbook =
    Globals.ThisAddIn.Application.ActiveWorkbook

    Private Sub Button2_Click(sender As Object, e As EventArgs)
        Handles Button2.Click

    End Sub
End Class

Answer: Wait for the Worksheet.SelectionChange event which is fired when the selection changes on a worksheet.
Comment: How should I integrate your suggestion with my VB.net project WinForm?
Comment: But in your post you've mentioned that a VSTO project was created in Visual Studio with a Ribbon and a Form.
Comment: Should I make a loop in my WinForm to catch the **Worksheet.SelectionChange** result? is _Worksheet.SelectionChange_ in a Workbook return anything or is it just Do something after an event occurs?
Comment: Anyway, your add-in can wait for the event and communicate with a standalone WinForms application using standard .net communication mechanisms like WCF and etc.
Comment: Users may click a button on the windows form application which can call the VSTO add-in to wait for the event in Excel. As soon as the event is fired your VSTO add-in can communicate back to the windows forms application.


sql insere o datetime errado na tabela

Question: eu to usando microsoft sql manager,e estava tentando inserir uma data na minha tabela,o problema é que ele está inserindo a data errada,eu faço o seguinte comando para a tabela do manager
<code>INSERT
INTO tblCliente /*minha tabela que eu criei*/
VALUES('Ze',1999-10-02,1) /*atributos da tabela:nome[nvarchar[50]],data[datatime],limiteCliente[decimal(18,2)]*/
</code>
porém ao clicar para rodar o codigo da tabela,acontece isso

alguém que possua experiência em sql poderia me dizer como resolver esse problema?
Comment: Faltam aspas na data
Answer: O que aconteceu é que o valor 1999-10-02 foi considerado como uma expressão numérica:
     1999 - 10 - 2 = 1987
E esse valor numérico foi acrescentado ao menor valor de data disponível para datetime, que é 1/1/1900. Algo semelhante a:
<code>-- código #3
SELECT dateadd(day, 1999-10-02, 0)
</code>
Como a segunda coluna está declarada como datetime, recomendo que informe o valor entre apóstrofos. Por exemplo:
<code>-- código #1
INSERT into tblCliente
   VALUES('Ze', '19991002', 1);
</code>
Dessa forma, sem hífens.
Pode também utilizar a função Convert, caso queira informar a data em outro formato. Por exemplo, para informar a data no formato dd/mm/aaaa:
<code>-- código #2
INSERT into tblCliente
   VALUES('Ze', convert(datetime, '2/10/1999', 103), 1);
</code>
Answer: Tente da seguinte forma
INSERT INTO tblClientes VALUES ('Zé', '1999-10-02', 1)
A data tem que ser inserido como string.

sql insere o datetime errado na tabela


Drupal permission table

Question: If u have seen drupal permission table, that is like
structure 1
pid rid    perm                   tid
1    1   view f1, edit f2          0
2    2   view f1                   0

what is good with  structure1?
why drupal doesn't use this structure?
table1

    fid      name
    1      view f1
    2      edit f2

table2

pid  rid  fid
1     1    1
2     1    2
3     2    1

Answer: Table 1 matches how it is in Drupal 6, table matches more or less how it is in Drupal 7.
So to answer your Question, the newest Drupal does indeed use a schema close to what you proposed.

Drupal permission table


angular-in-memory-web-api not finding created database

Question: I am building an Angular 2 application with a Node.js back-end. I am trying to use angular-in-memory-web-api to find a created database within a custom file "./in-memory-data-service," pretty much following the Tour of Heroes demo on Angular 2 except using Observables instead of Promises. However, I keep getting undefined returned in my console.
Here is my super-basic database:
in-memory-data.service.ts
import { InMemoryDbService } from 'angular-in-memory-web-api';

export class InMemoryDataService implements InMemoryDbService {
  createDb() {
      let comments = [
          {id: 1, author: "Katie", text: "Hows it going?"}
      ];
      return {comments};
  }
}

Here is my app.module.ts file. All my custom components and providers are declared.
/// <reference path="../node_modules/immutable/dist/immutable.d.ts" />
import { NgModule }               from "@angular/core";
import { BrowserModule }          from "@angular/platform-browser";
import { HttpModule, JsonpModule, XHRBackend } from "@angular/http";
import { FormsModule, ReactiveFormsModule }            from "@angular/forms";
import { ChartsModule }           from "ng2-charts/ng2-charts";
import { InMemoryWebApiModule, InMemoryBackendService }   from "angular-in-memory-web-api";
import { InMemoryDataService }    from "./in-memory-data.service";
// extra dependencies omitted

@NgModule({
  imports: [
    ...
    InMemoryWebApiModule.forRoot(InMemoryDataService),
    ...
  ],
  declarations: [
    ...
  ],
  providers: [
    ...
  ],
  bootstrap: [ AppComponent ]
})
export class AppModule { }

This is the relevant part of my comment.service.ts file which retrieves the created database. I suspect the private commentsUrl = "app/comments" could be the culprit, but I've changed it to many other addresses with a similar result.
import {Injectable} from "@angular/core";
import {Http, Response, Headers, RequestOptions, XHRBackend} from "@angular/http";
import {Observable} from "rxjs/Rx";

export class Comment {
    constructor(
        public id: number,
        public author: string,
        public text: string
    ){}
}

@Injectable()
export class CommentService {
    constructor (private http: Http) {}

    private commentsUrl = "app/comments";

    getComments(): Observable<Comment[]> {
        return this.http.get(this.commentsUrl)
            .map((res: Response) => res.json())
            .catch((error:any) => Observable.throw(error.json().error || "Server error"));
    }

}

And this is the comment-list.component.ts file which collects the data from the service. The console.log(this.comments) in the loadComments() method is where one can see the undefined:
import { Component, OnInit, Input, OnChanges } from '@angular/core';
import { Observable } from 'rxjs/Observable';
import {Comment, CommentService} from './comment.service';
import { EmitterService } from './emitter.service';

@Component({
    selector: 'comment-list',
    templateUrl: "./app/comment-list.component.html"
})

export class CommentListComponent implements OnInit, OnChanges {
    constructor(private commentService: CommentService) {}

    comments: Comment[];

    @Input() listId: string;
    @Input() editId: string;

    loadComments(){
        this.commentService.getComments()
            .subscribe(
                comments => this.comments = comments,
                err => {
                    console.log(err);
                }
            );
            console.log(this.comments);
    }
    ngOnInit(){
        this.loadComments();
    }

    ngOnChanges(changes:any){
        EmitterService.get(this.listId).subscribe((comments:Comment[]) =>
            {this.comments = comments});
    }
}

Any help on this issue would be appreciated. I can supply extra code if needed.
Comment: Did you import `CommentService` into `app.module.ts`? Also, I don't see `CommentService` listed as a provider either at the module level nor at the component level.
Answer: Try changing res.json() for res.json().data in your service


In a factorial experimental design model, does it make sense to remove statistically insignificant factors?

Question: I am trying to analyze data from a factorial design experiment with three factors: Nitrogen, Phosphorus and CO2, with 2 levels each. I ran a three-way ANOVA in R:
<code>> fit = aov( Chl ~ (Nitrogen*Phosphorus*CO2), data=rr2)
> print(drop1(fit,~.,test="F"))
Single term deletions

Model:
Chl ~ (Nitrogen * Phosphorus * CO2)
                        Df Sum of Sq    RSS     AIC F value  Pr(>F)
<none>                               21.168 -203.21
Nitrogen                 1   0.88545 22.053 -200.13  4.8523 0.02959 *
Phosphorus               1   0.07483 21.242 -204.77  0.4101 0.52321
CO2                      1   0.01695 21.185 -205.11  0.0929 0.76110
Nitrogen:Phosphorus      1   0.85426 22.022 -200.30  4.6814 0.03254 *
Nitrogen:CO2             1   0.02174 21.189 -205.08  0.1192 0.73057
Phosphorus:CO2           1   0.00376 21.171 -205.19  0.0206 0.88608
Nitrogen:Phosphorus:CO2  1   0.20799 21.376 -204.00  1.1398 0.28791
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
</code>
Since CO2 appears to have no effect, does it make sense to simplify the model like so, discarding the CO2 variable:
<code>> fit = aov( Chl ~ (Nitrogen*Phosphorus), data=rr2)
> print(drop1(fit,~.,test="F"))
Single term deletions

Model:
Chl ~ (Nitrogen * Phosphorus)
                    Df Sum of Sq    RSS     AIC F value    Pr(>F)
<none>                           23.289 -199.37
Nitrogen             1   2.11369 25.402 -190.59 10.8912 0.0012725 **
Phosphorus           1   0.20896 23.498 -200.26  1.0767 0.3015159
Nitrogen:Phosphorus  1   3.11497 26.404 -185.80 16.0506 0.0001074 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
</code>
Answer: In my view the answer is NO and YES — depending on your aims:
No: If the aim of your study was to test the effect of all factors on Chl in a Null-Hypothesis-Significance-Testing (NHST) sense, i.e. if the aim was to confirm/reject any main or interaction effect of Nitrogen/Phosphorus/CO2 on Chl. In this case altering your analysis model post-hoc will likely lead to spurious significances due to "the garden of forking paths problem": If you allow post-hoc changes in analysis, you inflate Type I error, as each of the numerous  possible analysis choices states a multiple-comparison.
I can think of two cases where simplifying the model is ok:
Yes 1: If the aim of your second analysis is merely to complement your main analysis and to show that the results hold true even when simplifying the model  then that is ok (when clearly labeled as a post-hoc analysis).
Yes 2: If the purpose of your experiment was to explore potential relationships between Chl and Nitrogen/Phosphorus/CO2 to find the optimal model ("model building"), then systematically dropping or adding variables may make sense. However, you should conduct an independent study to test the resulting model before arriving at any final conclusions.
There is a similar post discussing this question for linear regression. All points raised there apply here alike since both linear regression/ANOVA/ANCOVA are basically variations of general linear model analysis.


Loaded styles and css but not displaying properly

Question: So I tried to import a web HTML CSS template into my self-made MVC.
When I open the template itself it works properly. But when I access it through my controller it can not connect to CSS files and does not display proper style. Only blank HTML without any styles.
This is how I am trying to access it
class NewsController {

public function actionIndex() {
    $newsList = array();
    $newsList = News::getNewsList();

    require_once 'views/index.php';

    return true;
}

This is code from HTML head where it links the CSS
   <link rel="stylesheet" href="css/animate.css" >
        <link rel="stylesheet" href="css/icomoon.css">
        <link rel="stylesheet" href="css/bootstrap.css">
        <link rel="stylesheet" href="css/magnific-popup.css">
        <link rel="stylesheet" href="css/style.css">

this is how folders are located(highlited the controller, and views folders is a full template ) folders tree:

EDIT
I checked with inspec all the elements are loaded but do not display any styles. Whats the matter?
Answer: You need to add './' at start of CSS paths. This allow browser to use absolute file path.
Comment: Did it this way
nothing changed
Answer: ./ is has nothing to do with an absolute file path. It just marks the current direcory.
If you want to use the absolute filepath, do it with php or the equivalent in your template language:
<link rel="stylesheet" href="<?php echo __ DIR __ . DIRECTORY_SEPARATOR . 'blogv4' . DIRECTORY_SEPARATOR . 'views' . DIRECTORY_SEPARATOR . 'css' . DIRECTORY_SEPARATOR . 'animate.css'; ?>" />

?>
EDIT: Due to the image of your folder tree you can write:
<link rel="stylesheet" href="blogv4/views/css/animate.css" />

You need to enter the path from your bootstrap file (e. g. index.php), which contains your header.
Comment: Please could you make an example
I am not so advanced in php :(
Comment: So i wrote All of this in the same place it was before, just change the route?
Did not help either :(
Comment: It was just a suggestion in case you want/need to use an absolute URL. If you want to stick to a relative URL in HTML, then please try:

The location of your controller is not relevant in this case.
Comment: @Asisito If I may: the absolute path using `__DIR__` is not a good idea, because the css/js resources are visible to anyone.
Comment: the CSS resources are visible to anyone else because their path is in your sourcecode. by clicking on the link - not depending if url is absolute or not - everyone can see.
Comment: @Asisito I meant: no one must see an absolute path of the **file system**, like "C:\MyProjects\blogV4\views\..." (which is resolved by the magical `__DIR__`). One should use a **web** absolute path, like: "www.myblog.com/assets/css..." to access the css/js resources, or a **web** relative path.
Answer: Make sure the PHP script (in your case, the index.php handling the request) that's calling your Controllers lives in your document root where your assets are.
And, if I am the one, I'd write an helper function for handling my assets, like
<?php
    function asset($resource) {
        $link = preg_replace('/\//', DIRECTORY_SEPARATOR, $resource);
        return $link;
    }
?>

Like that, you can render the absolute path of where your assets are by adding the protocol and domain prefix in front on the return value.


AWS migrate vendor account to my organization

Question: Please consider that I'm new to AWS administration.
I have full access to my organisation's AWS account. One of our website(developed in Drupal) is developed by a vendor who hosts it in their AWS account(dev/prod env.). They have a master account and separate accounts for each clients.
My organisation wants to migrate the entire infrastructure in-house in our own AWS environment, which does seem to be a great idea. Meanwhile, I've found AWS Organisations and I know I can invite vendor to join our organisation and then we can restrict their access(only development access) and mange the website and respective infrastructure ourselves. But this means we will have the control of their root account, which won't happen as they have other client accounts as well.
In this scenario, please suggest the best way to get full access and control to only our account?
Answer: For the sake of clarity, I'm going to refer to the specific AWS account that the vendor has for your AWS resources as your client account.
You have two options:

Have the vendor transfer just your client account to your AWS Organization.
Use an IAM cross account role to assume a role in your client account.

Either of these options could be setup to allow you to administer the AWS resources that the vendor has in your client account.
Option 1: If your company should pay the AWS bill for your client account then transferring the account from the vendor's AWS org to your AWS org is most likely the best choice. The vendor does not need to make their AWS master account part of your org, only detach your specific client account from their AWS org and then accept a request for it to join your AWS org.
Option 2: If the vendor should pay the AWS bill for your client account then a cross account role is most likely the best choice. In this scenario the vendor will remain the ultimate authority over your client account. The cross account role can be granted any set of permissions up to and including administrator access to your client account.
Caveat: Your vendor may have a complex account structure setup for purposes such as centralized logging across all of its client accounts. Additional considerations would need to be factored in if this is the case.


If $f:\mathbb{R}\rightarrow \mathbb{{R}}$ and $f(x)$ satisfying $f(2x+3)+f(2x+5) = 2$ . The period of $f(x)$

Question: If $f:\mathbb{R}\rightarrow \mathbb{{R}}$ and $f(x)$ be a function satisfying $f(2x+3)+f(2x+5) = 2$. Then period
of function $f(x)$ is.
$\bf{My\; Solution::}$ Let $(2x+3) = t\;,$ Then equation is $f(t)+f(t+2) = 2$
Now Replace $t\rightarrow (t+2)\;,$ we get $f(t+2)+f(t+4) = 2$
So from these two equation, we get $f(t+4) = f(t)$
So period of function $f(t)$ is $=4$
But answer Given as $=2$  . But in Solution it is given as $ = 2$
I did not understand where I have done Wrong.
Help me
Thanks
Comment: @Goos not in x-space!
Comment: I think the OP is actually correct. Let's take an example. If we define $f(x) = \sin 2x$, then the period of $f(x)$ is $\pi$. But if we define $f(2x) = \sin 2x$, then the period of $f(x) = \sin x$ is $2\pi$, even if the period of $\sin 2x = f(2x)$ remains $\pi$. This is what's happening in this case. The period of $f(2x)$ is $2$, but the period of $f(x)$ is actually $4$.
Comment: Your answer $4$ is correct.  $2$ is wrong.
Comment: @Deepak Can you undelete your answer so I can upvote it?  You are correct.
Comment: @Goos To be perfectly honest, the reason I deleted my initial post is because I had made the same mistake that others had. When I thought a little harder, I realised that you (and the OP) were right. Anyway, I've edited my post with a fuller explanation, along with the analogy I gave in my comment above.
Answer: $\displaystyle f(2x+3) + f(2x+5) = 2$
Replace $x$ with $x+1$:
$\displaystyle f(2x+5) + f(2x+7) = 2$
Subtracting and rearranging, $\displaystyle f(2x+3) = f(2x+7)$
Hence $\displaystyle f(2x+3) = f((2x+3)+4)$
At this point, you can conclude that the period is $4$. If this is difficult to see, you can replace $(2x+3)$ with $t$, as the OP has done.
It might be tempting to do this:
$\displaystyle f(2(x+\frac{3}{2})) = f(2(x+\frac{7}{2}))$
and conclude that the period is $\displaystyle \frac{7}{2} - \frac{3}{2} = 2$.
But this would be a mistake. This ($2$) is the period of $f(2x)$. The period of $f(x) = 4$.
If you're still unconvinced, here's another example. If we define $f(x)=\sin 2x$, then the period of $f(x)$ is $\pi$. But if we define $f(2x)=\sin 2x$, then the period of $f(x)= \sin x$ is $2\pi$, even if the period of $\sin 2x=f(2x)$ remains $\pi$. This is what's happening in this case. The period of $f(2x)$ is $2$, but the period of $f(x)$ is actually $4$.
Answer: Period is usually defined as the smallest $p>0$ such that $f(x+p)=f(x)$.

$$\large\underline{\text{Counter-Example}}$$
Consider the function $$f(x)=1+\sin{\frac{3\pi x}{2}}$$
See for yourself that $$f(2x+3)+f(2x+5)=2$$
And that $$f(x+\frac{4}{3})=f(x).$$
By properties of $\sin$ function, it can be shown that $\frac{4}{3}$ is the period.

Conclusion
There is no point in arguing that this is the period or that is the period (unless you can show that it is the 'smallest'). None of the answers or comments have shown that either $2$ or $4$ is the smallest.

$$\underline{\text{Why 2 cannot be the period}}$$
By the way, we can prove that $2$ cannot be period of this function. That, if we assume $2$ is the period  then the given equation can be reduced to $$2f(2x+3)=2 \iff f(2x+3)=1$$ or simply $f$ is a constant function(which we don't consider periodic).
Comment: do you mean $\sin 3x\pi/2$?
Comment: Good point.  Certainly, however, the period is $4$ divided by some integer, and it is quite reasonable to assume the question is asking for the largest guaranteed period, i.e. $4$.
Comment: @DanZimm. Yes, that was a typo. Thanks for pointing out.
Answer: The function is $t$-periodic with period 4. However, as $t = 2x + 3$, this means that $f$ is $x$-periodic with period 2, since an increment of 2 to $x$ will give an increment of 4 to $t$.
Comment: Disagree, I think the OP is right. Please see my comment above.
Comment: This is absolutely incorrect.  A function is not "x-periodic" or "t-periodic", it is just periodic.  The function has nothing to do with the variable.  $f(2x + 3) + f(2x + 5) = 2$ is a *functional equation* which holds for all real numbers $x$, and expresses a property of the function $f$.  We conclude from the property that $f(t + 4) = f(t)$, or equivalently $f(x + 4) = f(x)$, i.e. $f$ is periodic of period $4$.  (It is of course true that if you set $g(x) = 2x + 3$, then $f(g(x))$ has period $2$.  But this is not the question at hand.)


Is there any elegent way to formally prove that the ring $\mathbb Z/(8)$ cannot be decomposed to a product of rings?

Question: So far, my idea is that:
As a additive group, $\mathbb Z/(8)$ is the cyclic group $\mathbb Z_8$. So to find two rings that give a product ring that is isomorphic to $\mathbb Z_8$. The order of the product ring must be 8.
We know that $|H*K| = {|H| \; |K|}$. As the factors of 8 are 1,2,4,8, and we cannot involve the trivial ring, we need to try groups of order 2 and order 4.
There is only one structure of group of order 2:$\mathbb Z_2$.
But there are two structures of group of order 4: $\mathbb Z_4$ and the Klein four group. Neither of these $2$ groups gives us a cyclic group of order $8$.
Hence the $\mathbb Z_8$ cannot be decomposed to the product of 2 rings.
But I find this method tedious...
Could someone give a simpler and nicer way to prove it? Thanks so much!
Comment: If you find this tedious, *get out while you still can.* It will probably only get more and more tedious in your opinion.
Comment: @MattSamuel So is that the only way to prove it? I am just searching for a nicer way if possible...
Comment: This is exactly the approach I would've taken.  Out of curiousity, how did you verify that $\mathbb{Z}_2 \times \mathbb{Z}_2 \times \mathbb{Z}_2 \ncong \mathbb{Z}_8$ and $\mathbb{Z}_2 \times \mathbb{Z}_4 \ncong \mathbb{Z}_8$ as additive groups?  There are definitely tedious (unnecessary) ways of going about this...
Comment: Calling an argument that can be completely done in 5 sentences tedious is quite extraordinary.
Comment: You can prove that there is no decomposition of the underlying additive group into a product of groups. Central idempotents are good for decompositions of rings into products of rings, but not really needed in this case.
Answer: My favorite way to do this is that ring decompositions of a ring correspond to central idempotents. Nontrivial decompositions correspond to nontrivial idempotents.
In this case, six trivial checks suffice to prove it can't be decomposed:
$$
2^2\equiv 4\\
3^2\equiv 1\\
4^2\equiv 0\\
5^2\equiv 1\\
6^2\equiv 4\\
7^2\equiv 1\\
$$
Or, if that's too ugly, think about what it would mean to be idempotent:
If $8$ divides $x-x^2 = x(1-x)$ would imply that $2$ divides one of $x$ or $1-x$. Of course it can't divide both, so once you decide which one is divisible by $2$, it is also divisible by $8$ (and hence is zero in $\mathbb Z/(8)$.) At that point you know one of $x$ and $1-x$ is zero and the other is $1$.
Another reason that this ring can't be decomposed is that it's a local ring, that is, it has a unique maximal ideal. Any ring with a nilpotent maximal ideal is local, and that's the case here.
However the check above works even for rings that aren't local, like integral domains. (And it is not always necessary to manually check every element as I did above.)
Answer: For this particular case, the order of an element of $\mathbb Z_2\oplus H$ it at most 4: $$4(0,x)=(0,4x)=(0,0)$$ and $$4(1,x)=(0,4x)=(0,0).$$
In $\mathbb Z_8$, the element 1 has order 8. So $\mathbb Z_8$ and $\mathbb Z_2\oplus H$ cannot be isomorphic.


How to determine the invariant factors of an abelian group of order $2^6 \cdot 3^5 \cdot 5^7$

Question: $$
A =
\mathbb{Z}/2\mathbb{Z} \oplus
\mathbb{Z}/2\mathbb{Z} \oplus
\mathbb{Z}/2\mathbb{Z} \oplus
\mathbb{Z}/2\mathbb{Z} \oplus
\mathbb{Z}/2^2\mathbb{Z} \oplus
\mathbb{Z}/3^2\mathbb{Z} \oplus
\mathbb{Z}/3^3\mathbb{Z} \oplus
\mathbb{Z}/5\mathbb{Z} \oplus
\mathbb{Z}/5^2\mathbb{Z} \oplus
\mathbb{Z}/5^2\mathbb{Z} \oplus
\mathbb{Z}/5^2\mathbb{Z}
$$
First of all, $A$ is isomorphic to an abelian group of order $121500000$.
Then, it can be split to an isomorphic group of
$\mathbb{Z}/64\mathbb{Z} \oplus \mathbb{Z}/243\mathbb{Z} \oplus \mathbb{Z}/78125\mathbb{Z}$.
But how should I proceed next?
Comment: @InanimateBeing: Hahaha! I know, right? I wasn't aiming for it when I hit around 950 days. Since then, I decided to make a note of it. It wasn't difficult. This site is fun for me and I learn a lot from it. A thousand days well spent, I say!
Comment: @Shaun  The OP has shared some thoughts, but unfortunately they are not correct!
Comment: @Shaun 1000+ days!!! Have you asked MSE for feature request of platinum badges?
Comment: In the original decomposition there are five factors of order a power of $2$, so there must be at least five invariant factors - in fact there are exactly five.
Comment: [Here's a MathJax tutorial](http://meta.math.stackexchange.com/questions/5020/mathjax-basic-tutorial-and-quick-reference) :)
Comment: What have you tried so far?
Comment: The OP has shared some thoughts, @CyclotomicField.
Comment: I see that a=2^3 * 3^2 * 5^3 divides b=2^3 * 3^3 * 5^4 where a*b=121500000, but we also have c=2^2 * 3^1 * 5^2 divides d=2^2 * 3^1 * 5^2 divides e=2^3 * 3^3 * 5^3 where cde=121500000. So it has two invariant factors??
Comment: Well, they don't have to be *right*, @DerekHolt . . .
Answer: There is a fact that serves to find them algorithmically.
Fact.
Let $G$ be a finitely generated group, then there exists $m\geq 0$, $p_1,\ldots,p_n\in \mathbb{Z}_{+}$, distinct primes and naturals
\begin{align}
 &\alpha_{11}\geq\cdots\geq\alpha_{1k_{1}}>0\\
&\vdots\\
&\alpha_{n1}\geq\cdots\geq \alpha_{nk_{n}}>0
\end{align}
such that
$$G\cong \mathbb{Z}^{(m)}\oplus\mathbb{Z}_{p_1^{\alpha_{11}}}\oplus\cdots\oplus\mathbb{Z}_{p_{1}^{\alpha_{1k_1}}}\oplus\cdots\oplus\mathbb{Z}_{p_{n}^{\alpha_{n1}}}\oplus\cdots \oplus\mathbb{Z}_{p_{n}^{\alpha_{nk_n}}}$$
Let $q_{i}=p_{1}^{\alpha_{1i}}p_{2}^{\alpha_{2i}}\cdots p_{n}^{\alpha_{ni}}$, these elements are called invariant factors, and satisfies that $q_{i}\mid q_{i-1}$ and
$$G\cong \mathbb{Z}^{(m)}\oplus \mathbb{Z}_{q_1}\oplus\cdots \mathbb{Z}_{q_n}$$
In your case $|G|=2^{6}\times 3^{5}\times 5^{7}$, computing the proper partitions you can verify that there are $1155$ finitely generated groups except for isomorphisms, in particular:
\begin{equation}
A\cong \mathbb{Z}_{2^2}\oplus\mathbb{Z}_{2}\oplus\mathbb{Z}_{2}\oplus\mathbb{Z}_{2}\oplus\mathbb{Z}_{2}\oplus\mathbb{Z}_{3^3}\oplus\mathbb{Z}_{3^2}\oplus\mathbb{Z}_{5^2}\oplus\mathbb{Z}_{5^2}\oplus\mathbb{Z}_{5}
\end{equation}
\begin{align}
p_{1}&=2:= 2 \quad 1 \quad  1 \quad 1 \quad 1\\
p_{2}&=3:= 3 \quad 2 \quad 0 \quad 0 \quad 0\\
p_{3}&=5:= 2 \quad 2 \quad 2 \quad 1 \quad 0
\end{align}
then, the invariant factors are
\begin{align}
 q_{1}&=2^3\times 3^3\times 5^2\\
q_{2}&=2^{1}\times 3^{2}\times 5^{2}\\
q_{3}&=2^{1}\times 5^{2}\\
q_{4}&=2\times 5\\
q_{5}&=2\\
\end{align}
Hence
\begin{equation}
A\cong \mathbb{Z}_{q_{1}}\oplus\mathbb{Z}_{q_{2}}\oplus\mathbb{Z}_{q_{3}}\oplus\mathbb{Z}_{q_{4}}\oplus\mathbb{Z}_{q_{5}}.
\end{equation}
Comment: Shouldn't the $n$ in $\Bbb Z^{(n)}$ be, say, $m$?
Comment: thanks you for the help! things are clear now
Comment: @Shaun It's correct
Comment: q3 should be 2*5*5 right?
Comment: @everwith, yes $q_{3}=2.5^{2}$


Counting edges in a specially defined graph

Question: $G$ has vertices $\{0, \ldots, pq - 1\}$, where $p, q$ are different primes. There is an edge between $x$ and $y$ if $p \mid x - y$ or $q \mid x - y$. How many edges does $G$ have?
I looked at the cases where $p=2$ and $q=3, 5, 7, 11, 13, 17, 23$ and the formula I got was $|E|=q^2$, but it doesn't apply to $p=3$. For $p = 3, q = 5$, there are $45$ edges.
How do I derive a closed form formula for this?
Comment: Maybe think about the case that the are vertices $\{0,...,pq-1,pq\}$ first.
Answer: In the set $\{0,\dots,pq-1\}$, there are $p$ elements in each equivalence class modulo $q$, and $q$ in each class modulo $p$.
Since there is an edge between $x$ and $y$ iff $x\cong y \pmod p$ of $x\cong y \pmod q$, each of the $p$ equivalence classes modulo $p$ contributes $\frac{q(q-1)}2$ edges, and each of the q equivalence classes modulo $q$ contributes $\frac{p(p-1)}2$ edges.
Therefore, overall, there are $\frac{pq(q-1) + pq(p-1)}2$ edges.
In the case $p=2$, this does in fact simplify to $q^2$, as you found.


How to remove reference from imported wikipedia table in R?

Question: I want to import table from Wikipedia, but it contains reference links in brackets.
I import this table from  https://en.wikipedia.org/wiki/Statistics_of_the_COVID-19_pandemic_in_Poland

My code:
library("rvest")
url <- "https://en.wikipedia.org/wiki/Statistics_of_the_COVID-19_pandemic_in_Poland"
xpathd <- "/html/body/div[3]/div[3]/div[5]/div[1]/table[2]"

dane <- url %>%
  read_html() %>%
  html_nodes(xpath = xpathd) %>%
  html_table()

dane <- dane[[1]]
head(dane)

When I use it I get data table with brackets with reference numbers:

How to remove it?
Answer: An easy way would be to use parse_number on the columns that you want to remove digits from.
library(dplyr)
new_data <- dane %>%
               mutate(across(`Confirmed daily[c]`:`Unofficial deaths daily[f]`,
                      readr::parse_number))

parse_number returns the first number from the string.
Answer: You can use gsub() to remove the pattern of reference parts.
library(dplyr)

dane %>%
  mutate(across(.fns = ~ gsub("\\[.*?\\]", "", .)))


Check the continuity at origin

Question: $$ f(x,y) = \begin{cases}
\frac{x^3 \cos\frac{1}{y}+y^3 \cos\frac{1}{x}}{x^2+y^2}, & x,y \neq (0,0)\\
0 , &\text{otherwise}
\end{cases}
$$
Now i write
$|f(x,y) - f(0,0)| = |\frac{x^3 \cos\frac{1}{y}+y^3 \cos\frac{1}{x}}{x^2+y^2}| < \frac {|x^3| + |y^3|}{x^2+y^2}$ . How should i proceed now ?
Comment: Your function should be defined for $x\neq 0$ and $y\neq 0$.
Answer: Hint:$$\frac{|x|^3+|y|^3}{x^2+y^2}=\frac{|x|^3}{x^2+y^2}+\frac{|y|^3}{x^2+y^2}\leq \frac{|x|^3}{x^2}+\frac{|y|^3}{y^2}=|x|+|y|$$
Comment: can we use polar corodinates to solve this
Comment: @Gathdi Sure, you have $\frac{|x|^3+|y|^3}{x^2+y^2}=r(|\cos^3\theta|+|\sin^3\theta|)\leq 2r$.
Comment: so we can say that r goes to 0 so the approximation goes to 0 and f is cts at origin ..is that right ?
Comment: @Gathdi Yes, what you are saying is $\lim_{(x,y)\to 0}f(x,y)=\lim_{r\to 0}f(x,y)=0=f(0,0)$. So $f$ is continuous at the origin.

Check the continuity at origin


Last 5 entries or more depending on where

Question: I want to run a query where the last 5 entries are always returned, which is simple: SELECT * FROM table ORDER BY id DESC LIMIT 5. But I want to get more than 5 if a certain condition applies, which by itself would simply be something like SELECT * FROM table WHERE field > value. Is there a way to do this as a single query or do I have to run 2?
Comment: What is the benefit of making it as a single query?
Comment: Nope. You will have a convoluted SQL instead of two simple queries.  In all probability the combined query will run longer.
Comment: With all due respect, you don't know the code. 1 SQL statement where I could just loop over all returned entries instead of having extra logic to process 2 sets while ignoring duplicates, merging, and re-sorting would greatly simplify my code, even if the SQL statement were more difficult.
Comment: You may want to clarify your question and provide sample data and the expected result.  It's not clear if you intend to use the two queries as alternatives or in combination.  I assumed the former.
Comment: Combination. I want the most recent 5 no matter what, in addition to any others that match certain conditions. There is an autonumber field named id for ordering. Just because the most recent 5 are chosen, doesn't mean they necessarily meet the conditions though
Comment: Then `UNION` should work.
Answer: Maybe with UNION?. Like:
(SELECT * FROM table ORDER BY id DESC LIMIT 5)
UNION
(SELECT * FROM table WHERE field > value)


Methods for storing metadata associated with individual files?

Question: Given a collection of files which will have associated metadata, what are the recommended methods for storing this metadata?
Some files formats support storing metadata internally (EXIF,ID3,etc), but not all file formats support this, so what are more general options?
Some of the metadata would almost certainly be unique (titles/descriptions/etc), whilst some would be repetitive to varying degrees (categories/tags/etc).
It may also be useful to group the metadata, if different types of attribute are required.
Ideally, solutions should cover concepts, rather than specific language implementations.
Answer: To store metadata in database has some advantages but main problem with database is that metadata are not directly connected to your data. It is more robust if metada stay with data - like special file in the directory or something like that.
Some filesystems offer special functionality that can be used for metadata - like NTFS Alternate streams. Unfortunately, this can be used for metadata storage in special cases only, because those streams can be easily lost when copying data to storage system that does not support it. I believe that linux filesystems have also similar storage mechanism.
Anyway, most common solutions are :

separate hidden file(s) (per directory) that hold metadata
some application use special hidden directory with metadata (like subversion, cvs etc).
or database (of various kinds) for all application specific metada - this database can be used also for caching purposes in most cases

IMO there is no general purpose solution. I would choose storage of metadata in hidden file (robustness) with use of the database for fast access and caching.
Answer: One option might be a relational database, structured like this:
FILE
f_id
f_location
f_title
f_description

ATTRIBUTE
a_id
a_label

VALUE
v_id
v_label

METADATA
md_file
md_attribute
md_value

This implementation has some unique information (title/description),
but is primarily targetted at repetitive groups of data.
For some requirements, other less generic tables may be more useful.

This has advantages of this being that relational databases are very common,
and obviously very good at handling relationships and storing lots of data.
However, for some uses a database server brings an overhead which might not be desirable.
Also, the database server is distinct from the files - they do not sit together, and require different methods of interaction.
Databases do not (easily) sit under version control - which may be a good or bad thing, depending on your point of view and specific needs.
Answer: I think the "solution" depends greatly upon what you're going to be doing with the metadata.
For example, almost all of the metadata we store (Multiple datasets of scientific data) are all chopped up and stored in a database. This allows us to create datasets to preserve the common metadata between the files (as you say, categories and tags) while we have file specific structures (title, start/stop time, min/max values etc.) While we could keep these in hidden files, we do a lot of searching and open our interface to outside consumers via web services.
If you're storing metadata that isn't going to be searched on, hidden files or a dedicated .xml file per "real" file isn't a bad route to take. It's readable by basically anything, can be converted to different formats easily, and won't be lost if you decide to change your storage mechanism.
Metadata should help you, not hinder you. I've seen (and been a part of) systems where metadata storage has become more burdensome than storing the actual data, and became a liability. Just keep in mind what you are trying to do with it, and don't over extend yourself with "what ifs."
Answer: Plain text has some obvious advantages over anything else. Something like
FileName = 'ferrari.gif'
Title = 'My brand new car'
Tags = 'cars', 'cool'
Related = 'michaelknight.mp3'

Picasa's Picasa.ini files are a good example for this kind of metadata. Also, instead of inventing your own format, XML might be worth considering. There are plenty of readily available DOM processors to deal with this format.
Then again, if the amount of files and relations between them is huge, databases may be better.
Comment: [There ain't no such thing as plaintext](http://www.joelonsoftware.com/articles/Unicode.html). In fact I am right now looking for a way to store text character set encoding as metadata about a file.
Comment: @MilindR You don't seem to understand the difference between physical and logical formats.
Answer: I would basically make a metadata DB which held this information:
RESOURCE_TABLE
  RESOURCE_ID
  RESOURCE_TYPE (folder, doctype, web link, other)
  RESOURCE_URL (any URL)
NOTES_TABLE
  NOTE_ID
  RESOURCE_NO
  RESOURCE_NOTE (long text)
TAGS_TABLE
  TAG_ID
  RESOURCE_NO
  TAG_TEXT
Then I would use the note field textual notes to the file/folder/resource. Choose if you would use 1:1 or 1:N for this.
The tags field I would use to store any number of searchable parameters like YEAR, PROJECT, and other values that will describe and group your content.
Then you could add tables for owner, stakeholders, and other organisation info etc.


Confusion about spark streaming's transform function

Question: I am a bit confused about the transform function of a DStream. For example, if I have the following.
val statusesSorted = statuses.transform(rdd => rdd.sortByKey())

Would the whole DStream be sorted by key or the individual RDDs inside the DStream would be sorted separately. If that is indeed the case, how can I sort keys of the whole DStream.
Comment: how should  sorting the entire Stream work? You can't know what elements will come next...
Comment: I mean for example when you use reduceByKeyAndWindow, you also specify the windows size. Is it possible to sort the whole window?
Answer: The transform function in Spark allows you to perform any Spark transformation on the RDDs within your DStream.
The map transform does a similar operation but on an element to element basis, whereas the transform operation on dstream allows you do the transformation on a complete RDD.
To answer your questions,

Would the whole DStream be sorted by key or the individual RDDs inside
  the DStream would be sorted separately.

It will sort the individual RDDs in your dstream.

If that is indeed the case, how can I sort keys of the whole DStream.

To answer this, understand that Spark processes one batch at a time and the records in a batch correspond to the RDDs. So sorting the records in a batch(i.e. an RDD) will make sense because they form the data for computation. Sorting a dstream is not logical.
Comment: Yes, that will be the output of your window operation.
Comment: OK, but when you use reduceByKeyAndWindow, you also specify the window size. For example you could be streaming at 1 second, but window is 60 seconds. Is it possible to sort data in that 60 seconds window. I would like to use such functionality because, for example, I want to list the top counts for a window.
Comment: OK, got it, In the reduceByKeyAndWindow's case, an RDD just become the one with 60 seconds interval of data.


I need to create substrings out of a main string with varying window length

Question: I have the following sequence
ABCDEFGHIJKLMNOPQRSTUVWXYZ

In this I have a substring JKLM.
I want to create sub strings of varying lengths. For example I want 6 to 12 sequence length on either side of JKLM
The answer is supposed to be like this
6 length
HIJKLM,
IJKLMN,
JKLMNO

7 length
GHIJKLM,
HIJKLMN,
IJKLMNO,
JKLMNOP

8 length..... and so on
I am very new to programming and it would be good if someone can provide me the source code in perl.
Comment: How do you want to specify the initial substring? By offset and length, or by content? And what about *4 length* and *5 length*, and do you want to go all the way up to *26 length*?
Answer: Use positive lookahead assertions to enable overlap between matches.
The following displays the results for strings of length 6 to 9, but this could easily be expanded:
use strict;
use warnings;

my $string = join '', 'A'..'Z';
my $search = 'JKLM';

for my $len (6..9) {
    my $dist = $len - length $search;
    while ($string =~ m/(?=.{0,$dist}\Q$search\E)(?=(.{$len}))/g) {
        print "$1\n";
    }
}

Outputs:
HIJKLM
IJKLMN
JKLMNO
GHIJKLM
HIJKLMN
IJKLMNO
JKLMNOP
FGHIJKLM
GHIJKLMN
HIJKLMNO
IJKLMNOP
JKLMNOPQ
EFGHIJKLM
FGHIJKLMN
GHIJKLMNO
HIJKLMNOP
IJKLMNOPQ
JKLMNOPQR

For less complex tools, it's also possible to just use index and substr to build this list of values.  In fact, if you are that new to programming, those are the tools that I'd advise that you learn first.
Answer: Something like this should suit you
use strict;
use warnings;

my $s = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';

for my $size (6, 7, 8, 13) {
  printf "%d length\n", $size;
  print "$_\n" for windows($s, 'JKLM', $size);
  print "\n";
}

sub windows {

  my ($str, $substr, $size) = @_;

  return unless $str =~ /\Q$substr\E/;

  my ($strlen, $substrlen) = map length, $str, $substr;
  return if $substrlen > $size;

  my $start = $-[0];
  my $end = $start + $substrlen;

  # Calculate the earliest offset so that the string contains
  # the whole window and the window contains the whole substring
  #
  my $wfirst = $end - $size;
  $wfirst = 0 if $wfirst < 0;

  # Calculate the latest offset so that the string contains
  # the whole window and the window contains the whole substring
  #
  my $wlast = $start + $size;
  $wlast = $strlen if $wlast > $strlen;
  $wlast -= $size;

  map { substr $str, $_, $size } $wfirst .. $wlast;
}

output
6 length
HIJKLM
IJKLMN
JKLMNO

7 length
GHIJKLM
HIJKLMN
IJKLMNO
JKLMNOP

8 length
FGHIJKLM
GHIJKLMN
HIJKLMNO
IJKLMNOP
JKLMNOPQ

13 length
ABCDEFGHIJKLM
BCDEFGHIJKLMN
CDEFGHIJKLMNO
DEFGHIJKLMNOP
EFGHIJKLMNOPQ
FGHIJKLMNOPQR
GHIJKLMNOPQRS
HIJKLMNOPQRST
IJKLMNOPQRSTU
JKLMNOPQRSTUV

Comment: Takes 50% of the time taken by Miller's solution for the usage pattern in Miller's answer. [Benchmark](http://pastebin.com/7LuaMwXe)


Image inside  tag

Question: I have a question about floating elements inside of
 tag.
I have following markup

2013 Toyota Tacoma

Price : 450000$

Year : 2013

more

In FF, IE works fine, but in Chrome list numeration floats image too.
How to fix it? Thanks
Answer: I would rewrite your example like this:

2013 Toyota Tacoma

Price : 450000$

Year : 2013

more

This is written with inline-style CSS attributes, which I usually tend to avoid, but as in your example, I wrote it like you did - inline.
Comment: That's not correct way

Image inside  tag


Which sport is played on this field?

Question: I have an extension in my web browser that brings up an aerial photo from different parts of the world each time I open a new tab. A couple of days ago it was an image from Tokyo (I think), with a kind of sports field that I haven't seen before. I'm not able to find that image now, as the browser extension only remembers the last 10 or so images.
Here's how I remember it:

It could have been a soccer field, if the penalty area didn't have rounded corners. Also, I am unable to say anything about the dimensions of the field.
What sport is played here?
Comment: Sure, but we don't know that until we identify it. I placed a "suitable" tag based on how we have handled such questions in the past.
Comment: Welcome to Sports SE. I tagged with `international-sports` as we have had "sport identification" questions tagged with this before ([example 1](https://sports.stackexchange.com/questions/3002/what-is-this-three-against-one-game-with-red-and-blue-balls), [example 2](https://sports.stackexchange.com/questions/870/identify-this-italian-sport)).
Comment: @MASTERMIND_ED: I see you added the `international-sports` tag. But could it be that it is a sport mostly played in Japan?
Answer: One place this type of field can be found is at the Tokyo Metropolitan Gymnasium, which includes a futsal court that is consistent with your findings.

Which sport is played on this field?

Futsal, a variant of football/fùtbol (also known as soccer in the US).
Comment: You are right! Now that I know I was looking at a futsal court, I was able to find back to the image I had been looking at. It was indeed the [Adidas Futsal Park](https://firstname.lastname@example.com,139.7005974,172m/data=!3m1!1e3!4m5!3m4!1s0x60188b57f2c00001:0x1f9914a8f3bf2609!8m2!3d35.6587073!4d139.7007901?hl=no).

Which sport is played on this field?


Android Studio: Set reminder for specific date and time

Question: I'm a student at high school who's dramatically underestimated how difficult it would be to wrap my head around android studio. After 3 months of struggling I've managed to pound out most of the issues, but I still have one major thing I need to complete:
I need to set a reminder for a user defined date and time that will appear as a notification or something. I've tried every tutorial out there but I don't really know what I'm doing. At this point, I just need someone to walk me through it step by step because I'm at my wits end and regretting ever trying to take this project on.
Any help at all would be appreciated.
Answer: Try this out:
final static int req1=1;
public String a = "0"; // initialize this globally at the top of your class.

private void setAlarm(Calendar target){
    Intent intent = new Intent(getBaseContext(), AlarmReceiver.class);
    PendingIntent pendingIntent = PendingIntent.getBroadcast(getBaseContext(), req1, intent, 0);
    AlarmManager alarmManager = (AlarmManager)getSystemService(Context.ALARM_SERVICE);
    alarmManager.set(AlarmManager.RTC_WAKEUP, target.getTimeInMillis(), pendingIntent);
    a ="1";
}

For calling this method:
Calendar cal = Calendar.getInstance();
cal.set(2016, 0, 23, 18, 5, 0);
setAlarm(cal);
if(a.equals("0")
{
  // do whatever you want to do.
}

Comment: Awesome, thanks for your reply. It's good to get some help! Just three questions: req1 shows up as an error, sorry if this is a dumb question, but what is is supposed to be? AlarmReciever also is an error, was I supposed to have done something else with it? Lastly, the setText shows up as an error.
Comment: Sorry I accidentally hit send without finishing the comment. I edited the comment with some questions
Comment: I have edited my answer pls check. Instead of AlarmReceiver replace with your class name.
Comment: Awesome you've been so helpful thanks, sorry for being a bother. One last thing: For the cal.set, what is the format for the time/date? I assume it's something like year/month/day/hour/minute/second? Is the hour in 24 hour time, or how is it done thanks.
Comment: This code will set the alarm/reminder to 23 January, 2016, 18:05:00.
Comment: Awesome it seems to be working fine now. I know this is getting old, but one last thing: How do I make the app do something when the alarm triggers?
Comment: `public String a = "0";` initialize this globally. And inside setAlarm method at the last line make a = "1"; and using this flag do what you want after alarm strikes. Pls see my edited answer.
Comment: You can even compare the system date and time and alarm set date and time. If they match then that is where the alarm will be triggered and you can do your functions there.


connection on tensor bundle

Question: I know that if $E$ and $F$ are two vector bundle with connection $\nabla^E$ and $\nabla^F$, then it is natural to define tensor connection $\nabla = \nabla^E\otimes 1 + 1 \otimes \nabla^F$ on $E \otimes F$.
By I have a stupid question: is is $\nabla^E \otimes 1$ also a connection?
It seems to satisfy the conditions: $\nabla(f\sigma_E\otimes \sigma_F) = \nabla^E(f\sigma_E)\otimes\sigma_F = (\nabla f) \sigma_E \otimes \sigma_F +f \nabla^E\sigma_E\otimes \sigma_F = \nabla f \sigma_E \otimes \sigma_F + f\nabla(\sigma_E \otimes \sigma_F)$.
I guess the reason is the above condition is true only if I put $f$ with $\sigma_E$; and this ambiguity is not desired. But I'm not sure.
Comment: I think $0$ is a connection.  So it looks like you are correct.
Comment: @Stephen: No, $0$ is not a connection. The Leibniz rule has to hold for multiplication by scalar functions.
Comment: @TedShifrin Yes, I missed that.  So in that case, the OP is probably wrong.
Answer: It can't be, because, on the one hand, for any vector field $X \in \Gamma(TX)$, you would have
$$
 (\nabla \otimes I)_X(f(\sigma_E \otimes \sigma_F)) = (\nabla \otimes I)_X((f \sigma_E) \otimes \sigma_F) = \nabla_X(f\sigma_E) \otimes \sigma_F = (X(f)\sigma_E + f\nabla_X\sigma_E) \otimes \sigma_F\\ = X(f)(\sigma_E \otimes \sigma_F) + f(\nabla_X\sigma_E \otimes \sigma_F),
$$
whilst on the other,
$$
 (\nabla \otimes I)_X(f(\sigma_E \otimes \sigma_F)) = (\nabla \otimes I)_X(\sigma_E \otimes (f\sigma_F)) = \nabla_x\sigma_E \otimes(f \sigma_F) = f(\nabla_X \sigma_E \otimes \sigma_F).
$$
You really do need an honest connection on $F$ to get a connection on $E \otimes F$. Indeed, even if $F$ is trivial, you need to take at least the flat connection $d$.
Answer: Only if $F$ is a trivial bundle. Note that $f(\sigma_E\otimes\sigma_F)=\sigma_E\otimes(f\sigma_F)$ as well!

connection on tensor bundle


Intermittent ssh_exchange_identification: Connection closed by remote host

Question: We encountered an issue where a series of git requests over ssh would sometimes fail with
ssh_exchange_identification: Connection closed by remote host
There are many examples on SE/SF of structural problems (tcp-wrappers, permission on key files).
Our problem was: What is a likely cause of intermittent connection failures with this message?
Answer: Our issue appeared to have been caused by a moderately high number of incoming requests.
As soon as the number of unauthenticated connections goes over the sshd:MaxStartUps parameter,
sshd starts rejecting those connections.
The solution lies in modifying MaxStartups in sshd_config
Comment: It can also indicate a brute force attack, please make use of /etc/hosts.{allow,deny} files to limit connections.                                                                       http://www.aboutlinux.info/2005/10/using-tcp-wrappers-to-secure-linux.html
Comment: @AndrewMcGlashan it depends on the definition of **moderately high number** in this context.  I think we had a Jenkins host checking out repositories from our git server, all within our own network.  This host could very well handle the bursts, so we increased that limit.


p value for monthly trend

Question: I have a distribution of values for each of 12 months.
When I draw the boxplots (one boxplot for each month) the median gradually increases, but the boxes overlap.
Which test do I have to use in order to tell whether this seeming increase is significant or not?
Also, how would I quantify the increase? e.g. if between January and Feb the median increased by 1, but the box plots for these two months largely overlap, is that even meaningful?
Comment: You would likely get a better answer if you told us more about your data - for example, are these the same units observed each month, or a new set of units each month? Are you interested primarily in whether the median is different, or the mean, or something else?
Answer: You could use time series regression, and test whether the slope/trend has a significant coefficient after accounting for months. This page might be a good place for more details.

p value for monthly trend


classical topology but with lattices

Question: I'm looking for a reference, if such a references exists.
So there are currently at least two approaches to topology.

The point-set or "classical" approach to topology, which concerns itself with ordered pairs $(X,\tau)$ called topological spaces.
The "pointless" approach to topology, which concerns itself with (particular kinds of) lattices $(\tau,\wedge,\vee)$ called frames. (For more information, see e.g. Wikipedia.)

I'm interested in a concept halfway between 1 and 2. We might call it "the classical approach, but with lattices."
In particular, rather than studying point-set topological spaces $(X,\tau)$, we concern ourselves with "lattice-theoretic" topological spaces $(P,\tau)$, where $P$ is a lattice that is isomorphic to a powerset lattice, and $\tau$ is a subset of $P$ that is closed with respect to arbitrary joins etc.
The main motivation: We may be able to weaken the requirement that $P$ needs to be isomorphic to a powerset, and thereby obtain a more general theory, which is still classical in flavor.
Has this idea been studied before? If so, a reference recommendation would be great.
Comment: ahhh, I see what you mean. Is the following the definition you have in mind: Suppose you have some functor F:Set-->CLat. Say that an F-topological space is a pair $(X,\tau)$ such that $X$ is a set and $\tau$ is a subset of $F(X)$, closed under arbitrary unions and finite meets. The case F(x)=powerset of X gives precisely ordinary topological spaces.
Comment: If you demand that your frame $\tau$ is embedded as a subframe of $P X$ for some set $X$, then you are asserting that $\tau$ actually is a topology on $X$. In effect, you are suggesting that we restrict the study of locales to those that actually come from topological spaces. In other words there is no added generality at all! So why not just study these things _qua_ topological spaces?
Comment: Can you perhaps elaborate on how what you are looking for is different to locale theory? The only way I can interpret the above is that $\tau$ is a subset of P that is closed under arbitrary joins and finite meets. Weakening P from being a powerset lattice to just a complete lattice gives you locales precisely. So, unless you have some special subclass of lattices in mind, your question is not precise enough.
Comment: @IttayWeiss You write, "The only way I can interpret the above is that $\tau$ is a subset of $P$ that is closed under arbitrary joins and finite meets." Yes this is precisely what I mean. This is different from a locale, which makes no mention of $P$ whatsoever.
Comment: To be honest, all that categorical stuff kind of defeats me... All I was thinking is that if $(X,\tau)$ is a topological space, then $\mathcal{P}X$ can be viewed as a lattice, so $(\mathcal{P}X,\tau)$ can be viewed as a "lattice-theoretic" topological space. Furthermore, we can forget about the fact that $\mathcal{P}X$ and $\tau$ are collections of sets. Who cares what they're collections of! What matters is how they relate. This is kind of like - who cares what the elements of a group are. Whether they're functions or matrices or whatever, doesn't matter. What matters is how they relate.
Comment: However, it's possible that what you're saying is more interesting than what I'm saying.
Comment: I don't know of any relevant references. It is an interesting exercise to see how far you can develop topology using lattice theory only. Good question!
Comment: But what is there to relate $\tau$ to $P$? As far as I can tell, $P$ is just some arbitrary lattice in which you can embed the frames/locales $(\tau,\wedge,\vee)$. This embedding cannot be unique (you can always add additional points to $P$ that is not in $\tau$ almost arbitrarily). Hence I don't expect that more can be said about $P$ then what can be said about $\tau$ in the theory of locales or frames.
Comment: We require that $\tau$ be closed with respect to finite meets, for example. What does "meet" mean? The notion of meet comes from $P$, not $\tau$. Whereas in the theory of locales, the notion of meet comes from $\tau$.
Comment: But $\tau$ is surely by definition a sublattice of $P$. How will you differentiate between, say, $(P,\tau)$ and $(P\cup \{\infty\},\tau)$ where $\infty \wedge p = \infty$ and $\infty \vee p = p$ for every $p\in P$?
Comment: I don't fully understand the problem. The "lattice-theoretic topological spaces" $(P,\tau)$ and $(P \cup \{\infty\},\tau)$ are different spaces. The notions of meet and join are different in these spaces. So even if $\tau$ is closed with respect to finite meets and arbitrary joins when the meanings of "meet" and "join" come from $P$, it needn't be closed in this way when the meanings of "meet" and "join" come from $P\cup\{\infty\}$. But, is this a problem?
Comment: By the definition I gave, the two will coincide. One can talk about whether it is worthwhile to carry extra useless baggage (things like $P\neq Q$ as lattices but their topologies $\tau = \sigma$ are isomorphic as lattices; of course, from the categorical point of view it seems that the indiscrete topology is useful, so I won't argue this point too far). But mainly I wanted to point out that perhaps your natural generalisation would require $P$ to be a bounded lattice and that $\tau$ contains $\sup P$ and $\inf P$ both, since those facts have generally been useful in topology.
Comment: Well if $P$ is isomorphic to a powerset, it will certainly be bounded. And if $\tau$ is closed with respect to finite meets and arbitrary joins, it will be closed with respect the empty meet and empty join.
Comment: Oh i see what youre saying. Yes, i agree.
Comment: I'm not suggesting anything, I'm just after a reference. You're right that unless the requirement that $P$ be isomorphic to a powerset is weakened, the concept is no more general than that of a classical topological space.
Answer: I do not have a precise reference. But this may give you something to look at and/or some people to ask.
If you had asked me the similar question about convexity instead of topology, I would've given you a positive answer. The classical point-set topology satisfies the following axioms on the topology of closed sets $\tau \subseteq \mathcal{P}(X)$:

$\tau \ni {\emptyset, X}$
for any finite subset $K\subseteq\tau$ we have $\cup K\in \tau$
for any subset $J \subseteq \tau$ we have $\cap J \in \tau$.

The collection of all convex subsets $C$ on an affine space $E$ satisfy a similar collection of properties:

$C \ni {\emptyset, X}$
for any subset $J \subseteq C$ we have $\cap J \in C$
(optional, depending on definition) for any directed (with respect to inclusion) subset $S\subseteq C$ we have $\cup S \in C$

So one sees that there are certain similarities between convex structures and topological structures.
Now, for convex structures, something like what you proposed has been developed. It is sometimes called "abstract convexity", and one formulation is something like this:

Defn: Let $(X,\leq)$ be a complete lattice. A convexity system $\mathcal{C} \subseteq X$ is a subset that is closed under infimum. The system is said to be inductive if it is in addition closed under directed supremums.

Note that $X$ does not have to be the powerset lattice for a set. It turns out that many facts of convex analysis can be reproduced in this more general context.  See, for example, Ivan Singer's Abstract Convex Analysis or MLJ van der Vel's Theory of Convex Structures.
Given the similarity between convex structures and topological ones (for example, the convex hull operator is almost a closure operator in the Kuratowski sense), maybe the sort of "topology as a subset of a complete lattice" point of view can be found linked to from the literature in abstract convexity, and maybe the experts in that area can point you in the right direction so-to-speak.
Comment: @user18921 that is more a matter of convention and brevity, which I am well aware of. For example in the more formal definition of the convexity system which I gave the membership requirement is conveniently omitted.
Comment: Cool I'll check it out. I've actually encountered this concept before, except defined in the less general "point-set" fashion (rather than a "lattice-theoretic" fashion). It goes under the name "aligned space," in Coppel's [Foundations of Convex Geometry](http://www.amazon.com/Foundations-Geometry-Australian-Mathematical-Society/dp/0521639700).
Comment: Also, I think a few of your definitions are slightly off. Like 1 should feature a superset symbol, not an ownership symbol.
Comment: @user18921 corrected. I was changing between two sets of notations and evidently didn't make everything line up. Thanks.
Comment: No worries. Also, and this is a technicality that EVERYONE get wrong, not just you, the collection of all closed sets in a topological space is not really closed under arbitrary intersections. In particular, the empty intersection gives the wrong answer - you wind up getting the universal class, rather than $X$. A little nicety of the lattice-theoretic approach is that this is automatically dealt with.
Comment: To put that less strongly, it's perhaps not "wrong," but at least in need of a mention. Taking a lattice theoretic approach, it's simply true without qualification. Also, let me just point out that $\tau \ni \emptyset$ (or, more precisely, $\tau \ni \bot$) is implied by closure with respect to joins, and $\tau \ni X$ (or, more precisely, $\tau \ni \top$) is implied by closure with respect to meets.
Comment: Certainly mention of $\tau \ni \emptyset$ and $\tau \ni X$ is a matter of convention. As for the "fluid" definition of empty intersection, I think it's rather problematic. For instance, if we're dealing with two topological spaces at the same time, with different underlying sets $X$ and $Y$, should we choose $\bigcap \emptyset = X$, or should we choose $\bigcap \emptyset = Y$? It's a little messy either way.
Comment: @user18921 What I meant is, one could say: "$X\in \tau$" and "closed under arbitrary nonempty intersections", or one could be more verbose and define closure under intersections to be "$\cap (K\cup\{X\}) \in \tau$". Everybody _knows_ that this is what is meant, and unless one actually works in foundations relatively little harm can come from an abuse of notation. I mean, next you will complain that $A^c$ is a bad notation for $X\setminus A$ because the set $X$ is not explicitly mentioned in $A$ (or for a less trivial example $S^\perp$ where $S\subseteq V$ an inner product space)....
Comment: Ha! Fair enough. I wan't really trying to say that anything is bad notation for anything else. I guess was trying to say: the definitions are SLIGHTLY asymmetric. When you ask yourself WHY??? you inevitably come to the conclusion that it's because you're using a powerset to "model" a lattice. The powerset is a means; the lattice is the end. Sorry, I wasn't trying to be insolent or argumentative or anything.
Comment: Ah, do you mean asymmetric as in the closed-set definition versus the open-set ones? Then I think I see better what you are getting at. Though I am not sure if one should blame the point-set formulation of topology, or the fact that foundations take the point of view that everything is a set.... In any case, I am relatively not interested in getting into foundational discussions. So this will probably be my last comment on the issue.
Comment: I think we're talking about the same thing. In the open-set definition of a topological space, we define that arbitrary unions of open sets are open. In the closed-set definition, we define that finite unions of closed sets are closed. But once we start talking about intersections, we're forever using the "non-empty" qualification. Coming at it from another angle, the meaning of "union" is consistent across contexts, whereas the meaning of "intersection" is slightly fluid, or as you put it, we really mean $\cap(K \cup\{X\})$. That's the asymmetry.
Comment: Yes I don't want to get into a foundational discussion either. Anyway, I appreciate your time.

classical topology but with lattices


Permission denied error in all.js

Question: I am using all.js for facebook implementation. I used a facebook button on signup page . by logging through  facebook i am getting user name and email of facebook user into the fields on sign up page.
It's working fine on Chrome, Safari and Firefox but its giving error on IE8.
the error is in all.js line 22
{FB.UIServer._loadedNodes[a.id]=b;if(a.params)b.fbCallID=a.id;

Please help me if any one knows why this problem is occuring .
Thanks
Udham
Comment: Tons of similar questions: [1](http://facebook.stackoverflow.com/questions/7343226/facebook-login-throws-permission-denied), [2](http://facebook.stackoverflow.com/questions/7346333/permission-denied-error-in-all-js), [3](http://facebook.stackoverflow.com/questions/6678376/facebook-connect-cant-login-in-using-ie-permission-denied), [4](http://facebook.stackoverflow.com/questions/6654892/facebook-connect-access-denied-on-line-22-all-js-in-ie7-8) — I've voted to close the others in an attempt to make this one the canonical question.
Comment: @apaidnerd From what I can tell... there are several reasons why you might get a permission denied error in the all.js file -- voting to close all but one is like voting to close all C questions that include a buffer overflow - the permission denied error is a symptom... not the actual bug
Comment: @Jiaaro - Your analogy is incorrect. The ones I voted on are related to one particular Facebook bug.
Answer: Try this out. It worked for me, may help some one.
FB.UIServer.setLoadedNode = function (a, b){FB.UIServer._loadedNodes[a.id] = b; }

Ideally attach debugger and see which method breaks up and hack it.
Comment: This should be marked as answer - it worked for me and a bunch of others
Answer: This is a known bug: http://bugs.developers.facebook.net/show_bug.cgi?id=19042
It is set as CLOSED FIXED, but many users are still reporting the error in the comments area.
A hack is suggested on comment #19:
FB.init({
....
....
});
FB.UIServer.setActiveNode = function(a,b){FB.UIServer._active[a.id]=b;} // IE hack to correct FB bug

It worked for some people. I must say it didn't work for me, but I thought it was worth to mention.
Comment: @apaidnerd the fact that this hack works for some and not others suggests that there are different underlying causes for why this error occurs... don't you think?
Comment: Update: Bug 19042 was updated on Facebook Bug Tracker to **STILL NOT FIXED**. So they're aware and hopefully working on it.
Comment: There are two Facebook bugs, [19042](http://bugs.developers.facebook.net/show_bug.cgi?id=19042) and [20168](http://bugs.developers.facebook.net/show_bug.cgi?id=20168).  Both have similar symptoms but different hacks to fix (one fixes setActiveNode, the other fixes setLoadedNode).
Answer: I just ran into this (or a similar problem). Mine was in all.js line 22, char 3160, right after document.documentElement.style.display='none'; It was a permission denied error.
In my case, it was because the channelUrl was using a different protocol (https) than my app's iframe was being loaded under (http). I tried //mydomain.com/channel.html, but that gave me a different error. I solved it by dynamically choosing http/https for channelUrl when generating the html.
Answer: I found a solution. Already posted an answer here https://stackoverflow.com/a/8504794/287604, but for the desperate the quick fix:
FB.init({
    appId:        'xxxxx',
    appSecret:    'xxxxxxxxx',
    status:        true
    cookie:        true
});

// this code solves the issue
FB.UIServer.setLoadedNode = function (a, b) {
FB.UIServer._loadedNodes[a.id] = b;
};

Answer: I am still struggling with this but an odd work around I just noticed is if when using fb:login-button when setting the show-faces attribute to "true" it suddenly works fine. I tried 2 side by side fb:login buttons and the button with show faces set to either false or not included would pass back the error every time.

Permission denied error in all.js


Averaging numbers comes out incorect

Question: I have tried looking this up on here and still haven't found the exact solution that I am looking for. The math keeps coming out incorrectly.
       //Local Constants
       int count = 0;          //Used to track the number of grades entered by the user

       //Local Variables
       double currentGrade = 0;   //User's current grade inputed
       double numberGrades;       //Total number of grades to be entered
       double totalGrade = 0;     //Total of all grades
       double gradeAverage;       //Average of all grades

       //Main Function
      //Ask the user for the amount of grades they would like to enter
       System.out.print ("How many grades would you like to enter? ");
       numberGrades = scan.nextInt();

      //If the user asks to enter 0 numbers, output an error
       while (numberGrades <= 0){
           System.out.print ("Please enter a valid number! ");
         numberGrades = scan.nextInt();
       }
      //Add the grades together as they are input by the user
        while (count < numberGrades){
          totalGrade += currentGrade;
          System.out.print ("Please enter your next grade: ");
          currentGrade = scan.nextInt();
        count++;
        }
      //Calculate and output the average to the user
        gradeAverage = (totalGrade / numberGrades);
        System.out.print ("\n");
        System.out.print ("The average of all grades is: " + gradeAverage);
  }

}
Thought this was fixed, but when I run it I'm still getting the same issue. The example I tried was to enter 3 numbers, each was 50. The average ended up being 33.0. I've run the debugger and it keeps giving me errors once the first number is entered. count, totalGrade and currentGrade all show as errors. No clue how to fix it.
Comment: Welcome to Stack Overflow!  It looks like you need to learn to use a debugger.  Please help yourself to some [complementary debugging techniques](https://ericlippert.com/2014/03/05/how-to-debug-small-programs/).  If you still have issues afterwards, please feel free to come back with more details.
Comment: @AndyTurner I think you tagged the wrong Joe, mate.
Comment: Just debug your code then you'll may notice why your programm doesn't actually care about your last input.
Comment: @JoeBurkhart two things: 1) you're missing out one of the grades (the last one); 2) you're using integer division.
Comment: @JoeC indeed! Retagged.
Answer: You are doing the sum before getting the input. You should do it after that.
Replace your while loop with following change:
        while (count < numberGrades){

          System.out.print ("Please enter your next grade: ");
          currentGrade = scan.nextInt();
          totalGrade += currentGrade;
        count++;
        }

Comment: Wow. Thank you. It sucks when it's something that simple that I just completely overlook after staring at the code for hours trying to move everything else around.

Averaging numbers comes out incorect


Private message prefill message body

Question: I'm using Drupal 7 Privatemsg module.
To prefill the message body, I use the following hook but don't get anything displayed.
function MyModule_privatemsg_message_view_alter(&$vars) {
  module_load_include('pages.inc','privatemsg');

  $vars['message']['body'] = "Dear Friend,\n you rock!";
}

Any help ?
Answer: One solution is to use the Prepopulate module and then customize the URL to send a privatemsg.  This works well if you are using Views to generate the links, for example.
Comment: I tried to use the URL but /message/new returns Access Denied even for admin

Private message prefill message body


Comparing the mean of two non-random sub-groups

Question: Students are assigned to group A or B based on two aspects: a grade from an exam and an evaluation of their teacher. Students with both a high grade and a positive evaluation get into group A, other students in group B. I know that the grade is much more influential than the teacher's evaluation.
I have the exam grades of 1,000 students. I also know whether students are assigned to group A or B.
I want to know whether the average grade of students in group A is higher than the average grade in group B.
I am thinking about this regression (which I believe is equivalent to a direct t-test):
$\text{Grade}_i=\alpha+\beta*\text{DumA}_i+\epsilon_i$
where $\text{Grade}_i$ is the grade of student i. $\text{DumA}_i$ equals 1 if student i is in group A and 0 otherwise.
Do students in group A have a higher grade if Alfa is significant according to a standard t-test? Note that students in group A tend to have a higher grade since assignment is mostly based on the grade. It seems to me that the dummy is endogenous and thus the estimate of Alfa is incorrect. Is my interpretation correct?
Does this mean that it is not possible to compare the mean of the two groups? Are there any alternatives?
Any thoughts are very much appreciated!
Answer: You can compare the means of the two groups using an ordinary t-test. But, if you want to generalize the results, it has to be to groups that are divided similarly to the ones in your sample.
You could also use your regression, but the means would be significantly different if $\beta_1$ is significant. $\alpha$ would be the mean of students in the group that was coded 0.
Comment: Dear Peter, Thank you very much for your helpful answer! Do you think that the endogeneity is not a problem? That is, the two groups of which I want to compare the mean grade are partly formed via the grades.
Comment: That depends on what you mean by a "problem". E.g. it is perfectly fine to (say) test whether jockeys and basketball players are the same height. But does it tell you anything useful when you find out that they are not? This is not a problem with the t-test, nor can it be solved by using another test. It's a question of whether it makes any sense to do the comparison.
Comment: Peter, thank you very much! Two points:
1) The comments of Peter imply that the size of the correlation coefficient and the p-value of the t-test on whether this coefficient differs from zero are valid in the regression above. Is the regression above only not valid if I claim causality between the assignment to a group and the grade?
Comment: 2) Basically, I would like to know whether the difference that I find between the mean of the grades in group A and group B is substantial. I  believe that by using the t-test, I define substantial as that the difference between the two means (which exists almost by definition) cannot be attributed to coincidence, where coincidence is defined as the possibility that the grades of students in group A are generated from the same distribution as the grades of students in group B. Is my application of the t-test sensible and is my interpretation correct?
Comment: 1) Causality has nothing to do with it. 2) Be careful about "substantial" vs. "significant". With a large enough N, very small differences can be significant; with a small N, even large differences may not be.


Asymptotic normality with weighted sum of objective function $\min_{x} \; f_n(x) + g_n(x)$

Question: Suppose

$f_n(x)$, $g_n(x)$ are convex functions w.r.t. $x$
the optimal point of the two problems $\min_x f_n(x)$
and $\min_x g_n(x)$ have asymptotic normality as $n \rightarrow \infty$
they converge to the same point $x_0$

$$
x_1 \overset{p}{\sim} N(x_0,\frac{1}{n}D_1), \quad where \quad x_1 = \arg\min_x f_n(x),
$$
$$
x_2 \overset{p}{\sim} N(x_0,\frac{1}{n}D_2), \quad where \quad x_2 = \arg\min_x g_n(x).
$$
Question:
Do we still have asymptotic normality for the following optimization problem
$$
\min_x \; f_n(x) + g_n(x) \;?
$$

Here is an example on how the asymptotic normality and $n$ appear.
Consider data pairs $(u_i,v_i)$, $i=1,\cdots,n$ randomly generated from a linear model
$$
u = k \cdot v + \epsilon , \quad where \quad \epsilon \sim N(0,1) .
$$
Note that the variance is fixed.
We can do least squares regression and obtain an estimator $k_1$. As $n \rightarrow \infty$, there is asymptotic normality.
We can also do least absolute deviation regression and obtain an estimator $k_2$. As $n \rightarrow \infty$, there is asymptotic normality.
$k_1$ and $k_2$ converges to the same true value $k$ as $n \rightarrow \infty$.
Consider the following problem
$$
\min_k \; \sum_{i=1}^n (u_i - kv_i)^2 + \sum_{i=1}^n |u_i - kv_i|.
$$
Comment: Although it's not clear exactly what you mean by "solutions to problems," it strikes me you might want to investigate functions like $h$ where $h(x)=-\exp(-x^2).$ If we set $f(x)=h(x+1)$ and $g(x)=h(x-1),$ we see each is globally convex and have global minima at $\pm 1.$ But $f+g$ has *two* global minima near $\pm 1$ separated by a local *maximum* at $0.$ Consequently, whatever you might mean by "asymptotic," it is evident solutions would be focused equally near $\pm 1$ and would avoid a neighborhood of $0,$ yielding a bimodal distribution and preventing them from being asymptotically normal.
Comment: Re the edit: you still need to explain how "asymptotic normality" arises.  This doesn't happen merely by optimizing functions.  What is your statistical model and what assumptions do you make about how these problems vary with $n$?
Comment: I tried to formula a general problem but it is still not clear. I guess it is easier to convey the idea by a simple problem like the linear regression.
Comment: @whuber your $h(x)=-exp(-x^2)$ isn't convex, at least in the sense I'm familiar with. The sum of convex functions **is** convex: https://en.wikipedia.org/wiki/Convex_function#Operations_that_preserve_convexity
Comment: @Thomas You're right--I apologize for mischaracterizing it.  (It is log convex.)  What matters is that it is smooth and has a unique local (therefore global) minimum in the interior of its domain.  With the changes to the question, though, this example is moot.
Answer: Ok, so your example has two convex functions. I'm going to divide each of them by $n$ so they are averages of $n$ random convex functions. I'm also going to write the argument as $\theta$, since it's a parameter to be estimated. I'm also going to assume iid data $(X,Y)$, which is stronger than necessary, but convenient.
Convexity implies that $h_n(\theta)=f_n(\theta)+g_n(\theta)$ is convex.  Also, since it is convex, the pointwise convergence (in probability) of $h_n(\theta)$ to a limit $h(\theta)$, which follows from a law of large numbers,  is uniform on compact sets. This is enough to ensure that the minimiser $\hat\theta_n$ converges in probability to the minimiser $\theta_0$ of $h(\theta)$. (Well, together with existence of some moments, which I'm usually happy to assume)
Asymptotic normality requires a bit of smoothness. The classical condition is on the third partial derivatives of the objective function, but that's too strong for LAD regression.  Write $h^*(\theta;x,y)$ for the function such that $h_n(\theta)=\frac{1}{n}\sum_i h^*(\theta; x_i,y_i)$ -- the single-observation objective function. Assuming independence of observations, it's enough (van der Vaart Asymptotic Statistics Theorem 5.23) that $h^*(\theta;X,Y)$ is differentiable at $\theta_0$ for almost every $(X,Y)$, that $h^*(\theta)$ is Lipschitz on a neighbourhood of $\theta_0$ (automatic for convex functions) and that the expected value $E_{X,Y}[h^*(\theta; X,Y)]$ has a second-order Taylor expansion at $\theta_0$. This does cover LAD regression (under some assumptions on $X$ and $Y$)
Now, if $f^*$ and $g^*$ (defined analogously to $h^*$) satisfy the smoothness conditions, then $h^*$ also will satisfy them, and $\hat\theta_n$ will be asymptotically Normal.
I don't know if it's automatically sufficient that $f^*$ and $g^*$ are consistent and asymptotically Normal (I suspect it would be possible to construct some pathological counterexample), but if you have a proof that the minimisers of $f_n$ and $g_n$ are asymptotically Normal, you probably have a proof of sufficient smoothness that translates to $h_n$ and thus a proof that the minimiser of $h_n$ is asymptotically Normal. And in particular it's true for LS and LAD regression.
There are, however, a couple of complications with this as an estimation technique. First, there's no guarantee that $f_n$ and $g_n$ are on remotely comparable scales, so a simple sum may well be a very suboptimal way to combine them.  Second, the combined function $h_n$ will have the same roughness as the rougher of the two functions near $\theta_0$ and the same long-range growth as the faster-growing of the two functions far from $\theta_0$. This is typically the opposite of what you want: eg, the Huber $M$-estimator is set up so its influence function looks like the mean when $\theta$ is near $\theta_0$ and like the median when $\theta$ is far from $\theta_0$.


Where do bitopological spaces naturally occur? Do they have applications?

Question: I am interested where bitopological spaces occur in various parts of mathematics (i.e., what are natural examples of bitopological spaces stemming from various areas of mathematics, not from the studying bitopological spaces for their own sake.)
I would also like to know where bitopological spaces have some applications in various parts in mathematics. I quote from the discussion which motivated me to ask about them (I am quoting Theo B.): "a reasonable definition of an application: a result that doesn't mention the objects of study in the statement but uses them in the proof" (edit T.B.: This is a paraphrase of Paul Balmer's strict applications mentioned by him e.g. in his ICM talk, p.2).
The discussion in the comments to Brian M. Scott's answer here motivated me to ask about bitopological spaces.

I am not sure to which extent my background is important, but I never studied bitopological spaces, although I have read two papers on quasi-metric spaces, which are a special case.
Comment: I don't know where they occur, but now they can get married here in NYC :).
Comment: The book Bitopological Spaces: Theory, Relations with Generalized Algebraic Structures and Applications
By B.P. Dvalishvili, Tbilisi, Georgia
http://www.elsevier.com/wps/find/bookdescription.cws_home/704996/description#description
http://books.google.com/books?id=Drp6v_Un0sMC&printsec=frontcover&dq=%22bitopological+spaces%22+applications&hl=en&ei=_R0tTsKXMsSq-Ab3ttHsDQ&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCgQ6AEwAA#v=onepage&q=applications&f=false
has a chapter called "Application of bitopologies". But it seems that I do not have sufficient background for going through them.
Comment: Many hyperspace topologies (e.g. Vietoris topology) have upper and lower part, so it is another situation where the same set comes with two topologies. However, I do not know whether they were studied as bitopologies or whether there are some applications of bitopological spaces in this area.
Comment: A related question was posted at MO: [Bitopological spaces and algebraic topology](http://mathoverflow.net/questions/100102/bitopological-spaces-and-algebraic-topology).
Answer: One of the situations where bitopological spaces occur naturally are asymmetric metric spaces or quasi-metric spaces. They are defined as metric spaces, but the symmetry in the definition of metric is omitted.
$$
\begin{gather*}
d(x,y)\ge 0\\
d(x,y)=0 \Rightarrow x=y\\
d(x,z)\le d(x,y)+d(y,z)
\end{gather*}
$$
We already had a discussion about quasi-metrics here.
Such spaces naturally bear two topologies: forward topology  $\tau_+$ generated by the sets
$$B^+(x,\varepsilon)=\{y\in X; d(x,y)<\varepsilon\}$$
backward topology $\tau_-$ generated by sets
$$B^-(x,\varepsilon)=\{y\in X; d(y,x)<\varepsilon\}$$
The papers

Wilson, On quasi-metric spaces American Journal of Mathematics 53 No. 3 (1931), 675–684.
J.C. Kelly, Bitopological spaces, Proc. London Math. Soc. (3) 13 (1963), 71–89, MR0143169.

appear frequently as refences in works on this topic. A natural generalization to quasi-uniform spaces has been studied, too.

As far as the applications of this concept are concerned, let's have a look what some authors publishing in this area can say:
Isaac Vikram Chenchiah, Marc Oliver Rieger, Johannes Zimmer: Gradient flows in asymmetric metric spaces Nonlinear Analysis: Theory, Methods & Applications; Volume 71, Issue 11, Pages 5820–5834
Not only applications in science and engineering suggest that
the symmetry requirement of a metric is often too restrictive; Gromov points out the limiting effects of this assumption [10, Introduction].
[10] M. Gromov, Metric structures for Riemannian and non-Riemannian spaces, in: Progress in Mathematics, vol. 152, Birkhäuser Boston Inc., Boston, MA, 1999, based on the 1981 French original [MR0682063 (85e:53051)], With appendices by M. Katz, P. Pansu and S. Semmes, Translated from the French by Sean Michael Bates.
J. Collins, J. Zimmer: An Asymmetric Arzela-Ascoli theorem,  Topology and its Applications Volume 154, Issue 11, 1 June 2007, Pages 2312–2322
In the realms of applied mathematics and materials science we find many recent applications of asymmetric metric
spaces; for example, in rate-independent models for plasticity [6], shape-memory alloys [8], and models for material failure [12].
[6] A. Mainik, A. Mielke, Existence results for energetic models for rate-independent systems, Calc. Var. Partial Differential Equations 22 (1) (2005) 73–99.
[8] A. Mielke, T. Roubíček, A rate-independent model for inelastic behavior of shape-memory alloys, Multiscale Model. Simul. 1 (4) (2003) 571–597 (electronic).
[12] M.O. Rieger, J. Zimmer, Young measure flow as a model for damage, Preprint 11/05, Bath Institute for Complex Systems, Bath, UK, 2005.
Answer: A situation where two
topologies on the same set occur naturally are epireflections and
coreflections in the category of topological spaces. In such cases,
one of the two topologies is finer than the other one.
Perhaps it's worth mentioning that this condition also appears in some papers on
quasi-metric spaces. Namely I mean the condition called "approximate
metric axiom" (AMA) in the paper
Santanu Bhunia and Pratulananda Das: Two valued measure and
summability of double sequences in asymmetric context, Acta Mathematica Hungarica, 130 (1–2), 167–187
 and without any name in
J. Collins, J. Zimmer: An Asymmetric Arzela-Ascoli theorem,  Topology and its Applications Volume 154, Issue 11, 1 June 2007, Pages 2312–2322
(AMA) implies $\tau_+\prec\tau_-$. I went briefly over these two
papers and I have the feeling that some of the results would still
hold when using the condition $\tau_+\prec\tau_-$ instead of (AMA).
In my opinion, if some facts
which hold in these setting could be stated and proven in a unifying
way using bitopological spaces such that one of the topologies is
finer than the other one, I would personally prefer such
formulation (even in a paper which deals with only one of these
settings). I do not know whether bitopological spaces with this property have been studied.