Scaling and Load testing Django Nginx and Gunicorn: 502 Server Error + Resource temporarily unavailable gunicorn.sock

Question: I'm doing load testing with locust to see if my server can handle 1500 users.
What I'm using: Django, Nginx, Gunicorn, Postgresql
My droplet: 24vCPUs, 128GB RAM, 25GB SSD
at about 1100 users I start getting the following errors in locust:
GET / HTTPError('502 Server Error: Bad Gateway for url: myurl.here ')
GET /aboutpage/     HTTPError('502 Server Error: Bad Gateway for url: myurl.here ')

In my nginx error.log I get the following errors:
2020/01/26 23:14:17 [error] 30465#30465: *167765 connect() to unix:/var/www/file/to/sock/gunicorn failed (11: Resource temporarily unavailable) while connecting to upstream, client: 8x.8x.1xx.3x, server: mysite.here, request: "GET // HTTP/1.1", upstream: "http://unix:/var/www/file/to/sock/gunicorn://", host: "mysite.here"

for some reason it tells me that my Resource is temporarily unavailable.
Here is my nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 2048;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        # Virtual Host Configs
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

And here is my server block configuration:

upstream mysite-production {
    server unix:/var/www/path/to/sock/gunicorn;
}
server {
    listen [::]:80;
    listen 80;
    server_name mysite.here;

    # set client body size to 100M #
    client_max_body_size 100M;

    location / {
      include proxy_params;
      proxy_pass http://unix:/var/www/path/to/sock/gunicorn;
      auth_basic "Restricted Content";
      auth_basic_user_file /etc/nginx/.htpasswd;
    }

    location /static/ {
        root /var/www/site/production/;
        expires 30d;
        add_header Vary Accept-Encoding;
        access_log off;
        gzip on;
        gzip_comp_level 6;
        gzip_vary on;
        gzip_types text/plain text/css application/json application/x-javascript application/javascript text/xml application/xml application/rss+xml text/javascript image/svg+xml application/vnd.ms-fontobject application/x-font-ttf font/opentype;
    }

    location /media/ {
        root /var/www/site/production/;
        expires 30d;
        add_header Vary Accept-Encoding;
        access_log off;
    }

}

Here is my gunicorn service file:
[Unit]
Description=mysite production daemon
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/path/to/site/src
ExecStart=/var/www/path/to/venv/bin/gunicorn  --workers=49 --bind unix:/var/www/path/to/sock/gunicorn --log-level DEBUG --log-file '/var/www/path/to/log/gunicorn.log' mysite.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

What could be causing this problem? We are predicting about 1500-2000 concurrent users that will be on the site at the same time.
Thank you for taking the time! I'm looking forward to your answers!
Comment: Can you post what logs says, `Resource temporarily unavailable` is generic statement reasons could be many such as memory, files, sockets etc.
Answer: When load testing you will always hit limits and bottlenecks.
In your case you need to increase in gunicorn configuration the backlog from the default value of 2048 to something that your server can handle.
Comment: Thank you for your answer, three years ago this was indeed the issue!


Imagem não está mudando quando clico (javascript)

Question: Estou fazendo um jogo da velha e adicionei um eventListenner à cada espaço, para que assim que eu clicasse, a imagem mudasse. Porém não está funcionando e não consigo entender o porquê.
Segue o jS,HTML,CSS e as imagens

const player1 ="X";
const player2 ="O";
var playTime = player1;
var gameOver = false;

updateDisplay();
initiateSpaces();

function updateDisplay(){

    if(gameOver){
        return;
    }

    if (playTime == player1)
    {
        var player =document.querySelectorAll("div.display img")[0];
        player.setAttribute("src", "./Assets/images/X.png");

    }
    else{
        var player = document.querySelectorAll("div.display img")[0];
        player.setAttribute("src","Assets/images/circle.png");
    }

}

function initiateSpaces(){
    var spaces = document.getElementsByClassName("space");

//Adicionando um EventListner para cada Espaço
    for (let index = 0; index < spaces.length; index++) {

        spaces[index].addEventListener("click",function(){
            if (gameOver){
                return;
            }
            //Adicionando a condição para o click

            if(this.getElementsByTagName("img").length ==0){
                if(playTime == player1){

                   this.innerHtml = ""
                   this.setAttribute("jogada",player1);
                   playTime = player2;

                }
                if(playTime ==player2){
                    this.innerHtml = ""
                    this.setAttribute("jogada",player2);
                    playTime = player1;
                }
                updateDisplay();
            }
        });

    }

}
.game{
    display: flex;
    justify-content: center;
    align-items: center;
    flex-direction: column;
}

.game > .table >.space{

    width: 80px;
    height: 65px;
    border:2px solid black;
    float: left;
    text-align:center;
    padding-top: 15px;
}

.table{
    width:calc(84*3px);
}

.display{

    margin-top:30px;
    margin-bottom:20px;
    display: flex;
    align-items: center;

}

.image{
    margin-right: 200px;
}

.game > .table > .a-line{
    border-top: none;
}

.game > .table > .c-line{
    border-bottom:none;
}

.game > .table > .column-1{
    border-left:none;
}

.game > .table > .column-3{
    border-right:none;
}
<!DOCTYPE html>

    <meta charset='utf-8'>
    <meta http-equiv='X-UA-Compatible' content='IE=edge'>
    <title>Page Title</title>
    <meta name='viewport' content='width=device-width, initial-scale=1'>
    <link rel='stylesheet' type='text/css' media='screen' href='./Assets/CSS/index.css'>

Vez do Jogador:

Answer: Robert, o erro está principalmente na propriedade innerHtml, na verdade é innerHTML, apenas essa questão de case sensitive.
Porém além disso, o seu if de jogador está gerando problemas, pois você troca para o jogador 2 e na sequência verifica se foi ele que jogou, então ali é melhor fazer um else if, caso contrário sempre será a vez do primeiro jogador.
Também mudei o endereço das imagens para as que você anexou, mas pode desconsidera essa parte, deixei assim apenas para ficar visualmente melhor na execução dentro do stackoverflow.

const player1 ="X";
const player2 ="O";
var playTime = player1;
var gameOver = false;

updateDisplay();
initiateSpaces();

function updateDisplay(){

    if(gameOver){
        return;
    }

    if (playTime == player1)
    {
        var player =document.querySelectorAll("div.display img")[0];
        player.setAttribute("src", "https://i.stack.imgur.com/Szfrh.png");

    }
    else{
        var player = document.querySelectorAll("div.display img")[0];
        player.setAttribute("src","https://i.stack.imgur.com/Y9poD.png");
    }

}

function initiateSpaces(){
    var spaces = document.getElementsByClassName("space");

//Adicionando um EventListner para cada Espaço
    for (let index = 0; index < spaces.length; index++) {

        spaces[index].addEventListener("click",function(){
            if (gameOver){
                return;
            }
            //Adicionando a condição para o click

            if(this.getElementsByTagName("img").length ==0){
                if(playTime == player1){

                   //Corrigi a propriedade aqui
                   this.innerHTML = ""
                   this.setAttribute("jogada",player1);
                   playTime = player2;

                }
                //Criei o else if para que seja efetuada a troca de jogadores
                else if(playTime ==player2){
                    //Corrigi a propriedade aqui
                    this.innerHTML = ""
                    this.setAttribute("jogada",player2);
                    playTime = player1;
                }
                updateDisplay();
            }
        });

    }

}
.game{
    display: flex;
    justify-content: center;
    align-items: center;
    flex-direction: column;
}

.game > .table >.space{

    width: 80px;
    height: 65px;
    border:2px solid black;
    float: left;
    text-align:center;
    padding-top: 15px;
}

.table{
    width:calc(84*3px);
}

.display{

    margin-top:30px;
    margin-bottom:20px;
    display: flex;
    align-items: center;

}

.image{
    margin-right: 200px;
}

.game > .table > .a-line{
    border-top: none;
}

.game > .table > .c-line{
    border-bottom:none;
}

.game > .table > .column-1{
    border-left:none;
}

.game > .table > .column-3{
    border-right:none;
}
<!DOCTYPE html>

    <meta charset='utf-8'>
    <meta http-equiv='X-UA-Compatible' content='IE=edge'>
    <title>Page Title</title>
    <meta name='viewport' content='width=device-width, initial-scale=1'>
    <link rel='stylesheet' type='text/css' media='screen' href='./Assets/CSS/index.css'>

Vez do Jogador:

O jogo ainda não tem um algorítimo de vencedor, isso eu não fiz :D


How would these nonchord tones from Beethoven's Pathétique Sonata best be analyzed?

Question:
Hi everyone.
I'm working on an analysis of Movement II from Beethoven's Pathétique Sonata.
In the measures shown in the image, I am trying to analyze the nonchord tones that occur in the chromatic descending lines.
For example, the first chord shown is B♭7. In that descending line, the first C can probably be labeled as an anticipation. But what about the second C? If that was a suspension, would it be labeled as something like 9-♮8? Is it better to simplify it and call each group of two of the same notes a passing tone?
Any insights would be appreciated. Thank you!
Answer: The opening E-flat (note that both hands are in the bass clef) is a 4–3 suspension held over from the prior measure. This resolves to a D, a chord tone of the B♭7.
The C and B♮ are both passing tones. I would argue that these are simply re-articulated passing tones, and that we don't need a separate name for their second appearance.
In any event, I would certainly hesitate to call either of those pitches a suspension; remember that true suspensions must first be a chord tone in the harmony immediately prior. In order for that C to be a suspension, we would need a chord immediately before it that included that C as a chord tone.
(The third measure is the same as the first, just using invertible counterpoint.)
As for the final measure, the F is a suspension (note that it's a consonant chord tone in the V7 chord immediately before the downbeat). But this suspension has an added chromatic passing tone; instead of resolving immediately to E♭ (as a 9–8 suspension here normally would), it passes through the F♭ first. We end, of course, with a chromatic neighbor D.
Answer: In measures 1,3,4 (as displayed, not counted from the beginning), there is a syncopated chain of suspensions which are also passing tones. To some extent, the chromatic descending melody can be analyzed as an object in itself. It looks like fourth species counterpoint. Analyzing as passing tones or suspensions (I think) may be too detailed to be useful and obscures the more important chromatic descent.


React Components Not Rendering

Question: I can't even get my root app component to render.  I load localhost and it loads index.html successfully with no js errors but nothing is rendered to the page.  I can't pinpoint the problem if there are no errors :).  I looked at my code and I don't know what the deal is.  I'm new to React so...could be a bunch of issues below, I don't know.
server.js
var express = require('express'),
    app = express();

app.set("view options", {layout: false});
app.use(express.static('.'));
app.set('views', __dirname + '/');
app.engine('html', require('ejs').renderFile);
app.set('view engine', 'html');

app.get('/', function (req, res) {
    res.render('index.html');
});

app.listen(4000, function () {
    console.log('Example app listening on port 4000!');
});

index.html
<!DOCTYPE html>

index.js
import React from 'react';
import { render } from 'react-dom';

import App from 'src/components/App';

const app = document.getElementById('app');
render(<App/>, app);

App.js
import React, { Component } from 'react';
import CompanyList from 'src/components/CompanyList';

class App extends Component {
    render() {
        return (

                <CompanyList />

        );
    }
}

export default App

CompanyList.js
import React, { Component } from 'react';

class CompanyList extends Component{
    render(){
        return (

No Companies Found

        );
    }
}
export default CompanyList

Answer: It doesn't work because of the type="text/babel" on you're script tag. This type was intended to use with the now deprecated babel browser to transpile ES6 and JSX in the browser.
You have to change the type to "text/javascript" to run the file. However, as the browser doesn't understand JSX, and some parts of ES6, this will mainly throw errors.
What you'll have to do in addition to make it work, is to use a build system, such as webpack, to transpile ES6 and JSX using babel, and bundle you're files, to something the browser can use. Try this guide.
If you just want to practice react without the hassle of a build system, you can use the React Base Fiddle.
Comment: but I should able to do this without a build still somehow
Comment: I'm going to be using browserify at first then move to webpack later..but I'll read that post
Comment: You'll have to use babel to transpile you're code either in webpack or browserify. Webpack is a bit easier to use.
Comment: lol some could debate the "webpack is easier"...maybe for transpiling but everyone else has found it a pain to figure out at first and get your head wrapped around the gigantic beast of a config framework.

React Components Not Rendering


Inaccessible cardinals under a Mahlo

Question: Is it true the statement that: if $\kappa$ is a Mahlo cardinal, then for all $\alpha<\kappa$ there is a cardinal $\lambda$ with $\alpha<\lambda<\kappa$ and $\lambda$ contains $\lambda$-many inaccessible cardinals?
It seems really obvious to me, but I tried to prove it and it feels a bit odd.
Comment: Please add context and attempts to this problem.
Answer: Yes, this is true. Here's one way to show this.
Lemma. If $\lambda$ is inaccessible, then $\lambda$ is the $\lambda$th inaccessible if and only if the set of inaccessible cardinals below $\lambda$ is unbounded if and only if $V_\lambda\models$"There is a proper class of inaccessible cardinals".
This lemma is easy enough to prove, so I will leave you to work it out.
Exercise. If $\kappa$ is Mahlo, then $\kappa$ is the $\kappa$th inaccessible cardinal.
Now, show that $C=\{\alpha<\kappa\mid (V_\alpha,\in)\prec (V_\kappa,\in)\}$ is a club, where $\prec$ means "elementary submodel" in the language of set theory. And now that "There is a proper class of inaccessible cardinals" is a first-order statement in the language of set theory.
Finally, since $\kappa$ is Mahlo, the inaccessible cardinals below it form a stationary set, so $\{\lambda\in C\mid\lambda\text{ is inaccessible}\}$ is a stationary set as well as the intersection of a club and a stationary set. In particular, it is unbounded. Now, apply the lemma.
Comment: If you liked the answer, it would be nice to mark it as correct @alvoi
Comment: @alvoi: Well, do you know that the increasing union of elementary submodels is an elementary submodel? That takes care of the closed part. To show that it's unbounded, simply iterate Löwenheim–Skolem by starting with some $V_{\alpha_0}$, taking the elementary submodel it generates, letting $\alpha_1$ be the supremum of the ordinals of that model, and adjoining $V_{\alpha_n}$ recursively. The limit will be of the form $V_\alpha$.
Comment: Thank you for the detailed answer! It was really helpful for me to understand better the objects in question. I think I proved everything you said, except for one thing: how can I prove that $C$ is a club? Maybe it's some general result that I don't recall, but it feels to me that my problem is concentrated in that point. Thank you a lot again!!
Comment: @AsafKaragila ooh yeah, you're right! I felt like Löwenheim-Skolem was needed but I didn't think of applying it that way! Thank you, you are always perfectly clear!

Inaccessible cardinals under a Mahlo


Vaadin maven-project stopped builds normally

Question: CVS rollbacks will not help. Builds failing on local machines and on jenkins server.
I use multimodule archetype (vaadin-archetype-application-multimodule, https://vaadin.com/maven#archetypes)
Steps to buid:
mvn clean install (work)
mvn project-production/pom.xml -Pproduction package (crash)
Vaadin version: 7.4.0
Error message:

[DEBUG] Execute command : C:\Program
  Files\Java\jdk1.7.0_51\jre\bin\java -Xmx512M -Xss1024k -classpath
  C:\Users\vbeglyanin\IdeaProjects\SberWeb\sber-web-production\target\classes;
  ... project specific stuff ...
  C:\Users\vbeglyanin.m2\repository\org\eclipse\jetty\jetty-servlets\8.1.12.v20130726\jetty-servlets-8.1.12.v20130726.jar;C:\Users\vbeglyanin.m2\repository\org\eclipse\jetty\jetty-client\8.1.12.v20130726\jetty-client-8.1.12.v20130726.jar;C:\Users\vbeglyanin.m2\repository\ant\ant-launcher\1.6.5\ant-launcher-1.6.5.jar
  com.google.gwt.dev.Compiler -logLevel INFO -style OBF -war
  C:\Users\vbeglyanin\IdeaProjects\SberWeb\sber-web-production\target\sber-web\VAADIN\widgetsets
  -localWorkers 4 -strict -XfragmentCount -1 -gen C:\Users\vbeglyanin\IdeaProjects\SberWeb\sber-web-production\target.generated
  ru.tsc.crs.web.MyAppWidgetset
  [ERROR] Unknown argument: -XfragmentCount
  [ERROR] Google Web Toolkit 2.3.0
  ... (list available arguments)

Empty vaadin project created from archetype builds normally.
.pom`s files not modified except add dependencies and repositories, but that was before there was a problem
Answer: According to this mailing list this happens, when you are using a different gwt version than expected. From your log we can see that you are using 2.3.0, the mailing list says you need at least 2.5.1.
You have probably added a new dependency which uses an older version of gwt than the archetype expects.
Run mvn -f project-production/pom.xml -Pproduction dependency:tree -Dverbose and grep for gwt.
Then look for something like
\- (com.google.gwt:gwt-dev:jar:2.5.1:compile - omitted for conflict with 2.3.0)

then you can see which dependency causes the conflict.


How to store ETH in the Smart Contract?

Question: I'm writing a LibraryPortal Smart Contract in which multiple users can rent their books to each other. So, in this contract the msg.value contain the total amount which is a combination of the Security Deposit & the Renting Rate.
What I have to do is transfer the Renting amount to the owner of the book instantly & store the remaining amount in the contract i.e., the Security Deposit.
If the Renter will not return the book with in the time specified then the security amount will be transfered to the Owner of the book, otherwise get returned to the Renter.
Here is my snippet:
function borrowBook(string _bName) payable returns (string){
    if(msg.sender != books[_bName].owner){
        if(books[_bName].available == true){
            if(getBalance()>=(books[_bName].amtSecurity + books[_bName].rate) ){
                books[_bName].borrower = msg.sender;
                books[_bName].available = false;
                books[_bName].owner.transfer(msg.value - books[_bName].amtSecurity);
                //  Code missing
                //  For storing
                //  ETH into the Contact
                return "Borrowed Succesful";
            }else{
                return "Insufficient Fund";
            }
        }else{
            return "Currently this Book is Not Available!";
        }
    }else{
        return "You cannot Borrow your own Book";
    }
}

Comment: Apart from the title i do not see a question in your post. If what you are asking is how to store the ether you get in the contract the the answer is pretty simple. You don't have to do anything to store the ether, the amount of a payable function is automatically transferred  to the contract. You can send what you want to the owner of the book and the rest will be stored in the contract.
Answer: You can achieve the result from something called as Escrow contract.
Following is the implementation of Escrow contract by open-zeppelin:
contract Escrow is Secondary {
  using SafeMath for uint256;

  event Deposited(address indexed payee, uint256 weiAmount);
  event Withdrawn(address indexed payee, uint256 weiAmount);

  mapping(address => uint256) private _deposits;

  function depositsOf(address payee) public view returns (uint256) {
    return _deposits[payee];
  }

  /**
  * @dev Stores the sent amount as credit to be withdrawn.
  * @param payee The destination address of the funds.
  */
  function deposit(address payee) public onlyPrimary payable {
    uint256 amount = msg.value;
    _deposits[payee] = _deposits[payee].add(amount);

    emit Deposited(payee, amount);
  }

  /**
  * @dev Withdraw accumulated balance for a payee.
  * @param payee The address whose funds will be withdrawn and transferred to.
  */
  function withdraw(address payee) public onlyPrimary {
    uint256 payment = _deposits[payee];

    _deposits[payee] = 0;

    payee.transfer(payment);

    emit Withdrawn(payee, payment);
  }
}

You can just instantiate the contract in your contract and forward the funds to the contract.
For full implementation of similar functionality have a look at the refundable crowdsale contract
Comment: yeah just write a method to send eth to your address
Comment: When contract owner unlocks its wallet and have access to the balance, I was wondering can the contract owner sent all those received amount to a different address? @Paradox
Answer: Thank you guys for answers but, later I came to know that the VALUE sent along with the transaction to the contract is stored in the Contract itself, & you can access that by using the address(this).balance which will always gives you the Balance available in that instance of the contract. Due to this you don't require any variable or other stuff to store ETHER in your contract.

How to store ETH in the Smart Contract?


Cassandra as Cache Front-end to RDBMS

Question: We are using Oracle RDBMS in our system. To reduce database load we plan to use a caching layer.
I am looking to see if we can use Apache Cassandra as a Caching Storage frontend to Oracle db.
From what I have looked so far Cassandra is more like a database with built-in caching features. So, using it as a caching layer to Oracle would be more like using another database. I feel it would be better to Cassandra itself as an alternative to Oracle and other RDBMS rather than using it along with Oracle.
Has anyone used Cassandra as a caching layer to RDBMS. But, I have not found any resources or examples for using it. If so can you help me on this.
Comment: If you need a cache why not just use a cache?
Comment: Yes. I am looking to use cassandra as a cache for the RDBMS. But not sure if that's possible.
Comment: Cassandra isn't a cache, it's a persisted log storage database. You probably want a Redis, or memcache
Answer: I'm not sure what you mean by a caching storage frontend.
Cassandra might be useful if you are expecting a large volume of writes that arrive at a rate faster than Oracle could handle.  Cassandra can handle a high volume of writes since it can scale by adding more nodes.
You could then do some kind of data analysis and reduction on the data in Cassandra before inserting the crunched data into Oracle.  You might then use Oracle for the tasks that suit it better such as financial reporting, ad hoc queries, etc.

Cassandra as Cache Front-end to RDBMS


For loop for read.table?

Question: I know this is an absolute mess and I apologise. In R I want to add a column to a set of large txt files and then make new txt files with the new columns. I don't get an error message but it takes a long time to run and eventually it gets killed. Is there a better way to run the below or maybe there is a bug still? I'm new to R so still don't know how it normally behaves.
files <- dir(".../rsid", pattern= glob2rx("final_rsid_chrom*.txt"))

rsid.db <- list()

for (i in files){
rsid.db[[i]]  <- lapply(i, read.table, header = T, sep=" ", stringsAsFactors = F)
rsid.db[[i]]$linkage.new <- apply(rsid_db[[i]], 1, create.alleles.col, "REF", "ALT")
write.table(rsid.db[[i]], "rsid_linkage_new[[i]].txt", quote = FALSE,row.names=FALSE)
}

Roland - I also ran the following separately and it has the same issue (i.e. taking a really long time).
 files <- dir("/home/sjk98/rds/rds-jmmh2-projects/iron/novel/novel_iron/cleaned/3_columns_std/rsid", pattern= glob2rx("final_rsid_chrom*.txt"))

 rsid.db <- list()

 rsid.db  <- lapply(files, read.table, header = T, sep=" ", stringsAsFactors = F)

 quit()

Comment: Please give us more background, eg. how many files you are reading, how many total MB/GB they consume on disk, how big is your RAM  size...
Comment: You need to benchmark/profile your code. There are several steps here that can be time consuming but we can't know what the bottleneck/s are. It could be file reading and writing (use `fread` and `fwrite` from package data.table instead). It could be the `create.alleles.col` function, which appears to be called extremely often (seems likely, nothing we can do easily, maybe use parallelization). It could even be `apply` itself (less likely).
Comment: Well, try the alternative I suggested. read.table is very slow and there are alternatives that are faster by orders of magnitude.
Comment: Also, you need to change the file name when writing to file. As it is, this will create exactly one file and overwrite it in each iteration. Use `sprintf` or something like that to create different file names.
Comment: Thank you Roland. Made some additions above
Answer: There were some bugs which I think are fixed in the below code. However, if create.alleles.col is the time consuming part, we can't help without knowing more about that function.
# someone correct me if I'm wrong, but I don't believe ".../" is a legit path;
# "../risd" looks for risd in the parent dir of the current working directory.
# I think you want a dot before the * in your regext as the original m* matches
# zero or more m characters
files <- dir("../rsid", pattern = "final_rsid_chrom.*\\.txt", full.names = TRUE)

for (i in files){
  # no need to call lapply on a single item; using a temporary variable
  # instead of a list will release the memory from each file instead of
  # building up in RAM if saved to a list
  rsid.tmp  <- read.table(i, header = TRUE, sep=" ", stringsAsFactors = FALSE)

  # if create.alleles.col is the time consuming part, we can't help without
  # knowing more about it
  rsid.tmp$linkage.new <- apply(rsid.tmp, 1, create.alleles.col, "REF", "ALT")

  # create a new file name with _update appended to the end, i.e., oldname_update.txt
  new_file <- sub("(\\.txt)$", "_updated\\1", i)
  write.table(rsid.tmp, new_file, quote = FALSE, row.names = FALSE)
}

Comment: I guess `create.alleles.col` is not the (dominant) time critical part (see the second part of the question where the code does not even use this function but is also working quite slow). I suppose "out-of-RAM" issues causing swapping of RAM on disk. It is IMHO generally not the best way to read in all files at once (if they are big) but process them sequentially (releasing the memory after processing each file), eg. re-using `rsid.tmp` instead of `rsid.db[[i]]` within the loop of your code
Comment: @R-Yoda: Oh right, I lost track of that second part. I'll update the answer to process sequentially without saving each file to a list.
Comment: Hello ngwalton and R Yoda,

Yes so .../rsid isn't the real file destination but the files that the script is meant to work on are in rsid folder.

It seems that ngwalton's code works! Thank you so much for that. Unfortunately the the job was killed due to consuming excessive CPU time on the login nodes. So I will have to run it again as a SLURM job. The code did get through Chromosome 1 out of 22. See below:
head -n3 final_rsid_chrom1_updated.txt
CHROM_POS ID REF ALT linkage.new
chr1_100000 rs1331956057 C T C_T
chr1_100000002 rs1462733795 C T C_T
chr1_100000004 rs373074226 G A A_G
Comment: @SpencerK: As @Roland suggested, you might consider using data.table and/or processing your files in parallel. It can be a pain to get your environment setup on the cluser, but changing the above code to use `foreach` should give you a big decrease in processing time if you have many files. You might want to run the analysis for a single file on SLURM to see how long it takes. Then you can decide if you want  to wait that long x n files or if you want to invest in getting parallel working.
Comment: @SpencerK: Also, if you're submitting your job to SLURM anyway, you could setup your job submission to loop over your files names and start a new job for each file. That way you could have a single R file that takes a file name as an input argument. This is a way of processing your files at the same time without using parallel in R. Not which will be more work on your setup.

For loop for read.table?


How is the integral calculator finding this $\int_{0}^{a}\frac{a^2-x^2}{(a^2+x^2)^2}dx$ while I get an undefined expression?

Question: My book's attempt:
$$\int_{0}^{a}\frac{a^2-x^2}{(a^2+x^2)^2}dx$$
Let us find the antiderivative first:
$$\int\frac{a^2-x^2}{(a^2+x^2)^2}dx$$
$$=\int\frac{x^2(\frac{a^2}{x^2}-1)}{(x(\frac{a^2}{x}+x))^2}dx\tag{1}$$
$$=\int\frac{x^2(\frac{a^2}{x^2}-1)}{x^2(\frac{a^2}{x}+x)^2}dx$$
$$=\int\frac{(\frac{a^2}{x^2}-1)}{(\frac{a^2}{x}+x)^2}dx\tag{2}$$
$$=-\int\frac{1}{(\frac{a^2}{x}+x)^2}d(\frac{a^2}{x}+x)$$
$$=\frac{1}{(\frac{a^2}{x}+x)}+C\tag{3}$$
$$=\frac{x}{a^2+x^2}+C\tag{4}$$

$$\left[\frac{1}{(\frac{a^2}{x}+x)}\right]_{0}^{a}=\text{undefined}$$
$$\left[\frac{x}{a^2+x^2}\right]_{0}^{a}=\frac{1}{2a}\tag{5}$$
$(5)$ gives the correct answer according to this integral-calculator.

Both $(3)$ and $(4)$ should be perfectly valid antiderivatives, as we have completed the integration process before $(3)$. However, I found that only $(4)$ is a usable antiderivative, and $(3)$ gives an undefined expression.
Question:

Why do we get an undefined expression using $(3)$?

My hunch:
My book's attempt of finding the antiderivative is wrong in that my book did $\frac{0}{0}=1$ multiple times throughout its working (for example, in lines $(1)$ and $(2)$). We should expect $(3)$ not to work since the process was wrong. When going from $(3)$ to $(4)$, the book did $\frac{0}{0}=1$ again. I am surprised that the book somehow managed to land at the correct antiderivative for our domain given that the process was wrong, and going from $(3)$ to $(4)$ is wrong as well.
Comment: When you reach $(4)$, even if you have questionable (or even false) steps in there, you can check that is it the antiderivative by differentiating.  Then proceed with $(4)$ and get the definite integral.
Answer: You get an error with (3) because evaluating the expression at 0 results into a division-by-zero error. Now the standard way to bypass this problem is to replace 0 by $\varepsilon$ and then let $\varepsilon$ goes to $0$. Note that both expressions (3) and (4) coincide when $0$ is replaced by $\varepsilon$. So they must have the same limit when $\varepsilon$ goes to zero and since expression (4) is continuous at zero, the limit is the value of (4) at zero.
Comment: The expression you called undefined  =$\frac{1}{2a}$, since at $x=0$, the denominator $=\infty$.
Answer: You could say:
$$\left[\cfrac{1}{\left(\frac{a^2}{x}+x\right)}\right]_0^a=\frac{1}{a+a}-\lim_{x\to0^+}\frac{1}{\frac{a^2}{x}+x}=\frac1{2a}$$
so you get the same result either way. We can make this assumption since the domain of the integral is $[0,a]$ and so we know that $0$ is approached from above.
Answer: What the author does in step $(1)$ (which needs to be reversed in step $(4)\,)$ is an informal 'trickery' that works because the improper integral that implicitly gets created is convergent.
This is somewhat analogous to saying that $\tan x$ and $\cot x$ are reciprocals of each other, even though technically this is false for multiples of $\displaystyle\frac\pi2,$ at which the same sort of division-by-zero magic is nonetheless acceptable.

Alternatively—quoting GEdgar above—When you reach $(4),$ even if you have questionable (or even false) steps in there, you can check that is it the antiderivative by differentiating. This is a good idea, and a valid proof-by-verification argument.
Alternatively, both cloudy's suggestion and the following method (the third step uses $\theta=\operatorname{arcsinh}\frac xa$ and integration by parts) also avoid having to divide by $0:$ \begin{align}&\int\frac{a^2-x^2}{(a^2+x^2)^2}\,\mathrm dx\\=&\int\frac{2 a^2}{(a^2 + x^2)^2} - \frac1{a^2 + x^2}\,\mathrm dx\\=&\frac2a\int{\operatorname{sech}^3\left(\operatorname{arcsinh}\frac xa\right)}\,\mathrm d\left(\operatorname{arcsinh}\frac xa\right)-\frac1a\arctan\frac xa\\=&\frac1a\left(\operatorname{sech}\theta\tanh\theta+\int\operatorname{sech}\theta\,\mathrm d\theta\right)-\frac1a\arctan\frac xa\\=&\frac1a\left(\sinh\theta\operatorname{sech}^2\theta+\arctan\frac xa\right)-\frac1a\arctan\frac xa+C\\=&\frac1a\left(\frac xa\times\frac1{1+\left(\frac xa\right)^2}\right)+C\\=&\frac x{a^2+x^2}+C.\end{align}
Alternatively, use $\theta=\arctan \frac xa$ or integrate by reduction formula.


Transparent Ethernet over GRE not forwarding traffic

Question: I am running on Linux 220.127.116.11 and have been trying to setup a transparent ethernet over GRE tunnel. The network topology used is as follows:

I want to bridge PC1 and PC2 using Transparent Ethernet over GRE. RouterA is connected to internet via ppp0 as RouterB. Both internet connections are PPPOE (PPPoEoA PVCs).
So i do:
In RouterA (linux 220.127.116.11):
ip link add testgre type gretap remote 220.127.116.11 local 184.108.40.206 ttl 255
brctl addif br0 testgre
ip link set testgre up

In RouterB (linux 188.8.131.52):
ip link add testgre type gretap remote 184.108.40.206 local 18.104.22.168 ttl 255
brctl addif br0 testgre
ip link set testgre up

Some commands output:
In RouterA (similar output in RouterB):
# ip link show testgre
36: testgre: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 5f:79:cd:4d:c6:57 brd c1:98:f3:ce:ff:ff

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.5f79cd4dc657       no              eth0.5
                                                        eth0.4
                                                        eth0.3
                                                        eth0.2
                                                        wlan0
                                                        testgre

Then i do PING request from PC1 to PC2:
ping 192.168.1.33

I run tcpdump in RouterA and I see the ARP Request frame over ppp0 (pcap file opened with Wirehark):

I run tcpdump in RouterB attached to ppp1 and i see that the ARP Request has arrived, but its not forwarded at L2 to PC2. I dont see the ARP request in PC2.
I try with statics ARP entries but the ICMP Request has the same problem, it is not forwarded to PC2.
There are not ebtables rules. Nor iptables rules (Default action is ACCEPT for ebtables and iptables). Should i add ppp0 to bridge interface or something like that? Any suggestions are welcomed.
Comment: since you see the arp request on router b, try to configure a loopback on routerb with ip address 192.168.1.33 to see if it will respond to the ARP
Comment: this link also might help : http://backreference.org/2013/07/23/gre-bridging-ipsec-and-nfqueue/
Comment: I have pinged from PC1 to routerB(192.168.1.2) while sniffing in ppp1 of routerB. I see the ARP Request incoming but i dont see the response. Seems that original frame is not decapsulated or packet is dropped some place in kernel. Where could i see inside kernel source the problem?I think `ip_gre.c` or `net/bridge/*.c` is a good place to start.
Comment: can you ping from RouteA, source 192.168.1.1 to destination 192.168.1.2 ?
Comment: I can ping to public WAN IP of RouterB from RouterA or from PC1 and vice versa and then i get the ICMP response (but in this case i see both ICMP request and response not travelling through GRE tunnel). If I ping from RouterA to destination 192.168.1.2 then i see the ARP request arriving through GRE tunnel but i dont get ICMP response. It seems source IP datagram is not decapsulated or maybe its dropped at some point in kernel.
Answer: The problem was related with the question I asked here on Server Fault IP/GRE compiled in kernel but gre0 interface doesnt exist.
I solved the first question by letting GRE demultiplexer driver in gre.c to register the GRE protocol handler in kernel. But I have found that when a GRE packet arrives GRE handler is not pointing to any ip_gre.c function (GRE demultiplexer driver is suposed to check GRE version in packet and pass the bucket to the right protocol handler but in my case it points NULL).
I commented the piece of code where gre.c registers GRE protocol handler and I have uncommented the piece of code where ip_gre.c registers GRE protocol handler. Now when a GRE packet arrives it is well decapsulated and commited to next layer, now i can ping from PC1 to PC2 without any problem. I think this issue is happening to many people in their embedded devices because I searched with google for the error message  https://www.google.com/search?q=ipgre+init%3A+can%27t+add+protocol when IP GRE driver can not start because GRE demultiplexer driver had yet registered a GRE handler and I have found several devices with this error.


How to run multiple commands with psexec?

Question: I need to run a command to uninstall forefront on some of our workstations and am having a bit of trouble formatting the command.
Here's what needs to happen (in multiple commands)
xcopy \\serverpath\Installer.exe C:\Windows
Install.exe /u /s
Del C:\Windows\Install.exe
I'm using psexec with a computer list but keep running into errors. This is what my command looks like as of right now.
psexec @ComputerList.txt -u domain\administrator cmd /c (xcopy "\\NetworkPath\Forefront Software\Install.exe" "C:\Windows" && Install.exe /u /s && del C:\Windows\Install.exe)

I think the spaces in the path name and quotes are messing things up...
EDIT:
Tried running it with a script with no luck. It looks like scepinstall.exe just hangs on the remote machine. I also tried using the exe that can be found in C:\Windows\ccmsetup\
Thanks for the help
Comment: Create a script pass the `-c` option so the script is copied to the remote system.
Comment: That would force us to run multiple commands, first copying the script to all the machines then running it. We are trying to do this in one seamless command since it will be deployed to 500+ machines. If that's the only way to do it I guess we have to then...
Comment: False, read the docs.  Passing the `-c` option causes psexec to copy the script as part of the task.  http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx
Comment: Tried doing it as a script, but it's now hanging on the uninstall. I also tried a script that would CD into Windows\CCMSetup\ and run the exe there, but it returns `script.bat exited on testmachine with error code -2147156221`
Answer: I don't think those parentheses are valid.  I think the /s switch in cmd.exe should preserve everything between the first and last ".  Does this work?
psexec @ComputerList.txt -u domain\administrator cmd /s /c "xcopy "\\NetworkPath\Forefront Software\Install.exe" "C:\Windows" && Install.exe /u /s && del C:\Windows\Install.exe"

I tested with the following simplified command and it worked (modified to /k so I could view output).
psexec \\remotemachine cmd /s /k "dir "c:\temp" && echo "hello""

Comment: It looks like this would be better suited doing multiple commands in a script, as it simplifies the process and psexec has arguments specifically for scripts. Thanks for the help!
Answer: Got it. The installer didn't like the path being hard coded in the script,
Changed from
C:\ccmsetup\scepinstall.exe /u /s

%windir%\ccmsetup\scepinstall.exe /u /s


Package 'effects': predictorEffects function customising the y axis with overprinting

Question: I have this predictorEffects finction, which results in overprinting of the labels on the y-axis, however, I have tried many things and they don't seem to be working. Here's the code I used:
pretty <- predictorEffects(newbinomial15)
plot(predictorEffects(newbinomial15,y=list(rotate=30)))
r <- plot(pretty,axes=list(y=list(rug=FALSE, rotate=30), lattice = list(layout=c(3,1))))

Any advice would be amazing! I really need to fix the labels!
Answer: One option would be to try recreating these same plots using the ggeffects package.  Here is a quick example of how you would do this for a binary logistic regression model without interactions among its predictors:
data(mtcars)

mtcars$cyl <- factor(mtcars$cyl)

m <- glm(am ~ wt + cyl, family = binomial(link="logit"), data = mtcars)

library(ggeffects)

g1 <- ggpredict(m, terms = "wt")
g2 <- ggpredict(m, terms = "cyl")

g1
g2

p1 <- plot(g1)
p2 <- plot(g2)

library(cowplot)

plot_grid(p1, p2)

The plots p1 and p2 can be customized further quite easily - see for example the ggeasy easy package, which makes customization of ggplots a breeze (https://cran.r-project.org/web/packages/ggeasy/vignettes/shortcuts.html).
For models with interaction terms, refer to the package vignette:
https://cran.r-project.org/web/packages/ggeffects/vignettes/ggeffects.html


Enabling HTTP2 for Spring Boot with Istio microservices

Question: We are planning to enable HTTP2 for microservice communication in a Kubernetes cluster. This microservices are built with Spring Boot 3.
Since HTTP2 requires HTTPS, the problem we wanted to avoid was configuring TLS on every Spring Boot application. Initially we thought of taking advantage of Istio's sidecar mTLS and enable HTTP2 directly there, making it transparent to the Spring Boot applications.
With it in place the communication would be:
MICROSERVICE1 ---[HTTPS(mTLS)-HTTP2]---> MICROSERVICE2(ISTIO SIDECAR CONTAINER ---[HTTP-HTTP1.1]---> SPRING BOOT)

Since the whole communication pipeline is not fully HTTP2, I'm not sure this would be enough or we would have to enable HTTP2 Spring Boot's side using server.http2.enabled=true and configure TLS.
Is there any way to enable HTTP2 transparently for the existing applications o there's no other way than configuring TLS directly on them?
Comment: HTTP/2 does not require TLS. Only browsers require TLS for HTTP/2. On backend systems you can typically use clear-text HTTP/2 (unless you have other requirements of confidentiality) to communicate among nodes.
Answer: For Using mTLS:
As you mentioned by adopting istio service mesh, Application / Micro Services need not to have any configuration w.r.t mTLS.  Istio will enable mTLS communication between pods.  For pod to pod mTLS communication following will be the high level flow.

Microservice container1 <--- http ---> IstioProxySidecar <--- https
---> IstioProxySidecar <--- http ---> Microservice container2.

To enable mTLS an PeerAuthentication, DestinationRule objects have to be created. For more details mTLS istio documentation can be referred.
For Using HTTP2:
Usually istio does automatic protocol selection.  If it needs to specifically done this can be achieved using below options.  For more details istio documentation can be referred

By the name of the port: name: [-].
By the appProtocol field: appProtocol: .

Both mTLS and http2 are separate and can be utilized respectively.


How to calculate the coefficients in an IBVP which is solved using separation of variables?

Question: So, I'm new to IBVP and I was looking at a basic heat problem, but I have a small question about the solution for the problem. I am attaching the problem and the solution.

In this problem $f(x) = sin (2πx) - 3 sin(6πx)$.
Based on the conditions we're given we know that $K = 1$ and $L = 1$
So our solution is

When I substitute in the solution I have $bn = \int_0^1 sin(2πx)sin(nπx) - 3sin(6πx) sin(nπx)dx$
We also know that
$\sum_1^\infty bn*sin(nπx) = sin (2πx) - 3 sin(6πx)$
From that, we conclude that $b2 = 1$ and $b3 = -3$. Can someone please explain how we got these coefficients from the above?
Comment: Well, $b_n$ are the result of the integral you correctly posed, e.g. $\int_o^1\sin^2(2\pi x)=1/2$. But, are you sure about $b_3$? it's zero as all others except $a_6$, tht has value $-3$.
Answer: The separated solutions of the your PDE have the form
$$
             f_n(x,t) = X_n(x)T_n(t),\\
          X_n(x) = \sin\left(\frac{n\pi x}{L}\right),\;\;\; T_n(t)=\exp\left(-k\left(\frac{n\pi}{L}\right)^2t\right)
$$
The eigenfunction solutions $X_n$ for the $x$ equation satisfy common endpoint conditions, and this forces
$$
             \int_{0}^{L}X_n(x)X_m(x)dx = 0,\;\;\; n \ne m.
$$
This is a general pattern for separation of variables problems. The general solution $u$ has the form
$$
            u(x,t) = \sum_{n=1}^{\infty}A_n X_n(x)T_n(t)
$$
The coefficients $A_n$ are determined by the initial condition
$$
            f(x) = u(x,0) = \sum_{n=1}^{\infty}A_n X_n(x)
$$
because $T_n(0)=1$. Using orthogonality,
$$
             \int_{0}^{L}f(x)X_m(x)dx = A_m\int_{0}^{L}X_m(x)^2dx \\
            A_m = \frac{\int_{0}^{L}f(x)X_m(x)dx}{\int_{0}^{L}X_m(x)^2dx}.
$$
Apparently $L=1$ in your problem. And the initial function $f$ was convenient chosen to be of the form
$$
         f(x) = \sin(2\pi x) - 3\sin(6\pi x) \\
              = X_2(x)-3X_6(x).
$$
So that was crafted to make it easy to find the coefficients $A_n$ because
$$
        \int_{0}^{L}f(x)X_m(x)dx = 0,\;\;\; m \ne 2,6.
$$
You can spot that the solution has to be
$$
                  u(x,t) = X_2(x)T_2(t)-6X_6(x)T_6(t).
$$


Why am I getting vector subscript out of range error in the Merge Sort?

Question: void merge(vector<int> dst,vector<int> first,vector<int> second)
{
    int i=0,j=0;

    while(i<first.size()&&j<second.size())
    {
        if(first[i]<second[j])
        {
            dst.push_back(first[i]);
            i++;
        }
        else
        {
            dst.push_back(second[j]);
            j++;
        }
    }
    while(i<first.size()
    dst.push_back(first[i++]);

    while(j<second.size())
    dst.push_back(second[j++]);
}

void mergeSort(vector<int> &a)
{
    size_t sz = a.size();
    cin.get();
    if(sz>1)
    {
        vector<int> first(&a[0],&a[sz/2]);
        vector<int> second(&a[(sz/2)+1],&a[sz-1]);

        mergeSort(first);
        mergeSort(second);

        merge(a,first,second);
    }
}

void MergeSort(int* a,size_t size)
{
   vector<int> s(&a[0],&a[size-1]);
   mergeSort(s);
}

Can some one help me what is the problem with this code ? I am getting vector subscript outof range error.
Answer: If sz == 2, &a[(sz/2)+1] will try to take the address of a[2], which will give you this error.
Comment: @brett: It's fairly common to write `if ( input is empty ) return input`. But see Martin's answer.
Comment: how can I avoid that case. One thing is to use a special case. But thats not a good code. any other way ?
Answer: Your sub vectors are specified incorrectly.
Remember the iterators specify the beginning to one past the end.
So this will misses the middle element and the last element in the vector.
And is also undefined for really short vectors of length 2
    vector<int> first(&a[0],&a[sz/2]);
    vector<int> second(&a[(sz/2)+1],&a[sz-1]);

Imagine if a is the vector { A, B, C, D}
    first:  {A,B}   0 -> 2 (where 2 is one past the end so index 0 and 1_
    second: {}      3 -> 3 (Since one past the end equals the start it is empty}

Or Try a bigger vector: { A, B, C, D, E, F, G, H, I}
    first:  {A, B, C, D}    0 -> 4 (4 is one past the end so index 0,1,2,3)
    second: {F, G, H}       5 -> 8 (8 is one past the end so index 5,6,7)

Or Try a smaller vector: { A, B}
    first:  {A}    0 -> 1
    second: {BANG} 2 -> 1

Should be:
    int* st = &a[0];
    // Using pointer arithmatic because it was too late at night
    // to work out if &a[sz] is actually legal or not.
    vector<int> first (st,      st+sz/2]); // sz/2 Is one past the end.
    vector<int> second(st+sz/2, st+sz   ); // First element is sz/2
                                           // one past the end is sz

The vectors passed into merge(). The dst parameter has to be passed by reference as it is an out parameter. But also note that first and second parameters are const so we can pass by const reference (to avoid the copy step).
void merge(vector<int>& dst,vector<int> const& first,vector<int> const& second)

Also the merge function:
Is pushing the value into dst. But dst is already full from the data that came in. So before we do the merge the destination must be cleared.
    mergeSort(first);
    mergeSort(second);

    // Must clear a before we start pushing stuff into.
    a.clear();   // Add this line.
    merge(a,first,second);

Comment: why are you passing st+sz/2 in both first and second arrays ? Also why are using ] it might give a compiler error right ?
Comment: @brett: The constructor for a vector takes two iterators. They point to the first element and one past the last element. Thus the vector 'first' we pass `st+sz/2' this is one past the end and thus should be equal to the first element in the second vector `st+sz/2'.
Comment: @brett: Following these rules: This `vector s(&a[0],&a[size-1]);` is also wrong. The last iterator should point one past the last element. `s(&a[0],&a[size])`
Comment: Thank you very much sir for the clean explanation.
Answer: Martin is right, the problem is the contructor of the auxiliar vectors:
Original vector: 1 9 7 9 2 7 2 1 9 8
iter1: 2, iter2: 8
   vector<int> v( iter1, iter2 ); //new vector: 2 7 2 1 9

http://www.cppreference.com/wiki/stl/vector/vector_constructors
And talking about merge-sort and other sorting algorithms, I´ve found a very useful web:
http://www.sorting-algorithms.com/merge-sort


What does substituting a population total for an estimate mean for the standard error?

Question: I have to calculate the minimum sample size for a proportion estimate required in order to meet a criterium on the standard error of this kind:
$$
\text{SE}_P = \sqrt{\frac{P(1-P)}{n}}  \sqrt\frac{N-n}{N-1} \lt \text{SE}_\text{req}
$$
Where $\text{SE}_\text{req}$ is a constant (calculated by a formula).
However, the population size $N$ is itself an estimate.
I am trying to find out whether I can just substitute this estimate into the formula for $\text{SE}_P$ or if the fact that $N$ is an estimate changes the standard error? I have no luck googling it.
Comment: The $\sqrt{(N-n)/(N-1)}$ term is also known as the _finite population correction_ (fpc) factor. I haven't seen an example of how the fpc changes if $N$ is unknown but that may be a good path forward to answer this question. There are cases where the fpc can be ignored but that doesn't really answer your question.
Comment: Can you give more info on how $N$ was estimated? Was it assumed to be _normally_ distributed?
Answer: I've been watching this question hoping someone would step in and drop some statistical theory for us but until then I can point to a possible applied solution to your problem.
I mentioned in my comment that the $\sqrt{(N-n)(N-1)}$ term is also known as the finite population correction (fpc) factor. The name is pretty self explanatory: it's used to correct the estimate when the population is finite. I found this answer gave a ton of intuition for the fpc as this being a case where you are sampling from a Hypergeometric distribution.
If we assumed the population was infinite then the classical way to estimate the sample size no longer depends on the fpc and is:
$$
n = \frac{p(1-p)}{\text{SE}^2}
$$
For this case, let's say we want to estimate the true proportion of $p$ to a standard error of no worse than $0.05$ (5 percentage points) and to be conservative we choose that $p=0.5$. Then we would have the following (code in Julia):
se = 0.05
p = 0.5
n = (p / se)^2
n

## 100

Now we can see the effect that a finite population has on this estimated sample size as well. Given the formula from your question we could solve this for $n$ and we'd have the following:
$$
n = \frac{N}{\frac{\text{SE}^2(N-1)}{p(1-p)} + 1}
$$
If we reuse our chosen $\text{SE}$ and $p$ and assume we know the total population and it is $N = 100$, then the estimated sample size would be:
N = 100
N / ((N - 1) * (se)^2 / (p * (1-p)) + 1)

## 50.25

This definitely jives with our intuition that we shouldn't need as many samples if the population is small.
Getting back to your question, there are a couple ways I could think of propagating the uncertainty of the estimated $\hat N$:

If we are dealing with a linear function then we could use linear uncertainty propagation. If you're familiar with the Julia programming language, there is a package called Measurements.jl that makes this very easy.
If we are dealing with a non-linear function then we could use Monte-Carlo sampling instead. In Julia the MonteCarloMeasurements.jl package has this implemented.

We can verify if the function we're working with is linear or non-linear with respect to $N$ by plotting the function:
using CairoMakie

xs = 0:0.01:600
ys = @. xs / ((xs - 1) * (se)^2 / (p * (1-p)) + 1)
lines(xs, ys; axis=(; xlabel=L"N", ylabel=L"n"))

The above plot shows us this function is non-linear with respect to $N$ and is approaching $100$ as $N \to \infty$. Since it's non-linear we should make use of simulations to better understand $n$ given $\hat N$:
using MonteCarloMeasurements

N_est = 100 ± 25 # Construct Gaussian uncertain parameter
n_est = N_est / ((N_est - 1) * (se)^2 / (p * (1-p)) + 1)
n_est

## Particles{Float64, 2000}
##  49.4265 ± 6.69

The result from above could be a bit misleading since it gives you back a $\pm$ but this does not mean the result is symmetric so it's good to plot the raw samples to have a better idea of what's going on:
samp_quant = pquantile(n_est, [0.025, 0.5, 0.975])
hist(Array(n_est), bins = 50, axis = (;xlabel=L"n", title = "Samples of estimated sample size after population uncertainty propagation"))
vlines!(samp_quant, color = :black, linestyle = :dash)
current_figure()

So via simulation we have a distribution for $n$ and we can now see this distribution is a bit skewed.
If we wouldn't have taken into account the uncertainty of $\hat N$ then we would have chosen $n \approx 50$ (note that's our median as well) but given that we can do uncertainty propagation we may want to choose a more conservative choice of $n \approx 60$.
Another thing this answer doesn't address is power but your question didn't mention it so I'll just leave this reference to Chapter 16 (Design and Sample Size Decisions) in Regression and Other Stories by Andrew Gelman, Jennifer Hill, Aki Vehtari. The pdf is free for personal use and can be found at the following link.


How to set the scrolltop() method to the sliding div?

Question: I have a sliding div, or we can say that its a on click expand and collapse div(toggle).
I want to apply the .scrollTop() to the div. when the button is clicked to expand the div
    before expand  set to top, and then expand. I tried it more then hundreds times
    with different ways but i unable to get the desired result. Any help will be appreciated.

 Click me

var currentscrollpos;
currentscrollpos = $(window).scrollTop();
$(document).ready(function(){
$(".slidingDiv").hide();
    $(".show_hide").show();
$('.show_hide').click(function(){
$(".slidingDiv").slideToggle();
    $('body').animate({scrollTop:100}, 'slow')
    return false;
  });

});
$(document).click(function()
{
$('body').animate({ scrollTop: currentscrollpos }, 0);
});

//This is my css

#mainarea{width:550px; height:300px;background-color: yellow;}

.slidingDiv {
height:200px;
background-color: white;
margin-top:10px;
width:261px;
background-color: black;

}

.show_hide {
display:none;
}
#notificationFooter {
background:#FFFFFF; margin:0 auto; height:8px; width:96px;
 }
This is the fiddle
http://jsfiddle.net/muheetmehfooz/wmwn5424/

Comment: fiddle is here: http://jsfiddle.net/muheetmehfooz/wmwn5424/
Comment: Please put it as an answer.:)
Comment: Nadeem what you mean i didn't understand.
Answer: What you are doing is not bring you to top, just move you bit up. However, if your question is only about how to make this actions in right order - try this.
jsfiddle
just use callback function

P.S.
 to get to the top use
{scrollTop:currentscrollpos}


How to extract values of a raster stack into separate columns?

Question: I have rasters of different vegetation measurements (i.e., canopy height, veg density). I want to stack the layers and use dplyr to extract values and assign to different columns. How can I do this? Sample code below. Sorry for not having a reproducible example.
Instead of this:
RSF_df.sf <- RSF_df.sf %>%
  mutate(can_cover5m = raster::extract(can_cover5m, RSF_df.sf),
         veg_density1mheight_5m = raster::extract(veg_density1mheight_5m, RSF_df.sf),
         veg_density2mheight_5m = raster::extract(veg_density2mheight_5m, RSF_df.sf),
         veg_density3mheight_5m = raster::extract(veg_density3mheight_5m, RSF_df.sf),
         veg_density4mheight_5m = raster::extract(veg_density4mheight_5m, RSF_df.sf),
         veg_density5mheight_5m = raster::extract(veg_density5mheight_5m, RSF_df.sf),
         veg_density_over5mheight_5m = raster::extract(veg_density_over5mheight_5m, RSF_df.sf))
summary(RSF_df.sf)

I want this, but with each veg. measurement in a separate column:
all_land_data<-addLayer(can_cover5m,veg_density1mheight_5m,veg_density2mheight_5m,veg_density3mheight_5m,
                        veg_density4mheight_5m,veg_density5mheight_5m,veg_density_over5mheight_5m)

del <- RSF_df.sf %>%
  mutate(all_land_data = raster::extract(raster stack, RSF_df.sf)) #I need values for each veg. measurment in seperate columns
summary(del)

Answer: Unless I misunderstood, I think you are making this more complicated than necessary. If you use the terra package, and read all rasters in together, a SpatRaster stack is created. Then, using terra::extract to extract values for each feature in a terra::vect object will produce a data.frame with columns for each raster layer in the stack.
i.e.
library(terra)
veg_stk <- c(rast(veg_density1mheight_5m), rast(veg_density2mheight_5m), rast(veg_density3mheight_5m), rast(can_cover5m)
RSF_vect <- vect(RSF_df.sf)

del <- terra::extract(veg_stk, RSF_vect, fun = mean)  # returns a DF

Comment: Micha, I appreciate your help. Will this work if ```RSF_df.sf``` is a simple features object?
Comment: It did extract the raster data to separate columns, but it did not retain all of the data in ```RSF_df.sf```.
Comment: Save the output of the extract into a new data.frame, then cbind the original `RSF_df.sf` with the new data.frame to get all columns in one DF.


Installing phpMyAdmin installs Apache

Question: When I install phpMyAdmin it automatically installs Apache. That happens even though I do not select Apache in the installation process of phpMyAdmin and choose not to install via db-config.
How do I avoid to install Apache since I am using Nginx?
My setup is:
- Debian Squeeze
- dotdeb.org setup in sources.list
- nginx 1.0.11
- PHP 5.3
Answer: apt-get has a download only option. So do the sudo apt-get install -d phpmyadmin and then choose the packages you want to install and install them manually with sudo dpkg -i <packages>.deb  These packages are usually downloaded to /var/cache/apt/archives/ .
Answer: What OS do you use?If it is using packages like Debian or Redhat, Apache  is installed as dependency since PHPadmin requires webserver.Consult your OS documentation for details how to ignore dependencies.
Comment: Check this howtoforge article that may help: http://www.howtoforge.com/running-phpmyadmin-on-nginx-lemp-on-debian-squeeze-ubuntu-11.04
Comment: howtoforge do not take into account the Apache get's install. And the webserver nginx is already installed

Installing phpMyAdmin installs Apache


How should I go about placing the content of a field from a content type into a block?

Question: My case is this:
I have a content type called 'Test Type' with 2 fields: Body (text area), Slideshow (Unlimited list of images)
When I create an instance of Test Type and it renders, my goal would be to have the content of the Slideshow field go to a different region, let's say the header. First thought was to create a custom block that would hold a reference to the field content, but that does not seem to be an option. Is this possible to do through the UI with certain settings or would I be better off writing a module. The module Field as Block is pretty much exactly what I was looking for but it seems not work right now.
Comment: You could create a view with a Block display that filters to a particular node and displays only the Slideshow field.
Answer: this can be possible by using views which is now in core in drupal 8
step 1 :: go to structure->views->add new views
step 2 :: Add new view ui will be opened
2.1.enter view name
2.2.under view settings select show content of type (the content type     whose field you would like to place in an block ex:your required content type)

3.only check the check box create a block under block settings
2.3.1.under BLOCK DISPLAY SETTINGS select the display format most likely
 (unformatted list) of fields
2.3.2 now save and edit and you are good to go...

step 3 :: now views ui is opened
3.1.on the top left hand side of the page under title
3.2. under fields select add (an pop up ui will open)
3.3. select type content ->select the field types of your content you want to display in your block ex-body and so on and so forth as required by you now apply and save..
3.4. configure field will now open configure as required..you can also rewrite the fields so better follow this link
rewrite fields
step 4 :: go to structure->blocks->place the block as per your required region->and find the block(it will be the same name as block name or view name unless changed)
4.1. place the block as per your wish there are many boundaries like selecting content type etc ...but for now select pages under visibility
leave the check box as default and inside pages type <front>
**remember to add less than and greater than sign as well
4.2.now save the block
step 5 :: in the block ui at the bottom also click save blocks and finally go back to the site
if all goes right you will see the content of a field from the content type in a block using views
Comment: Brilliant, had not ventured toward Views. Got exactly what I needed and more with some contextual filters. Thank you for the advice.


How do I pass a url into an asp.net repeater from an XML source?

Question: It's actually a bit more complicated than the title. I'm trying to figure out how to pass my own predefined url plus an ID from an XML source-- all as an href in an anchor tag.
Basically, I have a XML source with different titles and ID's:
    <root>
      <mainNode>
         <title>ABC</title>
         <ID>100</ID>
      </mainNode>
      <mainNode>
         <title>DEF</title>
         <ID>101</ID>
      </mainNode>
    </root>

I'm passing the Title to a repeater and want there to be an anchor tag as follows:
    <asp:Repeater ID="xPathRepeater" runat="server">
    <ItemTemplate>

<%#XPath ("title") %>

            Read More...

    </ItemTemplate>
</asp:Repeater>

Where 100 would be the ID I'm getting from the XML source.
I have no clue how to do this.
Here is my code behind:
    protected void XMLsource()
{
    string URLString = "http://ExternalSite.com/xmlfeed.asp";

    XmlDataSource x = new XmlDataSource();
    x.DataFile = URLString;
    x.XPath = String.Format(@"root/mainNode");

    xPathRepeater.DataSource = x;
    xPathRepeater.DataBind();
}

Any help would be appreciated. Thank you so much!
Answer: I think this is what you are asking for?
  <asp:Repeater ID="xPathRepeater" runat="server">
  <ItemTemplate>

<%#XPath ("title") %>

      <a href="preDefinedPathInCodeBehind.aspx?<%#XPath ("ID") %>">Read More...

  </ItemTemplate>
</asp:Repeater>

Answer: Did you try this?
<a href='preDefinedPathInCodeBehind.aspx?' + '<%# XPath("ID") %>'>Read More...

Answer: Or this..
   <asp:Repeater ID="xPathRepeater" runat="server">
            <ItemTemplate>

 <%#XPath ("title") %>

                    <a href=  "<%# "preDefinedPathInCodeBehind.aspx?" + XPath ("ID")  %>" >Read More...

            </ItemTemplate>
        </asp:Repeater>

Answer: Does it mean?
<asp:Repeater ID="xPathRepeater" runat="server">
        <ItemTemplate>

                    <%#XPath ("title") %>

                <a href="preDefinedPathInCodeBehind.aspx?<%#XPath ("ID") %>">Read More...

        </ItemTemplate>
    </asp:Repeater>

tried. it works.


How to change the width of list box in html?

Question: How to change the width of list box
<select id ="tagsList" class="ui-widget-content ui-corner-all" width="100" size="18" multiple="multiple">

I try width="100" but it doesn't work
Answer: You can set the width using inline css like this:
<select style="width: 100px;"></select>

In with the rest of your attributes.
There are other ways to specify the width of an element, and most revolve around CSS. Here's a link to Google's CSS education video that should get you more than primed and ready to go.
Comment: link no longer active because The University Consortium is no longer actively maintained.
Answer: You need to use CSS:
<select id="tagsList" style="width:100px;">

Answer: You have to use CSS's width property. Add style="width: 100px;"
Answer: <select id ="tagsList" class="ui-widget-content ui-corner-all" style="width:100;" size="18" multiple="multiple">

this code almost your i just added that style that is enough


Stationary matrix given a transition matrix

Question: I am given the following transition matrix
$$P= \pmatrix{ 1-\alpha & \alpha \\ \beta & 1-\beta}, \ \alpha,\beta \in (0,1)$$
with the states $S=\{1,2\}$.
I want to determine the stationary distribution $\pi$ of the Markov chain determined by a starting distribution and the transition matrix $P$.
We have the system:
$\pi(1)=(1-\alpha)\cdot \pi(1) + \beta\cdot\pi(2)$
$\pi(2)=\alpha\cdot \pi(1) + (1-\beta)\cdot\pi(2)$
When I substitute for $\pi(2)$ from the first equation in the second, I get $0=0$.
However, the answer should be:
$$\pi(1)=\frac{\beta}{\alpha+\beta},\pi(2)=\frac{\alpha}{\alpha+\beta}$$.
What am I doing wrong? Thank you for your time.
Comment: You are missing $\pi_1 + \pi_2 = 1$. This gives you the proper second equation.
Answer: You are missing $\pi_1 + \pi_2 = 1$ giving you the proper second equation.


Prediction of multiple time series of different lengths

Question: I have multiple store sales time series where seasonality is, or should be, correlated, but the time series are of differing lengths (e.g. 6 months vs 3.5 years). Forecasting the longer time series works well, but for the shorter time series obvious seasonality components are missing.
Is there a straight forward way to "borrow" seasonality information from the longer time series and apply it to the shorter time series?
Answer: Yes. Are you familiar with time series decomposition? Whatever software you're using will have the ability to do. It can separate your longer time series into components like level, error, trend and seasonality.
Let's assume your time series do not have equal means/levels. One important thing will be to get the decomposed seasonality in multiplicative terms not additive. This will make it a percent, rather than nominal. If you had annual data with weekly seasonality you can get 52 percentages, one per week.
If you then forecast and decompose your shorter series the seasonal index can be multiplied against the level to get a seasonally adjusted level.


Soccer players with over 100 international goals (IFFHS definition)

Question: Who are the soccer players with the most international goals? The international goal definition used is the one from IFFHS (international goals in national team + international goals in club) and not the one used by RSSSF who count international goals as those made in the national team only . This is a (incomplete) list that I've been making, but it is probably incomplete because players from Asian and Arabian leagues are hard to track and it could have some mistakes, plus it becomes outdated quickly since players such as Cristiano Ronaldo, Messi, Ibrahimovic keep scoring international goals. Is there any other players with more than 100 international goals I have not found?
<code>1  Cristiano Ronaldo  POR 158
2  Lionel Messi       ARG 146
3  Gerd Muller        GER 138
4  Puskas             HUN 126
5  Ali Daei           IRN 122
6  Raúl               ESP 119
7  Didier Drogba      CDM 115
   Andriy Shevchenko  UCR 115
9  Zlatan Ibrahimovic SUE 113
10 Ronaldo            BRA 111
11 Thierry Henry      FRA 110
12 Hossam Hassan      ARA 109
13 Romario            BRA 105
14 Samuel Etoo        CAM 104
15 Pelé               BRA 101
   Miroslav Klose     GER 101
</code>
Comment: Ali Daei is from Iran, also I am not sure what country ARA stands for..
Comment: Sorry, those are acronyms written in spanish :)
Comment: Do you count (e.g.) goals for [AS Monaco FC](https://en.wikipedia.org/wiki/AS_Monaco_FC) against French teams? (and all the other similar situations).
Answer: The difficulty of this classification lies on the fact that you have to figure out how many club level goals have been scored at international matches (and here I am assuming that international match means a match in an inherently international competition, so that it includes e.g., goals scored in a Champions League match against a team of the same country and excludes goals in national league matches against teams from just across the border).
Luckily, some other people have done some of the work for you, but even there you have to work with care. For example, you list Romário as having 105 international goals. When one checks his career statistics on Wikipedia, he gets credit for 55 goals with Brazil plus 50 goals in "continental" competitions at the club level. So far so good. But then, next to the "continental" column, there is an "other" column claiming 44 goals, and a note indicating that these goals correspond to a variety of competitions ("includes UEFA Super Cup 1988, Intercontinental Cup 1988, Dutch Super Cup 1991, 1992, Supercopa de España 1993, FIFA Club World Cup 2000"), only some of which are international. So the 105 figure is, if anything, a lower bound. If you go and investigate the listed competitions, you'll find out that Romário scored at least 4 goals. Bringing the figure up to at least 109. Were all other 40 "other" goals scored in domestic competitions? I don't know, since the relevant note doesn't claim to exhaust all of the relevant competitions. That's something for someone with more time than me to go and figure out.
tl;dr : there is some opacity in the domestic/international distinction at the club level that makes claims of "best international goalscorer" inherently unreliable. The only solution is honest, time-consuming detective work on individual players' careers.
Comment: yes, the difficult part is the club goals, mainly goals in supercups, recopas, and such minor championships


python scrapy decoding to json even when value field contains a single quotes

Question: I'm working on python web scraping and I would like to decode a string to json format. But I can't do it, since some of the value fields in the string contains single quotes in it.
For example, this is the string from the web page, I want to decode to json:
dataLayer = [{'place': 'paris',
        'description': 'its one of the world's beautiful places.' }]

Since the description field contains a single quotes inside it (world's), I can't decode it to json.
This is my code:
import re
import cjson
place_details = re.findall("dataLayer = \[(.*?)\];", response.body, re.DOTALL | re.MULTILINE)[0].replace("\'", "\"")
place_details_json = cjson.decode(("{"+product_details+"}"))

Please help me to solve this issue.
Answer: Well it's a big of a mess to try to untangle. Depending on the other places you have ' in your dataset, you may get away with something as simple as this:
>>> s="""{'place': 'paris',
...             'description': 'its one of the world's beautiful places.' }"""
>>> import re, json
>>> json.loads(re.sub(r"('\B)|(\B')", '"', s))
{'place': 'paris', 'description': "its one of the world's beautiful places."}

"scrapy" is the key word when text hasn't been escaped correctly!
Comment: Thank you john for your help. It worked. Now im wondering what if a double quote comes in place for a single quote? Is it possible to load it to json?
Comment: @nithinks, you'll need to create a test suite of samples that covers all the variations you're seeing. Then you can work on solving all those other special cases.
Answer: import json
print json.dumps(dataLayer)

Because dumps() for non-ASCII characters return value to unicode.


Htaccess Need: http to https - nonexisting files, folders, subdomains to 404 - Only first is working

Question: I need to add rewrites in .htaccess as described in title, but only the first one works, here is how my .htaccess looks now:
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [R,L]

RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /404.php [R,L]

ErrorDocument 404 /404.php

Examples that does not work
These two NON EXISTING SUBDOMAINS examples just hang then show "Unable to connect", but I want them to show REAL 404 error page:
http://nonexisting.example.com
https://nonexisting.example.com

If we enter NON EXISTING FILES or FOLDERS, for example:
https://example.com/nonexistingfile.php
https://example.com/nonexistingfolder/

it shows my 404.php page allright BUT they turn up as a SOFT 404 and I want it to show the REAL 404 error page.
I have tried around 30 different variations of the second Rewrite section without any success. Some code returned a REAL 404 code but showed a messed up page not my 404.php page.
Comment: So you need a wildcard DNS entry: `*.yourdomain.com`
Comment: What do you mean by `SOFT 404`?
Comment: The "non existing subdomains" are working correctly unless you have a wildcard DNS record.
Comment: @anubhava Accorind to google : The target URL doesn't exist, but your server is not returning a 404 (file not found) error
Comment: @Tigger The DNS is set to:

    **Type | Host | Value**
    A Record | @ | IP-ADDRESS
    URL Redirect Record | www | h t t p s : / / example.com
Comment: For the real 404, send the correct  header in 404.php: `header('HTTP/1.1 404 Not Found')` before any output is sent, e.g. at the very top of the file.
Answer: You don't need a rewrite rule here which is just redirecting all non-existing files or directories to /404.php with R=302.
Just try this code:
# will send correct 404 status as well
ErrorDocument 404 /404.php

RewriteEngine On

RewriteCond %{HTTPS} off
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [R=302,NE,L]

Comment: it seems to work like another one I tried which was that it is showing REAL 404 code but the page (404.php) seems messed up with style, html etc. any idea?
Comment: I have added full links to each css, js, images respectively without no effect. I use no other rewrites that your example.
Comment: how does your 404 php header look like except for header($_SERVER["SERVER_PROTOCOL"]." 404 Not Found"); OR header("HTTP/1.0 404 Not Found"); ?
Comment: My `404.php` doesn't have any `header` function. Just regular HTML data with custom message and usual css/js/images links
Comment: You know what, I made a completely new 404.php without any PHP header and now ALL seems to work excellent!


Formula for conversion scale starting from 1 to percentages.

Question: so my question is.
I have a scale ranging from 1 to 7 points. With 1 point beeing the worst and 7 points beeing the best. I need to convert this scale to percentages. E.g. 1 is 0%, and 7 is 100%
Can anybody help me out with the formula for coversion of this scale. For example 5 needs to be converted.
The problem that i run into is that 1 has to be 0%
Answer: What is wrong with $1$ going to $0\%$?  Intuitively, you can just subtract $1$ from your numbers so they run from $0$ to $6$.  Now you want $6$ to  be $100\%$.  Your percentage is $16.6666666 (\text{number}-1)$


Upsert Bulk vs Batch Upsert in Salesforce

Question: I am trying to upsert large amount of data into Salesforce. I found there are two methods available for this.
1. Use UpsertBulk. This will upsert data in a single shot
2. Use Batching
Create upsert job. create batches for upsert operation.
What is the difference between these two methods?
What is the best way to do bulk upsert?
Answer: The best way to insert/update/upsert data in bulk is via the recently released and renewed BulkAPIv2. Please see this link: https://resources.docs.salesforce.com/210/latest/en-us/sfdc/pdf/api_bulk_v2.pdf
Bulk API 2.0 simplifies uploading large amounts of data by breaking the data into batches automatically. All you have to do is upload a CSV file with your record data and check back when the results are ready
Answer: Both methods are using Bulk API
Differences:

Upsert bulk - single operation offered by Salesforce connector which creates a new Job in Salesforce and creates a Batch within the Job. After batch is processed you need to make sure Job will be closed.
Create Job -> Create Batch - two separate operations of the Salesforce connector to create a new Job and add a new Batch within the Job. After batch(es) is processed you need to make sure Job will be closed.

Both options are pretty similar.
Which one is better for you? I would check the Bulk API limits first and then decide -> Salesforce Bulk API Limits
Depending on the volume of data you are going to process you may want to create one or multiple batches (or maybe use multiple jobs).


What is wrong with my PDO prepared statement?

Question: Obviously, I am preparing the statement wrong, but I am not certain what I am doing wrong.
These 2 code segments are identical, except for the second line.
This fails:
<code>    $dbh = new PDO('mysql:host=localhost;dbname=' . $DB_Database, $DB_UserName, $DB_Password);
    $sth = $dbh->prepare("SELECT * FROM `PNB`.`Users` WHERE `Users`.`EMail` = :email OR `Users`.`Temp_EMail` = :temp_email");
    $sth->execute(array(':email' => $email, ':temp_email' => $email));
    $sth->setFetchMode(PDO::FETCH_ASSOC);
    $res = $sth->fetch();
    $dbh = null;
</code>
This hard-coded test works:
<code>    $dbh = new PDO('mysql:host=localhost;dbname=' . $DB_Database, $DB_UserName, $DB_Password);
    $sth = $dbh->prepare("SELECT * FROM `PNB`.`Users` WHERE `Users`.`EMail` = 'firstname.lastname@example.com' OR `Users`.`Temp_EMail` = 'firstname.lastname@example.com'");
    $sth->execute(array(':email' => $email, ':temp_email' => $email));
    $sth->setFetchMode(PDO::FETCH_ASSOC);
    $res = $sth->fetch();
    $dbh = null;
</code>
What am I doing wrong?
Thanks!
UPDATE: Solved!
The exact issue is still unknown, but seems to be related to the 'excessive naming' suggested by user 'Your Common Sense' in the comments below.
This works just fine:
<code>$dbh = new PDO('mysql:host=localhost;dbname=' . $DB_Database, $DB_UserName, $DB_Password);
$sth = $dbh->prepare("SELECT * FROM Users WHERE EMail=:email OR Temp_EMail=:temp_email");
$sth->execute(array(':email' => $email, ':temp_email' => $email));
</code>
Thanks to everyone. I learned lots AND resolved the issue.
Message to Your Common Sense; If you form your comment as an 'Answer', then I can accept it.
Comment: Are you getting an error?
Comment: Btw, there is no use for such excessive naming. `"SELECT * FROM Users WHERE EMail = :email OR Temp_EMail = :temp_email"` is okay.
Comment: Thanks. I suspected the excessive naming was no value add. Just wanted to be consistent with examples I was seeing while I debug this.
Answer: It's hard to answer on sight.
Your code seems okay to me. So, debugging seems the only way.

What am I doing wrong?

Always ask this question from your PDO.
Every time you're connecting to PDO, do it this way (also make sure you can see errors, either on-screen or logged):
<code>error_reporting(E_ALL);
$opt = array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION);
$dsn = 'mysql:host=localhost;dbname=' . $DB_Database;
$dbh = new PDO($dsn, $DB_UserName, $DB_Password, $opt);
</code>
if there is an error - you'll be notified.
If there isn't - check typo-like problems.
Comment: Where will I be notified? I tried it, and there is no difference. The query fails, and I don't know why.
Answer: Just a quick try - Do you get it right with these two lines ?
<code>$sth = $dbh->prepare("SELECT * FROM `PNB`.`Users` WHERE `Users`.`EMail` = :email OR `Users`.`Temp_EMail` = :temp_email");
$sth->execute(array(':email' => 'firstname.lastname@example.com', ':temp_email' => 'firstname.lastname@example.com'));
</code>
in other words... Did you set your $email variable ?
Comment: No. I get the same result. The query fails. Going to try the error reporting steps from user 'Your Common Sense'.
Answer: try this
<code>  $dbh = new PDO('mysql:host=localhost;dbname=' . $DB_Database, $DB_UserName, $DB_Password);
$sth = $dbh->prepare("SELECT * FROM `PNB`.`Users` WHERE `Users`.`EMail` = :email OR `Users`.`Temp_EMail` = :temp_email");
$sth->bindParam(':email', $email, PDO::PARAM_STR);
$sth->bindParam(':temp_email', $email, PDO::PARAM_STR);
$sth->execute();
$res = $sth->fetch(PDO::FETCH_ASSOC);
$dbh = null;
</code>


Machine learning methods for multi-dimensional input and output

Question: I have a large dataset where my input is an $M$-dimensional tensor, and each input has a corresponding $N$-dimensional output. My goal is to train a method to learn outputs from the millions of inputs (i.e. tensors) in my database. Each of the elements composing the input and output tensors are simply floating point numbers, and this is essentially a regression task. In my specific case, $M$ and $N$ are $3$ and $2$, respectively (e.g. input is $20 \times 20 \times 20$ and output is $100 \times 100$). But generalized approaches are sought, if possible.
Based on past questions, there are suggestions to possibly convert the data into a single vector, although this appears to lose information about structural features (e.g. profiles). I'm also looking into convolutional neural networks, but treatment of multi-dimensional outputs appears missing. Overall, I am searching for methods and/or packages that can handle tensors of variable size as inputs and outputs (i.e. to perform a mapping in $M$-dimensions to $N$-dimensions).
Comment: @Alex I may simply be mistaken, but after searching for examples, I could not find instances of neural nets taking tensors as both inputs and outputs. If you can provide particular code examples, that would be greatly appreciated. Also, it's perhaps worth adding that I am interested in learning of methods in addition to neural networks to tackle this problem. But any solution is still a solution.
Comment: "multi-dimensional outputs appears missing." Why do you say that? Neural Networks can easily have multiple output neurons instead of one output neuron.
Answer: From what it sounds to me you need to make a custom architecture for your task. Multi-dimensional outputs, however, should not bother you, as they are found in a multitude of tasks.
CNNs have been used for multi-dimensional outputs in many instances.
To name a couple:

Probably the most common form of multi-dimensional input and output model is an Autoencoder. While this has little to do with your task, as its main goal is data compression, it is an architecture that takes an image (e.g. $224 \times 224 \times 3$) as an input and produces the same image as an output ($224 \times 224 \times 3$).
Another task that requires both multi-dimensional input and output tesors is the task of image segmentation. Here the input, again, is an image (e.g. $224 \times 224 \times 3$), while the output is a number of segmentation masks ($224 \times 224 \times num\_classes)$. Some NN architectures that solve this task can be found here.

However, a CNN-based approach might under-perform if your input isn't image related, or doesn't have spatial features.
Design tips:
If you decide to try it out, I'd like to give out a few pointers. Most architectures can be split into two parts: the first half of the model aims at compressing the input and extracting all possible information, while the second takes this compressed form and tries to recreate the desired output.
For the first part, you can choose an architecture that has enough depth to sufficiently model the problem. Imagine you want to solve a simple binary classification task on your input data. What architecture would you choose? Drop the output layer and you have your first half. For the second part, try to gradually upscale this compressed form to yhe desired output. You can see how an architecture like U-Net handles this.
The output layer, in your case, needs to have a shape of $(100, 100)$ (+ the batch dimension). Also don't use any activation functions on this layer, because you are solving a regression problem.
Comment: This was incredibly helpful and I'm going to look into develop a custom architecture. Would you happen to have any suggestions for methods besides CNNs as well?
Comment: Unfortunately not, this works well with the "deep" nature of CNNs, where you have lots of layers, stacked one after another. If you use let's say fully connected layers, the model would be untrainable due to the large number of parameters (high memory requirements, slow computation time, etc.)...
Comment: This is something I will investigate as I move forward, but is there any limit I should keep in mind in terms of layers for the CNN?
Comment: You could go for a popular U-Net like style where you have blocks of 2 convolution layers followed by a pooling layer. These three layers result in lowering the size of the input dimensions by half. Because you start from 20, you don't want to go too low; I'd try two such blocks, where the first would go from 20 to 10 and the second from 10 to 5. The number of filters (i.e. the third dimension), however should be doubled in each such block (20 to 40 in the first and 40 to 80 in the second). Also something I just thought of, if you have volumetric data you can look into a 3D-UNet architecture.


How can I replace the for loop with enhanced for loop in the following code?

Question: I have this code for sorting an arraylist
    ArrayList<Worker> workersSorted= sort(workers);
    System.out.println(workersSorted.toString());
}

public static ArrayList<Worker> sort(ArrayList<Worker> _workers) throws CloneNotSupportedException {
    Worker temp;
    for (int i = 0; i < _workers.size() - 1; i++) {
        if ((_workers.get(i)).compareTo(_workers.get(i+1)) == -1){
            temp = (Worker) _workers.get(i).clone();
            _workers.set(i, _workers.get(i+1));
            _workers.set(i+1, temp);
        }
    }
    return _workers;
}

How may I rewrite the code using enhanced for loop?
Comment: You can't as you are modifying the list, and not processing the last item, and you need the index.
Comment: you can make an improvement by deleting `.clone()` there though. btw, u are not really sorting here.
Comment: I don't think you can use the enhanced for loop since the block inside your for loop depends on the indices of the workers that are next to each other and an enhanced for loop simply has access to a single worker in each iteration.
Comment: @pquest - you can use Google to find out - http://stackoverflow.com/questions/11685305/what-is-the-syntax-of-enhanced-for-loop-in-java
Comment: btw: this is sort in O(n)? :-D From the code it seems, that Worker implements `Comparable`, why not to use just `Collections.sort()` ?
Comment: Don't. Enhanced for loop is meant to be used for read only purposes, and since you want to update collection you are iterating on you would get `ConcurrentModificationException`. Also `_workers.set` needs information of index of data you want to use (replace) so you still need some `i`.
Answer: Although this probably isn't the answer you are looking for, my opinion is that even if you could rewrite this method to use an enhanced for loop, you shouldn't.
Your current method is fairly clear in it's intention and an enhance for loop will not be able to clearly access Worker instances at specific indexes. Chances are you will likely also run into issues with ConcurrentModificationException if you try to make changes during the iteration. On top of that enhanced for loops are marginally (you could argue insignificantly) slower.
Essentially, an enhanced for loop is not well suited to your requirement.
Answer: I think what you're looking for is this:
for(Worker w : _workers)

where you will use w instead of _workers.get(i) and _workers.get(_workers.indexOf(w) + 1) instead of _workers.get(i+1). As you can see, it will be quite ugly since you have to use indexOf in order to access the next element in the list and also check whether w is the last element of your list (because _workers.get(_workers.indexOf(w) + 1) will throw an error in that case). One other thing you need to consider - does your ArrayList contain duplicates? If it does, then indexOf will always return the first value it finds.
I suggest you stick with an ordinary for-loop in this case.
Comment: That's true. Even more reasons for which he should use an ordinary `for-loop`. I will add your point to my answer.
Answer: Its pretty gross, but it should work.  I shifted the indexes back, so now the i+1 is the current Worker object w and then I had to save the last worker in previous.
ArrayList<Worker> workersSorted= sort(workers);
    System.out.println(workersSorted.toString());}

    public static ArrayList<Worker> sort(ArrayList<Worker> _workers) throws        CloneNotSupportedException{

        Worker temp;
        Worker previous;
        int prevIdx=-1;
        ArrayList<Worker> wNew=new ArrayList<Worker>;

        for (Worker w: _workers){

            if (previous !=null && (previous).compareTo(w)==-1){
                temp=(Worker) w.clone();
                wNew.set(prevIdx, w);
                wNew.add( temp);
            }
         previous=w;
         prevIdx++;
        }
        return _workers;
    }

Comment: Looks like you'll run into concurrency issues. Will this actually run?
Comment: Can you elaborate on concurrency issues?  I'm not sure how you mean this.
Comment: Nevermind, I meant `ConcurrentModificationException` but I didn't notice you were adding and setting to a new `Arraylist`. Your still returning the original list though, so your not actually returning a sorted list. >.<
Answer: The only enhancement I would do to your loop is before the loop assign _workers.size to a variable.  Use the variable instead which removes the size check on every iteration.