PDO Paginação - Fatal error: Out of memory

Question: Estou fazendo uma paginação, porém a mesma está gerando um erro de memória.
Existe alguma maneira de otimizar ou corrigir a falha? Ou é no servidor mesmo?
Nota: o php.ini esta setado a memoria em 512.
if (!(isset($_GET['pagenum']))) {
    $pagenum = 1;
} else {
    $pagenum = $_GET['pagenum'];
}
$page_limit = ($_GET["show"] <> "" && is_numeric($_GET["show"]) ) ? $_GET["show"] : 30;

try {
    $keyword = trim($_GET["keyword"]);
    if (!empty($keyword)) {
        $sql = "SELECT * FROM noticias WHERE conteudo LIKE :keyword OR titulo LIKE :keyword ORDER BY Nid DESC";
        $stmt = $DB->prepare($sql);

        $likekeyword = "%".$keyword."%";
        $stmt->bindParam(':keyword', $likekeyword, PDO::PARAM_STR);

    } else {
        $sql = "SELECT * FROM noticias WHERE 1 ORDER BY Nid DESC";
        $stmt = $DB->prepare($sql);
    }

    $stmt->execute();
    $total_count = count($stmt->fetchAll());

    $last = ceil($total_count / $page_limit);

    if ($pagenum < 1) {
        $pagenum = 1;
    } elseif ($pagenum > $last) {
        $pagenum = $last;
    }

    $lower_limit = ($pagenum - 1) * $page_limit;
    $lower_limit = ($lower_limit < 0) ? 0 : $lower_limit;

    $sql2 = $sql . " limit " . ($lower_limit) . " ,  " . ($page_limit) . " ";

    $stmt = $DB->prepare($sql2);

    if ($keyword <> "" ) {
        $stmt->bindParam(':keyword', $likekeyword, PDO::PARAM_STR);
    }

    $stmt->execute();
    $results = $stmt->fetchAll();
} catch (Exception $ex) {
    echo $ex->getMessage();
}

codigo exibe contadores

   <?php
   for ($i = 1; $i <= $last; $i++) {
   if ($i == $pagenum) {
    ?>

<?php echo $i ?>

            <?php
          } else {
            ?>

<a href="noticias_listar.php?pagenum=<?php echo $i; ?>&keyword=<?php echo $_GET["keyword"]; ?>" class="links"  onclick="displayRecords('<?php echo $page_limit; ?>', '<?php echo $i; ?>');" ><?php echo $i ?>

            <?php
          }
        }
        ?>

Comment: Existe, e muita, sendo que o `count` itera cada posição da array, propriedades de um objeto, e o método [rowCount](http://php.net/manual/en/pdostatement.rowcount.php) é basicamente um aditivo à consulta, ou seja, ele já existe no momento em que é executada uma consulta, e além do mais, não se usa com o `fetchAll`, usa-se diretamente na consulta `$stmt->rowCount`, tal como a função [columnCount](http://php.net/manual/en/pdostatement.columncount.php). Outra opção é a uma função nativa do próprio SQL - [COUNT](http://www.w3schools.com/sql/sql_func_count.asp).
Comment: Você precisa adicionar o `limit` naquele select com dois `likes`.
Comment: mais o limit e difinifo pela variavel $page_limit , ou é a forma que fiz que ta errado?
Comment: Na sua consulta, depois de `ORDER BY Nid DESC` adicione `LIMIT 0, 70` apenas para teste veja se vai acontecer o erro. Do jeito que está a consulta é retornado o maior número de registros, com esse `limit` serão retornados apenas os 70 primeiros, claro vc precisa alterar esse mecanismo para funcionar de forma dinâmica. Veja [essa resposta](http://pt.stackoverflow.com/a/94890/91)
Comment: colocando no select com 2 likes o erro continou o mesmo ja colocando no outro foi o seguinte ->  SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'limit 0 , 30' at line 1
Comment: Coloca ai a consulta que deu erro.
Comment: E outra você está executando a mesma consulta duas vezes ... uma sem limit e outra com.
Comment: rray $sql = "SELECT * FROM noticias WHERE 1 ORDER BY Nid DESC LIMIT 0,30";  no caso estou executando 2 pq se o usuario nao digitar nada ele lista tudo se digitar no campo de pesquisa ele busca com a outra
Comment: Dei um tapa na pergunta :)
Comment: Do jeito que o sistema está é mais provável receberes erros do que o resultado esperado, um exemplo óbvio é este - ` $total_count = count($stmt->fetchAll());` - que capaz de causar sobrecarga, e ainda mantém os valores, e depois ainda é executada outra consulta, com o método `fetchAll()`. Existem funções próprias para contar o número de linhas encontradas, e existem também cláusulas próprias para contar o número de linhas, exemplos são a função `COUNT` do SQL e o `fetchColumns` ou simplesmente `rowCount` para o `PDO`.
Comment: ` $total_count = count($stmt->fetchAll());` mais aqui nao esta usando exatamente o count? existe diferenca entre rowCount($stmt->fetchAll()) e  count($stmt->fetchAll())?
Answer: É uma péssima ideia usar PDOStatement::fetchAll() se você não tem ideia da quantidade de registros que será retornada e armazenada na memória. Tenha em mente que a classe PDOStatement funciona também como um iterador, evitando o uso premeditado de memória. No seu caso, utilize PDOStatement::rowCount() para ter em mãos o número de registros e itere com o método PDOStatement::fetch().
Eu particularmente prefiro executar uma query com COUNT(*) antes, para obter o número total de registros, e outra query definitiva para recuperá-los, usando mecanismos com OFFSET e LIMIT. Não faz sentido requisitar linhas ao banco de dados que não serão utilizadas durante a execução.
Answer: Pelo erro, aquela com consulta com where parece ser o problema, o idela é paginar ela, outro problema é que essa consulta é executada duas vezes.
Veja:
} else {
    $sql = "SELECT * FROM noticias WHERE 1 ORDER BY Nid DESC";
    $stmt = $DB->prepare($sql);
}
$stmt->execute(); //primeira vez que a consulta é executada.
$total_count = count($stmt->fetchAll());

//... código omitido.

$sql2 = $sql . " limit " . ($lower_limit) . " ,  " . ($page_limit) . " ";
$stmt = $DB->prepare($sql2);

//código omitido.

$stmt->execute(); //segunda vez que a mesma consulta é executada.
$results = $stmt->fetchAll();

Acredito que o código pode ser simplificado dessa forma:
$pagenum = isset($_GET['pagenum']) ? $_GET['pagenum'] : 1;

$limite = ($_GET["show"] <> "" && is_numeric($_GET["show"]) ) ? $_GET["show"] : 30;
$offset = ($pagenum * $limit) - $limit;

try {
    $keyword = trim($_GET["keyword"]);
    if (!empty($keyword)) {
        $sql = "SELECT * FROM noticias WHERE conteudo LIKE :keyword OR titulo LIKE :keyword ORDER BY Nid DESC";
        $stmt = $DB->prepare($sql);

        $likekeyword = "%".$keyword."%";
        $stmt->bindParam(':keyword', $likekeyword, PDO::PARAM_STR);

    } else {
        $sql = "SELECT * FROM noticias WHERE 1 ORDER BY Nid DESC LIMIT :limit, :offset";
        $stmt = $DB->prepare($sql);
        $stmt->bindValue(':limit', $limite, PDO::PARAM_INT);
        $stmt->bindValue(':offset', $offset, PDO::PARAM_INT);
    }

    $stmt->execute();
    $results = $stmt->fetchAll();

    //Faz o calculo do número de registros por página
    $total_registros = $DB->query("SELECT COUNT(*) FROM noticias")->fetchColumn() or die($db->error);
    $paginas = floor($total_registros / $limite);

} catch (Exception $ex) {
    echo $ex->getMessage();
}

//imprime os resultados e os link das páginas(segundo for).
foreach($results as $item){
    printf("id: %d - título: %s - idade: %d
", $item['id'], $item['titulo']);
}

for($i = 1; $i <= $paginas; $i++){
    printf('%d|', $i, $i);
}

Referência:
Paginação script PHP
Comment: Bom não tive oportunidade de testar, entrei em contato com o pessoal da hospedagem e eles disseram que desbloquearam algo no apache quanto ao erro. Mesmo assim testei as alterações que vc postou na resposta e ok, funcionou quase perfeito exceto pelo contadores de paginas que ficou apenas na primeira pagina ( atualizei a pergunta com o codigo da onde exibe os contadores de pagina)
Comment: O link da sua página precisa ter duas informações, o limite por exemplo 30 e o off set que é a 'posição atual' do registro para frente que deve ser exibido. @ArsomNolasco
Answer: Experimenta separar algumas das variáveis das funções, assim consegues visualizar melhor o que se está a passar.
Como já antes disse nos comentários, existem várias alternativas à função count que estavas a usar juntamente com o método fetchAll, e que estavam a ser repetidas desnecessárias vezes.Lembrando também que o fetchAll possui como método predefinido de busca o PDO::FETCH_BOTH.
<?php

$dsn = "mysql:host=localhost;dbname=exemplo;charset=utf8;";
$pagina = isset($_GET['pagina']) ? (int)$_GET['pagina'] : 1;
$pagina = ($pagina > 0) ? $pagina : 1;
$termo_busca = isset($_GET['s']) ? (string)$_GET['s'] : '';
$por_pagina = 3;
$offset = ($pagina -1) * $por_pagina;
$dados = array();
$total = "";
$i = 1;

try {
    $pdo = new pdo($dsn, 'root', '');
    $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
    $sql = "SELECT * FROM noticias WHERE conteudo LIKE :busca OR titulo LIKE :busca ORDER BY Nid DESC LIMIT {$por_pagina} OFFSET {$offset}";
    if($prepare = $pdo->prepare($sql)){
        if($prepare->execute(array(':busca'=>'%'.$termo_busca.'%'))){
            while($linha = $prepare->fetch(PDO::FETCH_OBJ)){
                $dados[] = $linha;
            }
            # condição para corrigir a paginacao quando existe um termo de busca
            if($termo_busca):
                $where = "WHERE conteudo LIKE '%{$termo_busca}%' OR titulo LIKE '%{$termo_busca}%'";
            else:
                $where = "";
            endif;

            $count = $pdo->query("SELECT COUNT(*) FROM notcias {$where}")->fetchColumn();
            $total = ceil($count/$por_pagina);
        }
    }
} catch(PDOException $e){
    die($e->getMessage());
}

# imprimir os resultados e paginar
if($dados){
    foreach($dados as $object){
        print $object->titulo . "
";
    }

    do{
        if($termo_busca):
            print " {$i} ";
        else:
            print " {$i} ";
        endif;

        $i++;
    } while($i <= $total);
} else {
    print "Nenhum resltado encontrado voltar";
}

?>

Esta consulta é interna, e retorna o número de resultados correspondentes ao critério fornecido:
$pdo->query("SELECT COUNT(*) FROM notcias {$where}"); # retorna COUNT(*)=> #numero

A função fetchColumn retorna uma única coluna na seguinte linha do conjunto de resultados. Adicionando à consulta anteriror, ela retorna o resultado exato, sendo que a função COUNT(*) retorna apenas uma coluna para como resultado da consulta.
$pdo->query("SELECT COUNT(*) FROM notcias {$where}")->fetchColumn();

Ou podes ainda usar, a consulta desta forma:
$pdo->query("SELECT * FROM exemplo {$where}")->rowCount();

Existem ainda outras formas de contar o número de linhas existente no banco de dados sem sobrecarregar a memória, só tens de procurar.


Importing CSV data using PHP/MySQL - Mysqli syntax

Question: IN THE BOTTOM OF THIS QUESTION THE FINAL CODE THAT FINALLY WORKED!
Trying to implement this (Importing CSV data using PHP/MySQL). I must be almost there...
notes1: my $sql came straight from copy/paste phpmyadmin (generate php code) and ran just fine in the phpmyadmin.
note2: If I comment the line $sql="DELETE FROM dbase" the code runs just fine (and the table is cleaned).
So if i know my sql is right and my code can run other sqls, why does the below does not run?! Im getting:
Call to a member function execute() on a non-object - for the line
$stmt->execute();

Full code:
<?php
$mysqli  =  new mysqli('localhost','root','pass','dbase');
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}

$sql = "LOAD DATA INFILE \'./myfile.csv\' INTO TABLE tab\n"
. " FIELDS TERMINATED BY \',\'\n"
. " LINES TERMINATED BY \'\\r\\n\'\n"
. " IGNORE 1 LINES";

//$sql="DELETE FROM dbase";

$stmt=$mysqli->prepare($sql);
$stmt->execute();
$stmt->close();
$mysqli->close();
?>

tks in advance!
EDIT:
Made below changes and still not working!
new code:
<?php

$mysqli  =  new mysqli('localhost','root','pass','dbase');
/* check connection */
if (mysqli_connect_errno()) {
    printf("Connect failed: %s\n", mysqli_connect_error());
    exit();
}

/* return name of current default database */
if ($result = $mysqli->query("SELECT DATABASE()")) {
    $row = $result->fetch_row();
    printf("Default database is %s.\n", $row[0]);
    $result->close();
}

$sql = "LOAD DATA INFILE 'C:/xampp/htdocs/myfile.csv' INTO TABLE tab
        FIELDS TERMINATED BY ','
    LINES TERMINATED BY '\\r\\n'
    IGNORE 1 LINES";

echo "
";
echo "
";
echo $sql;
echo "
";
echo "
";
$stmt=$mysqli->prepare($sql);

/* Prepared statement, stage 1: prepare */
if (!($stmt = $mysqli->prepare($sql)))
{    echo "Prepare failed: (" . $mysqli->errno . ") " . $mysqli->error;
}

// NOTE HERE WE'RE DUMPING OUR OBJ TO SEE THAT IT WAS
// CREATED AND STATUS OF PREPARE AND THEN KILLING SCRIPT
var_dump($mysqli);
exit();

//$sql="DELETE FROM intrasdump

$stmt=$mysqli->prepare($sql);
$stmt->execute();
$stmt->close();
$mysqli->close();
?>

What i see in my browser when I ran this is the following:
Default database is dbname.
LOAD DATA INFILE 'C:/xampp/htdocs/myfile.csv' INTO TABLE tab FIELDS TERMINATED BY ',' LINES TERMINATED BY '\r\n' IGNORE 1 LINES
Prepare failed: (1295) This command is not supported in the prepared statement protocol yetobject(mysqli)#1 (19) { ["affected_rows"]=> int(-1) ["client_info"]=> string(79) "mysqlnd 5.0.11-dev - 20120503 - $Id: 40933630edef551dfaca71298a83fad8d03d62d4 $" ["client_version"]=> int(50011) ["connect_errno"]=> int(0) ["connect_error"]=> NULL ["errno"]=> int(1295) ["error"]=> string(68) "This command is not supported in the prepared statement protocol yet" ["error_list"]=> array(0) { } ["field_count"]=> int(1) ["host_info"]=> string(20) "localhost via TCP/IP" ["info"]=> NULL ["insert_id"]=> int(0) ["server_info"]=> string(6) "5.6.11" ["server_version"]=> int(50611) ["stat"]=> string(133) "Uptime: 7993 Threads: 2 Questions: 865 Slow queries: 0 Opens: 75 Flush tables: 1 Open tables: 68 Queries per second avg: 0.108" ["sqlstate"]=> string(5) "00000" ["protocol_version"]=> int(10) ["thread_id"]=> int(117) ["warning_count"]=> int(0) }
Note: If I copy paste the sql string echoed above in to mysql prompt, it runs just fine. That should mean that both the file location issue and the sql string itself are fine, no???
how can this be so hard?!
EDIT 3.
Tks for all answers and comments. Final code version below works:
<?php

$mysqli  =  new mysqli('localhost','root','pass','dbname');
/* check connection */
if (mysqli_connect_errno()) {
    printf("Connect failed: %s\n", mysqli_connect_error());
    exit();
}

$sql = "LOAD DATA INFILE 'C:/xampp/htdocs/myfile.csv' INTO TABLE tab
        FIELDS TERMINATED BY ','
        LINES TERMINATED BY '\\r\\n'
        IGNORE 1 LINES";

//Try to execute query (not stmt) and catch mysqli error from engine and php error
if (!($stmt = $mysqli->query($sql))) {
    echo "\nQuery execute failed: ERRNO: (" . $mysqli->errno . ") " . $mysqli->error;
};
?>

useful notes:

note the file path uses frw-slash instead of windows-default back-slash. Orderwise will just note work. God knows how I figured that one out...
take advantage of the many debugging codes offered in the answers. i guess one effective way to check if your sql is right to echo (echo $sql) it and copy/paste in your sql prompt. don't trust phpmyadmin 'create php PHP code' functionality.
keep in mind 'Prepared stmts don't support LOAD DATA'
Comment: @cerd I meant the \n at the end of the statement string parts, not the line terminator definition, but I now see that was not clear at all...
Comment: I think phpmyadmin assumed you would enclose the query in single quotes so it escaped them.  Just remove the \ before all the ' and I think it should work.  Or enclose the query in ' instead of double quotes "
Comment: As you're using double quotes on the outside you don't need to escape the single quotes.  And not the problem, but the \n and \r are not needed at all, and clutter your string.  You can safely remove them.
Comment: And what @fvu said, I wasn't looking at that :)
Comment: @fvu - actually it is better to specify the LINES. By default without a LINES.. statement, mysql looks for LINES TERMINATED BY '\n' and not '\r\n' in v5.6.
Answer: You should probably put guards in to detect when a prepare call fails:
/* Prepared statement, stage 1: prepare */
if (!($stmt = $mysqli->prepare("INSERT INTO test(id) VALUES (?)"))) {
    echo "Prepare failed: (" . $mysqli->errno . ") " . $mysqli->error;
}

Source: http://www.php.net/manual/en/mysqli.quickstart.prepared-statements.php
I'd also check the path for the LOAD DATA INFILE command. Phpmyadmin might be setting a different path variable. If you try to temporarily put the absolute unix path (and ensure mysql has appropriate permission to that directory), it may clear up the problem.
However, in either case, you should output the error from the mysql client to give you a better idea of what's causing the issue.
Lastly, I'd try to execute the command directly in the MySQL command line client. If there's any "magic" happening in phpmysql or the mysqli php client, it will show up by using the mysql cli client as two of the clients will succeed where one will fail (compared to your assumptions).
I've run into a similiar issue before and it ended up being MySQL's odd way of escaping the delimiter, so I'd look to see that you're escaping the TERMINATED BY parameters correctly.
Comment: The first step that I suggested was to output the error message. Which, as it turns out, is exactly what the issue was: "This command is not supported in the prepared statement protocol yetobject(mysqli)#1" Glad you found your issue though.
Comment: I followed you step-by-step and I'm still not there Homer6. Please take a look at my edit.
Comment: @Homer6 good call on the above code, I added it into the answer above as a means to illustrate the issue along with the work around.
Answer: EDIT for Probable Solution: Prepared stmts don't support LOAD DATA.
If you use mysqli_query - instead of mysqli->prepare ..->execute(); this should work.
So:
//Connect as normal above
$sql = "LOAD DATA INFILE '/myfile.csv' INTO TABLE tab"
. " FIELDS TERMINATED BY ','"
. " LINES TERMINATED BY '\r\n'"
. " IGNORE 1 LINES";

//$sql="DELETE FROM dbase";
// $stmt = $mysqli->query($sql);
// Integrate other posters good recc to catch errors:

//Try to execute query (not stmt) and catch mysqli error from engine and php error
if (!($stmt = $mysqli->query($sql))) {
    echo "\nQuery execute failed: ERRNO: (" . $mysqli->errno . ") " . $mysqli->error;
}

useful still to var_dump($mysqli) here to see result I got access denied as in my env we disallow LOAD FILE, but that tells me the engine successfully parsed and attempted to execute the query.
You need to get better error info, there is a bit that could be going on, here is how I would dig in:
Diagnosis Process to get to this Point:
$sql = "LOAD DATA INFILE ...";
echo $sql;
$stmt=$mysqli->prepare($sql);

// NOTE HERE WE'RE DUMPING OUR OBJ TO SEE THAT IT WAS
// CREATED AND STATUS OF PREPARE AND THEN KILLING SCRIPT
var_dump($mysqli);
exit();

Results on my Box (not the best representation):
  object(mysqli)#1 (18) {
  ...
  string(68) "This command is not supported in the
              prepared statement protocol yet"

2. Other possible errors you would see
Insufficient FILE privs or Improper Directory of file Loc
phpMyAdmin is pretty sandboxed and will work completely differently than mysqli/php.
Address the following from MySQL Docs, and read up on this section. Like I said, LOAD..FILE is a very sensitive operation with a lot of restrictions. LOAD...FILE MySQL Docs

For security reasons, when reading text files located on the server,
  the files must either reside in the database directory or be readable
  by all. Also, to use LOAD DATA INFILE on server files, you must have
  the FILE privilege. See Section 6.2.1, "Privileges Provided by MySQL".
  For non-LOCAL load operations, if the secure_file_priv system variable
  is set to a nonempty directory name, the file to be loaded must be
  located in that directory.
Comment: I followed every step of your answer cerd. I'm indeed getting the same error I read in your box = "This command is not supported in the prepared statement protocol yet". Pls take a look at my edits!
Comment: @Amariani I looked into this. See my edited answer above for probable solution.
Comment: Bingo! Working like a charm! I wonder how did you figured out that 'Prepared stmts don't support LOAD DATA'. Should I post my final solution as an edit in my question for other victims of LOAD DATA? Im a bit new to this stackflow thing. tks a zillion anyways!
Comment: @AMariani - glad that worked. Honestly, I could find no specific references for this error other than comments in the docs from MySQL. You cannot DROP tables either. Perhaps  please accept answer for others who may run into this. With mysql_ being deprecated in php 5.5 many others will hit this when going over to mysqli_, PDO etc.
Answer: What Homer6 states, and probably:
$sql = "LOAD DATA INFILE './myfile.csv' INTO TABLE tab"
. " FIELDS TERMINATED BY ','"
. " LINES TERMINATED BY '\\r\\n'"
. " IGNORE 1 LINES";

Might need to check the path to the file as well.  I don't know what the ./ is relative to in the query.  Maybe not the location of the PHP file(s).
Comment: I had bigger problems using prepare with LOAD DATA, but indeed the your sql string above is the one working!
Answer: You must use the root user of MySQL
If it still fails to run, use this code to check the error output
var_dump(mysqli_error($conn));


How to extract the data from Soap Response in C#

Question: Following is a part of the function which i use to extract data from.
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <MP0118_GetGridHeaderData_001_Result xmlns="http://schemas.datastream.net/MP_functions/MP0118_GetGridHeaderData_001_Result">
         <GRIDRESULT type="LIST.HEAD_DATA.STORED">

               <DATA jsonclass="array">
                  <ROW id="1">
                     <D n="6721">10128</D>
                     <D n="6724">CL</D>
                     <D n="6771">*</D>
                     <D n="6773">ACT</D>
                     <D n="6774">PHON</D>
                     <D n="6775">04-MAR-2018 21:54</D>
                     <D n="6779">MEP-IU</D>
                     <D n="6780">MEP-IU-010</D>
                     <D n="6782">CWP2B19-113</D>
                     <D n="6792">11410</D>
                     <D n="6809"/>
                     <D n="6880"/>
                     <D n="11651">Tap is not working in the Back of the Apt </D>
                     <D n="100410">40977</D>
                     <D n="101312">email@example.com</D>
                     <D n="101313">HANDOVER</D>
                  </ROW>
                  <ROW id="2">
                     <D n="6721">10132</D>
                     <D n="6724">CL</D>
                     <D n="6771">*</D>
                     <D n="6773">ACT</D>
                     <D n="6774">PHON</D>
                     <D n="6775">05-MAR-2018 11:27</D>
                     <D n="6779">GENERAL</D>
                     <D n="6780">GENERAL-005</D>
                     <D n="6782">CWP2B17-303</D>
                     <D n="6792"/>
                     <D n="6809"/>
                     <D n="6880"/>
                     <D n="11651">Flat 303, 304, 305 and 307 asking for Water Activation. Reporter is property manager. Please do the needful.</D>
                     <D n="100410">40981</D>
                     <D n="101312">email@example.com</D>
                     <D n="101313">CAM</D>
                  </ROW>
                  <ROW id="3">
                     <D n="6721">10177</D>
                     <D n="6724">CL</D>
                     <D n="6771">*</D>
                     <D n="6773">ACT</D>
                     <D n="6774">PHON</D>
                     <D n="6775">06-MAR-2018 14:41</D>
                     <D n="6779">MEP-IU</D>
                     <D n="6780">MEP-IU-006</D>
                     <D n="6782">CWP2B19-605</D>
                     <D n="6792">10600</D>
                     <D n="6809"/>
                     <D n="6880"/>
                     <D n="11651">City Walk Building 19, APT 605, The drainage inside whole apartment is smelling bad</D>
                     <D n="100410">41026</D>
                     <D n="101312">email@example.com</D>
                     <D n="101313">HANDOVER</D>
                  </ROW>

               </DATA>
               <FIELDS viewtype="LIST" visibleColumn="true" visiblerowcount="">
                  <FIELD aliasnum="6721" control="text" filterable="+" keyfield="+" label="ctr_code" lookup="" name="ctr_code" order="1" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6792" control="text" filterable="+" keyfield="-" label="ctr_contactinfoid" lookup="" name="ctr_contactinfoid" order="2" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6809" control="text" filterable="+" keyfield="-" label="ctr_contactnote" lookup="" name="ctr_contactnote" order="3" type="MIXVARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6774" control="text" filterable="+" keyfield="-" label="ctr_contactsource" lookup="" name="ctr_contactsource" order="4" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6775" control="text" filterable="+" keyfield="-" label="ctr_created" lookup="" name="ctr_created" order="5" type="DATETIME" visible="+" width="180"/>
                  <FIELD aliasnum="101312" control="text" filterable="+" keyfield="-" label="ctr_createdby" lookup="" name="ctr_createdby" order="6" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6880" control="text" filterable="+" keyfield="-" label="ctr_desc" lookup="" name="ctr_desc" order="7" type="MIXVARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="100410" control="text" filterable="+" keyfield="-" label="ctr_event" lookup="" name="ctr_event" order="8" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="101313" control="text" filterable="+" keyfield="-" label="ctr_mrc" lookup="" name="ctr_mrc" order="9" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="11651" control="text" filterable="+" keyfield="-" label="ctr_note" lookup="" name="ctr_note" order="10" type="MIXVARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6782" control="text" filterable="+" keyfield="-" label="ctr_object" lookup="" name="ctr_object" order="11" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6771" control="text" filterable="+" keyfield="-" label="ctr_org" lookup="" name="ctr_org" order="12" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6779" control="text" filterable="+" keyfield="-" label="ctr_servicecategory" lookup="" name="ctr_servicecategory" order="13" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6780" control="text" filterable="+" keyfield="-" label="ctr_serviceproblem" lookup="" name="ctr_serviceproblem" order="14" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6724" control="text" filterable="+" keyfield="-" label="ctr_status" lookup="" name="ctr_status" order="15" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6773" control="text" filterable="+" keyfield="-" label="ctr_type" lookup="" name="ctr_type" order="16" type="VARCHAR" visible="+" width="180"/>
               </FIELDS>
            </GRID>
         </GRIDRESULT>
      </MP0118_GetGridHeaderData_001_Result>
   </soapenv:Body>
</soapenv:Envelope>

this the code i m using for to filter the data,
#region Namespaces
using System;
using System.Data;
using System.IO;
using System.Net;
using System.Text;
using System.Windows.Forms;
using System.Xml;

using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;

#endregion

///
/// This is the class to which to add your code.  Do not change the name, attributes, or parent
/// of this class.
///
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{

    public override void PreExecute()
    {
        base.PreExecute();

    }

    public override void PostExecute()
    {
        base.PostExecute();

    }

    public override void CreateNewOutputRows()
    {

        IvokeAPICalls(Variables.CRMAPIEndPoint1, Variables.CRMSoapRequest1);

    }

    private static void IvokeAPICalls(string APIEndPoint, string XmlRequest)
    {
        ServicePointManager.SecurityProtocol = (SecurityProtocolType)3072;

        HttpWebRequest httpWebRequest = (HttpWebRequest)(WebRequest.Create(APIEndPoint));
        httpWebRequest.Headers.Add(@"SOAP:Action");
        httpWebRequest.ContentType = "text/xml;charset=\"utf-8\"";
        httpWebRequest.Accept = "text/xml";
        httpWebRequest.Method = "POST";
        httpWebRequest.Timeout = 99999999;
        var fileContent = File.ReadAllText(XmlRequest);
        StringBuilder stringBuilder = new StringBuilder();
        stringBuilder.Append(fileContent);//.Replace("@StartDate", StartDate).Replace("@EndDate", EndDate).Replace("@Operations", Operations);
        XmlDocument xmlDocument = new XmlDocument();
        xmlDocument.LoadXml(stringBuilder.ToString());

        using (Stream stream = httpWebRequest.GetRequestStream())
        {
            xmlDocument.Save(stream);
        }

        using (WebResponse response = httpWebRequest.GetResponse())
        {
            using (StreamReader streamreader = new StreamReader(response.GetResponseStream()))
            {
                string result1 = streamreader.ReadToEnd();
                 MessageBox.Show(result1);

                //Method1
                //xmlDocument = new XmlDocument();
                xmlDocument.LoadXml(result1);

                //foreach (XmlElement xmlElement in xmlDocument.DocumentElement.SelectNodes("//DATA/ROW[*]"))
                foreach (XmlElement xmlElement in xmlDocument.DocumentElement.SelectNodes("//GRIDRESULT/GRID/DATA/ROW[*]"))
                {
                    Console.WriteLine(xmlElement.OuterXml);

                    //MessageBox.Show(xmlElement.OuterXml);
                    string result2 = streamreader.ReadToEnd();
                }

                //Method2

                XmlDocument xmlDoc1 = new XmlDocument();
                xmlDoc.LoadXml(result1);

                XmlNamespaceManager xmlnsManager = new System.Xml.XmlNamespaceManager(xmlDoc1.NameTable);

                xmlnsManager.AddNamespace("soapenv", "http://schemas.xmlsoap.org/soap/envelope/");
                xmlnsManager.AddNamespace("xsi", "http://www.w3.org/2001/XMLSchema-instance");
                xmlnsManager.AddNamespace("xsd", "http://www.w3.org/2001/XMLSchema");
                xmlnsManager.AddNamespace("si", "http://schemas.datastream.net/MP_functions/MP0118_GetGridHeaderData_001_Result");

                XmlNode node = xmlDoc.SelectSingleNode("//si:DATA/ROW[*]", xmlnsManager);
                string name = node.InnerText;
            }
        }
    }

}

this is what I tried to filter the data. but i m not getting.
I want to extract list of all records from response. I m not sure how to go about writing a code for to filter the data.
i want to fetch data from this soap respons exml i tried xml node xpath but didn't work.Please help to fetch data
Answer: from the provided XML, it seems it's divided into two sections, rows and fields.
The fields are just a definitions for the rows. So, what you'll need to do is to make a class model with the fields using aliasnum and name and type of each field.
for instance this :
<FIELD aliasnum="6721" control="text" filterable="+" keyfield="+" label="ctr_code" lookup="" name="ctr_code" order="1" type="VARCHAR" visible="+" width="180"/>

would be converted to this :
public class SoapField
{
    public string Aliasnum { get; set; }
    public string Name { get; set; }
}

here we took only the Aliasnum and the Name, which will be storing aliasnum and name. these two are what we need to map the rows.
now, taking the list of name attributes, we will create another class model for the rows based on it.
public class SoapRow
{
    public string Id { get; set; }
    public string Code { get; set; }
    public string ContactInfoId { get; set; }
    public string ContactNote { get; set; }
    public string ContactSource { get; set; }
}

now you'll need two loops, one to read these fields, the other one will be for the rows.
I have reduced the XML to four fields, just for demonstration purpose, you'll need to complete the rest as required.
Example :
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
   <soapenv:Header/>
   <soapenv:Body>
      <MP0118_GetGridHeaderData_001_Result xmlns="http://schemas.datastream.net/MP_functions/MP0118_GetGridHeaderData_001_Result">
         <GRIDRESULT type="LIST.HEAD_DATA.STORED">
            <GRID>
               <DATA jsonclass="array">
                  <ROW id="1">
                    <D n="6721">10128</D>
                    <D n="6792">11410</D>
                    <D n="6809"/>
                    <D n="6774">PHON</D>
                  </ROW>
               </DATA>
               <FIELDS viewtype="LIST" visibleColumn="true" visiblerowcount="">
                  <FIELD aliasnum="6721" control="text" filterable="+" keyfield="+" label="ctr_code" lookup="" name="ctr_code" order="1" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6792" control="text" filterable="+" keyfield="-" label="ctr_contactinfoid" lookup="" name="ctr_contactinfoid" order="2" type="VARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6809" control="text" filterable="+" keyfield="-" label="ctr_contactnote" lookup="" name="ctr_contactnote" order="3" type="MIXVARCHAR" visible="+" width="180"/>
                  <FIELD aliasnum="6774" control="text" filterable="+" keyfield="-" label="ctr_contactsource" lookup="" name="ctr_contactsource" order="4" type="VARCHAR" visible="+" width="180"/>
               </FIELDS>
            </GRID>
         </GRIDRESULT>
      </MP0118_GetGridHeaderData_001_Result>
   </soapenv:Body>
</soapenv:Envelope>

then using the above classes we can do this :
var rowsList = new List<SoapRow>();
var fieldsList = new List<SoapField>();

//use XDocument instead of xmlDocument it's simpler and works very well with LINQ.
var doc = XDocument.Parse(xml);

// declare the namespace
var ns = doc.Root.GetNamespaceOfPrefix("soapenv");

// get Body
var body = doc.Descendants(ns + "Body").Elements();

// get MP0118_GetGridHeaderData_001_Result
var gridresultheader = body.ElementAt(0).Elements();

// get GRIDRESULT
var gridresult = gridresultheader.ElementAt(0).Elements();

// get GRID
var grid = gridresult.ElementAt(0).Elements();

// get DATA
var rows = grid.ElementAt(0).Elements();

// get FIELDS
var fields = grid.ElementAt(1).Elements();

// define the fields first
foreach(var field in fields)
{
    fieldsList.Add(
        new SoapField()
        {
            Aliasnum = field.Attribute("aliasnum").Value.Trim(),
            Name = field.Attribute("name").Value.Substring(4).Trim() //get the name without (ctr_)
        }
        );
}

// now loop over the rows, and map them to the fields.
foreach (var row in rows)
{
    // get the row id
    var id = row.Attribute("id").Value.Trim();

    //get the row elements
    var d = row.Elements();

    //set the row id first
    var dataRow = new SoapRow
    {
        Id = id
    };

    // go over each element in the row
    foreach (var element in d)
    {
        // get the row n attribute
        var n = element.Attribute("n").Value.Trim();

        // get the name of this element by the n attribute from the fields definition.
        var name = fieldsList.Where(e => e.Aliasnum == n).Select(x => x.Name).First();

        // now just map it using a simple switch statment.
        switch(name)
        {
            case "code":
                dataRow.Code = element.Value.Trim();
                break;
            case "contactinfoid":
                dataRow.ContactInfoId = element.Value.Trim();
                break;
            case "contactnote":
                dataRow.ContactNote = element.Value.Trim();
                break;
            case "contactsource":
                dataRow.ContactSource = element.Value.Trim();
                break;
        }

    }

    // add the row results into the list
    rowsList.Add(dataRow);

}

the results will be in the rowsList.
The code can be further simplified, but I wanted to show each step as clear as possible.
Comment: My heartly thanks for your response. I will try this one let you know
Answer: There is a website https://xmltocsharp.azurewebsites.net/ which makes extract to c# maybe it will helps you
Comment: Thanks bro,From the Soap Response
Comment: How i need to filter the data
Comment: using (WebResponse response = httpWebRequest.GetResponse())
        {
            using (StreamReader streamreader = new StreamReader(response.GetResponseStream()))
            {
                string result1 = streamreader.ReadToEnd();
                 MessageBox.Show(result1);
                xmlDocument = new XmlDocument();
                xmlDocument.LoadXml(result1);
Comment: For Filter the data from response
Comment: foreach (XmlElement xmlElement in xmlDocument.DocumentElement.SelectNodes("//DATA/ROW[*]/D[text()]"))
                {
                    Console.WriteLine(xmlElement.OuterXml);

                    string result2 = streamreader.ReadToEnd();
                }
Comment: I used above code to filter the records but not getting


What to do about heterodox answers in low-frequency questions?

Question: Have a look at this question. Due to poor formatting, the question has only received one upvote.
One of the posters, (who I've seen posting aggressive anti mainstream answers and comments on several occasions) does two things:

Posts a (I suppose) Marxist answer, where the typically labelled labor share is labeled as the inverse of the "exploitation rate".
Comments under the other answer:

Wrong. The labor share is an accounting abstraction that does not reflect actual class dynamics

There's several issues at hand here:

I personally believe we should allow answers from all parts of Economics, as long as they're somewhat credible. Up votes should be a signal of quality of the answer.
This leaves the issue on what to do when this happens on questions with low visibility. I have no clear suggestion on this part.
Disrespect, however, is out of the question. Disagreement with a specific answer should be based on facts. Instead of a long discussion on comments, a separate answer should be the way to go.
There should be a minimum threshold. I don't think we want truther-type people who just claim things that are too far off the basics (hard to define). Answers like the following (same user, deleted) should result in a strongly-worded warning and eventually to removal from the community

In reality GDP, like all economic statistics, are made up and arbitrary. So go for it, whatever you're suggesting.

These are of course initial suggestions, and I'd like to hear what others think.
Comment: Why acting upon "heterodox answers to low-frequency questions" needs a special/modified set of principles compared to the principles applied to, say, "orthodox answers to high-frequency questions"? Your post is not clear about this aspect, which it appears to me to be  critical.
Comment: My broad policy as a moderator has been to rely on the community's various voting mechanisms to demote low quality content and restrict interventions to cases of abuse (sabotage, rude or offensive behavior, etc.—mostly in response to flags). But this policy can be shaped to meet the community's needs.
Comment: @AlecosPapadopoulos Because I assume that badly written heterodox answers in high frequency questions will be punished by readers that understand this and care. In low-frequency questions, idiots can run havoc.
Comment: @FooBar I think the heterodoxy angle of this post is interesting and deserves a separate meta question. (When in doubt, should I defend the Establishment?) I am not a fan of any kind of doxy, but an obvious problem with heterodox users is that if they do persist they are likely to vote enthusastically which may distort the voting process. Especially in unclear questions where non-believers may not be inclined to vote.
Answer: After a comment exchange with the OP, it is my impression that the issue here is more accurately described as

"What to do with low-quality answers in low-frequency questions where the SE mechanisms like  comments or downvotes or flags for moderator
  attention have low probability of functioning effectively since site users
  won't be frequenting the said question and so said quality-preserving actions have low-probability of materializing?"

Well, in a fully-volunteered place like economics.se, the only "policy" I see is "volunteer more!": namely, those who care enough, to substitute for the missing "large general public" in these low-frequency Q&A threads.
To say it more concretely, monitoring economics.se by those members who care enough for the site itself and push their involvement beyond their immediate interests in the economics discipline. This is the only way I see.
As for "rules on excommunicating rogue members", I believe that the much larger than economics.se SE-universe has evolved and prospered without such rules in place, recognizing that it would be very difficult to agree on such rules in the first place (and agreed by whom?), and that what matters is to "know one when you see one" (under the "be nice-offer helpful content" fundamental principle), and then alert the moderators and the community to action.
I am not saying that this is necessarily typical for on-line communities: there maybe others where the degree of "command and control" is much higher, but this does not appear to be the SE-way.
Answer: Here's my decision flow (YMMV), in the hope that it might illustrate some of the issues in your question:
On flagging:
Is the answer (or comment) abusive, offensive, or spam? If so, flag accordingly.
Else
Does the answer at least try to answer the question? No matter how wrong-headedly, how outrageously wrong, however misguided - does it try to answer the question? If not, flag accordingly. Shog9 has written a helpful and characteristically witty post on the subject.
On voting: (this is independent of flagging)
Is it correct and useful, and am I in a good position to assess this? Upvote.
Is it wrong (and am I in a good position to assess this), not an answer, or spam? Downvote.
Examples:

[answer:] In reality GDP, like all economic statistics, are made up and arbitrary. So go for it, whatever you're suggesting.

Downvote. Flag as "not an answer" - it doesn't remotely try to address the question.

[comment]: Wrong. The labor share is an accounting abstraction that does not reflect actual class dynamics

It's terse, but I wouldn't call it "not constructive" or offensive - it is trying (rightly or wrongly) to point out an error in the answer, which is one of the things comments are for. If you're not the answerer whose answer is being commented on, then ignore and move on; or post a comment if you can clear up the commenter's misunderstanding kindly. Ideally, the answerer will then edit the answer to either explain kindly why the comment is incorrect, or to incorporate a correction if the comment is correct.
Comment: It is my answer that got the comment. I did not find it offensive. I also did not find it prudent to argue as the commenter is a troll. It is a shame he misled the OP (not the OP of this meta-post), but I think the OP might have been looking for this kind of answer anyway.
Comment: I agree with the first part, but not on the evaluation of the comment - so average vote of zero.


How to turn on .sass extensions in Ionic 2?

Question: I have set up a basic project and only .scss files are picked up. I would like to write my CSS with the .sass format. How can I turn that on?
Comment: Which version of Ionic2 are you using?
Comment: ionic version 2.1.1
Answer: You can override the app script config files:
https://github.com/driftyco/ionic-app-scripts#overriding-config-files
In my package.json, I added the following:
<code>"config": {
  "ionic_sass": "./config/sass.config.js"
}
</code>
I created the sass.config.js (copy from the github project) and added the sass extension:
<code>includeFiles: [
  /\.(scss|sass)$/i
],
</code>
It's unfortunately impossible to update the watch config file, so I directly edited it in the node module:
Under @ionic/app-scripts/config/watch.config.js
Add the following .sass line below .scss:
<code>'{{SRC}}/**/*.scss',
'{{SRC}}/**/*.sass'
</code>


How can I find simple guitar chords to fit a melody line?

Question: I worked out that the melody line to "Joy to the world" is basically just up and down the D major scale. In fact the first line "joy to the world, the lord is come" is literally the descending melody D, C#, B-A, G, F#, E, D.
It feels like finding guitar chords would be trivial as I know the key and melody but every time I try to work it out based on the notes it sounds wrong. I'm left playing D all the way through!
If I know the key and melody line, how can I safely assign some basic chord changes? Nothing fancy, just a couple of chords per line highlighting the key notes.
Comment: Moving to advanced level, have a listen to re-harmonisation of 'Just the Two of Us', done by Diego Figueiredo and Cyrille. And compare to the original, Grover Washington, I think.
Comment: Listen to existing recordings of the song and see what chords they use. Or look at examples in song books. I'm sure there are perfectly good arrangements that use the same chords that you've already tried, in some order.
Comment: I couldn't find a version that wasn't awful :) so I figured I'd give it a try
Comment: @Mr.Boy interesting ... can you post a link to a version you found awful? I've usually accompanied the beginning descending line with a sixth harmony note below each melody note, on the piano. On guitar you might need to use a separate chord for each note, following the melody, like: D - A/C# - G/B - D/A - G - D/F# - A7/E - D. Then (D/F#) - G - G - A - A - D, and a similar chord run for the next descending melody line. I hope you can decipher it from that textual description. I wouldn't recommend this song as one of the first simple songs to find chords to, for a beginner. :)
Answer: I think I know the trap you have fallen into.
In the score below I circled the first note of each measure in green.
Those four notes map out the D major chord. Naturally you might play D major for each bar. If you are using basic guitar chord, those D chords will be in root position.
However, in the classical style of this music the melody ^3 ^2 ^1 - that's the F# E D in measures 3 to 4 - would not be harmonized with chords D | D. Measure 3 needs to be a chord of the dominant or the D chord in 2nd inversion.
If you are not dealing with classical style, @ggcg's answer about I IV V will serve you well!
The rest of what I have written is just detail about this tune could be harmonized.

A common method for harmonizing a melody will tell you to match melody tones to corresponding chords from the scale. Example, in D major, take the note F#, it is found in the three triads Bm, D, and F#m. One of those chords is a possibility for a harmonization. However, if you do that for all of the tones of a melody, you quickly have a big, jumbled list of possible chords with no sense of how to select from the possibilities. The problem with that is it focuses on a matrix of pitches instead of starting with rhythm and phrasing as the basis for harmonization.
There is a deep connection between harmony, rhythm, and the barline that should be satisfied with a good harmonization.
Before selecting chords assess the rhythm of the music especially the latent harmony of the line. Where does the line pause? Are any chords outlined by the line?
Especially in this style of music phrases end with cadences. If you identify those phrases and select appropriate cadences first, you know where the music needs to go structurally.
Also, we want to think about levels of harmony. The idea is that harmony can be simplified or expanded without loosing the essence of the harmonic functions. Example, a single chord I could be expended to I IV6/4 I. We can use this idea to sort of sketch up from basic harmony to a final harmonization, or to break down an existing harmonization to see its broad organization.
Let's take this existing harmonization and work backwards from the finished product to see the principle applied.

Initially, forget about pitches. Where are the long notes? Measures 2 and 4 get the long notes. Both have a dotted quarter note on beat one. These are our resting points. They should be our harmonic destinations/goals. How about shortest notes? the last notes of measures 1 and 2 are a 16th note and 8th note respectively. Whatever happens on those notes can be considered harmonically subordinate to the main notes. They are details we can gloss over for the moment.
Now let's consider pitches. The first note is D the tonic. Long note, measure 2 is A the dominant. Long note, measure 4 is D the tonic. Also, the end of the phrase - measures 3 to 4 - gives ^3 ^2 ^1 that should leap off the page as a cadential formula.
A general sketch of our pitches so far would fit the rough sketch I  |V I which I tried to indicate with brackets at the bottom.
Work backward from the cadence and fill in the detail. Essentially we want to write the cadence and then connect it to the beginning. The melody segment ^3 ^2 ^1 is a common compound cadence formula. Zero creativity is involved, just fill in the formula. (I used parenthesis to show that the I6/4 can be viewed as a V with an appoggiatura above, but it really doesn't matter, it's just a cadential formula.) That takes care of 2 of 4 measures.
The next question is how to treat the melody segment ^1 ^7 ^6 ^5 for measures 1 and 2. The ^5 in bar 2 is the tricky note. It could be part of a tonic chord or a dominant chord. (We will skip over why the mediant chord wouldn't be used.) This is the moment where a mere matrix of melody tones and matching chords will fail us. We aren't simply matching melody to possible chords. We are trying to connect to whatever chord comes next! This is why it was important to find the cadence first. When we know where we are going we can decide how to get there..
The ^5 in measure 2 could be harmonized with a dominant chord, but then it would be setting up a repeat of the dominant chord in measure 3 and that would cause the harmony to become static. Better to use the tonic chord I in measure 2 as a prolongation of the opening tonic chord in bar 1. Now we have gone from a harmonic sketch to the particular harmony for each of our 4 bar phrase I | I | V | I.
The remaining material is harmonically subordinate. Because this material isn't essential we actually have more flexibility with how it could be handled. This harmonization gives us three common devices:

treat a melody note as a non-chord tone, the ^7 passing tone in measure 1.
prolong a chord with a pedal 6/4 (or passing chord), the IV6/4 between the two I chords of measures 1 and 2.
use a pre-dominant chord (chord ii, IV, or vi), the ii6 in measure 2.

Finally the basic method is...

identify phrases and fill in cadences
sketch in the primary chords
fill in the subordinate material
Comment: Thanks for the detailed answer. You use some notation I'm not familiar with though... What is `^n`? What is 6/4 in your chord notation?
Comment: ^ (the circumflex) means 'scale degree' and 6/4 means second inversion triad, literally a 4th and 6th above the bass note.
Answer: Sounds like you're asking for a couple years of harmony lessons.  ;-).
You could just strum the Dmaj chord.  Another approach is use the DMaj, GMaj and AMaj chords and pick one that has the note in question in it.  These 3 chords (I, IV, V) cover the entire major scale.
There are many rules and regulations w/r to harmonizing a melody in the classical sense, too many to quote here.  But the above statement is start.
Comment: Simple is good :)
Comment: Using I, IV, V to harmonize is the starting point.
Answer: If you are able (musically experienced) to sing the bassline to a given melody you will easily find the fitting chords and very often you'll find several solutions. Start playing "this old house of rocky docky", joyful, joyful, "silent night". go by googlesearch: songtitle and "chords", and by practice your analytic ear  will get trained.
most songs can be accompained with tonica, dominante subdominante, in your case D.G and A7. many songs have a dominante of the dominante (V/V) in the half-end, that means before the repetition of the first line.
there are a few patterns of chord progressions to accompain almost all songs:
the best way to find the right chords is to get skilled in playing following chord progression using the circle of fifths:
I-V-I   tonica dominante tonica
I-V  V-I
I-IV-V-I "cadence"
I-V-V-I; IV-I-V-I (the mozart scheme)
then the well known 12 bar blues scheme
and the sub-dominante cadence (jazz cadence)
I-I7-IV-IVm-I-V-I (e.g. oh when the saints go marchin' in)
most popular:
many pop songs have the following cadence
I-VIm-IIm-V7
and some variations of this ...
with a lot of practice of it you will learn to find out the right chords!
concerning joy to the world:
this is just very special and difficult .. as it is a barock tune of handel:
try to find the right bass line (think of counterpoint)
as the melody goes down you try to sing a bass line that goes up
and try to accompaigne  the scale do re mi fa ... up and down.
Answer: There is often a myriad of different ways in which to harmonise a melody. The 'most correct' one to a lot of people will be the one they heard most. Only because it's been imprinted into their brains.
One could play a different chord for each melody note. That gets tedious and impractical in a lot of songs. One could water it down to two or three chords. Yes, we've all heard those sort of renditions! But sometimes that's all that's necessary. We could check the (main) notes in each bar, and find an appropriate chord that fits. This exercise comes into every music course at some point at university, I guess.
Basically there's not one 'correct' way to harmonise a song. A lot of jazz proves this statement.
As an exercise, take a simple tune - Happy Birthday, Silent night, for example, and come up with half a dozen different harmonies. Don't gravitate towards 'the original' - except just once - but see (listen) what else works, without changing the tune itself. And, yes, you're probably well aware that you're allowed to use non-diatonic notes in your chords!
As a more specific piece of advice, playing Silent Night in C, there are opportunities for diminished chords, and a nice D9 towards the end - not used as V/V...   Very simple starter 'rule' - make the chord tone/s fit the melody.


Better way to generate random number that is not already used?

Question: I need to generate a random 9 digit number that is not already used.
This is the way I currently have and it's working perfectly, but maybe this won't be too great when the table grows:
while true
    random_number = 9.times.map{rand(9)}.join
    found = Document.find_by(dfe_number: random_number)
    return random_number if found.nil?
end

Is there a better way to do this? Or a faster way? Thank you all.
Comment: In general, if you add date/time of the generation to the output number it will be good, but... you  don't have enough space with only 9 digits :(
Comment: @J_P Thanks for the suggestion, maybe hour+minutes+day + 3 random numbers would work better, just to be more safe...
Comment: "Better" in what way? "Faster" than what? I had a very similar situation just this past week and went with (essentially) the approach you have. Only, I used SecureRandom to generate my number instead of rolling my own. And, I put it in an AR call_back. But, essentially the same.
Comment: @jvillian I guess this is one of the better ways. "Better and/or Faster" I think I was hoping to see some ruby on rails magic or something, you never know xD
Answer: I am making the assumption that you want to save new instances of Document each with a unique, random dfe_number.
In this case you should take an "optimistic" approach by assuming a collision is very improbable.
Therefore you should:

define a DB unique constraint on documents.dfe_number
get a new random dfe_number - using your method or '%09d' % rand(10 ** 9)
try to insert the new instance
catch any ActiveRecord::RecordNotUnique and retry

P.S.: You should make sure that the expected number of Document instances is orders of magnitude smaller than 10 ** 10 or, if that cannot be guaranteed, use some more digits.
Comment: @GustavoGabriel Is two-column uniqueness DB index not possible in your case? I imagined that perhaps you have another column like `company_id` alongside `dfe_number` in the `Documents` table? If you are using a multi-threaded Rails server like puma, or multi-process, or have multiple application servers, then it's best to add uniqueness index if this `dfe_number` is really really important to be unique.
Comment: I didn't post in the question to make it more simple, but dfe_number cant be unique, I can have the same dfe_number for multiple companies. If I went with your way, I'd have to validate uniqueness in multiple fields...
Comment: Your approach needs at least 2 SQL queries, one for checking, one (I assume) for inserting. You could bring this down to 1 if you could somehow create a DB unique index that prevents inserting when required uniqueness is violated, only then trying another random number. You would have to give more details on your requirements, though.
Comment: The problem is, I have to do some other processing after I generated the number, so I think there isn't too much of a choice here. I asked the question to see if I could improve the loop somehow, or see if it had a faster way...


Rule on Being fouled shooting at the wrong basket

Question: Let's say a player gets fouled while shooting at the opponent's basket either intentionally or unintentionally (deflects it in or makes it accidentally passing it cross-court while bringing the ball up, or they are just confused after exchanging ends, etc.).
Is it a 2-shot foul if they miss?  Or 3-shot if they are in 3-point range?  Does the basket count for the other team if they make it while being fouled?  Is it only one free throw if the shot at the wrong basket is made, like it is when you make a shot at your own basket?
Answer: A shooting foul can only occur when trying to shoot at the basket which will increase your own team's score.
Therefore, if you were somehow shooting at your own basket and got fouled, it would be the same as a normal non-shooting foul. You would not get free throws unless you were in the "bonus".
If you actually made the basket, I think it would be waved off because it is a foul on the team who the score would go to. So just as a basket does not count when an offensive player commits a charge (or other offensive foul), a "wrong-way" basket would not count when the defending team has committed a foul.
In fact, this is the most likely way that your scenario would occur. Player goes up for a rebound, is fouled and makes the ball accidentally go into the basket. Foul would be called with "no basket".

Here is the official definition of a "Field Goal Attempt" from the NBA (http://www.nba.com/analysis/rules_4.html?nav=ArticleList):

A field goal attempt is a player's attempt to shoot the ball into his
  basket for a field goal.

Attempt to shoot the ball into "his basket" means the basket which would give his team points. Trying to shoot the ball into the other team's basket does not qualify under this definition and therefore does not count as a "field goal attempt".
Free throws are granted for fouls during a field goal attempt, so since "shooting at the wrong basket" is NOT a field goal attempt, no free throw would be given.

In another section of the same rules (http://www.nba.com/analysis/rules_12.html?nav=ArticleList) it says:

A personal foul assessed against an offensive player which is neither an elbow, punching or flagrant foul shall be penalized in the following manner:
  (1) No points can be scored by the offensive team

Which I take to mean that NO TEAM can score points on a play where they also commit a foul.
Comment: do you have a rules cite for that? If you're going to answer a technical rules question, you'll need to back it up with actual rules citations.
Comment: fine, there is the citation.
Comment: Thanks for the feedback.  I wasn't aware of the stipulation in the rulebook that says "A shooting foul can only occur when trying to shoot at the basket which will increase your own team's score."   I know it is uncommon for players to intentionally shoot at the wrong basket (not unlike a football player running the wrong way to score a TD), but it does happen from time to time, most often I think after an opening tip of a game or overtime.
Comment: So if the defensive player in that case was also confused, and was trying to stop the shot, say even fouling the player with the ball while dunking the basketball into the wrong basket, the points would still be waved off?  So in that case the foul itself ends the play?
Comment: @DonaldC.Richmond here is the full Rules page: http://www.nba.com/analysis/rules_index.html - if you look at Rule 5 (Scoring), Section 1 it actually says that: d. A field goal accidentally scored in an opponent's basket shall be added to the opponent's score, credited to the opposing player nearest the shooter and mentioned in a footnote.
e. It is a violation for a player to attempt a field goal at an opponent's basket. The opposing team will be awarded the bal at the free throw line extended.
Comment: So E. means that if he does it intentionally, it will not count because it is a foul to even attempt the shot. And D. means that if it's an accident it would count, but in Rule 12 (Fouls) Section 3 it says: A personal foul assessed against an offensive player shall be penalized in the following manner:  (1) No points can be scored by the offensive team. Which I would take to mean that a team cannot score points if they commit a foul on the play. - I will add this into the answer above.


How useful is estimate of accuracy for cross-validation in case of imbalance in class distribution

Question: I have about 4000 instances of one class and 38000 instances of another. I used the DAAG library and I got the following result:
<code>> fit2 <- glm(formula=y~x1+x2,data=rec,family=binomial)
> library(DAAG)
> cv.binary(fit2)

Fold:  5 3 9 10 1 7 2 4 8 6
Internal estimate of accuracy = 0.911
Cross-validation estimate of accuracy = 0.911
</code>
In case of class imbalance in general, accuracy is not a very useful measure. Is this the same accuracy? How useful is the result I obtained?
Answer: You are correct that overall accuracy is not a useful measure in this case, since simply predicting the more common class all the time will give an accuracy of 38000/42000=0.905. There are several approaches you can use in this case:

Measure the accuracy separately for each class, and take the mean.
Use the Matthews Correlation Coefficient, which measures the correlation between your predictions and the real class labels.
Change your cross-validation procedure so that the same number of each class is held out for testing on each fold. This may require some recalibration of the learned model, since the model will be expecting unbalanced data (and so may have a strong prior about which class to predict).
Use more data about the problem domain to adjust your cost function. Are mistakes in either class equally bad? Rather than fitting a model which counts all mistakes equally, your model could weight mistakes in the smaller category higher than mistakes in the larger category.

I'm not very familiar with R and DAAG, so I can't offer specifics about how these would be implemented in this framework.


Nginx Rewite - Matching End Urls not working

Question: Trying to redirect urls like these:
http://example.com/vehicles/cars/author/xyz
http://example.com/plane/author/xyz

to these:
http://example.com/profile/xyz

I tried this Nginx Rewrite, but its not picking up the match and redirecting:
rewrite ^/author/(.*)$     http://example.com/profile/$1 last;

What have I got wrong here?
Answer: ^/author/(*)$ means that it starts with /author/. The URLs you want to rewrite don't start with /author/. You need something like this:
rewrite ^/.*/author/(.*)$ http://example.com/profile/$1 last;

But I think this is better and probably more robust:
rewrite /author/([^/]+)/?$ http://example.com/profile/$1 last;

Comment: `^` is the standard regular expression for the start of line
Comment: Aha.... that makes sense. I thought `^` starts the match from anywhere. Thank you @AntonisChrisofides


Trying to load in a url

Question: I'm trying to load in a url that is a zip file..
Right now I read in the file to get the size.  Then create an array of ints with that size,  and then read in the file again and store it in my array

Is there a way to get the size of the url so I do not have to read it in twice?
Is there a way to change the size of the array as I read it in so I do not have to get the size ahead of time.

It seems silly to read in the file 3 times
Comment: Use, for example, [`java.io.ByteArrayInputStream`](http://docs.oracle.com/javase/8/docs/api/java/io/ByteArrayInputStream.html) instead of using an array directly.
Comment: It is very silly to read it 3 times.  Use a buffer.  http://stackoverflow.com/questions/921262/how-to-download-and-save-a-file-from-internet-using-java
Comment: Are you trying to save the file to your computer or would you like to cache it in the code?
Comment: you should probably create an array of `byte` instead of `int`
Answer: It depends on how your server/service has been implemented. Try reading the "HTTP Header" Content-Length first e.g
URL url = new URL("http(s)://yourfileURL");
URLConnection urlConnection = url.openConnection();
urlConnection.connect();
int file_size = urlConnection.getContentLength();

or
urlConnection.getHeaderField("Content-Length");

See Content-Length

Trying to load in a url


Number of positive integers with precisely four mutually incongruent primitive roots

Question: How would one determine this?  I can see that 11 and 13 each have 4 primitive roots.  There are two others apparently.  11 and 13 are prime but the problem is not restricted to mod p.  I know that if n has a primitive root, then n is 2, 4, a power of p, or double a power of p.  I believe that the remaining two integers are 22 and 26, which happen to be 2x11 and 2x13. I am not seeing why 22 has to have the same number of primitive roots as 11 (after all the primitive roots are not the same), for example.
Answer: The group of units mod $n$ has order $\phi(n)$, where $\phi$ is Euler's totient function, and if this is cyclic the number of primitive roots is $\phi(\phi(n))$.  We have
$$\eqalign{\phi(\phi(11)) &= \phi(10) = 4\cr
   \phi(\phi(13)) &= \phi(12) = 4\cr
   \phi(\phi(22)) &= \phi(10) = 4\cr
   \phi(\phi(26)) &= \phi(12) = 4\cr}$$
so all these cases have $4$ primitive roots.
You might start by determining which $m$ have $\phi(m)=4$.  Then which of those can be $\phi(p^k)$ or $\phi(2 p^k)$ for odd prime $p$.
Comment: Thank you this is very helpful.  I see how this helps find them.  I see how we can find 11, 13, 22, 26 using your help above.  How do I know there is not another number larger than 26 which also works?  I have not yet taken abstract algebra.
Comment: I think this would be because applying Euler's totient function twice to n where n is p^k or 2p^k (and where n is larger than 26) yields numbers greater than 4.
Answer: If $p$ is an odd prime number, then the Chinese remainder theorem gives a ring isomorphism
$$\Bbb{Z}/2p\Bbb{Z}\ \longrightarrow\ (\Bbb{Z}/2\Bbb{Z})\times(\Bbb{Z}/p\Bbb{Z}):\ k+2p\Bbb{Z}\ \longmapsto\ (k+2\Bbb{Z},k+p\Bbb{Z}).$$
This in turn yields a group isomorphism between the groups of units
$$(\Bbb{Z}/2p\Bbb{Z})^{\times}\ \longrightarrow\ (\Bbb{Z}/2\Bbb{Z})^{\times}\times(\Bbb{Z}/p\Bbb{Z})^{\times}.$$
Of course $(\Bbb{Z}/2\Bbb{Z})^{\times}$ is trivial and so we get an isomorphism
$$(\Bbb{Z}/2p\Bbb{Z})^{\times}\ \longrightarrow\ (\Bbb{Z}/p\Bbb{Z})^{\times}:\ k+2p\Bbb{Z}\ \longmapsto\ k+p\Bbb{Z},$$
with inverse given by
$$(\Bbb{Z}/p\Bbb{Z})^{\times}\ \longrightarrow(\Bbb{Z}/2p\Bbb{Z})^{\times}:\ k+p\Bbb{Z}\ \longmapsto\ \begin{cases}k+2p\Bbb{Z}&\text{ if $k$ is odd}\\k+p+2p\Bbb{Z}&\text{ if $k$ is even}\end{cases}$$
Because these two groups are isomorphic, their numbers of generators (primitive roots) are equal.


Textboxes exceed width of  container

Question: I have a few textboxes set to width: 100% inside a
 with the width of 350px, they go outside of the div on the right side, and I don't understand, why?
See this fiddle
Also, code here (same as fiddle):

    <form method="post" action="~/AJAXcalls/InsWrkOAJAX.cshtml">

            Primary
            <input type="radio" name="Type" id="typ4" value="4" class="radioSelect">

            Secondary
            <input type="radio" name="Type" id="typ5" value="5" class="radioSelect" />

            Assistance
            <input type="radio" name="Type" id="typ6" value="6" class="radioSelect" />

                Squat
                <input class="radioSelect" type="radio" name="Exercise" id="hej1" value="1" />

                Benchpress
                <input class="radioSelect" type="radio" name="Exercise" id="hej2" value="2" />

                Deadlift
                <input class="radioSelect" type="radio" name="Exercise" id="hej3" value="3" />

                <input id="dailyPR" placeholder="Daily Max..." name="dailyPR" type="text" size="50" value="" class="hideThis insertWorkoutBoxes" />

                <input placeholder="Weight..." name="Kg" type="text" size="50" value="" class="insertWorkoutBoxes" />

                <input placeholder="Number of sets..." name="Sett" type="number" size="50" value="" class="insertWorkoutBoxes" />

                <input placeholder="Number of reps..." name="Rep" type="number" size="50" value="" class="insertWorkoutBoxes" />

            <input placeholder="" type="text" spellcheck="false" autocomplete="off"
                   class="datepicker workoutVariSelect capitalFirst" name="Date" value="" readonly="readonly" />

Add

    </form>

CSS:
.AddBtn {
    padding: 10px 70px 10px 70px;
    background-color: white;
    cursor:pointer;
}
.centerthis {
    text-align:center;
}
.radioSelect {
    width: 20px;
    float: right;
    margin: 0px 115px 0px 0px;
}
.radioLabel {
     float:left;
     margin-left: 80px;
}
.insertWorkoutFormHolder {
    width: 350px;
    margin:auto;
}
.workoutVariSelect {
    width: 100%;
    padding: 11px;
    border: 1px solid white;
    -webkit-appearance: none;
    -moz-appearance: none;
    text-indent: 1px;
    text-overflow: '';
    font-family: Verdana;
    font-weight: 500;
    background: #fff center right 10px no-repeat url('Images/pil.jpg');
    background-size: 12px 8px;
    cursor: pointer;
    user-select: none;
    margin: 0;
    vertical-align: middle;
}
.insertWorkoutBoxes {
    border: 1px solid white;
    width: 100%;
    margin: 5px 0px;
    padding: 11px;
}

Answer: There are some default paddings and borders set on most of the form element such as <input type="text">. You can apply box-sizing: border-box along with the width: 100%, so that padding and border will be part of total width.

div {
  width: 200px;
  outline: 1px solid aqua;
}

input {
  width: 100%;
}

.border-box {
  box-sizing: border-box;
}

  <input type="text">

  <input type="text" class="border-box">

Answer: The reason is the padding in the input field which is not considered in width 100%. You should use box-sizing: border-box;
Comment: please check existing answers before submitting yours

Textboxes exceed width of  container


2 outs with bases loaded, runner misses second

Question: This happened the other day in a game I was watching.
There were 2 outs with the bases loaded. The batter hit ball over the left fielder's head.  He reached the 3rd base, and 3 runs scored.  The defensive team appealed that the batter missed the second base, umpire later called the batter out.
How many runs should count?
Answer: This isn't a force play, so the batter is out either when he is tagged out or when the ball reaches the base.
If there were runners behind that would otherwise score, then per 5.09(d), they could not score if the runner ahead made the third out on appeal.  But in your scenario, there are no runners behind the play.
So all 3 preceding runners score.  The third out happened after they reached home.  If the defense noticed the miss immediately and threw to second rather than to try to tag someone at home, then it would depend on the timing of when the base is tagged and when the runners reach home.  Only the runners reaching the plate prior to the third out can score.
Comment: The runner missing second base does not institute a force play, so the out cannot be recorded simply by tagging second base after the runner has run by. The defense must either tag the runner, or call time after the runner scores to execute an appeal play.


SyntaxError (amazon-sagemaker-stock-prediction/dbg-custom-rnn.ipython)

Question: I'm running the code cell below from https://github.com/aws-samples/amazon-sagemaker-stock-prediction/blob/master/notebooks/dbg-custom-rnn.ipynb , on SageMaker Notebook instance. For more information, here is the link to the error I posted on GitHub: https://github.com/aws-samples/amazon-sagemaker-stock-prediction/issues/7.
<code>%%time
Instantiate estimator with container image of artifact and backend EC2 instance(s)
rnn = Estimator(image,
role, 1, 'ml.c5.18xlarge',
output_path=output_location,
base_job_name = base_job_name,
sagemaker_session=session)

rnn.set_hyperparameters(**hyperparameters)
Train the model
rnn.fit(data_location)
estimator_job = rnn.latest_training_job.job_name
model_archive = "{}/{}/output/{}/output/model.tar.gz".format(artifactname,interval,estimator_job)
print("Estimator created at completion of training job {}".format(estimator_job))
</code>
And I hit the error below:
<code>Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.
's3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.
2020-08-12 15:58:15 Starting - Starting the training job...
2020-08-12 15:58:17 Starting - Launching requested ML instances......
2020-08-12 15:59:38 Starting - Preparing the instances for training...
2020-08-12 16:00:16 Downloading - Downloading input data...
2020-08-12 16:00:22 Training - Downloading the training image...
2020-08-12 16:01:13 Uploading - Uploading generated training model.2020-08-12 16:01:08.786584: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-08-12 16:01:08.786650: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Starting the training.
Hyperparameters file : {"target_stock": "BMW", "lag": "10", "interval": "D", "batch_size": "4096", "covariate_columns": "StartPrice, MinPrice, MaxPrice", "percent_train": "85.0", "covariate_stocks": "CON, DAI, PAH3, VOW3", "dropout_ratio": "0.1", "num_epochs": "1000", "target_column": "EndPrice", "horizon": "5", "num_units": "256"}
Hyperparameters initialized
Loading data from : /opt/ml/input/data/training/resampled_stockdata.csv
Loading data from : /opt/ml/input/data/training/resampled_stockdata.csv
Training data loaded
100 Stock symbols found.
Records for 65 trading days found.
0-CON#0111-DAI#0112-PAH3#0113-VOW3
Exception during training: invalid syntax (core.py, line 314)
Traceback (most recent call last):
File "/opt/program/train", line 255, in train
traindata.to_csv(os.path.join(model_path, trainfile))
File "/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py", line 3020, in to_csv
formatter.save()
File "/usr/local/lib/python3.5/dist-packages/pandas/io/formats/csvs.py", line 157, in save
compression=self.compression)
File "/usr/local/lib/python3.5/dist-packages/pandas/io/common.py", line 344, in _get_handle
from s3fs import S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/__init__.py", line 1, in <module>
from .core import S3FileSystem, S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/core.py", line 8, in <module>
from fsspec import AbstractFileSystem
File "/usr/local/lib/python3.5/dist-packages/fsspec/__init__.py", line 10, in <module>
from .mapping import FSMap, get_mapper
File "/usr/local/lib/python3.5/dist-packages/fsspec/mapping.py", line 2, in <module>
from .core import url_to_fs
File "/usr/local/lib/python3.5/dist-packages/fsspec/core.py", line 314
out[0] = (f"{out[0][1]}://", out[0][1], out[0][2])
^
SyntaxError: invalid syntax
2020-08-12 16:01:19 Failed - Training job failed
UnexpectedStatusException Traceback (most recent call last)
<timed exec> in <module>

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
497 self.jobs.append(self.latest_training_job)
498 if wait:
--> 499 self.latest_training_job.wait(logs=logs)
500
501 def _compilation_job_name(self):

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
1114 # If logs are requested, call logs_for_jobs.
1115 if logs != "None":
-> 1116 self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
1117 else:
1118 self.sagemaker_session.wait_for_job(self.job_name)

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3075
3076 if wait:
-> 3077 self._check_job_status(job_name, description, "TrainingJobStatus")
3078 if dot:
3079 print()

~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
2669 ),
2670 allowed_statuses=,
-> 2671 actual_status=status,
2672 )
2673

UnexpectedStatusException: Error for Training job dbg-custom-rnn-D-BMW-2020-08-12-15-58-15-812: Failed. Reason: AlgorithmError: Exception during training: invalid syntax (core.py, line 314)
Traceback (most recent call last):
File "/opt/program/train", line 255, in train
traindata.to_csv(os.path.join(model_path, trainfile))
File "/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py", line 3020, in to_csv
formatter.save()
File "/usr/local/lib/python3.5/dist-packages/pandas/io/formats/csvs.py", line 157, in save
compression=self.compression)
File "/usr/local/lib/python3.5/dist-packages/pandas/io/common.py", line 344, in _get_handle
from s3fs import S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/__init__.py", line 1, in <module>
from .core import S3FileSystem, S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/core.py", line 8, in <module>
from fsspec import AbstractFileSystem
File "/usr/local/lib/python3.5/dist-packages/fsspec/__init__.py", line 10, in <module>
from .mapping import FSMap, get_mapper
File "/usr/local/lib/python3.5/dist-packages/fsspec/map
</code>
Could anyone explain for me how to solve this error.
Thank you so much!
Answer: Rename s3_input to TrainingInput and remove warning message.
May be this will help you https://github.com/aws/sagemaker-python-sdk/pull/1680
Comment: Thank you, I will run it again


Unexpected end of command aleatório no firebird

Question: Estou tendo um problema no sistema que desenvolvo, por algum motivo recebo a mensagem:

Unexpected end of command - line 1, column 214

Mas não encontro qual SQL está ocorrendo o problema. Pelo que achei na internet, geralmente se trata de problemas com uso de procedures ou SQLs executados em blocos, mas no meu caso não utilizo nenhum dos dois. Existe uma maneira de identificar qual é esse SQL problemático via Firebird ou algo do gênero?
Comment: Em qual momento ocorre o erro? Qual o comando que foi executado antes da ocorrência do erro?
Comment: O erro ocorre em alguma função especifica ? Pelo que me lembro (nem mais instalado tenho ...) um Query1.SQL.SaveToFile(´C:PastaArquivo.Txt´) salva o sql num arquivo , daí execute isto remotamente
Comment: Esse é o problema que estou tentando identificar, não consegui pegar em qual SQL que está ocorrendo esse problema, eu fiz uma vasculha nos fontes para verificar qual SQL, mas não achei.
Comment: try...except... captura o erro cara, coloca para salvar o sql executado
Answer: Opa, tudo bem ?
Você poderia postar o SQL que ajudaria muito.
Uma das técnicas que uso em casos desse tipo :
1 - Query.Savetofile('C:\sql.sql') -- Para obter o SQL e rodar a query para identificar o problema.
2 - Ou usar Try excepet para capturar o erro.
3 - Delphi Options >Debugger Option > Embarcadero Debugge> Language Exception, desmarca as opções que está em "Exception types ton ignore ".
4 - Quando der o erro clica em Break.
Comment: Sua resposta pode ser melhorada adicionando informações fundamentais. Basta [edit] para adicionar mais detalhes, como citações ou documentação para que outras pessoas possam confirmar que sua resposta está correta. Você pode encontrar mais informações sobre como escrever boas respostas [na Central de Ajuda](/help/how-to-answer).
Comment: o grande problema é que eu não sei exatamente qual SQL que gera o erro, só acontece em produção
Comment: Você precisa mapear passo a passo, para reproduzir o problema. Exemplo verificar se é na geração de algum relatório, ou se é ao abrir a tela, se for ao fazer a pesquisa.

Para poder te ajudar você poderia dizer em qual passo no seu sistema  ocorre o erro. em qual tela ...
Answer: Você pode tentar colocar um break point no evento que gera o erro, por exemplo, no click do botão, e vai dando steps, até chegar na linha que da o erro, feito isto poste novamente, vai dar pra ter uma ideia melhor... pode ser comando sql gerado pelo componente, como tambem pode ser algum problema na concatenação de comandos em tempo de execução, onde, olhando tudo parece bem, mas executando o script montado se torna problematico. fico no seu aguardo
Comment: Por favor, adicione mais detalhes para expandir sua resposta, como um código que funcione ou citações de documentação.
Comment: O problema em questão é que ocorre isso em produção, na máquina de um cliente aparentemente. Eu tentei fazer isso localmente, na minha máquina e meu ambiente, e não ocorre isso


colocar Thread.sleep no metodo Call do Retrofit sem travar a UI Thread (principal)

Question: Bom pelo que eu entendi o método call.enqueue(new Callback<Void>() { inicia uma thread para se comunicar com webservice. Então pensei em retardar o método colocando um thread.sleep(2000) para exibir meu ProgressDialog percebi que colocando o sleep não interrompia a thead criada pelo retrofit e sim a principal Ui thread  travando o ProgressDialog.
Tem algum jeito para colocar um Thread.sleep dentro do método call.enqueue para simular uma demora de resposta do webserver?
package com.amazongas.paulo.crudbasicandroid.dao;

import android.util.Log;

import com.amazongas.paulo.crudbasicandroid.api.ApiCallback;
import com.amazongas.paulo.crudbasicandroid.api.DataService;
import com.amazongas.paulo.crudbasicandroid.model.Contato;
import com.google.gson.Gson;

import retrofit2.Call;
import retrofit2.Callback;
import retrofit2.Response;

public class ContatoDao {

    public void salvar(Contato contato, final ApiCallback callback){

        Gson gson = new Gson();

        String json = gson.toJson(contato);

        DataService dataService = DataService.retrofit.create(DataService.class);

        Call<Contato> call = dataService.gravar(contato.getNome(),contato.getTelefone());

        call.enqueue(new Callback<Contato>() {
            @Override
            public void onResponse(Call<Contato> call, Response<Contato> response) {

                if(response.isSuccessful()){

                    //Simular retardo do server

                    try {
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }

                    callback.onResponse(response.body() != null);

                    //Boolean success = response.body().getSuccess();
                    int lastId = response.body().getCod();
                    Log.d("resultado","JSON: "+lastId);

                }
            }

            @Override
            public void onFailure(Call<Contato> call, Throwable t) {
                call.cancel();
                callback.onResponse(false);
                Log.d("resultado","JSON: ERROR");
            }

        });
    }
}

Comment: De acordo com https://square.github.io/retrofit/, `On Android, callbacks will be executed on the main thread`, ou seja, não há como simular um tempo de espera pelo callback do Retrofit sem travar a UI. Você pode mockar o webserver e fazer a request demorar 2segs... Parece que eles tem uma API de mock https://github.com/square/retrofit/tree/master/retrofit-mock.
Answer: Você pode usar o postDelayed:
Handler handler = new Handler();
handler.postDelayed(() -> salvar(contato, callback), 2000);


Como realizar o parse de uma string tendo como separador espaço duplo a esquerda de um caracter?

Question: Estou com dificuldade para realizar o parse de uma string, de modo a transformar a mesma em um array associativo. Para exemplificar, veja um exemplo da string citada:
Titulo  Valor  Desc
item1   10     Descrição aqui
novo aqui     AB  Mensagem
Label  text_1      Descrição...

Obs: cada linha da string acima está lendo processada horizontalmente, isso ocorre pois o processo transforma cada linha em um array e a processa individualmente.
Na primeira linha tenho o cabeçalho das colunas, já nas linhas seguintes tenho os valores, que podem conter múltiplos espaços a esquerda da coluna. Entretanto, ao observar, podemos notar que a esquerda de cada coluna existem 2 espaços em branco, o que acaba por ser um delimitador (separador) de colunas.
Não consegui criar uma regex que consiga extrair os valores. Meu objetivo final seria um array como este:
Array
(
    [0] => Array
        (
          'Título' => 'item1',
          'Valor' => '10',
          'Desc' => 'Descrição aqui',
        )
    [1] => Array
        (
        //...

Comment: Tem certeza que no arquivo são espaços em branco mesmo? Se forem espaços, o que deverá acontecer caso algum valor possua 2 ou mais espaços seguidos, algo como `"Anderson      Woss"`? É apenas um valor.
Comment: Sim, certeza que são espaços em branco. Os valores quando são inseridos não podem conter múltiplos espaços em branco entre os caracteres, porém, não sei por qual motivo, podem conter infinitos espaços a direita. Porém, antes de cada nova coluna sempre haverá 2 espaços em branco.
Answer: Regex nunca foi meu forte, mas depois de pesquisar varias exemplos, cheguei a uma possível solução:
preg_split('/\s\s+/', $string_aqui);

Nos testes iniciais funcionou.
Comment: Poderia ser apenas `\s{2,}` que também casa com 2 ou mais espaços em branco (incluindo quebras de linha, tabulações, etc)
Answer: //A string inicial
$data = "Titulo  Valor  Desc
item1   10     Descrição aqui
novo aqui     AB  Mensagem
Label  text_1      Descrição...";

//Divide a string pelas quebras de linha
$rows = explode("\n", $data);

//Array com o cabeçalho
$header = array_map(
    //Executa a função trim em todas as string resultantes
    "trim",
    //Divide por dois espaços
    explode(
        "  ",
        //Pega a primeira linha (isso já remove ela do array "$rows")
        array_shift($rows)
    )
);

//Cria uma variável para guardar os dados organizados
$parsed = array();

//Percorre as linhas
foreach ($rows as $row) {
    $row = array_map(
        //Executa a função trim em todas as string resultantes
        "trim",
        //Filtra o array para remover índices em branco
        array_filter(
            //Divide por dois espaços
            explode("  ", $row),
            function($v) {
                return $v !== "";
            }
        )
    );

    //Cria um novo array a partir de dois,
    //o primeiro será usado para os índices
    //e o segundo para os valores
    $parsed[] = array_combine($header, $row);
}

//Mostra o resultado
print_r($parsed);

O resultado do print_r:
Array
(
    [0] => Array
        (
            [Titulo] => item1
            [Valor] => 10
            [Desc] => Descrição aqui
        )

    [1] => Array
        (
            [Titulo] => novo aqui
            [Valor] => AB
            [Desc] => Mensagem
        )

    [2] => Array
        (
            [Titulo] => Label
            [Valor] => text_1
            [Desc] => Descrição...
        )

)

Como pode ver não é necessário expressões regulares
A função trim é executada para remover possíveis espeços após e antes da coluna, por exemplo, " text_1 " vira "text_1"
A função array_filter é executada para remover possíveis índices vazios, por exemplo, ["Label", "", "", "text_1", "Descrição..."] vira ["Label", "text_1", "Descrição..."]
Se quiser usar expressões regulares...
$data = "Titulo  Valor  Desc
item1   0     Descrição aqui
novo aqui     AB  Mensagem
Label  text_1      Descrição...";

$rows = explode("\n", $data);

$header = preg_split(
    "/\s{2,}/",
    trim(
        array_shift($rows)
    )
);

$parsed = array();

foreach ($rows as $row) {
    $parsed[] = array_combine(
        $header,
        preg_split(
            "/\s{2,}/",
            $row
        )
    );
}

print_r($parsed);

A ideia é mais ou menos a mesma, só que o preg_split já faz o trabalho do explode, array_filter e array_map + trim. O resultado final é o mesmo
Comment: Eu diria impossível, não tem como saber se aquele espeço vazio é uma coluna vazia ou não se a quantidade de espaços é variável (de 2 a infinito), você até pode saber, pela quantidade de itens no array, que está faltando uma coluna em algum lugar, mas qual lugar? Sugiro alterar o método que cria esses dados
Comment: Só tome muito cuidado com o `array_filter`, pois `array_filter(['0'])` retorna `[]`. Se existir na *string* a linha `foo  0  bar` o código acusará erro, pois após o filtro restaria apenas `['foo', 'bar']`, https://ideone.com/B39sFi.
Comment: Bom ter avisado @AndersonCarlosWoss, achei que, como string, não removia, apenas como inteiro
Comment: existe uma outra complicação, identifiquei que podem haver casos onde a coluna pode conter valor vazio, agora acredito que o parse tenha se tornado ainda mais complicado.


Kernel sizes for multiple convolutional layer neural networks

Question: In most examples I've seen of CNNs with multiple two or more convolutional layers the second layer has more kernels (feature masks) than the second layer, usually around twice as many. My intuition is telling me that the reason for this is that the first layer is picking out features from the input data and the second layer is picking out subsequent combinations of those features in the first convolutional layer. If there are 5 different possible features in the data then there are many more combinations of ways these features could show up in the data. If I'm not wrong there is $\sum\limits_{n=1}^{5}{5 \choose n}$ combinations. So why do we not have that many kernels in the second layer? Is it because of memory limitations, or is it because the number of kernels is more related to the number of possible classes?
Answer: Usually between two such convolutional layers there is a pooling layer which reduces the size of each of these feature maps. So the deeper into the network you go the smaller the dimensions become. On the contrary, the number of feature maps is actually increased. This is done because we generally want a trade-off between high-resolution maps and more features.
Furthermore, the deeper you go in a network the more abstract the features it can detect. Say you have a CNN for facial recognition. The first layers would detect simple geometric shapes (lines, curves, etc.), the subsequent layers would detect more high-level features from these first ones (eyes, noses etc.) while the final layers would combine these to detect the highest level features (faces) but in low-resolution. So it makes sense we would want the model to detect many of these high-level features because they have more detail, while it wouldn't make sense wanting to detect many low level features (because in these layers the model understands just lines).
The above example can be seen in the image below:
Comment: Interesting, thank you for your answer! So is there any kind of relationship or rule of thumb for how many kernels you have in each layer? Also would you happen to have a source for those images? I'm writing my masters thesis and would like to look into that more.
Comment: One thing I tend to do is to double the number of filters and then use a pooling operation that cuts each dimension in half, so that after those two operations you require the half number of parameters than the previous layer. I haven't found, however, anywhere saying that this way is the best, it's just what I like to do. I think I got the image either from [here](https://www.strong.io/blog/deep-neural-networks-go-to-the-movies) or [here](https://medium.com/machine-learning-for-humans/neural-networks-deep-learning-cdad8aeae49b). Search google images for "convolutional neural networks faces".


Is any linear transformation between topological vector spaces of finite dimension a closed map?

Question: So in the text Functional Analysis by Walter Rudin the proposition 1.21 states

If $n$ is a positive integer and $Y$ is an $n$-dimensional subspace of a complex topological vector space $X$, then every isomorphism of $C^n$ onto $Y$ is a homeomorphism and $Y$ is a closed set

So can I argue thorugh the above theorem that any linear map between topological vector spaces of finite dimension is a closed map? In particular is any linear transformation $f$ of $\Bbb C^n$ in $Y$ a closed map? So could someone help me, please?
Comment: See https://www.mathcounterexamples.net/continuous-maps-that-are-not-closed-or-not-open/
Comment: @RishiSonthalia Okay, but at least is $f$ closed?
Comment: No, only linear injective maps are. Your (standard BTW) example shows that 1-1 is quite needed.
Answer: Let be $\pi_1:\Bbb R^2\rightarrow \Bbb R$ the projection onto the first coordinate and so let be $C$ set given by the identity
$$
C:=\Big\{(x,y)\in\Bbb R^2:y=\frac 1 x\,\,\,\text{for}\,x>0\Big\}
$$
So $C$ is closed, being the graph of a function,  but however $\pi_1[C]$ is not closed.