Haskell: dealing with ambiguous imports

Question: I'm getting ambiguity warnings, sometimes errors, all the time. This is just a clear cut example. I'm using doctest:
$ stack exec doctest -- src/Main.hs

<interactive>:6:14:
    Ambiguous occurrence 'stdout'
    It could refer to either 'Turtle.stdout',
                         imported from 'Turtle' at src/Main.hs:6:1-13
                         (and originally defined in 'Turtle.Prelude')
                      or 'System.IO.stdout',
                         imported from 'System.IO'
                         (and originally defined in 'GHC.IO.Handle.FD')
...
Examples: 2  Tried: 2  Errors: 0  Failures: 0

It obviously works, at least for the time being, but these warnings are voluminous, annoying, and probably dangerous.
I tried to cope with hiding, like this (Main.hs, all the imports):
import Turtle
import Prelude hiding (FilePath)
import System.IO hiding (FilePath, stdout, stderr) -- trying to hide std...
import Text.Regex.TDFA

but it makes no difference.
Is there a correct general approach?
Comment: [Johann Tibell recommends](https://github.com/tibbe/haskell-style-guide/blob/master/haskell-style.md) to "[a]lways use explicit import lists or qualified imports for standard and third party libraries".
Comment: Do all-encompassing standard libraries, like Prelude or System.IO, really lend themselves to the explicit import list approach?
Comment: @AlexeyOrlov IMO not for prelude - but for `System.IO` etc. it's even a nice documentation - your user can see where some function he/she might not know is imported from
Answer: Well you could edit your .ghci file and include
:m - Prelude

this way GHCi should not preload the Prelude but like Jubobs I would suggest that you don't reuse those names in the first place or that you import (qualified) with some combination of
import Turtle hiding (stdout)
import qualified Turtle as T
import Turtle ( noOverlappingFunc, NonOverlappingType )

it's just the minor inconvenience of writing
T.stdout

instead.
IMHO it's not a good idea to hide stuff from the Prelude to much - aside from the problems you saw there is always the problem of peaple reading your code and their expectations
Comment: It does not look like 'minor inconvenience'. It looks like impossibility, as there's no explicit references to stdout in my source code.
Comment: err ... if you don't need it then just use `import Turtle hiding (stdout)` Problem solved ... of course this raises the question why you tried `import System.IO hiding (stdout)` because obviously you did use `Turtle.stdout` in this case
Comment: the *inconvenience* is prepending `T.` which seems not to much burden on the programmer IMO
Comment: In my case Turtle stdout is working: `echo (format ("verbose: "%w) verbose)` is my only output, and that's Turtle through and through. I have to get rid of the standard stdio.

Haskell: dealing with ambiguous imports


Turn numerical filter into ON/OFF

Question: I'm using uc_best_sellers to count the number of times a product has been sold.
The module provides a numeric views filter like this:

I'd like to have a view like this:

A product would be a best seller if it was sold more than 50 times (in my case).
What would I need to achieve this filtering style?
Answer: Instead of using exposed filters for this, I think it will be easier to make separate views displays with overridden filters. Then either use quicktabs to display the different filter options, or make each one a page display with a path and output the links to the filter displays in the headers.
Comment: Because it says 'filter' doesn't mean that it's an exposed filter, so, yes, Jody, is correct that you need three different views. You can use QuickTabs which are nice or create three different secondary tabs.

Turn numerical filter into ON/OFF


Passing D3DFMT_UNKNOWN into IDirect3DDevice9::CreateTexture()

Question: I'm kind of wondering about this, if you create a texture in memory in DirectX with the CreateTexture function:
HRESULT CreateTexture(
  UINT Width,
  UINT Height,
  UINT Levels,
  DWORD Usage,
  D3DFORMAT Format,
  D3DPOOL Pool,
  IDirect3DTexture9** ppTexture,
  HANDLE* pSharedHandle
);

...and pass in D3DFMT_UNKNOWN format what is supposed to happen exactly? If I try to get the surface of the first or second level will it cause an error? Can it fail? Will the graphics device just choose a random format of its choosing? Could this cause problems between different graphics card models/brands?
Answer: I just tried it out and it does not fail, mostly
When Usage is set to D3DUSAGE_RENDERTARGET or D3DUSAGE_DYNAMIC, it consistently came out as D3DFMT_A8R8G8B8, no matter what I did to the back buffer format or other settings. I don't know if that has to do with my graphics card or not. My guess is that specifying unknown means, "pick for me", and that the 32-bit format is easiest for my card.
When the usage was D3DUSAGE_DEPTHSTENCIL, it failed consistently.
So my best conclusion is that specifying D3DFMT_UNKNOWN as the format gives DirectX the choice of what it should be. Or perhaps it always just defaults to D3DFMT_A8R8G8B.
Sadly, I can't confirm any of this in any documentation anywhere. :|
Answer: MSDN doesn't say. But I'm pretty sure you'd get "D3DERR_INVALIDCALL" as a result.

If the method succeeds, the return
  value is D3D_OK. If the method fails,
  the return value can be one of the
  following: D3DERR_INVALIDCALL,
  D3DERR_OUTOFVIDEOMEMORY,
  E_OUTOFMEMORY.
Comment: No, I tried it and it does not fail, at least not in all cases.
Comment: Ok, at this point it would be worth letting us know what you expect to happen, and what you are trying to do.
Answer: I think this falls into the "undefined" category.  Some drivers will fail the allocations, while others may default to something.  I've never seen anything in the WDK that says that this condition needs to be handled.  I'm guessing if you enable the debug DX runtime you will see an error message.


JavaScript – Continuidade de uma array usando operações aritméticas para encontrar o índice

Question: No JavaScript, eu, usando no caso o for loop comum de iteração de array com a variável i, estou tentando criar um arranjo com sub-arranjos que tenham dados de outros arranjos a partir de uma ordem relativa à posição de i.
Este exemplo de código descreve situação:

var arranjoBase1 = ["a", "b", "c", "d", "e"]
var arranjoBase2 = ["f", "g", "h", "i", "j"]
var arranjoBase3 = [1, 2, 3, 4, 5]

var arranjoFinal = [ [], [], [], [], [] ]

for(i = 0; i < arranjoFinal.length; i++)
 { arranjoFinal[i].push(arranjoBase1[i], arranjoBase2[i], arranjoBase3[i]);
   if(i != 0) { arranjoFinal[i].push(arranjoBase3[i - 1]) }
    else { arranjoFinal[i].push(arranjoBase3[arranjoBase3.length - 1]) };
   if(i != arranjoBase3.length - 1) { arranjoFinal[i].push(arranjoBase3[i + 1]) }
    else { arranjoFinal[i].push(arranjoBase3[0]) };
   if(i == 0) { arranjoFinal[i].push(arranjoBase3[arranjoBase3.length - 2]) }
    else if(i == 1) { arranjoFinal[i].push(arranjoBase3[arranjoBase3.length - 1 ]) }
    else { arranjoFinal[i].push(arranjoBase3[i - 2]) };
   if(i == arranjoBase3.length - 1) { arranjoFinal[i].push(arranjoBase3[1]) }
    else if(i == arranjoBase3.length - 2) { arranjoFinal[i].push(arranjoBase3[0]) }
    else { arranjoFinal[i].push(arranjoBase3[i + 2]) }  };

//Resultado Final
  var arranjoFinal = (5) [Array(7), Array(7), Array(7), Array(7), Array(7)]

//Exemplo de Arranjo Final
  var  arranjoFinal[0] = (7) ["a", "f", 1, 5, 2, 4, 3]

Esse código já me fornece os resultados desejados.
O ponto é que, como precisei usar muitas condicionais if para contornar o problema de se o resultado de uma das operações aritméticas para encontrar o próximo elemento a se adicionar no arranjoFinal estiver fora do índice do arranjoBase3 – caso em que seria retornado undefined no lugar do elemento desejado –, sinto que deve haver uma solução muito mais prática para se chegar a esse resultado, que, por exemplo, também seja sustentável para arranjos maiores do que cinco elementos.
Assim, haveria de fato uma forma mais [prática/concisa] de se alcançar resultados como esses? Como, por exemplo e preferencialmente, uma forma de dizer que, se o resultado de uma operação aritmética para encontrar o índice do elemento de uma array ultrapassar o índice dessa array isso significa que é para ela dar continuidade à operação a partir do seu [início/final]?
Comment: Poderia explicar mais detalhadamente o que o código deveria fazer ou faz atualmente? se possível como se fosse a explicação dada para algum desenvolvedor que fosse desenvolver essa tarefa pra você. Assim surgirá uma solução possívelmente bem diferente da sua e pode ser que seja melhor.
Answer: Certo, vamos lá:
Acredito que suas condições sejam os menores dos seus problemas.
Não tentei melhorar as condições porque estava um pouco confuso, acredito que você precisa pensar numa lógica melhor.
Mas a questão é melhorar as práticas de javascript nesse código.
Uma função que vai ajudar muito sua vida é a map, ela retorna um novo array a partir do seu array de entrada (o array original não é modificado, o que significa muito em programação funcional).
Suas comparações deveriam ser === ao invés de == para evitar conversão de tipo já que você sabe que as variáveis que você está comparando são números.
Use const ou let, var já está em desuso. No meu exemplo é utilizei só const porque apesar de ter que alterar os elementos de um array eu não alterei o endereço de memória dele.
Melhore sua identação: seus condicionais estão bem difíceis de entender, não coloque ponto e vírgula no final de um else. Te aconselho a utilizar o eslint que fornece regras para codificação e acaba te ensinando a programar melhor.
Como última dica, aprenda programação funcional e veja porque não mutar uma variável de entrada é importante para melhor manutenção e debug do seu código e aprenda javascript na versão 6.
Eu fiz um código utilizando arrow functions que é da versão 6 e já funciona no chrome e no firefox:
const arranjoBase1 = ["a", "b", "c", "d", "e"]
const arranjoBase2 = ["f", "g", "h", "i", "j"]
const arranjoBase3 = [1, 2, 3, 4, 5]

const arranjoBase = [undefined, undefined, undefined, undefined, undefined]

const arranjoMap = arranjoBase.map((el, i) => {
    const arranjoParte1 = [arranjoBase1[i], arranjoBase2[i], arranjoBase3[i]];
    const arranjoParte2 = [];

  if (i !== 0) {
    arranjoParte2.push(arranjoBase3[i - 1]);
  } else {
    arranjoParte2.push(arranjoBase3[arranjoBase3.length - 1]);
  }

  if(i !== arranjoBase3.length - 1) {
    arranjoParte2.push(arranjoBase3[i + 1]);
  }
  else {
    arranjoParte2.push(arranjoBase3[0]);
  }

  if (i === 0) {
    arranjoParte2.push(arranjoBase3[arranjoBase3.length - 2]);
  } else if (i === 1) {
    arranjoParte2.push(arranjoBase3[arranjoBase3.length - 1 ]);
  } else {
    arranjoParte2.push(arranjoBase3[i - 2]);
  }

  if (i === arranjoBase3.length - 1) {
    arranjoParte2.push(arranjoBase3[1]);
  }
  else if (i === arranjoBase3.length - 2) {
    arranjoParte2.push(arranjoBase3[0]);
  } else {
    arranjoParte2.push(arranjoBase3[i + 2]);
  }

  return arranjoParte1.concat(arranjoParte2);
});

Como eu vou usar o map, só preciso de um array com o número de elementos que você precisa. Como você precisa somente dos valores dos outros 3 arrays o valor do array não importa.
O arranjoParte1 é sempre fixo e o arranjoParte2 é tratado nas condição. A cada interação eu retorno um array com a concatenação dos dois.
Na versão 5 eu uso concat para concatenar, na versão 6 eu posso simplesmente usar esse comando:
[...arranjoParte1, ...arranjoParte2]

Aqui está um exemplo funcionando, porque afinal falar é fácil né ;)


Meteor accounts {{> atForm }} not showing

Question: I have a problem displaying the standard Meteor user accounts login template {{> atForm }} in my React based Meteor application.
I am using a wrapper component to display Blaze based templates in my React application. I am trying to implement the standard login form of Meteors useraccounts package. But when I am using {{> atForm }} in the Blaze Template the login form is not displayed. But when I lock a specific state like {{> atForm state='signUp'}} then the form is displayed.
Wrapper Component
import React, {Component} from 'react';
import ReactDOM from 'react-dom';
export default class AccountsUI extends Component{

componentDidMount() {
    this.view = Blaze.render(Template.LoginTemplate,
    ReactDOM.findDOMNode(this.container));
}

componentWillUnmount(){
    Blaze.remove(this.view);
}

render () {
    return  this.container = ref} />
}
}

Blaze Template
<template name="LoginTemplate">
    {{> atForm }} //Not displayed
    {{> atForm state='signUp'}} // Displayed
</template>

But also other internal states like {{> atForm state='signIn'}} are not displayed as well.
Packages in .meteor
useraccounts:unstyled
accounts-password

Do you have a suggestion why this is the case and how I can resolve this problem ?
Thanks in advance.
Answer: Answer is solved by myself. The problem was, that I was already logged in when i rewrote my code. So the only thing that was missing was to log out. So if anyone has the same problem, always check if you are already logged in, in your application. If so, be sure to log out, before adding the {{> atForm}} into your template.
Answer: Uninstalling accounts-ui seemed to fix this for me in my case


How to add or insert ' (single quotes) for every string in a list in which strings are separated by commas using Java

Question: I have a list as below
[url1,url2,url3,url4]

This list will be based on multiple selection from drop down list of HTML. So list size i.e., list elements change dynamically. My problem is I am unable to get the logic for getting single quotes to the strings.
I want the above string list to be displayed as
'url1','url2','url3','url4'

That is single quotes (') for each string and commas (,) should not be eliminated. Please help me how to achieve this using Java.
Comment: Please share what you have tried.
Comment: Are we talking about a `List` containing your "url" variables (in that case they would be displayed between double-quotes) or a `String` representing your list as "[url1, url2, etc]"?
Comment: Not able to get the logic for keeping quotes.But able to remove the [ and ] using replace()
Comment: You can take each element from list and put it in between single quotes and add it back..Try this in code and let us know if you have any problem..
Comment: One option is to split the string on comma, then append single quote to each word and then append all words back
Answer: for list  you can convert this by Java 8
 list.stream().collect(Collectors.joining("','", "'", "'"));

Answer: import org.apache.commons.lang.StringUtils;
...
...
String arr[] = new String[4];
arr[0] = "my";
arr[1] = "name";
arr[2] = "is";
arr[3] = "baybora";

String join = "'" + StringUtils.join(arr,"','") + "'";

Result:
'my','name','is','baybora'

Answer: You can use Java 8 utility class java.util.StringJoiner.
This class specifically introduced to format list of string values.
For example:
StringJoiner joiner = new StringJoiner("','", "'", "'");
        joiner.add("url1");
        joiner.add("url2");
        joiner.add("url3");
        joiner.add("url4");
        System.out.println(joiner);

Output
'url1','url2','url3','url4'

Note that, in constructor of StringJoiner,
StringJoiner(CharSequence delimiter, CharSequence prefix, CharSequence suffix)

we are passing single quote as prefix and suffix to satisfy your specific requirement.
Answer: Using Java 8 Streams:
String[] array = {"url1", "url2", "url3", "url4"}

Stream.of(array).collect(Collectors.joining("','", "'", "'"));

Answer: Most fast and efficient way to do this
Ex.
ArrayList<String> a=[url1,url2,url3,url4];

String s  ="'"+a.toString().replace("[","").replace("]", "").replace(" ","").replace(",","','")+"'";

Answer:
Iterate the list (for/while).
For each element in the list append <element-of-list>. Hint: use append() on StringBuilder.
Truncate/substring the list to remove the last element added. Hint: use substring on String class.
Comment: StringBuilder b=new StringBuilder(); Now b.append("''")will not append ' at begining and ' at ending of string right??
Answer: Have one Util.java file or if you already have all common util stuff then just add these static methods

for single quote

public static String singleQuote(String str) {
        return (str != null ? "'" + str + "'" : null);
}

for double quotes

public static String doubleQuotes(String str) {
        return (str != null ? "\"" + str + "\"" : null);
}

Answer: If you have your elements in an array, you can do something like this:
import java.util.ArrayList;

class Test1{
    public static void main(String[] args) {
        String[] s = {"url1","url2","url3","url4"};

        ArrayList<String> sl = new ArrayList<String>();
        for (int i = 0; i < s.length; i++) {
            sl.add("'" + s[i] + "'");
        }
    }
}

Answer: Try this
    String[] str = {"url1","url2","url3","url4"};
    ArrayList<String> strList = new ArrayList<String>();
    for (String k:str)
        strList.add("'" +k+ "'");
    System.out.println("Printing the list");
    for(String k:strList)
        System.out.println(k);

Answer: Also you can use Java8 built-in features for String concatenation and third-party libraries for wrapping each string in quotes.
import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.runners.MockitoJUnitRunner;
import org.springframework.util.StringUtils;

import java.util.ArrayList;
import java.util.List;
import java.util.StringJoiner;

@RunWith(MockitoJUnitRunner.class)
public class StringJoinerTest {
    @Test
    public void testStringJoiner() {

        //init test data
        String s1 = "team1";
        String s2 = "team2";
        List<String> teams = new ArrayList<>(2);
        teams.add(s1);
        teams.add(s2);

        //configure StringJoiner
        //when some values will be added to joiner it will return string started with prefix "(" an finished with suffix ")"
        //between prefix and suffix values will be separated with delimiter ", "
        StringJoiner teamNames = new StringJoiner(", ", "(", ")");
        //if nothing has been added to the joiner it will return  this value ('')
        teamNames.setEmptyValue("(\'\')");

        //fill joiner with data
        for (String currentString : teams) {
            //if you need to wrap each string in single quotes you can do this via org.apache.commons.lang3.StringUtils#wrap
            // or org.springframework.util.StringUtils#quote
            teamNames.add(StringUtils.quote(currentString));
        }

        System.out.println(teamNames.toString());
    }
}


In an integrally closed, Noetherian, local, integral domain of dimension $1$, the maximal ideal $P$ is eventually principal

Question: Let $R$ be an integrally closed, Noetherian, local, integral domain of dimension 1 with unique maximal ideal $P$.
Take an element $a \in P$ that is non zero. Show that for some $n$, $P^n$ is contained in $aR$ (the ideal generated by $a$).
The point of this exercise was to prove unique factorisation in a Dedekind Domain. A step in this is to show that the localisation of $R$ with respect to $P$ is a PID.
Comment: And your attempts?
Comment: @Don Thousand not sure if this counts as an attempt but presumably the noetherian condition should be of use. I imagine assuming that the powers of P are never contained in aR allow you to construct an increasing chain of ideals which must terminate.
Comment: What is $ a $$?$
Comment: @Bernard a is a general non zero element in P
Answer: Hint:
The quotient ring $R_P/aR_P$ is a local noetherian ring of dimension $0$ so $PR_P/aR_P$ is its nilradical.


Feature creation of Print module for path to print stylesheet

Question: Using the print module, I need to create a Feature to deploy in order to set the path to the .css file to my themes directory on my production site.
In Features, what do I need to check to make sure I get these settings?
When creating the Features, the closest thing I can find is a checkbox under permissions:

Thanks
Answer: You also need strongarm enabled to export the path variable. Then it will be available as print_css variable.


Jest don't instantiate a sequelize model

Question: I'm working in an backend testing task, and now I'm struggling with Jest and Sequelize integration.
We have an existing (and working) Express JS API, which needs to be tested with Jest. This API create/read/update/delete resources with Sequelize, in an PostgreSQL database.
As I said, everything works fine in the localhost and in the production env. But, in test environment, i have some issues. Jest can upload my application into an docker container (using npm's 'testcontainers' lib), make requests and receive responses, but apparently cannot instantiate my models and repositories.
My project folder tree is something like this:

src
config-env
db

migrations
models
repositories
seeders

routes

controllers
dtos
middlewares
schemas

services
tests

My tests are running from /tests/integration and /tests/unit.
My jest.config.js is something like this:
require('@babel/register');
require('@babel/polyfill');

module.exports = {
  verbose: true,
  bail: true,
  clearMocks: true,
  collectCoverage: true,

  testEnvironment: 'node',
  collectCoverageFrom: ['./src/**'],
  coverageDirectory: './tests/jest-coverage',
  coveragePathIgnorePatterns: [
    ...
  ],

  coverageProvider: 'v8',

  moduleFileExtensions: [
    'js',
  ],

  transform: {
    '^.+\\.(js|jsx)?$': 'babel-jest',
  },
  transformIgnorePatterns: ['<rootDir>/node_modules/'],

  testMatch: [
    './**/*.test.js',
  ],

  globalSetup: '<rootDir>/tests/jest.setup.js',
};

My jest.setup.js file is something like this:
import { GenericContainer, Wait } from 'testcontainers';

import Constants from '../src/utilities/constants';
import {
  initializeDatabaseOnly, initializeConectionDatabaseOnly, runPendingMigration, runPendingSeed,
} from '../src/db/database';

module.exports = async () => {
  process.env.DEBUG = 'true';
  process.env.NODE_ENV = 'test';
  //Multiple env variable declarations...

const container = await new GenericContainer('postgres')
  .withEnv('POSTGRES_DATABASE', 'my-database')
  //More ommited environment definitions for the container
  .withExposedPorts(5432)
  .withWaitStrategy(Wait.forLogMessage('database system is ready to accept connections')
  .start();

  await new Promise(r => setTimeout(r, 1000));
  await initializeDatabaseOnly();

  await new Promise(r => setTimeout(r, 1000));
  await initializeConectionDatabaseOnly();

  await new Promise(r => setTimeout(r, 1000));
  await runPendingMigration();

  await new Promise(r => setTimeout(r, 1000));
  await runPendingSeed();

  await new Promise(r => setTimeout(r, 1000));
};

This configurations kind of works. I can upload a container and i can access my seeders data inside a postgresql database, via dbeaver for an example. So, my models and repositories are working, here, right? When i try to run the tests, i got an error in my repository call, apparently because the node process of the test don't have that instantiated.
My .test.js files are being constructed like so:
import PaymentOrderService from '../../src/services/payment-order';
describe('Create payment order', () => {
  it('should return error when create a payment order with invalid customer', async () => {
    let response;

    try {
      response = await PaymentOrderService.create({
        order: { //The body of request, working both in Postman and cURL requests });
    } catch (err) {
      console.log(err);
    }

    expect(response.text).toBe('user_not_found');
  });

But when the container process tries to create some data into the db, it crashes with a TypeError: Cannot read property 'findOne' of undefined. The findOne() is a function that I call to find a user and then add the payment order data into the db (with this user id as foreign key).
We also have some tests with the following architecture, returning the same error:
describe('Create payment order with credit card parceled', () => {
  it('should return error to create a payment order installments amount', async () => {
   const response = await supertest(app)
      .post('/payment-order')
      .send(body)
      .set('api-access-key', 'dreqweq-1231239-aqeasd-aqwede-fakeapikey');
   const data = JSON.parse(response.error.text);
   expect(response.status).toEqual(415);
   expect(data.error).toBe('invalid-order.config.method-payment');

We believe that is a jest parallelization related problem, because the same calls works outside the jest env. We got stucked with a jest.isolateModules(() => { beforeAll() }); problem too, so we changed the jest.config.json from an setupFiles:[] setupFilesAfterEnv:[] scenario to an globalSetup: '' one, because we just needed one container running (not one for each describe). I don't believe that is the cause, but...
Someone have a clue about how to resolve this? If anyone needs more info to help me, just ask!
Answer: have such a same problem.

Remove transform from jest.config.js:
module.exports = {
preset: 'ts-jest',
testEnvironment: 'node',
roots: ['<rootDir>/server/'],
setupFiles: ['<rootDir>/server/testsetup.ts']
// transform: {
//  '^.+\\.(ts|tsx|js)$': 'babel-jest'
// }

};

Change all imports like

import * as express from 'express' to
import express from 'express'

For me, it helps. Good Luck.
Comment: Thanks a lot! I have solved my problem actually changing some parts of the project pattern, accepting that jest and sequelize would never get along together the way it was being build. But now I got another breath!

I will test your answer soon. Thank you again!


Linear regression with aggregated data or not?

Question: I am working on a simple linear regression model that predicts cost-per-truck based on distance between origin and destination. Let's assume there's a good linear relationship between Cost and Distance. I have historical data for ~250K shipments which corresponds to around ~2000 lanes (origin, destination pairs).
The goal of this model is to predict cost for new lanes (for which I have no historical data) and get the confidence/prediction intervals.
Some random dummy data -
# A tibble: 10 x 4
     Lane   Distance ShipmentNo  Cost
    <chr>      <dbl>      <int> <dbl>
 1 A to B        100          1  812.
 2 A to B        100          2 1055.
 3 A to B        100          3  749.
 4 A to B        100          4 1479.
 5 A to B        100          5 1099.
 6 C to D        500          6  754.
 7 C to D        500          7 1146.
 8 C to D        500          8 1221.
 9 C to D        500          9 1173.
10 C to D        500         10  908.

I am confused between 2 approaches -

Build a model at the Shipment level i.e. every shipment is a data-point (250K points) on Cost vs. Distance scatter plot. Data to be divided into train and test data-sets i.e. test data represents unseen (new) shipments. In this case, the regression line represents mean shipment cost for a given distance. How would this translate into lane level prediction? Would it make sense to use the predicted mean shipment cost along with 95% confidence interval as a prediction for new lanes?
Build a model at the Lane level (aggregated level) i.e. average lane rate (Total Cost / Total Shipments for each lane) is a data-point (now 2000 lanes/points) on Cost vs. Distance scatter plot. Aggregated data to be divided into train and test data-sets i.e. test data represents unseen (new) lanes. I suppose, in this case, the regression line represents the mean of average lane rate for a given distance. Would it make sense to use this mean of average lane rates along with 95% prediction interval as prediction for new lanes?

Not sure but option 2 seems right in terms of level of detail I need. However, I am concerned that by aggregating the data I may be losing information on (or even misrepresenting) the variation in cost of trucks for individual shipments. Also, in option 2 all data-points (lanes) seem to have same weight in spite of some lanes potentially having lot more shipments than others which doesn't seem right.
Based on other answers on this site, general opinions indicate that aggregation is a bad idea - however I am curious to know if it is justified in my case considering the goal is to predict lane rates and not individual shipment rate.
Looking for answers to my above questions and advise on best way to handle this problem. Thanks for the help!
Comment: You could consider a linear model with a *random* effect for `Lane`.
Comment: Not wanting to make more work for you, but there is something to be learned from doing both models, and comparing the results.  If they give similar answers, you have a relatively well-behaved problem space (for example, different lanes having similar variability).  On the other hand, if the two models give you very different answers, that in itself tells you something more than either one alone.
Answer: I would suggest taking advantage of all the data by fitting a linear mixed effects model. This is a model that includes fixed effects and random effects. If you aggregate the data then you lose information, while if you fail to account for the probability that costs in one lane are more similar to costs in that lane than other lanes then you will obtain biased results.
To do this, you simply fit random intercepts for lane. Using the lme4 in R package as an example the would look like:
lmer(cost ~ distance + (1 | lane), data = mydata)

The lane variable is treated as random in the sense that it can be thought of as a variable for which you have more than just a few (5-10 is often mentioned as being the minimum, although there is no consensus on the matter: you have 2000 so this is not an issue at all), and each lane then has it's own intercept which varies randomly around the global intercept (which is equivalent to just "the intercept" in a model without random effects). We do this in order to control for the non-independence of observations/measures within each lane, otherwise biased results are very likely.
You can also allow the effect of distance to be different for each lane by adding random slopes:
lmer(cost ~ distance + (distance | lane), data = mydata)

And finally you can also allow distance to have a non-linear effect on cost by including nonlinear terms in the model equation, or using splines.
The above models will help you to understand the association between distance and cost for existing lanes, and it will also help you to understand the variation in costs by lane, and in the case of the random slopes model, to also see if the association between cost and distance differs between lanes. However, you say that you are interested in prediction for new lanes that have not been seen before. To get to grips with this problem, more information is needed about what the lanes actually are.
Comment: Good advice, but few sentences about what mixed effects models are could be useful, since OP does not seem to mentioning it, so we can guess they are not familiar with mixed effect models.
Comment: @Tim thank you - I have edited my answer accordingly
Comment: @RobertLong Thanks for the answer. I'll read up on mixed effects model. As for *"more information is needed about what the lanes actually are"* - the analysis is limited to mainland USA. If it helps, the number of distinct origins is limited to ~50 i.e. new lanes will likely appear due to new destinations. Also, some additional attributes could be developed for all lanes like East-to-West or State-to-State or Zip3-to-Zip3 etc. Does that help?
Comment: @Shree thanks ! I am still not quite clear about what *exactly* is a lane.
Comment: @RobertLong Lane is origin-destination pair. For example - NYC-to-Chicago, NYC-to-Boston, Cleveland-to-Dallas etc.
Comment: @Shree hmm. OK thanks. So surely the lanes also have have an element of distance to them ? The lanes *could* be coded with a numeric value, but it would not be exact (hence you have the actual distance variable). Not only that, but some lanes such as zip to zip would be far more accurate, than say, state to state, So you are ignoring that, and just coding them as categorical. Is that right ?
Answer: First, I'd try to answer the questions. Are the cost/distance relationships between each lane statistically different? If yes, are the differences large enough to matter for our application / problem? If no, use the aggregated data. Otherwise, we need to do some simple thought experiments to see what using aggregated / non-aggregated data means.
Suppose 50% of lanes cost \$10/km and 50% \$2/km. The average would be \$6/km. But does this mean anything for making predictions? I'd say a \$6/km prediction for a previously unseen lane is wrong. If all lanes are similar, than it's either going to cost \$10/km or \$2/km. But the average of hundreds of new lanes will be close to \$6/km.
Using aggregated or non-aggregated data answers different questions.
Hope this helps a little.
Comment: Thanks for the answer. Could you elaborate on *"Using aggregated or non-aggregated data answers different questions"* i.e. what question each answers. As mentioned in the question I am looking to predict the mean cost of truck and corresponding confidence/prediction intervals for a new lane with known distance.


Simulation of a truncated normal distribution over two intervals

Question: Given $X$ a random variable with a normal distribution, what is the best procedure to simulate $X|X\in[a;b]\cup[c;d]$, i.e. we want to simulate the truncated normal distribution only on the intervals $[a;b]$ or $[c;d]$.
I have already seen a question asked on this forum in Truncated normal distribution over a union of intervals, but the answer gives no indication of how to simulate a sample of this law.
Comment: I am astonished by your claim that the answer "gives no indication of how to sample," because it *includes working code!*  Please take another look.  However, that code implements the (potentially lethally inefficient) rejection method.  Instead, just invert the CDF.  Alternatively, sample from a mixture of two Normals, each truncated on one of the intervals.
Answer: Assuming wlog that the truncated Normal is a standard $\mathcal N(0,1)$ Normal, the density of the distribution to be simulated writes as
$$f(x)\propto e^{-x^2/2}\mathbb I_{(a,b)}(x)+e^{-x^2/2}\mathbb I_{(c,d)}(x)$$
or, with the normalising constant
$$f(x)= \dfrac{e^{-x^2/2}\mathbb I_{(a,b)}(x)+e^{-x^2/2}\mathbb I_{(c,d)}(x)}{\sqrt{2\pi}[\Phi(b)-\Phi(a)+\Phi(d)-\Phi(c)]}$$
when assuming that $a<b<c<d$. As noted by whuber, this writes as a mixture of two truncated Normal distributions,
$$f(x)= \dfrac{\Phi(b)-\Phi(a)}{\Phi(b)-\Phi(a)+\Phi(d)-\Phi(c)}\dfrac{e^{-x^2/2}\mathbb I_{(a,b)}(x)}{\sqrt{2\pi}[\Phi(b)-\Phi(a)]}\\
\qquad\qquad\qquad+\dfrac{\Phi(d)-\Phi(c)}{\Phi(b)-\Phi(a)+\Phi(d)-\Phi(c)}\dfrac{e^{-x^2/2}\mathbb I_{(c,d)}(x)}{\sqrt{2\pi}[\Phi(d)-\Phi(c)]}$$
Since the probability weights
$$\dfrac{\Phi(b)-\Phi(a)}{\Phi(b)-\Phi(a)+\Phi(d)-\Phi(c)}\quad\text{and}\quad\dfrac{\Phi(d)-\Phi(c)}{\Phi(b)-\Phi(a)+\Phi(d)-\Phi(c)}$$
are available, one can pick a component (first or second) according to its probability and then simulate the corresponding truncated Normal (for which I once wrote an efficient accept-reject algorithm that has been implemented in an R package).
Answer: Honestly, I don't think you need to do anything more sophisticated than simulating from an unconstrained Normal distribution, and then exclude values outside of your intervals (example in R):
<code>mu = 0
sigma = 1
low1 = -1
high1 = -.5
low2 = 0
high2 = 1
x = rnorm(n, mu, sigma)
in_interval = (x > low1 & x < high1) | (x > low2 & x < high2)
trunc_x = x[in_interval]
</code>
There are also more efficient, but more complicated solutions, that let you sample directly from an arbitrary distribution.
Comment: This can be *incredibly* inefficient.  Try it with a standard Normal distribution truncated to $(-10,-6)\cup(6,10),$ for instance.


Return file with status code from remote server in Nginx

Question: I am trying to set up my SPA to return 404 for invalid resources by having nginx do a backcheck for the resource before responding. All this is to avoid invalid pages coming with 200 and thus having spiders crawling soft 404 pages.
My desired behaviour
Browser                           Nginx                           API
  | -- GET myapp.com/users/12 -->   |                              |
  |                                 | -- GET myapi:8000/users/12   |
  |                                 |   <----------- 404 --------  |
  | <-- 404 index.html ------------ |
  |                                 |

I expect my configuration to be something like below
server {
        listen       80;
        server_name  localhost;
        root   /usr/share/nginx/html;
        index  index.html;

        location /users/ {
          # 1. Fetch from myapi:8000
          # 2. Return index.html with status from fetched URI
        }
}

I've been digging around for a while now but I'm a bit lost on how to achieve this. I've seen many simple examples out there with try_files etc. but nothing that seems to fit my case, since it seems most examples do very simplistic forwarding.
How can I achieve the above behaviour?
Comment: nginx doesn't have this kind of functionality built-in. Getting  status code and response body from separate sources is in strong disagreement of nginx fundamental design. You might be able to hack something with nginx lua module.
Comment: @TeroKilkanen thanks for the reply. I tried with Lua but it fastly became too cumbersome. I ended up making my backend API instead to serve the index.html.
Answer: I managed to solve this by having my backend API to always respond with index.html for Accept: text/html. I did try the Lua way with OpenResty but it felt too hacky and quirky and too complex to maintain.
Then my config looks like below:
    location / {
      try_files $uri $uri/ @forward_to_api;
    }

    location @forward_to_api {
      rewrite ^(.*) /api$1 break;
      proxy_set_header "Accept" "text/html";
      proxy_pass http://localhost:8000;

      # Follow any redirects
      proxy_intercept_errors on;
      error_page 301 302 307 = @follow_redirect;
    }

    location @follow_redirect {
      set $moved_location $upstream_http_location;
      proxy_pass http://localhost:8000$moved_location;
    }

This will redirect all $uri to localhost:8000/api/$uri and follow redirects.


Node insertion binary tree C++

Question: I believe i implemented inserting a new node into a binary search tree, however my testing script shows my tree as being in valid
My code where the value is inserted into the tree if there is no current node.
bool InsertRecursive(BinaryTreeNode* tree_node, int value) {

 if (tree_node == NULL)
 {

BinaryTreeNode * pNode = new BinaryTreeNode;
pNode->data = value;   // inserting node in
pNode->left_child = NULL;
pNode->right_child = NULL;
tree_node = pNode ;
return true;
 }
       if(tree_node->data == value )   // if value already exists return false
{
 return false ;
  }
   else if (tree_node->data > value)   // traverses to find the location for the value
    return InsertRecursive(tree_node->right_child, value);
  else
return InsertRecursive(tree_node->left_child, value);

 }

the actual method of inserting the node inside
 bool BinaryTree::Insert(int value) {

if (InsertRecursive(head, value)) {
return true;
} else
  return false;
}

What my testing script says
|   Test: Insert
|   |   -Should be a valid tree of size 1 after inserting '4'
|   |   |   You built an invalid tree. Why would you do that to me?
|   |   -Should be a valid tree of size 2 after inserting '2'
|   |   |   You built an invalid tree. Your code needs to get it's life together
|   |   -Should be a valid tree of size 3 after inserting '3'
|   |   |   You built an invalid tree. More coffee maybe?
|   |   -Insert(1) should return false for: [ d2-1 d1-2 d2-3 d0-4 d2-5 d1-6 d2-7
]
|   |   |   It returned true. Unlike a boss
|   Failed!
So the problem should be with in where i actually insert the node as the tree is being read as in valid.  I am lost on how to solve this however and would like some guidance.
Comment: `tree_node` is a local variable. All you've done is create a memory leak.
Comment: I think you're traversing the wrong way..  if tree_node->data is >, you need to traverse left (and vice versa)
Comment: I don't know what you mean by that. I suggest you 1) think about what "`tree-node` is a local variable" means until it's clear what's going wrong, and 2) implement some linked lists flawlessly before you attempt trees.
Answer: when leave fuction, "tree_node = pNode ;" effect nothing. change tree_node type to "BinaryTreeNode**"  if you want change it's value
Answer: You have to split the inserts into both the helper and the method for each case of when head is equal to nullptr and when tree_node is equal to nullptr. Then add your implementations to both of those.
Your insert functions look like they are are on the right track so just think about how the two functions in the helper and the method are similar.
Comment: and just as a note, you will get a seg fault if you write BinaryTreeNode* pNode = new BinaryTreeNode. change it to BinaryTreeNode* pNode = new BinaryTreeNode{0}.

Node insertion binary tree C++


Is it possible to change the content database of an existing SharePoint '07 web app?

Question: Disclaimer: I'm very much an accidental SharePoint admin so I greatly appreciate your patience.
My SharePoint farm's database server is EOL and about to die. I have managed to get the config database (SharePoint_Config) moved over to the new database server and even managed to get the Central Administration site to use the new database server. I have even added the new database server to the farm as a content database server. My problem right now is with the web app's content database (WSS_Content). I already have a copy of the content database setup and ready to go on the new database server but, for the life of me, I can't figure out how to change the SharePoint web app's database over to it.
Is this even possible? If so, how would I do it? If not, would I have to create a new web app and then restore (and overwrite) its WSS_Content database with my existing one?
Answer: Read Microsoft's advice on this, and follow it!
http://technet.microsoft.com/en-us/library/cc512725%28office.12%29.aspx


Powershell eliminate files with exact extension in -Exclude

Question: I used this cmdlet in my PowerShell script to get all files in the path except the rdl files :
Get-childitem $path -recurse -Exclude *.rdl | select -expand fullname

The problem is that the command also eliminates files with extension ".rdl.rss" and that's not what I want.
How can I eliminate only files with the exact extension ".rdl" ? Thanks Guys
Answer: Use Where-Object rather than -Exclude:
Get-ChildItem $path -Recurse |Where-Object {$_.Extension -ne '.rdl'} |Select-Object -ExpandProperty FullName


gostaria de saber como eu faço pra remover uma div class com js

Question: gostaria de saber como eu faço pra remover um certo elemento do meu site com js
já tentei varias coisas.
Consigo remover por id. mas class eu não sei e já procurei em vários lugares.
<code>var node = document.getElementById("formCalcularCep");
if (node.parentNode) {
  node.parentNode.removeChild(node);
}
</code>
gostaria de remover isso porque o template do meu site e pago, e não consigo ter acesso total do código, e não quero pagar a empresa pra fazer essa remoção.
Comment: basicamente a mesma coisa, mas um seletor por id (`getElementById`) retorna um único elemento, um seletor por classe vai retornar várias, uma vez que a mesma classe pode ser utilizada por vários elementos, então precisa outra característica para selecionar o elemento certo. Por exemplo, se fosse o primeiro elemento encontrado, poderia fazer algo como `var divs = document.getElementsByClassName("nome-da-class"); divs[0].remove();`
Comment: entendi, muito obrigado pela ajuda, deu certo!
Answer: já procurou ocultar o elemento pela classe utilizando JQuery?
Segue um exemplo:

   $(document).ready(function(){
      $('.classe').hide();
   });


WSOD need help!

Question: I am having problem with WSOD on my site. I have a lot of modules so my php execute time is 60s and allocated memory is 512MB. This error comes not regularly this is the biggest problem, and I cannot reproduce it.
The http request at that time is 200 what means its ok, my memory and processor usage is fine. But I have no error on php and apache2 level that means something dies even before it could log anything. If I try to debug it and put code in the index.php the problem dissapears and the site is visible, with the apache restart as well.
Other strange thing, I have a test copy of the site, on the same server. If I load the test site the problem dissappears like on apache restart.

I went throught the Drupal WSOD document (https://drupal.org/node/158043) and tons of forum posts.

I have this in my settings.php, and index.php file as well.
<code>error_reporting(E_ALL);
$conf['error_level'] = 2;
ini_set('display_errors', TRUE);
ini_set('display_startup_errors', TRUE);
</code>
The error reporting is setted on php and server level as well.
Still no sign of errors and solutions.
Thanks,
Adam
Comment: possible duplicate of [How to see the error messages when I get the white screen of death?](http://drupal.stackexchange.com/questions/7560/how-to-see-the-error-messages-when-i-get-the-white-screen-of-death)
Comment: Relevant: http://drupal.stackexchange.com/questions/104371/blank-pages-or-wsod-on-a-recently-migrated-site-to-a-godaddy-server-only-logging
Comment: You're not seeing errors on the white screen, however did you check the log files? `/var/log/apache2` or `/var/log/httpd` or whatever is set in your httpd.conf or apache2.conf or vhosts.conf files.
Comment: Sure, I have a full logging over the webserver in /var/log/
Answer: Your best bet would be to take a local copy of your site and leverage XHProf (there's a module for that) to determine what exactly is causing your site to throw a WSOD (memory leak? too many DB calls? etc.).
That being said, you should probably start by identifying which modules aren't absolutely required for your site and uninstall them completely. Remember that any module enabled is being parsed on each page load and this causes unnecessary memory consumption. One way to mitigate this, though, is to make sure you have APC enabled on the server, or better, that you're running PHP 5.5.x which comes with Zend Opcode cache built-in (it might be more involved, though, for compatibility reasons).
Then, you should definitely try to find why you need as much as 512M of PHP memory to run the site. This is way too much for regular use cases. Part of it comes from the fact that you must have too many modules indeed, but it could also be caused by too many (uncached) Views and/or Panels or anything that is taking too long to load.
I'd recommend that you install the Memory Profiler module prior to getting to XHProf profiling (if you're not familiar with this already). It's a quick and lightweight way to determine paths that are possible culprits.
Comment: I just installed memcache, and varnish on the server and the site to work together and see wath's going to happen in the following days. Since my site is very huge and custom I need a lot of modules. I took in account that to check everything with XHProf and I will do it by the next week.

Thanks for your answer and helpful advices.

WSOD need help!


predict all test batches in keras / tensorflow

Question: i am trying to predict all my test batches in keras / tensorflow to then plot a confusion matrix.
The current BATCH_SIZE is: 32
My test dataset is generated with the following code from a big dataset:
test_dataset = big_dataset.skip(train_size).take(test_size)
test_dataset = test_dataset.shuffle(test_size).map(augment).batch(BATCH_SIZE)

After model.compile() and model.fit() i get the predictions and the correct labels with this code:
points, labels = list(test_dataset)[0]
preds = model.predict(points)
preds = tf.math.argmax(preds, -1)
points = points.numpy()

This method only predicts one batch --> 32 predictions.
Is there a way to predict all test batches in keras / tensorflow?
Thanks in advance!
Answer: You could pass the entire dataset to model.predict according to the docs:

Input samples. It could be: A Numpy array (or array-like), or a list
of arrays (in case the model has multiple inputs). A TensorFlow
tensor, or a list of tensors (in case the model has multiple inputs).
A tf.data dataset. A generator or keras.utils.Sequence instance. A
more detailed description of unpacking behavior for iterator types
(Dataset, generator, Sequence) is given in the Unpacking behavior for
iterator-like inputs section of Model.fit.

points = test_dataset.map(lambda x, y: x)
labels = test_dataset.map(lambda x, y: y)
preds = model.predict(points)
preds = tf.math.argmax(preds, -1)

Or with numpy:
points = np.concatenate(list(test_dataset.map(lambda x, y: x))
labels = np.concatenate(list(test_dataset.map(lambda x, y: y))
preds = model.predict(points)
preds = tf.math.argmax(preds, -1)


Gulp inserir arquivo minificado na index

Question: Introdução
Estou implemento o automatizador de tarefas Gulp, está tudo indo bem, arquivos minificados, concatenados e otimizados.
Problema
Agora o que me falta é pegar o arquivo minificado ( all.min.js ) e inseri-lo em minha index.html removendo as importações antigas, tentei pesquisar algo que fizesse isso  mas não encontrei.
Alguém sabe qual usar?
Gulpfile.js
var gulp = require('gulp'),
    sass = require('gulp-ruby-sass'),
    autoprefixer = require('gulp-autoprefixer'),
    minifycss = require('gulp-minify-css'),
    rename = require('gulp-rename'),
    imagemin = require('gulp-imagemin'),
    pngcrush = require('imagemin-pngcrush'),
    concat = require('gulp-concat'),
    htmlmin = require('gulp-htmlmin'),
    plumber = require('gulp-plumber'),
    jshint = require('gulp-jshint'),
    notify = require('gulp-notify'),
    open = require('gulp-open'),
    uglify = require('gulp-uglify')
    karma = require('gulp-karma');

// karma
var testFiles = [
    'app/scripts/main.js'
];

gulp.task('test', function() {
  // Be sure to return the stream
  return gulp.src(testFiles)
    .pipe(karma({
      configFile: 'karma.conf.js',
      action: 'run'
    }))
    .on('error', function(err) {
      // Make sure failed tests cause gulp to exit non-zero
      throw err;
    });
});

// servidor
gulp.task('express', function() {
  var express = require('express');
  var app = express();
  app.use(require('connect-livereload')({port: 4002}));
  app.use(express.static(__dirname+"/app"));
  app.listen(4000);
});

var tinylr;
gulp.task('livereload', function() {
    tinylr = require('tiny-lr')();
    tinylr.listen(4002);
});

function notifyLiveReload(event) {
    var fileName = require('path').relative(__dirname, event.path);

    tinylr.changed({
        body: {
            files: [fileName]
        }
    });
}
// Controle de erro pumbler
var onError =function(err){
    gulp.beep();
    console.log(err);
};
// Open url localhost:4000 in google chrome
gulp.task("url", function(){
    gulp.src('app/index.html')
    .pipe(open("", {app: "google-chrome", url: "http://localhost:4000"}));
});

// Compile SASS and minify
gulp.task('styles', function() {
      return gulp.src('app/styles/**/*.scss')
        .pipe(plumber({
            errorHendler: onError
        }))
        .pipe(sass({ style: 'expanded' }))
        .pipe(autoprefixer('last 2 version', 'safari 5', 'ie 8', 'ie 9', 'opera 12.1'))
        .pipe(gulp.dest('app/css'))
        .pipe(rename({suffix: '.min'}))
        .pipe(minifycss())
        .pipe(gulp.dest('app/css'))
        .pipe(gulp.dest('dist/css'));
});

// lint jshhnit
gulp.task('lint', function(){
    return gulp.src('app/scripts/*.js')
        .pipe(plumber({
            errorHendler: onError
        }))
        .pipe(jshint())
        .pipe(jshint.reporter('jshint-stylish'));
})
// observador
gulp.task('watch', function() {
    // Compila .sass
    gulp.watch(['app/styles/**/*.scss', 'app/styles/*.scss'], ['styles']);
    // livereload alter html file
    gulp.watch(['app/*.html', 'app/*/*.html'], notifyLiveReload);
    // live reload alter css file
    gulp.watch('app/*/*.css', notifyLiveReload);
    // duplicado arrumar depois
        // Lint alter JS file
        gulp.watch(['app/scripts/*.js', 'app/*/*.js'], ['lint']);
        // livereload alter JS file
        gulp.watch(['app/scripts/*.js', 'app/*/*.js'], notifyLiveReload);
    // ..
});

// BUILD
// minify img
gulp.task('imagemin', function(){
    return gulp.src('app/image/*')
        .pipe(plumber({
            errorHendler: onError
        }))
        .pipe(imagemin({
            progressive: true,
            svgoPlugins:[{removeViewBox: false}],
            use: [pngcrush()]
        }))
        .pipe(gulp.dest('dist/image/'));
});

// concatenar js
gulp.task('scripts', function(){
    gulp.src('app/scripts/**/*.js')
        .pipe(jshint())
        .pipe(jshint.reporter('jshint-stylish'))
        .pipe(concat('all.js'))
        .pipe(rename({
            suffix: '.min'
        }))
        .pipe(uglify())
        .pipe(gulp.dest('dist/js'));
});
// minify html
gulp.task('minifyhtml', function(){
    gulp.src(['app/*.html', 'app/*/*.html'])
        .pipe(htmlmin({collapseWhitespace: true}))
        .pipe(gulp.dest('dist/'))
});

// Run '$ gulp'
gulp.task('default', [
                    'styles',
                    'express',
                    'url',
                    'livereload',
                    'watch',
                    'imagemin',
                    'minifyhtml',
                    'lint'
        ], function() {});
gulp.task('build', [
                'lint',
                'scripts',
                'minifyhtml',
                'imagemin',
                'styles'
    ])

Comment: Pode colocar o código que já tem? Está a integrar o js dentro do `index.html`? por que não ter um ficheiro à parte?
Comment: @Sergio opa acho que me expressei mal, tenho ele em um ficheiro a parte all.min.js (apos processo de concat e minify), oque quero agora é pegar o endereço do documento e inserir em meu html (  )
Comment: @Sergio inseri o Gulpfile.js na pergunta;
Comment: Porque voce não dá overwite no seu arquivo JS que voce está substituindo?
Answer: Eu acredito que o que você esta buscando é exatamente o que o plugin gulp-usemin oferece.
Com o gulp-usemin voce pode criar um bloco de construção (build block) dentro do seu index.html com todos os arquivos JS que devem ser minificados.
O plugin além de fazer toda a minificação vai inserir a  exatamente aonde e como você quiser.

    |
    |  depois da minificação
    |
    V

Da mesma forma que você pode criar blocos para JS files, você criar para CSS files também.
Comment: Exatamente isso!


Handle change and select list of options

Question: I'm new to React and I'm trying to select an option from the list and click the button to confirm the selection. I save the selected option in the "pickedUser" object. But when I change the state of the "pickedUser", I think the render is start again and the list is like at the beginning. I just want to select the option and click on the button without restarting select list. If you can help solve the problem and point out the mistakes I need to correct in order to get better. Thanks!
//pickedUser saving selected option from list.
constructor(props){
    super(props);
    this.state = {
      users: [],
      pickedUser:{
        name:"",
        email:"",
        uloga:""
      },
      isLoading: true,
      errors: null
    };
    this.handleChange = this.handleChange.bind(this);
    this.routeChange = this.routeChange.bind(this);
  }

//In handleChange I'm trying to set a value I get from select to pickedUser.
async handleChange(event) {

    const eName = event.target.value.split(',')[0];
    const eEmail = event.target.value.split(',')[1];
    const eUloga = event.target.value.split(',')[2];
    await this.setState({
      pickedUser:{
        name : eName,
        email: eEmail,
        role: eUloga
      }
    });
  }

//And this is a part of my render.
render() {
    const { isLoading, users, pickedUser } = this.state;
    return (

        <select
          id="list"
          value={pickedUser}
          onChange={event => this.handleChange(event)}
        >
          {!isLoading ? (
            users.map((user, key) => (
              <option key={key}>
                {user.name},  {user.email},   {user.role}
              </option>
            ))
          ) : (

Loading...

          )}
        </select>
        <button id="b1" type="button"
          value={pickedUser}
          onClick={event => {
            this.handleSubmit(event);
            this.routeChange();
          }}>
          Sign in
        </button>

I wanted to function normally, that when I select the option, it stays selected, but it happens that when I select it, it is refreshed again.
I just have to tell that the value is good when the option is selected but the display is not.
Comment: What does `routeChange` method do?
Comment: Note that each time the state changes using `setState` the component is re-rendered. https://reactjs.org/docs/state-and-lifecycle.html#do-not-modify-state-directly
Comment: Is there any need for `handleChange` when clicking on the select? Since it will call the `setState` that will in return re-render the component. FYI, there is no need for an async function for the `handleChange`, you can remove the async and await.
Comment: 1)I call on the routeChange at onClick in button and I used it to transfer to the other page.
Comment: 2)I note that the component is re-rendering and I want to stop that when I select option.
Comment: As far as I can see there should not be any problem with this code, the DOM should not be changed when you click on the select. Can you create a reproducible code, using e.g. https://codesandbox.io/ ?
Comment: How you mean is there any need for handleChange? I must somehow to save the option I have chosen.
Comment: Create a form and pick up the selection when you click sign in. No need to store it in the state. https://reactjs.org/docs/forms.html
Comment: Yes, but I want to pick, for example, the name from the selected option, and on the next page I write a welcome {name}! That's why I would keep it in the some object.
Answer: Technically you only have to correct this line
<select
    id="list"
    value={pickedUser.name + "," + pickedUser.email + "," + pickedUser.role}
    onChange={event => this.handleChange(event)}
>

value should not be object (pickedUser), but it should be string.
This is working example
But I can suggest following:

Make state.users object (not array). Email should be unique, so it can be used as key. For example:
this.state = { users: { "email@example.com": {name: "Jack", uloga: "aaa"},
    "email@example.com":  {name: "Mark", uloga: "aaa"} } }

In this case you'll be able to extract user from users by it email.
Object also support iteration like arrays useng Object.keys(users) or Object.values(users)
Use email as key for <option>. Index as keys is not good practice in React.
Add id to each <option> to easily identify selected option in event handler

Suggested version is here
Comment: Thanks a lot! You helped me and solved my problem. Also thank you for tips that are very useful.


AJAX Form using a sub-class of ContentEntityForm fires/invokes/calls the WidgetBase::massageFormValues() twice

Question: I'm using a AJAX form. This form I build from a sub-class of ContentEntityForm. The Entity edited by this form has a field MyField. To bring all the field values in a appropriate structure I'm using massageFormValues() in MyFieldWidget class (sub-class of WidgetBase).
The little problem is: the method massageFormValues() is called twice on every submit. The are no other problems with the form. I can properly edit values on this form.
So it's just the question of understanding: why Drupal does call massageFormValues() twice on every submit.
The mentioned and my (very simplified) code:
My ajax controller
class AjaxController extends ControllerBase {
  ...
  ...

  public function myController($eid) {

    $myEntity = $this->entityTypeManager()->getStorage('my_entity')->load($eid);
    $form = $this->entityFormBuilder()->getForm($myEntity, 'my_mode');
    $renderer = \Drupal::service('renderer');
    $output = $renderer->renderRoot($form);

    $response = new AjaxResponse();
    $response->addCommand(new OpenModalDialogCommand($title, $output));

    return $response;
  }
  ...
  ...
}

MyForm is a sub-class of the ContentEntityForm and it's a mode form for the mode my_mode of the Entity my_entity:
class MyForm extends ContentEntityForm {

 ...
 ...
  protected function actions(array $form, FormStateInterface $form_state) {
    $actions = parent::actions($form, $form_state);

    $actions['submit'] = [
      '#type' => 'submit',
      '#value' => t('Save'),
      '#ajax' => [
        'callback' => '::ajaxCallback',
        'event' => 'mousedown',
      ],
    ];

  return $actions;
  }
  ...
  ...
}

My field widget class:
class MyFieldWidget extends WidgetBase implements ContainerFactoryPluginInterface {
...
...

  public function massageFormValues(array $values, array $form, FormStateInterface $form_state) {

    // THIS CODE IS CALLED TWICE ON EVERY SUBMIT OF MY SUBMIT BUTTON DEFINED ABOVE
    // BUT WHY TWICE?

    return $rightStructureValues;
  }

...
...
}

Answer: In an Ajax request the form is rebuilt from scratch. This was often asked here why the form build method is called two or even three times. It seems like the entity is rebuilt as well when the form is submitted via Ajax:
Drupal\Core\Entity\EntityForm::afterBuild
  /**
   * Form element #after_build callback: Updates the entity with submitted data.
   *
   * Updates the internal $this->entity object with submitted values when the
   * form is being rebuilt (e.g. submitted via AJAX), so that subsequent
   * processing (e.g. AJAX callbacks) can rely on it.
   */
  public function afterBuild(array $element, FormStateInterface $form_state) {
    // Rebuild the entity if #after_build is being called as part of a form
    // rebuild, i.e. if we are processing input.
    if ($form_state->isProcessingInput()) {
      $this->entity = $this->buildEntity($element, $form_state);
    }

    return $element;
  }

This will trigger the mentioned method in the field widgets.
Comment: Unfortunally I wasn't able to prevent calling massageFormValues() twice with behalf of afterBuild(). But if you can describe in detail, how it works, it would be nice. I will provide my own solution as an answer.
Comment: There is nothing wrong with that this method and the entire form build is being called twice. The core maintainers decided to rather rebuilt the form (and the entity in this case) than to fill up the database with cached Ajax forms which never get submitted.
Comment: I understand. Thank you, 4k4. I recognize, it's micro optimization I make :-).
Answer: My workaround/solution. massageFormValues() is still called twice by Drupal, but my logic in it will be executed just once, after the form validation is complete:
MyFieldWidgetClass.php:
class MyFieldWidget extends WidgetBase implements ContainerFactoryPluginInterface {
...
...

  public function massageFormValues(array $values, array $form, FormStateInterface $form_state) {

    // massageFormValues() is called twice: on validateForm and on submitForm
    // we just need massageFormValues once, after $form_state->isValidationComplete() is TRUE
    if(!$form_state->isValidationComplete()) {
      return [];
    }
    ...
    ...
    return $rightStructureValues;
  }

...
...
}


Automorphism group of a functor

Question: Could someone tell me how the automorphism group of a functor is defined? I tried to google it for quite some time but nothing useful came up.
Answer: In any category, the automorphism group of an object is just the set of isomorphisms from that object to itself, with the group operation being composition. A functor $F:\mathcal{C}\to\mathcal{D}$ is an object of the category $\mathcal{D}^{\mathcal{C}}$, where the isomorphisms are natural isomorphisms of functors, so the automorphism group is the set of all natural isomorphisms $F\to F$ under composition.

Automorphism group of a functor


When doing principal components regression, do I need to standardize independent variables and/or dependent variable?

Question: I want to run PCA on a set of variables and then regress my dependent variable on the PCA scores.
I have the following questions:

Should I scale and center my variables?
If yes, should I also standardize my dependent variable before running linear regression?
What if I don't standardize my dependent variable?
Comment: `regress the scores on my dependent variable` Not clear enough so far. Are you regressing the "dependent variable" on the components? Or regressing a chosen component on that variable?
Comment: @ttnphns after I run PCA, I will get **principal components** and the corresponding **scores**. Now, I want to regress these principal component scores on my dependent variable.
Comment: If you are not shrinking or penalising the regression in "PCA space" this will achieve nothing.  PCA is an "affine" transformation (I think) of your X matrix, and it is reversible/invertible.  Its like modelling $ Y=A^T (X-c)\gamma + e $ (where $A$ and $ c $ are functions of $ X $) instead of $ Y=X\beta +e $.
Comment: You'll usually just feed the (small number of) components to the regression that account for, say, 90% of the variation in the original data, as given by the cumulative sum of squares of the eigenvalues. I assume this is what the OP had in mind.
Comment: @probabilityislogic would you please explain a little more
Answer:
Yes. If you don't scale and center, your PCA will not capture covariances, but the directions of largest variation. Put differently: your PCA should really not depend on whether lengths are measured in meters or inches, or your temperatures in Celsius and Fahrenheit.
PCA works on the independent data only, it doesn't care about the dependent variables. You will usually not transform the dependent variables. Or you may want to transform them independently of the PCA, e.g., to stabilize variances or take logs, depending on the underlying science.
See 2. You will usually not transform the dependent variable just because you do a PCA on the independent variables. So the question of whether or not to transform the DV and whether or not Bad Things will happen if you transform (or don't transform) is orthogonal to the PCA.
Comment: 1.(a) Sometimes when the variables are measured on comparable scales the directions of largest variance are thought to be more interesting.
Comment: The interpretation of the model will depend on the interpretation (if any) of your principal components. Examine the loadings and see whether you can interpret the PCs in a meaningful way. PCA is notoriously hard to interpret. You may want to consult your favorite textbook. The fitted values have the same interpretation as always: they are fitted values.
Comment: @Scortchi and @Stephan Kolassa  : Now after I regress the scores of the first PC on the dependent variable, I get a model
`Y = intercept + a*PC1`(note that PCA was performed on standardised X matrix)
what will be the interpretation of the model? and how will I interpret the fitted values obtained?


What is a strict definition of U-shaped relationship?

Question: I now have seen several papers that analyze U-shaped or inverse U-shaped relations among variables (in a regression framework). The general understanding I have from there is that it is a specific type of a non-linear relationship that we can all easily visualize.
However, I am a bit confused about how exactly people mathematically define U-shaped regression functions. Suppose for simplicity there is only regressor $x$.
Does having U-shaped regression function mean the regression function is convex and decreasing in $x$ up to some point $c$ and then after $c$ is convex and increasing in $x$?
Or does it simply mean that the regression function is decreasing in $x$ up to some point $c$ and then after $c$ is increasing in $x$?
Comment: Different authors may have different definitions--should the relation be continuous? Differentiable? Convex? The most general definition consistent with the idea of "increasing then decreasing" or "decreasing then increasing" is: A map $f:A\to\mathbb{R}$ with $A\subset\mathbb{R}$ is "U-shaped" means there exists a decomposition of $A=B\cup C$ where (1) every element of $B$ is less than or equal to every element of $C$; (2) $f$ is monotonic on both $B$ and $C$; (3) the images $f(B)$ and $f(C)$ have at least two values each; and (4) the directions of monotonicity of $f$ differ on $B$ and $C$.
Comment: I don't think "U-shaped" is a mathematically well-defined term; there is no universally accepted definition and I don't think you should be looking for one. I changed the first two sentences of my answer to stress that.
Comment: I couldn't attest to a general agreement--and I'm sure that many authors would object that my definition is broader than they intended.  That's why I have left it as a comment.
Comment: @whuber This is exactly what I am looking for -- if there is a general agreement on how to define it....
Answer: The short answer to your question (as stated elsewhere) is that
there is no single mathematical definition of a U-shape. The comment by @whuber is the
best general definition that I have seen.
I do research on tests of U-shapes and for my presentation I have a slide with the title
"What does a U mean to you?", meaning that it is subjective what people mean by
the term "U-shape". The most important thing is that when you use the term
"U-shape", you define exactly what you mean by it, without assuming that
others will know what you mean.
Since you specified the case of only one regressor, I'll focus on that. I've
seen the following definitions used in various articles:

A U-shape is a quadratic.
A U-shape means convexity (for an application along these lines, see Van Landeghem's 2012 "A test for the convexity of human well-being over the life cycle: Longitudinal evidence from a 20-year panel").
A U-shape is a function with weighted average derivative negative until a point, and weighted average derivative positive after that point (see Uri Simonsohn's Two-Lines: The First Valid Test of U-Shaped Relationships).
A U-shape is a function with exactly one turning point. This corresponds to a function that is quasi-convex but not monotone.

One complication that comes up is what if the turning point is close to the
ends of the range of the x variable? Should we still consider such a function a
U-shape? In my opinion, such a discussion should be had when you define what a
U-shape means to you for your application, and when you specify your null
hypothesis.
The definition I use in my paper, Non-Parametric Testing of U-Shaped Relationships, is the following:
Let $m(x)$ be the regression function and let $S\left(X\right)$ be the support of $X$. For a specified set $A_{0}\subset S\left(X\right)$, we are interested in testing the following:
$$\begin{align*}
H_{0}\colon & \exists a\in A_{0}\mbox{ st }\forall x\in S\left(X\right)\\
 & m^{'}\left(x\right)\left(x-a\right)\ge0\\
\text{versus}\\H_{A}\colon & \forall a\in A_{0},\,\exists x\in S\left(X\right)\mbox{ st}\\
 & m^{'}\left(x\right)\left(x-a\right)<0
\end{align*}$$
For example, in an application I test for a U-shape of life satisfaction in age
from age 20 to 70, where the turning point is between ages 30 and 60. Arbitrary
decisions are necessary with this proposed framework. The important thing is to be open
about them and check how sensitive results are to changes (and to challenge others to do the same).
In addition to stating the null hypothesis, as always you should state the
assumptions you rely on. For example, a common assumption is that the
regression function is either U-shaped on monotone. See, for example, Lind and
Mehlum's 2009 "With or Without U? The Appropriate Test for a U-Shaped
Relationship", where they propose an improvement on the vanilla OLS quadratic test by testing that the derivative of a specified functional
form is negative at the beginning of the range, and positive at the end.
An additional point to consider is: Do you want a test that rejects the null hypothesis because of a small violation of U-shapedness? If yes, consider the R package qmutest, which implements a non-parametric tests based on splines of the null hypotheses that the regression function is quasi-convex, and separately that it is monotone. If you do not want a test that gives inference against a U-shape because of a small violation, Uri's two lines test might be best if you want to test that a regression function is mostly decreasing and then mostly increasing.
Since your question was about the use of the term "U-shape" and the
definition, I find it relevant to list some terms here that are used often to
refer to the same thing that "U-shape" and "inverted U-shape" are used to refer
to: "valley-shaped", "trough-shaped", "hill-shaped", "unimodal",
"single-peaked", and "bell-shaped". There is no inherent reason why "U-shape"
is a better term than the others, but its use seems to have caught on.
I'm working on a general R package that will just be an interface to
specific R packages (such as qmutest) that test for U-shaped relationships however they choose to
define them. The goal will be to help users compare different tests and to
think hard about the exact null hypothesis they want to test, and which
assumptions they are prepared to make.
Comment: (+1) Very nice, thoughtful, authoritative overview.  Welcome to our site!
Comment: +1. I am a bit confused by this sentence: "Do you want a test that rejects the null hypothesis because of a small violation of U-shapedness?" I assume that the null is that there is no U-shapedness, so that a sufficiently small p-value were an evidence of U-shapedness, is that correct?
Comment: (I am glad to see that you favourably mention Uri's paper: I mentioned it in my answer here and it was heavily criticized in the comments.)
Comment: @amoeba When I use "U-shape" I'm referring to definition 4 above (a function with exactly one turning point). For my test, the null is U-shapedness. What I mean is that asymptotically the null of U-shapedness will be rejected if there is any violation of U-shapedness in the underlying regression function (e.g. there are two turning points). This is not the case for Uri's test because the two lines test is about the average derivative. So there can be wiggles without necessarily leading to asymptotic inference against U-shapes.
Comment: @amoeba As an example, see the function labeled "sin" in Figure 2 of my paper. I believe (although I have not checked) that the two lines test would give asymptotic inference suggesting that "sin" is a U-shape, even though it has three turning points.
Comment: @amoeba the nulls of the qmutest package are quasi-convexity and monotonicity (corresponding to the R functions `cbquasi` and `cbmono`). Combined, the tests give inference about U-shapes. Indeed this is different from the OLS quadratic specification and Uri's two lines test. I made edits to add information on the null hypotheses.
Comment: I am sorry I still don't understand this bit that I quoted above: "Do you want a test that rejects the null hypothesis because of a small violation of U-shapedness?" Can you please explain what this means? Are you asking if one wants a test that rejects the null [of no U-shape; i.e. a test that concludes that there is a U-shape] if there is "a small violation of U-shapedness" i.e. if there actually is no U-shape but almost a U-shape? I am pretty confused by what you mean here.
Comment: @scottkosty, any progress on the package you reference at the bottom of your post?
Comment: @Parseltongue unfortunately I've made zero progress directly on that package. I would like to work on better non-parametric methods before that general package. It's only indirectly related, but I've been working on a paper for a new quasi-convexity restriction method (https://arxiv.org/abs/1809.01038). However, it will take time for that work to translate into U-shapes. Probably I won't work on the general R package still for another couple of years.
Answer: "U-shaped relationship" is not a mathematically precise term and there is no universally accepted definition. It usually means that the relationship is first decreasing and then increasing, or vice versa.
In other words, it means that the relationship is not monotonic (non-monotonic), but instead has exactly one extremum (maximum or minimum). In computer science this is sometimes called "bitonic".
Uri Simonsohn has recently written an interesting paper about testing U-shaped relationships. See his preprint Two-Lines: A Valid Alternative to the Invalid Testing of U-Shaped Relationships with Quadratic Regressions which is very readable and amusing. Here is how the paper starts:

Is there such thing as too many options, virtues, or examples in an opening
  sentence? Researchers are often interested in these types of questions, in assessing if the effect of $x$ on $y$ is positive for low values of $x$, but negative for high values of $x$. For ease of exposition, I refer to all
  such relationships as 'u-shaped,' whether they are symmetric
  or not (i.e., U or J shaped), and whether the effect of $x$ on $y$ goes from negative to positive or vice versa (i.e., U or inverted-U).

This supports the definition I gave above.

For a short overview of Uri's paper, one can read his DataColada post Two-lines: The First Valid Test of U-Shaped Relationships. The main point is that using quadratic regression  to test the presence of a U-shaped relationship is very very wrong. Apparently quadratic fits are often used in some fields to argue in favor of a U-shaped relationship (i.e. t-test for the quadratic term is taken to be the test of U-shape-ness); this is troubling.
Here is the key figure:

Update: There is some criticism of Uri's paper in the comments. I would like to stress that he never suggests that discontinuous two-line fits are supposed to model the data well (or that the jump at the discontinuity has some physical meaning). No. This fit is used for the sole purpose of providing a statistical test of U-shape-ness.
Of course I agree with @FrankHarrell that it makes much more sense to use a spline model to fit such nonlinear relationships. But splines do not provide a test of U-shape-ness, whereas Uri's two-line fit does.
Comment: I'd say that a quadratic curve points towards, in the first place, a varying slope. Which I believe is a very very good way (or at least easy way, in many circumstances) to check that. Yet a very very bad way to represent the (true) underlying relationship, especially the, if you could say, U-ness of a relationship.
Comment: I just read it.  He says 'forcing two lines to connect introduces bias'.  What a strange argument.  Allowing them not to connect introduces impossibilities.  I find the whole two-line argument weak.  It seems to just be avoiding splines.
Comment: I don't see his arguments are strong.  Splines are more likely to fit; why stop at bilinear or even present it seriously?  With splines, testing for association (flatness) and nonlinearity are trivial.  Testing for non-monotonicity is a challenge; would like to see a reference on that.  Regarding just testing for nonlinearity (but ignoring accuracy of predictions) quadratics do a pretty decent job.  The two-lines method is very dependent on where you place the discontinuity.
Comment: Even though I love Bayesian modeling I'm not convinced that the thought experiment of imagining a change point is the straightest way to go.  I would rather see a flexible smooth fit with a prior distribution for the degree of non-monotonicity.
Comment: @FrankHarrell Actually there is a very good reason why they don't enforce continuity of the two-line fits, see Figure 9 in the linked preprint. They never claim that these two-line linear fits are supposed to model underlying relationship well; the fits are only used for the purpose of testing U-shape-ness. I agree that splines make a lot more sense in a more general situation.
Comment: @FrankHarrell Well, I imagine it's hard (if at all possible) to come up with a p-value for U-shape-ness based on a splines model. I guess in many cases it's enough to build a good splines model and then just eyeball it to see if there is any evidence of U-shape-ness. And you don't like p-values anyway. So that's fine. But this paper is trying to develop some instrument for the researchers that want to compute a p-value for U-shape-ness; and this instrument should not have obviously ridiculous false positive rate like quadratic term in regression does... At least that's my understanding.
Comment: @Frank "Non-monotonicity" is a good term! Just look at the Sterling et al paper referenced in the figure; abstract says that they "observed a quadratic association, which indicated that ideological moderates were more susceptible to bullshit than ideological extremists" -- which is just bullshit. Two-lines-test would allow them to test non-monotonicity and see that it's not even significant at 0.05 level. The two-lines paper discusses optimizing discontinuity in some detail, but I don't recall the details. It would certainly be nice to have some test for non-monotonicity with splines, I agree.
Comment: Agree with @FrankHarrell - I think a much more natural way to model this kind of relationship (on the premise of just trying to capture the first increase then decrease) is a Bayesian Latent model, which you model the change point as a latent variable and two linear models for before and after this latent change point. And the two intercept of the linear models should link together so at the change point they share the same value. Allowing two lines to disconnect is not appropriate.
Comment: My point puts nuance on the bad bad behavior of quadratic fits. In the end, by some shift of variables, we could make any "L" relation ship into "U" shape. This does not make the critique on the sloppy language about U shaped relationships and related (symmetric) quadratic fits less sloppy. However it may tell that the critique on (indeed ridiculous) conclusions, can be at some times pedantic. In the case of the two images from Sterling and Swaab, the U-shape might be ridiculous, and strictly speaking there is more like an L/J-shape, however the U shape conveys the story to non professionals.
Comment: The two-line fits in his examples do not even have zero-order continuity, which is almost impossible in real relationships between continuous variables.  The quadratic fit seems to be better than D.  But all of this is why we use regression splines to fit smooth relationships; they relax the assumptions of quadratic shapes.
Comment: @CharlesLao No reason, but the point is that some people *in practice* obviously don't look at their data at all (or maybe don't think at all?): see two examples on the figure that I reproduced in this answer. I think you don't fully appreciate where Uri is coming from; this paper is supposed to suggest a specific actionable alternative to the moronic practice of fitting a quadratic and using the t-test for the quadratic term as a test for non-monotonicity.
Comment: @CharlesLao Just look at the Figure 9 in the linked paper please. If you do what you suggest and if the model is badly misspecified (like shown on that figure), then you can get what looks like a significant increase-decrease breakpoint whereas in reality there is no non-monotonicity.
Comment: @amoeba I dont think that really qualified as an example of "first increase then decrease" :-) If your data looks like that why not just fit it with a logarithm function?


Overdispersion parameter

Question: I am modelling a zero-truncated process with a count model, and am trying to determine whether the data are overdispersed. The Poisson distribution has a variance equal to its mean,
$$\newcommand{\Var}{\operatorname{Var}} \Var(y) = E(y) = \lambda $$
The negative binomial model relaxes this assumption by estimating an overdispersion parameter $\alpha$. There are two regimens for this method, NB1
$$\Var(y) = \lambda  \alpha$$
and NB2
$$\Var(y) = \lambda  (1 + \lambda/\alpha)$$
I have seen the second version more often. I have estimated three models with the same covariates but the three different link functions (Poisson, NB1, NB2).
Model   logLik   alpha  p(alpha)
Poisson -7942    1      NA
NB1     -6399    1.001  0
NB2     -7944    403.4  0

Clearly the overdispersion parameter is significant and I should therefore use one of the NB models. But how can the log-likelihood for NB2 be lower than for Poisson when there is an additional parameter in the model?
When I estimate all three models, I get a warning that my variance-covariance matrix may not be positive semi-definite. But I don't have highly collinear variables, and all of the relevant matrices have strictly positive eigenvalues, so I don't see why I'm getting this warning. Could this be related to the likelihood problem?
I am using the glmmadmb function from the R package of the same name.
Comment: I would recommend that you also try to estimate the variance-covariance matrix of the Poisson estimates using the Huber/White/Sandwich linearized estimator. That estimator of the variance-covariance matrix does not assume $E(y_i) = Var(y_i)$, nor does it even require that $Var(y_i)$ be constant across $i$. That would be less of apples-to-orangutangs comparison with the NB model. I can't tell if glmmadmb allows you to calculate the errors that way or not.
Answer: You are right, the NB2 model should not have a worse fit than the poisson model, because the latter is a special case of the former, where $\alpha=0$. There is definitely a problem. It sounds to me like your numerical maximum likelihood maximizer had problems with your specification and therefore didn't really hit the global maximum.
Now I don't use R, but in Stata the NB2 model often has convergence problems. Stata has an option "difficult", which uses a different algorithm to jump from nonconcave regions of the likelihood function. Using this often solves the problem. Maybe R too has different options for your maximizer; perhaps, you could try different starting values and see whether that makes a difference.

Overdispersion parameter


Python generator function confusion

Question: Hey so I am working on a program that requires a day/night cycle. Here is the function that I made to get it to start working:
def epoch():
    for i in range(0,varb.run_number):
        print("it is now day")
        epoch_state = 1
        yield epoch_state
        time.sleep(varb.day_night_length)
        print("it is now night")
        epoch_state = 0
        yield epoch_state
        time.sleep(varb.day_night_length)

I can find nothing wrong with it, but when I call it I get this:
<generator object epoch at 0x01036670>

Any ideas on how to fix this?
P.S. The idea here is to run the cycle while printing out the state and returning the state
P.P.S. anything with varb. is a global with an unimportant numerical value
Comment: That's not an error. You created a generator function...
Comment: OK how could I change my code to do what I need it to?
Comment: Ok I just did list(epoch()) and it worked. What did I do and how do I recreate that if I need to do the same kind of thing again?
Comment: @unknown: `list()` takes an iterator (a generator is a special kind of iterator) and takes all the elements that produces.
Comment: when ever you want it call list(epoch)
Comment: Thanks for the help. I changed the tittle of the question to a tittle that reflects the problem better. Thanks again!
Answer: There is no error here. The function is working correctly.
You created a generator function by using yield expressions. You now need to iterate over your generator. Until you do, the generator is paused.
You could use list() for that:
result = list(epoch())

or a for loop:
for result in epoch():
    print(result)

Comment: Would `result = list(epoch())` run the loop that is in the function?

Python generator function confusion


need to print current row position number in block for most popular article

Question: I created a view mode named "small-sidebar" and here I need to print current row position number in this block...
I'm using this code :

        <?php echo $content['small_image']; ?>

        <?php echo $label; ?>

      <?php echo $content['title']; ?>

        <?php  echo $content['read_more_link']; ?>

    <?php render($node); ?>

but it not show the current position
in the div class lebel-no I need to print the current position
Comment: what is your mean `current row position` ?
Comment: means views-row position
Comment: can u get a screenshot of your views and put here?
Answer: You can do this in two steps below,
STEP : 1
Add the views result counter under the fields option
Ex:
Fields
Global: View result counter
Content: Title
Content: Body

STEP : 2
If you want to theme it in your own way add the following tpl.php file in your theme folder,
views-view-field--custom-view-theming--counter.tpl.php
I have added the following html to theme it like a tab

    View result counter:<?php print $output; ?>

Thats all about bringing a view counter in your block.


How to use the Paypal sandbox

Question: I guess I'm not understanding the fundamentals of the PayPal sandbox.
I've created a preconfigured personal buyer account with a balance.
I've logged in, and now I'm going to my site to "purchase" something.
Q: Do I have to point to sandbox.paypal.com in order to test with my sandbox user?
Answer: Yes, you do.
You'll also need to create a preconfigured seller account and modify the 'business' of your PayPal , if you use the  buttons.
Sandbox is a bit cumbersome and requires a lot of changes if you use Website Payments Standard (the  buttons), but it's awesome for testing API integrations!
Comment: As for 2013 March, Sandbox is not working anymore for setting IPN of Sandbox accounts.
Comment: What seems to be the problem?
Comment: Now the SandBox functions work only under IE. I had a lot of discussion with Paypal Tech Support's guys.
Comment: Feel free to send on the ticket number(s) and I'll have a look for you.

How to use the Paypal sandbox


How do I send a large form to a REST API in express?

Question: In the front-end (React) there is a form with almost 50 fields. Now, my API in Express, is taking all the fields as x-www-form-urlencoded. Front-end says it's too many fields and there should be a simpler way to send them and accept them in back-end.
I'm not too sure what the solution would be. How can I minimize those 50 fields into just a few keys?
I understand the question is quite broad, but here's an example of how the keys might look like:
name,
surname,
locationName_1,
locationAddress_1,
Latitude_1,
longitude_1,
someOtherField,
otherfieled,
locationName_2,
locationAddress_2,
Latitude_2,
longitude_2,
...
I'm returning a big object with all those form data too. I would like to know if there are some design patterns which I should look into.
Thanks.
Answer: Well, I've never sent 50 fields at once, but I'm curious, do you have
app.use(express.urlencoded()) in your app.js (Express API) with certain configs?
I do know that when a server does not have app.use(express.urlencoded()) the data is not usefull because of this: x-www-form-urlencoded
Comment: Yes I have the `app.use(express.urlencoded({
    extended: true
}));` in my server. What would be a good way of sending 50 fields at once? Does it even make sense as a best practice (or good practice)


Drupal 7, Dev Site, will not Aggregate Javascript or CSS

Question: I have two installations of the same site. The first is sub.website.com, and is the main website. The second is sub.website.com/drupaldev.  On the /drupaldev site I cannot get the javascript or CSS to aggregate, which is causing myriad odd problems. I have checked the tmp directory, changed it, modified permissions and re-installed the site from scratch. None of these things has allowed me to get the website to aggregate.
Is the problem that I am hosting the development site as a folder of the main site?
Answer: This could really be a myriad of problems.  More than likely, your files directory isn't writable.  Browse to the Status Report (admin/reports/status) and make sure the File System section says Writable.  While you are at it, make sure everything on the report is green.
Running a second Drupal site in a subdirectory of the first site is a tricky situation.  The main concern is the way Apache handles .htaccess files in a directory tree.  htaccess rules apply to the directory that the file is in, plus all subdirectories.  This means that your site in /drupaldev is going to inherit the directives from the root htaccess file.  This may or may not cause problems, but in general, this is a big red-flag.


Uploadify Not Working

Question: EDIT:
It seems the issue is to do uploadify not having a session, therefore it could not access the page I was sending it to.
So make sure you point it to a page with no admin login security ;)
This was due to an issue with the url that uploadify was accessing. As it has no session it couldn't retrieve anything.
Answer: Check the documentation for the "on" functions. Looks like you've got the wrong arguments for onAllComplete. If you're only uploading single files then just use the onComplete function. That's what I ended up doing. The video example I watched had the arguments interchanged as well so maybe they changed the signature in a recent version which would explain part of your trouble.
Answer: The flash file is the object doing the communicating, so you need to submit the session ID with the scriptData option:
'scriptData': { 'session': '<?php echo session_id();?>'}
See: http://www.uploadify.com/faqs/how-do-i-send-the-session-id-to-the-back-end-script/
for more info.

Uploadify Not Working


Filtering special characters in SQL Server

Question: This is question pertaining to SQL Server 2014. I have a table xxx. There is a column col1 of type varchar. The values in this column can have alphanumeric characters like 1A324G. There can also be special characters along with alphanumeric like !2A93C or #AC934D, etc.
There can be any special character (eg: !$#^().-_) in a value for this column. I wanted to extract data with only alphanumeric values and NOT any special characters in it. I was trying to use the LIKE clause with wildcard search pattern but I am not able to weed out the ones with only alphanumeric values.
Can someone please help me and let me know how I can do it?
Comment: Please post example of the code that you've tried in a question. It will help you clear your mind, and direct me where not to go.
Comment: Hi, for example assuming i've 2 values in the col1 column: 1GHD5343HS3435368 and !GE9343DGE5934435. i want to extract only 1GHD5343HS3435368 in the result. i tried something like select * from xxx where col1 like '%[0-9][A-Z][^!]%', but it doesn't seem to work. i'm not sure how i can use the wildcard pattern search in the LIKE clause. Also instead of the ! there could be a $ in the col1 value. There could be combo of special chars and alphanumeric. Can you please let me know how i can filter only the alphanumeric values in the select statement. Or should i write a stored proc for this.
Answer: It's been a while since I've played with sql but something like this should work.
SELECT *
FROM xxx
WHERE col1 NOT LIKE '%!%' OR '%$%';
Comment: This does not exclude all fields with non-alpha values.
Comment: i tried something like the below and it seems to work:

    select distinct col1 from xxx
    where col1  like '%!%'
    or col1 like '%@%'
    or col1 like '%#%'
    or col1 like '%$%'
    or col1 like '%&%'
    or col1 like '%*%'
    or col1 like '%`%'
    or col1 like '%''%'
    or col1 like '%?%'
    or col1 like '%.%'
    or col1 like '%-%'
    or col1 like '%+%'


If statement comparing 2 conditions of ints runs as boolean

Question: My if statement is not working correctly. The purpose is to compare a user given date with the current date, and if both the month and week match (a user's birthday), then getBonus = true.
However, my if statement is giving me the following error:
bad operand types for binary operator '&&'
first type:  int
second type: int
incompatible types: int cannot be converted to boolean
Why is my if statement trying to run as a boolean when all of the variables involved are int?
public static boolean getBonus ( int Week, int Month, int bMonth, int bWeek
) {

boolean getBonus = false;

/**************************************************************************
The following statement is used to determine if the user's birthday is
this week, using the month and week of the month. bMonth/bWeek are generated
from user input, Month/Week are generated from a real time calendar.
**************************************************************************/

if(bWeek = Week && bMonth = Month)
  {
    getBonus = true;

  }

return getBonus;
}//end class getBonus

Comment: `=` vs `==`.............closing as typo
Comment: You can simply `return bWeek == Week && bMonth == Month;` (no need for the local variable `getBonus` of the `if` condition).
Answer: Do this
 if(bWeek == Week && bMonth == Month)//used to compare
 // true && true --> true

Instead of this
 if(bWeek = Week && bMonth = Month)//used to assign

Comment: That is better, but it doesn't fix the incompatible types.
Comment: It would surely fix incompatible type problem because this is a comparison which either returns **true** or **false**.
Answer: Use If Statement as follow
if(bWeek == Week && bMonth == Month)
{
     getBonus = true;

 }

== is used to compare
= is use to assign value