How Java "pointers" work?

Question: Lets say this is the C++ code:
void change(int& x){
    x++;
}

or
void change2(int* a){
    *a++;
}

Both will change the global x, right?
So how can I do something like that in java?
Specifically, I want to point to a Vector object

But since Java has no pointers, I'm not sure what to do.
From searching the internet I saw people saying that Java does that in some other way, but I haven't found any real example.
Thanks for
help!
Comment: Your second example does not make any sense.  You are assigning the address of a global variable to a local pointer variable, then increment the pointer.  This will neither effect the global `x` nor the argument `a` on the caller's side.
Comment: possible duplicate of [How to pass by reference in Java](http://stackoverflow.com/questions/5614562/how-to-pass-by-reference-in-java)
Comment: Java is always pass by value.
Comment: The answer is likely, "you can't" - but start by give us the relevant Java code ...
Comment: possible duplicate of [Is Java pass by reference?](http://stackoverflow.com/questions/40480/is-java-pass-by-reference)
Comment: So, you just want a global [Vector](http://docs.oracle.com/javase/6/docs/api/java/util/Vector.html) that you can keep stuff in? (What kind of objects?) Or is it something else that you're ultimately trying to do?
Answer: In Java, instead of pointers you have references to objects. You cannot pass a primitive type by reference, but you can wrap a primitive type inside an object and then pass a reference to that object.
Java provides the type Integer which wraps int, however this type is immutable so you cannot change its value after construction. You could however use MutableInt from Apache Commons:
void change(MutableInt x) {
    x.increment();
}

The change to x will be visible to the caller.

Specifically, I want to point to a Vector object

When you write Vector v = ...; you are assigning a reference to a vector to the variable v. A reference in Java is very similar to a pointer. References are in fact implemented internally using pointers.
Java uses pass by value. When you pass a vector to a method, you are actually copying a reference to that vector. It does not clone the vector itself. So passing a reference in Java is very similar to passing a pointer in C++.
Comment: What I don't like about this is that it gives the impression that you need a 3rd party lib to do something extremely basic in Java. I'm pretty sure that OP does **not** really need MutableInt or anything else from Apache Commons; instead, he'd just need to learn idiomatic Java. Admittedly the question is hard to answer better, as it isn't clear what he *really* wants his code to do.
Comment: I also do not like that this is the accepted answer. See http://stackoverflow.com/questions/5614562/how-to-do-the-equivalent-of-pass-by-reference-for-primitives-in-java/5614630#5614630 for a thorough explanation of 4 ways to do this, without importing any third party library.  Or see dash1e's answer for a common way of doing this.
Answer: With Java you cannot pass primitive types like int by reference, they are passed only by value.
The only things you can do is to find artifices to do that, because instead Objects are passed by reference. Here two examples.
Use an array of single value, like this
int[] value = new int[1];
value[0] = 2;

// call a method
obj.setValue(value);

// and in setValue
public void setValue(int[] value) {
  value[0] = 5;
}

Or second approach use an holder class:
public class Holder<T> {
  public T value;
  public Holder(T value) {
    this.value = value;
  }
}

// then use it in this way
Holder<Integer> h = new Holder<Integer>(2);

obj.setValue(h);

// and in setValue
public void setValue(Holder<Integer> h) {
  h.value = 5;
}

In this case I use an holder class implements with generics but you can have a simple holder too, only for integer. For example:
public class IntHolder {
  public int value;
  public IntHolder(int value) {
    this.value = value;
  }
}

Comment: dear SmRndGuy, this is the answer you are looking for. (PS: I didnt know about Holder class, so thank you dash1e .)
Comment: @bluebrain pay attention, is not a Java standard class, you have to write your Holder by yourself, but it is very simple. Maybe, sometimes, you can find Holders in different libraries because it is an usual way to solve that problem.
Comment: That generic `Holder` class a nice way of doing a "wrapper".
Answer: Java always passes by value and there are no global variables as in the C++ sense. So if you want to do the same as in C++ you need to return the new value.
Thusly:
public int change(int x) {
    return ++x;
    // or
    // return x + 1;
}

To test it:
int x = 2;

change(x);

System.out.println(x); // returns 2

x = change(x);

System.out.println(x); // returns 3

So it doesn't make any sense to let the method be called change, it is more sensible along the lines of calculateThisInt.

Java does pass objects by value. But as Mark Byers mentions the Integer class is immutable and you could use MutableInt from Apache Commons library. To describe how this works you could implement it yourself for your example:
public class MyInt() {

    public int i;

    public void setInt(int i) {
        this.i = i;
    }

    public int getInt() {
        return this.i;
    }

    public int increment() {
        this.i++;
    }

}

You need to change your change function to have the above MyInt object as argument:
public void change(MyInt i) {
    i.increment();
}

Usage:
MyInt x = new MyInt();
x.setInt(2);

change(x);

System.out.println(x.getInt); // returns 3

In your case you want to change a Vector object...
public void changeVector(Vector v) {
    // anything you do with 'v' will change it even
    // for the scope that called this method
}

// Usage:
Vector v = new Vector();
changeVector(v);
// v should be changed after calling change vector method

Hope this all makes sense.
Comment: you need to use return ++x;  instead of return x++;
Comment: Objects are not passed by reference in Java. The reference to an object is passed by value.
Comment: @his: Okay, I got concepts mixed up. I guess I need more coffee this morning. :-P
Comment: Still a little misleading: "Java does pass objects by value". As there are no value objects in Java this is not quite exact. "Java does pass objects _references_ by value" would be better.
Comment: IMHO "Java does pass objects by value" is more confusing than the pre-edit version "Java passes objects by reference". To a casual reader "pass by value" makes it sound like any changes made to the fields on an object, within a method, would not affect the original object, as if a COPY of the object were sent to the method. Personally, I would drop that sentence. Also drop "But" from the following sentence. The important point is that Integer is IMMUTABLE, and what is needed is a MUTABLE instance.
Answer: While you can't replace an object that's been passed to a function, you can change its state by altering fields directly or calling methods. If you need something like a pointer to a primitive, wrap it in an object. To follow your code, you could do this:
public class IntPointer {
    public int value;
    public IntPointer(int value) {
        this.value = value;
    }
}

Then elsewhere you could say:
public static void change(IntPointer ipoint) {
    ipoint.value++;
}
public static void main(String[] args) {
    IntPointer a = new IntPointer(10);
    change(a);
}

This might seem a bit awkward, but it hasn't come up for me as often as you'd think.  I'd be more likely to do something like this:
public class ABPair {
    private int a = 0;
    private int b = 0;
    public static void changeA() {
        a++;
    }
    public static void changeB() {
        b++;
    }
}

So that elsewhere I can say:
public static void main(String[] args) {
    ABPair ab = new ABPair();
    if (ACondition) {
        ab.changeA();
    }
}

In other words, my data tends to already be wrapped in some sort of object, and I tend to use the data object's methods to mediate any changes.
Answer:
Both will change the global x, right?
So how can I do something like that in java? Specifically, I want to
  point to a Vector object

The question is somewhat vague, but I got the impression that you ultimately want a global Vector that you can keep stuff in?
Many ways to do that, but one of the simplest is to have a static field in a class, with public static methods for accessing it. (Or simply a public static field which is accessed directly, but that really wouldn't be idiomatic in Java.)
public class Foo {
    private static List<Integer> globalVector = new Vector<Integer>();

    public static void add(int number){
         globalVector.add(number);
    }

    // ... plus whatever other accessors to the global list that you need
}

Anywhere else in code:
Foo.add(23); // modifies the global vector

(Btw, Vector is kinda obsolete, and typically we'd use ArrayList in its place now. As the Javadoc says, it's been retrofitted to implement the List interface, which I also used in the example.)
Comment: **Disclaimer**: I may have totally misinterpreted what OP wanted, and will delete this if that turns out to be the case. The question really should be clearer on what it is that OP actually wants to achieve with his code.
Answer: Java supports what it calls "references". References act alot like pointers in C/C++-like languages. They don't act the same way "references" work in those languages.
The major differences between a pointer in C and a reference in Java are:
You can't do pointer arithmetic in Java (i.e. you can't "add" or "subtract" from a Java reference, you can only dereference it or compare it with another one).
You can't cast it to an incompatible type: Java is strongly type-safe, you can't "re-interpret" the bytes in memory as some other object.
For some uses of pointers this has no real effect (for example linked lists work pretty much the same in both languages), for others the difference is quite major (arrays in C are just fancy pointer arithmetic, in Java they work quite differently).
So in a way Java references could be called "restricted pointers".

How Java "pointers" work?


LINQ - Group DataTable by multiple columns determined at runtime

Question: Using .NET 3.5, I have the need to group a DataTable by multiple columns, where the column names are contained in an IEnumerable.
// column source
IEnumerable<string> columns;
DataTable table;

IEnumerable<IGrouping<object, DataRow>> groupings = table
    .AsEnumerable()
    .GroupBy(row => ???);

Typically ??? would be an anonymous type as described here, but I need to use columns as the column source.  Is this possible?
Answer: The simplest way to do this is to create a function which selects the required columns and creates a hash for comparison. I'd do something like this:
Func<DataRow, IEnumerable<string>, string> f = (row, cols) =>
    String.Join("|", cols.Select(col => row[col]));

This is a function taking a DataRow and an IEnumerable<string>. It projects the IEnumerable<string> (the column names) into the corresponding column values (cols.Select(col => row[col]))), and then joins those values with a | character. I chose this character because it's an unlikely candidate to be included in your fields, but you might want to swap for another delimiter.
Then simply
IEnumerable<IGrouping<object, DataRow>> groupings = table
    .AsEnumerable()
    .GroupBy(row => f(row, columns));

Ideally we would select into a better type - not a string tied together with a arbitrary delimiter. But I expect that selecting into an object will cause problems due to the comparison of reference types, as two objects aren't equal even if they have identical properties.
Comment: @Joanna glad it helped. The reason it works is because the string that the function returns isn't just 'any string', it's a string containing the column values. If the column values match for any two rows, then the strings will match, and they'll be 'grouped'.
Comment: I had to add a cast in the projection to make it work:
            `Func, string> f = (row, cols) =>
                    String.Join("|", cols.Select(col => (string)row[col]).ToArray());` - but I wonder if you could explain how this line actually works ? `.GroupBy(row => f(row, columns))` - the function returns a string, but if you replaced it with a string it wouldn't work! How does the grouping know to look at the column values once it constructed the string in func? I'd be happy if you can point me to any resources explaining this.
Comment: so it's doing `(row => f(row, columns))`, ends up with values like `row => 1|10` or `row => 2|987` and these become Keys for the groupings, so that if any of them is repeated it will be grouped with the previous occurence. Ok, that makes perfect sense. Thanks!


How to solve the system of differential equations?

Question: Let $x_i=x_i(t,s),i=0,1,2$ be  functions of $t,s$ and consider the system of differential equations
\begin{cases}
\frac{\partial x_0}{\partial t}=0,\\
\frac{\partial x_1}{\partial t}=x_0,\\
\frac{\partial x_2}{\partial t}=2x_1,\\
\frac{\partial x_0}{\partial s}=2x_1,\\
\frac{\partial x_1}{\partial s}=x_2,\\
\frac{\partial x_2}{\partial s}=0,
\end{cases}
How to solve it?
My attemt.
First of all  I have solved the first 3 equations and get
\begin{align}
 &x_{{0}}=C_1 \left( s \right) ,\\
&x_{{1}}  =C_1 \left( s \right) t+C_2(s),\\
&x_{{2}}  =C_1 \left( s
 \right) {t}^{2}+2\,C_2 \left( s \right) t+C_3 \left( s
 \right),
\end{align}
and substitute into the last 3 equation
$$
\begin{cases}
\displaystyle {\frac { d}{{ d}s}}C_1 \left( s
 \right) =2\,C_1 \left( s \right) t+2\,C_2 \left( s
 \right),\\
\\
 \displaystyle t  {\frac { d}{{ d}s}}C_1 \left( s \right)+{\frac { d}{{ d}s}}C_2 \left( s \right) =C_1 \left( s \right) {t}^{2}+2\,C_2 \left( s \right) t+C_3 \left( s \right) ,\\ \\
 \displaystyle {t}^{2}\, {\frac { d}{{ d}s}}C_1
 \left( s \right)  +2\,t  {\frac { d}{{ d}s}
}C_2 \left( s \right) +{\frac { d}{{ d}s}}C_3 \left( s \right) =0,
\end{cases}
$$
or normalize it:
$$
\begin{cases}
\displaystyle{\frac { d}{{ d}s}}C_1 \left( s
 \right)=2\,C_1 \left( s \right) t+2\,C_2 \left( s \right) ,\\ \\
\displaystyle{\frac { d}{{ d}s}}C_2 \left( s
 \right)=-C_1 \left( s \right) {t}^{2}+C_2 \left( s \right) ,\\ \\
\displaystyle{\frac { d}{{ d}s}}C_3 \left( s
 \right)=-2\,C_2 \left( s \right) {t}^{
2}-2\,C_2 \left( s \right) t
\end{cases}
$$
Then I solve the system for $C_1(s),C_2(s),C_3(s)$ and get
\begin{align*}
&C_1 \left( s \right) =-{\frac {2\,C_1'\,{s}^{2}{t}^{2}+4\,C_2'\,s{t}^{2}+2\,C_1'\,st+4\,C_3'\,{
t}^{2}+2\,C_2'\,t+C_1'}{4{t}^{3}}},\\
&C_2 \left( s
 \right) =\frac{1}{2}\,C_1'\,{s}^{2}+C_2'\,s+C_3', \\
&C_3
 \left( s \right) =-\,{\frac {2\,C_1'\,{s}^{2}{t}^{2}+4\,C_2\,s{t}^{2}-2\,C_1'\,st+4\,C_3'\,{t}^{2}-2\,C_2'\,t+C_1'}{4t}}.
\end{align*}
Here $C_1',C_2',C_3'$ are constants.
Then I substitute it  into the above expression for $x_0,x_1,x_2$  and get
\begin{align*}
&x_{{0}} \left( t,s \right) ={\frac { \left( -2\,C_1'\,{s}^{2}-4\,C_2'\,s-4\,C_3' \right) {t}^{2}+ \left(
-2\,C_1'\,s-2\,C_2' \right) t-C_1'}{4{t}^{3}}}, \\&x_{{1}
} \left( t,s \right) ={\frac { \left( -2\,C_1'\,s-2\,{C_2'} \right) t-C_1'}{4{t}^{2}}},\\ &x_{{2}} \left( t,s \right) =-
\,{\frac {C_1'}{2t}}
\end{align*}
But obviously it is not solutions of the initial system!
Where is my mistake?
Note, the system have solution, for example $x_1^2-x_0 x_2$ is its first integral.
Comment: I guess you missed something, since $x_0$ can be expressed only in terms of $s$ $x_0=2C_1(s)t+2C_2(s)$ would only be true if $C_1(s)$ is nil and this deadlocks the system, or maybe i did.
Comment: no, I have cheked all my steps
Comment: wel tbh this is a mystery!!
Comment: leox, notice that the derivate of $x_1$ in terms of $t$ gives $x_0$ then in term of $s$ gives itself $2x_1$, which kind of function you derivate it twice it yields to itself ? we are not dealing with polynomials here, its either sine or exponential.
Answer: $$
\frac{\partial x}{\partial t}=M_1 x\Rightarrow x = C_1e^{M_1 t}+\phi(s)\\
\frac{\partial x}{\partial s}=M_2 x\Rightarrow x = C_2e^{M_2 s}+\psi(t)\\
$$
then
$$
x = C_1e^{M_1 t}+C_2e^{M_2 s}
$$
but for this solution we have
$$
\frac{\partial x}{\partial t}-M_1 x = -M_1C_2e^{M_2 s}=0\Rightarrow C_2 = 0\\
\frac{\partial x}{\partial s}-M_2 x = -M_1C_1e^{M_1 t}=0\Rightarrow C_1 = 0\\
$$
hence the solution is $x = 0$
Answer: The first and last equations give $x_0(t,s) = x_0(s)$ and $x_2(t,s) = x_2(t)$. Therefore, the system becomes
\begin{aligned}
\partial_{t} x_1 &= x_0 \\
\partial_{t} x_2 &= 2x_1 \\
\tfrac{\text d}{\text d s} x_0 &= 2x_1 \\
\partial_{s} x_1 &= x_2
\end{aligned}
Applying $\partial_{t}$ to the third line yields $0 = 2\partial_{t} x_1$ and the first line gives $\partial_{t} x_1 = x_0$. Hence we know $x_0=0$ and $x_1(t,s) = x_1(s)$. The system
becomes
\begin{aligned}
\partial_{t} x_2 &= 2x_1 \\
0 &= 2x_1 \\
\tfrac{\text d}{\text d s} x_1 &= x_2
\end{aligned}
Since $x_1=0$ as well, we end up with $x_2=0$. The solution must be
$$
x_0 = x_1 = x_2 = 0.
$$


ActionView::MissingTemplate, how can I resolve?

Question: After sending the file, the controller executes the render nothing: true, and should show the registered image on the screen.
In a show view, to add a banner field image to a banner category, when uploading the file after refresh, has ActionView :: MissingTemplate response error and headers error 500.
https://i.stack.imgur.com/OIBOO.png
https://i.stack.imgur.com/vwDAi.png
change render nothing: true, to render head :ok
banners_controller.rb
class Admin::BannersController < Admin::PositionUpdaterController
  belongs_to :banner_category, parent_class: BannerCategory
def create
    @gallery = BannerCategory.find params[:banner_category_id]
<code>@image = @gallery.banners.build

@image.image = params[:file]

@image.save

render nothing: true
</code>
end
In a show view, to add a banner field image to a banner category, when uploading the file after refresh, has ActionView :: MissingTemplate response error and headers error 500.
Answer: I was able to change and render nothing: true, to render json: nil, status :: ok


Updating node title inside Rules Action

Question: I am creating a Rules Action that will update the node title after it is created.
Here is my code, but when I try to create new content using Postman, it keeps loading. I dont get response in Drupal logs.
...
class ContentTitle extends RulesActionBase {

  protected function doExecute(EntityInterface $entity) {

    $entity_id = $entity->id(); // Getting dynamic entity ID
    $controller = \Drupal::entityManager()->getStorage('node');
    $node = $controller->load($entity_id);

    $title = $entity->label();
    $item = $node->get('field_xyz')->value;
    $new_title = "Title is " . $title . $item;

    \Drupal::logger('custom_rules')->notice('new_title: ' . $new_title);

    $data = array(
      'type' => 'custom_content',
      'title' => [
        'value' => $new_title,
      ]
    );
    $node = Drupal::entityManager()
      ->getStorage('node')
      ->create($data);
    $node->save();
  }
}

Im getting the expected output in the logger.
I dont think I am setting the title in $data properly. Any suggestion is greatly appreciated. Thanks!
Answer: You can set a node's title with the "Set a data value" action - you don't need to write your own action. I'm going to assume that you're putting this into your own action because there are other things you want to do that you haven't mentioned or because you are learning how to write an action - if this is ONLY about setting the title then I don't see the point of a custom action.
Problems with the approach and the partial solution given in the initial post:

You don't need to get the entity ID then load the entity from
storage, because Rules is already passing you the loaded entity.

You're using entity instead of node, which wrong - Nodes have
titles, Entities in general DON'T.

You're hardwiring in a title taken from a specific field 'field_xyz'.
Instead, you should pass in a string variable which holds the title.
That way the Rule configuration can get the value out of 'field_xyz'
or out of 'field_nlm' or from a thousand other places - writing the
action with a string parameter will make it far more useful.

The EntityManager has been deprecated since Drupal 8.0 and should not be used. You should be using EntityTypeManager instead. But as I said in #1, this isn't even needed.

You don't have to save the node, Rules will handle the saving
automatically.

This really only requires one line of code (plus some boilerplate code, like all plugins...). Here's a complete solution that takes two context variables:

The node to change.
The title to use.

namespace Drupal\my_module\Plugin\RulesAction;

use Drupal\node\NodeInterface;
use Drupal\rules\Core\RulesActionBase;

/**
 * Sets the title of a content item.
 *
 * @RulesAction(
 *   id = "rules_node_set_title",
 *   label = @Translation("Sets the title of the selected content item"),
 *   category = @Translation("Content"),
 *   context_definitions = {
 *     "node" = @ContextDefinition("entity:node",
 *       label = @Translation("Node"),
 *       description = @Translation("Specifies the content item to change."),
 *       assignment_restriction = "selector"
 *     ),
 *     "title" = @ContextDefinition("string",
 *       label = @Translation("Title"),
 *       description = @Translation("The new title.")
 *     ),
 *   }
 * )
 */
class NodeSetTitle extends RulesActionBase {

  /**
   * Executes the action with the given context.
   *
   * @param \Drupal\node\NodeInterface $node
   *   The node to modify.
   * @param string $title
   *   The new title.
   */
  protected function doExecute(NodeInterface $node, $title) {
    $node->setTitle($title);
  }

  /**
   * {@inheritdoc}
   */
  public function autoSaveContext() {
    // The node should be auto-saved after the execution.
    return ['node'];
  }

}

Answer: You can create node like below also:
use \Drupal\node\Entity\Node;

class ContentTitle extends RulesActionBase {

  protected function doExecute(EntityInterface $entity) {
    $entity_id = $entity->id(); // Getting dynamic entity ID
    $controller = \Drupal::entityManager()->getStorage('node');
    $node = $controller->load($entity_id);
    $title = $entity->label();
    $item = $node->get('field_xyz')->value;
    $new_title = "Title is " . $title . $item;
    \Drupal::logger('custom_rules')->notice('new_title: ' . $new_title);
    $node = Node::create([
      'type' => 'content_type',
      'title' => $new_title,
    ]);
    $node->save();
  }
}

In your code "title" should be only $new_title not in the array as value. Even your code also looks fine except that title. I didn't any try of your code but I am suggesting to use above one.
Comment: Thanks for your reply. Im getting `172.18.0.1 - - [19/Jun/2018:10:11:13 +0000] "-" 408 - "-" "-"` in my access log
Comment: This answer has all the same problems as the origin post, and doesn't provide any significant help or guidance on how to properly write a RulesAction.
Answer: Simple answer from https://drupal.stackexchange.com/a/249771/70281 did the trick for me.
$node->set('title', $new_title);
$node->save();

Comment: This is not good, because if the "entity" you pass in doesn't have a 'title' property, this will fail. Entities do not in general have a title.

Updating node title inside Rules Action


Add Resources to IngressRule as IPeer

Question: const ecsAdminService = new ecs.FargateService(this, "AdminService", {
  cluster,
  taskDefinition:taskDefinitionAdmin,
  desiredCount: desiredCount,
  vpcSubnets: props!.vpc.selectSubnets({ subnetType: ec2.SubnetType.PUBLIC }),
  assignPublicIp: true,
  securityGroups:[adminServiceSg],
  enableExecuteCommand:true,
  serviceName: serviceName
});

props!.dbSecurityGroup.addIngressRule(ecsAdminService, ec2.Port.tcp(3306),'allow mysql port');

I want to add the fargate to addIngressRule and addIngressRule required the IPeer
However I am not sure how to change the fargate to IPeer.
How can I make this?
Answer: Refer to this answer on the topic: Best way to organize security group rules in AWS CDK
The recommended way to do this is to use CDK's abstractions. It would look like this:
ecsAdminService.connections.allowToDefaultPort(props.db);

You would need to pass the database instance/cluster in the props, not the security group. The required security group rules will be created automatically.
Comment: Ah I see. I think it is relevant with the basic structure philosophy of CDK. Thank you very much.

Add Resources to IngressRule as IPeer


Regex, Match uppercase characters not between brackets

Question: In RegEx, I search a pattern that selects multiple uppercase characters (more than 1), that are not enclosed by curly braces.
It should match:
ABC
AB
XYZABC

but not:
{ABC}
{AB}
{XYZABC}

Answer: The below regex would match one or more uppercase letters only if it is not followed by a closing curly } bracket.
^[A-Z]+(?!.*?})$

DEMO
OR
You could use perl regex verbs,
{.*?}(*SKIP)(*F)|[A-Z]+

DEMO
Comment: character class must be [A-Z] and not [A-z]
Answer: try this pattern
[A-Z]+(?![^}{]*})

Demo
Answer: Try this pattern:
{.*?}|([A-Z]+)

Then test group1 if not empty
Demo


How to use data augmentation in the context of model evaluation in machine learning?

Question: I'm trying to use data augmentation for training a model for a classification task.
But I'm not sure about how to use data augmentation in a fair and meaningful way in the evaluation of a machine learning model.
I have considered at least three different options for doing this and I would like to know which one is the best.
Firt of all, let us suppose that we have a labeled dataset D.
Option 1:
<code>-We can use a data augmentation approach over D, for generating a novel set of synthetic data S.
-We can create a new dataset D2, which combines D and S.
-With D2, we can use a 10-fold cross-validation approach, where:
    -We divide D2 into 10 random subsets (folds).
    -We use each fold as test data and the remaining 9 folds as training data.
    -This is repeated for each fold.
    -We obtain the average performance (on the 10 folds) for evaluating the model.
</code>
Option 2:
<code>-We can use a data augmentation approach over D, for generating a novel set of synthetic data S.
-With D, we can use a 10-fold cross-validation approach, where:
    -We divide D into 10 random subsets (folds).
    -We use each fold as test.
    -We create a set T including the remaining 9 folds.
    -We create a new set T2 that combines T and S
    -We use T2 as the training data.
   -This is repeated for each fold.
   -We obtain the average performance (on the 10 folds) for evaluating the model.
</code>
Option 3:
<code>-From D, we can use a 10-fold cross-validation approach, where:
       -We divide D into 10 random subsets (folds).
       -We use each fold as test.
       -We create a set T including the remaining 9 folds.
       -We use a data augmentation approach over T for generating a set S of synthetic samples.
       -We create a set T2 that combines T and S.
       -We use T2 as the training data.
       -This is repeated for each fold.
       -We obtain the average performance for evaluating the model.
</code>
What is the best option and why?
Answer: Option 3.
Options 1 and 2 have an information bleed-through issue between your train/validation data. Because you are creating S BEFORE splitting into folds, you will have synthetic data that was created from observations in your validation fold. By having data that is in some way only a slight permutation of your validation data, you haven't properly separated your validation/train data.
Option 3 is the only one that does not have the bleed-through issue because S is only created based on T.


PHP array_slice or equivalent on array of arrays that are keyed by ID

Question: I have a multi-dimensional array that is keyed by uuid's and need to slice/pop/unset an element by uuid (i.e., if I had a410463e-7fe2-4fba-8733-a812c0ee8c54 and wanted to remove that item by that uuid) so that the result is essentially the same minus the one item that was removed:
array:5 [
  "5fc29794-9e08-4944-ba6d-4a5fcde5c88b" => array:3 [
    "id" => "5fc29794-9e08-4944-ba6d-4a5fcde5c88b"
    "name" => "fuga"
    "value" => 0
  ]
  "a410463e-7fe2-4fba-8733-a812c0ee8c54" => array:3 [
    "id" => "a410463e-7fe2-4fba-8733-a812c0ee8c54"
    "name" => "nihil"
    "value" => 0
  ]
  "c141d973-91fe-4227-8985-04bd0665f4a8" => array:3 [
    "id" => "c141d973-91fe-4227-8985-04bd0665f4a8"
    "name" => "eaque"
    "value" => 0
  ]
  "17030897-1aa9-487d-a4be-d574dd0c9d9b" => array:3 [
    "id" => "17030897-1aa9-487d-a4be-d574dd0c9d9b"
    "name" => "eveniet"
    "value" => 3
  ]
  "901d9f8f-573f-444f-8562-0cdf5888ba6e" => array:3 [
    "id" => "901d9f8f-573f-444f-8562-0cdf5888ba6e"
    "name" => "in"
    "value" => 6
  ]
]

I know how to slice by index, but am having trouble finding resources on how this might be achieved. This is for a phpunit test. I've tried unset, but can't seem to store that in a variable or just call it in an assertion:
unset($array1[$id]);
unset($array2[$id]);

Does not persist the change.
$newUnchanged = unset($array1[$id]);
$oldUnchanged = unset($array2[$id]);

Throws syntax error, unexpected 'unset' error. Ultimately I want to assert that all of the unchanged items remained the same as prior to a single item being updated. I've also tried this ugly business which is removing a single item, but not the correct one:
$keyOne = array_search($id, array_keys($array1), true);
$oldUnchanged = array_slice($array1, $keyOne, null, true);

$keyTwo = array_search($id, array_keys($array2), true);
$newUnchanged = array_slice($array2, $keyTwo, null, true);

// Shows that the item that I wanted to slice still exists in both arrays
dd($id, $oldUnchanged, $newUnchanged);

// ^ Causes this test to fail
$this->assertEquals($oldUnchanged, $newUnchaged);

Comment: So you're just wanting to remove an item based on it's key?
Comment: Yes. But the key is an id and not an index.
Comment: Ok. I'm a little confused at the minute since the keys in your example are the same as the ids. Can you explain why something like `unset()` wouldn't work? Also, are you able to provide the code you have so far?
Comment: What item(s) do you want to remove? Can you include the array as an var_export instead?
Comment: I started with unset, but I'm getting a `syntax error, unexpected 'unset'`. I'm trying to remove any item I have the id for. The use case is that I am updating just one of the items and then attempting to test that all of the others were unchanged. The above array was cast from a jsonb postgres column.
Comment: I'm assuming that `$service` is a model and  `all_service_data` is a collection?
Comment: $service is an Eloquent model and all_service_data is an array cast from a jsonb column.
Comment: I updated the above code to attempt to be even more clear. Forget the origin of the arrays. They are in the above format, keyed by a uuid, and I have a single uuid and need to remove the element that is keyed by that uuid from both arrays to compare the two arrays, minus the one element that I updated and then removed from them to ensure that the items I did not want to update did not get updated.
Answer: I figured out I have to clone the arrays before I can unset them
$oldUnchanged = $array1;
unset($oldUnchanged[$id]);

$newUnchanged = $array2;
unset($newUnchanged[$id]);


Estimating the number of nodes of a discretization scheme

Question: Working on a problem in partial differential equations, I have come across a function
$$f\colon [0, T]\to \mathbb{R}_{\ge 0}$$
which is continuous and non-decreasing, and whose maximum is $M=f(T)$. I need to have a control over its growth, so I discretized the time interval:
$$[0, T]=[0, \tau_1)\cup [\tau_1, \tau_2)\cup\ldots\cup [\tau_{K-1}, T],$$
where the nodes $\{\tau_k\ :\ k=0\ldots K\}$ are chosen in such a way that
$$f(\tau_{k+1})-f(\tau_k)\le \frac{1}{C}$$
for a fixed constant $C$.

Question. Can we give an upper bound over $K$, the number of nodes?

Specifically, I hope that $K$ can be bounded above by some quantity which depends only on $C$ and $M$ and not on $f$ and/or $T$. Is this true?
Comment: Is it a lower bound that you want?
Comment: @Tomás:  No, sorry, I am interested in an upper bound. The least nodes, the better.
Answer: It is pretty clear that the nodes can be chosen in such a way that $K\leq MC$. This is sharp, for instance, for linear functions.


Uniformly distributed points over the surface of the standard simplex

Question: I would like to generate points that are uniformly distributed over the SURFACE of a standard $k$-simplex ($k$ dimensions, $k+1$ vertices).
One way to efficiently generate points that are uniformly distributed over the ENTIRETY of the $k$-simplex is to generate samples of $k$ iid exponential random variables, $E_i$, $i\in \left[1,k\right]$, then normalize, and $\vec{X}=\frac{\left\langle E_1,E_2,\dots,E_k\right\rangle}{\sum_{i=1}^k E_i}$, will be uniformly distributed over the entire $k$-simplex.
My problem is that the probability that one of these points lands on the surface (i.e. at a vertex, or along a hyper-edge, or on hyper-face, etc.) is nearly $0$, so that in practice all of the points end up being in the INTERIOR of the $k$-simplex.
In an attempt to sample the SURFACE of the $k$-simplex I have considered each hyper-edge, hyper-face, etc. separately and generated uniformly distributed samples on each (since they are themselves lower dimensional simplices). However, the problem becomes computationally unfeasible as $k$ increases because the number of hyper-edges, hyper-faces, etc. blows up.
Does anyone know of a method to explicitly generate samples that are uniformly distributed over the SURFACE (or BOUNDARY) of a standard $k$-simplex?
Answer: Notice that faces of higher codimension than $1$ are negligible (they have measure zero), so to generate your random point you just need to:

Pick a random integer from $0$ to $k.$ That will tell you which (top dimensional) face of the surface of the simplex you are working with.

The top dimensional faces are themselves simplices, so

Use the algorithm you describe to generate a random point on the face you chose.

EDIT For the OP's nonuniform question, the simplest solution is to do what I said before, but if the point is too close to the boundary of the face (so, in three dimensions, you are picking a point from a triangular face, and seeing if it is too close to the edge), go to the closest point of that boundary, and then iterate the process.
Comment: That is a great insight. However, in spite of the fact that faces of higher codimension that 1 have measure zero, points on such faces are often more important for my application. In fact, my interest in a given face decreases slightly with decreasing codimension :), so I somehow need to uniformly sample all faces regardless of co-dimension, without explicitly considering each face, since the number of faces blows up.
Comment: @okj in that case, I am not sure what you are looking for, because any uniform sampling method will get you $0$ points on higher codimension faces on average.
Comment: @okj so you are looking for something non-uniform, in which case you need to be precise about what it is...
Comment: A geometrical example might help. Consider a 3-simplex (a tetrahedron), the final sample set should consist of points on edges, and faces (if it doesn't get the vertices that's ok, they are easy to add), and the number of points from each edge (or face) should be roughly equal. The average number of points on an edge should be less that the average number of points on a face, for example, if, on average, there are $10^1$ points on an edge, then, on average, there should be maybe $10^2$ points on a face.
Comment: @okj see the edit...


What legitimate reason would an IT Admin have for requesting a user's password to setup a new laptop with Windows 10?

Question: The IT admin person where I work is currently setting up a new laptop for me and they've asked for my user password so they can finish the job. I'm obviously uncomfortable with that.
I suspect not, but is there any reasonable reason to need this information from me? And if not, is there a plausible reason why an inexperienced admin might prefer to just log in as me?
Comment: Configuring your user profile. Setting up your email client. Installing your printers. Installing your applications. Verifying that you have access to the resources you need. Saving you from having to do all of that yourself. I can think of many reasons why the MIGHT ask you for your password.
Comment: Well... not entirely. In order to configure your user profile they need to log on as you... with your password. In most cases it's easier to perform these steps while logged on as the user rather than have the user perform them themselves while IT guides them through the process. Tools like Autopilot and Intune can make some of this more "zero touch", but it sounds like your team isn't using those kinds of tools, or at least not fully.
Comment: Can't any of that be done without my password? That seems like a massive security risk.
Answer: This is a common error some IT do by lacking to explain why.
If the request is authentic, an actual laptop change.
Technically speaking he could just change your password and give it to you when the job is done.
In COVID Time if working from home it might break your VPN ressource if the IT change it.
The best way of action is to change it yourself and give a temporary password to them.
He possibly want to setup all your app and copy back your data under your own username.
If working from home, that will allow your profile to be cached too before he ship the laptop
Comment: Thanks yagmoth. I'm 100% convinced that the person in question isn't up to something. It just seems unlikely to me that he can't complete his work without my password - but this isn't my domain, hence the question.
Comment: @JoeH He need to log to copy from data if you want all your data copied over (desktop data, favorite, etc), but like I told he can do it by resetting your password or with you around. If you are able to copy your data you can state it to him too. Some application do it without user intervention, but many enterprise don't have such as it has cost related to that, its why the manual step
Comment: He won't be copying any data. I'll just be getting a standard issue laptop and the only data from the old one will be from dropbox.
Answer: The real reason behind the request is lack of knowledge and professionalism. There is no excuse to do it like that, but it is often an easier way for IT support than doing things correctly. They honestly don't understand that they are compromising their own security by asking users for passwords.
If users should be taught one thing about security, it is: IT support should NEVER ask for your password.
Comment: Thank you Jevgenij. This confirms my suspicions. I'm really not sure what it is he needs my password for.
Comment: @Joe H you have a legitimate concern. Why don't you ask them why they need your password and have them explain it to you? We can only give you possible reasons why they might ask you for your password, but we can't tell you the specific reasons they're asking you for it.


Duration of articulations (e.g. staccato, tenuto) for piano

Question: QUESTION
Staccatissimo, staccato, mezzo-staccato, portato, non-legato, tenuto, legato.*
For piano, what would be a (rough) numerical representation of these articulations' gap duration for note duration of 0.5 and 1 second?
*Of them, staccato, non-legato, and legato are the more critical (as occurring more often in sheet music).

BACKGROUND
What the question is about, or what an answer should look like.
I am looking for an answer that might look like:

Sample answer 1
note 0.5s : . . staccato (sound 250ms, gap 250ms) . . legato (sound 500ms, gap 0ms)
note 1.0s : . . staccato (sound 750ms, gap 250ms) . . legato (sound 1s, gap 0ms)

where ". ." represents the space for the other articulations (e.g. staccatissimo to the far left).
Please note that there are two elements to this answer:  (a) an ordering of the articulations in terms of gap length and (b) an indication of what happens when the note becoming longer.
In the above example, when the note becomes longer, the gap remains constant.  But we could imagine answers in which sound and gap are proportional or where the sound duration remains constant:

Sample answer 2 (proportional)
note 0.5s : . . staccato (sound 250ms, gap 250ms) . . legato (sound 500ms, gap 0ms)
note 1.0s : . . staccato (sound 500ms, gap 500ms) . . legato (sound 1s, gap 0ms)

Sample answer 3 (constant sound duration)
note 0.5s : . . staccato (sound 250ms, gap 250ms) . . legato (sound 500ms, gap 0ms)
note 1.0s : . . staccato (sound 250ms, gap 750ms) . . legato (sound 1s, gap 0ms)

I realize that the precision of numbers like 250ms may be ridiculous and am using them only to illustrate my purpose.
Your answer may therefore simply (a) order the articulations from short sounding to long sounding and (b) indicate whether "sound" or "gap" should tend to remain constant (or whether a proportionality applies).
What I am not looking for, and what an answer should not be made of.
I am not asking what e.g. a staccato is.  Therefore if you tell me e.g., "A staccato is not just a matter of duration, but it also involves. . . .," that might be very useful information (and I would appreciate it) but would not be responsive.
I am also not asking how to play e.g. a staccato.  So again if you tell me that the finger, the wrist, or the arm should go up or not go up, that too may be very useful but not responsive.
A quick way to contextualize may question, I believe, would be to say that I am entering the articulations into computer in MIDI or some other numerical notation and need a rough guideline on appropriate sound/gap duration.
One final note on non-legato.  I have read here and there that non-legato is not a separate articulation, but its absence, or the way articulation is left to the performer.  If you are of this opinion, I realize that you could not give any sound/gap duration for non-legato.
Thank you very much.
Comment: Unfortunately, you explicitly ban the answer that reflects human performance best, because, well, "staccato is not just a matter of duration." Aside from various genres or contexts that pursue homogeneity, like maybe some minimalism, most human-performed music values variability for expressive purposes. I could face a measure of staccato notes and quite intentionally make some longer than others.
Comment: (I realized what I'm thinking of: Debussy violin sonata, 2nd mvt. Many measures of repeated 16th notes on one pitch, all marked staccato; my goal would be for no two consecutive notes to be the same duration.)
Comment: @AndyBonner Alas, a beginner has to learn first the *un*varied so he'd know what he is varying *from*.  Otherwise his variability will be random rather than thoughtful or instinctive.  Thanks for turning me on to the Debussy! Internet had Oistrakh.
Answer: TL;DR
In order to have unique durations for each articulation, use a formula along the lines of PerformedDuration = NotatedDuration * ListOrderOfArticulation / NumberOfDistinctArticulations (See General Answer). Or use a variation on MuseScore's default duration percentages (See MuseScore definitions).
General Answer
Since all articulations are interpretive and contextual, all answers to this question will be arbitrary. However, given the idea that a computer needs a fixed indication of how to "perform" each marking, the primary decision is whether the relationships should be linear (i.e., constant differences between "adjacent" durations) or logarithmic (i.e., adjacent durations differ by, say, a factor of 2).
Given the "computerization" context, I think linear relationships will be most successful, as they give the clearest delineation between one articulation and another.
The shortest (pitched) duration the human ear can discern is about 100ms. (See, for example, What's the shortest sound perceptible to the human ear?) So, let D be the duration in milliseconds of some note such that D > 100ms. Then the following table emerges:

articulation
duration (ms)

staccatissimo
100 + (D-100)/7

staccato
100 + 2(D-100)/7

mezzo-staccato
100 + 3(D-100)/7

portato
100 + 4(D-100)/7

non-legato
100 + 5(D-100)/7

tenuto
100 + 6(D-100)/7

legato
D

There is also a convention that staccato is 1/2 the notated duration. Going with that definition, I would say:

staccatissimo = 1/4 of the notated duration
staccato = 1/2 of the notated duration
mezzo-staccato = portato = non-legato = tenuto = 3/4 of the notated duration
legato = exactly the notated duration

MuseScore definitions
These are the default percentages of a note's full duration used for various articulations. They come from MuseScore's master instrument definitions file.

articulation
duration percentage

staccatissimo
33

staccato
50

portato
67

tenuto
100

marcato
67

sforzato
100

sforzatoStaccato
50

marcatoStaccato
50

marcatoTenuto
100
Comment: Huh, I always thought that Musescore portatos were 75% instead of 67%.
Comment: Thank you. Exactly what I was looking for (indeed an "Oh my God" answer).  I cannot upvote while sub 15. As the site discourages non-substantive comments, I will delete this one after some time.


Binary addition using JOLT

Question: How do I transform below JSON using JOLT

Input will be array of strings
This string will be binary number , 0 or 1.

I want to perform binary addition and then split resultant array as split and every element should be a seperate property
<code>{
  "binaryarray": [
    "0100",
    "0010"
  ]
}
</code>
Output
<code>{
  "field1" : 0,
  "field2" : 1,
  "field3" : 1,
  "field4" : 0
}
</code>
Comment: Can you explain , using words, how this transformation should work ?
Comment: @Luuk : Edited my question , let me know if this is enough
Answer: I was able to partially achieve it

Convert array of string to array of int

perform Addition of array

Convert above to string

Split above string
<code>[
   {
     "operation": "modify-overwrite-beta",
     "spec": {
       "binaryarray": ["=toInteger", 0]
     }
     },
   {
     "operation": "modify-overwrite-beta",
     "spec": {
       //
       // Sums
       "sumIntData": "=intSum(@(1,binaryarray))"
     }
   },
   {
     "operation": "modify-overwrite-beta",
     "spec": {
       "sumIntData": ["=toString", 0]
     }
     },
   {
     "operation": "modify-overwrite-beta",
     "spec": {
       "PID3": "=split('', @(1,sumIntData))"
     }
     }
 ]
</code>
Comment: Please read: [How do I format my code blocks?](https://meta.stackoverflow.com/questions/251361/how-do-i-format-my-code-blocks)
Comment: still not able to edit my answer

Binary addition using JOLT


Erro ao executar programa em python

Question: Eu fiz esse programa em python :
//////////color.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

class Color(object):

    string = {
        "RED": "\x1b[31m",
        "GREEN": "\x1b[32m",
        "BLUE": "\x1b[34m",
        "YELLOW": "\x1b[33m",
        "CYAN": "\x1b[36m",
        "GREY": "\x1b[38;5;247m",
        "ENDC": "\x1b[0m"
    }

    if use_colors in ["off", "OFF"]:
        string = {
            "RED": "",
            "GREEN": "",
            "BLUE": "",
            "YELLOW": "",
            "CYAN": "",
            "GREY": "",
            "ENDC": ""
        }

//hello.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from color import Color as colors

class HelloWorld(object):

    def __init__(self, colors):

           print("%s".format
           (
             self.colors.string["RED"], "Hello ", self.colors.string["ENDC"],
             self.colors.string["GREEN"], "World!"],
             self.colors.string["ENDC"])
           )

porem ao fazer:
$python hello.py

ele gera o seguinte erro abaixo:

File "hello.py", line 13
      self.colors.string["GREEN"], "World!"],
                                           ^ SyntaxError: invalid syntax

Qual é o erro? Pois eu testei criando uma variável chamada msg="Hello World"
e dá erro também.
Answer: Tem um colchete a mais nessa linha:
self.colors.string["GREEN"], "World!"],

Comment: Não entendi o voto negativo. O que o Gustavo falou está certo.
Comment: Deu certo aqui rodei o programa e funfou legal..

Erro ao executar programa em python


How do I troubleshoot Active Directory on MacOS 10.4 clients?

Question: We have three MacOS machines, all running 10.4.x, bound to our Active Directory network. The domain has two DCs on a single subnet (192.168.1.0/24). One of the DCs (the secondary one, which does not hold any of the 5 FSMO roles) is currently off-line. It's physically shut-down because its boot volume is hosed.
Since the secondary DC went off-line, two of the three Macs have been unable to do standard Active-Directory authentication. I can't log in to either of them as an AD user, I can't use AD credentials in a UAC prompt, and I can't even run "Directory Access" -- it starts up but then becomes unresponsive and I need to force quit to close it.
I can, however, get a Kerberos ticket for an AD user, with the "Kerberos" GUI utility. And when I run 'dsconfig ad -show' on one of the offending Macs, I see the primary DC (the one that's actually running right now) in the "Preferred domain controller" line.
And all of our Windows machines (running XP Pro) are authenticating just fine against AD -- which would rule out any replication problem, I think.
Just wondering what I can do next to troubleshoot this.
Update on Friday afternoon, August 7
Both machines just stopped having AD trouble. I did restart them both, but I had done that twice before with one of them, to no avail. So I guess the question has become academic at this point. Still interested in hearing how anyone else would troubleshoot this.
Answer: Not being able to run Directory Access is a problem.  You definitely shouldn't be seeing that.  I've run into many issues with macs and AD over the years and I've never seen that.  I'd recommend unbinding and rebinding them.  Since you are unable to use Directory Utility you're going to have to use the DSCONFIGAD command in terminal.  You can man it or view the following article:(there are many on the net that explain this)
http://www.peachpit.com/articles/article.aspx?p=1246089
Unbind and bind using the instructions in that article and see if that fixes it.  In theory, losing a DC should not have affected it's connectivity, however I've seen macs cry when and Open Directory Master goes down even though it should be able to connect to a replica, so I'm not surprised it's pissed.  You may also want to check your DNS.  If the dead DC can still be resolved by doing an NSLOOKUP on your DOMAIN then I bet the mac might be trying to hit it.  It's probably not as resilient as windows clients are when a DC dies.
Comment: "I've seen macs cry when and Open Directory Master goes down even though it should be able to connect to a replica" -- only in this case it's the replica that's down... but the master was also down briefly this morning, so maybe the Macs just got in a tizzy and never came out.
Comment: There are definitely still live DNS records for the dead DC -- I was also wondering what the best way was to temporarily nix those, but I guess that's a separate question.
Comment: If you had split dns you might be able to do it.  Binding might be an issue though as your srv records would probably get delegated down to AD DNS, those would be tought to clean out and I wouldn't recommend doing that ever anyways.  It'd be interesting to see though.  Luckily(knock on wood) I haven't had a DC failure like that yet to test it out.


Symfony return a response to export excel

Question: How do I return a response in Symfony that will output an excel file? I am getting an error that I must return a response and it needs to be in a string correct? Help!
 $excel = new PHPExcel();

 $excel->setActiveSheetIndex(0);
 $excel->getActiveSheet()
        ->setCellValue('A1', 'hi');

 $objWriter = \PHPExcel_IOFactory::createWriter($excel, 'Excel2007');
 $objWriter->save(str_replace('.php', '.xlsx', __FILE__));

 return $objWriter;

Error:

The controller must return a response (Object(PHPExcel_Writer_Excel2007) given).
Answer: Controller must return an instance of Response class, so do the following:
ob_start();
$objWriter->save('php://output');

return new Response(
    ob_get_clean(),  // read from output buffer
    200,
    array(
        'Content-Type' => 'application/vnd.ms-excel',
        'Content-Disposition' => 'attachment; filename="doc.xls"',
    )
);

If you use Excel 2007 or upper set content type to "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" and file extension to xlsx.
Comment: this is giving me a page with gibberish characters.
Comment: @StormParker Yes, I was wrong with the returning method. You may use the solution from the updated answer.
Answer: In my case, I've got a route and a controller to export an XLS generated (in my case a button access the route and returns the XLS file)
Also, I read a already generated and stored file, but it should work even if you're not storing the file, you should just use the $objWriter; in this line
$response = $this->get('phpexcel')->createStreamedResponse($writer);

Also, I'm not using the Response object that VisioN indicated I should.
The Action:
public function exportAction() {
    $readerObject = PHPExcel_IOFactory::createReader('Excel5');
    $phpExcelObject = $readerObject->load('files/downloads/reports/' . $this->getUser()->getName() . '.xls');
    $writer = $this->get('phpexcel')->createWriter($phpExcelObject, 'Excel5');
    $response = $this->get('phpexcel')->createStreamedResponse($writer);
    $response->headers->set('Content-Type', 'text/vnd.ms-excel; charset=utf-8');
    $response->headers->set('Content-Disposition', 'attachment;filename=' . $this->getUser()->getName() . '.xls');
    $response->headers->set('Pragma', 'public');
    $response->headers->set('Cache-Control', 'maxage=1');

    return $response;
}

Please note, 'Excel5' is for XLS files, if you desire XLSX files you should replace those with 'Excel2007'
Comment: That doesn't releate to the question at all. I wrote *FYI*, which means *"For Your Information"*: it doesn't state that you should use the proposed way, just to keep in mind that there is a shorter alternative.
Comment: FYI There is a `$this->getUser()` shortcut method.
Comment: Happy now? I don't see how it's relevant to the user's question.
Comment: use 'application/vnd.ms-excel' and not 'text/vnd.ms-excel', which does not exist.
( full list of mimetypes: https://github.com/apache/tika/blob/main/tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml#L1898 )


Cross compiling gcc

Question: Im trying to get gcc running on my Kindle 3.  I have a native terminal and ssh working, and I found that the Sourcery G++ toolchain for arm eabi linux produces working binaries, so it seems like I should be able to just configure gcc to build using arm-none-eabi-gcc and compile it, but Im obviously misunderstanding something...  When I configure with --target=arm-none-eabi --host=arm-none-eabi and run make, it compiles using gcc, not arm-none-eabi-gcc. Unless theres some pre-compilation phase that uses gcc?  I havent gotten all the way through the build yet, since Ive had other errors im working through, but I dont want to waste all this time if im configuring something wrong in the beginning.  I found this question, Compile GCC with Code Sourcery where the op seems to want the same thing, but it was never answered...  Also, Ive run into problems when compiling lua with the sourcery compiler, it seems to be using a different version of GLIBC, which I assume will be a problem.  Is there any easier way to do this?
Answer: /pathToStuffifNotPathed/arm-none-linux-gnueabi-g++ foo.c
Should you not invoke the compiler via it's tool-chain invocation? I believe so.

Cross compiling gcc


How do you practice Audiation (playing music in your head)?

Question: Time for a tricky question.
Audiation appears to be the practice of playing music in your head.  I found an answer referring to what audiation is in https://music.stackexchange.com/questions/9621/to-sing-is-good-for-ear-training-but-is-it-crucial, and I'm intrigued!
A quick search also found Audiation, Mental hearing, Composing/Reading scores without sound reference.
I imagine that creating a simple melody in your head, and then using knowledge of relative intervals and rhythm could be used to practice.
I'm also imagining playing a scale in your head while visualizing the sheet music, and in the past I've learned chord patterns on the guitar in my head to help with memorizing the shapes, but I've never really considered the practice of in your head practice before!
So let's start simple:
How can you practice Audiation?
and then get more heavy
What are the benefits of practicing Audiation?
I've heard that some composers can create entire symphonies in their head, and I myself am capable of hearing 2-3 simple instrument voices in my head while composing.  It's something that applies to any instrument and potentially allows for some pretty amazing feats! So I think this topic needs some exploration!
I look forward to seeing these answers
Comment: @Kaz - that is correct, however, audiation falls into the category of musical pedagogical practice, which applies to this site.
Comment: @Kaz - yes, because audiation technique determines your intonation and your ability to produce correct intervals.  If we were in a sight-singing test and you hadn't practiced your audiation, you would sing out of tune and I would give you a bad grade. :)
Comment: Imagining sound isn't music performance; there is no external evidence of any music.
Comment: If it's pedagogic, can I be graded on an exam that tests how well I ... audiate?
Comment: It is impossible to conclude whether or not someone practiced audiation from whether or not they failed a music performance test. Audiation cannot be directly observed. This question cannot be answered. "How do you practice audiation" is akin to "how do you imagine", "how do you feel" or "how do you go about thinking".
Comment: @kaz It seems that it is something we do anyway, but it's more like how do you practice feeling, imagination and thinking.  I'd argue that writers practice imagination,  Artists practice feeling, philosophers and scientists practice thinking.  just because most people do it passively it doesn't mean you can't do it actively, and grow from it.
Comment: @Kaz - when you type or write words, you usually have an inner-monologue speaking the words in your head, correct?  This is non-musical audiation.  You're "hearing" the words in your head and then transcribing them into letters and words to form sentences and thoughts.  With respect to music, either the person is practicing audiation, has absolute pitch (a form of audiation,) or is merely guessing at the notes.  I can tell you from years of experience that students guessing at the notes are not successful.  Improvement in audiation can be and *is* monitored as progress over time.
Comment: This should probably be two separate questions, but aside from that, it seems perfectly fine to me.
Comment: If you get really good at audiation, would you start making mistakes on the instruments like you do in real life?!
Answer: I explain audiation to my students like this:
Audiation is just like transcribing a given external melody or rhythm, only it happens internally.
When you compose music, you are constantly audiating - your "inner ear" "hears" something, you write it down, you check it, and if it matches, you move on, if it doesn't, then you modify your understand of what it is you're hearing until you're satisfied.
You practice audiation the same way you practice your other aural skills, because it is one in the same.  I almost think the real question here is How can you develop the inner-ear?
In my experience, I have found that there are two effective ways that yield dramatic results with diligent practice. (Note here that I am not discounting other methods, but am simply describing two that I have found to be effective.)
They are as follows:

Transcription

And I don't mean writing down a dumb melody in ear-training class.  But transcribing pieces of music - especially "non-classical" music to be precise.  Often this type of music is in simple or basic duple meter, uses popular chord progressions and intervals, and is transparent in its architecture.  Such material is great to work with when transcribing since it does not clutter the aural dictive process.
I recommend that students transcribe songs from their favorite bands - not rap artists as the harmonic language is too sparse, however, it is good for rhythmic training.  Transcribing one song in its entirety (voice, guitars, drums, bass, etc,) is equivalent to approximately one year of aural skills at the collegiate level.

Singing

Singing is vitally important to developing the ears, though, oddly enough, many vocalists are very poor at sight-singing (that is the subject for another time.)  As opposed to transcription, I recommend to my students not to sing popular music for many reasons (not listed here.)  Instead, I have them sight-sing using fixed and movable solfege systems through melodic etudes that require them to alter specific pitches within each etude.  The alterations are simple at first, eventually graduating to more difficult alterations, modal mixture, and finally atonality or pantonality.
I highly encourage all instrumentalists to sing, and all vocalists to learn rudiments of one melodic instrument.  Pairing audiation with kinesthetic coordination creates more neurological connections in the brain, therefore further reinforcing the learned concept.
This is why children's brains are the most active when they play music and why it is such a terrible idea for schools to cut arts funding.  But I digress.

Audiation is beneficial for many reasons, some of which are listed below:

Improves overall musicianship - by practicing audiation, other shortcomings will be improved as well.  For example, let's say you're having a hard time slurring some intervals on the trumpet.  Your problem could be one of audiation and not actually physical lip technique.
Reduces Learning Time - you don't need to spend as long learning the notes because you already know the notes and can focus on expressivity.
Reduces Composition Time - as a composer, this is a big one for me.  The better my ears get the less time I spend figuring out what I want, which means I can write or music or do other stuff that I want to do.
It helps you appreciate music more deeply - think about how much more awesome that Beethoven Sonata or that Radiohead song sounds because you can identify the augmented sixth chords.
Increases musical memory - you can retain more sounds in your head more accurately for a longer period of time, thus helping you appreciate music more deeply / remember cool things that you want to steal and put in your own music.
Opens up more opportunities - say you and some friends wanted to do some impromptu barbershop singing at a pub (like me recently) but you can't join in because you can't hear the intervals you're supposed to sing from memory.  That makes everyone sad.  Incidentally, we sounded great.
Improves ensemble playing - by being able to hear intervals ahead of time, you are better able to not only compensate for your instrument, but also compensate for the group as well, thus resulting in a better overall sound.

Some composers hear everything instantaneously (Mozart, Mendelssohn) while some labor agonizingly (Beethoven, Brahms, Joan Tower, most people) and some can even work very steadily (Britten, Stravinsky.)  One way isn't better than any other way, and it doesn't make the music any more incredible or the composer any more worthwhile.  As you can see, I mentioned a bunch of famous composers with wildly different working aesthetics.
What's important is to develop our shortcomings so that we may improve at whatever and however we do what we do - whether it is playing, singing, composing, or just listening.
Hope that helps.
Comment: I read an article that Opera singers who go deaf stop being able to sing certain notes in key, not because the voice can't do it, but because their ear can no longer hear the note.  Would that be a side-effect of lost audiation/the loss of being able to mentally hold a note?
Comment: @AlexanderTroup - there are also articles about musicians with absolute pitch losing their ear after learning to play the violin - all of the microtonal adjustments blurred their understanding of the pitches.  It is hard to say why an operatic vocalist would lose pitch as there are many factors involved - age, improper training, hearing damage, and poor nutrition could all be contributing factors.  There are many others, but I have only mentioned a few here.
Comment: @AlexanderTroup The inability to sing in key due to deafness is the simple consequence of the control loop being open. It is not possible to apply the exactly right amount of muscle tension in the vocal cords to produce an exact note, without monitoring the result to make corrections.
Comment: What instruments do you play jj? I'm guessing you play the trumpet, anything else?
Comment: You mentioned some exercises using the solfege technique, is there any perticular book or books of sheet music you would reccommend for that? I have Bona's rhythmical articulation which i've heard they sing at college sometimes.
Comment: @Anthony - I'm classically trained on tuba, though I play all instruments with rudimentary proficiency. To Alexander - I personally use materials that are unpublished.  For recommendations, every school has a particular text they prefer - to my knowledge there is no nationally revered text for solfege.  The most important thing is that you're singing consistently - daily - if only for 10 minutes.


How to preformat inside textarea

Question: I was wondering if it were possible to preformat text that is inside a textarea.  Right now I have a textarea code that I want to add syntax highlighting and also add linenumbers so I am trying to wrap the text inside a pre tag.  Is this correct or should I be doing something completely different?
<textarea id="conversation" class="codebox" style="font-family:courier;">

//  Start typing...

</textarea>

Comment: Unfortunately you can't do this with a textarea. Anything between the `` tags is taken literally as text. You'd have to write your own editor, or use a pre-built one, that gives textareas some sort of RTE capability.
Comment: I think you're looking for an in-browser code editor. Try [CodeMirror](http://codemirror.net/mode/javascript/index.html).
Answer: textareas are not able to render content like you're wanting to do, they only display text. I would suggest an in-browser code editor. A good one is CodeMirror, which is fairly easy to use:
HTML
<textarea id="code" name="code">
// Demo code (the actual new parser character stream implementation)

function StringStream(string) {
  this.pos = 0;
  this.string = string;
}
...
</textarea>

Javascript
var editor = CodeMirror.fromTextArea(document.getElementById("code"), {
    lineNumbers: true,
    matchBrackets: true
});

And CodeMirror insert an editable block with that content within the new editor.
There are other options. Wikipedia has a comparison of Javascript-based source code editors entry.
Answer: Having the Pre tag contain the textarea works in Chrome
Answer: The textarea element content is treated as preformatted in implementations, though browsers by default line wrap the text if a line is longer than specified by the cols attribute. The wrapping can be disabled using the nonstandard but widely supported attribute wrap=off.
The textarea element is not adequate for mere display of content, though. Neither is it suitable for setting up an input editor with formatting features, since all markup inside textarea is taken literally.

How to preformat inside textarea


Getting the highest correlation coefficient from the given data set

Question: I would like to compare the following given data set
a = '235 148 89 19 222';
b = '112 128 144 160 176';
c = '192 192 192 192 192';
d = '64 64 64 64 64';

with
y = [230 138 79 15 212];

Then calculate the correlation coefficient by comparing each of the given data set with y. Then, display the string with the highest correlation coefficient found.
I can find it for calculating it for two values with the command
c = corrcoef( a, y );
c = abs(c(2,1));

but how do I iterate through each data set using a for loop and display the result with highest the corrcoef?
Here is the piece of code which I have written, but I don't know how to proceed with the 'for loop'
a = '235 148 89 19 222';
b = '112 128 144 160 176';
c = '192 192 192 192 192';
d = '64 64 64 64 64';

y = '230 138 79 15 212';

s = {a;b;c;d};
s = cellfun(@strsplit, s, 'UniformOutput', false);
s = vertcat(s{:});

for i = 1:size(s,1)

end

Answer: Is there a special reason for using strings and cells instead of integer values and matrices?
What about the following solution:
a = [235 148 89 19 222;...
    112 128 144 160 176;...
    192 192 192 192 192;...
    64 64 64 64 64];

b = zeros(size(a));

y = [230 138 79 15 212];

for i=1:length(a(:,1))
    b(i,:) = a(i,:)- y;
end;

[~, minLine] = min(sum(abs(b')));

disp(minLine);

Here every line in the given dataset a(matrix) is compared with the given vector y, by calculating the difference and storing the values in a second matrix b.
After the loop the minimal sum is calculated and gives you the line in a, which is correlating most to y
Comment: Thank you! It almost works, I mean it detects that the closest value to y is line 1 and it displays it correctly but can it give the result [235 148 89 19 222] instead of displaying just 1.
Comment: And could you please do it by using corrcoef?
Answer: First of all the easy way to transform a string of number to array is using
str2num like this:
>> an = str2num(a)

an =

   235   148    89    19   222

to concatenate stings in a matrix as rows use char and then convert it to a matrix:
>> S = char(a,b,c,d)

S =

235 148 89 19 222
112 128 144 160 176
192 192 192 192 192
64 64 64 64 64

>> N = str2num(S)

N =

   235   148    89    19   222
   112   128   144   160   176
   192   192   192   192   192
    64    64    64    64    64

then the only thing you need is to go trought the matrix:
>> [rows,columns] = size(N)

rows =

     4

columns =

     5

we need to iterate over all the rows
>> N(1,:)

ans =

   235   148    89    19   222

in the matlab help:
R = corrcoef(A,B) returns coefficients between two random variables A and B.

>> R = corrcoef(N(1,:),y)

R =

    1.0000    0.9995
    0.9995    1.0000

so then applying your measure to the loop
>> for i = 1:rows
R = corrcoef(N(i,:),y);
rr(i) = abs(R(2,1));
end
>> rr

rr =

    0.9995    0.2789       NaN       NaN

finally the max of that vector is the row that you want
>> [value,position] = max(rr)

value =

    0.9995

position =

     1

>> N

N =

   235   148    89    19   222
   112   128   144   160   176
   192   192   192   192   192
    64    64    64    64    64

>> N(position,:)

ans =

   235   148    89    19   222

Comment: How can I display the result in an array; say `y = [63 63 63 63 63]` so the closest output should be `d = [64 64 64 64 64]`. So how can I display that result (entire matching vector)?
Comment: I added the final solution, it was only to find the max value, please if this answer heps you upvopte and check as valid


Is there a specific reason why R GLM does not return warnings, while anova(glm) does?

Question: It is a replica of my question at https://stackoverflow.com/questions/46895017/is-there-a-specific-reason-why-r-glm-does-not-return-warnings-while-anovaglm
I was advised to ask it here instead.
It is not about glm.fit: fitted probabilities numerically 0 or 1 occurred; it is about why anova warns while glm doesn't.
I have wanted to see contrasts inside a specified model:
is_service ~ action_count * document_entropy

The full dataset is loaded in the code. The data are somewhat pathological, as you could see below.
Overall the data are these:
> str(dat)
'data.frame':   6432 obs. of  3 variables:
 $ action_count    : num  0.0759 0.1505 0.1435 0.1535 0.2067 ...
 $ document_entropy: num  -0.667 -0.667 -0.667 -0.667 -0.667 ...
 $ is_service      : int  0 0 0 0 0 0 0 0 0 0 ...

The target column has this binomial distribution:
> table(dat$is_service)

   0    1
6291  141

Input columns are z-normalized and distributed as follows:

> cor.test(dat$action_count, dat$document_entropy)

    Pearson's product-moment correlation

data:  dat$action_count and dat$document_entropy
t = 20.477, df = 6430, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.2243432 0.2702313
sample estimates:
     cor
0.247426

It is interesting to see that when I fit this model (1st part of the code) the procedure ends without a warnings.
However, when I run contrasts with the stats::anova (2nd part of code) it does return warnings.
Question: Why is that happening, and which level is more alarming: single model or the anova analysis of it?
Is this library specific question or statistics specific?
Output result:
> summary(
+      glm(formula = is_service ~ action_count * document_entropy
+          , family = binomial(link = 'logit'),
+          data = dat
+      )
+ )

Call:
glm(formula = is_service ~ action_count * document_entropy, family = binomial(link = "logit"),
    data = dat)

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-4.0972  -0.1808  -0.1638  -0.1477   3.1019

Coefficients:
                              Estimate Std. Error z value Pr(>|z|)
(Intercept)                   -4.06877    0.10143 -40.116  < 2e-16 ***
action_count                   6.04157    0.57882  10.438  < 2e-16 ***
document_entropy               0.29721    0.06866   4.329  1.5e-05 ***
action_count:document_entropy -0.43928    0.04456  -9.859  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1356.2  on 6431  degrees of freedom
Residual deviance: 1034.0  on 6428  degrees of freedom
AIC: 1042

Number of Fisher Scoring iterations: 7
### GLM LOOKS OK
> anova(
+      glm(formula = is_service ~ 1
+          , family = binomial(link = 'logit')
+          , data = dat
+      )
+      , glm(formula = is_service ~ action_count
+            , family = binomial(link = 'logit')
+            , data = dat
+      )
+      , glm(formula = is_service ~ action_count + document_entropy
+            , family = binomial(link = 'logit')
+            , data = dat
+      )
+      , glm(formula = is_service ~ action_count + document_entropy + action_count:document_entropy
+            , family = binomial(link = 'logit')
+            , data = dat
+      )
+      , test = "Chisq"
+ )
Analysis of Deviance Table

Model 1: is_service ~ 1
Model 2: is_service ~ action_count
Model 3: is_service ~ action_count + document_entropy
Model 4: is_service ~ action_count + document_entropy + action_count:document_entropy
  Resid. Df Resid. Dev Df Deviance  Pr(>Chi)
1      6431     1356.2
2      6430     1051.1  1  305.135 < 2.2e-16 ***
3      6429     1043.2  1    7.867  0.005035 **
4      6428     1034.0  1    9.199  0.002422 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Warning messages:
1: glm.fit: fitted probabilities numerically 0 or 1 occurred
2: glm.fit: fitted probabilities numerically 0 or 1 occurred

And the reproducible code:
list.of.packages <- c('RCurl')
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages)

library(RCurl)

x <- getURL("https://rawgit.com/alexmosc/FX_Big_Experiment/master/service_train_saved.csv")
dat <- read.csv(text = x)
dat$X <- NULL

str(dat)
# first part
summary(
     glm(formula = is_service ~ action_count * document_entropy
         , family = binomial(link = 'logit'),
         data = dat
     )
)
# second part
anova(
     glm(formula = is_service ~ 1
         , family = binomial(link = 'logit')
         , data = dat
     )
     , glm(formula = is_service ~ action_count
           , family = binomial(link = 'logit')
           , data = dat
     )
     , glm(formula = is_service ~ action_count + document_entropy
           , family = binomial(link = 'logit')
           , data = dat
     )
     , glm(formula = is_service ~ action_count + document_entropy + action_count:document_entropy
           , family = binomial(link = 'logit')
           , data = dat
     )
     , test = "Chisq"
)

Update:
Let me specify which exactly function I call.
> summary(
+      stats::glm(formula = is_service ~ action_count * document_entropy
+          , family = binomial(link = 'logit'),
+          data = dat
+      )
+ )

Call:
stats::glm(formula = is_service ~ action_count * document_entropy,
    family = binomial(link = "logit"), data = dat)

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-4.0972  -0.1808  -0.1638  -0.1477   3.1019

Coefficients:
                              Estimate Std. Error z value Pr(>|z|)
(Intercept)                   -4.06877    0.10143 -40.116  < 2e-16 ***
action_count                   6.04157    0.57882  10.438  < 2e-16 ***
document_entropy               0.29721    0.06866   4.329  1.5e-05 ***
action_count:document_entropy -0.43928    0.04456  -9.859  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1356.2  on 6431  degrees of freedom
Residual deviance: 1034.0  on 6428  degrees of freedom
AIC: 1042

Number of Fisher Scoring iterations: 7

>
> m1 <- stats::glm(formula = is_service ~ 1
+     , family = binomial(link = 'logit')
+     , data = dat
+ )
> m2 <- stats::glm(formula = is_service ~ action_count
+       , family = binomial(link = 'logit')
+       , data = dat
+ )
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred
> m3 <- stats::glm(formula = is_service ~ action_count + document_entropy
+       , family = binomial(link = 'logit')
+       , data = dat
+ )
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred
> m4 <- stats::glm(formula = is_service ~ action_count * document_entropy
+       , family = binomial(link = 'logit')
+       , data = dat
+ )
>
> stats::anova(m1
+       , m2
+       , m3
+       , m4
+       , test = "Chisq")
Analysis of Deviance Table

Model 1: is_service ~ 1
Model 2: is_service ~ action_count
Model 3: is_service ~ action_count + document_entropy
Model 4: is_service ~ action_count * document_entropy
  Resid. Df Resid. Dev Df Deviance  Pr(>Chi)
1      6431     1356.2
2      6430     1051.1  1  305.135 < 2.2e-16 ***
3      6429     1043.2  1    7.867  0.005035 **
4      6428     1034.0  1    9.199  0.002422 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Update 2:
> summary(stats::glm(formula = is_service ~ action_count * document_entropy
+                    , family = binomial(link = 'logit')
+                    , data = dat
+ ))

Call:
stats::glm(formula = is_service ~ action_count * document_entropy,
    family = binomial(link = "logit"), data = dat)

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-4.0972  -0.1808  -0.1638  -0.1477   3.1019

Coefficients:
                              Estimate Std. Error z value Pr(>|z|)
(Intercept)                   -4.06877    0.10143 -40.116  < 2e-16 ***
action_count                   6.04157    0.57882  10.438  < 2e-16 ***
document_entropy               0.29721    0.06866   4.329  1.5e-05 ***
action_count:document_entropy -0.43928    0.04456  -9.859  < 2e-16 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 1356.2  on 6431  degrees of freedom
Residual deviance: 1034.0  on 6428  degrees of freedom
AIC: 1042

Number of Fisher Scoring iterations: 7

>
> stats::anova(
+      stats::glm(formula = is_service ~ 1
+                 , family = binomial(link = 'logit')
+                 , data = dat
+      )
+       , stats::glm(formula = is_service ~ action_count
+                    , family = binomial(link = 'logit')
+                    , data = dat
+       )
+       , stats::glm(formula = is_service ~ action_count + document_entropy
+                    , family = binomial(link = 'logit')
+                    , data = dat
+       )
+       , stats::glm(formula = is_service ~ action_count * document_entropy
+                    , family = binomial(link = 'logit')
+                    , data = dat
+       )
+       , test = "Chisq")
Analysis of Deviance Table

Model 1: is_service ~ 1
Model 2: is_service ~ action_count
Model 3: is_service ~ action_count + document_entropy
Model 4: is_service ~ action_count * document_entropy
  Resid. Df Resid. Dev Df Deviance  Pr(>Chi)
1      6431     1356.2
2      6430     1051.1  1  305.135 < 2.2e-16 ***
3      6429     1043.2  1    7.867  0.005035 **
4      6428     1034.0  1    9.199  0.002422 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Warning messages:
1: glm.fit: fitted probabilities numerically 0 or 1 occurred
2: glm.fit: fitted probabilities numerically 0 or 1 occurred

I finally understand this behaviour. The warnings actually happen for models with one and two independent covariates, and these warnings are also issued inside anova. However, when I build a more complex model with interaction there are no warnings in either glm or anova.
> m2 <- stats::glm(formula = is_service ~ action_count
+       , family = binomial(link = 'logit')
+       , data = dat
+ )
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred
> m3 <- stats::glm(formula = is_service ~ action_count + document_entropy
+       , family = binomial(link = 'logit')
+       , data = dat
+ )
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred

> stats::anova(
+      # stats::glm(formula = is_service ~ 1
+      #            , family = binomial(link = 'logit')
+      #            , data = dat
+      # )
+      #  , stats::glm(formula = is_service ~ action_count
+      #               , family = binomial(link = 'logit')
+      #               , data = dat
+      #  )
+      #  , stats::glm(formula = is_service ~ action_count + document_entropy
+      #               , family = binomial(link = 'logit')
+      #               , data = dat
+      #  )
+       stats::glm(formula = is_service ~ action_count * document_entropy
+                    , family = binomial(link = 'logit')
+                    , data = dat
+       )
+       , test = "Chisq")
Analysis of Deviance Table

Model: binomial, link: logit

Response: is_service

Terms added sequentially (first to last)

                              Df Deviance Resid. Df Resid. Dev  Pr(>Chi)
NULL                                           6431     1356.2
action_count                   1  305.135      6430     1051.1 < 2.2e-16 ***
document_entropy               1    7.867      6429     1043.2  0.005035 **
action_count:document_entropy  1    9.199      6428     1034.0  0.002422 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Warning messages:
1: glm.fit: fitted probabilities numerically 0 or 1 occurred
2: glm.fit: fitted probabilities numerically 0 or 1 occurred

One more question: is this expected that a more complex glm model proceeds well, but its less complex version is warning prone?
Update 3:
I am checking a relation between fitted probabilites and covariates, lookin specifically for probability equal 1 or 0.

Update 4:
I removed extreme values from predictor space:
list.of.packages <- c('RCurl', 'data.table')
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)) install.packages(new.packages)

library(RCurl)
library(data.table)

x <- getURL("https://rawgit.com/alexmosc/FX_Big_Experiment/master/service_train_saved.csv")
service_dat <- read.csv(text = x)
service_dat$X <- NULL

service_dat <- as.data.table(service_dat)

service_dat_cleaned <- service_dat[action_count < 10, ]
service_dat_cleaned <- service_dat_cleaned[document_entropy < 10, ]

> summary(service_dat_cleaned)
  action_count       document_entropy     is_service
 Min.   :-0.084589   Min.   :-0.66652   Min.   :0.00000
 1st Qu.:-0.079289   1st Qu.:-0.66652   1st Qu.:0.00000
 Median :-0.057901   Median :-0.66652   Median :0.00000
 Mean   :-0.018093   Mean   :-0.01084   Mean   :0.02101
 3rd Qu.: 0.008299   3rd Qu.: 0.61200   3rd Qu.:0.00000
 Max.   : 4.902059   Max.   : 9.97527   Max.   :1.00000

Analysis of Deviance Table

Model: binomial, link: logit

Response: is_service

Terms added sequentially (first to last)

                              Df Deviance Resid. Df Resid. Dev  Pr(>Chi)
NULL                                           6425     1310.1
action_count                   1  259.038      6424     1051.1 < 2.2e-16 ***
document_entropy               1    7.867      6423     1043.2  0.005035 **
action_count:document_entropy  1    9.968      6422     1033.2  0.001593 **
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Comment: First, check which of the *four* `glm` function calls in your `anova` function call are producing warnings. There may be no discrepancy at all.
Comment: Well that's presumably because you get the warning from model fitting. When you call `anova(m1, m2)` the models are not being re-fit so you don't get any warnings. In the original code, you had all four models fit inside the `anova()` call. In short, there is no contradiction to be seen. CC @Scortchi.
Comment: It's not a general expectation that a more complex model will produce fitted probabilities equal to 0 or 1, no. Suggest you check *which* fitted probabilities are 0 or 1 (your histograms suggest likely candidates), examine influence diagnostics, & think carefully even about the model with interaction.
Comment: It might help to have more detail about the extremes of the distributions you have plotted as histograms. As @Scortchi has suggested Cook's distance or the hat values would be well worth looking at for all your models together.
Comment: Thank you @Nick Cox. What I am asking is -- without a profound knowledge of statistics -- I wonder if my data create a good model which can be estimated well by this function, but I do not know the bordeline data quality which will not fit well to the function (to the math behind it). So I rely on warnings as a user. I see a contradiction in that two closely related functions behaved differently.
Comment: Do you think submitting a question to R-Devel is a good idea?
Comment: Let's see, maybe I will get more opinions. As a second step I can try to dig into function's code myself.
Comment: @ Scortchi, thank you. I updated my question (after **Update**). It seems confusing now. The model with interaction **m4** did not produce warnings neither in glm nor inside anova, but models **m2** and **m3** with independent terms gave me warnings. When I tried anova(m1, m2), I did not get warning either. Do I mix different things together?
Comment: @amoeba, thanks for pointing this out. I got it. I will once again fitiing models inside and outside of anova and will report results.
Comment: @amoeba, after **Update 2**, I finally understand this behaviour. The warnings actually happen for models with one and two independent covariates, and these warnings are also issued inside anova. However, when I build a more complex model with interaction there is no warnings in both glm and anova.
Comment: However, I seem to have generated a spin-off question: why a more complex glm model proceeds well, but its less complex version is warning prone.
Comment: @ mdewey  & @ Scortchi: Thank you, visually (after **Update 3**) I can tell model's output saturates very fast on action count and the interaction term. Is this alarming?
Comment: How confident are you in the linear relation you've assumed between each predictor & the logit response? Might there be something awry with an action count of around 80? What are you going to do with the model? (BTW put '@' right next to the user-name to ping people.)
Comment: @ Scortchi, I did not have relation kind assumptions; a goal is to have a bearable predictive model. My covariates are z scores, and each point is the weekly action count of corporate users, so in the course of 2 month this outlier is possible, and there is no measurement error as far as I can tell. It is just some admin who moved thousands of files, producing the heavy tailed distribution. I want to use a model in the form is_service ~ action_count + document_entropy as a classifier for admin users. Interaction showed no improvement in terms of the AUC, so I don't need it.
Answer: The coefficient for action_count is about 6. This means that the predicted value for action_count == 50 is going to be about 300 on the log scale. $e^{300}$ is about $10^{130}$ so it is not surprising that when converted from log odds to odds to probabilities it is effectively 1. The same is true to a lesser extent for document_entropy. When you fit the interaction which has a negative coefficient this is presumably sufficient to pull the predicted log odds down to a value which can be converted to a value less than one.
What you do next is to some extent a scientific rather than statistical issue.
Some suggestions:

Drop the extreme observations. This is almost certainly a bad idea. The only case in which it might be helpful is if you have external evidence as to why they differ. It might be for instance that they should never have been included, they were measured using a completely different scale, or something else. Not fitting the model is not a good reason to drop them.

Take better account of the properties of the variables. In this case the explanatory variables appear to be counts. Do we really think that the difference between a count of 1 and 2 is the same as the difference between a count of 50 and 51? If the answer is no then we may in fact be thinking of counts as on a logarithmic scale so transforming them (after possibly ading a small constant, might be a good idea.

The model may differ between the upper and lower parts of the range of the predictors. In this case you could fit a model with separate slopes in the two parts of the range.

There may be other predictors which could be included, possibly with their interaction with existing variables to cope with the model being different for different sub-groups of the population.

No doubt there are other possibilities.
Note that all these suggestions are post hoc as you have made your choice based on having seen the data so really you need to do a completely new study to validate them
Comment: @AlexeyBurnakov why perform a selective exclusion of values which seem extreme, when the techniques I mentioned in my comment above do these for you in a manner which is less arbitrary?
Comment: I'd agree the warnings are probably due to 'ones' predicted by the models, but I think a 'statistical' (which IMO is also scientific ;) ) addition might be that this extremely high coefficient is due to outliers / data points specific to the exact sample these analyses were based on. In other words, I suspect **overfitting**. As such, resampling techniques, cross-validation and/or shrinkage could be used to mitigate the effects of such datapoints and give a less-overfitting prone model.
Comment: @IWS I agree it is a fine line between science and statistics here, I will see what the OP has to say before expanding the answer with suggestions (if necessary).
Comment: @mdewey, thank you. What I will do is remove some points with most extreme predictor values from my sample so that I do not get z score above 20, and I will check whether the model has changed significantly. I don't think it will hurt much if I omit extreme behavior, since I cannot know in any way what the maximum value can be in the future given the shape of the distribution (non-finite variance of it). I think the answer is solid. Just for curiosity I will report again after doing the tuning.
Comment: @ IWS, they sound valid. I need to design good cross validation for such unbalanced data. By shrinkage do you mean regularized version of regression?
Answer: Not an answer to your original question (glm vs anova(glm), which was more like a code debugging question, since the glm function also gives the same error as the anova(glm) if you just supply the same specific model that has this error/warning),
but anyway, you may be interested in the following plot:
plot(0.1 + dat$action_count,
     2/3 + dat$document_entropy,
     log="xy"
    )
points(0.1 + dat$action_count[which(dat$is_service == 1)],
       2/3 + dat$document_entropy[which(dat$is_service == 1)],
       pch=21,bg=2)

The image shows the distribution of the 1s in your dummy variable, as function of your two regressors (which needed to be transformed slightly to allow easier view on a log-scale).

Yes you can try to fit this mess with your glm model. But it will be a distorted result since the goodness of fit is very bad.
And because of this distorted fitting you can get these surprising results that adding or removing some regressors will result in this model that is warning to be outside of range due to predicting 1s or 0s.

Adding the parameter 'action_count' is predicting 1s because, as mdewey answered, this variable has a large range and the precision becomes bad (not so much over-fitting, since the prediction of p=1, in that upper right corner is actually not so bad, and not really a problem for further evaluation such as evaluation of sums of residuals).
Then adding the cross term is removing the predictions of 1s. That is because the prediction of the upper right corner can now be somewhat decoupled, from the fit in the other three corners.

See for these effects also the difference in the two images below with the predicted p value for the binomial distribution plotted as contours. The first model is without and the second model with the cross term.

So this cross term does not really improve your predictions, except for the upper right corner where the p-value changes from 1 to 0.9999782

Useful for debugging these type of questions:
It is simple to find the source by entering the function int the command line of R, which provides you the object.
And for glm.fit we see the following piece of code:
    eps <- 10 * .Machine$double.eps
    if (family$family == "binomial") {
        if (any(mu > 1 - eps) || any(mu < eps))
            warning("glm.fit: fitted probabilities numerically 0 or 1 occurred",
              call. = FALSE)
    }

So, it may be possible to improve the code (if this would be ever necessary for someone). Namely, the precision near 0 is higher than near 1 (see eps(0) vs eps(1). However, somehow the glm code still gives me the machine eps also if I invert the numbers.
> model1 <-  glm(dat$is_service ~ 1 + dat$action_count + dat$document_entropy, family = binomial(link = 'logit')
> model2 <-  glm(1-dat$is_service ~ 1 + dat$action_count + dat$document_entropy, family = binomial(link = 'logit')

> range(predict(model1), type="response"))
[1] 0.009288089 1.000000000
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred

> range(predict(model2), type="response"))
[1] 2.220446e-16 9.907119e-01
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred

> eps(1)
[1] 2.220446e-16

> eps(0)
[1] 2.225074e-308

Comment: I suggest that you put the answer of mdewey as the accepted answer since he provided the original thought about the reason why the floating point precision warning occurs. Also his answer is much more brief in this aspect than mine, which is more extending the answer and moving into a direction which is actually a different question. On that aspect this question and their answers are a bit confusing, because the topic has shifted from the initial difference between glm and anova(glm), which did not exist, into reasons for the warning and how to deal with it.
Comment: Wow, I need to process it mentally. Looks great, thank you!
Comment: Yep, I have done that. Ijust wanted to flag both. Got it now. I will return later. Lots of things to think on. Thank you.


Test needing a mock of RabbitTemplate

Question: I have a implementation class which acts as a rabbitMQ sender class, I am trying to write unit test cases for this, but i am having doubts about mocking rabbitmq template.
This is my sender class code:
@Service
public class Car implements CarDelegate {

    @Autowired
    private RabbitTemplate rt;

    @Value("${exchange}")
    private String exchange;

    @Value("${queue}")
    private String queue;

    @Override
    public ResponseEntity<String> createCar(String model, String name) {
        Car car = new Car();
        car.setModel(Model);
        car.setName(Name);
        String jsonString;
        jsonString = new ObjectMapper().writeValueAsString(car);
        try {
            rt.convertAndSend(exchange, queue, jsonString);
        } catch (AmqpException e) {
            //to implement
        }
        return new ResponseEntity<>(HttpStatus.ACCEPTED);
    }
}

My sender class is also my implementation method.
The test class for it is as below:
@RunWith(MockitoJUnitRunner.class)
public class CarTest {

    private Car car;
    @Mock
    private RabbitTemplate rt;

    @Test
    public void create_valid() {
        Car car = new Car(rt);
        car.create("sedan", "arison");

        String jsonString = "";
        Mockito.doReturn("")
           .when(rabbitTemplate.convertAndSend(null, null, jsonString))
           .myMethod(Mockito.any(createLeadTest_valid.class));
        Mockito.when(rabbitTemplate.convertAndSend(null, null, jsonString)).thenReturn("");
    }

}

What is the correct way to mock rabbit template
Answer: For your specific case, no need to add behavior to your mock.
public class CarServiceTest {

    @Test
    public void create_valid() {
        RabbitTemplate rt = Mockito.mock(RabbitTemplate.class);
        CarService car = new CarService(rt);
        ResponseEntity<String> response = car.create("sedan", "arison");

        assertThat(response).isNotNull();
        assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
    }
}

FYI, it is not good practice to manipulate ResponseEntity outside an HTTP adapter (typically a bean annotated with @Controller).
And RabbitTemplate#convertAndSend is supposed to provide a conversion mechanism, so you do not have to use Jackson directly.
Hoping this will help you !

Test needing a mock of RabbitTemplate