Strip SpamAssassin headers before processing

Question: I have a server running cPanel/WHM with exim and SpamAssassin. I've been noticing an issue where emails coming in with forged spamassassin headers bypassing some of the filtering. I want to strip out all SpamAssassin headers before it goes through spamassassin and then filtered into the inbox/spam folders.
Searching the net, the only similar instance I could find was from 2004. However, the exim config by that user and by me are very different. I am not sure how to apply it. I can run formail against a file containing the message to remove the headers, but I don't know how to make exim do that.
Just to provide an example, a message will come in with headers like this:
X-Spam-Status: No, score=1.3
X-Spam-Score: 13
X-Spam-Bar: +
X-Ham-Report: Spam detection software, running on the system "serv02.example.com", has
        identified this incoming email as possible spam.  The original message *snip*
X-Spam-Flag: NO

My SpamAssassin will add these headers to the message:
X-Spam-Status: Yes, score=6.8
X-Spam-Score: 68
X-Spam-Bar: ++++++
X-Spam-Report: Spam detection software, running on the system "serv02.example.com", has
        identified this incoming email as possible spam.  The original message *snip*
X-Spam-Flag: YES

But because the exim vfilter rules read the first X-Spam headers, the email ends up in the user's inbox instead of in the spam folder.
Answer: Check out if SpamAssassins remove_header or clear_headers configuration file options will do what you want.
Comment: After trying these two options, they don't appear to have any affect.
Comment: Apparently, you also can remove headers in exim using the [headers_remove list](http://www.exim.org/exim-html-current/doc/html/spec_html/ch44.html#SECTheadersaddrem) - although I am not sure if headers actually will get removed before checking the vfilter rules.
Answer: Exim for cPanel uses vfilters stored in /etc/vfilters and relies on the X-Spam-Bar header. Other systems may use X-Spam-Flag-Status or the Score to determine whether to save the message in the inbox, a spam folder, or delete it. Unfortunately, vfilter will use the very first match, which is the crux of this issue.

Outdated solution found     http://wiki.exim.org/ExiscanExamples#Replacing_foreign_Spamassassin_headers_with_local_ones
Add into ACL's somewhere
   warn
    spam = exiscan:true
    set acl_m0 = ($spam_bar) $spam_score
    set acl_m1 = $spam_report
This will set ACL variables appropriately for the following processing in the system filter:
Add into system filter:
if first_delivery then
  # ...

  headers remove X-Spam-Score:X-Spam-Report:X-Spam-Checker-Version:X-Spam-Status:X-Spam-Level
  if $acl_m2 is not ""dd then
    headers add "X-Spam-Score: $acl_m0"
    headers add "X-Spam-Report: $acl_m1"
  endif
endif

In cPanel, the system filter is /etc/cpanel_exim_system_filter and the acl_mX variables are different, so lets figure those out. Add this to the end of
cpanel_exim_filters
# work out our acl variables!
if first_delivery then
    headers add "X-acl-m0: $acl_m0"
    headers add "X-acl-m1: $acl_m1"
    headers add "X-acl-m2: $acl_m2"
    headers add "X-acl-m3: $acl_m3"
    headers add "X-acl-m4: $acl_m4"
    headers add "X-acl-m5: $acl_m5"
    headers add "X-acl-m6: $acl_m6"
    headers add "X-acl-m7: $acl_m7"
end

--- did this and got:
    X-acl-m1: user1
    X-acl-m2: 1
    X-acl-m3:
    X-acl-m4:
    X-acl-m5:
    X-acl-m6:
    X-acl-m7:
So i find the section of exim.conf that adds headers and see all the $spam variables. I wonder if they carry over? Let's find out.
headers add "X-Test-Spam-Subject: ***SPAM*** $h_subject"
headers add "X-Test-Spam-Score:$spam_score"
headers add "X-Test-Spam-Score-Int: $spam_score_int"
headers add "X-Test-Spam-Bar: $spam_bar"
headers add "X-Test-Spam-Report: $spam_report"

So $spam_score_int is carried over along with $h_subject, but not spam_bar, which is vital
X-Test-Spam-Subject: ***SPAM*** test
X-Test-Spam-Status:
X-Test-Spam-Score: 33
X-Test-Spam-Bar:
X-Test-Spam-Report:

But.. I realized that headers carry through, which might work
headers add "X-Test-Spam-Bar: $h_X-Spam-Bar"

results in:
X-Test-Spam-Bar: +
+++

So the $h headers have both the headers received and headers added.
Solution?
Add custom spam filters into exim.conf and then swap them out in cpanel_exim_filter
log_message = "SpamAssassin as ${acl_m1} detected message as spam ($spam_score)"
add_header = X-989Spam-Subject: ***SPAM*** $h_subject
add_header = X-989Spam-Status: Yes, score=$spam_score
add_header = X-989Spam-Score: $spam_score_int
add_header = X-989Spam-Bar: $spam_bar
add_header = X-Spam-Report: $spam_report
add_header = X-989Spam-Flag: YES

add_header = X-989Spam-Status: No, score=$spam_score
add_header = X-989Spam-Score: $spam_score_int
add_header = X-989Spam-Bar: $spam_bar
add_header = X-Ham-Report: $spam_report
add_header = X-989Spam-Flag: NO
log_message = "SpamAssassin as ${acl_m1} detected message as NOT spam ($spam_score)"

Ok, now to make this more persistent.
First, copy your cpanel_exim_system filter:
cp /etc/cpanel_exim_system_filter /etc/custom_exim_system_filter

Log into WHM  Main >> Service Configuration >> Exim Configuration Editor, and click Advanced Editor
Scroll down and change the spam header section to look like below. Note that you're just changing X-Spam headers to X-989Spam headers (you can of course use any custom name you like as long as your custom-exim-system-filters match. Since the configuration file may change over time, make sure you are modifying only the add_header lines.
add_header = X-989Spam-Subject: ***SPAM*** $h_subject
add_header = X-989Spam-Status: Yes, score=$spam_score
add_header = X-989Spam-Score: $spam_score_int
add_header = X-989Spam-Bar: $spam_bar
add_header = X-Spam-Report: $spam_report
add_header = X-989Spam-Flag: YES

-- and --
add_header = X-989Spam-Status: No, score=$spam_score
add_header = X-989Spam-Score: $spam_score_int
add_header = X-989Spam-Bar: $spam_bar
add_header = X-Ham-Report: $spam_report
add_header = X-989Spam-Flag: NO

Save your configuration and wait for WHM to refresh.
Go back to Main >> Service Configuration >> Exim Configuration Editor >> Filters
Change the system filter file to point to /etc/custom_exim_system_filter
Answer: The accepted answer may have been the best you could do in 2012, or with the commonly available exim versions in 2012, but nowadays exim does have a remove_header directive that can be used in ACLs.  It was added with version 4.82:

New ACL modifier "remove_header" can remove headers before message gets
  handled by routers/transports.


High CPU usage of IIS process (w3wp.exe) because of many slow clr.dll!CopyPDBs

Question: I am using Windows Server 2008 R2 Enterprise with IIS 7.5, hosting ASP.NET MVC 5 applications.
I've noticed lately that one of the w3wp.exe processes is using extremely high CPU bandwidth:

After investigated little bit more, I've also noticed that the reason of this is the function clr.dll!CopyPDBs:

Does anybody know what can be the problem?
If the reason is not related to .NET Framework and clr.dll!CopyPDBs how can I investigate further to find the actual reason for this
Answer: This is usually because you have a custom application on top of IIS. If it is an in house built application, speak with the Developers about it.
If it is not developed in house, please have a look at .net CLR with counters https://msdn.microsoft.com/en-us/library/x2tyfybc(v=vs.110).aspx and more tips on CLR performance https://msdn.microsoft.com/en-us/library/ee851764(v=vs.110).aspx and https://msdn.microsoft.com/en-us/magazine/cc163528.aspx .
Comment: Unfortunately, the answer makes no sense at all. The issue happens sometimes after you build in Visual Studio an application which is configured to use IIS (Not IIS Express which is default for VS web projects). The issue is not in the application code but somewhere in IIS. Killing the process in task manager fixes the problem (IIS will start a new process for the application and it will work fine).
Answer: That simply means it's a .NET application that's taking all the CPU, but Process Explorer can't find the .pdb files for it. See https://stackoverflow.com/a/31705019/8479 for more detail.
If you can't get Process Explorer to say what's going on then try other avenues like log files, event log and Process Monitor.
Answer: This is a guess at best, but perhaps your development team is building and deploying the application in debug mode, in stead of release mode.  This will cause the occurrence of .pdb files.  The implication of this is that your application will take up additional resources to collect system state and debugging information during the execution of your system, causing more processor utilization.
So, it would be simple enough to ensure that they are building and deploying in release mode.


Rails tests with activeadmin

Question: I am using activeadmin on my Rails project and I would like to know how to make test on it. I looked over on internet but there is no documentation about it.
I have a settings resource, and I disabled the possibility to create a new one or to delete it.
I would like to know how to test these routes doesn't exists or another way to be sure it can't be accessible.
In my actual though I was trying to assert this page to be a 404 but the test seems to throw this error (which is what I tried to test !!!): ActionController::UrlGenerationError: No route matches {:action=>"new", :controller=>"admin/settings"}
I don't use Rspec but Minitest from Rails.
Thanks for your help !
My activeadmin file:
ActiveAdmin.register Setting do
    actions :all, except: [:new, :destroy]
    …
end

My tests:
require 'test_helper'

class Admin::SettingsControllerTest < ActionController::TestCase
   include Devise::TestHelpers
   setup :initialize_settings

   test 'should throw 404 if trying to access new action' do
     get :new
     assert_response 404
   end

   def initialize_settings
     sign_in users(:one)
   end
end

My project:

Rails 4.2
Ruby 2.2.0
Minitest (NOT RSPEC)
Comment: if you remove the :new from this line actions :all, except: [:new, :destroy] do you still get that same error? Have you tried instead of assert_response ... doing something like this value(response).must_respond_with 404
Answer: What you are needing are integration tests, specifically ones that will test the routes, or in your case, the absence of routes. The non-existent actions will raise ActionController::RoutingError.
# test/routes/admin/settings_test.rb
require 'test_helper'

class Admin::SettingsRouteTest < ActionDispatch::IntegrationTest
  test 'create' do
    assert_raises(ActionController::RoutingError) do
      post '/admin/settings'
    end
  end

  test 'destroy' do
    assert_raises(ActionController::RoutingError) do
      delete '/admin/settings/1'
    end
  end

  test 'edit' do
    assert_routing '/admin/settings/1/edit', controller: 'admin/settings', action: 'edit', id: '1'
  end

  test 'index' do
    assert_routing '/admin/settings', controller: 'admin/settings', action: 'index'
  end

  test 'new' do
    assert_routing '/admin/settings/new', controller: 'admin/settings', action: 'show', id: 'new'
  end

  test 'update' do
    assert_routing({ method: 'patch', path: '/admin/settings/1' }, { controller: 'admin/settings', action: 'update', id: '1' })
  end
end

Notice that the new action falls back to a show action with an id of 'new'.
OBSERVATION: If you are removing the :new action, you probably intended to remove the :create action too so that no records can be created through that route.
# app/admin/setting.rb
ActiveAdmin.register Setting do
  actions :all, except: [:new, :create, :destroy]
end

Comment: Actually, this post solved the problem for Rails 4: http://stackoverflow.com/a/21662764/3700317

Rails tests with activeadmin


When adding entity the EntityFramework wants to create instead of reference related entities

Question: I have DDD entities as below:
public class Grant
{
    public string GrantId { get; set; }
    // Other properties omitted
}

public class AuthorizationCode
{
    public string Code { get; set; }
    public Grant? Grant { get; set; }
    // Other properties omitted
}

And that's code to add the entity:
public async Task StoreAsync(AuthorizationCode authorizationCode, CancellationToken cancellationToken = default)
{
    await _dbContext.AddAsync(authorizationCode, cancellationToken);
    await _dbContext.SaveChangesAsync(cancellationToken);
}

When adding the new AuthorizationCode then EntityFramework instead of referencing existing Grant, it wants to create new and I get exception:

SqlException: Violation of PRIMARY KEY constraint 'PK_Grants'. Cannot insert duplicate key in object 'dbo.Grants'. The duplicate key value is (1l7xVpoEEhgHmZVZf28PvpO3lGg_xV59H69LCB4lPkQ). The statement has been terminated.

Grant configuration:
public class GrantConfiguration : IEntityTypeConfiguration<Grant>
{
    public void Configure(EntityTypeBuilder<Grant> builder)
    {
        builder.HasKey(p => p.GrantId);
    }
}

AuthorizationCode configuration:
public class AuthorizationCodeConfiguration : IEntityTypeConfiguration<AuthorizationCode>
{
    public void Configure(EntityTypeBuilder<AuthorizationCode> builder)
    {
        builder.HasKey(p => p.Code);
    }
}

The Grant is created before AuthorizationCode, at different HTTP endpoint. The relations in database are valid. So what's wrong here?
Comment: Use `Update` instead of `AddAsync`. This is a very common issue. BTW the async version of `Add` should only be used in special cases, please see the docs.
Comment: Can you provide the code that is doing the update or the insert?
Comment: I edited my question and added it. I don't update this entity
Comment: I do `_dbContext.Attach`, then `Update` and `SaveChangesAsync`.  But I get `DbUpdateConcurrencyException: The database operation was expected to affect 1 row(s), but actually affected 0 row(s); data may have been modified or deleted since entities were loaded`. Btw this appropach seems strange to me - could you explain why `Update` instead of `AddAsync`?
Comment: First of all, `AddAsync` isn't needed unless you use a strange key generator like HiLo. `Add` will attach the detached (ie created in memory, not loaded from DB) object and *all other reachable objects* in the `Added` state. If the new object already existed, you'd need to attach in the `Modified` state. That's what` Updated` does. `Update` is smarter than that though, and will attach in the `Added` state if the object has no PK values (eg a long ID of 0). Neither `Add` nor `Update` talk to the database though. They do what the docs say - they attach objects in the specified state
Comment: If you *don't* know whether the data exists in the database you *have* to check. `SaveChanges` won't generate a check to select INSERT or UPDATE. Most of the time though you *should* know whether the objects exist or not, eg because you loaded them yourself, or they were the payload of a PUT or DELETE. You really need to explain what scenario results in objects with unknown state
Answer: Try this:
modelBuilder.ApplyConfiguration(new GrantConfiguration());
modelBuilder.ApplyConfiguration(new AuthorizationCodeConfiguration());

in OnModelCreating method of your context.


How do you sync a Roland Boss RC 300 Loop Station to external rhythm sources?

Question: I'm looking for ways to fatten the sound of the combo I play with. But the band plays tight tempos in strict 32 bar choruses.
Is there any way to sync the Roland Boss RC 300 Loop Station to external rhythm sources such as a live drummer or an external drum machine?
The RC 300 has MIDI master but not MIDI slave capabilities.  And its rhythm tracks are awful.
Answer: I just took a pass at reverse engineering the RC300 midi output so I could slave it. Short answer is - no can do. Mainly this is because there is no track identifier in the sysex data. It basically sends the same sysex message to start and stop recording for each of tracks 1, 2 and 3. I stopped my investigation upon discovering that.
Answer: Okay, I now know how to do this, although it will take me some time to implement.
This case is for having an acoustic drummer set the tempo on the RC 300.

Get an piezoelectric acoustic drum trigger and mount it on the snare rim.

Hook the drum trigger up to a midi percussion module with a midi out that has configurable midi messages.

Edit the outgoing midi message from the percussion module to be CC#64 with a value of 127.

On the RC-300, go into Memory Edit | Assign Controllers. Find an unassigned controller, like 8. Set it to ON, Target to MEM Tempo (tap), and Source to CC#64.

First caveat: Don't use this to trigger the Rhythm Track or it will restart on every beat.
Second caveat: The drummer will need a way to turn off the trigger when he's not hitting every (other) beat on the snare.


How to make a React component 'communicate' with another React component?

Question: I have a parent jsx component that has 2 different jsx components within it.
The one component is a button and the other is a div that opens and closes itself when you click on it (it has a click handler and a state of open or closed).
I now want to add the ability for the button to open and close the div as well.
Is the only way to accomplish this is to pass a handler function down to the button from the parent, moving the div's open and closed state to the parent component, and pass the state down to the div as props? The reason I ask is that this particular div component is used in a number of different components and removing the open and closed state would affect a lot of different parent components.
Comment: You can pass it an isOpen prop and on componentWillReceiveProps  set the state inside the component. Thats a way to leave the functionality to the component but also affect it through it's parent.
Comment: Your usecase is a perfect case for lifting the state up, and even though it might require some changes in your app you should go ahead with it. Check this question on when to lift the state up https://stackoverflow.com/questions/46594900/reactjs-lifting-state-up-vs-keeping-a-local-state/47349693#47349693
Comment: "Is the only way to accomplish this is to pass a handler function down to the button from the parent, moving the div's open and closed state to the parent component, and pass the state down to the div as props?" No you can use the strategy to allow external manipulation as suggested by @MatanBobi
Answer: Here's a code example of allowing external state manipulation where you can mix the usage of the button or the div to toggle the state. You extend your Collapsible component to use passed props to update the state.
class Collapsible extends React.Component {
  constructor(props){
    super(props);
    this.state = { isOpen: this.props.isOpen !== false };
    this.toggleOpen = this.toggleOpen.bind(this);
  }
  componentWillReceiveProps({ isOpen }) {
    this.setState({ isOpen });
  }
  toggleOpen(){
   this.setState((prevState) => ({ isOpen: !prevState.isOpen }))
  }
  render() {
    let display = this.state.isOpen ? null : "none";
    return (

         header

{this.props.children}

    );
  }
}

class Parent extends React.Component {
  constructor(props){
    super(props);
    this.state = { isOpen: true };
    this.toggleOpen = this.toggleOpen.bind(this);
  }

  toggleOpen(){
   this.setState((prevState) => ({ isOpen: !prevState.isOpen }))
  }

  render() {
    return (

        <Collapsible isOpen={this.state.isOpen}>content</Collapsible>
        <button onClick={this.toggleOpen}>
          toggle
        </button>

    );
  }
}

Answer: Here is another code example, hope it helps:

I use the local state of the Container and pass this down to the child components. In a bigger app I'd advice you to use something like Redux to manage your state.
The central idea is that the parent component passes a function which can "change it's state" to the button child. It also passed the current isOpen state to the panel. Clicking the button will change the state of the parent, thus triggering a reflow, thus updating the collapsable.
For future reference:
import React from "react";
import { render } from "react-dom";

const Collapsable = ({ isOpen }) =>
  isOpen ? (

      {" "}
      Hey, I'm open{" "}

  ) : (

Oh no...closed :(

  );

const Button = ({ openPanel }) => (
  <button onClick={() => openPanel()}>Open Panel</button>
);

class Container extends React.PureComponent {
  state = { open: false };

  openPanel = () => {
    this.setState({
      ...this.state,
      open: this.state.open ? false : true
    });
  };

  render() {
    return (

        <Button openPanel={this.openPanel} />
        <Collapsable isOpen={this.state.open} />

    );
  }
}

const App = () => (

    <Container />

);

render(<App />, document.getElementById("root"));

Comment: @mr-baudin You shouldn't be using arrow functions inside a render as it creates a new function on each render. instead, bind the function in the constructor or use curry to bind. Also, there's no need to spread the last state when your setting the new state is setState changes only the properties it needs to change and when you are editing the previous state you should be using `setState((prevState) => ({isOpen: !prevState.isOpen}))`
Comment: @MatanBobi Thanks for the feedback, you are right. Just for some bookkeeping: [don't use arrow functions in render](https://medium.freecodecamp.org/why-arrow-functions-and-bind-in-reacts-render-are-problematic-f1c08b060e36).
You are also right about the state. I didn't know that thank you!

How to make a React component ‘communicate’ with another React component?


PHP mkdir() won't work on Mac OS X even if folder permissions are 777

Question: I'm facing a strange issue. I cannot get mkdir() to work on Mac OS X in a directory that is under my home, using nginx + php70-fpm webserver. nginx and php-fpm are installed from MacPorts. Coming from the Linux world, I have tried the following:
Edit /opt/local/etc/nginx/nginx.conf and add the user myusername directive and reload nginx, but no success. The server won't even start.
Through the get info menu option, give permission to the folder I want to write to to everyone, but no success.
From the console, chmod 777 the folder, but no success.
Is there something I am missing?
Code:
    if(!file_exists( dirname( __FILE__ ) . $ds. $storeFolder . $ds . $_SESSION["lid"] . $ds)) {
        // No directory, so create it
        if(mkdir(dirname( __FILE__ ) . $ds. $storeFolder . $ds . $_SESSION["lid"] . $ds, 0777, true)) {
            die("Folder creation failed");
        };
    }

Answer: Edit:
As an observation, after doing this you may face issues with phpMyAdmin saying incorrect permissions are applied. Therefore, some may find it useful to grand everyone "Read & Write" access only to the htdocs folder, or better yet, the specific folder for the project, not the whole XAMPP folder.
Here is the solution:

Open the applications folder

Locate the XAMPP folder

right-click, and get info (as described above)

In the pop-up window, locate the 'sharing & permission' section

Click the 'locked' padlock symbol

Enter the administrator password

Change 'Everyone' permissions to read & write

In the get info window still, select the 'cog' icon' drop down
option at the very bottom and select 'Apply to enclosed items'.
This will adjust the permission across all sub-folders as well.

Re-lock the padlock symbol

Close the 'Get Info' window.

The task is complete. This will now allow you to make directories.
Answer: I had the same problem. I'm not sure if your problem is the same as mine but I changed the permissions on the root folder to 777 and then everything worked fine.
Comment: What is that "root folder"?


What is the explanation of this expected value result?

Question: Given $X \sim Uni(0,1)$, and $Z = \mathbb{I} \{a < x < 1\}$, $Y = \mathbb{I} \{0 < x < b\}$
for the parameters $0 < a < b < 1$. And we know that $Y$ and $Z$ are dependent.
So basically what I can understand from this, is that $Z = 1$ if $a < x < 1$, otherwise $Z = 0$.
And the same for $Y$, as $Y = 1$ if $0 < x < b$, otherwise $Y = 0$
I need to find $E[Y|Z]$. The answer for this is the following, although I don't understand almost anything out of it:
For $Z = 1$:
$$P(y = 0|z = 1) = \frac{P(y = 0, z = 1)}{P(z = 1)} = \frac{1-b}{1-a}$$
$$P(y = 1|z = 1) = \frac{P(y = 1, z = 1)}{P(z = 1)} = \frac{b-a}{1-a}$$
$$P(Z = 1) = 1-a$$
for $Z = 0$:
$$P(y = 0|z = 0) = \frac{P(y = 1, z = 0)}{P(z = 0)} = \frac{a}{a} = 1$$
$$P(Z = 0) = a$$
Finally:
$$E[Y|Z] = E[Y|Z = z] * P(Z = z) = P(Y = 1| Z = 1) * P(Z = 1) + P(Y = 1|Z = 0)*P(Z = 0) = \frac{b-a}{1-a} a + 1 (1-a)$$
Can someone please help me understand how did we get to the final formula? why is $P(y = 0, z = 1) = 1 - b$  and why is $P(z = 1) = 1 - a$ ?
N
Answer: Answering the final questions out of order:
Let $f(x)=1$ be the density of $X \sim Uniform(0,1)$
$$P(Z = 1) = \int_a^1 f(x)dx = \int_a^1 1dx = 1 -a$$
$$P(y = 0, z = 1) = P(y = 0 | z =1) P(z=1) = \frac{1-b}{1-a}(1-a) = 1 - b$$
By the meaning of conditional expectation (I think there was a mistake in the notation)
$$E(Y) = \sum_z E[Y|Z = z] P(Z = z)$$
By the meaning of the expected value
$$E[Y|Z=z] = \sum_y yP(Y=y|Z=z) = (1) P(Y = 1|Z = z) + (0) P(Y=0|Z=z)$$
Now
$$E(Y) = P(Y = 1|Z = 0) P(Z=0) + P(Y=1|Z=1)P(Z=1)$$
$$ = (1)(a) + \frac{b-a}{1-a} (1-a) = b$$
Also
$$E(Y|Z=0) = P(Y = 1|Z = 0) = 1$$
$$E(Y|Z=1) = P(Y = 1|Z = 1) = \frac{b-a}{1 - a}$$
You can simulate these in R
<code>X <- runif(100000)

a <- 0.5
b <- 0.7

Y <- ifelse(X < b, 1, 0)
Z <- ifelse(X > a, 1, 0)

mean(Y)
b
mean(Y[Z == 0])
1
mean(Y[Z == 1])
(b-a) / (1-a)
</code>
Answer: If $U \sim \mathrm{Unif}(c, d)$ and $c<u_1<u_2<d$, then
$$ P(u_1<U<u_2)=\frac{u_2-u_1}{d-c} $$
[Sketch it; it's just the area of a rectangle.]
To answer your questions:

why is P(y=0,z=1)=1−b ?

Because $Y=0, Z=1$ is equivalent to $X>b$ and $a<X<1$, i.e. $b<X<1$.

and why is P(z=1)=1−a ?

Because $Z=1$ is equivalent to $a<X<1$.
Comment: Hey, back to this question again. Any chance you explain more why is $Y=0,Z=1$ is equivalent to $X>b$ and $a<X<1$? I still dont get what is the concept of $Z$ and $Y$ and how do their values affect the output. What is$ Z=1$ and $Z=0$? what does it mean? I don't get their meaning here, the same for $Y$.


Equivalent condition for $ (X_n, \mathcal{F}_n) $ to be a martingale

Question: I've encountered an interesting problem and am not quite able to solve it. It is to prove the following statement ($ X_n $ denotes a sequence adapted to a filtration $\mathcal{F}_n) $:
$$ (X_n,\mathcal{F}_n) \text{ is a martingale} \iff \text{ for any bounded stopping moment } \tau ~ \mathbb{E} X_\tau = \mathbb{E}X_0$$
I'm not quite sure where to begin with - it's possible to prove that for a martingale it holds that
$$ \mathbb{E}\left( X_n - X_{n-1}\right) = 0 $$
but $ \tau $ is a random variable, so I can't think of a way to skip that;
I would appreciate some help
Answer: Hints:

"$\Rightarrow$": Use Doob's optional stopping theorem and the fact that $\tau$ is bounded.
"$\Leftarrow$": It holds that $$\mathbb{E}(X_{\tau}) = \mathbb{E}(X_{\sigma})$$ for any two bounded stopping times $\tau$, $\sigma$. Show that $$\tau :=n \cdot 1_{B} \qquad \quad \sigma := k \cdot 1_{B}$$ are bounded stopping times for $k \leq n$ and $B \in \mathcal{F}_k$. Conclude from $$\mathbb{E}(X_n 1_B) = \mathbb{E}(X_k 1_B), \qquad \text{for all} \, \, k \leq n$$ that $(X_n,\mathcal{F}_n)_{n \geq 0}$ is a martingale.
Comment: +1. (And more direct than the suggestion in my (now deleted) comment.)


Difficulty in calculating joint probability

Question: I have a table.
<code>           CASH   CREDIT CARD   DEBIT CARD   TOTAL
UNDER 20   0.09          0.03         0.04    0.16
20-100     0.05          0.21         0.18    0.44
OVER 100   0.03          0.23         0.14     0.4

TOTAL      0.17          0.47         0.36       1
</code>
The question is: "Find the probability that a debit card purchase was $20 and over".
In working this question, I became confused. I came up with several possible answers. I would like to know which one is right. How do I work out this question?
Is the probability:

$0.18+0.14=0.32$
$ \frac{0.18}{0.44} + \frac{0.14}{0.40} = 0.41+0.35= 0.76$
$ \frac{0.18+0.14}{0.36} = 0.89$
$ \frac{0.14+0.18}{0.84}= 0.38$
Comment: Does anyone know how to add an attachment? I did not see that option. I saw for links but not attachments. The formatting has changed for the table. It's jumbled up now :\
Comment: I tried to fix the table, you should avoid tabs.
Comment: Is this a homework question for some university course? Also, please pay attention to spelling and avoid using too many abbreviations.
Comment: thanks for fixing it. It's a past paper exam question. I'm due to write Economic Statistics exams next week. Do you know the answer?
Answer: For homework questions, the site's policy is to provide hints rather than complete solution. Here's one:
The question suggests a conditional probability (probability of A given B), which can be computed as
$$P(A|B) = \frac{P(A \bigcap B)}{P(B)}$$
What's B? What's A? Then, where is $P(B)$ in the table (you can read it directly) and what is $P(A \bigcap B)$ (you have to compute it).
Comment: Ok. Doesn't seem like a conditional probability to me though.
Comment: Laurans , Conditional probabilities tend to have 'given' as the big clue to the question being one on conditional probability. Even then,I would not know which event is the condition. Eg, the question doesn't say ""Find the probability that a debit card (was used) GIVEN that the purchase was $20 and over" or '"Find the probability that a purchase was $20 and over GIVEN that a debit card purchase was made".
Comment: @Laurans ,I tried again but I am still confused. I used the conditional probability. A=debit card purchase, B= $20 and over. P(B|A)=P(A intersect B) / P(A). P(B)=0.44. P(A intersect B) is what I am confused about again. I came here because I am unsure of what to do. I am still unsure. For P(A intersect B), do I take .18+.14=0.32
OR .18/.44 + .14/40 = 0.41+0.35=.76
OR .18/.36+.14/.36=
OR .14+.18=.32 therefore, .32/ .36 OR .32/.84
Comment: The top part of the equation is the probability of A intersect B. The probability of debit card purchases and $20 and over. HOW do I calculate it? That is all I am asking. This is what has me confused. I am not asking for an answer, just the methodology.
Comment: @aboabo It would be too easy if you just needed to look for the word "given" without trying to understand the problem. In your case, wouldn't the sentence imply that you know the purchase was made with a debit card and you want to know the probability that it was under 20$? What are A and B in "A given B" then?


Need a solution to search by sales person based on unique territory

Question: I've been researching for hours for a solution, but I can't find a module or example of someone else needing this feature.
I have a list of 20-ish sales people, each person is assigned a range of zip codes. Some of them are entire states, but there are a few sales reps where the state is split between 2-3 people. I need a way for the user to enter in their own location and show which sales rep would be their point of contact.
Anyone have an idea of how I would go about doing that? Location module?
Thanks!
Answer: Found something that might help you Locator tutorial and use Location Map if you want to display the results in a map.
Answer: https://www.drupal.org/node/1944648 This link might help you out..
Comment: I've used this tutorial before but for this case, I need each dealer/sales rep to be "in charge of" hundreds of zip codes. So the proximity wouldn't really matter. I'm looking for a way to import all of these zip codes (and an easy way to maintain them) without having to manually input each one.
Comment: let me be clear first you need 20 sales person having multiple zip codes assign each of them. and on search if a user inputs their zip code. the corresponding sales person will be listed in the list????
Comment: That's my initial plan-- but I see two downsides:
- Zip codes can change and I don't want to have to actively maintain the database (especially because Canada will be included)
- It'd be nice if they could search by their street address/state as well like with the Location module/geofield combo (but it isn't necessary)
Comment: I suggest you to use some kind of refrence module like user refrence or entityrefrence.. You have to relate each node with with any one user.once you acheive that you can refine your search easily based on that refrences
Comment: one solution i found while working for your question is that you have to user node refrence or refrence or user refrence module for drupal 7.This provide the user refrence field in mange fields of your content type which shows a list of users on the site that you create using views and refrening in this newly created field which list down all the users to refer while creating new node . and while creating your view you can refer those users or sales men to list down with that node on search..


LXC Networking: Access other container from another container throug host's domain

Question: This is my first question here so sorry for any mistakes I make. I've not found any solution for my question yet here in serverfault.
I setup two LXC Containers: one ("A") with Dovecot and Postfix, the other one ("B") with nginx and Roundcube. Both are connected using veth to the host's main NIC which has 1 public IPv4 adress.
I can ping the host's domain from both containers, but if I do curl [host's domain or ip adress] it will fail with connection refused. (I think it connects to the host, where there isn't anything running on port 80)
From any external computer I'm forwarded to B, because I setup the following iptables rule: -A PREROUTING -i ens3 -p tcp --dport 80 -j DNAT --to 10.0.3.xxx:80
I want to access the mailserver container from the roundcube's one through the host domain because of the SSL certificate.
Answer: I found the anwser myself:
You need to add the rule also for the local-only interface
-A PREROUTING -i lxcbr0 -p tcp --dport 80 -j DNAT --to 10.0.3.xxx:80


set focus in an ember application

Question: I want to be able to set the focus to a textarea as a result of a mouse click not in that task area.
As a minimal example, let's say that I'm starting with a text and if you click on it, it gets replaced with a textfield. I can achieve this by a handlebars script:

with a controller of
App.ApplicationController = Ember.Controller.extend({
    isActive: false,
    foo: function(){
       this.set("isActive", true);
    }
});

This works to create the textfield on the click, but does not give focus to that text area (it takes a second click to be able to actually enter text).
Is there a good way to achieve this end? I could do something hacky by setting an ID in the template and selecting it with jquery, but that seems inelegant.
Answer: Furthermore, to reduce the didInsertElement and this.$().focus(); which can seem as though you're mixing jQuery into your Ember modules -- and something I hate doing, you can use the Ember.JS way of specifying additional attributes for the Ember.TextField.
We can specify that we're interested in the HTML5 autofocus attribute:
Ember.TextSupport.reopen({
  attributeBindings: ["autofocus"]
});

We can then place the standard Ember.TextField onto our page without having to create another view to extend Ember.TextField:
{{view Ember.TextField autofocus="autofocus"}}

See JSFiddle: http://jsfiddle.net/MdzhN/
Comment: Good suggestion. A note to others, this property does not work in Internet Explorer versions < 10. [Autofocus Attribute](http://www.w3schools.com/tags/att_input_autofocus.asp)
Answer: Consider extending Ember.TextField as follows:
App.FocusedTextField = Em.TextField.extend({
  didInsertElement: function() {
    this.$().focus();
  }
});

Then change you handlebars template to use it instead:

set focus in an ember application


How can I get a Windows client not in the domain to get dns from the DC?

Question: I have Server 2012 as Domain controller and DNS server.
Now if I have a machine joined to the domain, they get all servers names defined in dns zone.
I have a Windows 7 client which is not joined to the domain, but I want it to get the IP of the DNS servers.
I tried manually putting the IP 192.168.0.100 in the DNS of the network adapter, but it does not get the IP of the DNS servers.
Answer: When not a part of the domain, you need to use the FQDN of the host you are trying to resolve. Example:
A server named app01 on domain.lab with ip 192.168.0.69 being pinged from a machine on the same network, but not in the domain.
ping app01
 - host cannot be resolved
ping app01.domain.lab
- Reply from 192.168.0.69: time=blah blah blah
Comment: can't i use the dns server to be server 2008 for all computers on network. does not matter if they are part of domain or not. i mean for e,g i mean for `app01` the window 7 computer will query dns server which will be server 2008 , which then converts ti ip. currently i use my router gateway as my dns server
Comment: You "can" set it up to use any dns server you want, but the implementation will vary greatly based on your specifc situation. Your description of your current environment will need to be a lot more robust to get a more specific answer.
Answer: If you want a non-domain joined computer to resolve internal names, you should be putting the domain controller (which has DNS on it) as the DNS server and setting the DNS prefix to whatever your domain name is. Even computers not on the domain will still be able to work with the DNS server.
What won't work is other computers resolving the name of the non-joined machine since it won't be in DNS, but that wasn't your requirement.
Comment: I tried that. i put the dns server as my DC. but my computer still can't resolve the internal ip. is there any way to check if my dns server is receiving any query from my computer
Comment: Yes, `nslookup someinternalhost 10.x.x.x` where 10.x.x.x is the IP address of your DNS server. You may need to use the FQDN if you don't have the prefix set.
Comment: thanks nathan , its working now. can i add two DNS on winod0w 7 so that first my computer goes to DNS server of router , and if that does not find any ip for host , then it asks for my DC to gte internal ip. I find that DNS resolution has slowed down the internet speed if i use internal


Relacionamento de 1 atributo com tabelas diferentes

Question: Galera, boa tarde.
Estou Modelando meu Banco de Dados (MySQL) e surgiu uma dúvida.
No banco, entre outra, tenho as seguintes tabelas:
Funcionarios - Setores - Filiais
Um Funcionário trabalha OU em um Setor da sede OU em uma Filial. Eu queria criar um relacionamento entre um atributo (local_de_trabalho) da tabela Funcionario com os códigos (id) da tabela Setor OU da tabela Unidade.
Isso é possível? Como faria?
Comment: Você colocou `OU`, mas só para desencargo de dúvida, um funcionário pode ter SEDE e FILIAL ao mesmo tempo?
Answer: Segue um simples exemplo que acabei de criar tendo em vista sua questão. Segue imagem abaixo:

Observando o exemplo creio que vc possa criar uma de acordo com suas necessidades.
Espero ter ajudado.
Answer: Uma vez que o campo local_de_trabalho está relacionado a uma ou outra table, não é possível tratá-lo como uma foreign key, o que você pode fazer, nesse caso, é ter um outro campo (ex: tipo_local_de_trabalho) indicando a qual tabela o campo local_de_trabalho está sendo relacionado


DOM Jquery não funciona com formulário

Question: Tenho esse script que abre uma página html com o DOM do Jquery
$('.abas').on('click', function(){
   var paginas = ["planejamento_estrategico2.php","mentoria_fast_food.php","terceirizacao.php","marketing_full-time.php","mastermind.php"];
   aba_index = $(this).attr('tabindex');
   $('.abas').removeClass('active');
   $(this).addClass('active');
   $("#texto").load ("http://dominio.com.br/"+paginas[parseInt(aba_index) - 1]);
});

E esse onde faz uma requisição para um arquivo PHP e mostra o resultado dentro de uma determinada div.
$(document).on('click', '#reg-form_3', function(e){

            e.preventDefault();

            $.ajax({
                url: 'email_planejamento.php',
                type: 'POST',
                data: $(this).serialize()
            })
            .done(function(data){
                $('#form-content_2').fadeOut('slow', function(){
                    $('#form-content_2').fadeIn('slow').html(data);
                });
            })
            .fail(function(){
                alert('Ajax Submit Failed ...');
            });
});

Meu problema está na hora em se que clica no input do formulário,  já executa o script acima sem clicar no botão submit do form.
Preciso que quando o usuário coloque o email e ao clicar no botão submit do formulario, aí sim, faça a requisição do arquivo php e mostra o resultado na div. Como poderia resolver esse problema com o DOM do jquery?
Meu html
<form action="email_planejamento.php" method="post" class="wpcf7-form" id="reg-form_3" novalidate>

<input type="hidden" name="_wpcf7" value="6">
<input type="hidden" name="_wpcf7_version" value="4.9.2">
<input type="hidden" name="_wpcf7_locale" value="en_US">
<input type="hidden" name="_wpcf7_unit_tag" value="wpcf7-f6-p1065-o1">
<input type="hidden" name="_wpcf7_container_post" value="1065">

<input type="text" name="email" value="" size="40" class="wpcf7-form-control wpcf7-text wpcf7-validates-as-required" aria-required="true" aria-invalid="false" placeholder="Digite seu e-mail">

<input type="submit" value="Ver um exemplo de sucesso" class="wpcf7-form-control wpcf7-submit">

</form>

Comment: O HTML é parte essencial para resolução do seu problema, edite sua pergunta e inclua seu HTML para que possamos analisar  o que está acontecendo.
Comment: desculpe, esqueci do html mesmo, já está lá
Answer: O problema é que você está capturando todo clique no elemento com id reg-form_3 o que ocorre é que esse elemento é o formulário, ou seja, toda vez que clicar no formulário seu script será executado.
Para solucionar mude o seletor do método para o <input type="submit" />
$(document).on('click', '#reg-form_3 input[type="submit"]', function(e){
   ...
});

Ou se preferir pode adicionar um id no <input type="submit" /> e capturar o click nesse elemento:
<input type="submit" value="Ver um exemplo de sucesso"
   id="btnSubmit" class="wpcf7-form-control wpcf7-submit" />

$(document).on('click', '#btnSubmit', function(e){
  ...
});

Comment: Não precisa nem colocar um id no botão. Basta alterar o seletor para `'#reg-form_3 input[type="submit"]'`
Comment: Boa! Sua solução é mais simples, vou adicioná-la a resposta.

DOM Jquery não funciona com formulário


read a file and redirect it

Question: Im trying to find a file with x type and store that in a variable. It is working fine when i try to execute manually. If i try the same in a bash script it is displaying test_count='1 because of this "if" condition is failing.
Please find the code below :-
`
usage_backup_dir=/test/usage_backup
DT_FILENAME=$(date +%Y%m%d);
export DT_FILENAME
test_count=$(find $usage_backup_dir -type f -name "Summary_$DT_FILENAME*.bdm_all_cases.lg.log" -exec cat {} + )
echo $test_count`

Answer: This command looks weird:
find $usage_backup_dir -type f -name "Summary_$DT_FILENAME*.bdm_all_cases.lg.log" -exec cat {} +

I would end it as follows:
find $usage_backup_dir -type f -name "Summary_$DT_FILENAME*.bdm_all_cases.lg.log" -exec cat {} \;

(I've replaced + by \;)
I know there's some discussion about this ending (backslash or not, semicolon or not) but I'd advise you to have a look there.
Comment: [Using semicolon (;) vs plus (+) with exec in find](https://stackoverflow.com/q/6085156/3282436)
Comment: Same outout it is displaying bkup_count='1
Comment: Did you try adding a backslash before the plus-sign?

read a file and redirect it


Find $f$ and $g$ homotopic s.t. induce different homomorphisms

Question: Let $X$ and $Y$ be two topological spaces. Are there continuous functions $f,g:X\to Y$ satisfying the following conditions?

$f(a)=g(a)=b$ for some $a\in X$,
$f$ and $g$ are homotopic, and
the induced homomorphisms $f_*,g_*:\pi_1(X,a)\to\pi_1(Y,b)$ are different.

As far as I know, if $f$ and $g$ satisfy conditions 1 and 2, then there is a loop $\delta$ at $b$ in $Y$ such that $f_*=F\circ g_*$, where $F:\pi_1(Y,b)\to\pi_1(Y,b)$ is the isomorphism $F([\beta])=[\delta\beta\delta^{-1}]$.
Therefore if $f_*$ and $g_*$ are different then we know that:

$\pi_1(Y,b)$ is not commutative, and
The homotopy from $f$ to $g$ is not relative to $\{a\}$.

I've been trying with bouquet of circles and some reflections but without getting the desired example.
Answer: You can do this with a wedge of two circles.  Let $\alpha(x), \beta(x): S^1 \to S^1 \vee S^1$ be the two standard generators of $\pi_1(S^1 \vee S^1)$, given by the inclusion of the two summands.  Now, we describe $H: I^2 \to S^1 \vee S^1$ by the following:
$H(s,t) = \left\{ \begin{array}{ll}\beta(t - 4s) & \textrm{if } 0 \le s \le t/4 \\ \alpha\left(\frac{4s-t}{4-2t}\right) & \textrm{if }  t/4 \le s \le 1-t/4 \\ \beta(t + 4s - 4) & \textrm{if } 1 - t/4 \le s \le 1\end{array}
\right.$
This provides a homotopy between elements representing $\alpha$ and $\beta^{-1}\alpha\beta$ in $\pi_1( S^1 \vee S^1)$.  The intuition is to derive the formula from a square with the desired behavior on the edges.  You want the top edge to be $\alpha$ and the bottom edge to be a suitably parametrized version of $\beta^{-1}\alpha\beta$, and the two side edges will be $\beta$.  Form a trapezoid on the interior by connecting the upper corners with the points on the bottom mapped to the basepoint.  At each time $t$, the loop starts at $\beta(t)$, moves to the basepoint, loops around via $\alpha$, and then goes back to $\beta(t)$.
Actually, I think the constructed formula will work for any two noncommuting elements in the fundamental group of any space.
This is part of a standard result on equivalence classes of loops via free homotopies (e.g. see Hatcher Chapter 1 Exercise 6).
Comment: So if $\alpha$ and $\beta$ are homotopic then they do not induce the same  homomorphism. But I don't see why $\alpha$ and $\beta$ are homotopic.
Comment: Here the homotopy is between $\alpha$ and $\beta^{-1}\alpha\beta$, not between $\alpha$ and $\beta$.  Two elements of the fundamental group are homotopic via free homotopies if and only if they are conjugate.


Differentiate between back/ forward in jquery address plugin

Question: As can be seen in the documentation (http://www.asual.com/jquery/address/docs/), the plugin has an event to detect when the back/forward button of the browser was pressed.
$.address.externalChange(fn)
My question is if there is a way to know when the back is pressed and when the forward is pressed. For example, this will work in both cases:
$.address.externalChange(function() { console.log('back/forward pressed'); });

Thanks.
Answer: To my knowledge, no history plugin can detect which browser button exactly has been pressed.
The best you can do is keep a history of visited pages, then check if the new page is further back in the history than the current one. When pages are revisited without the back/forward button, you'll probably get wrong results with below code.
Untested example code:
var visitedPages = [$.address.path()];
var currentPage = $.address.path();

$.address.internalChange(function() {
   visitedPages.push($.address.path());
});

$.address.externalChange(function() {
   var newPage = $.address.path();
   var currentPageIndex = -1;
   var newPageIndex = -1;

   for (var i = 0; i < visitedPages.length; i++) {
      if (visitedPages[i] == currentPage) currentPageIndex = i;
      if (visitedPages[i] == newPage) newPageIndex = i;
   }

   if (newPageIndex == -1 || currentPageIndex == -1) {
      console.log("unkown button pressed");
   } else {
      if (newPageIndex > currentPageIndex) {
          console.log('forward pressed');
      } else
      if (newPageIndex < currentPageIndex) {
          console.log('back pressed');
      } else {
          console.log('page reloaded?');
      }
   }
});

Comment: I can't really get it to work. I'm trying to create a jsfiddle but I am neither capable. Thanks
Comment: I gave you the bounty,but still cant get it to work. Could you post a fiddle,please?
Comment: Maybe you could start a fiddle and we'll see whats the problem there?
Comment: I couldn't, but i will try again. Here is a more complex question about the same issue more elaborated where i just opened a bounty: http://stackoverflow.com/questions/16808014/jquery-address-certify-that-user-is-pressing-forward-button-and-not-back-button
Answer: I finally figured out. This code works for me in a normal page navigation, where I navigate between: Page1, Page2, Page3, etc. The code recognises back and forward in those cases.
var visitedPages = [$.address.path()];
var first_load = true;

$.address.internalChange(function() {

   visitedPages.push($.address.path());

});

$.address.externalChange(function() {

    var newPage = $.address.path();

    if (visitedPages[visitedPages.length-2] == newPage) {

        console.log('back');
        visitedPages.splice(visitedPages.length-1 , 1);

    } else if (first_load === false) {

        console.log('forward');
        visitedPages.push($.address.path());

    }

    first_load = false;

});


Matplotlib, 3rd set of values as colors of colorbar

Question: I have 3 sets of values. It's like I have the x-y coordinate plane which constitutes first two sets of values. And I have divided the plane region into small squares of unit size or some size in particular. Then I have another 2-D array that contains values corresponding to each square or small region.
Now, the problem:
I could plot the 2-D array as color points using a colorbar but the x axis and y axis how the column and row indices respectively! Rather than that I wanted to have x and y coordinates shown. I tried searching a lot and didn't get the solution.
It is similar to this. It's just that I want my set of x-y coordinate values instead of the row and column values shown.
Please help. And suggest improvements if needed through comments instead of downvoting. That is very discouraging.
EDIT:
After following the answer provided, I got this
.
Instead I should have got the figure like this. Of course the axes as the previous figure and unlike below.
Comment: @MycrofD: it may be a problem of the order you do things. First `imshow` then `axis('normal')` and only after that `colorbar`. Look at the edited example code below. If you cannot make it work, please show the code.
Comment: Isn't that what [`pcolor(X, Y, C)`](http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.pcolor) does?
Comment: can you please give an example. I am really not comfortable with the docs written like in the link you provided.. please
Comment: I can't give an example in the comment. But if you provide examples of `X`, `Y` and `C` in the question, as well as describe specifically the desired result for those values, then I or someone else may be able to provide a more confident answer.
Comment: Ok. I am on it. But can u help me as how to give you the array of 50*50 values here? any link I should use or follow?
Comment: I'd just suggest to make it 5x5.
Comment: Yes. I was doing that, when the answer came up. And it almost solved my problem. But the figure is squeezed along vertical axis. How to fix that?
Comment: @MycroftD: Please not that my comment above contained a hideous error, I edited it. The correct command is `plt.axis('normal')`.
Answer: If you want to have something similar to the imshow example you linked to but with different coordinate axes, you may want to use the extent keyword of imshow:
import numpy as np
import matplotlib.pyplot as plt

# some random data (10x10)
image = np.random.uniform(size=(10, 10))

plt.figure()

# draw the pixel image
#   interpolation='nearest': draw blocky pixels, do not smooth
#   cmap=pl.cm.gray: use gray scale colormap (autoscale, no vmin or vmax defined
#   origin='lower': draw first row of the array to the bottom if the image
#   extent=[-3,3,-10,10]: draw the image so that it covers area (-3,-10)..(3,10)
plt.imshow(image, cmap=plt.cm.gray, interpolation='nearest', origin='lower', extent=[-3,3,-10,10])

# this is needed to make the pixels non-square if needed
plt.axis('normal')

plt.colorbar()

plt.show()

This way you can create "pixels" whose size is exactly what you want:

Of course, you may plot more information onto the same plot if you want just by using plot or something else.
The image here is an array of scalars and the colouring is defined by the cmap, but it may also be an array of RGB or RGBA values if you want to do fancier coloring. For example:
How to create colormap of confidence estimates for k-Nearest Neighbor Classification
If you want to have transparent areas in your map, put nan values into image.
Comment: Almost got what I needed..
But now, the y axis is shrunken. How to elongate it?
I mean x varies from 4 to 22. And y varies from -75 to -70.5. So the prob is it looks like a rectangle with the figure having squeezed along vertical axis, i.e. y axis.
Comment: @MycrofD: The `extent` keyword gives the size of the image in the plot area coordinates. Remember to give the `plt.axis('normal')` and then you can scale the axis ranges to be what you want by `plt.axis([-100,100,-50,50])` or whatever range you want to have.
Comment: thank you very much. Thnx to you and @Lev Levitsky for such a quick response. :)
Comment: I was using cbar.ax.set_ylabel('something'). It is not working. I can't label the colorbar. Help plz..
Comment: @MycrofD: Are you sure you want to set the Y label for the colorbar? That works well with `ax.set_ylabel`. But if you want to set the Y axis tick labels manually, you need `get_yticks().to_list()`, edit the list and `set_yticklabels`. The process has several kinks, see: http://stackoverflow.com/questions/11244514/matplotlib-modify-tick-label-text
Comment: cbar = plt.colorbar()..
cbar.set_label('something')..

I did that and it works. :)


Como criar uma barra de progresso dinâmica no R?

Question: Para acompanhar o status do processamento em minhas rotinas utilizo as barras de progresso do pacote pbapply, mas não encontrei uma forma dinâmica de acompanhar mais de um processo. Para ilustrar o problema considerei uma lista contendo três elementos de dimensões diferentes e que se queira junta-los levando em conta o elemento em comum entre eles (ID):
 lista<-list(A=data.frame(ID=1:100, Med=rnorm(100,50,100)),
        B=data.frame(ID=1:50, Med=rnorm(100,50,50)),
        C=data.frame(ID=51:100, Med=rnorm(100,50,25)))

 result<-data.frame(ID=1:100,matrix(NA,nrow = 100,ncol = length(lista)))

 for(t in 1:length(lista)){
     local<-dim(lista[[t]])[1]
     for(t2 in 1:local){
         posilocal<-which(result$ID==lista[[t]]$ID[t2])
         result[posilocal,(t+1)]<-lista[[t]]$Med[t2]
     }
  }

Como posso criar uma barra de progresso que mostre a situação do processo, considerando a evolução de t e t2?
Answer: Não é o ideal, mas você pode tentar fazer assim, usando o pacote progress.
library(progress)
total <- sum(sapply(lista, function(x) dim(x)[1]))
pb <- progress_bar$new(total = total)
for(t in 1:length(lista)){
  local<-dim(lista[[t]])[1]
  for(t2 in 1:local){
    posilocal<-which(result$ID==lista[[t]]$ID[t2])
    result[posilocal,(t+1)]<-lista[[t]]$Med[t2]
    Sys.sleep(3/100)
    pb$tick()
  }
}

Primeiro calcular o tamanho total do loop e depois fazer uma barra que o percorra dessa forma.
Parece ser mais complicado colocar uma barrinha que apareça para cada processo. Essa é uma limitação do RStudio pelo que eu entendi. Ler essa discussão pode ajudar a entender. Parece que o suporte a múltiplas barras está nos planos do pacote progress.


On a property of continuous functions

Question: Let $f$ and $g$ be continous functions and $f(x)\neq 0$ forall $x$. Prove that $f^2(x)=g^2(x)$ forall $x$ then $f(x)=g(x)$ forall $x$ or $f(x)=-g(x)$ forall $x$.
It 'is so apparent, so I don't know what must I do, and with similar problems, do we have the general solution? Thank you!
Comment: Think about what might happen in the case where f is not continuous. It may give you some clues as to why continuous required in the question, and may help your construct your proof.
Comment: Without the assumption of continuity, it could be the case that $f(x)=g(x)$ and $f(y)=-g(y)$ for some other point $y$, which is what you are asked to show can't happen.
Comment: yeah, I got your hints, thankyou :D
Answer: The continuous function $\frac g f$ takes only the values $-1$ and $+1$. What can you conclude from this?

On a property of continuous functions


Showing that a proposition is a substitute of the Euclidean fifth postulate.

Question: I'm needing some guidance in order to prove that the following proposition:
Any point in the interior of an (not $\pi$) angle $\angle AÔB$ is in a segment whose extremes are in the rays $\vec{OA}$ and $\vec{OB}$.
is a substitute to Euclid's fifth postulate, which I can use the Playfair substitute, since that equivalence I already proved:
Given a line $r$ and a point $P \notin r$, there is exactly one line $s$ such that $P \in s$ and $r \cap s = \phi $.
Here is my attempt:
Suppose that there is a point $P$ in the interior of $ \angle AÔB$. Since $P$ is not in the support line of $OA$, there is exactly one line through $P$ not intersecting sup. line $OA$. consider the perpendicular line to said line. My aim is to show that this perpendicular will intersect the ray $OB$, but I'm having trouble doing this.
Can someone say if this is a good approach and how to fix it? I would also accept different solutions to this problem.
Thanks.
Comment: Well 5th post => prop is easy.  Take point in interior.  Draw 2 lines through it parrallel to each ray.  Draw a third line through it.  The third line is parallel to neither ray. So the point of intersection are the segment.  (there are infinite number of them.  prop => 5th post might be harder.
Comment: Don't worry about the perp line.  Do three lines.  At most one is parallel to OA, one is parallel to OB so at least one is parellel to neither.  The one parellel to neither will form the desired segment.  Not sure how to go the other direction though.
Comment: Wait.  That *can't* be a substitute for the 5th postulate!  The proposition is true in spherical geometry in which the 5th postulate is false.
Comment: To prove proposition $\implies$ 5th postulate you can prove instead NOT(5th postulate) $\implies$ NOT(proposition).
Comment: @fleablood Spherical geometry also has other postulates different from euclidean.
Comment: @Aretino unless those other postulates *break* euclidean axioms that doesn't matter.  And the whole point of non-euclidean geometry only the fifth postulate is removed (or broken).  If stuff must be *added* well that doesn't matter.  At worst it makes the proposition unprovable but it can't make it false as it *is* true without it (albeit it with maybe extra axioms added.)
Comment: @fleablood As far as I remember spherical geometry has axioms which conflict with those of euclidean geometry. For instance, two different lines can have more than one point in common.
Comment: Yes, you are right.  The first: two points determine a line, and the third circles exist for all radii.  are violated.  I don't think the third matters but the first *might*.  I'll have to look up *elliptical* geometry.  I'm still not sure how we can formulate the negation of either the 5th or the prop as 5th refers to a single object whereas the prop allows for infinite.
Comment: Okay.  I take back my objection that spherical geometry the prop is true as the spherical geometry breaks post 1.  But I still think the other direction will be hard.  We can image OA and OB and point in the interior which will have a line that intersects neither OA nor OB but I don't see how we can find a point  in *no* lines through it intersect both.
Comment: @fleablood I tried in my answer to exhibit a proof based on my idea.
Comment: I expanded my answer: hope it is clearer now.
Answer: Let's recall first of all that an immediate consequence of Euclid's fifth postulate is the following: if two lines are intersected by a common transversal and form with it non supplementary consecutive interior angles, then these lines meet on that side of the transversal where the sum of consecutive interior angles is less than $\pi$ (as a matter of fact, this is exactly how the postulate was presented by Euclid himself).
But even without Euclid's fifth postulate we can state a weaker result:
LEMMA - If two lines are intersected by a common transversal and form with it non supplementary consecutive interior angles, then these lines cannot meet on that side of the transversal where the sum of consecutive interior angles is greater than $\pi$.
This can be proven by the exterior angle theorem: if $\alpha+\beta>\pi$ (see picture below) then $\gamma=\pi-\beta<\alpha$; if $AA'$ and $BB'$ would meet at $P$ on the right, then triangle $PAB$ would have an external angle $\gamma$ greater then an internal angle, which is impossible.

Let's now prove that Euclid's fifth postulate implies your proposition. Consider a convex angle $\angle AOB\ne\pi$ and a point $P$ in its interior. Draw from $P$ the parallel to $OB$, which meets ray $OA$ at $A$ (see picture below), and take a point $Q$ on $OA$ such that $A$ is between $O$ and $Q$. We have $\alpha>\phi$ and $\alpha=\pi-\theta$, whence $\phi+\theta<\pi$: it follows that line $PQ$ meets ray $OB$.

Let's now show the converse, in its contrapositive form: supposing that uniqueness of a parallel line is not guaranteed, I'll prove that one can construct an angle, and a point inside it, for which your proposition does not hold.
Let $a$ be a line in the plane and $P$ a point not belonging to $a$ and $O$ a point on $a$. Through $P$ we can construct a line $a'$ forming equal alternating interior angles with $OP$, so we know that $a'\parallel a$. Suppose now that uniqueness of parallel line is violated, so that there exists another line $b'$ through $P$ parallel to $a$. We can construct a line $b$ through $O$ forming equal alternating interior angles with $OP$ and thus parallel to $b'$. Of the four convex angles formed by $a$ and $b$ choose the one containing $P$: I'll call this $\angle ab$ and from now on I will use $a$ and $b$ to denote those rays of the lines which are the sides of $\angle ab$. I want to prove that there exists no segment with endpoints on $a$, $b$ and containing $P$.
To this end, take any point $A$ on $a$, and let $\angle AOP=\alpha$ (see picture below). Denote then by $\beta$ the angle formed by $b$ with $OP$ and notice that $\alpha+\beta<\pi$ (because the angle is convex). Moreover, we have $\angle APO<\beta$, because if $\angle APO=\beta$ then $A$ would belong to $b'$ by construction (impossible, as $a$ and $b'$ are parallel) and if $\angle APO>\beta$ then $b'$ would intersect $AO$ (impossible again).
It follows that line $AP$ and ray $b$ form with $OP$ consecutive interior angles greater than $\pi$: by the lemma stated above, that proves the assertion.


Why TFDBatchMove raises exception ELocalTimeInvalid for date field with value "03/11/2019"?

Question: I'm using Delphi 10.3 Rio Update 1 on a Windows 7 SP1 machine.
My program's purpose is to convert a TFDMemtable into a JSON format.   For a reason that I can't understand, when the date field of this TFDMemTable has the value '03/11/2019', using the DisplayFormat "day/month/year", it raises an exception:

Project ProjMemtabJSON.exe raised exception class ELocalTimeInvalid with message 'The given "03/11/2019" local time is invalid (situated within the missing period prior to DST).'.

Any other dates different than "Nov, 3rd 2019" work fine.
I have no clue what is going on here!
program ProjMemtabJSON;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils,
  System.Classes,
  System.JSON,
  FireDAC.Comp.DataSet,
  FireDAC.Comp.Client,
  FireDAC.Comp.BatchMove,
  FireDAC.Comp.BatchMove.DataSet,
  FireDAC.Comp.BatchMove.JSON,
  Data.DB;

Var
  Fmemtable   : TFDmemtable ;
  FJSONArray  : TJSONArray;
  FDBatchMoveJSONWriter1     : TFDBatchMoveJSONWriter;
  FDBatchMoveDataSetReader1  : TFDBatchMoveDataSetReader;
  FDBatchMove1               : TFDBatchMove;

procedure  CreateMemtable;
begin
  Fmemtable := TFDmemtable.Create(nil);
  FMemTable.FieldDefs.Add('ID', ftInteger, 0, False);
  FMemTable.FieldDefs.Add('Name', ftString, 20, False);
  FMemTable.FieldDefs.Add('Status', ftString, 20, False);
  FMemTable.FieldDefs.Add('Duedate', ftdatetime,0, False);
  FMemTable.CreateDataSet;
end;

procedure FillMemtable;
begin
  FMemtable.Append;
  FMemtable.Fields[0].Value := 10;                           // ID
  FMemtable.Fields[1].Value := 'John';                       // Name
  FMemtable.Fields[2].Value := 'Active';                    // Status
  { ==> HERE IS THE PROBLEM :  change this date to 03/11/2019  i.e.  03/Nov/2019 and an error will raise }
  FMemtable.Fields[3].Value := StrToDate('02/11/2019');     // Due date dd/mm/yyyy
end;

procedure PrintMemtable;
begin
  writeln('ID       : ' ,Fmemtable.Fields[0].AsString);
  writeln('Name     : ' ,Fmemtable.Fields[1].AsString);
  writeln('Status   : ' ,Fmemtable.Fields[2].AsString);
  writeln('Due Date : ' ,Fmemtable.Fields[3].AsString);
end;

function TableToJson : TJSONArray;
begin
  Result := TJSONArray.Create;
  try
    FDBatchMoveDataSetReader1 := TFDBatchMoveDataSetReader.Create(nil);
    FDBatchMoveJSONWriter1    := TFDBatchMoveJSONWriter.Create(nil);
    FDBatchMove1              := TFDBatchMove.Create(nil);

    FDBatchMove1.Reader       := FDBatchMoveDataSetReader1 ;
    FDBatchMove1.Writer       := FDBatchMoveJSONWriter1;

    try
      if not FMemtable.Active then
        FMemtable.Active := True;
      FDBatchMoveDataSetReader1.DataSet := FMemtable;
      FDBatchMoveJSONWriter1.JsonArray  := Result;
      FDBatchMove1.Execute;
    except
      on E: Exception do
        raise Exception.Create('Error Message: ' + E.Message);
    end;
  finally
    FDBatchMoveDataSetReader1.Free;
    FDBatchMoveJSONWriter1.Free;
    FDBatchMove1.Free;
  end;
end;

begin
  try
    { TODO -oUser -cConsole Main : Insert code here }

    Creatememtable;
    FillMemtable;
    PrintMemtable;

    FJSONArray   := TableToJSON;

    readln;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Comment: I'm unable to reproduce this, no matter how I set the short date format on my Windows 10 machine (`dd/mm/yyyy` or `mm/dd/yyyy`) and no matter how I set the month and day (`03/11/2019` or `11/03/2019`).
Comment: What happens if you use EncodeDate instead of StrToDate? Also, why use the BatchMove process. The TFDMemDataset.SaveToFile can also save JSON files.
Comment: @Ken, thks.  I forgot to mention that I'm using Windows 7 SP 1.
Comment: @JacalarRick, I'm using BatchMove because I need to get a TJSONArray as Result   to be used as Body for a TRESTRequest POST method call !  I'll try EncodeDate and then let you know the result. Thks.
Comment: @JacalarRick, EncodeDate doesn't work as well.
Comment: *EncodeDate doesn't work as well* Are you sure? It appears that `StrToDate` isn't working at all - how could `EncodeDate` work less well than not at all? :-)
Comment: @KenWhite, I interpreted "as well" as "either".  In other words, fails the same way on the same date.  What is interesting is the full text of the error, which refers to Daylight Savings Time. Sunday, Nov 3, 2019 is the date, in the US, when DST ends (at 2am).  I wonder if that has something to do with the issue JRG is having.
Comment: The [`ELocalTimeInvalid` doc](http://docwiki.embarcadero.com/Libraries/en/System.DateUtils.ELocalTimeInvalid) says: "*The ELocalTimeInvalid exception is raised whenever a given local time is situated in an invalid period of time, usually when the hour shifts due to the DST (Daylight Savings Time) rules.*" Clearly something is trying to take the `TDateTime` stored in the MemTable and convert it to a UTC date/time that doesn't exist. `StrToDate()` and `EncodeDate()` don't do that, it has to be one of the `TFDBatchMove...` components. What does the stack trace look like when the error is raised?
Comment: Just curious, what happens if the field is defined as ftDate instead of ftDateTime?  Do you need/use the time portion?
Comment: @JacalarRick, you're right, I meant "either" above. Sorry for my poor English. One important information :  I'm using Windows 7 and my Region settings is set to Brazil GMT -3:00 . Last year the start date of Brazilian DST has changed.  As Ken said  he had no issue and n Windows 10,. I tried in Windows 2012 R2 and it worked fine.  I'm confused now !!  Thanks all.
Comment: Aha! That seems to be the issue. In the April 2018 DST update from Microsoft, the DST time change for Brazil is set at midnight on the 1st Sunday of November. Therefore, the datetime with a time portion=0 doesn't exist (since midnight becomes 1:00am). Since you have declared the field as ftDateTime, set the time to 1200 (noon) for all your dates. Or, try declaring the field as just a ftDate and see if that works.
Comment: @JacalarRick, I changed the duedate field to ftDate , but the problem remains !
Comment: Was worth a shot. The exception is being raised due to the DST transition occurring at midnight (whole number datetime) versus some other hour. A small number of countries use midnight, rather than the more typical 1/2/3am; and Brazil is part of that group. Personally, I wouldn't have coded the exception to be thrown for an exact midnight datetime, but the quickest way around this is to adjust the setting of your DueDate field to be `StrToDate()+0.5`
Answer: There is a System.DateUtils routine that can check for an invalid time:
TTimeZone.IsInvalidTime(const ADateTime : TDateTime) : Boolean;
If you add DateUtils to your Uses clause, then update your FillMemTable to something like:
procedure FillMemtable;
var  MyDate : tDateTime;
begin
  FMemtable.Append;
  FMemtable.Fields[0].Value := 10;                           // ID
  FMemtable.Fields[1].Value := 'John';                       // Name
  FMemtable.Fields[2].Value := 'Active';                    // Status
  { ==> HERE IS THE PROBLEM :  change this date to 03/11/2019  i.e.  03/Nov/2019 and     an error will raise }
  MyDate := StrToDate('03/11/2019');
  if TTimeZone.local.IsInvalidTime(MyDate) then MyDate := MyDate + 0.5; //noon won't be invalid
  FMemtable.Fields[3].Value := MyDate;     // Due date dd/mm/yyyy
end;

Or, as mentioned in the comments, if you don't want the overhead of the IF statement, just force all dates to be noon.
I had never realized that there were time zones which switched to/from DST at midnight. Unfortunate, what with how Dates (without times) are defined.
Comment: thank you so much !  Very useful code.  I changed my code following your suggestions and it works now.  I hope to fix this definitively by aplying OS patch or upgrade.  Thanks a lot !
Comment: @jrg, if you could please accept my answer...  Of course, this is just a workaround for an underlying issue.  Even though there are certain dates in certain time zones where midnight falls in the Invalid Time period, I believe that an exact "0" for the time portion of a datetime value should never throw this exception: maybe the InvalidTime should start at time 00:00:00.01 to allow for the date without time.


Taxes is not shown in the cart. Using Drupal Commerce module

Question: I'm using the Drupal Commerce module and I need to add some taxes to be shown in the cart.
Then, I added 2 taxes in admin/commerce/config/taxes following this video directions: video
First one is a VAT tax type which is shown correctly
Second one is a Sales Taxes tax type which is not shown at all in the cart
What am I missing? As far as I can see is not a permission related problem.
Answer: Drupal commerce will by default not show taxes on the cart, the cart is more simple minded with focus on products. The review and checkout pages, will however have taxes displayed by default.
The shopping cart is based on views, so it's pretty easy to change how the order total is displayed. Instead of using Commerce Line Item: Line item summary area handler you should instead use the Commerce Order: Total
Comment: Thanks, I didn't know that I had to make changes to the cart views to make the Sales Taxes visible.
Comment: @chefnelone Right, I had forgotten about that. The easiest solution is to use CSS to position them differently. I talked with Ryan at Drupalcon London and looked into this. The short explanation, is that the way Views work, it's not possible to change the markup, since the buttons comes from a views form and the summary from an area handler. (It's possible but I believe it requires changes in Views and a a lot of rewrite of some Commerce Views integration)
Comment: It works but the button (update and checkout) are now above total and taxes amounts. Look a the capture I've done: http://img27.imageshack.us/img27/3563/screenshot03zq.jpg . Where do I change this?
Comment: I understand, so I will let the css do its magic. thanks.


Calculus- Simplifying Derivatives using formal definition

Question: I'm currently working on trying to solve for a derivative using the formal definition $\lim_{h\to0}\frac{f(x_0 +h)-f(x_0)}{h}$ I'm solving for $\frac{\text{d}y}{\text{d}x}$for $y=\frac{4}{x}$. Using the rules of derivatives I know the derivative is $y=\frac{-4}{x^2}$, but I'm confused on how to get this from the definition. After plugging into the formula and simplifying to get rid of the complex fraction I got $\lim_{h\to0} \frac{\frac{-4h}{x^2+hx}}{h}.$ How can I simplify that expression to $\frac{-4}{x^2}$?
Thanks for any advice you may have.
Comment: As you mentioned, you have
$$\lim_{h \rightarrow 0} \frac{\frac{-4h}{x^2+hx}}{h} = \lim_{h \rightarrow 0} \frac{-4h}{h(x^2+2h)}=$$
$$\frac{-4}{(x^2+2(h=0))}=\frac{-4}{x^2}$$
Answer: $$\frac{f(x+h)-f(x)}{h}=\frac{4}{h}(\frac{1}{x+h}-\frac{1}{x})$$
$$=\frac{4}{h}\frac{-h}{x(x+h)}$$
$$=\frac{-4}{x(x+h)}$$
and the limit when $h\to 0\;$ is
$$\frac{-4}{x^2}=f'(x).$$


Create a wrapper div within field groups on node displays

Question: For ease of formatting I would like to add an inner wrapper div inside of field groups displaying on content nodes and node forms. I'm using the Field Group module to display groups in div format.
Currently it's this:

…

Would like to output this:

 …

Answer: Try hook_field_group_build_pre_render_alter() since node content solution seems not longer working.
/**
 * Implements hook_field_group_build_pre_render_alter().
 * Adds additional wrappers
 */
function [your_theme_or_module_name]_field_group_build_pre_render_alter(&$element) {
  if (!empty($element['group_tabs'])) {
    $element['group_tabs']['#prefix'] = '
';
    $element['group_tabs']['#prefix'] .= '
';
    $element['group_tabs']['#suffix'] .= '

';
  }
}

Check Field group for developers for more info.
Answer: You can use the theme_preprocess_node hook in your custom theme and add your inner div using the #prefix and #suffix of the fieldgroup.
It should be in $variables['content']['name_of_the_fieldgorup']['#prefix'] .
Answer: I found that the best way to handle this is to use Views Display Modes from Display Suite.


Which distance function do you have to use for miRNA expression data?

Question: For a set of miRNA gene expression data from different cancer patients I produced some heatmaps with R. The goal is to examine which miRNAs are up/downregulated depending on different parameters (state, occurence of brain metastases, etc). I could not figure out whether to use euclidian distance for the calculation or a correlation analysis (spearman).

The patients didn't receive any treatment so I wouldn't consider the shape to be that important but rather the actual distance between the datapoints. On the other hand I read that a lot of people are using correlation analysis for gene expression data. Is it just a  matter of personal choice which function to use here or is there a right and wrong way? I am quite new to statistical analyses so I would appreciate any help!
Answer: You can use correlation or distances to see which patients are similar in terms of gene expression. I don't quite understand why you would use distances or correlation for your problem.
I think what you're looking for is a differential expression analysis since you want to find up/down regulated genes in a certain group against another one.
You can find packages on Bioconductor such as DESeq2 or limma to do that. But there's also some specific packages adapted for microRna like AgiMicroRna but are maybe a bit more complicated to use.
You can have a look at this tutorial for generating a heatmap of differential expression analysis.
https://www2.warwick.ac.uk/fac/sci/moac/people/students/peter_cock/r/heatmap/
Comment: Thank you very much for your answer. I asked about correlation and distances not for the analysis itself but for the distance function in the heatmap() argument. The data is from an RNA-microarray and we just examine a subset of miRNAs (55) so I thought it's a viable option to have a look at the similarity of the miRNA-gene expression.


Eigenvalues and eigenvectors of a matrix.

Question: I needed to know what are the eigenvalues and eigenvectors of this matrix:
$$\left[\begin{array}{cc}+\cos\theta&-\sin\theta\\ +\sin\theta&+\cos\theta\end{array}\right]$$
First time having this matter, so I'm sorry if this sounds too easy
Comment: Have you tried using the usual method for finding eigenvalues?
Comment: This is a $2\times 2$ rotation matrix.. Eigenvalues/eigenvectors are easy to search for.
Comment: take the determinant of $(A-I\lambda)$, solve the characteristic polynomial for lambda, and go from there
Answer: The eigenvalues $\lambda$ of the matrix can be easily found by solving:
$$|M-\lambda I| = 0,$$
where $I$ is the $2\times 2$ unit matrix. This equations give us the spectrum of $M$:
$$\lambda_{1,2} = \cos{\theta} \pm  i \sin{\theta}$$
On the other hand, the eigenvectors are to be determined as the nullspace of the following matrices:
$$ \vec{v}_{1,2} = \text{Ker}(M-\lambda_{1,2}I) = \left\{ (x,y)\in \mathbb{C}/ \ (M-\lambda_{1,2}) \left(\begin{array}{cc} x \\ y \end{array} \right) = 0 \right \},$$
which leads to the eigenvectors:
$$ \vec{v}_1 = (i,1), \quad \vec{v}_2 = (i,-1)$$
Cheers.


Mahalanobis distance gives counterintuitive results

Question: I have generated 100 sample time series, each 24 items long, and each with an exponential distribution with a different scale for each of the 24 time points. This is the scale parameter per time point:

My 100 time series look like this:

This is the sample covariance matrix:

Now this is the first day:

Now I will artificially create two new days: One where I add a lot to one time point where the variance is generally high (day2 gets an increase at 8 o'clock), and another one where I add the same amount to a time point where the variance is low (day3 gets the same increase at 2am).
I will expect that the distance dist(day1, day2) is a lot smaller than dist(day1, day3), because day2's increase happened in a high-variance region (8am, that is).

But the output I get is:
mahalanobis(day1, day2, Sigma)  # should be "small"
62.9029

mahalanobis(day1, day3, Sigma)  # should be larger
15.0200

Why is the distance dist(day1, day2) larger than dist(day1, day3)?
Edit: Python code to reproduce the figures and results:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

one_day_length = 24
n_days = 400

x = np.array(range(one_day_length))
scales = 0.2 + 2 * np.sin(x / 5) ** 2

# plt.plot(scales)

np.random.seed(20181106)
one_random_day = np.random.exponential(scale=scales, size=one_day_length)

# plt.plot(one_random_day)

random_days = pd.DataFrame([np.random.exponential(scale=scales, size=one_day_length) for _ in range(n_days)])

# random_days.head(20).T.plot(legend=False)

Sigma = random_days.cov()

from scipy.spatial.distance import mahalanobis

day1 = random_days.iloc[0]

# plt.plot(day1)
# plt.title('Day 1')

# plt.imshow(Sigma)

mahalanobis(day1, day1, Sigma)  # 0 of course

day2 = day1.copy()
day2[9] += 30
# plt.plot(day2)
# plt.title('Day 2 (8am += 30)')

day3 = day1.copy()
day3[2] += 30
# plt.plot(day3)
# plt.title('Day 3 (2am += 30)')

mahalanobis(day1, day2, Sigma)  # should be "small", but is 64.61
mahalanobis(day1, day3, Sigma)  # should be larger, but is 15.02

Comment: Your intuition is the opposite of how Mahalanobis distance works.  See https://stats.stackexchange.com/questions/62092 for explanations.  Note that this generalizes the basic concept of "standardization" or "z scores" from one dimension to many.  Specifically, adding a large amount to a variable that has a large variance will scarcely change its z score, while adding the same amount to a variable with small variance will make a large change in its z score.
Comment: Thanks! But your explanation is exactly my intuition, as far as I understand. Your last sentence is exactly what I expected here - and what didn't happen. In the "Day 2" plot, I add a large amount (30) to a variable (x=8am) that has large variance (see 2nd and 3rd image). That should scarcely change the z-score, i.e. result in a small distance of mahalanobis(day1, day2), relative to mahalanobis(day1, day3), right?
Comment: Nope--you have it exactly backwards.  Adding $30$ to a variable with a variance of, say, $100^2$ adds $0.3$ to its z score.  Adding 30 to a variable with a variance of $10^2$ adds $3$ to its z score.  The latter is a much greater change.
Comment: I am confused :/ In my opinion you and I are saying the same thing. I understand that adding $30$ to a variable with a variance of $100^2$ is a smaller change than adding $30$ to a variable with variance $10^2$. That's exactly what I did in my plots (right?).
Now I'm confused why the distance d(day1, day2) is larger than d(day1, day3), even though I added to the high-variance variable in day2, and I added to the low-variance variable in day3. I expected day3 to be more "extreme" (like you just explained above).
Comment: Could you add a reproducible example, i.e., code generating the data, adding 30, computing the distances? What is the distance between day1 and unmodified day2  vs. the distance between day1 and unmodified day3?
Comment: @JuhoKokkala I've added Python code at the end that reproduces the results and figures in this question. The distance between unmodified day2 and day1 is zero (I just copied the array).
Answer: The strange result is due to a programming error - according to https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.spatial.distance.mahalanobis.html the mahalanobis-function in question takes inverse covariance matrix as input. Fixing the code as, e.g.,
invSigma = np.linalg.inv(Sigma.values)
mahalanobis(day1, day2, invSigma)  # 14.41
mahalanobis(day1, day3, invSigma)  # 61.93

produces results matching with the expectation.
Indeed, since we are here adding $\Delta=30$ to $j$th element of the vector (day1), keeping the other elements constant, the Mahalanobis distance simplifies as
\begin{equation}
\sqrt{(\mathbf{x}+\Delta\mathbf{e}_j-\mathbf{x})^T\,V^{-1}\,(\mathbf{x}+\Delta\mathbf{e}_j-\mathbf{x})} = |\Delta|\,\sqrt{(V^{-1})_{j,j}}.
\end{equation}
In the setting of the question, the covariance matrix is pretty close to diagonal since it's a sample covariance of data produced from a distribution where the components are independent, and thus $(V^{-1})_{j,j}$ is close to $V_{j,j}^{-1}$. Hence, OP's expectation that modifying a component with high variance should produce a smaller Mahalanobis distance is correct. In presence of correlation, the diagonal element of  $(V^{-1})$ measures the residual variance controlling for the other variables (https://stats.stackexchange.com/a/73499/24669). That is to say, the order of the distances could have been different if the high-variance point 8 am was highly correlated with other components.
Comment: Oh god, thank you. Unfortunately I must have stopped reading before the last sentence, because the second-to-last one was "clear enough" for me.


How to interpret the results of a chi square and p-values of a distribution uniformity test

Question: I am reading about a method called consistent hash designed to distribute load among servers.
The best case scenario would be a discrete uniform distribution where each server would get the same amount of requests.
In a specific implementation I am considering using there is a table of describing the chi squared and p-values test results:

I am having difficulties understanding how to interpret the results: From what I understand a Chi square test tests a null hypothesis and that the p value is the probability that we would accept the null hypothesis even if it is wrong.
What I don't understand is how to interpret a row in the results table.
How do I associate the chi-squared test results and the value in the p-value[0.05/0.01] values to the probability that the distribution is not uniform ?
Answer: They're showing that the distribution of load is statistically indistinguishable from the uniform distribution, by (I'm guessing) partitioning the servers into 2 through 18 sets and conducting a chi-squared goodness of fit test, comparing the observed requests for each bucket with expected requests.
$\chi^2$ goodness of fit test
A $ \chi^2$ test comes up anytime you have a test-statistic that follows the $\chi^2$ distribution, i.e. the statistic is the sum of squared standard normal random variables.
One classic such situation is the chi-squared goodness of fit test of whether an observed sample came from a particular distribution.

Let $X$ be some random variable.
Let's partition the range of $X$ into $r$ buckets. (This partition is arguably somewhat arbitrary.)

If we assume that $X$ follows some distribution for our null hypothesis, we can calculate the probability $p_i$ that $X$ falls into bucket $i$ for each bucket. If we have $n$ observations, we'll expect $np_i$ entries in bucket $i$ under the null hypothesis. Let $v_i$ be the actual number of observations in bucket $i$.
As Dmitry Panchenko's lecture notes show here, the following test statistic converges to the chi-squared distribution with $r-1$ degrees of freedom.
$$ \sum_{i=1}^r \frac{\left(v_i - np_i \right)^2}{np_i} \rightarrow \chi^2_{r-1} $$
Answer: The table reports the chi-square statistic from the test.  The values under "p-value = 0.05" are the chi-square value you would need to get to get a p-value of 0.05.  So, for two buckets you would need a chi-square value of at least 3.84 to get a significant result, and they got a chi-square value of only 0.5.  So obviously, the p-value is quite greater than 0.05 for this test.
We used to use tables of p-value thresholds like this before we all had computers that could tell us the precise p-value.
Comment: I assume that the null hypothesis of the test is that the distribution is uniform.  If so, the table would suggest that there is ***no*** good evidence that the distribution is ***not*** uniform.
Comment: Thank you. From your answer it seems that the test results show that even for 20 buckets it doesn't seem that the distribution is uniform, right?


What are "Grids" and Detection at different scales" in YOLOV3?

Question: I've recently started working with Yolov3 and the more I go in depth, the more confused I get. In the simplest terms what I think about YOLOV3 (On 416 input, 80 classes, 3 BB) is that:

It Extract features from DarkNet-53 and then again use 53 Convolution networks throughout.
At 3 different levels (at layers number: 82,96,106), there are feature maps created of size :13,26,52 respectively with same depth 256. The features at 82 are upscaled till layer 96 and features from 96 upscaled till 106.( **What are these? Feature Map size?)

What I want to know from now here is that :
what are Grid cells?  Are these pixels or slices of image?
Are there 3 different outputs predicted (one for each of the 3 layers) and total loss is calculated. If so? How is this achieved? Because we input 1 image with annotations? How would that give us Ground BB for the downscaled images?
OR Is the size same at each level? IF so, what are we up sampling?
In the end, How is the total loss calculated?
Apart from reading the original paper, I have referred to many blogs and videos but I could not get the meaning of how exactly works. Could someone please explain in simple terms of how an example would go in and get predicted.
Comment: Thanks a lot buddy!
Comment: I edited my answer, hope it helps
Answer: Regarding the first comment of the question:

It Extract features from DarkNet-53 and then again use 53 Convolution networks throughout.

The network used to extract features is just DarkNet-53 as stated by the author in the YOLOv3 paper:

We use a new network for performing feature extraction [...]
It has 53 convolutional
layers so we call it.... wait for it..... Darknet-53!

Regarding the second comment of the question:

At 3 different levels (at layers number: 82,96,106), there are feature maps created of size: 13, 26, 52 respectively with same depth 256. The features at 82 are upscaled till layer 96 and features from 96 upscaled till 106. (What are these? Feature Map size?)

Yes, the sizes of 13, 26 and 52, refer to the feature map dimensions at the prediction layers. These sizes are directly related to the grid cells as we are going to see in the following lines. Just a sidenote, the correct depth (number of feature maps) is 255 and not 256 (this will also  be explained hereafter).
Grid cells
Grid cells were first explained in the original paper You Only Look Once. They are part of a regular grid (of simensions $S\times S$) into wich the input image is divided. Concretely, in YOLOv1 the author uses a grid of 7x7 cells. This is an example:

Images extracted from YOLO CVPR slides
The purpose of this type of grid (which is mantained in YOLOv3) is to serve as the output of the architecture $\Rightarrow$ $B$ bounding boxes are being predicted at each of the grid cells. Specifically, in YOLOv3, $B=3 \Rightarrow$ We are predicting 3 bounding boxes at each grid cell.
But how can this be encoded in the network?
The authors used what you mentioned in the question i.e. 3 different prediction layers/ grids composed of $13\times 13, 26\times 26$ and $52\times 52$ grid cells.
Now, we should remember that $B=3$ bounding boxes are being predicted at each of these grid cells. These boxes are encoded as a vector of $85$ features/ dimensions $\Rightarrow$ The depth (number of feature maps) needed is $85\times B = 255$, as you said in the question.
To visualize this, first we can focus on how a certain box of a certain cell is encoded:

$\mathbf{C}$ :  Is the objecteness of the associated bounding box (sort of a confidence measure of how likely a box is containing, or not, an object).
$\mathbf{(t_x, t_y)}$ : Features related to the offsets of the bounding box center w.r.t. a corner of the grid cell.
$\mathbf{(t_w, t_h)}$ : Features related to the dimensions of the bounding box.
$\mathbf{p_1,...,p_{80}}$ : Conditional probabilities, $P(\text{Class}| \text{Object})$ , for each of the $80$ classes being considered in the paper.

By doing this we end up predicting a tensor of dimensions $S\times S\times B \times 85$, which represents the output of the architecture, as commented above:

However, we are predicting 3 bounding boxes for each grid cell $\Rightarrow$ the total output is the result of concatenating three of these tensors together, making a dimensions total of $S\times S \times 85 \times 3 = S\times S \times 255$.
Moreover, as you said in the question, YOLOv3 uses three of these regular grids as outputs $\Rightarrow$ We are predicting 3 of these $S_i\times S_i \times 255$ tensors at 3 different depths of the architecture.
Predictions and training with YOLOv3
Due to the 3 different grid resolutions $(S_i = \{13, 26, 52\})$, we will be predicting outputs at 3 different depths of the architecture. This is shown in the next image extracted from the great post "What's new in YOLO v3?":

At these 3 different layers (outputs), the same object can be detected, however this is not very likely to happen. This is because, as said by the authors, each of these regular grids uses anchor boxes with different sizes $\Rightarrow$ The lowest resolution grid ($S=13$) uses bigger anchor boxes, and the highest resolution grid ($S=52$) uses smaller anchor boxes.
The purpose of this, is to be able of detecting smaller objects which are difficult to detect in a low resolution grid. This way, in order to detect bigger objects at he highest resolution grid, the difference w.r.t. these anchor boxes would be significant, difficulting the learning procedure of the network.
However, having a look at the label generation for VOC dataset:

Now we need to generate the label files that Darknet uses. Darknet wants a .txt file for each image with a line for each ground truth object in the image that looks like:
<object-class> <x> <y> <width> <height>
Where x, y, width, and height are relative to the image's width and height

There isn't distinction of scale for the ground-truth objects. This means that the network will be learning to predict the ground-truth bounding boxes at the 3 different grids $\Rightarrow$ A ground-truth object will have influence at the loss function of all 3 output grids.
This influence in learning is by backpropagation of the gradients of the 3 loss function w.r.t. the weights/ biases. One loss function for each output grid. However, note that, for example, not all the weigths/ biases will have an influence in the first and second grid loss functions.
This is because, for example, a change in the weights and biases values of one of the final layers won't affect the predictions of the first output grid $\Rightarrow$ The associated gradient will be zero.
Comment: @Deshwal NMS is applied globally. Suppose you have multiple detected BBs in different layers for the same object $\to$ This lead to a *global* image where all these detections are checked. Note that different grids have differente sizes, but they correspond to the same image $\to$ We don't get as output 3 different images, we get one $\to$ All detections are related to the same image.
Comment: What a thoughtful answer man. Appreciate it.
Comment: One more question. If we have 3 different detections at 3 levels, how do we interpret results? Let us suppose an object `A` was detected in all of the 3 stages, object `B` in 2 and object `c` in just ` stage, how do we differentiate / normalise  or combine the results in the very end?
Comment: @Deshwal That can be dealt via [Non-Maximum Suppresion (NMS)](https://towardsdatascience.com/non-maximum-suppression-nms-93ce178e177c). Its goal is to filter redundant bounding boxes that have significant IOU with the correct bounding box (BB). As it's stated in the link, this is done by getting the most confident BB of a certain class that has significant [IOU](https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/) with others of the same class. An official implementation can be viewed at: https://github.com/pjreddie/darknet/blob/master/python/darknet.py
Comment: But `NMS~ works on the same layer and same grid. I am asking about different level of detection. How would the model know that it's the same object or not? [See this elaborate questuion](https://stats.stackexchange.com/questions/530734/yolov-3-how-are-features-detection-at-3-different-level-arranged-to-give-1-f)
Comment: Oh! Okay! It'll some time for me to digest that. Just not able to visualise how are these merged? Grid sizes are different so grid centres will be different too. I mean, thanks to you I've imagined almost everything except this last one.
Comment: @Deshwal that's right, grid sizes are different, but the image is the same. Imagine that the first figure of the above answer had size of $20\times 20\to$ This would be a more refined discretization of the image, but there are no changes in the image size.


SMS PDU format - how to extract message part

Question: How can I extract a message from SMS PDU?
I need to take a message from SMS PDU. When I use some online services, they work fine. For example, here - http://www.diafaan.com/sms-tutorials/gsm-modem-tutorial/online-sms-pdu-decoder/ - message from PDU 0791448720003023240DD0E474D81C0EBB010000111011315214000BE474D81C0EBB5DE3771B is diafaan.com.
I found some SMS PDU Java implementations to do PDU->Text conversion, but it seems they didn't work as I expect because I don't extract message part from whole PDU (in another words I don't cut the service information - From, SMSC...) - so how can I do this on Java? Or just an algorithm would be a great help too. Thanks!
Answer: Finally I used SMSLib library:
            //String resultMessage = ...
            Pdu pdu = new PduParser().parsePdu(resultMessage);
            byte[] bytes = pdu.getUserDataAsBytes();
            String decodedMessage;
            int dataCodingScheme = pdu.getDataCodingScheme();
            if (dataCodingScheme == PduUtils.DCS_ENCODING_7BIT) {
                decodedMessage = PduUtils.decode7bitEncoding(null, bytes);
            } else if (dataCodingScheme == PduUtils.DCS_ENCODING_8BIT) {
                decodedMessage = PduUtils.decode8bitEncoding(null, bytes);
            } else if (dataCodingScheme == PduUtils.DCS_ENCODING_UCS2) {
                decodedMessage = PduUtils.decodeUcs2Encoding(null, bytes);
            } else {
                log.error("Unknown DataCodingScheme!");
                ...
            }

Comment: It could be done even better - there's a `getDecodedText` method in `PduUtils`


Tool to maintain IBM x3650 - get info from faulty HW

Question: I am have a server IBM x3650 M4 running XenServer and Windows guest. The server have a problem with harddrive, yellow exclamation mark blinking. I am able to turn off the server go to BIOS and get some brief info. The info is very brief and not explanatory :/
I would like to have some tool allowing me to check server status from windows guest. Something like DELL server manager. I Have found IBM® ToolsCenter Suite unfortunately it asks me for user name / password and defeaults does not work. Is this the right tool or should I use something else? Can this bypass XenServer between hardware and guest system? Or is there any other way how to get info from the server without restarting it?
Thank you!
Answer: A hypervisor, Xen in this case, is there to "protect" your hosts hardware from the guest OS'es under all circumstances.
So, no, you will not be able to get detailed hardware information out of your guest machines. The oly machine you can use to 'speak' to the hardware ist the host OS, So Xen in this case. Look for Xen management agents which you can then acces from the guest VMs.
Answer: For constant monitoring, the answer by bjoster is your solution. For a one shot troubleshooting attempt: Boot Linux from a live CD/USB drive and use it to check your hardware.