Mac source control option with both working and live repositories being remote?

Question: I've been doing LAMP development for roughly ten years now. I've occasionally used CVS and SVN. I code mostly with BBEdit.
I am trying to setup some sort of source control to manage LAMP web apps. I prefer not to have a local repository. I usually have a dev server and a live server. I am looking for a source control option that will work with that. This is largely because I work on multiple machines and don't want to configure PHP, MySQL... on each of them for testing the working copy. Plus I can't test it on other machines if the working copy is on a local machine (without additional legwork.)
What options are out there for this? For SVN, so far I've only been able to figure out how to do a remote repository with a local copy.
Additionally, these are rarely on the same servers with the same users. I've got a half dozen live sites on a half dozen different hosts. Then I have the dev sites on my own server.
Thanks.
Answer: Git works well on OSX. Due to its nature, you will also have a local repository, but this should allow you to manage what you want (local, production and dev repositories). There is indeed a learning curve, but it was worth it, for me.
It depends on the tools you're happy to use and your general work habits, but I don't mind using git from the Terminal and gitk as a tool to visualize the commits in X11. This way, gitk is not fully integrated with the Mac, but it's OK (if you want to, I think there is a flag for the Tcl/Tk compilation to use the native interface rather than X11, but I think there was a bug whereby you couldn't select and copy quickly from the diff window, if that's useful). You can install Git via MacPorts or directly from the source.
Comment: Is there a way to not have a local copy? Seems like with SVN at least one must have working/dev copy be local.
Comment: No, there isn't. Why is the local copy such a problem? It's not a bad thing to have your files locally (and a history of their modifications, in this case). The local copy is within the working directory, in a hidden `.git` directory. Would you rather edit the files on the server directly?
Comment: My issues with a local copy:
1. Dev sites are already remote. Rather not change it, but am willing to.
2. Using local copies would require me to setup PHP, MySQL... on every box I use. I would have to worry about compatibility issues.
3. Some of what I need to edit need domain names. Eg, WordPress. If my dev site was local, I'd have to setup DNS for my local box for each site. Now if I go to edit this on multiple boxes, I'll run into issues.
4. Browser testing is MUCH easier when the dev site is remote. I've got ~4 systems I test on.

If there's some easier way to go about this, I'm all ears.
Comment: I'm not sure I understand here. How to you want to edit your files here? I'm not talking of a local working copy for testing, I'm talking of a local copy where you edit the files and then push them to a remote repository on the dev/prod server. You could even push directly to a working repository (non-bare), but that's not necessarily the best/cleanest way.
Comment: This sounds similar to what Alec suggested as well. I'd prefer to be able to just hit save in whatever app I'm working in, and have the changes be immediately available on the dev server. I'll play with it some more and see if I can get that setup, but it sounds like I'd have to have SVN or Git setup on each server I need to push to.
Comment: I'm not sure how you do it normally, some editors (perhaps BBEdit) can edit over SSH or FTP semi-transparently, they download the file in a temporary location and upload it on save automatically. If your editor doesn't do that, you could either edit your files on your server directly with an editor there (e.g. using X11, this works out of the box on the Mac) or mount your remote area using MacFuse/sshfs: it would look local. That's independent of version control, though.
Comment: Thanks Bruno. That's exactly how I edit now. I went back to Versions and tried to figure out some way to make it work with what I want to do, but can't figure it out. Obviously there's a disconnect somewhere. So I'm searching for someone I know to discuss it with since this method of communication doesn't lend itself to easily figuring this out. Thanks for the help.
Answer: Just because you have a local repo with your working copy does not mean you have to deploy from the local system.
When changes are complete and checked in locally then push your changes to which ever remote repos you want (including a remote dev repo)and then deploy from there.
Does that work for you?
Comment: I'll try that and see how it works. I am constantly saving, then hitting reload in a browser... wouldn't want to have to push the changes to the dev server for testing after each save. Thanks.
Comment: I wonder if the problem is the way you have dev set up to always be deployed from CM. I would look to copy my changes to dev (or have dev pointing to my work files) for unit testing. Then I'd commit and deploy to QA and PROD from CM

(CM -> Configuration Management)
Answer: Developing on a remote server is not something I do. I much prefer to develop locally on a clone of the repo (git/hg) or working copy (svn).  Setting up a Virtual Machine with Vagrant is simple and there are many prerolled boxes or you can configure one yourself easily.
That said, you can use Transmit (there are other options) to mount a remote volume over SSH. The volume then works like any other volume and you can edit files in your favorite editor and interact with a version control system on via a terminal.


Wilcoxon rank-sum test report results when Mdns are the same

Question: I understand that Medians should be reported when writing results of Wilcoxon rank-sum test. For example:
"For alcohol users depression levels were significantly lower on Wednesday (Mdn = 7.50) than on Sunday (Mdn = 16.0), p = .012, r = −.45."
But suppose that p=.012 but Mdn=7.50 on Wednesday and Mdn=7.50 also on Sunday, but the Mean= 6.45 on Wednesday and Mean = 14.56 on Sunday.
How do I report results? How would be the text in the example then? Because depression levels are indeed lower on Wednesday, but results should be reported using Medians..
Comment: Why do you understand that? The Wilcoxon rank-sum test (aka Mann-Whitney test) does not test the difference in medians.
Comment: @JeremyMiles That's true, but it is common to report medians along with test results. And OP is confused what to report with equal medians and p<0.05
Answer: It's always worth reporting some measure of disperssion along with measure of central tendency. You probably report standard deviation with mean, when you use it. So maybe just report quartiles or IQR (interquartile range) with median?
Comment: I would report median and quartiles like "Wednesday (Mdn=7.50, Q1=5, Q3=10)"
Comment: So it would be ok like this? For alcohol users, the Wilcoxon Signed-Ranks test indicates that the ranks of the depression levels were statistically significantly lower on Wednesday (Mean = 6.45, sd=1.2 Mdn=7.50) than on Sunday (Mean = 14.56, sd= 1.3 Mdn = 7.50), p = .012, r = −.45."


Probability of getting a specific hand in Hearthstone (Online Card Game)

Question: For the sake of learning, I am trying to understand how to calculate the odds of getting a winning combination of cards within a online card game (Hearthstone, for you Blizzard fans out there).  My knowledge of statistics & statistics notation is minimal, but I am hoping to use this an an opportunity to expand my horizons a bit.
Here's the problem:  Let's say that I have a winning hand combination if I am able to have 3 specific cards by turn 3 in the game.  Let's call them Card A, Card B, & Card C. The goal is to calculate the odds of starting my 3rd with one copy of A, B, and C in my hand.
Here are some of the parameters:

I have a total of 30 cards in my deck.
Within the deck, I have 2 copies of A, 2 copies of B, and only 1 copy of C.
I start the game by drawing 8 cards (In the game, I actually draw 4 and then
can return & redraw any unwanted cards, but I don't want to muddy
the water too much).
At the beginning of each turn, I draw 1
card.

What are the odds of my getting the "winning hand"?  Can guide me where to learn more about the method that you use to answer the problem?
Comment: The subject is combinatorics.  It is a big one.  Getting things counted accurately has many pitfalls.
Answer: The winning combinations are ABC, AABC, ABBC, AABBC, where the unshown cards are among the other $25$.  This gives ${2 \choose 1}{2 \choose 1}{25 \choose 8}+2{2 \choose 1}{25 \choose 7}+{25 \choose 6}=6426200$.  Dividing by ${30 \choose 11}$ gives about $11.76\%$
Comment: What does (21) mean? The 2 above the 1
Comment: @Draken:  $25 \choose 8$ is the number of [combinations](https://en.wikipedia.org/wiki/Combination) of $8$ things chosen from $25$.  The formula would be $\frac {25!}{8!(25-8)!}$  The Wikipedia link explains it.
Comment: Appreciated, thanks!


Score function of poisson distribution

Question: When counting the score for poisson distribution I get the log likelihood
$$S(\mu ) = \frac{\partial \ell(\lambda )}{\partial \lambda } = \sum_1^n \left(\frac{y_i}{\lambda}-1\right)$$
Textbook says that it is equivalent to:
$$\frac{n(\overline{y}-\lambda)}{\lambda}$$
I can get easily solve the fisher score from there on, but I'm not quite sure about this equation. Why does it switch to the mean of y? If you take the $1/{\lambda}$ on the front of the sum, wouldn't it then be
$$S(\mu ) = \frac{\partial \ell(\lambda )}{\partial \lambda } = \left(\frac 1 \lambda \right) \sum_1^n (y_i-1) \text{ ?}$$
Comment: Two Greek letters appear here: $\mu$ and $\lambda$. Is that a typo? Did you mean $S(\lambda)$ rather than $S(\mu)\text{?} \qquad$
Answer: The summation notation is a linear operator. So expand the fraction:
$$\begin{eqnarray}
 \sum_{i=1}^n \left( y_i / \lambda - 1 \right) &=&  \sum_{i=1}^n \left( (y_i - \lambda) / \lambda  \right) \\
&=&  \left( \left(\sum _{i=1}^ny_i - n \lambda \right) / \lambda  \right) \\
&=&  n \left( \left(\bar{y} - \lambda \right) / \lambda  \right) \\
\end{eqnarray}$$
Note that the score takes the canonical form of (the sum of) sample mean minus the theoretical mean over the variance.
You will find a similar canonical form of the score function in most of the exponential families like binomial, gamma, normal, and so on.
Answer: \begin{align}
& \sum_{i=1}^n \left( \frac {y_i} \lambda -1\right) = \sum_{i=1}^n \frac {y_i-\lambda} \lambda \\[10pt]
= {} & \frac{y_1-\lambda}\lambda + \cdots + \frac{y_n-\lambda} \lambda \\[10pt]
= {} & \frac{(y_1 + \cdots + y_n) - n\lambda} \lambda = \frac{n\overline y - n\lambda} \lambda.
\end{align}
Comment: @Guest192 : The sum did not switch to $\overline y,$ but rather to $n\overline y.$ The sum is $n$ times the mean. $\qquad$
Comment: I still don't understand why does the
$$({y_1} +...+{y_n})$$

switch to the mean?

Isn't
$$({y_1} +{y_2} + {y_3} + ...+{y_n})$$
 just the sum of y's over 1 to n?

Score function of poisson distribution


Domain joining debate for Outlook 2010 with Exchange 2007 on windows SBS 2008 for a user on a laptop that will travel a fair amount of the time

Question: I'm basically debating on whether or not to join the Domain on a Laptop, and was wondering if anyone has had a similar experience.
If the computer were staying in the office, its a no brainer.  Join the domain.  In this case I have a user who will come into the office a few days a week, and work remotely the rest of the time.  There is a working VPN using OpenVPN client/server, but it's not site-to-site.
My knee jerk reaction is to not join the domain, so that the user can have 1 profile that they always use.  In this configuration, should Outlook work properly with the user's domain account, and should the shared calendar still work (at least once inside the VPN)?
My concern with joining the domain would be the inability to login to it when elsewhere.  Is there maybe a way around this with caching or something?  Would creating a second local login make sense for a user like this in any way?  If so, why not just skip the domain join to begin with?
Any thoughts on or experiences with this would be appreciated.
Laptop OS Windows 7 (Not purchased yet.. pro if domain needed)
Server SBS 2008,  Exchange 2007
Outlook version 2010
Thanks for any help,
Mike
Answer: Join it to the domain. You want to be able to manage it easily, don't you? It'll also make the user's life easier when they open outlook and try to access resources on the server.
Passwords are cached automatically. I believe the default is 10 logins cached but I may be wrong about that. You can configure more or less caching through group policy.
Comment: Yes, Join it to the domain - There is no need for a second profile/local user. He can still login to the domain account when he is outside of your LAN because of caching as Jason mentioned.
Comment: Great, Thanks!  I'll go ahead and do that then, and knock it up to more than 10 logins.
Comment: just to add to this, make sure to have a plan to rejoin this computer to the domain if it becomes disjoined.
Comment: How do you manage passwords expiring?  How do users change their passwords?
Answer: As long as your laptop user attaches to the DC's LAN every once and while, or you launch the VPN session before the user logs in, then your remote machines will behave as if they were in the office all the time.


How to construct hyperbolically equidistant points on a line?

Question: In Stillwells' "Sources of Hyperbolic Geometry " page 66 figure 3.3 shows an ((incomplete?) construction of  hyperbolically equidistant points on a line.

I tried to reconstruct the figure but did not manage it can anybody tell me how this figure is constructed?
Or another way to construct hyperbolically equidistant points on a line.
Comment: The picture doesn't work. If its a vertical line, you just want the ratios of the heights to be equal.
Comment: @BrianRushton: This is the Klein model of hyperbolic geometry, so verticality of lines is completely irrelevant.
Answer: I'm looking at the original source: Felix Klein "Vorlesungen über nicht-euklidische Geometrie", page 172. The general idea is that the cross ratio $\operatorname{CR}(P_1,P_2;A,B)$ must be equal to the cross ratio $\operatorname{CR}(P_2,P_3;A,B)$ since length can be computed from that cross ratio. Now following Kleins description, the line $BA'$ is chosen arbitrarily through $B$, and the point $S$ is arbitrary as well. Then $A',P_1',P_2'$ follow by projection with center $S$ from line $AB$ to line $A'B$. Furthermore, $P_2P_1'$ intersects $AS$ in a point $M$, which can be used as the center of a second projection back onto $AB$. Since these projections preserve cross ratios, we have
$$
\operatorname{CR}(P_1,P_2;A,B)
\overset S=
\operatorname{CR}(P_1',P_2';A',B)
\overset M=
\operatorname{CR}(P_2,P_3;A,B)
$$
These steps can be repeated to form an arbitrary number of equidistant points. As the dashed lines indicate, you can use the same points $S$ and $M$ for all of them, only the intersection points on the line $A'B$ are different for every new point you transform. In essence, projection onto $A'B$ via $S$ then down to $AB$ via $M$ is a unit step of your sequence.


Preloader disappears when sidebar is loaded

Question: For example: i have 2 controll-view container user.cv.jsx and sidebar.cv.jsx
Screen consist of User and Sidebar. Sidebar rendering in User screen.
User container:
import React from 'react'
import {Link} from 'react-router-dom';
import { connect } from 'react-redux';
import UserTypeComponents from '../components/user_type.jsx'
import Sidebar from '../../sidebar/containers/sidebar.cv.js'

import * as showList from '../action/list.action.js';
import * as userLimit from '../action/limit.action.js';

import PropTypes from 'prop-types'

function mapStateToProps (state) {
  return {...state}
}

class UserType extends React.Component {

    constructor (props, context) {
        super(props);
        this.context = context;

        if(!this.props.oauth.isAuthenticating) {
            this.context.router.history.push('/login');
            return;
        }
    }

    componentDidMount() {

    }

    render() {
        console.log(this.props);
        return (

                    <Sidebar />
                    <UserTypeComponents {...this.props} />

);
    }

}

UserType.contextTypes = {
    router: PropTypes.object
}

export default connect(mapStateToProps)(UserType);

And Sidebar Container:
import React from 'react'
import {Link} from 'react-router-dom';
import ShowSidebar from '../components/sidebar.jsx';
import { connect } from 'react-redux'
import { bindActionCreators } from 'redux'
import Preloader from '../../../helpers/preloader.helper.js'

import * as active from '../action/active.action.js'
import * as list from '../action/list.action.js'
import * as show from '../action/show.action.js'

import {DEFAULT_COMPONENTS} from '../constant/sidebar.const.js';

function mapStateToProps (state) {
  return state.sidebar
}

function mapDispatchToProps(dispatch) {

  return bindActionCreators({
        ...active,
        ...list,
        ...show
    }, dispatch);
}

class Sidebar extends React.Component {

    constructor (props) {
        super(props);
  }

  listOfLinks(){
    const makeRequest = async () => {
        try {
          const data = await (await fetch('http://localhost:3000/sidebar')).json(),
                active = this.activeComponent(data);

          this.props.list(data);
          this.props.active(active);

        } catch (err) {
          console.log(err)
        }
      }

      makeRequest()
  }

  activeComponent(data){

    for(let key of data){

      if(location.pathname.indexOf(key.name.toLowerCase()) != -1){
        return key.name.toLowerCase();
      }
    }

    return DEFAULT_COMPONENTS;

  }

  componentWillMount() {

      this.listOfLinks();

  }

  activeSidebarState(event){

      let parent = event.target.parentNode,
          target = _$('.site-sidebar__name', parent),
          text = target.innerText.toLowerCase();

      this.props.active(text);
  }

    render() {

    const loading = this.props.sidebar.links.length;

    return (loading ? <ShowSidebar changeActive={::this.activeSidebarState} active={this.props.sidebar.active} links={this.props.sidebar.links} /> : <Preloader />);

    }

}

export default connect(mapStateToProps, mapDispatchToProps)(Sidebar);

For all this, action and redusers are written. The sidebar sends a request to the server and requests all the modules and forms links to them, too. The user module is accessing the server and requires all users. The problem is that the preloader is being formed in the sidebar, and when the sidebar is loaded the preloader disappears. But the users still could not boot.
The question is: How to control the loading of the sidebar and the user, so that when these two components are updated, the state remove the preloader.
Comment: It is not recommended to make async calls at `componentWillMount`, rather use `componentDidMount`
Comment: it's just an example. The question is different.
Answer: A common practice is to store isFetching flag in the reducer and update it in respond to fetch actions. For example:
function users(state = { users: [], isFetching: false }, action) {
  switch (action.type) {
    case 'FETCH_USERS_START':
      return { ...state, isFetching: true };
    case 'FETCH_USER_SUCCESS':
      return { ...state, isFetching: false, users: action.payload.users };
    default:
      return state;
  }
}

Then you can access it from both your components via mapStateToProps and show the preloader.
A main thing here is that you need to move the async call to an action, so reducer will be able to react to it. You can use redux-thunk middleware.
Comment: Yes, but my preloader to be in sidebar. Maybe I'm calling wrong preloader
Comment: `code`
render() {

    const loading = this.props.sidebar.links.length;

    return (loading ?  : );

    }
`code`
Comment: You will be able to access `isFetching` flag from the sidebar component and show the preloader.
Comment: that is now I need to check when rendering:

`code`
(!this.props.sidebar.isFetching && !this.props.user.isFetching) ?  :


Get database secrets (username and password) from hashicorp vault using python

Question: I want to get credentials (username and password)from Hashicorp vault using hvac python library and print them out or store them in some variable. However, I am getting an error TypeError: 'Response' object is not subscriptable after the last print statement. But, my credentials are getting authenticated.
My codes is as follows
import hvac

vault_token = 's.MYBDmBvO5I.....' # Copying my token from vault

vault_url = 'https://vault.corp.foxbase.de/ui/vault/secrets'

client = hvac.Client(url=vault_url, token=vault_token)

res=client.is_authenticated()

print('here is : ',res)

secrets_list = client.secrets.kv.v1.read_secret(
    path = 'databases/creds/data-science-access'
)

print("secrets list: ", secrets_list['username'])

output is
    here is :  True
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-52-3a463dc902b7> in <module>()
     23 #secrets_list1 = secrets_list.json()
     24
---> 25 print("secrets list: ", secrets_list['username'])

My credentials in Vault sit in databases/creds/data-science-access . When I run this command in vault terminal "vault read databases/creds/data-science-access" it generates the credentials as follows
Key      Value
password xxx
username xxx

When I login to my vault, it looks like this

Can someone please rectify where I am getting it wrong? Perhaps some parameters like "vault_url","vault_token" or "path" values I have wrongly passed.
Comment: I am not sure why anyone would be asking for this but outputting passwords is not right and it may lead you to have legal issues in the future. I would stir away from doing this
Comment: @Nelson
Basically I need to connect to the database and for that i need to fetch credentials from vault (username and password). If there is a workaround which can directly fetch credentials from vault without getting them in some variables and then passing them to
    conn = pymysql.connect(
    host='prod-read-replica.eu-central-1.rds.amazonaws.com',
    user='',
    password='< password from vault>',
    db= 'analytics
Please let me know, how to do that?
Thanks'
Comment: I am also facing error while fetching data from vault, Below is my error

"TypeError: 'Response' object is not subscriptable"
Answer: You need to query result inside "data", basically below as last line:
#print("secrets list: ", secrets_list['username'])
print("secrets list: ", secrets_list['data']['username'])


Views Slideshow vs list View+jquery cycle lite?

Question: The rotating images on my work site's home page are created as a View in list format. I'm using the jQuery Cycle Lite plugin for the rotator effect. The cycle.lite.js file is set in the scripts array in my theme's .info file, and there's a bit of code in page.tpl.php to check whether this is the homepage, and if so, set up the rotator.
Now I'm working on a second rotator for another section, and I've been trying out the Views Slideshow module.
What are the pros & cons of each approach (theme vs module), especially from a server load or page weight point of view?
Answer: I personally try to avoid using any Views jQuery modules for popular jQuery plugins and prefer just to add the jQuery plugin directly to my theme by adding the .js file to my theme.info file. I will then create another .js file that runs on load that instantiates the jQuery plugin.
I use Semantic Views module which allows you to tune the output of the view to your liking. I will use this module to output the markup in a compatible format for use with the jQuery plugin.
I find that if you use a Views plugin module for something like jQuery Cycle or jCarousel for example, you are limited by the options exposed by that plugin. I prefer not to rely on a 3rd party module which may not support the latest changes or all the features of that latest jQuery plugin.
I would expect there would be a very minimal overhead using if any at all using a Views plugin instead of using the jQuery plugin natively.
Comment: I think this is the direction I'm going to go, especially since I had a lot of trouble configuring/fine-tuning Views Slideshow. Thank you for the pointer to Semantic Views -- that should help me generalize my classes to make the JS tidier!
Comment: And actually, I didn't even need Semantic Views for this, although I'm going to keep it as an idea for other stuff. Just (!) discovered the "CSS class" setting, which is enough to create a hook to use the JS on very different Views. And this way, if I decide I want to use a different jQuery plugin, I don't have to change Drupal modules.
Comment: Great I'm glad you've found a good solution. I think it's much easier to control and support long term when you don't to rely on other peoples code.
Answer: I can't speak to any specifics on performance of Views Slideshow, but it is a very widely used module, and performs well on many large sites.
In general I find it best to go with the most popular solution when working with Drupal if it fits your needs, this usually means others have put it through its paces and helped iron out any weaknesses.
There is also the added benefit that maintaining any necessary updates will be far easier with the less custom approach of Views Slideshow.
Answer: When I started to build Drupal sites I did what Camsoft is suggesting. I used views tpl files to generate only the html I needed and I used jQuery Tools to add effects. The end result was very clean code and less reliance on contrib modules. However, I ended up with many tpl files I had to maintain. Also, since I created custom solutions, I was the only one testing them. After building many sites, I have come to believe that using contrib modules is the way to go for the reason pointed out by Mark Ferree. Using Semantic Views gets me almost the code I want but the speed advantage is tremendous. So if you build Drupal sites for a living I would recommend to use contrib modules and to look at the underlying jQuery plugins so you understand the settings.
Comment: Some links with references would be nice here.


Displaying images from firefox add-on's data directory in webpage

Question: I am writing a firefox addon and have a content script that does my DOM manipulation as I want, but I now want to display some images from my add-on's data directory in the manipulated webpage (rather than link to them on a remote server) but I'm not sure how to do it.
I see that communication between the add-on module and content script should be done via port.emit, but the API says that the parameter has to be JSON serializable.
Has anyone done this before? I feel that I've missed something.
Comment: Thanks for the suggestions guys, not forgotten about this, just haven't had a chance to check out your suggestions, then I'll mark an answer
Answer: Although you can hard code the resource: uri to the data directory, it is better to construct it at runtime. This is done with the help of the @loader/options pseudo module, but a forward compatible way would be through the self module. If you don't have other reason to use the port mechanism, then you can pass the uri to the content script through the contentScriptOptions parameter. Your content script can access it from self.options. The following code will insert the mylogo.jpg at every page of the mozilla.org domain.
// var options = require("@loader/options");
// options.prefixURI + options.name + "/data/" works now but may change in the future
var pageMod = require("sdk/page-mod");
var self = require("sdk/self");

pageMod.PageMod({
  include: "*.mozilla.org",
  contentScriptOptions: {prefixDataURI: self.data.url("")},
  contentScriptWhen: "ready",
  contentScript: "var img = document.createElement('img');" +
                 "img.src = self.options.prefixDataURI + 'mylogo.jpg';" +
                 "document.body.appendChild(img);"
});

Comment: This is true - the resource path format has changed in the past and could change in the future, so you should always derive it using self.data.url().
Comment: @canuckistani you 're right, I changed the code to reflect this.
Answer: I've done this in a bootstrap addon with the contentaccessible param in the chrome.manifest here: https://gist.github.com/Noitidart/9406437
For addon-sdk, which it sounds like you are doing, I think do this: just drop your images into your data folder. Then with the self.data.url('img_name') it gives you a resource:// path, use that path in the webpages. It should work, resource paths don't have the security issue Im pretty sure.


The likelihood function of related samples

Question: I know how to get the likelihood function for two independent samples from normal distribution. What will happen to the likelihood function if the samples are related? What will be its form?
Answer: This depends on what you mean by related but for example they can be correlated so that:
$$
y_1=\mu_1+\epsilon_1 \Sigma_{1,1} + \epsilon_c \Sigma_{1,2},
$$
$$
y_2=\mu_2+\epsilon_2 \Sigma_{2,2} + \epsilon_c \Sigma_{2,1}.
$$
where $\epsilon_1 \sim N(0,1)$, $\epsilon_2 \sim N(0,1)$ and $\epsilon_c \sim N(0,1)$ independent and with $\Sigma_{i,j}$ positive real values. Here it is easy to see that $y_1$ and $y_2$ are related in a certain form: they both depend on the gaussian noise term $\epsilon_c$. This noise model can be rewritten as:
$$
[y_1,y_2] \sim N(\mu,\Sigma)
$$
where $N$ stands here for the multivariate normal distribution with mean $\mu$ (here a 2d vector) and the covariance matrix $\Sigma$ (here a 2 by 2 positive definite symmetric matrix). Then from this well described distribution, the likelihood function simply writes:
$$
L(\mu,\sigma;y=[y_1,y_2])=\frac{1}{\sqrt{(2\pi)^{2}|\Sigma|}}
\exp\left(-\frac{1}{2}({ y}-{\mu})^{T}{\Sigma}^{-1}({ y}-{\mu})
\right).
$$


O que seria Pro*C/C++?

Question: Pelo o que vi ele é utilizando no Oracle, Mas posso utilizar com outro Banco?
Pro*C/C++ é útil no dias atuais?
Poderia passar um Exemplo de código?
Ele tem diferença do PL/SQL? Ou são utilizado em conjunto?
Answer:
O que seria Pro*C/C++?

Um mecanismo que permite usar essas linguagens dentro do banco de dados como se fossem códigos PL/SQL. Pode dar algum ganho de performance e permitir algoritmos mais poderosos que seria difícil fazer na linguagem do Oracle. Muitas vezes é usado só para performance, tanto que ele pode compilar o SQL em um código C para ser usado.
O ganho muitas vezes ocorre porque não ter mais que ficar interpretando. Mas pode ser maior quando você mexe no código gerado otimizando algo que o otimizador do Oracle não consegue. É uma API suportada e deixa você fazer o que quiser.
Documentação.

Pro*C/C++ é útil no dias atuais?

É útil sim. Nunca foi de enorme utilidade, mas ainda pode ser interessante aqui ou ali.
É claro que se você é desenvolvedor e não um DBA tenderá a considerar que essas melhorias deveriam estar na aplicação e não no banco de dados. Se é um DBA em geral não mexerá com C. Mas algum consulta otimizada ainda é útil.

Poderia passar um Exemplo de código?

<code>/* ... */
for (;;) {
    printf("Give student id number : ");
    scanf("%d", &id);
    EXEC SQL WHENEVER NOT FOUND GOTO notfound;
    EXEC SQL SELECT studentname INTO :st_name
             FROM   student
             WHERE  studentid = :id;
    printf("Name of student is %s.\n", st_name);
    continue;
notfound:
    printf("No record exists for id %d!\n", id);
}
/* ... */
</code>
Coloquei no GitHub para referência futura.
Fonte.

Ele tem diferença do PL/SQL? Ou são utilizado em conjunto?

Podem ser usados em conjunto. A API permite fazer tudo o que dá para fazer no PL/SQL, mas de uma forma bem diferente. Na verdade a linguagem C ou C++ junto com a API substitui a PL/SQL nas partes que você quiser.

O que seria Pro*C/C++?


Should I use IN or OUT degree in Network Diffusion Model of Twitter network using iGraph?

Question: I'm trying to run Independent Cascade Model for my Twitter graph to see who I have to stimulate to get the maximum cascade. This code is inspired by http://php.scripts.psu.edu/hxc249/code_segments/independent_cascade.py which uses Networkx. However, I modified a bit to be able to run with iGraph.
I create my network just OUT degree which corresponds to who I follow in Twitter. However, as far as I understand in order for a node to get infected, that node has to see the information. For example, only my followers would see that I tweeted something and if that node decided to retweeted my status that means the node is infected and the followers of that node can keep retweeting.
So, should I build my graph based on my followers instead of who I follow?
So, if I use G.successors(mynode) that would be all the nodes that potentially might be infected of the tweet I tweeted?
Here's my code
def independent_cascade_igraph(G, seeds, steps=0):
    # init activation probabilities
    for e in G.es():
        if 'act_prob' not in e.attributes():
            e['act_prob'] = 0.1
        elif e['act_prob'] > 1:
            raise Exception("edge activation probability:", e['act_prob'], "cannot be larger than 1")

    # perform diffusion
    A = copy.deepcopy(seeds)  # prevent side effect
    if steps <= 0:
        # perform diffusion until no more nodes can be activated
        return _diffuse_all(G, A)
    # perform diffusion for at most "steps" rounds
    return _diffuse_k_rounds(G, A, steps)

def _diffuse_k_rounds(G, A, steps):
    tried_edges = set()
    layer_i_nodes = [ ]
    layer_i_nodes.append([i for i in A])
    while steps > 0 and len(A) < G.vcount():
        len_old = len(A)
        (A, activated_nodes_of_this_round, cur_tried_edges) = _diffuse_one_round(G, A, tried_edges)
        layer_i_nodes.append(activated_nodes_of_this_round)
        tried_edges = tried_edges.union(cur_tried_edges)
        if len(A) == len_old:
            break
        steps -= 1
    return layer_i_nodes

def _diffuse_one_round(G, A, tried_edges):
    activated_nodes_of_this_round = set()
    cur_tried_edges = set()
    for s in A:
        for nb in G.successors(s):
            if nb in A or (s, nb) in tried_edges or (s, nb) in cur_tried_edges:
                continue
            if _prop_success(G, s, nb):
                activated_nodes_of_this_round.add(nb)
            cur_tried_edges.add((s, nb))
    activated_nodes_of_this_round = list(activated_nodes_of_this_round)
    A.extend(activated_nodes_of_this_round)
    return A, activated_nodes_of_this_round, cur_tried_edges

def _prop_success(G, src, dest):
    '''
    act_prob = 0.1
    for e in G.es():
        if (src, dest) == e.tuple:
            act_prob = e['act_prob']
            break
    '''
    return random.random() <= 0.1

independent_cascade_igraph(twitter_igraph, ['121817564'], steps=1)

Answer: If A is following B, then A has an OUT edge to B, that's right.
However, I agree it is not intuitive, since in the case of twitter the propagation of information goes from B to A. It is ok, provided you take into account that there is no propagation probability for out edges; i.e. you can influence your followers, but not the people you are following.
Thus, for the propagation you should use G.in_edges (that's in networkx, I haven't used iGraph but in the docs I see it should be G.predecessors). G.sucessors would give all the neighbors in networkx, nodes with both in and out edges; in iGraph, however, it gives only out edges. (Confusing!)
It is true that in the original code you provide it uses G.sucessors, i.e. all the neighbors in networkx. Do you have any context for that code, when it was used?
This is another piece of code, but this one if for twitter. Unfortunately it is in java but you can see it uses G.in_edges. But the algorithm is also described in English,

Each node is evaluated as per the following steps,
1. Push it to the stack
2. while stack is not empty
2.1 pop the node (v) from stack
2.2 consider it as active and put it into active sec
2.3 Repeat following for each **inbound neighbor** (u) of node
2.3.1 generate a random floating point number (rand) between 0 to 1.
2.3.2 if (propagation_probability(u, v)  <  rand)
184.108.40.206 if (u is not in active set) push u to stack

Hope it helps.
EDIT:
After comments, I add my very simple code for the implementation using networkx. Of course, any feedback would be more than welcome. The algorithm itself is debugged, however I must say I am still trying to figure out better ways for the following:

setting the probabilities for the propagation of each of the users
validating the results
how good the retweets are a proxy for the structure of the network (since downloading the followers is too slow due to Twitter's rate limits)

This is the method for the diffusion,
def single_diffusion(G, seeds, tot_users, debug=False):
    targets = list()
    actives = set()
    results = {}

    for seed in seeds:
        targets.append(seed)
        actives.add(seed)
        while True:
            target = targets.pop()
            if debug:
                print("t", target)
            for follower, u in G.in_edges(target):
                if debug:
                    print("f", len(actives), follower)
                    print(G[follower][target]['weight'])
                if random.random() < tot_users.loc[follower, 'p']:
                    if follower not in actives:
                        if debug:
                            print("a", len(actives), follower)
                        actives.add(follower)
                        targets.append(follower)
            if len(targets) == 0:
                if debug:
                    print("number of actives: ", len(actives))
                results[seed] = len(actives)
                break
            if len(actives) > len(tot_users):
                raise RuntimeError("infinite loop!")

    return results

where tot_users is a dataframe for all the users in the graph, and contains a column for the probability of distribution.
The loop for the results, and an example for calculating the mean.
results = {}
for i in range(num_iter):
    new_results = single_diffusion(G, seeds)
    if results != {}:
        for k,v in new_results.items():
            results[k] = results[k] + [v]
    else:
        for k,v in new_results.items():
            results[k] = [v]

for k,v in results.items():
    print(k)
    print(np.mean(results[k]))

Comment: I have do that in java as well. Can you provide the source code?
Comment: I meant that the link provides code in `java`. Well, it's kind of pseudo-code I guess. I'm working in python with `networkx`. However, I'm not using information from followers (rest api is too slow to get them for a big enough number of users), I'm using info from retweets to build the graph, but still not sure about the results, honestly!
Comment: Ok, the code too dirty to post it right now, but I will post it by the end of the week (i.e. the part that corresponds to the algorithm)
Comment: Done. Hope it helps.


How to get the first group from a table?

Question: I have a itemlocation table like:
item     address  date
a        x        1
b        x        1
a        y        2
b        y        2
...

I want to return the result set of
item     address  date
a        x        1
b        x        1

where it includes all the items at the first address group order by date.
I was looking at something similar to rownum but it's for the first in the group, not the first group of a table. Something like:
select * from (select * from itemlocation group by address order by date) where group_number <= 1;

How can I achieve this?
Answer: We use rank() to find all the rows with the earliest date and then filter by rnk_dte = 1
select  item
       ,address
       ,"date"
from   (
       select  t.*
              ,rank() over(order by "date") as rnk_dte
       from   t
       ) t
where  rnk_dte = 1

ITEM
ADDRESS
date

a
x
1

b
x
1

Fiddle
Comment: In my case I need to rank() on address, but it works thanks!


WebView Direct Download returns HTML which use the original file extension

Question: I am trying to perform a direct download inside the WebView, not linking to the browser.
webview.setDownloadListener(new DownloadListener() {
    @SuppressLint("DefaultLocale")
    @Override
    public void onDownloadStart(String url, String userAgent, String contentDisposition, String mimetype, long contentLength) {
        MimeTypeMap mtm = MimeTypeMap.getSingleton();
        DownloadManager downloadManager = (DownloadManager) getSystemService(Context.DOWNLOAD_SERVICE);
        Uri downloadUri = Uri.parse(url);

        // get file name. if filename exists in contentDisposition, use it. otherwise, use the last part of the url.
        String fileName = downloadUri.getLastPathSegment();
        int pos = 0;

        if ((pos = contentDisposition.toLowerCase().lastIndexOf("filename=")) >= 0) {
            fileName = contentDisposition.substring(pos + 9);
            pos = fileName.lastIndexOf(";");

            if (pos > 0) {
                fileName = fileName.substring(0, pos - 1);
            }
        }

        // predict MIME Type
        String fileExtension = fileName.substring(fileName.lastIndexOf(".") + 1, fileName.length()).toLowerCase();
        String mimeType = mtm.getMimeTypeFromExtension(fileExtension);

        // request saving in Download directory
        Request request = new DownloadManager.Request(downloadUri);
        request.setTitle(fileName);
        request.setDescription(url);
        request.setMimeType(mimeType);
        request.setDestinationInExternalPublicDir( Environment.DIRECTORY_DOWNLOADS, fileName);

        Environment.getExternalStoragePublicDirectory( Environment.DIRECTORY_DOWNLOADS).mkdirs();

        // request in download manager
        downloadManager.enqueue(request);
    }
});

And to open it,
// download complete toast
private BroadcastReceiver completeReceiver = new BroadcastReceiver() {
    @Override
    public void onReceive(Context context, Intent intent) {
        Resources res = context.getResources();

        // make toast
        Toast.makeText(context, res.getString(R.string.download_complete), Toast.LENGTH_SHORT).show();

        // go to download finished window
        startActivity(new Intent(DownloadManager.ACTION_VIEW_DOWNLOADS));
        }
    };

    @Override
    protected void onPause() {
        super.onPause();

        // if app stops, stop reciever
        unregisterReceiver(completeReceiver);
    }

    @Override
    protected void onResume() {
        // app start, start reciever
        IntentFilter completeFilter = new IntentFilter(DownloadManager.ACTION_DOWNLOAD_COMPLETE);
        registerReceiver(completeReceiver, completeFilter);
        super.onResume();
    }
}

However, the result is not the original file, but the HTML format of the source using the original file extension; e.g. it uses .pdf, but is a HTML file.
What is causing this problem, and How can I fix it?
Files I get
First of all, it is impossible to open with a normal PDF viewer, and when I don't open it as PDF, I get a normal HTML document:
<!DOCTYPE html>

…

There is NOTHING SPECIAL in the HTML document.
Comment: Give us some sample of the HTML you get, it's most likely something related to the header. What kind of back end returns the file you request, is there a url?
Answer: I found that your filename is coming with quote so just replace it.
fileName=fileName.replaceAll("\"", "");

you will get the proper file. I have also used that code in mine and it's successfully worked.
Comment: @jyoon Please add it above "// request saving in Download directory".
Comment: @jyoon Is the downloading of files encrypted in android?
Comment: `fileName=fileName.replaceAll("\", "");`
Comment: @jyoon fileName=fileName.replaceAll("\"", ""); this worked for me


Why do we divide by the standard deviation and not some other standardizing factor before doing PCA?

Question: I was reading the following justification (from cs229 course notes) on why we divide the raw data by its standard deviate:

even though I understand what the explanation is saying, it is not clear to me why dividing by the standard deviation would achieve such a goal. It says so that everyone is more on the same "scale". However, its not entirely clear why dividing by the standard deviation achieves that. Like, whats wrong with dividing by the variance? Why not some other quantity? Like...the sum of absolute values? or some other norm... Is there a mathematical justification for choose the STD?
Are the claims in this extract a theoretical statement that can be derived/proved through mathematics (and/or statistics) or is it more one of those statement that we do because it seems to work in "practice"?
Basically, can one provide either a rigorous mathematical explanation of why that intuition is true? Or if its just an empirical observation, why we think that works in general before doing PCA?
Also, in the context of PCA, is this the process of standardizing or normalizing?

Some other thoughts I had that might "explain" why the STD:
Since PCA can be derived from maximizing the variance, I guessed that dividing by a related quantity such as the STD, might be one of the reasons we divided by the STD. But then I considered that maybe if we defined maybe a "variance" with any other norm, $\frac{1}{n} \sum^{n}_{i=1} (x_i -\mu)^p$, then we would divide by the STD of that norm (by taking the pth root or something). Though, it was just a guess and I am not 100% about this, hence the question. I was wondering if anyone knew anything related to this.

I did see that there was maybe a related question:
PCA on correlation or covariance?
but it seemed talk more about when to use "correlation" or "covariance" but lacked rigorous or convincing or detailed justifications, which is what I am mainly interested in.
Same for:
Why do we need to normalize data before analysis
related:
"Normalizing" variables for SVD / PCA
Comment: There is a mathematical reason - dividing the (centered) data by the SD for each variable produces a transformed data set whose covariance matrix is simply the correlation matrix of the original (centered) data. After that, we're on correlation vs covariance matrix territory again. Are you seeking proof of how normalizing the data turns the covariance matrix into a correlation matrix?
Comment: Regarding the terminology, "normalizing" is not a precise term and can refer to various things. Whereas "standardizing" means subtracting the mean and dividing by standard deviation, which is what you are referring to.
Comment: I should add that I think your question is very good (+1). One can indeed normalize by dividing by something else; for example, standard deviation is a very non-robust measure and can be misleading in the presence of strong outliers. So one can choose to divide by some robust measure of spread instead (see e.g. "median absolute deviation"). There is no "rigorous mathematical explanation" of why using STD is the best way to normalize, and you are right on the mark that it is "just an empirical observation" that it often works well.
Comment: Re "why not divide by variance instead" - that can be fairly easily explained by the dimensional inconsistency. It would give you strange results if you changed the units one of the variables was in, for instance. Re "why not divide by MAD" - if the data were normally distributed, then since (in the population) MAD is proportional to SD, it would be possible to divide by [an appropriate multiple](http://en.wikipedia.org/wiki/Median_absolute_deviation#Relation_to_standard_deviation) of MAD and get an (inefficient but possibly robust?) estimate of the correlation. That's more interesting.
Comment: Your title question reads as if you are asking what the purpose of normalizing it (as opposed to not normalizing). This would be a duplicate of "PCA on correlation or covariance". However, what you actually seem to be asking is why normalization is done via dividing by STD (as opposed to dividing by variance, or range, etc.). If so, do you perhaps want to edit to make the title question more precise?
Comment: @amoeba I would't mind changing the title. What do you suggest a better title would be? Something like: "Justifying why one divides by the standard deviation instead than by the variance or some other norm?" Btw, feel free to change the title if you find a better way to express it. Its sometimes hard to summarize everything I want to ask about in one sentence.
Comment: http://stats.stackexchange.com/q/12200/3277 - one more link to add.
Comment: I did not want to edit myself, because I was not sure that I correctly understood your central question. Perhaps something like "Why normalizing the variables for PCA is done by dividing by standard deviation?" (or maybe "... by dividing specifically by standard deviation?").
Comment: The described form of standardization is called z-standardization (= converting to z-scores). The mean becomes 0 and st. deviation (and variance) becomes 1.
Comment: ... and [thence](http://stats.stackexchange.com/a/22520/3277) MSCP (mean sum of squares & cross-products) matrix decomposed in PCA becomes the correlation matrix.
Comment: @amoeba thanks :) While I was thinking about this, since PCA can be derived from maximizing the variance, I guessed that dividing by a related quantity such as the STD, might be one of the reasons we divided by the STD. I then thought that maybe if we defined maybe a "variance" with any other norm, say L1, then we would divide by the STD of that norm. Though, it was just a guess and I am not 100% about this, hence the question.
Comment: Funny, it does seem like an exact duplicate of ""Normalizing" variables for SVD / PCA" question, but unfortunately that one does not contain any answers discussing other possible normalization factors (apart from $\sigma$), so I am reluctant to mark this question as a duplicate of that one... On the other hand, for potential answerers it might be a good idea to provide a new answer in *that* thread and then mark this one as duplicate :)
Comment: Suppose we aren't discussing to center or not to center (agreed to center), then why scale by dividing by a variable's s.d. and not other variable statistic (such as variance or mean abs deviation)? My answer is: Linear PCA _is_ the least-squares diagonalizer tool for the [X'X-type](http://stats.stackexchange.com/a/22520/3277) matrix. Then, _if_ we want the matrix in its input view to have equal diagonal elements (we may or may not want, depends on our reasons) then division each variable by its s.d. (or value proportional to it) is the _only_ way to achieve that.
Answer: This is in partial answer to "it is not clear to me why dividing by the standard deviation would achieve such a goal". In particular, why it puts the transformed (standardized) data on the "same scale". The question hints at deeper issues (what else might have "worked", which is linked to what "worked" might even mean, mathematically?), but it seemed sensible to at least address the more straightforward aspects of why this procedure "works" - that is, achieves the claims made for it in the text.
The entry on row $i$ and column $j$ of a covariance matrix is the covariance between the $i^{th}$ and $j^{th}$ variables. Note that on a diagonal, row $i$ and column $i$, this becomes the covariance between the $i^{th}$ variable and itself - which is just the variance of the $i^{th}$ variable.
Let's call the $i^{th}$ variable $X_i$ and the $j^{th}$ variable $X_j$; I'll assume these are already centered so that they have mean zero. Recall that $$Cov(X_i, X_j) =\sigma_{X_i} \, \sigma_{X_j} \, Cor(X_i, X_j)$$
We can standardize the variables so that they have variance one, simply by dividing by their standard deviations. When standardizing we would generally subtract the mean first, but I already assumed they are centered so we can skip that step. Let $Z_i = \frac{X_i}{\sigma_{X_i}}$ and to see why the variance is one, note that
$$Var(Z_i) = Var\left(\frac{X_i}{\sigma_{X_i}}\right) = \frac{1}{\sigma_{X_i}^2}Var(X_i) = \frac{1}{\sigma_{X_i}^2} \sigma_{X_i}^2 = 1$$
Similarly for $Z_j$. If we take the entry in row $i$ and column $j$ of the covariance matrix for the standardized variables, note that since they are standardized:
$$Cov(Z_i, Z_j) =\sigma_{Z_i} \, \sigma_{Z_j} \, Cor(Z_i, Z_j) = Cor(Z_i, Z_j)$$
Moreover when we rescale variables in this way, addition (equivalently: subtraction) does not change the correlation, while multiplication (equivalently: division) will simply reverse the sign of the correlation if the factor (divisor) is negative. In other words correlation is unchanged by translations or scaling but is reversed by reflection. (Here's a derivation of those correlation properties, as part of an otherwise unrelated answer.) Since we divided by standard deviations, which are positive, we see that $Cor(Z_i, Z_j)$ must equal $Cor(X_i, X_j)$ i.e. the correlation between the original data.
Along the diagonal of the new covariance matrix, note that we get $Cov(Z_i, Z_i) = Var(Z_i) = 1$ so the entire diagonal is filled with ones, as we would expect. It's in this sense that the data are now "on the same scale" - their marginal distributions should look very similar, at least if they were roughly normally distributed to start with, with mean zero and with variance (and standard deviation) one. It is no longer the case that one variable's variability swamps the others. You could have divided by a different measure of spread, of course. The variance would have been a particularly bad choice due to dimensional inconsistency (think about what would have happened if you'd changed the units one of your variables was in, e.g. from metres to kilometres). Something like median absolute deviation (or an appropriate multiple of the MAD if you are trying to use it as a kind of robust estimator of the standard deviation) may have been more appropriate. But it still won't turn that diagonal into a diagonal of ones.
The upshot is that a method that works on the covariance matrix of standardized data, is essentially using the correlation matrix of the original data. For which you'd prefer to use on PCA, see PCA on correlation or covariance?
Comment: I think this answer does not really touch on the *actual* (and non-trivial) question of why standard deviation is taken as a measure of spread and used for normalization. Why not taking median absolute deviation instead? Granted, the resulting covariance matrix will not be the "default" correlation matrix, but perhaps it will be better, e.g. a more robust estimation of the correlation matrix. See also my last comment to the OP.
Comment: @ameoba On the "medium deep" point, the fact that we get variances of one down the diagonal of the new covariance matrix is essentially what we mean by getting the transformed data to have variables "on the same scale" from the PCA perspective. On the "very deep" issues raised by this question, I'm not sure there is much difference between asking "well why do we use variances as our measure of scale in PCA?" and asking "why does PCA concern itself with (co)variances?" - or at least, that the two issues would be intimately related.
Comment: @amoeba, why not divide by MAD or by variance instead of SD is, essentially, the same question as why to differentially scale at all: that is, why not to do PCA on covariances instead? I support this idea in the preceding comment.
Comment: @ttnphns: If different variables have completely incomparable scales (temperature, length, weight, etc.), then the desire to somehow normalize the variables is quite understandable. That's the common argument for using correlation matrix instead of covariance matrix. But if somebody is worried about outliers, I see nothing wrong with subtracting the median instead of the mean and dividing by MAD instead of SVD... I never did it myself, but I think it does sound like a reasonable thing to do.
Comment: @amoeba, Linear PCA needs the matrix to be the [SSCP-type](http://stats.stackexchange.com/a/22520/3277) matrix. Any linear transform of the original variables preserves this type. Of course, you could do any nonlinear transform as well (such as, for instance, ranking, to get Spearman rho matrix), but then component scores and loadings loose their direct (in sense of least squares minimization) connection with the data: they now represent the transformed data instead!
Comment: @ameoba That is correct. At present I think the question is not quite clear, this is only an answer on the narrow point: the OP asks "how does dividing by the standard deviation achieve that" (the "that" refers to the text, which mentions things like how does the variance become unit).
Comment: I am confused by your latest update. Are you saying that if the data cloud is rotated then the eigenvalues of the correlation matrix will always stay the same? I am either missing something, or this cannot be true. Imagine a 2D Gaussian data cloud strongly elongated along the x-axis. Standardizing both x and y will make it spherical, i.e. with eigenvalues 1 and 1. But if we first rotate it 45 degrees and then standardize, it will remain strongly stretched along the diagonal, with eigenvalues close to 2 and 0. Not sure how does fits to your simulation (which I only briefly looked at).
Comment: @amoeba That's quite correct; I happened to pick an unfortunate choice for the covariance matrix in the simulation matrix which obscured that. I rolled back to avoid confusion.
Answer:
Why do we divide by the standard deviation
  whats wrong with dividing by the variance?

as @Silverfish already pointed out in a comment, the standard deviation has the same unit as the measurements. Thus, dividing by standard deviation as opposed to variance, you end up with a plain number that tells you where your case is relative to average and spread as measured by mean and standard deviation.
This is very close to the idea of $z$-values and the standard normal distribution: If the data are normally distributed, standardization will transform them to a standard normal distribution.
So: standardization (mean centering + scaling by standard deviation) makes sense if you consider the standard normal distribution  sensible for your data.

Why not some other quantity? Like...the sum of absolute values? or some other norm...

Other quantities are used to scale data, but the procedure is called standardization only if it uses mean centering and dividing by standard deviation. Scaling is the generic term.
E.g. I work with spectroscopic data and know that my detector has a wavelength-dependent sensitivity and an (electronic) bias. Thus I calibrate by subtracting the offset (blank) signal and multiplying (dividing) by a calibration factor.
Also, I may be centering not to the mean but instead to some other baseline value, such as the mean of a control group instead of the grand mean.  (Personally, I almost never standardize as my variates already have the same physical unit and are in the same order of magnitude)
See also: Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?
Comment: +1. Nice example with spectroscopic data. By the way, congratulations with reaching 10k rep!
Answer: This link answers your question clearly, I guess: http://sebastianraschka.com/Articles/2014_about_feature_scaling.html
I quote a small piece:

Z-score standardization or Min-Max scaling?
"Standardization or Min-Max scaling?" - There is no obvious answer to this question: it really depends on the application.
For example, in clustering analyses, standardization may be especially crucial in order to compare similarities between features based on certain distance measures. Another prominent example is the Principal Component Analysis, where we usually prefer standardization over Min-Max scaling, since we are interested in the components that maximize the variance (depending on the question and if the PCA computes the components via the correlation matrix instead of the covariance matrix; but more about PCA in my previous article).
However, this doesn't mean that Min-Max scaling is not useful at all! A popular application is image processing, where pixel intensities have to be normalized to fit within a certain range (i.e., 0 to 255 for the RGB color range). Also, typical neural network algorithm require data that on a 0-1 scale.


Batch : Echo Two parameters with space

Question: I am working on Batch scripting and facing some problem while echo two variables. For this I created the file
test.bat

echo %2%
echo "Calling Two Paramters"
echo - %1% %2%

When calling, what output comes:
test.bat 1235 899
899
"Calling Two Paramters"
- `12352`

Now Expected Output must be:
1235 899

PLease help
Comment: Okay... so the correct last line is: `echo - %~1 %~2` (as @npocmaka already pointed out in [his answer](http://stackoverflow.com/a/37112843) there is only one `%` for arguments, and the `~` modifier removes potential surrounding qouotes; obviously your question was not clear enough to him though -- like also for me -- so he interpreted the two numbers as a *single* argument)...
Comment: Do you want `1235 899` to be treated as a single parameter, or do you want it to be seen as two arguments `1235` and `899`?
Comment: Two different Arguments
Comment: Thanks for the help...
Answer: Your error:
When using arguments in batch files, there is no trailing percentage character. So the first argument is %1, and not %1%.
Bonus tip:
I like to use @echo off as the first line in batch files, so the command interpreter doesn't echo all lines before parsing (and executing) them.
Example:
@echo off
echo %2
echo "Calling Two Paramters"
echo - %1 %2

You can ditch the @echo off if you don't want it ;)
Answer: Alter your test.bat like this:
echo "Calling Two Paramters"

echo - %~1

and call it like this:

test.bat "1235 899"

Subroutine and batch file arguments are have % only at the beginning.
Arguments with delimiters need to be enclosed with double quotes.
%~1 the tilda here is used for dequoting.
Comment: No. I have two different argunents as I need to pass those two to calling script in which first one is my streamname and second is version.
Comment: @R.Sha - then you can use just `echo %1 %2`
Answer: Modify your script's last line like this:
echo %1 %2%

Comment: This would works, but because of a hack that causes the last `%` to be interpreted as the start of an empty variable. Better would be to use `echo %1 %2`, or even better `echo %~1 %~2`

Batch : Echo Two parameters with space


closed formula for determinant

Question: Consider the following matrix
$$
\begin{equation}A_{r-1} :=
\begin{bmatrix}
\frac{1}{x_{1}} & -p & \dots  & 0  &\dots &0    \\
-q & \frac{1}{x_{2}} & -p &0  & \dots  & 0    \\
0 & -q & \frac{1}{x_{3}} &-p  & ~...  & 0    \\
0 & 0 &-q &\frac{1}{x_{4}} &-p   & 0    \\
0  &\vdots    & \ddots & -q & \frac{1}{x_{r-2}}  & -p \\
0  &0 &0 &\dots &-q  &\frac{1}{x_{r-1}}
\end{bmatrix}
\end{equation}
$$
where $x_{i},p,q \in \mathbb{R}$, $x_{i} \neq 0$ for all $i = 1,2,...,r-1$ and $r \in \mathbb{N}$, $r \geq 3$. I want to find a closed formula for $\det(A)$. For $r=3$ we have
$$
\begin{equation}
\det(A) =
\begin{vmatrix}
\frac{1}{x_1} & -p   \\
-q & \frac{1}{x_2} \\
\end{vmatrix} = \frac{1 -pq(x_1 x_2)}{x_1
x_2}\end{equation} .
$$
For $r = 4$ we have
$$
\begin{equation}
\det(A) =
\begin{vmatrix}
\frac{1}{x_1} & -p & 0   \\
-q & \frac{1}{x_2} & -p \\
0  &-q &\frac{1}{x_3}  \\
\end{vmatrix} = \frac{1 -pq(x_1 x_2 + x_2 x_3)}{x_1 x_2 x_3}
\end{equation} .
$$
Up to this point I think the formula is given by
$$
\det(A) = \frac{1 - pq(x_{r-1}x_{r-2} + x_{r-2}x_{r-3})}{x_{1}x_{2}...x_{r-1}}.
$$
But this is not correct. For $r = 5$ I get
$$
\begin{equation}
\det(A) =
\begin{vmatrix}
\frac{1}{x_1} & -p & 0 &0   \\
-q & \frac{1}{x_2} & -p &0 \\
0  &-q &\frac{1}{x_3} &-p  \\
0 & 0 &-q &\frac{1}{x_4 } \\
\end{vmatrix} = \frac{1 -pq(x_1 x_2 + x_2 x_3 + x_3 x_4) + p^2 q^2x_1 x_2 x_3x_4}{x_1 x_2 x_3 x_4}.
\end{equation} .
$$
Has someone has an idea how to find a closed formula? Thanks for any hints!
Answer: Since the matrix is tridiagonal, its determinant satisfies the following recurrence:
$$
\det A_n = \frac{1}{x_n} \det A_{n-1} -pq \det A_{n-2}
$$
Not a closed formula, but probably just as good.

closed formula for determinant


Pushforward of Hamiltonian vector field by (reverse) Hamiltonian flow is Hamiltonian

Question: Let $M$ be a smooth manifold with a Poisson tensor $\pi$.
Let $X_f = \pi^{\#}(df)$, $X_g=\pi^{\#}(dg)$ be two Hamiltonian vector fields.
Let $\Phi_g^u:M\to M$ be the time-$u$ flow of $X_g$.
(That is, $\Phi_g^u(x)$ is the result of integrating $X_g$ for time $u$ starting from the point $x$.)

Claim: The pushforward of $X_f$ by $\Phi_g^{-u}$ is Hamiltonian in the following way: $(\Phi_g^{-u})_*X_f=X_{f \circ \Phi_g^u}$.

Attempt: Let $Y = (\Phi_g^{-u})_*X_f$ and let $\Phi_Y^t$ denote its time-$t$ flow.
We know that $\Phi_f^t \circ \Phi_g^{u} = \Phi_g^u \circ \Phi_Y^t$ by pushforward, hence $\Phi_Y^t = \Phi_g^{-u} \circ \Phi_f^t \circ \Phi_g^{u}$.
Also $X_{f \circ \Phi_g^u} = \pi^{\#}(d(f \circ \Phi_g^u)) = \pi^{\#}(df \circ d\Phi_g^u)$.
Now I just need to take a derivative and/or view $\Phi_g^{\pm u}$ as a change of coordinates, but how exactly?
Answer: The time-$u$ flow $\Phi_{g}^{u}$ of $X_{g}$ is a Poisson diffeomorphism, i.e. $(\Phi_{g}^{u})_{*}\pi=\pi$. Expressing this in terms of the sharp map gives
$$
\pi^{\sharp}=(\Phi_{g}^{u})_{*}\circ\pi^{\sharp}\circ(\Phi_{g}^{u})^{*}.
$$
In particular, we get
$$
(\Phi_{g}^{u})_{*}X_{f\circ\Phi_{g}^{u}}=(\Phi_{g}^{u})_{*}X_{(\Phi_{g}^{u})^{*}f}=(\Phi_{g}^{u})_{*}\big(\pi^{\sharp}(d(\Phi_{g}^{u})^{*}f)\big)=(\Phi_{g}^{u})_{*}\big(\pi^{\sharp}((\Phi_{g}^{u})^{*}df)\big)=\pi^{\sharp}(df)=X_{f}.
$$
Composing both sides with $(\Phi_{g}^{-u})_{*}$ gives the desired equality
$$
X_{f\circ\Phi_{g}^{u}}=(\Phi_{g}^{-u})_{*}X_{f}.
$$


How can I tell when a function is increasing or decreasing?

Question: I have the function $f(x) = 5x^{2/3} - 5x^{5/3}$. I can differentiate it and find the zeros of $f'(x)$ which equals $\frac{10}{3\sqrt[3]{x}} - \frac{25\sqrt[3]{x^2}}{3}$, but I'm not sure how to tell when it is increasing or decreasing. I have even graphed it in Desmos, but even though it only seems to be increasing on the interval $(0, \frac{2}{5})$ this is apparently not the case.
How can I tell when this function is increasing or decreasing?
Differentiate:
$$\frac{d}{dx} (5x^{2/3} - 5x^{5/3}) = \frac{10}{3}x^{-1/3}-\frac{2}{3}x^{2/3}=\frac{10}{3\sqrt[3]{x}} - \frac{25\sqrt[3]{x^2}}{3}$$
Find Zeros:
$$\frac{10}{3}x^{-1/3}-\frac{2}{3}x^{2/3}=0 \to \frac{10}{3\sqrt[3]{x}} =\frac{25\sqrt[3]{x^2}}{3} \to 30 = 25(x^{2/3})(3x^{2/3}) \to 30 = 75(x^{2/3})(x^{1/3}) \to \frac{30}{75}=x$$
$$x = \frac{2}{5}$$
Comment: There's a theorem to the effect that if the derivative is positive throughout an interval, then the function is strictly increasing in that interval, and analogously for "derivative is negative" and "strictly decreasing". One common method that uses these results is to determine where the derivative is zero and where the derivative is undefined, and if what's left over separates into non-overlapping intervals, then you can use a standard precalculus "sign chart" method. Several such examples [are described in detail here](http://mathforum.org/kb/plaintext.jspa?messageID=5324603).
Comment: Try this [link](https://www.cliffsnotes.com/study-guides/calculus/calculus/applications-of-the-derivative/increasing-decreasing-functions) they show you how to use derivatives to find where a function is increasing or decreasing and also have two examples.
Answer: Your function is strictly increasing between the points where it has a derivative of 0.
Therefor, let's say you set $f'(x) = 0$ and got x = 1 and x = 3 as a solution.
Between $-\infty$ and $1$, it would be either strictly growing or falling.
Same counts for the Intervall between $1$ and $3$ and the Intervall from $3$ to $\infty$
Now to check if the function grows in these intervals, take any x value inside that Intervall and check if the derivative at that point is greater than $0$. if it is, it's growing. If not, it's falling. If it's $0$, you made a mistake.
Answer: Factor your function first.
$$f(x) = 5x^{2/3}(1-x).$$
At $x = 0$, $f(x)=0$, and at $x = 1$, $f(x) = 0$.
This function has only two roots at $x = 0,1$.  You found that $f'(2/5) = 0$.
Therefore, it must be decreasing on $(2/5,\infty)$.


A question on $L^{p}$ spaces

Question: Consider the space of essentially bounded functions from $\mathbb{R}^{n}$ to $\mathbb{C}$ , denoted by  $L^{\infty}(\mathbb{R}^{n})$.
Note that the $p$-norm is defined by $\| g\|_{p}=[\int_{\mathbb{R}^{n}}|g|^{p}]^{1/p}$, for $p\geq1$.
Let $g\in L^{1}(\mathbb{R}^{n}) \cap L^{\infty}(\mathbb{R}^{n})$. Show that $\| g\|_{p}$ is finite for all $p\geq1$.
Any hint or help will be greatly appreciated.
Comment: Thanks for having accepted my answer. Please, let me know if you have any question regarding my answer. If my answer provides relevant / helpful information regarding your question, please, upvote it too. To upvote, click the triangle **pointing upward above the number** (of votes) in front of the question.
Comment: Hint: If $g$ is bounded, can you find an inequality like $|g|^p \le K |g|$ for some constant $K$?
Answer: Actually with the same effort, we can prove a more general result

Let $r \geq 1$ and $g\in L^{r}(\mathbb{R}^{n}) \cap L^{\infty}(\mathbb{R}^{n})$. Then  $\| g\|_{p}$ is finite for all $p\geq r$.

Proof:
If $p = +\infty$ the result is trivial. So, suppose that  $1 \leqslant r <\infty$ and $1 \leqslant p <\infty$.
Since $g\in  L^{\infty}(\mathbb{R}^{n})$, there is $M >0$ such that $|g| < M$ a.e.. So,
for $p$ such that $r \leqslant p <\infty$, we have
$$ |g|^p < M^{p-r}|g|^r$$
So
$$ \int |g|^p < M^{p-r}\int |g|^r = M^{p-1}\|g\|_r^r $$
Since $g\in L^{r}(\mathbb{R}^{n})$, we have that $\|g\|_r<\infty$. So we have
$$\|g\|_p^p= \int |g|^p < M^{p-r}\int |g|^r = M^{p-r}\|g\|_r^r <\infty$$
So $\|g\|_p<\infty $.
Answer: I think the answer given by Ramiro completely solves your question. I would like to state a more general result as follows.

Let $\Omega\subset \mathbb{R}^n$,$1\leq p\leq r\leq q <\infty$. If $f\in L^p(\Omega)\cup L^q(\Omega)$, then $f\in L^r(\Omega)$.

Proof. We have the chain of equalities.
\begin{align}
&\int_\Omega|f|^r\mathrm{d}x\\
=&\int_{\Omega\cup \{|f|\geq 1\}}|f|^r \mathrm{d}x+ \int_{\Omega\cup \{|f|<1\}}|f|^r \mathrm{d}x\\
=& \int_{\Omega\cup \{|f|\geq 1\}}|f|^q \mathrm{d}x+ \int_{\Omega\cup \{|f|<1\}}|f|^p \mathrm{d}x\\
\leq&\|f\|_q^q+\|f\|_p^p<\infty
\end{align}

A question on $L^{p}$ spaces


Migrate SBS 2003 to Server 2008 Std

Question: I'm looking to migrate from SBS 2003 to Server 2008 Std.  Are there any guides out there similar to Microsoft's sbs to sbs guides?  Thanks in advance!
Answer: Migrating from SBS to a non-sbs platform is not as straight forward as an SBS to SBS migration due to the lack of Exchange.
If you perform a standard swing migration by joining the 2008 server to the domain, seizing the roles, and demoting the SBS you will have lots of exchange errors as those parts of the schema cannot copy over.
You could also use the ADMT 3.1 to transfer the roles, but again, dragging the SBS schema to the new server will cause issues unless you remove all of the exchange attributes beforehand.
You can tackle this in one of 3 ways:
.#1: Do the normal swing and clean up the mess after - running ntdsutil and doing a metadata cleanup on that server
.#2: Use the ADMT tool and Adsiedit to remove all the exchange attributes before migrating. This might work, but you will still probably have to perform cleanup as well.
.#3 Start from scratch, export AD user info with Csvde to anexcel file, build the new server, and reimport. At least that will get your users info.
Either way, good luck :-)
Comment: I think option 2 sounds like it may be the easier way to go.  How do I go about removing the exchange attribs?
Comment: Here is the link to the Adsiedit page, you will find what you need here:

http://technet.microsoft.com/en-us/library/cc773354(WS.10).aspx
Answer: I am just getting ready to go through the process, but two things that I found were helpful were these two links.
http://demazter.wordpress.com/2010/04/29/migrate-small-business-server-2003-to-exchange-2010-and-windows-2008-r2/
http://www.kdbsystems.ca/?p=60
They give step by step instructions for moving to server 2008 r2 WITH exchange 2010.
Answer: You did not say if you were also migrating to Exchange on the same new server, or to a second seperate server, or at all. Either way, the swing migration is certainly the way to go.
If you do not intend to migrate Exchange, you should probably migrate your data to your new system, and then remove it first following this document: http://technet.microsoft.com/en-us/library/bb288905(EXCHG.80).aspx.
I do not believe that the errors the other guys references with regards to Exchange are a factor. When you add the new 2008 server as a domain controller, the current schema (the tables and fields in a database), including for Exchange 2003, is copied to that server, and you will even be making it the schema master. So all of the fields, including for Exchange, which are in the current schema will still be there on the new server even if Exchange is not in use, and that is nothing you can do anything about: You can not un-extend a schema. So whether you upgrade Exchange to the new 2008/2010, when you will further extend the schema; or if you remove Exchange 2003 totally, all that will happen is that you have empty fields in the database (Schema).
Comment: I apologize for not giving the full picture.  We are not using the exchange portion of SBS and we are going to a new physical server.  Let me see if I understand you correctly.  I should be able to go ahead with a swing migration and just ignore the parts of exchange that were moved over?

Migrate SBS 2003 to Server 2008 Std


mysql select latest record only for record

Question: I have a MySQL database with the following structure:
custodian | counta | countc | countc | total | date
-------------------------------------------------------
ed        | 1      | 2      | 3      | 6     | 1/1/2016
ed        | 2      | 3      | 5      | 10    | 1/2/2016
ed        | 2      | 3      | 6      | 11    | 1/3/2016
ed        | 1      | 3      | 5      | 9     | 1/4/2016
fred      | 1      | 2      | 3      | 6     | 1/1/2016
fred      | 2      | 3      | 5      | 10    | 1/2/2016
fred      | 2      | 3      | 6      | 11    | 1/3/2016
fred      | 1      | 3      | 5      | 9     | 1/4/2016

How do I return the latest record for a custodian? I've been playing around with this condition where date >= DATE_SUB(NOW(),INTERVAL 59 MINUTE) since the table is updated hourly, but if I update the script twice in an hour, I would return more than one result per custodian.
Any advice?
Answer: You need to combine ORDER BY and LIMIT:
SELECT *
FROM yourTableName
WHERE custodian = 123
ORDER BY `date` DESC
LIMIT 1

Comment: Just got in before me! On a related note, I'm suprised that `LIMIT` isn't syntax highlighted.
Answer: You could try this
SELECT * FROM tbl ORDER BY date DESC LIMIT 1

The most recent date will be the first record when ordered decendingly, and limiting the select to 1 means you get only the latest record.


Criação de Programa em Python 3

Question: Eu estou fazendo um exercício:

Uma fábrica tem 10 representantes. Cada um recebe uma comissão calculada a partir do número de itens de um pedido, segundo os seguintes critérios:
  – para até 19 itens vendidos, a comissão é de 10% do valor total do pedido;
  – para pedidos de 20 e 49 itens, a comissão é de 15% do valor total do pedido;
  – para pedidos de 50 a 74 itens, a comissão é de 20% do valor total do pedido; e
  – para pedidos iguais ou superiores, a 75 itens a comissão é de 25%.
Faça um programa que lê a quantidade de itens de pedidos de cada representante e imprime o percentual de comissão de cada um.

Eu pensei em criar uma variável para cada entrada, ou seja, uma variável para cada representante e no final para gerar as comissões colocando vários if elif e else. A minha pergunta é com deixar esse código mais enxuto sem eu ter que criar as 10 variáveis?
Mais ou menos eu fiz esse pedaço de código que me da as 10 entradas. Só que daqui para a frente eu me enrolei na hora de criar os condicionais para gerar o valor de cada comissão de cada representante.
Meu esboço de código, com as entradas:
print ("\n--------------------------------------------------------")
print ("\n--------------------------------------------------------")

def exercicio():
    itens = []
    item = 0

    while item == 0:
        item = float(input("\nQuantidade de Itens do 1º Representante: "))
        if item < 0:
            print("\nAs Quantidades não podem ser menores que Zero.")
            itens.append(item)

    for i in range(9):
        item = float(input("\nQuantidade de Itens do %dº Representante: " % (i+2)))
        itens.append(item)

   print("\n",item)
   print ("\n",itens)

exercicio()

print ("\n--------------------------------------------------------")
print ("\n--------------------------------------------------------")

Answer: Acho que você se perdeu na lógica do seu código. Você imprime uma mensagem de que valores menores que 0 são inválidos, mas o armazena mesmo assim. Além disso a mensagem de erro só é imprimida se o primeiro valor for inválido, e ignora todos os demais.
A quantidade poderia ser tratada dessa maneira
for i in range(10):
    item = int(input("\nQuantidade de itens do %dº representante: " % (i+1)))
    while item < 0:
        item = int(input("\nA quantidade não pode ser menor que zero, digite novamente: "))

E as comissões poderiam ser guardadas num array, assim como a quantidade de itens vendidos
comissoes = []
for item in itens:
    comissao = '10%' if item < 20 else \
               '15%' if item < 50 else \
               '20%' if item < 75 else \
               '25%'
    comissoes.append(comissao)

Comment: Poxa. Muito Obrigado Ajudou Muito.

Criação de Programa em Python 3


What's the difference between stacked LSTM and encoder-decoder LSTM

Question: I wanted to learn about encoder-decoder LSTM and after some digging around I get that the first LSTM layer in an encoder-decoder-LSTM outputs its hidden state and then the next LSTM layer uses that hidden state as its initial hidden state, I get this bit but what I don't understand is that what's the difference between this and stacked LSTM in terms of code?
the code for a normal stacked LSTM like the one I'm currently using for a time series forecasting problem is:
<code>lstm = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(16, return_sequences=True),
    tf.keras.layers.LSTM(24, return_sequences=True),
    tf.keras.layers.Dense(units=1)
])
</code>
now what I dont understand is that what do I change here to make this an encoder decoder?
some articles I read added something called a RepeatVector layer however that went be necessary for me since im returning sequences on the first layer right..?
I apologise if this is a naive question, im new to DL and LSTMs
Answer: Good question. Yes, to turn this into an encoder-decoder, you need to turn off return_sequences in the first LSTM. This will create a 2D output, and so you need to use a RepeatVector(sequence_length) layer after this to convert to a 3D shape, so it can be passed onto the second LSTM. The rest of your network you can leave as is.
<code>    lstm = tf.keras.models.Sequential([
        tf.keras.layers.LSTM(16),
        tf.keras.layers.RepeatVector(sequence_length),
        tf.keras.layers.LSTM(24, return_sequences=True),
        tf.keras.layers.Dense(units=1)
    ])
</code>
I believe both the stacked LSTM and the encoder-decoder LSTM should both run fine for seq2seq problems where the inputs and outputs are of the same sequence length. But only the encoder-decoder LSTM will run when the input and output sequences are of different lengths. Note that combining stacked LSTMs and encoder-decoder might improve your results further.
Comment: Thank you for the answer! I never actually ended up figuring it out but still got an A. This is super useful though I'm glad I get it now. (this was for a senior HS project)  how come you found this question after so long?


How the Markov Chain Monte Carlo ensure the stationary distribution converge to the target distribution?

Question: I am reading about the MCMC but now I got a lot of questions.
Firstly, It says we could construct a markov chain which satisfy the detailed balance:p(z)T(z,z')=p(z')T(z',z) and we could sample from the proposal distribution T after we converge to the stationary distribution p(z). But why shouldn't we sample from p(z) but from the T(z,z')?
Secondly, why after we constructed this markov chain, it will ensure converge to the target distribution we want but not some other stationary distribution? The book says it satisfy the aperiodic and ergodicity. How the two properties make sure the stationary distribution is the distribution we want?
I am getting stuck here for several days. Thank you for somebody helping me out.
Answer: First - of course we should sample from p(z), but in some cases it is impossible and that is where MCMC is applicable.
Second - the correct stationary distribution is ensured by detailed balance condition. If further conditions are met (in simple case of finite state space it is enough that state is aperiodic and irreducible) the stationary distribution is unique, moreover regardless of the starting distribution there will be convergence to stationary distribution. It is quite nicely explained in Wikipedia https://en.wikipedia.org/wiki/Markov_chain#Time-homogeneous_Markov_chain_with_a_finite_state_space


Composite Variables

Question: I am getting a little confused with the idea of composite variables. If you have a bunch of (hyptertheical) features for house price regression such as #rooms, location, proximity to school, house area and yard area. I know you could combine some of these into composite variables, such as total house area (Yard + house) or maybe some empirical classification which takes house area, number of rooms and proximity to school, applies ratings to the input values and returns a number.
I know, in some situations such variables may improve regressor performance (as mentioned by Andrew Ng in one of his excellent videos on ML Regression).
However, I assume you would then not include the original features if you use the compound features? Does it matter how the composite feature was calculated or relates the independent features? I have been trying to find some references for this problem, but I have yet to find any. Can anyone else point me in the right direction?
Thank you
BJR.
Answer: Other than references, I'd suggest some thought experiments to convince yourself:

However, I assume you would then not include the original features if you use the compound features?

No. Consider you generate random data from model $y=x_1+x_2+x_1x_2$, you pretend as if you don't know the actual model, but have $x_1$ and $x_2$ as your features. You decided to apply linear regression and somehow you decided to use $x_1x_2$ as an extra feature. If you remove the original features, your model assumption will be $y=ax_1x_2+b$, in which you won't be able to fit as good as the model $y=ax_1+bx_2+cx_1x_2+d$.

Does it matter how the composite feature was calculated or relates the independent features?

Yes, in the above example, imagine generating your composite feature as $x_1/x_2$ instead of $x_1x_2$. Would you still think you could catch up with the original model? Or, assume you generated a probably worse feature: $(x_1/x_2)^{0.000001}$, would you catch up as good as the above one?
A note: If you have yard area, $x_1$, house area: $x_2$, and create total area: $x_3=x_1+x_2$ as a new feature, your original model assumption $y=ax_1+bx_2+c$ doesn't really change:
$y=ax_1+bx_2+cx_3+d=ax_1+bx_2+c(x_1+x_2)+d=kx_1+lx_2+d$, i.e. still same features. (Weighted) Summing of features do not add extra capacity to linear regression.

Composite Variables


How can I pass a value from one constructor to another constructor?

Question: I'm working with Java. I have a class with 2 constructors. The first constructor takes an int value as a parameter and sets an int variable as that value. The second constructor takes a string and prints it out. The idea is that when I call the first constructor from my main class, it sets an integer value. And when I call the second constructor in the main class, it takes the string representation of int variable of the first constructor and prints it out.
Here's how I made the constructors:
public class Test
{
   int val;

   public Test(int x)
   {
      val = x;
      return val; //I know this won't work. So I'm looking for an alternative
   }

   public Test(String y)
   {
      System.out.println("The value is " + y);
   }
}

And the main method (in a different class) looks like this:
public static void main(String [] args)
    {
        Test t1 = new Test(6);
        Test t2 = new Test(String.valueOf(t1)); //This won't work because the first constructor can't return a value
    }

So how exactly can I change the contents of the constructors so that I can pass val into the 2nd constructor?
Comment: `public Test(int x) { this(String.valueOf(x)); this.val = x; }`
Comment: + from @ernest_k a constructor can't return values.
Comment: I don't think this is a duplicate -- the OP wants to create `t1`, and then use the value in a subsequent call.
Comment: `int six = 6; Test t1 = new Test(six); Test t2 = new Test(String.valueOf(six)); `
Comment: Not a dupe.  The dupe is a symptom but there are *other* problems.
Comment: Possible duplicate of [How do I call one constructor from another in Java?](https://stackoverflow.com/questions/285177/how-do-i-call-one-constructor-from-another-in-java)
Comment: @fantaghirocco:  We've already established that this isn't a dupe of that.
Answer: Override toString() to return value so when you so String.valueOf(t1) it will do the toString() method;
public class Test
{
   int val;

   public Test(int x)
   {
      val = x;
   }

   public Test(String y)
   {
      System.out.println("The value is " + y);
   }

   @Override
   public String toString()
   {
      return String.valueOf(val);
   }
}

Comment: Didn't see this one, we said the same thing.
Answer: I think what you are probably actually trying to do is to override the toString() method of Test.
public class Test
{
   int val;

   public Test(int x)
   {
      val = x;
   }

   @Override
   public String toString() {
       return "Test:"+val;
   }
}

Then you can do this:
public static void main(String [] args)
    {
        Test t1 = new Test(6);
        String s = t1.toString();
        // or this
        System.out.println( t1 ); // prints "Test: 6"
    }

Answer: What you're describing is actually impossible without some changes.
First and foremost, t1 and t2 are two separate instances and the values inside of them have no bearing on one another.  So t1 has x=6 and t2 has x=0 (because of default values).
If you want your second constructor to have a value of x that isn't 0, then you'll need to pass that in too.
public Test(int x, String s) {
    super(x);
    System.out.println(x);
}

Answer: I think you don't really want two constructors. It seems like you're wanting to do something like the following:
public class Test
{
   int val;

   public Test(int x)
   {
      val = x;
   }

   public void printVal()
   {
      System.out.println("The value is " + val);
   }
   public static void main(String [] args)
   {
      Test t1 = new Test(6);
      t1.printVal();
   }
}

Comment: I would not use `printVal()`, but instead override the `toString()` method. In that case, you can do `System.out.println(t1.toString())` to print the value (including the message) to the screen.
Comment: Yes, I agree. Answer by @markspace is a better end result. I was trying to stay closer to the original code.
Answer: Your requirement is kinda weird. But this will work even it is kinda weird
public class Test {
    private static int val;

    public Test(int x) {
        val = x;
    }

    public Test() {
        System.out.println("The value is " + String.valueOf(val));
    }

    public static void main(String[] args) {
        Test t1 = new Test(6);
        Test t2 = new Test();
    }
}


the shortest path through the entire point in Kotlin

Question: I am a beginner in kotlin, I am developing an algo in kotlin.
My goal is to find the shortest path through all the points.
For the moment my algo is trying to find the shortest distance from one point to another but I can't improve it.
NOTE: a MapDataPoint is an object with coordinate(latitude and longitude), and a a status represented by string
<code> fun findClosestPoint(latLng: LatLng, points: List<MapDataPoint>): MapDataPoint {

        var latestClosestPoint: MapDataPoint? = null
        points.forEach { latestPoint->
            if (latestPoint.status == "present") {
                latestClosestPoint = latestPoint
                points.forEach { point ->
                    if (point.status == "present") {
                        val differenceLatestLng: Double = kotlin.math.abs(latestClosestPoint!!.lng - latLng.longitude)
                        val differenceLatestLat: Double = kotlin.math.abs(latestClosestPoint!!.lat - latLng.latitude)
                        val differenceActuelLng: Double = kotlin.math.abs(point.lng - latLng.longitude)
                        val differenceActuelLat: Double = kotlin.math.abs(point.lat - latLng.latitude)
                        if (differenceActuelLat < differenceLatestLat && differenceActuelLng < differenceLatestLng) {
                            latestClosestPoint = point
                        }
                    }
                }
                return latestClosestPoint!!
            }
        }
        return latestClosestPoint!!
}
</code>
<code>fun displayEpurationPath(mapDatapoint: List<MapDataPoint>) {

            val locationComponent = mapboxMap?.locationComponent?.lastKnownLocation
            val latLng = LatLng(locationComponent!!.latitude, locationComponent.longitude)
            var newCoordinate: LatLng? = null
            val points: MutableList<MapDataPoint> = mapDatapoint.toMutableList()
            var sortedList: MutableList<MapDataPoint> = mutableListOf()
            val list: MutableList<MapDataPoint> = emptyList<MapDataPoint>().toMutableList()

            while (points.isNotEmpty() && !manageStatusListPoint(points)) {
                val point: MapDataPoint = if (newCoordinate == null) {
                    findClosestPoint(latLng, points)
                } else {
                     findClosestPoint(newCoordinate, points)
                }
                newCoordinate = LatLng(point.lat, point.lng)
                sortedList.add(point)
                points.remove(point)

}
</code>
this is the graphic representation, we can see that it is not the shortest path
Comment: There are well known (and rather old) algorithms for solving this; did you try looking for them?
Comment: Of course I searched before asking my question, that's what I did. Would you have an algorithm to advise me?
Answer: your problem is a variant of the traveling salesman problem. You just do not want to return to the start point at the end.
I must inform you that there is no 'fast' algorithm known to solve this optimally.
But a lot of methods to find good solutions quite fast.
Common ways to find solutions is backtracking, genetic algorithms, ...
Comment: Yes, that's exactly what I'm trying to do, thank you very much for your help, but I've done a search on the internet for examples, tracking is generally used in the resolution of mazes and I don't see how to adapt it to the GPS coordinates of my points.
Comment: The only part of this that's specific to GPS co-ordinates is being able to calculate the distance between any two of your points.  Assuming you can do that, the rest is a completely general TSP, and so _any_ TSP algorithm should be usable.
Comment: Your points are simply vertices in a fully-connected graph.
Comment: yes that's right all the points are connected to each other, I created a class that contains two points with a distance to them. Then I create a list of this object which corresponds to all the possible connections
Comment: The point is that is EXACTLY what the Travelling Salesman Problem solves.
Comment: Thank you very much, but do you have any idea how I can implement the travelling salesman algo or a site that could help me to do so ? I've done my research and none of them take into account the GPS coordinates
Comment: I followed this tutorial: https://www.geeksforgeeks.org/traveling-salesman-problem-tsp-implementation/
 but I'm having a little trouble initializing my graph for the distance calculation I used "TurfMeasurement.distance".


How to get UIWebView page number

Question: I used following code to paginate the UIWebView.
self.webView.paginationMode = UIWebPaginationModeLeftToRight;
self.webView.paginationBreakingMode = UIWebPaginationBreakingModePage;
self.webView.scrollView.pagingEnabled = YES;

Now I want to know the total number of pages it created. The IOS7 document says UIWebView has a property called pageCount. I tried it, but it always returns 1.
How can I get the number of pages it created after pagination?
Thanks.
Answer: If you want only the number of pages you can use CoreGraphics
CGPDFDocumentRef pdf = CGPDFDocumentCreateWithURL((CFURLRef)[NSURL fileURLWithPath:@"your path"]);
int pageCount = CGPDFDocumentGetNumberOfPages(pdf);
CGPDFDocumentRelease(pdf);

If you want to count pages for HTML content you need to set delegate
self.webView.delegate = self;

and to implement UIWebViewDelegate method
- (void)webViewDidFinishLoad:(UIWebView *)webView
{
    NSUInteger count1 = webView.pageCount; // for iOS7 only!

    NSUInteger count2 = webView.scrollView.contentSize.width / webView.scrollView.frame.size.width; // for horizontal paging

    NSUInteger count3 = webView.scrollView.contentSize.height / webView.scrollView.frame.size.height; // for vertical paging
}

Note that you can get pageCount only after full rendering of the requested HTML page.
Comment: I am using UIWebView to display HTML instead of PDF. Is there a way to show the total paginated page numbers?
Comment: Thanks. This works. I also notice I can put pageCount inside `viewDidAppear` and it also works.
Comment: Exactly. Rendering is done by the time of viewDidAppear. But if you want to load pages by some user request you need delegates.
Comment: Note that you can set the pageLength property to split in multiple columns
Comment: @malex: Will this work for document as well like word, excel which i am displaying in UIWebView. I am looking for Pagecount and horizontal/Vertical scrolling of these documents.
Comment: @ArunGupta, it should work for any web content. Anyway It's better to try.
Comment: @Malex: I tried for different documents and pagecount (webView.scrollView.contentSize.height / webView.scrollView.frame.size.height) property seems to be working in case of vertical scrolling.  It is possible to get the current page or page number when scrolling.
Answer: UIWebView is a subclass of UIScrollView. You can detect UIWebView page change event for scroll view delegate method:
- (void)scrollViewDidScroll:(UIScrollView *)scrollView

This delegate will be called each time the webView scrolls. Maintain your page number yourself. You can use scrollView.contentOffset.x and scrollView.contentOffset.y to determine the scrolling direction.
You can find more help here.
EDIT
We can find page number of scrollview from contentoffset. Try something similar for webView.
    int page = scrollView.contentOffset.x / scrollView.frame.size.width;
Comment: I need to know the total number of pages before I scroll the page. I need to pass this value to the PageControl display the dots. In fact, I used similar function to get page `self.pageControl.numberOfPages = self.webView.scrollView.contentSize.width / pageWidth;`, but the `scrollView.contentSize.width` only shows one page width. I would expect it shows the width of all pages.
Comment: Did you tried scrollView.contentOffset.x inted of contentSize.width??

How to get UIWebView page number