Proof that $M(a)$ is a regular domain in $\mathrm{Int} M$ in Prof. Lee's Introduction to Smooth Manifolds errata for Theorem 9.26

Question: In Professor John Lee's Introduction to Smooth Manifolds, Second Edition,
Theorem 9.26 has, as its hypothesis, a smooth manifold with nonempty boundary $M$.
In his errata for the book, there is an entry (dated 8/19/14) for the proof of
this theorem:

There's a gap in this proof, because it is not necessarily the case that $M(a)$
is a regular domain in $\mathrm{Int}M$. To correct the problem, we have to choose
our collar neighborhood more carefully. Replace the first sentence of the proof
by the following paragraph:
"Theorem 9.25 shows that $\partial M$ has a collar neighborhood $C_0$ in $M$,
which is the image of a smooth embedding $E_0\colon[0,1)\times\partial M\to M$
satisfying $E_0(0,x)=x$ for all $x\in\partial M$. Let $f\colon M\to\mathbb{R}^+$
be a smooth positive exhaustion function. Note that $W=\{(t,x)\colon f(E_0(t,x))>f(x)-1\}$ is an open subset of $[0,1)\times\partial M$ containing
$\{0\}\times\partial M$. Using a partition of unity as in the proof of
Theorem 9.20, we may construct a smooth positive function
$\delta\colon\partial M\to\mathbb{R}$ such that $(t,x)\in W$ whenever
$0\leq t<\delta(x)$. Define $E\colon[0,1)\times\partial M\to M$ by
$E(t,x)=E_0(t\delta(x),x)$. Then $E$ is a diffeomorphism onto a collar
neighborhood $C$ of $\partial M$, and by construction
$f(E(t,x))>f(x)-1$ for all $(t,x)\in[0,1)\times\partial M$. We will show that
for each $a\in(0,1)$, the set $E([0,a]\times\partial M)$ is closed in $M$.
Suppose $p$ is a boundary point of $E([0,a]\times\partial M)$ in $M$; then there
is a sequence $\{(t_i,x_i)\}$ in $[0,a]\times\partial M$ such that
$E(t_i,x_i)\to p\in M$. Then $f(E(t_i,x_i))$ remains bounded, and thus
$f(x_i)<f(E(t_i,x_i))+1$ also remains bounded. Since $\partial M$ is closed
in $M$, $f|_{\partial M}$ is also an exhaustion function, and therefore the
sequence $\{x_i\}$ lies in some compact subset of $\partial M$. Passing to a
subsequence, we may assume $(t_i,x_i)\to(t_0,x_0)$, and therefore
$p=E(t_0,x_0)\in E([0,a]\times\partial M)$."
Then at the end of the first paragraph of the proof, add the following
sentences:
"To see that $M(a)$ is a regular domain, note first that it is closed in $M$
because it is the complement of the open set $C(a)$. Let $p\in M(a)$ be
arbitrary. If $p\notin E([0,a]\times\partial M)$, then $p$ has a neighborhood
in $\mathrm{Int}M$ contained in $M(a)$ by the argument above. If
$p\in E([0,a]\times\partial M)$, then $p=E(a,x)$ for some $x\in\partial M$,
and $C$ is a neighborhood of $p$ in which $M(a)\cap C$ is the diffeomorphic
image of $[a,1)\times\partial M$."

In between the above two paragraphs we have this sentence:

For any $a\in(0,1)$,
let $C(a)=\{(s,x)\in[0,1)\times\partial M\colon 0\leq s<a\}$ and
$M(a)=M\setminus E(C(a))$.

where I have taken the liberty of undoing the author's effort to "simplify the
notation" by identifying $C$ with $[0,1)\times\partial M$.
I understand and can verify everything stated in the errata. I am having trouble,
though, proving that $M(a)$ is a regular domain in $\mathrm{Int}M$. By the
errata paragraph beginning "To see that $M(a)$ is a regular domain", it appears
that it was his intention to use Theorem 5.51 to show that $M(a)$ is an
embedded codimension-$0$ submanifold with boundary of $\mathrm{Int}M$. Indeed,
in the first case, where $p\in M(a)$ is not in $E([0,a]\times\partial M)$, and
we have the $\mathrm{Int}M$-neighborhood $V$ of $p$, which is contained in
$M(a)$, we can find a smooth chart $(U,\phi)$ at $p$ for $\mathrm{Int}M$ such
that $U\subseteq V$, and then $M(a)\cap U=U$ so
$$\phi(M(a)\cap U)=\phi(U)$$
so that $M(a)\cap U$ is an $n$-slice of U$.
The problem arises in the second case, where $p=E(a,x)$. All my attempts at
finding a smooth chart $(W,\lambda)$ at $p$ for $\mathrm{Int}M$ such that
$\lambda(M(a)\cap W)=\lambda(W)$ or $\lambda(W)\cap\mathbb{H}^n$ have failed.
From the
last sentence of the errata, I was supposed to leverage the fact that
$E([a,1)\times\partial M)=M(a)\cap C$. Here is an example of one of my failed attempts:
Let $E_C$ denote the diffeomorphism $E_C\colon[0,1)\times\partial M\to C$,
($E$ with its codomain restricted to $C$) where $C$ has been given the unique smooth structure to make $E_C$ a
diffeomorphism.
Let $(U,\phi)$ be a smooth chart at $x$ for $\partial M$.
$(0,1)\times U$ is open in $[0,1)\times\partial M$,
so by Proposition 2.15(d), the restriction of $E_C$,
$\tilde{E}_C\colon(0,1)\times U\to E_C((0,1)\times U)$, is also a
diffeomorphism. Let $\Lambda\colon\mathbb{R}\times\mathbb{R}^{n-1}\to\mathbb{R}^{n-1}\times\mathbb{R}$ be the diffeomorphism which exchanges
its arguments. Let
$W=E_C((0,1)\times U))\text{ open}\subseteq C\text{ open}\subseteq M$.
Since $(\{0\}\times\partial M)\cap((0,1)\times U)=\varnothing$,
$W\text{ open}\subseteq\mathrm{Int}M$. Also, let
$$\lambda=\Lambda\circ(\mathrm{id}_{(0,1)}\times\phi)\circ\tilde{E}_C^{-1}\colon W\to\phi(U)\times(0,1)\text{ open}\subseteq\mathbb{R}^n.$$
Then $\lambda$ is a diffeomorphism, so $(W,\lambda)$ is a smooth chart for
$\mathrm{Int}M$. Crucially, $p\in W$ since $p=E(a,x)$ and
$(a,x)\in(0,1)\times U$. Then
$$M(a)\cap W=(M\setminus E([0,a)\times\partial M))\cap E((0,1)\times U)
=E([a,1)\times U),$$
$$\lambda(M(a)\cap W)=\lambda(E([a,1)\times U)=\phi(U)\times[a,1),\text{ and}$$
$$\lambda(W)=\lambda(E((0,1)\times U))=\phi(U)\times(0,1),$$
so we don't quite have $\lambda(M(a)\cap W)=\lambda(W)$ or $\lambda(W)\cap\mathbb{H}^n$. If I try to modify this by shrinking the left end of $(0,1)$
towards $a$, I can't go past $a$ since then $p=E(a,x)$ will not be in $W$, and
I can't use $[a,1)$ because that is not open in $[0,1)$. I've tried to use
a diffeomorphism from $[a,1)$ to $[0,1)$, but have not been able to get that to
work either.
I'd really appreciate some help in proving that $M(a)$ is a codimension-$0$
embedded submanifold with boundary of $\mathrm{Int}M$.
Answer: Oh, I just figured it out! Let $\tau_a\colon\mathbb{R}\to\mathbb{R}$
be defined by $\tau_a(r)=r-a$. $\tau_a$ is a diffeomorphism. Let
$W$ be defined as in the post and let
$$\lambda=\Lambda\circ(\tau_a\times\phi)\circ\tilde{E}_C
\colon W\to\phi(U)\times(-a,1-a)\text{ open}\subseteq\mathbb{R}^n.$$
Then
$$\lambda(M(a)\cap W)=\lambda(E([a,1)\times U)=\phi(U)\times[0,1-a),\text{ and}$$
$$\lambda(W)\cap\mathbb{H}^n=\lambda(E((0,1)\times U))\cap\mathbb{H}^n=(\phi(U)\times(-a,1-a))\cap\mathbb{H}^n
=\phi(U)\times[0,1-a).$$
So $\lambda(M(a)\cap W)=\lambda(W)\cap\mathbb{H}^n$ as required.


Configure Munin graphs to show different time periods

Question: I'm running a server with Munin installed through WHM cPanel. I've been googling around and I can't seem to find out how to do the following:
A: Change a graph to show a different time period (currently I can only view, day, week, month, year). I would like to possibly view per minute, per hour... or specific intervals such as per 30 minutes, per 4 hours, etc...
B(optional, but preferred): Add new graphs with the above criteria(A).
I am a novice with all things that will possibly be required to do this (such as ssh). I wish I could just go edit a php file as I'm highly experienced in that subject.
Answer: Here is documentation on the munin capabilities to change the sample periods:
Keep more data with custom retention plans
It depends on which version of munin are you are using.  1.4 has some limited ability to store higher precision data, but what you really want is what will be in 2.0 when it is released.  The second alpha was released around a month ago, so it'll likely be a while before it's out and packaged for your use.
Comment: I noticed this. I also noticed in the faq doesn't say much (or isn't specific enough for me to understand) about customizing graphs. The stuff they had was vague on where to even put it... I was also disappointed that there wasn't a forum I could join. There's a mailing list, but I'm very unfamiliar with those and the ones I've seen are annoying to follow. I figured I could go edit a config file through ssh and at least change the time periods of the current graphs.
Comment: My blog post at http://www.systemajik.com/blog/monitoring-with-munin/ has some override examples.  These were added to the configuration of the individual servers. Depending on where they are in the file they override at different levels, server, group, or site.  Time periods are fixed as they tend to be in such monitoring software.  Same as for the versions of n2rrd and mrtg I have worked with.
Comment: FYI: The last couple of days I've been playing with Zabbix, and it supports setting different intervals for different services and intervals other than 5 minutes.  The default is 5 seconds for the network graphs and 300 seconds for system uptime...  The thing that it's missing from Munin is a default "all graphs" screen, so it takes a little more setup via the "Screens" configuration to make something like that.  However, it can also combine Nagios-like monitoring functionality.  Might be worth a look.
Answer: The code that builds the images is in Perl.  I believe you will find period shorter than 1 day will give disappointing results.  Sample period is fixed at 5 minutes.  Per hour would only have 12 samples.
The rrdtool package should be installed so you can try to generating the images using rrdgraph.  Some of the rrd configuration comes from the plugins.  Telnet to localhost 4949 to talk to the plugins.  The node software has a help command.  The rrd data files can be found under /var/lib/munin.
The munin projects site has some help on generating graphs.
EDIT: The information you need to generate a graph or data dump is available from the rrd data files.  The structure and file naming under /var/lib/munin is pretty easy to figure out.  Directory tree matches your domain structure.  File names match plugins on hosts.
Munin-node will give you the information it uses to format a graph for a particular plugin.  The command telnet localhost 4949 will connect you to the munin node software on the local host.  Once connected the commands most useful for your use are:

help - List commands available
list - Lists configured plugins
config cpu - Lists rrd formatting information for cpu plugin.  Substitute your desired plugin for cpu.  This is not required to generate the graph, only to match the formatting used by munin.  Formatting can be overridden in the munin.conf file.
Comment: I have to disagree with the value of the results. Even though it only updates every 5 minutes, my graphs still show a great deal of detail. I would just like to ZOOM in on smaller times. As for the rest of what you said, that is beyond my ability right now unless someone has a quick guide?
Comment: I can ssh and edit text files fine, just don't know the right files to edit/etc. Munin's doc is written in a way that you would have to have a decent understanding of how munin is setup already.
Comment: The man pages for the rrdtools should allow you to quickly generate a graph.  If you want to zoom, you may want to look at the raw datapoints, which you can get from rrdtools as well.  Once you have the graph generation or dump working, it should be trivial to generate a page in PHP to show the output.
Comment: I would get the zooming functionality working first, then look at adding it in to the munin code.  It would need generate graphs dynamically.  I believe the daily graph gives one pixel width per data point in the raw data.  You may find that zooming the image gives you what you are looking for.


PowerChute appliance 4.4.1 for ESXi issues 302 redirect to localhost after login

Question: I've been using PowerChute 4.3 network appliance on my ESXi host for a while now. I followed the instructions to download and install the appliance update to 4.4.1 this morning using the official tar.gz file. The update appeared to go well.
Now when I browse to powerchute:6547 it requests my login as usual. After I login the appliance issues a 302 redirect to localhost:6547/index.html. Obviously that doesn't work because localhost is my computer. If I try to browse to the appliance in a separate tab it just immediately redirects to localhost because I'm already logged in. I did try rebooting the appliance to see if that would help but it did not.
Any idea how to fix this or do I need to delete the appliance, install a fresh copy, configure it, and commit to never doing it by an in-place upgrade again no matter how easy it seems like it should be.
Edit: Setting up a new 4.4.1 appliance I notice that the configuration asks for a hostname. Maybe the original 4.3 has the hostname unset or set for 'localhost'. If I knew where that setting is held I could SSH into the vm and update that setting.
Answer: Did you find a solution for this?  I'm way behind, but finding the same issue.  Looking for the settings to provide PS.  4.3 didn't have this issue, assuming it's new for the version.
Comment: Actually I never did get it working and had totally forgotten that it's broken. Thanks for the reminder. By the way, questions about a question should be posted as a comment, not as an answer.
Comment: I think I ultimately reinstalled the VM appliance but I never finished configuring it. I just completed the configuration and it seems to be running fine now. Since I had upgraded from 4.3 it still had enough configuration that it's been protecting my servers anyway, that's why I had not noticed that I never really finished the upgrade.


Changing Attribute Onclick Javascript

Question: Is there a special way to modify the attribute onclick?
    var btn = document.getElementById("deletebutton" + j);
    btn.id='deletebutton'+mreplace;
    btn.onclick = 'deleteDiv('+mreplace+')';

The id will change but the onclick remains constant.
I'm trying to switch out some Id's and data to reconstruct a page after an element has been deleted from the page. In this case I have 7 divs, when i erase say the 3rd one, I want to modify the other divs so that their text is updated--- This i have accomplished --- now the problem is just updating the onclick function of the delete button, so it will delete the correct div when clicked.
full function being called.
function deleteDiv(id) {

    var div = document.getElementById("idBox" + id);
    div.parentNode.removeChild(div);

    // If countbox > id

    if (countBox() > id) {

        var j = countBox();

        while (j > id) {

            var div = document.getElementById("idBox" + j);
            var mreplace = j-1;
            var replace = j+1;
            var html = div.innerHTML;

            var curInnerHTML = html.replace("Question "+replace,"Question "+j);
            div.innerHTML = curInnerHTML;

            div.id='idBox'+mreplace;
            var btn = document.getElementById("deletebutton" + j);
            btn.id='deletebutton'+mreplace;
            console.log(mreplace);
            btn.onclick = 'deleteDiv('+mreplace+');';

            j = j - 1;

        }

    }

}

here is the html generated of one of the divs that Can be deleted

Question 2:

Questijjon # 2

                            <select class="super" name="3" onchange="getCombo(this)">
                                <option value="MC" name="0" data-field="0">
                                    Multiple Choice</option>
                                    <option value="TF">
                                        True/False</option>
                                        <option selected="true" value="INT">
                                            Integer</option>
                            </select>High: 50  Low: 15
                            <form action="" method="post">

                                <input class="deletebutton" id="deletebutton1" value="-" name="delete" onclick="deleteDiv(1);" 1'="">
                            </form>

Comment: *"The id will change but the onclick remains constant."* In what way? How are you checking? Why do you think it hasn't changed?
Comment: Why do you want to use inline events to bind a handler.  Use javascript to bind them.
Comment: chrome element inspecter
Comment: @Sushanth--: Being pedantic, that **is** JavaScript.
Comment: http://puu.sh/7rV0p.png these numbers should be the same, delete and deletebutton
Comment: @user2647561: What you're doing *will* change the event handler, provided the string can be compiled. My guess is that `mreplace`, when substituted into a string, creates a string that doesn't do what you want. We'd need a lot more context to help you with that.
Comment: @user2647561: *"puu.sh/7rV0p.png these numbers should be the same, delete and deletebutton"* Nope, that's not how SO works. Links rot. Put the relevant information **in** the question.
Comment: @T.J.Crowder. What I meant was to avoid mixing javascript with HTML. I used the wrong wordings :)
Comment: @Sushanth--: The above is pure JavaScript. What you meant (I suspect) was to use a DOM2 handler, not a DOM0 handler. And I'm not at all sure it would make a difference, because the question lacks crucial information.
Comment: What more info would you need?
Comment: Why not just do: `btn.onclick = function() { deleteDiv(mreplace) }';`
Comment: It doesnt work with this set-up. So is it impossible to do what I'm trying to do?
Comment: @user2647561: *"What more info would you need?"* Gosh, I don't know. What does `mreplace` contain? What result do you expect? What result do you see instead? What should the button do? What does it do instead? What's in the off-site picture you've linked but not included in the question?
Comment: One sec I'll upload everything then.
Comment: @user2647561: Usually, creating a [small, self-contained, replicating example](http://sscce.org) is the best way to go. Nine times out of ten, you figure out the problem in the process; the tenth time, you have something well-contained that people can reasonably help you with.
Comment: Okay I know, but usually I just mess up syntax or something. I can't find any examples online of people changing the attribute on-click, i thought there might be a special way to do it.
Answer:         btn.onclick = function() {
        deleteDiv(mreplace);
         };

Changing Attribute Onclick Javascript


VAR lag length vs Johansen cointegration test outcome?

Question: First puzzle:
I am taught that the lag order of VECM does not affect the cointegration rank because the lag order is for the differenced regressors. But, I see the contrary: I experimented with sample data of 4 variables, using lag orders between 1 and 12. At low lag orders the cointegration rank is low, but as I increase the lag length, cointegration rank goes up. Does anyone know of a paper or source discussing this issue? Any suggestions as to what to do in this situation?
Second puzzle:
If I go by what information criteria indicate as an optimal lag length, establish the coinegration rank and then set up a VECM model, the residuals of the model are heteroscedastic and serially correlated. The info criteria based lag order is 2. To circumvent the issue, I have two options:

Go back, increase the lag order and set up a new VECM and do diagnostic tests
Just keep the cointegration rank as it is and increase the lag order
until the residuals are homoscedastic and free of autocorrelation

What would be the optimal course of action?
Comment: How did you get lags order for both 4 variable, get lags as a model or lag for each of them? I also didnt quite understand about this. thanks in advance
Answer: This is a usual problem with the two steps procedure, where one selects first the lag, then the cointegration rank depending on the lag chose in the first step.
Puzzle 1: The claim that the lag order does not matter for the cointegration test depends on the type of test:

VAR based test: if you use a Johansen test, the lag matters!
Residual-based test (cf Engle-Granger, Phulips-Ouliaris), then yes, the lag order does not matter

Puzzle 2: My favourite approach is the simultaneous-selection approach, where you choose the lag and rank based on a simlutaneous criterion selection (This is implemented in package tsDyn in R for example, see http://www.inside-r.org/packages/cran/tsDyn/docs/rank.select). See references below.
Now what should you do if the ideal criterion selects a model where there is still some auto-correlation in errors? Great question, and I am not aware of papers tackling this directly. So one option would be to increase lags till you get a model with good residuals. Another option would be to stick with the AIC/BIC choice, and use heteroskedasticity and autocorrelation robust estimators (HAC) for any test you do (t-test, Granger causality, etc). Cheng an Philips (2009) show that their AIC/BIC lag/rank procedure is robust even in presence of auto-correlation (hence the name of semi-parametric). The issue is probably that few softwares will allow you do to this for all tests, in particular, IRF, usually computed with bootstrap, might not include a bootstrap scheme mimicking auto-correlation.

Aznar A and Salvador M (2002). Selecting The Rank Of The Cointegration Space And The Form Of The Intercept Using An Information Criterion. Econometric Theory, 18(04), pp. 926-947. .
Cheng X and Phillips PCB (2009). Semiparametric cointegrating rank selection. Econometrics Journal , 12(s1), pp. S83-S104.
Comment: Thanks. But, what if the info criterion suggested lag order does not remove serial correlation? Should we increase the lag order whilst keeping the rank order from previous step?
Comment: true, I did not discuss this directly. I edited my text for this point!


Implementing form control validation inside custom Angular component

Question: I'm implementing my own components for my Angular project. I'm trying to be smart about implementing validation messages.
My current implementation has multiple boolean inputs for "required", "invalidFormat" and such.
What I want to do is, user form control validations inside my custom inputs. So I want something like this:

      This field is required

And I want that inside the custom component and for multiple validations, such as format, minLength etc.
But I want to do that without having a form control input. My custom input class implements ControlValueAccessor so it works inside a reactive form but I want to know is there any way to access the control that is bound to the component I'm working on?
Also is that considered best practice? It seems to me like the best approach to error handling. I just want to provide formControlName and let the component deal with validation under the hood.
I tried implementing using boolean input fields and it works, but it requires of me to do a lot of code repetition such as this:
<custom-input type="email"
formControlName="email"
[label]="t('email.title')"
[requiredError]="formGroup.controls['email'].errors?.['required']"
[formatError]="formGroup.controls['email'].errors?.['email']"
[required]="true">
</custom-input>

<custom-input type="password"
formControlName="password"
[label]="t('password.title')"
[requiredError]="formGroup.controls['password'].errors?.['required']"
[required]="true">
</custom-input>

And I would like to just pass formControlName and that would be enough for handling validation
Comment: if in your constructor inject the ngControl you have the control: `constructor(private control:NgControl){}`. BTW, I feel that a better aproach use viewProvider like in this [SO](https://stackoverflow.com/questions/74771994/parent-is-not-passing-form-values-to-cva-child-form-group/74801573#74801573). If only want to makes an input with labels, errors, etc it's not necesary implements ControlValueAccessor
Answer: How about passing the FormControl object to your child component like this:
<form [formGroup]="form">
  <custom-input [control]="form.controls['theControlName']">
  </custom-input>
</form>

Then your custom-input component would look like this:
import {Component, Input} from '@angular/core';
import {FormControl} from '@angular/forms';

@Component({
  selector: 'custom-input',
  templateUrl: './input.component.html',
  styleUrls: ['./input.component.scss']
})
export class InputComponent {
  @Input() control: FormControl;
}

And in the template you can check for errors on the FormControl:
<input [formControl]="control">

      This field is required

Comment: That is my second option. This can get ugly especially in formArrays. That's why I was wondering if there's a direct access from the component to the form control.

But I will do as you said if nothing else.


Is $\{0\}$is borel set in $\mathcal{B}(\mathbb{R})?$ yes/no

Question: Is   $\{0\}$is  borel set in   $\mathcal{B}(\mathbb{R})?$
My attempt : since  $\mathcal{B}(\mathbb{R})$ is generated by $(-\infty, a)$ and $(b, \infty )$
so my answer is  No
Comment: You should be more explicit in how **No** follows from your attempt. It's easy to see that some sets that belong to the Borel sets if you're given sets that generate it (those sets, at most countable unions of those sets, at most countable intersections of those sets will belong), but I don't think you can determine easily from a collection of generating sets what ***doesn't*** belong. This aspect of your question is more problematic to me than not being able to solve this problem, by the way (i.e. not recognizing an incorrect/incomplete proof is worse than not being able to give a proof).
Comment: Al open sets and all closed sets are Borel sets. It is hard find  a non-Borel set!
Comment: Regarding the comment by @Kavi Rama Murthy see this [18 January 2007 sci.math post](https://groups.google.com/g/sci.math/c/e-Du6qn-oKs/m/OIYIEjZEbpIJ).
Comment: Notice that $(-\infty,0)$ and $(0,\infty)$ are Borel sets. Therefore so is the complement of their union, which is...?
Comment: @DaveL.Renfro im sorry for that...actually  i was using the definition of Borel set  that  it is  the smallest  $\sigma$ algebra which contain  all of the open set .And here $\{0\}$ is singleton set and singleton set is not open   so  i say **No**
Comment: u mean $A=((-\infty,0) \cup (0,\infty) )^C= 0$ @Bungo $ \bigcup A = \{0\} $
Answer: Hint:
The correct answer is Yes.
To see why, remember $\mathcal{B}$ allows countable intersections of generating sets. Can you write $\{ 0 \}$ as such a countable  intersection? You might want to consider the sequence $\frac{1}{n}$.

I hope this helps ^_^
Comment: I would use $r \in \mathbb{N}$ instead. But yes, that's the idea. This is a countable intersection of open sets (and thus is borel), and is also $\{ 0 \}$.
Comment: u mean  $ \cap_{ r\in \mathbb{Q}} (-1/r,1/r)= \{0\}$ ? @HallaSurvivor  can u elaborate more i am not getting
Comment: okss thanks u got it @HallaSurvivor


Why does a VBA macro that opens another workbook not work when callend from dot.net

Question: When working in BluePrism, I have a problem running an Excel macro that opens another workbook.
The Excel-VBA routine
I have a Sub in a macro enabled excel workbook \\myCompany\myFolder\myMasterbook.xlsm that opens another workbook
Public Sub OpenMyFile(Optional book2open As String = "")
    Dim openedBook As Workbook
    Debug.Print "About to open", book2open

    Set openedBook = Application.Workbooks.Open(book2open)
    If openedBook Is Nothing Then
        Debug.Print "Could not open it"
    Else
        Debug.Print "I found", openedBook.Worksheets(1).Cells(1, 1).Value
        ' Actually, I do a lot more with that workbook, of course
        openedBook.Close
    End If
End Sub

When, in the Immidiate window (which you open with Ctrl-G), I enter
call OpenMyFile ("\\myCompany\myFolder\myWorkbook.xlsx")

I see the workbook open and close and get the response in the Immediate window
About to open   \\myCompany\myFolder\myWorkbook.xlsx
I found   Cell A1 content

Calling the VBA from BluePrism
In a Robotic Automation Process (RPA), using BluePrism, I need to open the workbook that contains this VBA Sub and run it.
In the Immidiate window, I now get
About to open   \\myCompany\myFolder\myWorkbook.xlsx
Could not open it

I tried two things to resolve it
Call without arguments
If I define my Sub in VBA as
Public Sub OpenMyFile(Optional book2open As String = "\\myCompany\myFolder\myWorkbook.xlsx")

and call it with OpenMyFile, but that does not resolve my problem, as I need to pass book2open to VBA.
Make the parameter mandatory
If I define my Sub in VBA as Public Sub OpenMyFile(book2open As String'), I get this error: `Internal : Could not execute code stage because exception thrown by code stage: Cannot run the macro
'OpenMyFile("C:\Users\P01549\Documents\TestHorsten.xlsx", -1)'. The macro may not be available in this workbook or all macros may be disabled.`
The MS Excel VBO - Extended object uses interop
Part of the dot.net code can be found in BluePrisms Object Studio. I run the following "Actions" of this object
Create Instance
Dim excel as Object = CreateObject("Excel.Application")

' Create a GUID with which we can kill the instance later
' if we have to play hardball to get rid of it.
excel.Caption = System.Guid.NewGuid().ToString().ToUpper()

handle = GetHandle(excel)

Open Workbook
Dim wb as Object = GetInstance(handle).Workbooks.Open(filename)
name = wb.Name
wb.Activate()

With

handle : the handle returned by Create Instance
filename : \\myCompany\myFolder\myMasterbook.xlsm

Run Macro
GetInstance(Handle).Run(Macro_Name)

With

handle : the handle returned by Create Instance
Macro_Name : OpenMyFile("\\myCompany\myFolder\myWorkbook.xlsx")

Does anyone have an explanation or a workacound?
Comment: It is the Excel application, and it obviously can, because it does so when called from the imediate window
Comment: Most probably.That is why it is not in the title.
Comment: Ineed, I first instantiate Excel from BluePrism, then open the macro enabled workbook. (Actually I open a macro enabled template.) All this runs some snippets of C# code, including the `GetInstance(Handle).Run(Macro_Name)`
Comment: The problem may be with your Macro_Name input format. If you want to pass path as string parameter within macro name string parameter, then it should look like this: "OpenMyFile(""\\myCompany\myFolder\myWorkbook.xlsx"")".
Comment: I have no idea why it does not work, however you can try some workaround to provide the path. Instead of passing the path through sub parameter, pass it to Excel file with macro (for example to 1st cell of 1st sheet) using Set Cell Value action, then get path from cell in the macro (book2open = ThisWorkbook.Sheets(1).Cells(1, 1).Value) and use it to open the target Workbook.
Comment: Set openedBook = Application.Workbooks.Open(book2open)
When you are running this from C#, what is the application object? Do you think it can open a workbook?
Comment: So, pretty much, the BluePrism is not relevant to the question? E.g. the question could be **how to call the "macro" from C#**? It really seems interesting, thus you may consider edititng it a bit and provide some [mcve] of how/what you tried in a C# console application? Furthermore, you may make your "macro" to just one robust line `MsgBox "Something"` to make sure you can access it from C#.
Comment: The immediate window in the visual basic window in excel? When you have the excel application running? But you are calling your macro from BluePrism via MS Excel VBO - Extended - So is there an instance of Excel running at that point? or is the running application some sort of vb hosting app? Does BluePrism launch Excel?
Comment: Can you not open the workbook directly from Blue Prism? why do you need to open a workbook that opens another workbook?. Also, whats the value of `book2open` when it attempts to open from Blue Prism?
Comment: I have to call that VBA Sub because it not only copies data from the source, it also validates and preprocesses it.
Comment: Indeed, you have to write `"OpenMyFile(""\\myCompany\myFolder\myWorkbook.xlsx"")"` to pass `OpenMyFile("\\myCompany\myFolder\myWorkbook.xlsx")`. That is what I do.
Answer: It is usually the case that Application.Run takes parameters to functions as separate arguments:
This argument is not appropriate for Application.Run:
OpenMyFile("\\myCompany\myFolder\myWorkbook.xlsx")
You can try to amend the VBO to have a run parameter with arguments:
Write another code stage called 'Run Macro With Parameter' (or whatever you want to call it)
GetInstance(Handle).Run(Macro_Name, param1)
where Macro_Name is "OpenMyFile", and param1 is "\\myCompany\myFolder\myWorkbook.xlsx"


Looking for a function with parameters and output on the unit interval...

Question: I'm looking for a function $f(x, y)$ with both parameters on the interval $[0,1]$ such that half of all possible values of $x$ are $<y$, and half are $>y$.
For instance, if I apply an $x$ of $0.25$, I want the function $f(y)$ it produces to be:

$<0.25$ if $y<0.5$
$>0.25$ if $y>0.5$
$=0.25$ if $y=0.5$

Output values should also fall on the interval $[0,1]$.
A smooth function would be best (ideally the graph of any $f(y)$ would be diagonally symmetrical), but I'm also looking for something that's easy to compute.
Comment: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Answer: This appears to be what you want but, if not, please clarify what you require.
Let $f(x,y)=2xy$ if $y\le 0.5$ and $f(x,y)=1-2(1-x)(1-y)$ if $y>0.5$ then
$f(x,y)<x$ if $y<0.5$,
$f(x,y)>x$ if $y>0.5$,
$f(x,y)=x$ if $y=0.5$.
A smooth function that might be of interest is
$f(x,y)=0$ if $xy=0$ and, otherwise, $f(x,y)=y^{-\log_2(x)}.$
Comment: That's a function that does satisfy the criteria, but it's not what I would ideally want, since the change of slope is quite abrupt. I've been playing around with $f(x, y) = 1-(1-y^x)^{(1/x)}$ since it has pretty much the ideal shape, but I can't get it to fit the requirements I outlined above.
Comment: (As an aside, I've also been looking into applying e.g. $arctanh$ to $x$ beforehand for that equation but it doesn't quite work.)
Comment: I've added a smooth function that looks good for $0<x<1$.
Comment: Thank you! That looks close to ideal.


Hypothesis Testing Theory

Question: Ok really simple, but I must be missing something in the theory.
If my $$H_0:p=.5$$
and $$H_1:p>.5$$
What happens if the sample data implies that $p<.5?$ Clearly my $H_0$ would be rejected, but then I would have to accept my $H_1$, which would contradict my known value of $p.$
So what am I missing?
Answer: A null hypothesis is usually state stated in the form of an equality. In the present example, we are testing
(1)$\qquad\qquad$  $H_{0}: p = 0.5$  against  $H_{1}: p > 0.5$.
In this case, even though the null hypothesis is stated as a simple equality, implicitly we are testing that
(2)$\qquad\qquad$  $H_{0}: p \leq 0.5$ against  $H_{1}: p > 0.5$.
Any test procedure that decides between $H_{0}$ and $H_{1}$ in (1), is also a reasonable test for (2).
Answer: If you have only two hypotheses as above, you are sure that no other cases is possible. Sample data cannot implies smth about $p$ with complete confidence. Even if $\overline X<0.001$, this supports the null hypothesis.
But in practice, if you get sample values as in this example, this will be a reason to reconsider the grounds on which these two hypotheses were put forward.

Hypothesis Testing Theory


HTML: upload-form in an other form

Question: I have a little problem with an upload-form within an other form (call it data-form).
I know it is not possible to put a form into an other.
So I would need to put it after my data-form.
But I need the upload-form controls in the middle of my data-form because of optical and structural reasons.
The file-upload should also perform other actions and not the same than the data-form.
So any idea how can I make the upload-form after my data-form but visible in it or any other ideas to handle this?
I am using javascirpt and php also.
thanks and best wishes for 2011!
br,chris
Answer:
I know it is not possible to put a form into an other.

Nesting HTML forms is not valid HTML and results into undefined behavior which could vary between browsers. Here's a similar post.
Answer: You could leave some space in your design and then nest the form (visually) in the other one with CSS and absolute positioning..
Answer: Apart from position: absolute that might work depending on your layout, I had the idea of using a Flash based uploader like SWFUpload. That process works separately from the surrounding form. However, it creates a dependency on Flash and does not degrade gracefully (i.e. if Flash is not present, it does not work at all).

HTML: upload-form in an other form


In Django how to getbinary field data from POST form

Question: I have one HTML file, where I am using one image upload button. Now this image is stored in the MySql database as Blob. I need to get or read this image data somehow in Django through post method. Can anyone please help how to do?
Icon is defined like :
icon = models.BinaryField(null=True)

My Html:
 <input type="file" id="toolicon" accept="image/*" data-type='image'  >
<button id="OpenImgUpload" style="margin-left: 100px">Image Upload</button>

In JQuery:
$('#OpenImgUpload').click(function(){ $('#toolicon').trigger('click'); });

Image:

Now I want to get this file as Binary Field data. Till now I have used :
tool_icon = request.POST('toolicon', '')
tool_icon = request.POST.get('toolicon', '')
tool_icon = base64.b64encode('toolicon', '')

Nothing works ... Can any one please help me.
Answer: Uploaded files are contained in request.FILES with the key corresponding to the name attribute on the input element.
So you should add a name attribute to your input:
<input type="file" name="toolicon" ...

And then access the data using request.FILES:
tool_icon = request.FILES.get('toolicon', '')

The request must have a content type of multipart/form-data which you should set on your form:
<form enctype="multipart/form-data" ...

Comment: Hi, It's showing multiValueDictKey error on toolicon
Comment: Did you set your form to `multipart/form-data`? If the form doesn't contain an icon then you'll need to use the `FILES.get()` method - I have updated the answer
Comment: Now its showing 'string argument without an encoding'. I want the image as binary field. Do I need any encoding?
Comment:
ToolIcon
   that is my html now
Comment: I don't think your form is posting the binary data correctly. Check in your browser's network panel. Also shouldn't your Image Upload button submit the form? e.g. `$('#OpenImgUpload').click(function(){ $('#formId')[0].trigger('submit'); });`
Comment: No, I have a signup form for example, name, Empid, etc, plus icon to upload photo. Now At last there is one button save , onclick which one user will be created.So basically I want to upload image as when clicking final save it will go to my create_user method


Posição do Elemento Antes de Arrastar - Droppable - jQuery

Question: Estou fazendo um jogo e preciso pegar a posição de um elemento antes de arrastá-lo. Estou usando a função Droppable do jQuery UI.
jQuery
$("div.cesto").off().droppable({
    accept : "span.piece",
    // Quando coloca o mouse em cima do cesto
    over : function(){
        $(this).addClass("amp");
        media.replay("boeing-02");
        media.play("boeing-02");
    },
    // Quando sai com o mouse de cima do cesto
    out : function(){
        $(this).removeClass("amp");
    },
    drop : function(event, ui){
        var cesto = $(this);

        // Posições atual do elemento
        var positionTopPiece  = ui.position.top;
        var positionLeftPiece = ui.position.left;

Nas duas últimas linhas do código notem que estou pegando a posição atual do elemento. Ou seja, eu clico sobre o elemento, arrasto e solto e me dá a posição.
Mas não é isso o que eu quero. Eu quero que ao clicar no elemento, antes de arrastá-lo, guarde a sua posição TOP e LEFT para usar posteriormente em uma função.
Isto é, o elemento vai voltar para sua posição de origem dependendo de um resultado.
Estou na documentação do Droppable - http://api.jqueryui.com/droppable/
Mas não encontro uma função que faça o que preciso.

position
Type: Object
Current CSS position of the draggable helper as { top, left } object.
offset
Type: Object
Current offset position of the draggable helper as { top, left } object.
Answer: Consegui resolver, usando o callback activate.
    // Ao clicar sobre o elemento
    activate : function(event, ui){
        positionTopPiece  = ui.draggable.css("top"); // Posição de Origem - Para caso errar, voltar
        positionLeftPiece = ui.draggable.css("left"); // Posição de Origem - Para caso errar, voltar
    },

Criei duas variáveis globais.
positionTopPiecee positionLeftPiece.
Pego as posições top e left e depois uso em uma função para voltar o item à sua origem.
Obrigado.


Angular2 - pass ASYNC data to child component

Question: I have issue, with passing async data to child component. I trying to write dynamic form generator. Issue starts when I try to call json via Observable and pass it into child component.
service:
generateSearchFields2(): Observable<any> {
        return this.http
            .get(this.API + 'searchFields')
            .map((res:Response) => {
                res.json().data as any;
                for (var i = 0; i < res.json().data.length; i++) {
                    var searchField = res.json().data[i];
                    switch (searchField.component) {
                        case "TextboxQuestion":

                            let TXQ: TextboxQuestion = new TextboxQuestion({
                                key: searchField.key,
                                label: searchField.label,
                                value: searchField.value,
                                required: searchField.required,
                                order: searchField.order
                            });
                            this.searchFieldModels.push(TXQ);
                            console.log("TXQ: ", TXQ, this.searchFieldModels);
                            break;
                        case "DropdownQuestion":

                            let DDQ: DropdownQuestion = new DropdownQuestion({
                                key: searchField.key,
                                label: searchField.label,
                                required: searchField.required,
                                options: searchField.options,
                                order: searchField.order
                            });
                            this.searchFieldModels.push(DDQ);
                            console.log("TXQ: ", DDQ, this.searchFieldModels);
                            break;
                        default:
                            alert("DEFAULT");
                            break;
                    }
                }
                return this.searchFieldModels.sort((a, b) => a.order - b.order);
            })
            .catch(this.handleError);
    }

Component Parent:
generateSearchFields2() {
    this.service.generateSearchFields2()
      .subscribe(res => this.searchFields = res)
  }

Iam passing variable via INPUT directive in parent template to child: [searchFields]="searchFields"
Issue is in child component, where searchField has undefined value. In this child I pass value to another service, to create formContros, but I got undefined there also. Data missing starts here, in child:
@Input() searchFields: SearchBase<any>[] = [];

ngOnInit() {
    this.form = this.qcs.toFormGroup(this.searchFields);
    console.log("ONINIT DYNAMIC FORM COMPONENT: ", this.searchFields);

  }

Please for hint how I can pass async variable, to not loose data meantime
Answer: You can make @Input() searchFields a setter
private _searchFields: SearchBase<any>[] = [];
@Input() set searchFields(value SearchBase<any>[]) {
  if(value != null) {
    this.form = this.qcs.toFormGroup(this.searchFields);
    console.log("ONINIT DYNAMIC FORM COMPONENT: ", this.searchFields);
  }
}

get searchFields() : SearchBase<any>[] {
  return this.searchFields;
}

You can also use ngOnChanges() which is called every time an input is updated, but a setter is usually more convenient except perhaps when the executed code depends on multiple inputs being set.
Comment: can you please  explain more about the difference between setter and ngOnChanges() ?
Answer: In the ngOnInit event the data which comes from the parent is not bound yet. So your searchFields is undefined yet. You can use it in NgAfterViewInit component lifecycle event.
@Input() searchFields: SearchBase<any>[] = [];

ngAfterViewInit() {
    this.form = this.qcs.toFormGroup(this.searchFields);
    console.log("ONINIT DYNAMIC FORM COMPONENT: ", this.searchFields);
}

For other cases you can see Angular2 Component Lifecycle events


How can submitted values be translatable?

Question: Like for the node, we can add the translations and the same node ID can hold all language translations. In a similar fashion, is it possible in Drupal 8 webform?
Comment: Translating webform submissions? Or webforms? What exactly are you trying to achieve? Can you maybe edit your questions and add some more context, or something like a user story?
Answer: Webforms are configuration entities and they need to be translated using a different UI/UX.
@see https://www.youtube.com/watch?v=dfG37uW5Qu8


What is the best way to distribute n forms in c categories between u users?

Question: I have asked this question in cstheory too
I have a form distribution problem. There is n forms in c categories (each form in 1 category). And there is u users, which each user can receive forms from at least one category (but maybe more than one category).
The goal is to distribute forms between users, so each user receive the same amount of forms. I prefer to equally use categories.
For example:
If categories are:

C1 : 20 forms
C2 : 3 forms
C3 : 8 forms
C4 : 2 forms

And users are:

U1 with access to C1 and C2
U2 with access to C2
U3 with access to C3
U4 with access to C1 and C3
U5 with access to C2 and C4

The answer should be:

U1: 1 x C1 + 1 x C2 | 2 x C1 (preferred)
U2: 2 x C2
U3: 2 x C3
U4: 1 x C1 + 1 x C3 | 2 x C1 (preferred) | 2 x C3
U5: 2 x C4

And 23 forms remains.
Do you have any suggestion on how can I write such algorithm?
There could be a second question, which in that some Categories have a SHOULD CONTRIBUTE option. Which if set, all remaining forms in that category will distribute between users who have access to that. for example if C1 have this option enabled, the answer should be:

U1: 1 x C1 + 1 x C2 + 9 C1
U2: 2 x C2
U3: 2 x C3
U4: 2 x C3 (to minimize remaining forms in C3 category) + 10 C1
U5: 2 x C4

and remaining forms would be 0 in C1, 0 in C2, 4 in C3 and 0 in C4.
I think its kinda Bin Packing algorithm, but I am not sure and I don't know how to solve it! :(
Note: The above answers are not best answers, these are just what I think!
Comment: -1 for cross posting. Next time, wait patiently for an answer on the original site you have asked on.
Comment: OK. I just thought that this could be related to stackoverflow too! I am a programmer, not a scientist!
Comment: I came to this problem in one of my Web applications! This is exactly what my client needs! So I think its really practical!
Comment: I am not saying it doesn't have practical applications, but the question, as posed, is very much in the realm of theory.
Comment: Possibly - but cross posting is just confusing. This question is indeed theoretical in nature, not practical (as such not suitable to SO). It might be suitable for [programmers](http://programmers.stackexchange.com), but only post there if it gets rejected from cstheory.
Comment: Yes, I write it in math language ;)
Answer: It seems to me that if you fix a number N of forms per user and ask the question: can we give N forms to each user? then you can turn this into a http://en.wikipedia.org/wiki/Maximum_flow_problem problem, where each user can receive flow/forms from their subset of categories, and there is an outflow of capacity N from each user. Also, if you can solve this problem for N you can solve it for all lesser values of N.
So you could solve the first problem by running max-flow lg (maximum N) times, using a binary chop to find out what the best possible value of N is. Since you can solve it by max flow, you can also solve it by linear programming. Doing it this way, perhaps just for the critical value of N, might allow you to favour some assignments over others, or perhaps to see where there are neighbouring feasible solutions, and then see if you can mix them to use categories equally.
Example - Create a source, and link it to each of the categories Ci, with the capacity of the link being the number of forms available in that category, so C1 gets a link from the source of capacity 20. Create links with their source's capacity between users and categories, where the user has access to the category, so U1 gets links to C1 and C2, but U2 only gets a link to C2. Now create links of capacity N from each user to a single sink. If there is an assignment of forms to users that gives every user N forms, then this will produce a maximum flow that fills every link from user to sink, and you can look at the flows between users and categories to see how to assign forms. You could start off with N = 3, because user 2 only has access to a total of 3 forms, so the answer can't be greater than that. That won't work because you have said the right answer has N = 2, so the max flow won't fill all the N=3 capacity links. So your program tries again at 3/2 = 1, and finds a solution - you have provided a solution for N = 2, so there must be one for N = 1. Now the program knows there is a solution for N = 1 but not one for N = 3 so it tries one halfway between at N = (1 + 3) / 2 = 2, and finds your solution. There is one for N = 2 but not for N = 3 so the N = 2 solution is the best you can do.
Comment: Thanks mcdowella. Your answer seems to be really close to what I need. But, can you explain more? To be honest, I don't get much from your answer and related wikipedia articles. Can you suggest any implemented code?
Comment: There appears to be a basic implementation of max flow at http://en.wikibooks.org/wiki/Algorithm_Implementation/Graphs/Maximum_flow/Edmonds-Karp. I found this fairly quickly, so you may be able to find more sophisticated versions like http://www.maths.lth.se/matematiklth/personal/petter/pythonmaxflow.php, and I am sure there are commercial implementations. I will edit the article to add an example of the network required.
Comment: Many many many thanks! I'm going to code this :)
Comment: just to mention, I found [this](http://kunuk.wordpress.com/2010/11/09/graph-flow-ford-fulkerson-algorithm-example-with-c/) and its exactly what you've said and what I need :)


What is the difference between the OPC-Router and Kepserverex?

Question: What is the difference between the OPC-Router and Kepserverex? I have seen a solution where both of them were used - Kepserverex connecting  to machines and sending data to the OPC-Router which then sends data to IT Applications. Couldn't one of them do both tasks together?
I did compare the features of both products, but couldn't figure out why you would use one or the other - or both together. Both products seem to support data exchange between different protocols and platforms.
Answer: The KepServerEX is an OPC Server. Connecting over 150 different kinds of machines and providing their data as OPC UA data for OPC Clients.
The OPC Router is an OPC Client and is a tool for system integration. With the OPC Router it is possible to transfer data between systems whenever the data is needed (triggered). So the OPC Router connects systems like SAP, databases, label printers and a lot more and moves data between them. This also includes workflow-like data transfers, like calling a Stored Procedure in a database with a defined set of input parameters and transferring the returned values back to another system (maybe to the PLC over OPC UA => over the KepServerEX). In general you enable systems to interact with each other by the OPC Router. And this is where Industry 4.0 starts.
Greetings from the inray-Team
Comment: Thank you very much for the explanation - now I can see the difference. Since the Kepserverex can read OPC UA data from other OPC-Servers too, it also acts as an OPC client but this this not its core functionality?


If condition inside preg_replace in PHP

Question: I want to add comment inside a replaced text only if condition is true.
This is the basic code I work with.
<?php
$keepcomments="yes"; //yes or no
$str="abcdefgh";
$str = preg_replace("~abcd~", 'dcba "if($keepcomments=="yes"){ echo"-- some comments here --"} ', $str);
echo $str;
?>

I tried with :
if ($keepcomments=="yes"){
$str = $str."<-- some comments here -->";
}

The result is :
dcbaefgh-- some comments here --

But I have to preg_replace in more lines, problem is when I do that, all the comments are appearing at the end of result string.
dcbaefgh
dcbaefgh
dcbaefgh
-- some comments here --
-- some comments here --
-- some comments here --

and I don't know how to use preg_replace_callback() in this case.
Any help would be appreciated.
EDIT
When the string is "abcdefgh", i want to replace "abcd" in that string with "dcba". at the same time, if $keepcomments is yes, then i want to add a comment at the end of that string.
and then i want to duplicate same code in few more lines.
Expected output is :
dcbaefgh-- some comments here --
dcbaefgh-- some comments here --
dcbaefgh-- some comments here --

Comment: What's the expected output?
Comment: abcdefgh -- some comments here --
abcdefgh -- some comments here --
abcdefgh -- some comments here --
Comment: Your matching pattern only covers `'abcd'` so at best it would produce `'abcd-- comment here--'`; what exactly are you trying to match?
Answer: Use a callback function:
$str = preg_replace_callback("~(abcd)(.*)~", function($match) use ($keepcomments) {
    $r = strrev($match[1]) . $match[2];

    if ($keepcomments == 'yes') {
        return $r . '-- some comments here --';
    } else {
        return $r;
    }
}, $str);

Comment: It gives me output like this => -- some comments here --efgh
Comment: @MuhammedAslamC It's not clear what you want? Should the comment only appear at the end of the line, or ... ?
Comment: dcbaefgh -- some comments here --
dcbaefgh -- some comments here --
dcbaefgh -- some comments here --

i need to output like this :)
Comment: and now your code gives me output => abcd-- some comments here --efgh
Comment: @MuhammedAslamC I know what the output should be like, but I'm asking why that is ... should it match `'abcdefgh` completely or just `'abcd` ` followed by zero or more characters ... I can't tell.
Comment: @MuhammedAslamC I've updated the answer with what I think is the right approach ...
Comment: Thanks a lot, it worked!  i had to change something- and i don't know how it worked ;)

$str = preg_replace_callback("~(replace-string-here)(.*)~", function($match) use ($keepcomments) {
    if ($keepcomments == 'yes') {
        return  'string -- comments -- ';
    } else {
        return 'string';
    }
}, $str);
Answer: $str=array("abcdefgh","dfgvdfg","rgerhg");

foreach($str as &$row)
{
if ($keepcomments=="yes"){
$row= $row."<-- some comments here -->";

}
}

Answer: This answer from jack did the trick : https://stackoverflow.com/a/22777242/1940720
i had to change something. i submit the final answer here, maybe it will help someone.
$str = preg_replace_callback("~(replace-string-here)(.*)~", function($match) use ($keepcomments) {
    if ($keepcomments == 'yes') {
        return  'string -- comments -- ';
    } else {
        return 'string';
    }
}, $str);

If condition inside preg_replace in PHP


Confidence interval does not include 0, but Wilcoxon test is sgnificant

Question: I've created an error plot with CI 95% to visualize the difference (post test - pre test) between achieved scores.

n = 64
Judging from the confidence intervals one could conclude that there's is no significant improvement between the pre and post test scores (the CI does include 0).
When I use a paired t-test for the pre and post scores in combination, none of them return significant as expected.
But since the differences of the scores are not normally distributed for Total Score Change (p < .001) and Sub Score Change I (p < .05) according to Shapiro-Wilk, i've use the paired Wilcoxon test, which turns out significant for Sub Score Change I (p < .05).
So which test should I believe and why?
Comment: Does the rule for determining significance by whether the confidence intervals include 0 or not apply only for normally distributed data?
Answer: You need to use a point estimate (pseudo median) and confidence interval that is consistent with the Wilcoxon signed-rank test.  Here is an example in R, where there are no ties in absolute values of diff so that exact calculations are available (otherwise use the R coin package).
diff <- c(-3.1, -2.2, 1.3, 2.4, 3.5, 4.6, 5.7)
wilcox.test(diff, exact=TRUE, conf.int=TRUE, conf.level=0.95)

Wilcoxon signed rank test

data:  diff
V = 22, p-value = 0.2188
alternative hypothesis: true location is not equal to 0
95 percent confidence interval:
 -2.2  4.6
sample estimates:
(pseudo)median
           1.8

Note that the pre-post design does not allow cause and effect type of inferences.
Comment: Right about the first part, but you still don't understand the big picture.  If you are going to be using statistics you should study some standard texts.  My point was that the parametric confidence interval is not supposed to be concordant with the nonparametric test.  One deals with the mean and the other with the pseudomedian.  Whether the assumptions of a parametric test are satisfied or not is another matter.
Comment: Thank you! But to be honest I don't know how to apply answer to my example.
I still don't understand why I can't use the Wilcoxon-signed-rank test? It yields a more desirable result (significant) although it is a non-parametric test, which normally are supposed to be less powerful.
Comment: If you have paired data, the Wilcoxon test may a good one.   But you were asking why the inconsistency between parametric confidence intervals and the nonparametric test.  I think I answered that.  If you want a confidence interval that is consistent with the test (which is desirable) then go with the example above.  This confidence interval can be thought of as the set of pseudomedians (median of all possible pairwise means) that if hypothesized to be the population pseudomedian would not be rejected at the $1 - \alpha$ level.
Comment: Okay, thanks. So you suggested to use the CIs from your example, when I use the Wilcoxon test. Same goes for plotting. Instead of mean the pseudo median with the adjusted CIs.

So the rule for determining significance by whether the confidence intervals include 0 or not, does probably only apply when all assumption of a parametric test are met.


Accessing double from .txt file

Question: I have a .txt file that looks like:
<code>(0.781860352, -11.6927013, 7.20992613)
(0.364501476, -9.41256046, 6.87873077)
(0.394773483, -7.85253429, 6.90275288)
</code>
I need a code to access each double from each line individually. I am having troubles solving this problem. Any ideas?
Thanks!
UPDATE:
So, like I said, I am accessing each double from each line using the suggestion by Kerrek SB, then I store these values into the components of a vector, then I access each component and perform an operation (say multiply them together), and store the result into an array; finally I add the contents of the array.
Anyway, here is the code:
<code>int main() {

    //=============================
    //Initial Declarations
    //=============================

Vector3 r(0,0,0);
int num = 0 , i;
string line;
char c, d1, d2, d3, d4;
double v1, v2, v3 , b[num];

   //=============================
   //Counting Lines in File
   //=============================

ifstream is;
is.open ("data.txt");
while (is.good())
{
c = is.get();
if (c=='\n')num++;
}
is.close();
cout<<"Number of lines in file is "<<num<<endl;

   //=============================
   //Getting Data
   //=============================

ifstream infile("data.txt");

for (line ,i=0 ; getline(infile, line), i<=num-1;i++ )
{

    istringstream iss(line);

    if (!((iss >> d1 >> v1 >> d2 >> v2 >> d3 >> v3 >> d4 >> ws) &&
          iss.get() == EOF                                           &&
          d1 == '(' && d2 == ',' && d3 == ',' && d4 == ')'))
    {

    }

    // Placing data in vector
    r[0] = v1;
    r[1] = v2;
    r[2] = v3;

    //Placing data from vector into array
    b[i] = r[0]*r[1]*r[2];

}

  // Adding contents of array
  valarray <double> bfinal (b,num);
  double b_final = bfinal.sum();

cout<< b[0] << "," << b[1] << ", " << b_final << endl;
return 0;
</code>
}
The code works (actually it is not adding the contents of the array correctly) , but it seems a little crude. I feel that it could be simplified and/or made more concise. Any ideas?
Comment: This is pretty basic. What have you tried?
Comment: With your edit, this might fit on http://codereview.stackexchange.com.
Answer: The standard C++ idiom:
<code>#include <string>
#include <fstream>
#include <sstream>
#include <iostream>

std::ifstream infile("data.txt");

for (std::string line; std::getline(infile, line); )
{
    char d1, d2, d3, d4;
    double v1, v2, v3;
    std::istringstream iss(line);

    if (!((iss >> d1 >> v1 >> d2 >> v2 >> d3 >> v3 >> d4 >> std::ws) &&
          iss.get() == EOF                                           &&
          d1 == '(' && d2 == ',' && d3 == ',' && d4 == ')'))
    {
        // error parsing "line"
        std::cerr << "Sorry, could not parse '" << line << "'. Skipping.\n";
        continue;
    }

    std::cout << "You said: " << v1 << ", " << v2 << ", and " << v3 << "\n";
}
</code>
Comment: I can't help but snicker a little Kerrek. +1.
Comment: @CaptainGiraffe: How so? Because it's paste+rep? :-) I normally don't like being cheap, but there are *so many* alleged C++ answers on this site that teach wrong stuff that I'd rather spell this out redundantly one more time :-S
Comment: That too. The funnies of sophistication vs. expectation is an extra bonus. E.g. will OP put this in a proper context, or assume script style parsing.
Comment: Exactly! Thank you!
I'll show the context in which this is being used once I work it out. Thanks again.
Comment: You're missing an `else`

Accessing double from .txt file


Quantifiers as Adjoints in Generalized Logics

Question: It is a well known fact that the classical universal and existential quantifiers can be seen as adjoints in certain categories.
In the continuous model theory of metric structures (see http://ptmat.fc.ul.pt/~alexus/papers/mtfms.pdf ) and more generally, the continuous model theory developed by Chang and Keisler (http://books.google.co.uk/books/about/Continuous_Model_Theory.html?id=uTGdPSI5rI4C&redir_esc=y), sets of truth values are (roughly) ordered, compact, Hausdorff spaces with two distinguished elements (0 and 1) that act like true and false. In this setting, the analogues of $\forall$ and $\exists$ are $\sup$ and $\inf$.
I have a two part question:
1) In the continuous model theory setting, how does one describe quantifiers in a similar categorical way?
2) Is there a natural analogue of quantification in the setting of truth values in a non-ordered, compact, (and Hausdorff if you really want it) topological space. I appreciate that if there is, it might be quite strange since I'm not requiring there be a ''true'' and ''false''
Answer: First, let's be clear about how we can view quantifiers as adjoint functors. Here's one way to look at it.
What is a predicate $P(x)$, with the variable $x$ coming from the sort $X$? Viewing $X$ as a set, we can view $P(x)$ as a subset of $X$. The (definable, if you like) subsets of $X$ come with a natural (partial order) category structure, where there is an arrow $P(x) \rightarrow Q(x)$ if and only if $P\subseteq Q$. Note that we can also view this category structure as the partial order of implications between predicates. Let's call this category $\text{Pred}(X)$.
Now there is a functor $D_y:\text{Pred}(X)\rightarrow \text{Pred}(X\times Y)$ corresponding to adding a dummy variable $y$ of type $Y$. So $D_y(P(x)) = \{(x,y)\in X\times Y\mid x\in P(x)\}$. Let's write this as $P(x,\dot{y})$, with the dot denoting that $y$ is a dummy variable. This is a functor, since if $P(x)\rightarrow Q(x)$, then $P(x,\dot{y})\rightarrow Q(x,\dot{y})$ (again, you can think of this as containment or implication).
There are also functors $\exists y:\text{Pred}(X\times Y)\rightarrow \text{Pred}(X)$ and $\forall y:\text{Pred}(X\times Y)\rightarrow \text{Pred}(X)$, which do what you expect: $\exists y\,P(x,y) = \{x\in X\mid \exists y\, (x,y)\in P(x,y)\}$, and $\forall y\,P(x,y) = \{x\in X\mid \forall y\, (x,y)\in P(x,y)\}$.
Now $\exists y\dashv D_y\dashv \forall y$. The adjunctions correspond to the following statements $$\exists y\, P(x,y) \rightarrow Q(x) \iff P(x,y)\rightarrow Q(x,\dot y)$$
$$P(x)\rightarrow \forall y\,Q(x,y) \iff P(x,\dot{y})\rightarrow Q(x,y)$$
Okay, but now instead of viewing a predicate $P(x)$ as a subset of $X$, we can turn the picture around and view it as a (definable, if you like) map $X\rightarrow \{0,1\}$. What's the corresponding partial order structure on maps $X\rightarrow \{0,1\}$? Well, $P(x) \rightarrow Q(x)$ if and only if for all $x\in X$, $P(x) \leq Q(x)$. That is, the partial order is pointwise comparison of functions.
Now the functor $D_y$ just takes a map $X\rightarrow \{0,1\}$ and composes with the projection $X\times Y \rightarrow X$, and its adjoints are given by
$$(\exists y\, P(x,y))(x) = \begin{cases} 1\text{ if }\exists y\,P(x,y) = 1\\ 0\text{ otherwise}\end{cases} = \text{max}_{y\in Y}P(x,y)$$
$$(\forall y\, P(x,y))(x) = \begin{cases} 1\text{ if }\forall y\,P(x,y) = 1\\ 0\text{ otherwise}\end{cases} = \text{min}_{y\in Y}P(x,y)$$
But now it's easy to guess (and to check) that you can replace $\{0,1\}$ with $[0,1]$, take the same category structure on predicates (pointwise $\leq$ in $[0,1]$), and get the continuous model theory quantifiers as adjoints to the "dummy variable functor", replacing max and min by sup and inf.
It may seem like I've gotten inf and sup mixed up, but that's just because in continuous model theory $0$ is true and $1$ is false, contrary to the convention when viewing a subset of $X$ as a map $X\rightarrow \{0,1\}$.
What about your second question? If the compact Hausdorff space isn't ordered, it's not clear to me what the right category structure should be, since the one in which the quantifiers appear naturally comes directly from the order. Of course, one can more easily generalize this situation replacing $\{0,1\}$ and $[0,1]$ with any complete lattice - this is what happens in topos theory.
Comment: It's because the intended semantics is metric, not probabilistic. The logical symbol $=$ is replaced by a "distance function" $d$, so "x=y is true" is replaced by "d(x,y) has value 0".
Comment: Why in a continuous model is $0$ true and $1$ false? I am sure that this is fine in a formal sense. However one interpretation of a logic valued in $[0,1]$ as some sort of probability.
Comment: It looks like the probabilistic interpretation and the metric interpretation are dual. I wonder what kind of work has been done on this type of duality. Perhaps some intuition for this is that  the farther away an event is, the smaller the probability of observing it.
Comment: @AlexKruckman So in the book I mentioned, if $X$ is a compact Hausdorff space (say with distinguished elements $0$ and $1$, though not necessarily ordered) quantifiers are defined as continuous functions from $X*$ to $X$, where $X*$ is the set of all non-empty subsets of X endowed with the Vietoris topology. They go on to show that they behave as you would expect with respect to ultraproducts.
Comment: @BabyDragon in the paper I mentioned in the question (http://ptmat.fc.ul.pt/~alexus/papers/mtfms.pdf )they address probability algebras. You might find it interesting.
Comment: Okay, I guess that makes some sense - although I don't at all understand the role of the Vietoris topology. Using that definition allows quite a lot of freedom in what you call a quantifier (similarly to the definition of connective), but you'll need a good category of formulas before you can ask which quantifiers are "natural"!


Are there speed and resource allocation differences between 1x 300 GB Azure VM data disk vs 3x 100GB Data disks (and striping those)

Question: I'm researching playing around with a Sql Server VM on Azure.
Our DB is around 30-40GB.
So i was thinking about making a 300GB data disk to put the 1 Database on it (or a few more, later).
I believe i have a few options here.

A single Data Disk which is 300GB in size.
Multiple Data Disks with striping that total 300GB (Eg. 2x150 or 3x100 GB)

Questions:

Is it worth striping?
Does either option consume more page blobs than the other?

I'm under the impression that:

Going the striping option offers us more IOPS at the cost of higher risk or HD failure which means higher risk of data loss (for the sake of this discussion, lets leave backups out of this).
The multiple data disks will consume more page blobs but that number is tiny and not really worth worrying about. (but I have ZERO proof of this nor do i know how to figure out this metric answer).
Comment: After the revision, I fear this is mainly a matter of opinion and your application.  Also, it's not "stripping"; it's "striping", as in "stripe".
Comment: @FalconMomot - feel free to you know; edit it
Answer: The number of pages should be roughly equivalent. Probably not exact since you're formatting multiple disks, but in the range of noise that won't really be noticed from a billing perspective.
Regarding disk-crashes. Each independent disk is a vhd stored in a page blob, which is part of Azure Storage. Azure Storage is durable - each blob is triple-replicated with a datacenter (and optionally geo-replicated to a secondary datacenter). So you're more likely at risk to have a programming error where you accidentally delete files than having a disk failure.
Regarding IOPS: You'd need to do some benchmarking. You'll likely see better throughput on a striped volume since you'd be moving data across multiple storage partitions.
You may want to simply start with a single disk, since a single disk may be up to 1TB and supports your 300GB requirement. This keeps your solution simple, especially if you want to move your disk to another VM.
Comment: Regarding performance, one thing to bear in mind is the scalability targets: http://blogs.msdn.com/b/windowsazure/archive/2012/11/02/windows-azure-s-flat-network-storage-and-2012-scalability-targets.aspx. Additionally, if you were to span your disks across multiple storage accounts you might see better performance.
Comment: Thank you heaps @davidMakogon! This really helps answer some of my questions. Cheers and keep up the good work!


ColdFusion 9 CFZIP Unzip Error On a Zip File Created From ColdFusion 11 CFZIP

Question: The ColdFusion application that I support is installed at two different locations.  One location is running Windows Server 2008 with ColdFusion 9 and MS SQL Server 2008 and the other is running Windows Server 2012 with ColdFusion 11 and MS SQL Server 2012.  The application provides an export process, using CFZIP action = "zip" and then the zip file is imported on the destination machine, using CFZIP action = "unzip".
Here is the code I have to generate the zip file for export:
<cfzip file="exportFileName.zip"
       source="#exportDirectory#"
       action="zip"
       overwrite="yes"
       recurse="yes">

The .zip file gets generated properly and I can open it in Windows Explorer and 7Zip with no issues.
Here is the code to unzip the zip file created above:
<cfzip action="unzip"
       file="exportFileName.zip"
       destination="#destination#\xml"
       recurse ="yes"
       storepath="yes">

This is the same code on both the ColdFusion 9 and ColdFusion 11 instances, however when we try to unzip a zip file generated by the ColdFusion 11 instance on the ColdFusion 9 instance I receive the following error:

Ensure that the file is a valid zip file and it is accessible.  Cause:  java.util.zip.ZipException:  only DEFLATED entries can have EXT descriptor

We're only seeing this issue when using CFZIP on a ColdFusion 9 server to unzip a zip file generated using CFZIP on a ColdFusion 11 server.  I can extract the contents of the zip file generated from ColdFusion 11 on the ColdFusion 9 server, create a new Zip file, using 7Zip, that contains the exact contents of the original ColdFusion 11-generated zip file, and I don't get the error.
The process works fine whenever we test it going from a ColdFusion 11 source to a ColdFusion 11 destination or from a ColdFusion 9 source to a ColdFusion 9 destination.  We're only getting the issue when zipping on ColdFusion 11 and trying to unzip on ColdFusion 9.  I've scoured Google, but can't seem to find any issues like this.  Any help would be greatly appreciated.
Comment: i would suggest you should use java because of coldfusion version complication. ... `java.util.zip.ZipFile`  or `java.util.zip.ZipInputStream`  `Fz = createObject("java","java.io.BufferedOutputStream").init(fos,arrayLen(buffer));`
Comment: sources:

•  http://stackoverflow.com/questions/12346707/java-util-zip-zipexception-invalid-cen-header-bad-signature

•http://www.kodejava.org/examples/334.html

•http://blogs.adobe.com/cantrell/archives/2004/01/byte_arrays_and_1.html
Comment: FWIW, it worked fine for me with CF9.0.2 and CF11. Does this occur with *all* zip files, or just some of them? What jvm versions?
Comment: @Leigh It is zip files which contain zero-length entries where those entries do not have the compression flag set but do have another flag set... pretty specific circumstances but `CFZIP` can generate the correct set of flags when it zips directories.
Comment: @MT0 - That is good info. Inundated with work right now, but cannot wait to run some tests with CF9. Thanks!
Comment: The underlying issue in Java's handling of `ZipInputStream` has been reported to Oracle (see bug report: [JDK-8143613](http://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8143613).
Answer: [TL/DR]

CFZip on ColdFusion 11 complies with a zip specification later than PKZIP 2.04g
Java's ZipInputStream (which CF9 appears to use) complies strictly to the PKZIP 2.04g (or earlier) specification.

A subtle difference between the two versions is their handling of zero-length entries (i.e. sub-directories or mime-types among others). CFZip in CF11 marks these entries as being STORED with no compression (which, intuitively and by later versions of the specification, is fine as they are zero-length so compression won't do anything to them) but ZipInputStream (called by CFZip in CF9) expects them to be marked as compressed using the DEFLATE method.
If you have no zero-length entries then the files generated is ColdFusion may be able to be read (I think they will but I can't prove it definitively) using java's ZipInputStream; however, if you have zero-length files then it will throw errors.
Alternatively, either:

Use something other than CFZip in CF9 which is able to read zip files compliant against later standards (i.e. you could use cfexecute to run 7zip external to ColdFusion or use the java org.apache.commons.compress library inside ColdFusion); or
Use something other than CFZip in CF11 which will restrict the zip files it creates to the earlier standard (see below).

Detailed answer:
From the Zip Format Specification section 4.4.4 general purpose bit flag:

Bit 3: If this bit is set, the fields crc-32, compressed
         size and uncompressed size are set to zero in the
         local header.  The correct values are put in the
         data descriptor immediately following the compressed
         data.  (Note: PKZIP version 2.04g for DOS only
         recognizes this bit for method 8 compression, newer
         versions of PKZIP recognize this bit for any
         compression method.)

This bit is set when the local file header indicates that there is a zero-length entry in the zip file (i.e. a directory or certain other things in a zip-file like an embedded mime-type).
Under the PKZIP version 2.04g (or earlier) standard it would then expect the compression method flag to be set to DEFLATE (method 8 compression). Java's ZipInputStream complies strictly to this standard and throws a ZipException (with the message only DEFLATED entries can have EXT descriptor) if it does not find this compression method (see source code here).
cfzip appears to set the compression method to STORED (method 0 compression - or no compression) when the General Purpose Flag bit is set. From what I have read this complies with later versions of the PKZIP standard but is not backwards compatible with the version of the standard implemented by Java in the ZipInputStream class.
How to mitigate for this:

Either don't store zero-length files in the zip (which means I don't think you can use the recurse option in CFZip); or
Don't use CFZip, in CF11, if you want strict backwards compatability and use something else (see below for a suggestion).
Use CFZip in CF11 but use something else to unzip the files in CF9 that is compatible with a greater range of compression algorithms (i.e. use cfexecute to call an external program to handle the zip files; use the java org.apache.commons.compress library; etc).

If you go for option 2 and want to use something compatible with the eariler versions then this worked for me and I can unzip the files using ZipInputStream (tested directly in Java as I don't have CF9):
package zip;

import java.io.*;
import java.util.zip.*;

public class Zip {
    private static void processFolder(
            final File folder,
            final ZipOutputStream zos,
            final boolean recurse,
            final int prefixLength,
            final byte[] buffer
    )
            throws IOException
    {
        for ( final File file : folder.listFiles() )
        {
            if ( file.isFile() )
            {
                final String name = file.getPath().substring( prefixLength );
                // System.out.println( name );
                final ZipEntry entry = new ZipEntry( name );
                zos.putNextEntry(entry);
                try (FileInputStream is = new FileInputStream( file ) ){
                    int read;
                    while( (read = is.read( buffer ) ) != -1 )
                    {
                        zos.write( buffer, 0, read );
                    }
                }
                zos.closeEntry();
            }
            else if ( recurse && file.isDirectory() )
            {
                processFolder( file, zos, recurse, prefixLength, buffer );
            }
        }
    }

    public static void zipFolder(
            final String folderPath,
            final String outputName,
            final boolean recurse,
            final boolean overwrite
    ) throws IOException
    {
        final File folder = new File( folderPath );
        if ( folder.exists() && folder.isDirectory() ) {
            final File output = new File( outputName );
            if ( overwrite || !output.exists() )
            {
                try ( ZipOutputStream zos = new ZipOutputStream( new FileOutputStream( output ) ) )
                {
                    processFolder( folder, zos, recurse, folder.getPath().length() + 1, new byte[1024*4] );
                }
            }
        }
    }
}

Based on this answer
(Note: the error handling is minimal so you probably want to make this more robust if you are going to use it in a production environment).
If you compile that java class and put the .class file in a zip subdirectory on your class path (which you can add entries to in the ColdFusion Administration panel).
Then you can call it using:
<cfscript>
  zip = CreateObject( "java", zip.Zip" );
  zip.zipFolder(
    "/path/to/folder/to/be/zipped/",
    "/path/to/output/zip/file.zip",
    true, // recurse
    true  // overwrite existing file
  );
</cfscript>

Testing
If you want something to test how this occurs then this will generate a valid zip file under the later specifiction which will generate the error:
import java.io.*;
import java.util.zip.*;

public class TestZip {
    public static void main( final String[] args ) throws IOException {
        final File file = new File( args[0] );
        if ( file.exists() )
        {
            System.out.println( "File already exists" );
            return;
        }
        final boolean general_purpose_bit_flag_bit3_on = true;
        final byte gpbf = general_purpose_bit_flag_bit3_on ? 0x08 : 0x00;

        final byte[] contents = new byte[]{
            // Local File header
            'P', 'K', 3, 4,     // Local File Header Signature
            13, 0,              // Version needed to extract
            gpbf, 8,            // General purpose bit flag
            ZipEntry.STORED, 0, // Compression method
            'q', 'l', 't', 'G', // Last Modification time & date
            0, 0, 0, 0,         // CRC32
            0, 0, 0, 0,         // Compressed Size
            0, 0, 0, 0,         // Uncompressed Size
            12, 0,              // File name length
            0, 0,               // Extra field length
            'F', 'o', 'l', 'd', 'e', 'r', '_', 'n', 'a', 'm', 'e', '/',
                                // File name
            // Central directory file header
            'P', 'K', 1, 2,     // Central Directory File Header Signature
            13, 0,              // Version made by
            13, 0,              // Version needed to extract
            gpbf, 8,            // General purpose bit flag
            ZipEntry.STORED, 0, // Compression method
            'q', 'l', 't', 'G', // Last Modification time & date
            0, 0, 0, 0,         // CRC32
            0, 0, 0, 0,         // Compressed Size
            0, 0, 0, 0,         // Uncompressed Size
            12, 0,              // File name length
            0, 0,               // Extra field length
            0, 0,               // File comment length
            0, 0,               // Disk number where file starts
            0, 0,               // Internal File attributes
            0, 0, 0, 0,         // External File attributes
            0, 0, 0, 0,         // Relative offset of local header file
            'F', 'o', 'l', 'd', 'e', 'r', '_', 'n', 'a', 'm', 'e', '/',
                                // File name
            // End of Central Directory Record
            'P', 'K', 5, 6,     // Local File Header Signature
            0, 0,               // Number of this disk
            0, 0,               // Disk where CD starts
            1, 0,               // Number of CD records on this disk
            1, 0,               // Total number of records
            58, 0, 0, 0,        // Size of CD
            42, 0, 0, 0,        // Offset of start of CD
            0, 0,               // Comment length
        };
        try ( FileOutputStream fos = new FileOutputStream( file ) )
        {
            fos.write(contents);
        }
        try ( ZipInputStream zis = new ZipInputStream( new FileInputStream( file ) ) )
        {
            ZipEntry entry = zis.getNextEntry();
            System.out.println( entry.getName() );
        }
    }
}


Pooled (or combined?) variance - is it suitable to use in the below scenario?

Question: I have a dataset with measurements and their deviations from nominal value. These measurements have been made on 3 different categories of a component. The components differ in physical properties (size, weight, volume etc.) Below are some more details:

3 categories: A,B,C
Count of number of observations for each category: $n_A$ = 2025, $n_B$ = 13507,$n_C$ = 21511
Each component has it's own nominal value : 0.8, 0.6, 1.00 respectively
The measurements & deviations can be between -${\infty}$ &
+${\infty}$
The simple means of the measurements : $\overline{x}_A$ = 0.976, $\overline{x}_B$ =0.908 , $\overline{x}_C$ =0.806
The standard deviation of the measurements : $\sigma_A$ = 0.062, $\sigma_B$ =0.069, $\sigma_C$ = 0.062
The simple means of the deviations from nominal value : $\overline{d}_A$ = 0.062, $\overline{d}_B$ = 0.035, $\overline{d}_C$ = -0.024
The data for each of the categories is close to normal (visual inspection using normal qq plot)
I have performed Bartlett's test for homogeneity of variances. There are significant differences between the sample variances.

The aim is to find a single statistic which can be used as a single representative value for the variances of the 3 categories. In the sense that if tomorrow the this single statistic increases (or decreases) then it can be used an indication to investigate further for the causes of variation.
I have gone through several questions on Cross Validated related to the topic of pooled variance and this one, I believe, is very close to the question I am asking. However, I am not sure if the formula given in the answer can be applied to my case. If not, then which statistic can I use in this case? If yes, then are there any assumptions that I should check for violation before applying the formula?
I cannot post the entire data, but I have attached a screenshot of a proportion of the data to give you an idea of what it looks like.
Answer: Assuming that the three true underlying variances are different from each other, there's no "correct" statistical method to aggregate them into one number. From a statistical modelling perspective you should have all three.
If for whatever reason you want a single number (index) anyway, this needs to involve decisions regarding the subject matter. There is more than one approach that could make sense.

You could compute a mean of all the category standard deviations. This would give all the standard deviations the same weight, indicating that they all have the same importance for whatever decision you want to make based on them.
You could compute the mean of all category variances. Same characteristics as before, but slightly different, because this gives higher weight to the larger standard deviations/variances.
You may want to weight by number of observations (you may think that the more observations there are, the more relevant a value is; also the more observation there are, the more reliable are the individual variances/sds). One way of doing this is to add up the squared differences from the respective category mean for all observations and divide by $n-3$ or $n$ (chances are the last thing doesn't make a relevant difference).
If you want to pick up situations where at least one category variance changes strongly, you could look at the maximum change out of all three from one time point (or some initial reference value) to the next.
You may want to bring in information from nominal values and deviations from them somehow, even though without understanding of the meanings of those and the exact aim of computing the index, I have no idea how.

The thing that you need to understand here is that the task is not to estimate some statistically assumed "truth", but rather that you need to decide how to aggregate the information that you have in order to be fit for the purpose you have in mind. (One could trace this back to a statistical modelling problem if there was a more detailed model about how the values are produced, and what kind of problem the index is meant to detect, but that seems out of reach based on the given information.)
You may also want to ask yourself whether you could make things work monitoring all three standard deviations separately rather then breaking them down to a single number.


Unbiased estimate for treatment effect in block randomized experiment where probability of treatment varies by block

Question: I want to analyze the results of a block randomized experiment. Within each block, units are randomized to treatment and control as if that block were a completely randomized experiment. Specifically, I want an unbiased estimator for the overall treatment effect and an estimate of the variance of this estimator.
If the probability of receiving treatment is the same in each block, then an unbiased estimate of the overall average treatment effect ATE can be obtained like so:
$$\hat{ATE} = \sum_{j=1}^J \frac{N_j}{N}\hat{ATE_j}$$
Where:

N = Total Number of units in the experiment
$N_j$ = Number of units in block j
$\hat{ATE_j}$ is the Average Treatment Effect in block j. Specifically,
$$ \hat{ATE_j} = \bar{Y}_{j,treatment} - \bar{Y}_{j,control}$$

$\bar{Y}_{j,treatment}$ is the average observed outcome in the treatment group in block j and $\bar{Y}_{j,control}$ is the average observed outcome in the control group in block j.
The difficulty is that the probability of treatment varies by the strata/blocks. So, some blocks have 50% of their units treated and others have, for example, 10% treated. I've been told that the above estimator for ATE is only unbiased if the probability of treatment is constant across the blocks and that, otherwise, I need to use some kind of Horvitz-Thompson weighting, weighting the units by their inverse probability of treatment, in order to get an unbiased estimator.
I can't find any literature on this. Do I need to use horvitz-thompson or some other kind of weighting? Or can I still use the estimator above? Any references would be especially appreciated.
Thanks!
Answer: I don't think this is the case. Each $ATE_j$ is unbiased, so weighted average of them should be unbiased as well because the expectation is linear operator.
Here's a 500 rep simulation in Stata where the treatment effect is set to 15 with 50 blocks of 100 observations (on average):
<code>capture program drop mybs
program define mybs, rclass
clear
qui set obs 5
gen block = _n
gen p_treat = uniform()
gen n = rpoisson(100)
qui expand n
gen treated = uniform()<p_treat
gen y = rnormal(100,15)
qui replace y = y + 15 if treated == 1
collapse (mean) y (count) n=y, by(block treated)
qui reshape wide y n, i(block) j(treated)
qui gen TE = y1-y0
qui gen N=n0+n1
qui sum TE [fw=N]
return scalar ate = r(mean)
end

set seed 123456
simulate ate = r(ate), reps(500): mybs
sum ate, detail
</code>
There does not appear to be any bias here:
<code>. sum ate, detail

                           r(ate)
-------------------------------------------------------------
      Percentiles      Smallest
 1%     9.240335       7.301056
 5%     11.53554       7.853343
10%     12.34418       8.115672       Obs                 500
25%     13.81046       8.936302       Sum of Wgt.         500

50%     15.07377                      Mean           15.03785
                        Largest       Std. Dev.      2.191757
75%     16.31851       20.99205
90%     17.74527       21.72955       Variance         4.8038
95%     18.50914       21.98358       Skewness      -.0549813
99%     20.30977       24.24625       Kurtosis       4.139082
</code>


Disappearing Content

Question: I just inherited a Drupal 7 website from another agency. The site was fairing well in our DEV and STAGE environments. The problem started occurring when we switched DNS over to our PRODUCTION site.
We have all the nodes (from the same content type) that seem to be "randomly" disappearing.
I've made sure to upgrade the core to the latest and any security fixes for any modules that are being used.
I was wondering what I could do to troubleshoot this issue - as I'm exhausting all my efforts and nothing seems to be working.
Comment: define randomly disappearing please. are those nodes deleted? or just showing 404s sometimes?
Comment: have you tried clearing the cache and rebuilding the node permissions? if you are login to your site would you still have the problem with those?
Comment: The nodes were deleted from a non-drupal related script. I'll make sure to post a follow up answer to this question. Thanks for you help.
Answer: It looks like a cache problem. I will suggest to do this, as usual backup is always a good idea:

See if this happens also if you are login. You might have your node permissions corrupted. If you rebuild and have a lot of node, it might take a little while to do this.
Turning cache off for page and blocks.
is your content generated by a view? if so check the permission
Comment: I found the issue. It was previous agency-related code that was running from an external source on a cron job hitting this script.

    function drop_villas() {
        $result = db_query("SELECT n.nid FROM {node} n WHERE n.type = 'villa'");
        foreach ($result as $n) {
     node_delete($n->nid);
        }
    }

Needless to say - the offending agency isn't very professional about transitioning this site.
Comment: can you answer your question and mark as accepted later. It will help our community.
Answer: Local module is disabled. Enable the Local module. It will solve it.
Comment: The correct answer was added as a comment by the OP about a year ago. Downvoted, as this answer does not make sense.

Disappearing Content


Fractal dimension of a dense subset

Question: Let $M$ be a metric space and $S\subset M$ a dense subset. For vague reasons (below), it seems to me that the upper box-counting dimension of $S$ should be equal to that of $M$, but I don't quite see the argument, and I am skeptical of my reasons anyway.

Is the upper box-counting dimension of $S$ equal to that of $M$? If so, what's the argument? If not, what's a counterexample?

Definitions and thoughts:
The upper box-counting dimension of a metric space $M$ is defined as follows: for $\varepsilon > 0$, let $N(\varepsilon)$ represent the minimal cardinality of a cover of $M$ by $\varepsilon$-balls. Then the upper box-counting dimension is
$$\limsup_{\varepsilon\to 0} \frac{\log N(\varepsilon)}{\log 1/\varepsilon}$$
Now if $S$ is dense in $M$, a cover of $S$ by $\varepsilon$-balls is "almost" a cover of $M$; the complement of the cover is a closed set with no interior. The number of extra $\varepsilon$-balls needed to cover the rest of $M$ should thus be "small". This is the intuition behind my question.
But I can't think of a way to make this argument at all quantitative because all the facts I have about the complement of the cover are topological - it is closed with empty interior, i.e. it is closed and nowhere dense - but not metric. In $\mathbb{R}$ we can easily have closed, nowhere dense sets that are not "small" in other ways - e.g. fat Cantor sets are closed and nowhere dense but of positive measure. So although this doesn't directly speak to the case at hand, it does suggest that the intuition mentioned above is not reliable.
Comment: If you can show that your definition doesn't depend on if your balls are open or closed, you've basically solved the question, because every (finite) cover of $S$ by closed balls cover $M$.
Comment: @Tryss - that's more or less exactly the question, isn't it?
Comment: @BenBlumSmith : no, it's a more general question, and easier to answer.
Comment: @Tryss - Ah! I think you're suggesting I note that open $\varepsilon$-balls contain closed $\varepsilon/2$-balls, etc.
Answer: Let's prove that the definition doesn't depend if the ball is open or closed.
The cover number for closed balls, noted $N_c$ verify
$$N(\epsilon(1+\epsilon) ) \leq N_c(\epsilon) \leq N(\epsilon)$$
Hence
$$\frac{\log N(\epsilon(1+\epsilon) ) } {\log (\frac{1}{\epsilon}) }
\leq \frac{\log N_C(\epsilon ) } {\log (\frac{1}{\epsilon}) }
\leq \frac{\log N(\epsilon ) } {\log (\frac{1}{\epsilon}) }$$
$$\frac{\log (\frac{1}{\epsilon(1+\epsilon)})}{\log (\frac{1}{\epsilon})} \frac{\log N(\epsilon(1+\epsilon) ) } {\log (\frac{1}{\epsilon(1+\epsilon)}) }
\leq \frac{\log N_C(\epsilon ) } {\log (\frac{1}{\epsilon}) }
\leq \frac{\log N(\epsilon ) } {\log (\frac{1}{\epsilon}) }$$
And as $\frac{\log (\frac{1}{\epsilon(1+\epsilon)})}{\log (\frac{1}{\epsilon})}  \to 1$, we can see that
$$\lim_{\epsilon \to 0}  \frac{\log N_C(\epsilon ) } {\log (\frac{1}{\epsilon}) }
= \lim_{\epsilon \to 0} \frac{\log N(\epsilon ) } {\log (\frac{1}{\epsilon}) }$$
Hence the definition doesn't depend if the balls are closed or open.
Then the initial problem is solved trivially by using the definition with closed balls
Comment: I buy this in essence, although there's something to be careful of since I am defining the dimension in terms of $\limsup$ and not assuming that a limit exists...
Comment: It shouldn't change anything to the demonstration, let me check

Fractal dimension of a dense subset


How to exclude a signer from receiving the completed/signed email

Question: We would like to prevent our Sales Rep(secondary signers) from receiving the completed/signed email with the signed documents as attachments. Only our external customers should be receiving the email.
Wondering if this is possible through the API or within the admin console setting.
We are using SOAP API by the way.
Thanks in advance.
Answer: It's not possible to exclude specific (remote) recipients from receiving the "envelope completed" email from DocuSign.  ("Embedded/Captive" recipients won't receive any emails from DocuSign by default -- but this isn't the scenario you're asking about.)
Additionally, whether or not the "envelope completed" email from DocuSign contains the signed/completed documents as attachments is controlled via an account-level setting (located in Preferences >> Features within the DocuSign web console):

Since this is an account-level setting, it's not possible to control this behavior (i.e., whether or documents are attached to the "envelope completed" email) on a per-envelope basis, let alone on a per-recipient basis for a specific envelope -- it's either enabled or disabled for the entire account (at any given point in time).
Keep in mind that even if you disable this setting at the Account-level (so that documents are never attached to "envelope completed" emails), all recipients will still be able to access the completed documents via the link in the "envelope completed" email (which displays the completed documents in the DocuSign web console), in addition to being given the opportunity to print and/or save the documents during their signing session. So, it's not like you're preventing a recipient's access to completed documents merely by not having the documents attached to the 'envelope completed' email.


transforming a specific equation

Question: how does the following equation:
$$\frac{1}{\sqrt{1+z^2}-z}$$
get transformed into:
$$\sqrt{1+z^2}+z$$
I'm sure I'm missing something obvious. Thanks.
Answer: $$\frac1{\sqrt{1+z^2}-z} = \frac{\sqrt{1+z^2}+z}{(\sqrt{1+z^2}-z)(\sqrt{1+z^2}+z)} = \frac{\sqrt{1+z^2}+z}{(1+z^2)-(z^2)} = \sqrt{1+z^2}+z$$
Comment: very nice, thanks for the quick reply

transforming a specific equation