Detect weak points of change (temperature profile)

Question: I am working on a method to detect a finite set of significant points of temperature change within a data series. Although my first pass does an okay job of detecting significant temperature transitions, the results are less than ideal when working with a noisy signal or weak temperature variance.
Sample Temperature Profiles

The X axis represents temperature and Y represents a sensor reading within the data set
Current Approach

Apply Savitzky–Golay Filter using multiple passes with a small window size to smooth data.
Calculate rate of change on smoothed data series and calculate standard deviation.
Starting with z-Score of 1.96 (~95% confidence) slowly increase or decrease confidence until I locate 2-5 significant temperature shifts that best fit the result of the previous data series or give up if I reach 0.84 (~60% confidence).

As illustrated in the left and middle profiles, this works okay; however there are times like the right profile where I need to detect changes in temperature that are weaker than my minimum allowed confidence of 60%. I know I could just continue to lower the bar (i.e., 50% or lower), but I am hoping there is a better approach.
Changes Considered

Reduce the minimum confidence level to 50% or lower to detect weaker transitions.
Given that there is normally a significant temperature increase on the lower half of the profile; break the data up into two halves at the maximum temperature reading (idea being variance on top half is typically weaker than the bottom half).
Don't use standard deviation and find the most significant transitions by identifying peaks and valleys ranked by overall temperature change.
Other…

I have been reading a lot of the related questions as well as scouring Wikipedia. I am fairly certain this is a well understood problem, and I am hoping someone can point me in the right direction on how to correctly identify the most significant changes in temperature given a small data series (72-120 values) while dealing with noise and weak variance.

EDIT 1
I neglected to mention that one of the challenges I am currently facing is that by applying the smoothing filter, I am negatively affecting my ability to detect weaker points of significant change.

EDIT 2
A raw sample data series (°C) representing the right temperature profile if needed…
<code>21.00,  21.90,  22.70,  22.90,  23.60,  23.60,  23.70,  23.70,  23.80,  23.80,   24.60,  24.70,  25.00,  27.00,  36.70,  53.90,  57.60,  57.80,  57.90,  57.90,  57.90,  57.80,  58.00,  57.90,  58.00,  58.00,  57.80,  58.00,  58.60,  59.00,  55.60,  45.90,  42.60,  41.00,  40.70,  40.70,  40.70,  40.90,  41.70,  41.60,  40.80,  40.00,  40.60,  40.70,  38.80,  36.80,  34.80,  33.60,  32.80,  32.70,  32.60,  31.90,  30.90,  29.90,  29.00,  28.80,  27.60,  26.00,  25.60,  24.00,  23.80,  23.80,  23.70,  23.80,  23.70,  23.60,  23.00,  22.90,  22.80
</code>
Answer: Various comments:

Your discussion is free with technical words such as "significance" and "confidence", but note that the relationship with the formal senses of those terms is tenuous. In particular, you have no formally defined generating process and your use of multipliers based on standard normal distributions includes no adjustments for dependence structure, either in your raw data or as smoothed. I don't have concrete suggestions for alternative inference procedures, not least because it seems that you are much better off regarding what you do as just exploratory or heuristic. But it's best in any report to put " " around those terms at least once, or even to avoid them altogether. Note that in my view exploratory and heuristic are not dirty words. Finding structure in data is one of the most vital parts of statistical science.
A heuristic of smoothing the data a little and then looking for jumps in the smoothed series sounds a good idea, although what you are doing already sounds a bit too complicated. Much depends on the sociology of your situation, i.e. who you have to explain this to and what they will understand or expect.
Your method of presenting the data and your general pitch implies to me that you are looking at temperature changes with depth e.g. below the land surface. If so, beware anyone trying to sell you anything based on time series analysis, as if these data are spatial, heat can (presumably) be propagated in both directions, unlike a time series in which past and future are quite unequal.
A simple exploratory plot of one-step temperature changes (temperature $-$ previous temperature) in your sample series suggests that most of the biggest temperature changes are short ramps over 2-3 or so intervals. You may be able to relate that to the physics of your problem. This may have implications for your degree of smoothing, for example implying that Savitzky-Golay smoothing smooths too much.

Much depends on your relative emphasis on (a) detecting jumps as compared with say (b) detecting zones or epochs within which temperature (or temperature change) is approximately constant. There are literatures on both. For example, a previous thread at How can I group numerical data into naturally forming "brackets"? (e.g. income) describes procedures for dividing series into subseries. You get to choose on the trade-off between number of subseries and internal homogeneity. Here is an arbitrary sample result from such a procedure. The criterion was to look for subseries with minimum internal variability in temperature, as defined by a sum of squared deviations from the mean. The numerals 1, ..., 7 indicate subseries identified. You could apply the same method to temperature change, or any other series.
Comment: Thank you for the detailed insight. Your feedback on terminology is appreciated (not my area of expertise). The profile represents temperatures within a 24'-32' tank; beyond knowing that *most* increases should occur at or before the 10' mark (heat source), I do not have other concrete details available. I did replace the Savitzky–Golay filter with a Median filter that seems to be producing better results; however I think you raise an excellent last point and I will explore that further today.
Comment: This is a great answer and after much reading on cluster analysis and related concepts, it ultimately lead me down a different path to achieve the results I required. I opted to use inflection points along with the angles to break down and analyze the temperature profile with the accuracy required. Thanks for your help!


Why is the output different in both cases comparing floats

Question: PYTHON PROGRAM:
a = 0.2
if a == 0.2:
    print('*')

OUTPUT:
*

C PROGRAM:
#include <stdio.h>

int main(void)
{
    float a = 0.2;
    if(a == 0.2)
    {
        puts("*");
    }
}

OUTPUT:

Why is the output different in both cases? Is there a difference between the working of == operator?
Comment: In the `a == 0.2` in C, 0.2 is a _double_ while a is a _float_. Change it to `if(a == 0.2f) { .. }`. If that works / changes the behavior, search for "Is floating point math broken?"
Comment: [Very related question](https://stackoverflow.com/questions/588004/is-floating-point-math-broken). Short summary: Don't ever do "exact comparison" using floating point values. Always use an *epsilon*.
Comment: Also, it's often hard to do direct comparison between different languages with different semantics. Even for small limited areas where syntax might seem the same, the behavior might be vastly different.
Comment: @NSR No not even in Python can it be reliable. Lets say you have made two different functions that take different input and do some calculation on the input. Using a specific set of inputs should mathematically produce an exact equal result, but due to inaccuracies in representation and possibly compounded rounding errors, the two results are not *exactly* equal, just very close.
Comment: Alse [Why Are Floating Point Numbers Inaccurate?](https://stackoverflow.com/questions/21895756/why-are-floating-point-numbers-inaccurate)
Comment: @Some programmer dude Not even in Python?
Comment: @Some programmer dude Yeah, but while printing Python just approximates, right?
Comment: @Someprogrammerdude "Don't ever do "exact comparison" using floating point values. Always use an epsilon." is an overstatement.  Comment space too small for a thorough explanation.
Answer: It is because the types float and double have different width reserved for the mantissa. The type double can represent a floating number more precisely. In this case that matters as 0.2 can't be represented exactly and has a very slightly different representation when stored as a double vs. a float.
In the condition
if(a == 0.2)

the left operand has the type float while the right operand has the type double, as the default type of a number literal with a "." in C is a double.
So change the declaration the following way
double a = 0.2;

Or alternatively change the condition like
if(a == 0.2f)

Here is a demonstrative program
#include <stdio.h>

int main(void)
{
    float a1 = 0.2;

    if ( a1 == 0.2f )
    {
        puts( "*" );
    }

    double a2 = 0.2;

    if ( a2 == 0.2 )
    {
        puts( "*" );
    }
}

Its output is
*
*


Configuring Aegir to Set Custom File Paths

Question: What is the best way to modify the file paths for a site deployed using Aegir?
Aegir installs sites with the file paths pointing in this format:
sites/<sitename>/files

(This path being relative to the site's platform)
What is the best way to change it to a NAS mount, mounted at, say, /var/nasmount ? For e.g., for a site called firstsite.com, the file paths would be:
/var/nasmount/firstsite.com/files
/var/nasmount/firstsite.com/files/tmp

and so on, along with the the corresponding entries in the site's apache conf file.
These lines appear in /var/aegir/.drush/provision/platform/provision_drupal_settings.tpl.php, but I am reluctant to change here as this is from 'core' Aegir (if you will):
  $conf['file_directory_path'] = 'sites/<?php print $this->uri ?>/files';
  $conf['file_directory_temp'] = 'sites/<?php print $this->uri ?>/files/tmp';

Should this be done in:

provision_drupal_settings.tpl.php file above?
Somewhere else?

Ideally, this change would be done once so all sites deployed via Aegir have this in place, and these changes survive the re-verify process of a site and/or platform.
Thanks!
Answer: The answer is to mv siltes/firstsite.com/files to /var/nasmount, then create a symbolic link to it.
(from inside your sites/firstsites.com directory,  mkdir /var/nasmount/firstsite.com, mv ./files /var/nasmount/firstsite.com, ln -s /var/nasmount/firstsite.com/files files).
I would put this in a script so this becomes a drush command, then make it run at the end of the aegir create a site process.
I'm not an expert of where to put this script and/or how to create it.
Comment: Thanks iStryker, looks like we're heading the sym link route. We tried editing the provision_drupal_settings.tpl.php file but to no avail -- the file paths got over-written.
Comment: I am trying to do the exact same thing. Can anyone please tell me where and in which hook to put this script?
Answer: Edit /var/aegir/config/includes/global.inc and put in it:
<?php
$conf['file_directory_path'] = '/var/nasmount/<?php print $this->uri ?>/files';
$conf['file_directory_temp'] = '/var/nasmount/<?php print $this->uri ?>/files/tmp';

This will be applied globally to all sites. If you only want to make this change for one or two sites, create a new file called 'local.settings.php' in the site dir (same level as settings.php) and put the change in there.
This is the best way of doing this without hacking templates or messing up your system with symlinks.
See also the documentation at http://community.aegirproject.org/node/71
Comment: Thanks @mig5. We tried this exactly (made the above change, verified the platform, verified the site), and it did not work. It appears that the file paths get overwritten later ...  ?
Comment: That should not happen if you edited the global.inc or the local.settings.php (note: *not* the settings.php) - there is nothing in the Aegir codebase that writes to these files (they are yours to override). If your local.settings.php / global.inc files got overwritten, please open a ticket at [drupal.org](http://drupal.org/project/provision)
Answer: If you edit edit /var/aegir/config/includes/global.inc, as mig5 is proposing, and the sites are running on remote servers (not on Aegir master), then you should additionally verify the remote servers in Aegir, in order to get the changes in /config synchronised on remote servers.
See http://community.aegirproject.org/node/71


AngularJS testing error on adding name parameter to scope?

Question: Error received when running karma start configs/karma.conf.js:
INFO [PhantomJS 1.9.2 (Linux)]: Connected on socket PK-IbFeI1qpFq0fix6WI
PhantomJS 1.9.2 (Linux) IndexController should add name parameter to scope FAILED
Error: [$injector:modulerr] http://errors.angularjs.org/1.2.6/$injector/modulerr?p0=myapp.controllers&p1=Error%3A%20%5B%24injector%3Anomod%5D%20http%3A%2F%2Ferrors.angularjs.org%2F1.2.6%2F%24injector%2Fnomod%3Fp0%3Dmyapp.controllers%0A%20%20%20%20at%20http%3A%2F%2Flocalhost%3A9877%2Fbase%2Fpublic%2Fapp%2Flib%2Fangular%2Fangular.min.js%3F1387689655000%3A20%0A%20%20%20%20at%20http%3A%2F%2Flocalhost%3A9877%2Fbase%2Fpublic%2Fapp%2Flib%2Fangular%2Fangular.min.js%3F1387689655000%3A21%0A%20%20%20%20at%20http%3A%2F%2Flocalhost%3A9877%2Fbase%2Fpublic%2Fapp%2Flib%2Fangular%2Fangular.min.js%3F1387689655000%3A29
    at /home/Projects/abbrevi8-karma-tests/public/app/lib/angular/angular.min.js:29
TypeError: 'undefined' is not an object (evaluating 'scope.name')
    at /home/Projects/abbrevi8-karma-tests/test/unit/controllerSpec.js:19
PhantomJS 1.9.2 (Linux): Executed 1 of 1 (1 FAILED) ERROR (0.178 secs / 0.006 secs)

public/app/controllers.js
var myApp = angular.module('myApp', []);

myApp.controller('IndexController', function ($scope) {
        $scope.name = 'bob';
});

test/unit/controllerSpec.js
'use strict';

describe('IndexController', function() {
    //initialise module
    beforeEach(module('myapp.controllers'));

    //params initialised in scope for tests
    var ctrl, scope;

    beforeEach(inject(function($controller) {
        //get controller from $controller provider
        scope = {};
        ctrl = $controller('IndexController', {
            $scope: scope
        });
    }));

    it ('should add name parameter to scope', function() {
        expect(scope.name).toBeDefined();
    });
});

Full project (same as above, only with my karma.conf.js and setup.sh): https://gist.github.com/AlecTaylor/8078807
Answer: Try:
beforeEach(module('myApp'));
instead of:
beforeEach(module('myapp.controllers'));

Comment: Thanks for that. Annoyed with the blog post that told me to write the other >.<
Comment: Ha! I didn't notice that.
Answer: I'm just guessing here, as I don't have Phantom to run on Linux... but $scope is expected to be a Scope object.
Try injecting $rootScope, and either use that, or create a $new() child scope from it:
beforeEach(inject(function($controller, $rootScope) {
    //get controller from $controller provider
    scope = $rootScope.$new();
    ctrl = $controller('IndexController', {
        $scope: scope
    });
}));

Comment: Thanks, but that still hasn't fixed the problem. `scope` and `ctrl` are still resolved as `undefined` :\


How to establish asymptotic normality of unbiased estimator and find asympotic variance?

Question: Given a linear regression model with deterministic regressors $$y_{i} = x'_{i}\beta + \epsilon_i, \quad \epsilon \sim (0,\sigma^2) i.i.d., \quad i = 1, ...,n, \quad with \quad \mu_4 := \quad \mathbb{E}(\epsilon{^4_1})< \infty $$
where the regressor matrix has full rank $k$, and $e$ denotes the vector OLS residuals. How do I establish the asymptotic normality of of the unbiased estimator $\hat{\sigma}^2 = \frac{e'e}{n-k} \quad of \quad \sigma^2 $, that is, $$ \sqrt{n}(\hat{\sigma}^2{-\sigma^2}) \xrightarrow{d} N(0,v)  $$ and find the asympotic variance $v$, using the Central Limit Theorem?
Answer: Since $k$ is fixed and $\dfrac{n}{n-k} \rightarrow 1$, it's not W.L.O.G to consider the estimator, $\hat{\sigma}^2 = \dfrac{1}{n}e^Te$ instead. After some matrix algebra, you can write,
$$ \dfrac{1}{n}e^Te = \dfrac{1}{n}\epsilon^T\epsilon - \dfrac{1}{n}\epsilon^TH\epsilon$$
where $\epsilon$ is the vector containing the errors, $\epsilon_i$ and $H = X(X^TX)^{-1}X^T$ is the projection matrix in the OLS estimator. The second term converges to $0$ in probability by Chebyshev's inequality,
$$\begin{eqnarray}
P\left(\frac{1}{n}\epsilon^TH\epsilon > \delta\right) \leq \dfrac{1}{n\delta}E[\epsilon^TH\epsilon] = \dfrac{\sigma^2}{n\delta}\text{tr}(H) \rightarrow 0, \;\; \forall \delta > 0
\end{eqnarray}
$$
So you only have to consider the first term,
$$ \dfrac{1}{n}\epsilon^T\epsilon = \dfrac{1}{n}\sum\limits_{i=1}^n\epsilon_i^2 $$
Since $E[\epsilon_i^4] < \infty$, you can directly apply the CLT,
$$ \sqrt{n}(\hat{\sigma}^2 - \sigma^2) = \sqrt{n}\left(\dfrac{1}{n}\sum\limits_{i=1}^n\epsilon_i^2 - \sigma^2\right) \xrightarrow{D} N(0, v)$$
where $v = E[\epsilon_i^4] - \sigma^4$.


VPS inaccessible after a few seconds of boot

Question: I got a debian VPS last day and installed on it a few things (nodejs, nginx, redis) and as of last night it's mostly inaccessible by ping, after each reboot it works for a few seconds and then goes to blackout, I have no idea what happens. With each reboot I have a few seconds to ssh to it and do a command or two.
I know this is a very vague question, but it's because I don't know what I'm looking for. If you have any idea of what might be the cause, please say.
UPDATE:
It was a network issue after all, after restarting networking (eth0) daemon for a few seconds everything is fine, but after that I can't even ping the gateway!
iptables are empty, no policy is added.
Comment: Use the virtual console to log in and diagnose the issue. If you have a VPS that doesn't provide one, get rid of it and get another one that does..
Answer: Sounds like a firewall issue. I would try:
Create file:
/root/stop.fw
<code>#!/bin/sh
echo "Stopping firewall and allowing everyone..."
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
</code>
then:
<code>$ sudo chmod +x /root/stop.fw
$ sudo /root/stop.fw
$ sudo iptables -L
</code>
ALSO TRY:
/etc/ssh/sshd_config:
SET:
<code>TCPKeepAlive yes
ClientAliveInterval 30
ClientAliveCountMax 999999
</code>
restart SSHd
Comment: nmap says Note: Host seems down, this is why I didn't think of it as a network issue. Any idea?


merge two csv file using python

Question: i have two csv with common headers i need to merge two csv file into one csv file column wise
my first csv file looks like
 step   item    path_of_logfile value
adi landlord    /home/abcd.txt  1000
vc  abcd    /hom/efgh.txt   2000

another csv  looks like this
step    item    path_of_logfile value
adi landlord    /home/txt        4500
VC  abcd    /home/efgh.txt      3000
DC  abcd    /home/efgh.txt      pass

and desired output is
step item path_of_logfile value path_of_logfile value
 adi  landlord  /home/abcd.txt 4500 /home/abcd.txt1000
 VC abcd     /home/efgh.txt 3000/home/efgh.txt 2000
 DC   abcd     /home/efgh.txt pass

i did through python using pandas
code
   import pandas as pd
  first = pd.read_csv('1st.csv')
  second = pd.read_csv('2nd.csv')
  merged = pd.concat([first, second], axis=1)

please help me to get desired output csvfile
Answer: You are close.  You need to merge the data frames instead of concatenating them.  You want to merge on the same key value in this case.
Your problem is a bit more complicated (not much) because your .csv files are not in standard form as you have multiple spaces between values instead of commas (the 'c' in csv).
This works.  Of note, you have a lowercase 'vc' and and upper 'VC' in the other file that don't match.  Perhaps that is a typo?
import pandas as pd
first = pd.read_csv('1st.csv', sep= '\s+')
second = pd.read_csv('2nd.csv', sep= '\s+')

merged = first.merge(second, how='outer', on='step')

print(merged)

merged.to_csv('merged.csv', na_rep='pass', index=False)

Yields:
  step    item_x path_of_logfile_x  value_x    item_y path_of_logfile_y value_y
0  adi  landlord    /home/abcd.txt   1000.0  landlord         /home/txt    4500
1   vc      abcd     /hom/efgh.txt   2000.0       NaN               NaN     NaN
2   VC       NaN               NaN      NaN      abcd    /home/efgh.txt    3000
3   DC       NaN               NaN      NaN      abcd    /home/efgh.txt    pass

CSV file:
step,item_x,path_of_logfile_x,value_x,item_y,path_of_logfile_y,value_y
adi,landlord,/home/abcd.txt,1000.0,landlord,/home/txt,4500
vc,abcd,/hom/efgh.txt,2000.0,pass,pass,pass
VC,pass,pass,pass,abcd,/home/efgh.txt,3000
DC,pass,pass,pass,abcd,/home/efgh.txt,pass

Comment: If your match criteria is both 'step' and 'item' you can pass that in as a list for the `on` field in merge.
Comment: Can we have  another method without using pandas
Comment: sure.  You used pandas in the orig, so it seemed like that was the path you wanted.  You need to update your post with how you want to join the files.  are you just matching "step"?  Also, give it a try with basic reading and dictionaries....
Comment: Can u  provide me the code so that I can use that as a start for my learning
Answer: Code:
import pandas as pd
first = pd.read_csv('1st.csv', sep='    ', engine='python')
second = pd.read_csv('2nd.csv', sep='    ', engine='python')
merged = pd.concat([second, first] ,axis=1, keys = 'step')
print(merged)

Output:
s                                    t
  step      item path_of_logfile value step      item path_of_logfile   value
0  adi  landlord       /home/txt  4500  adi  landlord  /home/abcd.txt  1000.0
1   VC      abcd  /home/efgh.txt  3000   vc      abcd   /home/fgh.txt  2000.0
2   DC      abcd  /home/efgh.txt  pass  NaN       NaN             NaN     NaN

You need to define the keys inside pd.concat() so that it can target the column on which it have to perform the concatenation. Also, i used engine='python' here otherwise it will throw a warning. I am also considering here that the data here is having 4 space separator that's why i have assigned 4 spaces to sep method.
Comment: Can we do without using pandas
Comment: @humblefool78 i do not see any other method to perform the concatenation because dataframe is still needed to operate concatenation, however i do not know if there is any other method or not and if there is one then it will be great to learn that new method

merge two csv file using python


Django - Filter the prefetch_related queryset

Question: I am trying to reduce my complexity by doing the following. I am trying to get all the teachers in active classrooms.
teacher/models.py:

Teacher(models.Model):
    name = models.CharField(max_length=300)

classroom/models.py:

Classroom(models.Model):
    name = models.CharField(max_length=300)
    teacher = models.ForeignKey(Teacher)
    students = models.ManyToManyField(Student)
    status = models.CharField(max_length=50)

admin/views.py

teachers = Teacher.objects.prefetch_related(Prefetch('classroom_set',queryset=Classroom.objects.filter(status='Active'))

for teacher in teachers:
    classrooms = teacher.all()
    # run functions

By doing this I get teachers with classrooms. But it also returns teachers with no active classrooms(empty list) which I don't want. Because of this, I have to loop around thousands of teachers with empty classroom_set.  Is there any way I can remove those teachers whose classroom_set is [ ]?
This is my original question -
Django multiple queries with foreign keys
Thanks
Comment: What if a teacher has one active and one passive classroom?
Comment: Can you provide the models to this?
Comment: What is "classroom" in this? What defines if a classroom is active?
Comment: Shouldn't this be `Classroom` instead of `Class`?
Comment: @WillemVanOnsem Yeah my bad. Guys Classroom and Teacher are two different apps
Comment: @AnkitSingh: the fact that there are two different apps does not matter.
Answer: If you want all teachers with at least one related active class, you do not need to prefetch these, you can filter on the related objects, like:
Teacher.objects.filter(class__status='Active').distinct()
If you want to filter the classroom_set as well, you need to combine filtering as well as .prefetch_related:
from django.db.models import Prefetch

Teacher.objects.filter(
    class__status='Active'
).prefetch_related(
    Prefetch('class_set', queryset=Class.objects.filter(status='Active'))
).distinct()
Here we thus will filter like:
SELECT DISTINCT teacher.*
FROM teacher
JOIN class on class.teacher_id = teacher.id
WHERE class.status = 'Active'


Cloning Two Windows Servers

Question: Basically I have a server running MSSQL, MYSQL, APACHE, IIS, ORACLE, PERL and other windows services. I am looking for some hardware solution that would allow me to clone a server on to a back up and provide some fail-over mechanism.
Can anyone point me into a direction for such a venture? What are the hardware needs?
How does the server automatically switch to the other? What is the hardware required?
I am novice in this area.
Proposed Venture:
http://i52.tinypic.com/5f56w1.png
Comment: how much down time can you afford?
Answer: Wow, you're making life hard for yourself there - all that on one machine?
Anyway, the problem you'll run into is that while all of that is clusterable it's done by lots of other bits of code that will only add to the complexity/fragility of your existing, already non-optimal, setup.
One thing you could think about doing is setting up a pair of virtualisation hosts using either VMWare's ESXi, MS's Hyper-V or KVM. Then build VMs for each of those functions, let the two servers spread the VMs between them both and then setup either a high-availability type system such as VMWare's HA or Hyper-V's clustering capability - or go the whole hog and run them in fault-tolerent mode (if Hyper-V or KVM have those features, ESXi does).
This would be a very flexible and modern way of doing what you need, plus spreading the complexity and supportability of your setup out into a more modular approach.
Of course none of this is free and you need to understand how much instant failover is worth to you compared to pretty quick failover but ultimately that's the direction I'd personally be looking at.
Comment: Thanks for your insight. If i was to run VMs and the main server fails I would be in problems. The server runs mission critical services. I am thinking about replicating two physical server via File and Database replications and using double take down for failover ip switching/hostname.
Comment: But you could set things up to help with outages
Answer: We achieve your desired solution by using 2 mirrored SANs and 2 ESX host machines in high availability. The storage for the VMs is on the SANs. If there is a hardware failure on the ESX machines the VMs running on that host will automatically fail over to the operable host. The only negative is that the VM will experience a reboot during the fail over. The mirrored SANs allows for either SAN to fail and both ESX hosts will be able to communicate to the functioning SAN.
Hardware & Software used:

2x HP G5 DL380
2x HP Storeworks Lefthand SANS
2x HP ProCurve 2810 Linked via Fiber
VMware vSphere 4

Cloning Two Windows Servers


How does one calculate $\text{Var}(E[Y|X])$? and other related quantities

Question: I know how to calculate $E[Y|X=x]$ from knowing how to calculate $P(Y|X=x)$.
What I don't understand is the meaning of $P(Y=y|X)$. Wouldn't be $P(X)$ = 1 since it is taken over the entire random variable? Or is it constraining X to some implicit set like $x_1 < X < x_2$, so you can then apply Bayes' theorem?
I have seen people suggest calculating $E[Y|X] = \int y p(y|X) dy$ which I don't understand.
My understanding on calculating $\text{Var}(E[Y|X)$ as it stands is calculating $E[Y|X=x]$ and then marginalizing over X for the variance part, but I think I am missing something when it comes to $P(y|X)$ or going from $E[Y|X]$ to $\text{Var}(E[Y|X)$.
Comment: Related: https://stats.stackexchange.com/questions/601921/formula-of-conditional-probability-when-we-have-discrete-and-continuous-random-v
Comment: Is there actually a difference between the event Y|X and Y|X=x? If X is "given" do we not pretend to know its specific value?
Answer: When you specify the conditional expectation (or other moment) of a random variable $Y$ given a specific value of the conditioning variable $X=x$, the result is a function of the argument $x$.  In generic function notation, we have:
$$\mathbb{E}(Y|X=x) = f(x).$$
When you replace the argument value $x$ with the actual random variable $X$ (by declining to specify a value for the conditioning variable) the result is a function of the random variable $X$.  In generic function notation, we have:
$$\mathbb{E}(Y|X) = f(X).$$
Because this latter form is a function of a random variable, it is itself a random variable.  That is, the conditional expectation of $Y$ conditional on $X$ is a random variable.  The distribution of this random variable is obtained through the appropriate transformation rules using the distribution of $X$.

An example: Suppose we have $Y|X=x \sim \text{N}(2x, 1)$ and $X \sim \text{N}(\mu, 1)$.  Then we have the conditional expectation function:
$$f(x) \equiv \mathbb{E}(Y|X=x) = 2x.$$
The random variable version of this is:
$$f(X) = \mathbb{E}(Y|X) = 2X \sim \text{N}(2 \mu, 4).$$
So in this case you would have:
$$\mathbb{V}(\mathbb{E}(Y|X)) = \mathbb{V}(2X) = 4.$$


Pasting Html text in Edittext

Question: How to restrict HTML content while pasting some text from a website to our EditText?.
Example I copy some text from a website, when I paste the text in my EditText the HTML font and text size also comes along. What i want is to restrict any HTML content while I paste the text. Tried several methods using TextWatcher and InputFilters, but none of them are working. Whatsapp is handling it somehow. Please help.
Edit: The string can be converted to non-HTML but how to make changes in EditText?
Comment: @Nj I do not think this question is duplicate of tagged post. Please read once before marking duplicate to any answer.
Answer: This is likely a Rich Text Formatting issue.
According to this question you can add a TextWatcher to your EditText and use this method:
public void afterTextChanged(Editable string)
{
    CharacterStyle[] toBeRemovedSpans = string.getSpans(0, string.length(),
                                                MetricAffectingSpan.class);
    for (int index = 0; index < toBeRemovedSpans; index++)
        string.removeSpan(toBeRemovedSpans[index]);
    }
}

Reference:
https://stackoverflow.com/a/24777290/5457878
Comment: It still leaves all the spaces around the text
Comment: Could you give a small example?
Comment: if i try to paste a text from website. The extra empty spaces still show up.
Comment: It's ugly, but you could use `trim()` or a `Regex` to remove whitespace.

Pasting Html text in Edittext


Ajax callback is not working on views exposed filters

Question: I am implementing dependent filters in views exposed filter for content:datasource search(Solr search). I have a requirement that in exposed dropdown field when a category is selected, by ajax i need to fetch the sub-categories and show up in another dropdown.For this I tried like below but its not going to callback function only. There is nothing logging in the network console also. Can anyone help me here to resolve.
use Drupal\Core\Ajax\AjaxResponse;
use Drupal\Core\Ajax\HtmlCommand;

function module_form_alter(&$form, &$form_state, $form_id) {
$form['field_categories']['#ajax'] = array(
        'callback' => 'module_get_subcategories',
        // 'callback' => '::module_get_subcategories',
        'event' => 'change',
        'wrapper' => 'edit-field-subcategory',
        'progress' => array(
          'type' => 'throbber',
          'message' => NULL,
        ),
      );
}

function module_get_subcategories(&$form, FormStateInterface $form_state) {
   $response = new AjaxResponse();
   $message = 'test';
   $response->addCommand(new HtmlCommand('#edit-field-subcategory', $message));
   return $response;
}

Answer: Something is weird in the code,
What is that return statement in the last two lines?
For the ajax, try using $form['field_categories']['widget']['#ajax']
Make sure that you have cleared the cache too and that your module is actually called module.
Comment: Hi @Ahmed, I have removed last line from my code. I tried with $form['field_categories']['widget']['#ajax'] this also and cleared cache. but still not calling.
Comment: What is your module name please?
Comment: I'm also facing the same problem. Trying to make an #Ajax callback to views form exposed filter.


How to have a scrollable list inside a SliverToBoxAdapter

Question: I am using the below SliverToBoxAdapter inside the CustomSrollView:
<code>        SliverToBoxAdapter(
            child: Stack(
            children: <Widget>[
              Container(
                width: double.infinity,
                height: 50,
                decoration: BoxDecoration(color: pink),
              ),
              Padding(
                padding: const EdgeInsets.symmetric(horizontal: 30),
                child: ListView.builder(
                  padding: const EdgeInsets.all(0),
                  shrinkWrap: true,
                  itemBuilder: (context, index) {
                    return buildSongRow(songs[index]);
                  },
                  itemCount: songs.length,
                ),
              )
            ],
          ))
</code>
The issues is that the list doesn't scroll, which is obvious as it has box constraints. I am using SliverToBoxAdapter here as I have to stack my list over some Container at the top of the list. How can I make my list scroll?
If it cannot be done using SliverToBoxAdapter, do I have any other options?
Answer: You can make Listview scrollable by adding the same scrollController of the parent CustomScrollView.
<code>class ScrollExample extends StatelessWidget {
  final scrollController = ScrollController();
  @override
  Widget build(BuildContext context) {
    return CustomScrollView(
      //  add scrollController to sliver root
      controller: scrollController,
      slivers: <Widget>[
        SliverToBoxAdapter(
          child: Card(
            child: ListView.builder(
              //  add the same scrollController here
            controller: scrollController,
              shrinkWrap: true,
              itemCount: 19,
              itemBuilder: (BuildContext context, int index) {
                return Text("ibnShamas");
              },
            ),
          ),
        ),
      ],
    );
  }
}
</code>


Is it bad form to join against a subselect like this, to avoid duplicate bind variables?

Question: Here's a simplified sample query, done the "traditional" way (sample queries tested in PostgreSQL 8.3):
PREPARE a AS SELECT *
FROM users
WHERE uid=$1
    OR parent=$2;
EXECUTE a(0,0);

The trouble is, it's cumbersome to pass the same bind variable twice (moreso for more complex queries when it's needed half a dozen times or more).
Of course, SQL allows the same bind variable to used multiple times in the same query:
PREPARE b AS SELECT *
FROM users
WHERE uid=$1
    OR parent=$1;
EXECUTE b(0);

But many DB API layers don't support this (or don't easily), thus making it even more cumbersome to use this approach.
I can get around this by adding a JOIN against a sub-select as such:
PREPARE c AS SELECT uids.*
FROM users
JOIN (SELECT $1::INT AS uid) AS x ON true
WHERE uids.uid=x.uid
    OR uids.parent=x.uid;
EXECUTE c(0);

Explain shows that the latter option has, as one would expect, the most complex execution plan, however, at least in this example, it consistently executes faster than a, and barely slower than b (although preparing the query takes longer than either of the other two options).
So my question is:
Is this sort of JOIN/sub-select construct considered good or bad form for more complex queries, as a way to avoid repeating bind variables?
Answer: Not a direct answer to your question, but in this particular case, you should be able to rewrite your query like this:
PREPARE b AS SELECT *
FROM users
WHERE $1 IN (uid, parent);

EXECUTE b(0);

Comment: Indeed, that would work *in this case* (it wouldn't work in the more complex case I actually care about :) ), but even so it won't use indexes on the `uid` and `parent` columns, if I'm not mistaken (and if my rudimentary tests are accurate).
Comment: @Flimzy Logically, it should be the same thing as `uid=$1 OR parent=$1`, so it should be executed in the same way. That being said, perhaps you have hit some peculiarity of ProstgreSQL query optimizer? In any case, I recommend measuring on realistic amounts of data to be sure - I'm no expert on PostgreSQL optimizer, but many DBMSes will resort to table scans on small tables even when indexes are available.
Comment: Indeed... I did my tests against a larger table.  But, sadly, that wasn't the only variable that changed, so I can't be sure my test was valid.  This is a good strategy to keep in my bag-o-tricks.


Propriedade computada Polymer

Question: Saudações.
Meu problema é o seguinte. Tenho uma página de detalhes do produto. Ela é acessada após clicar num produto específico em outra página de listagem, é exibido o conteúdo através do ID do produto. Até aí tudo bem. Dentro dessa página de detalhe, tenho um elemento  onde coloco informações sobre o produto naquela plataforma de venda. Esse elemento pede 5 propriedades básicas (logo, nome, descricao, url e preco). Consigo mostrar os dados sem problemas quando faço o caminho básico do firebase {{item.platform.plataforma.value}} , porém quero manipular o valor que está vindo e formatá-lo, mas quando faço a propriedade computada ela não retorna valor. Estou usando a mesma tática para a listagem dos produtos e tem funcionado por lá.
<template is="dom-if" if={{item.platform.teste.url}}>
   <box-plataforma logo="*imagem*" nome="{{nome}}" descricao="{{descricao}}" url="{{item.platform.teste.url}}" preco="{{item.platform.teste.value}}"></box-plataforma>
</template>

Quando utilizo o exemplo acima o valor é exibido normalmente.
Porém quando uso o código abaixo é retornado valor vazio para o preço que é o que desejo formatar.
<template is="dom-if" if={{item.platform.teste.url}}>
   <box-plataforma logo="*imagem*" nome="{{nome}}" descricao="{{descricao}}" url="{{item.platform.teste.url}}" preco="{{valueTeste}}"></box-plataforma>
</template>

O valor que deixo no properties é o seguinte:
valueTeste: {
  type: String,
  computed: 'pricePlatform(item.platform.teste.value)'
}

E a função para transformar o valor é:
pricePlatform(value) {
 if (value == undefined) {
  value = 0;
 }
 return (value != 0) ? '$ ' + value.toFixed(2).replace('.', ',') : 'Grátis'
}

Desde já grato, e caso faltem informações ou não esteja bem claro o que desejo peço desculpas, não sou muito experiente e não sei muitos termos específicos.
Answer: Consegui resolver, ao invés de criar propriedades computadas fiz uma função
_pricePlatform(preco) {
    if (preco == undefined) {
      preco = 0;
    }
    return (preco != 0) ? 'R$ ' + preco.toFixed(2).replace('.', ',') : 'Grátis'
  }

e no elemento usei algo que a documentação do Polymer chama de "Computed binding" aqui
<box-plataforma logo="*imagem*" nome="Teste" descricao="Teste" url="{{item.platform.teste.url}}" preco="{{_pricePlatform(item.platform.teste.value)}}"></box-plataforma>

Propriedade computada Polymer


What statistical test to use, Dichotmous IV and DV, with multiple sets of data

Question: I have previously run an experiment relating to a number of measurements from the internet.
I have run the experiment from multiple hosts / participants, multiple times. ie. several sets of data distributed over time per host.
The data I have concerns connections to a collection of hosts conducted in two separate ways. During these connections. A binary event does or does not happen.
I would like to deduce whether the choice of method to connect influences whether the event during the connection happens.
I have already looked at logistic regression models, however I am unsure if this is the correct approach.
Comment: Please change your title to something more informative.
Comment: And also include some statistical methods tags in addition to r tag
Answer: A multilevel logistic regression is necessary because of the multiple participants and multiple time measures. I am not sure what the model would look like given the few details, but it is a good starting point.
Comment: Would it also be possible to average the results over time allowing for more simplistic analysis through a paired t-test or repeated measures?

That is for a given participant,on average event A happened on N connections?
Comment: If you only want a really rough idea of the results for yourself, you can pooled time measures and see with logistic regression. You will violate many statistical assumptions and pooled different (inter and intra) variances together and potentially bias estimator. If it is for a scientific or commercial purpose, I would go for the appropriate analysis.


Sharing private data and methods between classes

Question: I have a bunch of classes for a specific feature. Only one of the classes should be instantiated by user code as the others are "helper" classes. However, these helper classes do expose useful information that is user code consumable.
So I could do with the constructors of the helper classes to only be accessible from the main class.
Likewise, it would be useful to have methods and data accessible only from the appropriate classes, and not from the 'user code'.
Solutions I've thought of so far:
Place all the classes for a feature into a single file and use 'static' to create variables that are local to only that file. This would work as there will only ever be a single instance of the main class and all of the 'helper' classes, but it doesn't seem a "nice" solution.
Have the 'helper classes' friend the main class. This would work also, but also give the main class access to all private data in the helper classes which is not ideal.
Inheritance from a virtual base class: Thinking this is possibly the best solution. Shared data via 'protected', private data stays private and public data is exposed. Not punched any holes in this theory yet. Need to consider performance aspect though as this is an embedded system with a clock speed of 72Mhz.
Comment: I'm not an expert, but I think considering virtual functions in a embedded system, wanting performance, is not a good idea.
Answer: Use Composition.
Inherited classes should only inherit functionality and not inherit state. Protected members/functions will possibly divide the state logic into two units. I would say use composition, and access functionality via public functions.
Regrading that your classes will never be good for the user you never know, so I would say don't restrict yourself.
Comment: So basically have the helper classes in the private section of the main class and then have the main class access them on behalf of the user via public methods?
Comment: yup, that's actually the "correct" design. This way you still have encapsulation, you don't restrict yourself and you don't have redundant coupling.


Associar ApplicationUser do Identity à minha classe PessoaFisica

Question: Meu sistema Asp.Net MVC usa o Identity na sua forma default, com algumas personalizações simples. Também possuo uma tabela de Pessoas Físicas (que herda algumas informações de Pessoas, mas acho que isso não vem ao caso).
Gostaria de associar o ApplicationUser à PessoaFisica, de forma que todo ApplicationUser tenha uma PessoaFisica (PessoaFisica podem ter ou não um ApplicationUser).

public class ApplicationUser : IdentityUser
{
   [ForeignKey("PessoaFisica")]
   public int PessoaFisicaId { get; set; }

   public virtual PessoaFisica PessoaFisica { get; set; }

   public async Task GenerateUserIdentityAsync(UserManager manager)
   {
      ...
    }
}

Na classe PessoaFisica tenho o seguinte:

[Table("Pessoas")]
public partial class PessoaFisica : Pessoa
{
   public int Id { get; set; }

   public string Nome { get; set; }

   ...

   public virtual ApplicationUser Usuario { get; set; }
}

O erro que obtenho é:

One or more validation errors were detected during model generation:
ApplicationUser_Pessoa_Source: : Multiplicity is not valid in Role 'ApplicationUser_PessoaFisica_Source' in relationship 'ApplicationUser_PessoaFisica'. Because the Dependent Role properties are not the key properties, the upper bound of the multiplicity of the Dependent Role must be '*'.

Colocando o relacionamento com fluent API na model:

protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
   modelBuilder.Entity‹PessoaFisica›()
   .HasOptional(f => f.Usuario)
   .WithRequired(s => s.PessoaFisica);
}

Também tenho um erro, mas é outro:

Unable to determine the principal end of an association between the types 'WebApplication5.Models.PessoaFisica' and 'WebApplication5.Models.ApplicationUser'. The principal end of this association must be explicitly configured using either the relationship fluent API or data annotations.

O que estou fazendo errado?
Comment: O seu problema aí é que várias pessoas físicas podem se referir ao mesmo ApplicationUser.
Answer: Achei muito complicado implementar isso. Muitos erros relacionados ao Identity.
Acabei fazendo um relacionamento meio a forma, consultando minha tabela cada vez que quero consultar o usuário de uma pessoa física. O mesmo faço para consultar a pessoa física de um usuário. Espero que ajude alguém.
Classe ApplicationUser:

public class ApplicationUser : IdentityUser
{
   public int PessoaFisicaId { get; set; }
   public virtual PessoaFisica PessoaFisica
   {
      get
      {
         MinhaModel db = new MinhaModel();
         var pessoaFisica = db.PessoasFisicas.Find(PessoaFisicaId);
         return pessoaFisica;
      }
   }
   ...
}

Classe PessoaFisica:

public class PessoaFisica : Pessoa
{
   public string Cpf { get; set; }
   public virtual ApplicationUser Usuario
   {
      get
      {
         ApplicationDbContext db = new ApplicationDbContext();
         var usuario = db.Users.FirstOrDefault(u => u.PessoaFisicaId == Id);
         return usuario;
      }
   }
}

Uso em um Controller:

public ActionResult Details(int id)
{
   PessoaFisica pessoaFisica = db.PessoasFisicas.Find(id);
   var nomeUsuario = pessoaFisica.Usuario.UserName;
   return View(pessoaFisica);
}

Uso em um View:

@using Microsoft.AspNet.Identity
@using NomeAplicacao.Models
@using Microsoft.AspNet.Identity.EntityFramework
@{
    var manager = new UserManager(new UserStore(new ApplicationDbContext()));
    var usuario = manager.FindById(User.Identity.GetUserId());
}
…
@Html.ActionLink("Olá " + usuario.PessoaFisica.Nome + "!", "Index", "Manage", routeValues: null, htmlAttributes: new { title = "Manage" })
…


General Relationship Between Standard Error and Sample Size

Question: Suppose I empirically estimate the standard error of some statistic to be 10% (perhaps I do this by bootstrapping).
Now, I want to know how much I need to increase the sample size to reduce the error to 5%. The estimate and sample are arrived at through complex procedures, and theoretically determining the relationship between sample size and standard error is not feasible. However, I believe that the standard error decreases as sample sizes increases.
Is it plausible to assume that standard error is proportional to the inverse of the square root of n (based on the standard error of a sample mean using simple random sampling)?
se = s / sqrt( n )
Do standard errors behave (very) roughly in this way in general in relation to  sample size, regardless of the estimate and the sampling procedure? How bad of an assumption is this?
Answer:
regardless of the estimate and the sampling procedure?

No, you will not have a "root-n" effect regardless of those things, since at least some standard errors do not scale with $\sqrt{n}$.
Many do -- quite possibly all the ones you will be likely to use -- but that's not all of them.
For things that do scale with $\sqrt{n}$ then you expect to halve the standard error by quadrupling sample size. So (at least if we're ignoring sampling variation in the estimate of $\sigma$), that's probably what you need.
One example of something that isn't proportional to $\frac{1}{\sqrt{n}}$ is the standard error of a kernel density estimate when the bandwidth is itself chosen as a function of $n$. [For some common choices of bandwidth formula the standard error goes down as $n^{-2/5}$ instead.]
Comment: Interesting. Even for your example where the error is proportional to n^(-2/5), n^(-1/2) is a good enough approximation for my purposes. Thank you.
Comment: @Iggy25 Off the top of my head I can think of examples that scale as $n^{-1}$ and $n^{-1/3}$ that you could run into with real data analysis problems. It really depends on what classes of problems you can encounter. Even the suitability of ignoring the difference between $n^{-1/2}$ and $n^{-2/5}$ depends on what you're doing; I'd regard the situations in which I'd be prepared to ignore that difference as pretty limited.
Answer: Yes, generally, more samples would lead to smaller standard error. However, since adding more data is generally a random process, the decrease in standard error is not monotonic.
Consider the special case of Monte Carlo. Suppose you draw i.i.d samples from a distribution $F$, $X_1, X_2, \dots, X_n$. Suppose $Var_F(X_1) = \sigma^2$ and the mean is $E_F(X_1) = \mu$. Consider the task of estimating $\mu$ with the sample mean. Then,
$$ \hat{\mu} = \dfrac{1}{n} \sum_{i=1}^{n} X_i.$$
The variance of $\hat{\mu}$ can be estimated having obtained the sample variance,
$$\hat{\sigma}^2 = \dfrac{1}{n-1} \sum_{i=1}^{n}(X_i - \hat{\mu})^2. $$
Note that $\hat{\sigma}^2 \overset{a.s.}{\to} \sigma^2$ as $n\to \infty$. Due to this convergence, with more samples, we keep bettering our estimate of $\sigma^2$, and the general trend is a decrease in the standard error $\hat{\sigma}/\sqrt{n}$.
However, if we add a new observation $X_{n+1}$ that lies in the tails of $F$, it is possible for that the new estimate of the variance $\sigma^2$ is larger than the old estimate, so that $\hat{\sigma}/\sqrt{n+1}$ is larger that the previous standard error.
If I understand the rest of your question correctly, you want the standard error to reduce from 10 units to 5 units. Suppose you have $n_1$ samples that gave you a standard error of 10 units. At the cost of being very approximate, lets equate 10 to the standard deviation of the estimate.
$$10 = \dfrac{\sigma}{\sqrt{n_1}} $$
You want to find $n_2$ so that
$$5 = \dfrac{\sigma}{\sqrt{n_2}} $$
Dividing the two equations,
\begin{align*}
2 & = \sqrt{\dfrac{n_2}{n_1}}\\
\end{align*}
and now you can solve for $n_2$ since you know $n_1$. Of course, this will be an approximate sample size calculation since we used the truth, $\sigma$.
Comment: This is right, but it doesn't quite answer my question. Yes, it is possible for the estimated standard error to increase as sample size increases as per your explanation. However, the true standard deviation of the estimate (I think) cannot decrease. What I really want to know is, if my current standard error is 10% how many observations do I need to add before I can expect it to drop to 5%, and is my approach a reasonable way of getting this number.
Comment: Can you explain the usage of %s when using standard error. In other words, you say standard error of 10%, but 10% of what?
Comment: Sorry, I can see how that is confusing. My statistic itself is a ratio, and is expressed as a percentage. So I have been thinking of the standard error as a percentage as well. I am asking about reducing the standard error by some ratio - ie from X to 0.5X.
Comment: I have edited the answer, and hopefully this answers your question.


Zero divisors in an algebra with two generators

Question: Let $k$ be a field, and $R = k\langle x,y \mid x^2 = 0\rangle$. The elements $x$ and $y$ are not supposed to commute with each other. Is the only case where nonzero elements $a, b \in R$ satisfy $ab=0$ when $a\in Rx$ and $b\in xR$?
Comment: Welcome to MSE. Your question is phrased as an isolated problem, without any further information or context. This does not match [many users' quality standards](https://math.meta.stackexchange.com/q/9201), so it may attract downvotes, or be closed. To prevent that, please [edit] the question. [This](https://math.meta.stackexchange.com/q/9959) will help you recognize and resolve the issues. Concretely: please provide context, and include your work and thoughts on the problem. These changes can help in formulating more appropriate answers.
Answer: The question is answered in Example 9.3 in the paper Victor Camillo and Pace P. Nielsen, McCoy rings and zero-divisors, J. Pure Appl. Algebra 212 (2008), no. 3, 599–615.


VBA Select Pivot Table

Question: I am trying to select a pivot table using VBA, and have encountered an issue in that the pivot table may start at A3 (if there are no report filters activated) or A6 (if 3 report filters have been chosen) etc. Accordingly the pivot table will not always be on the same point on the worksheet. Clearing the pivot only removes all the data and report filters, although does not move the pivot table back up to its original position. Is there any way to use VBA to select the pivot table (so as to be able to copy and paste the data into a new worksheet) regardless of where it may be positioned on the worksheet?
Thanks.
UPDATE - I have since figured out how to do this. Its quite simple really:
Dim PT As PivotTable
Set PT = ActiveSheet.PivotTables(1)
PT.TableRange1.Select

Comment: Now that you have a handle on how to select the PivotTable, you may very well want to know how to reference the different parts of a PivotTable. Samples can be found here: http://peltiertech.com/WordPress/referencing-pivot-table-ranges-in-vba/
Comment: You should write it out as an answer instead of an edit, and confirm it.
Comment: Thanks - I will do, as soon as the time limit for answering my own questions expires!
Answer: Use the following code (given for a pivot table names PivotTable1):
Dim PT As PivotTable
Set PT = ActiveSheet.PivotTables(1)
PT.TableRange1.Select

VBA Select Pivot Table


MySQL: Changing a value in a column

Question: The title nearly explains itself to be honest. I am trying to change my Delete column from 0 to 1, depending on what player has been selected from my listplayers drop down.
My listPlayers.php file is working 100%. The players are showing up fine in the dropdown. I should add that the error is happening within my $sql variable.
<?php
include 'dbconnect.php';
$sql = "UPDATE players SET Delete = 1 WHERE playersID = $_POST[playersIDHidden]";

if(! mysql_query($sql, $conn))
{
    echo "Error " . mysql_error();
}

else
{
    if(mysql_affected_rows() != 0)
    {
        echo $_POST[pName] . " has been deleted
";
        echo "Person ID: " . $_POST ['playersID'] . "
";
        echo "Players Name: " . $_POST ['pName'] . "
";

        echo "Deletion complete";
    }

    else
    {
        echo "No records were changed";
    }
}

mysql_close($conn);
?>

<form action = "viewPlayers.php" method = "POST" />
    <input type = "submit" value = "View Players">
</form>

And finally my listPlayers.php that is showing my players names in the drop down.
<?php
include "dbconnect.php"; //Opening Database Connection

$sql = "SELECT * FROM `players` WHERE `Delete` = 0;";

if (!$result = mysql_query($sql, $conn))
{
    die('Error in querying the database' . mysql_error());
}

echo "
<select name = 'listPersons' id = 'listPersons' onclick = 'populate()'>";

while ($row = mysql_fetch_array($result))
{
    $id = $row['playersID'];
    $pName = $row['playersName'];
    $dob = $row['playersDateOfBirth'];
    $dob = date  ("d-m-Y", strtotime($dob));

    $allText = "$id, $pName, $dob";
    echo "<option value = '$allText'> $pName </option>";
}

echo "</select>";
mysql_close($conn);
?>

Sorry for the relatively long post. I am generally able to fix 99% of undefined errors. This one has me lost.
Cheers!
Comment: *"Could this be happening because my playersID text box is disabled?"* - Try it without the disabled and see what results you get.
Comment: Makes sense. It works, but the textboxes are not allowed to be edited. Any way I can leave the textboxes enabled, but still not allow the user to edit them?
Comment: You can always try using a hidden attribute.
Comment: I would use the hidden attribute, but when I select a player in the listPlayers drop down, the players data is to be shown in the corresponding textboxes before the "delete" takes place. So hidden is not going to work in this case unfortunately.
Answer: Use your disabled textbox to display it for the user.
Use another hidden field to hold the playersID.
Populate both of them with your script and then you can use the value in the hidden field when you need to delete.
For clarity I'm showing you what your HTML might look like - here's the script:

MySQL: Changing a value in a column


Rails check if URL is already encoded

Question: I am using URI.encode to encode url's in my rail's application. But sometimes these url's can be already encoded and I do not want to encode them again. I can see one solution here: How to find out if string has already been URL encoded?
However, was wondering is there any API already available in ruby for doing this ?
Also as suggested in https://stackoverflow.com/a/2295286/1475737 is decoding the string and checking its length with original a correct solution ?
Appreciate any suggestions!
Answer: The 'uri' module has a regular expression holding the list of characters which should be escaped for URI encoding:
require 'uri'

URI::REGEXP::UNSAFE

=> /[^-_.!~*'()a-zA-Z\d;\/?:@&=+$,\[\]]/n

So one solution is to test whether the given string matches this regexp and encode it only if does.
Comment: This has been rebuild in 1.9.3
You can try the following: ***URI::Parser.new.pattern*** but not sure if this returns what you want....
Comment: This does not seem to be defined for ruby 1.9.3 :(.
Comment: True. In fact, 'uri.encode' has been deprecated and the 'cgi' module should be used instead. Sadly, CGI does not provide a similar constant.
Comment: Also, This was suggested here as well http://stackoverflow.com/a/2295317/1475737 but its have a non-working example as well.

Rails check if URL is already encoded


OO Design vs Database Design

Question: Suppose I am developing an application for a product distributor in C#.
The distributor does the following 3 types of transactions:
(1) Indent
(2) Sell
(3) Stock
I am designing my classes as follows:
public abstract class Transaction
{
}

public class Indent : Transaction
{
}

public class Sell : Transaction
{
}

public class Stock : Transaction
{
}

Now if I want to save these three types of information in three separate tables then how should I design my DA layer?
Should I build separate DA classes like
(1) IndentDA
(2) SellDA
(3) StockDA

or a single class TransactionDA and perform CRUD operations by checking their types by using as/is operators?
Or what else can I do? Any suggestions?
Comment: I wouldn't use TRanasction as an Entity name; could lead to confusion...
Comment: Depends on how much the methods in the DA classes are similar. What methods do you use in the DAs?
Answer: Firstly, if you created a single class TransactionDA and checked types inside the class to perform CRUD operations, you would be violating the Open/Closed Principle, so I would definitely not go down that path.
As for suggestions on how to accomplish building out your DAL, I would suggest following some blog posts on people much smarter than I on what they think about this topic.
Repository is the new Singleton
Repository is Dead: Long Live Repository
Night of the Living Repositories
The conversation continues, I believe, but that should get you started.
Answer: I would use entity subtypes here.  Create one table for the transactions (and as an earlier poster said, perhaps a different term would be better) and store everything that's common in there.  Then create one "sub-type" table for each specialization.  These subtype tables should have the same primary key as the main table (the "strong" entity) and the fields that are unique to that particular specialization.  Each subtype is related to the strong entity in a one-to-one manner, with optional participation on the subtype end and required participation on the strong entity end.
Then, to make querying easier, define a view that (outer) joins the strong entity with all of the entity subtypes so that you can easily "see" everything.
Here's a simple (and common) example of how to set this up:
create table Employee (
  eid        int primary key,
  first_name text,
  last_name  text not null
)

create table Salaried (
  eid             int primary key,
  annualSalaryUSD money not null
)

create table Hourly (
  eid             int primary key,
  hourlyRateUSD   money not null
)

Answer: I would use an ORM such as NHibernate and use it's multi-table inheritance capabilities and then not have to worry about it myself.
Answer: If you know that your data is going into three separate tables, then I would normally have three separate DA classes.
However, if your tables are pretty much exactly the same, then you could genericize the TransactionDA and simplify your data layer.  I would only do this if you know that you will have high volume of transactions and are going to be separating out your tables into different files or something, otherwise I would probably just simplify things and combine it all.
Don't try to create a TransactionDA unless all your separate transaction types are extremely similar.
Answer: You can use dependency injection, create a DA class for each and have them all implement the same interface ITransactionDA with your CRUD operations.
public interface ITransactionDA
{
  void Read();
  void Update();
...
}

public class StockDA : ITransactionDA
{
  //implement interface methods
}

Stock stock = new Stock(new StockDA());

Answer: I will do something like that
public abstract class DistributerTransaction
{
    DistributerDA dataaccess;
}
public class Indent : DistributerTransaction
{
}
public class Sell : DistributerTransaction
{
}
public class Stock : DistributerTransaction
{
}

public abstract class DistributerDA
{
   /*Read();
     Update();*/
}
public class IndentDA : DistributerDA
{
}
public class SellDA : DistributerDA
{
}
public class StockDA : DistributerDA
{
}

Answer: Check out the Gang of 4 design patterns.

OO Design vs Database Design


How can I format an etude book and use my custom header using lilypond?

Question: I am creating several etude books and would like to be able to use the custom header I've developed. The code for the header I use for solo and other works looks like this:
<code>evenHeaderMarkup = \markup {
\fill-line {
  \center-column {
    \line { \on-the-fly #not-part-first-page \override #'(font-name . "Libre Franklin Light") \fromproperty #'header:piece }
    \line {
      \on-the-fly #not-part-first-page \override #'(font-name . "Libre Franklin Light") \fromproperty #'header:instrument \override #'(font-name . "Libre Franklin Light") "- PG." \override #'(font-name . "Libre Franklin Light") \on-the-fly #print-page-number-check-first \fromproperty #'page:page-number-string }}}}
</code>
This is so the header will look like this:

Each \score block to this etude book will have it's own piece and instrument:

Where I am getting stuck is I do not know how to format this project as an etude book. Ideally, the code for the entire etude book output will be in one file of code, with the book having a title, and each \score block also would have a specific title, piece and instrument. I would then call that specific information from each individual \score block into the header for each individual etude contained in the book.
Is there anyway to do this?
Comment: Could you say more about what are you exactly having problems with? It seems to me that you have done everything correctly. You can either put the individual `\scores` one after another, or put them all in a `\book` construct. Was that the fact you were missing?
Comment: The problem that I am having is that it isn't working. If I designate a "piece" in a \header block inside of a \score block, that piece does not appear in the custom header of the page.
Comment: Apparently, you have to put each score in a separate book part. That's the best I could get -> http://lilybin.com/8w3o4q/1
Answer: you need to be using \book and \bookpart . Set up things that repeat like fonts and text strings as globals or variables.


Gallery with Album Covers

Question: I currently have two node types: Album (CCK), and Image (image module, using Image FUpload module).
A user creates an album, simply a title. When they create Image(s), the album is node-referenced.
I have a view of Albums, and I would like to pick one Image from each album to show as a cover photo. I cannot just add a CCK field to the Album for this; the image needs to appear in the view of album images, and every image can be voted and commented on, as well as have their own references (tags).
I tried a few relationships in the View, nothing seemed to pull the image for an album. Is this possible using the Image module?
Answer: I finally decided to add a 'Cover' cck field to the Album type. I could not find a way in Views to correctly relate an Album to a View of photos for this purpose using Relationships.
Comment: With Flag you have a possible solution for that. Just flag a particular item,add a relationship using the flag. List it :-)

Gallery with Album Covers


svelte CSS file isn't generate on run

Question: I'm following this tutorial trying to create a svelte frontend.
I don't get any errors but when I run npm run dev i get:

which I'm supposed to see the new extra.css file created in the public folder. and so the page displayed is without any CSS and just plain li items instead of having materialize-css attributes.
my files:
rollup.config.js
import svelte from 'rollup-plugin-svelte';
import resolve from '@rollup/plugin-node-resolve';
import commonjs from '@rollup/plugin-commonjs';
import livereload from 'rollup-plugin-livereload';
import { terser } from 'rollup-plugin-terser';
import css from 'rollup-plugin-css-only';

const production = !process.env.ROLLUP_WATCH;

export default {
    input: 'src/main.js',
    output: {
        sourcemap: true,
        format: 'iife',
        name: 'app',
        file: 'public/build/bundle.js'
    },
    plugins: [
        svelte({
            // enable run-time checks when not in production
            dev: !production,
            // we'll extract any component CSS out into
            // a separate file - better for performance
            css: css => {
                css.write('public/build/bundle.css');
            }
        }),
        css({ output: 'public\extra.css'}),
...
...

index.html
<!DOCTYPE html>

    <meta charset='utf-8'>
    <meta name='viewport' content='width=device-width,initial-scale=1'>

    <title>Svelte app</title>

    <link rel='icon' type='image/png' href='/favicon.png'>
    <link rel='stylesheet' href='extra.css'>
    <link rel='stylesheet' href='global.css'>
    <link rel='stylesheet' href='build/bundle.css'>

App.svelte

<Router>
    <Navbar>

            <Route path="/" component={Home} />
            <Route path="/about" component={About} />

    </Navbar>
</Router>

Navbar.svelte

        <Link to="/" class="brand-logo">svelte app</Link>

          <Link to="/">Home</Link>

          <Link to="/about">About</Link>

and i also found this question/answer on SO that seem to say that the tutorial is ok and working for people so i'm confused.
Comment: `css({ output: 'public\extra.css'}),` I guess it's just esacping the `e`. Check wether it did write the css to `'publicextra.css'`; and use slashes `/`
Comment: Hi @Thomas thank but i tried both ways. i thought it was a path-ing issue because I'm working with windows but both `/` and `\` have the same result - there are no additional files in the folder after running
Answer: I had the same issue while following along with the tutorial. After hitting my head for some time I realized that inside the public/build directory it was creating this public/build/extra.css, so first see if your file structure is similar to mine (picture below)
file structure initially
Then what I did was in the rollup.config.js, I changed the output from public/build/extra.css to  extra.css
...

    svelte({
        compilerOptions: {
            // enable run-time checks when not in production
            dev: !production,
                  // we'll extract any component CSS out into
  // a separate file - better for performance
  css: (css) => {
    css.write('public/build/bundle.css');
  }
        }
    }),
    // we'll extract any component CSS out into
    // a separate file - better for performance
    css({ output: 'extra.css' }),
...

I hope this solves your problem as well.
Answer: I have a similiar setup and it works for me. I only noticed that I import materialize.css instead of materialize.min.css and that I have no matierialize.min.css in my node_modules/materialize-css/dist/css/ folder.
Maybe that's the solution to that riddle: you may try importing a non-existing css file.
Comment: hi @andreas_D thanks for the reply, i checked and i have both the minified files and the normal css/js files. i tried importing the non-minified files and got the same result. i also thought it might be related to caching in the browser and made sure that my browser (opera) is not caching
Answer: I got stuck in the same problem. It can be fixed by using Rollup's callback function instead of the simpler file name definition:
css({
    output: function (styles, styleNodes) {
        console.log('Writing bundled CSS.');
        fs.writeFileSync('static/bundle.css', styles);
    }
})

This requires Filesystem to be defined beforehand:
const fs = require('fs');

svelte CSS file isn't generate on run


How to allow users to create groups with Organic Groups

Question: How do I allow users on my site to create groups. I've done the initial configuration of Organic Groups, there is no 'Create Group' button by default. I want users to be able to create groups under a Parent Group.
Comment: What exactly do you mean by "Parent Group"?
Answer: Groups are just a special kind of nodes, so you just need to go to Permissions and grant users the right to create nodes of that bundle. Users can create groups by going to yourwebsite.com/node/add/nameofyourgroupnode. There is also a menu link by default.


How do I change option name?

Question: I have sort link 'Best' in exposed form. I want after user click this link, link title changes to 'New'
I'm try this, but it don't work:
function helper_form_views_exposed_form_alter(array &$form, FormStateInterface $form_state, $form_id) {
    if($form['#id']== 'views-exposed-form-pozdravleniya-taxonomy-term-page-1') {
      $form['sort_by']['#title'] = '';
      $form['sort_by']['#options']['created'] = '';
      if ($form['sort_by']['#value'] == 'count') {
        $form['sort_by']['#options']['count'] = 'New';
      }
    }
}

Also I'm try this, it don't work too:
function helper_form_views_exposed_form_alter(array &$form, FormStateInterface $form_state, $form_id) {
    if($form['#id']== 'views-exposed-form-pozdravleniya-taxonomy-term-page-1') {
      $form['sort_by']['#title'] = '';
      $form['sort_by']['#options']['created'] = '';
      if ($form_state->getValue('sort_by') == 'count') {
        $form['sort_by']['#options']['count'] = 'New';
      }
    }
}

Comment: I think maybe you need achive it by javasript instead of form.
Answer: I get answer here https://stackoverflow.com/a/65888182/13738862
function helper_form_views_exposed_form_alter(array &$form, FormStateInterface $form_state, $form_id) {
    if($form['#id']== 'views-exposed-form-pozdravleniya-taxonomy-term-page-1') {
      $form['sort_by']['#title'] = '';
      $form['sort_by']['#options']['created'] = '';
      if ($form_state->getUserInput()['sort_by'] == 'count') {
        $form['sort_by']['#options']['count'] = 'New';
      }
    }
}

How do I change option name?


Java - URL to File

Question: I am trying to create a method that displays a list of files in a given directory. This works fine for normal directories (on disk) but when I enter a url my list of files is null.
    public void getListOfFiles(String folderLocation){
    File folder = new File(folderLocation);
    File[] listFiles = folder.listFiles();

    for(int i = 0; i < 10; i++){
        System.out.println(listFiles[i]);
    }
}

I think my problem is because the File 'folder' is removing one of the '/' in my folderLocation (http://...)
I have tried using URL and URI but have had no luck! Can anyone help?
Comment: is the location an FTP server?
Comment: If yes, I think class File is not able to handle, you need to use some kind of FTP client library.
Answer: First of all, File won't work for this as it's not networking-aware.
Secondly, in general there's no mechanism to list files over plain HTTP. If the HTTP server gives you some kind of a listing page when you present it with the URL, you'll have to download the page using, for example, URLConnection and parse it yourself.
To list files over FTP, you could use FTPClient from Apache Commons Net.
Comment: So I have to download the contents of the directory and then look through them myself?

Java - URL to File


Why do guitarists wave their guitars?

Question: I'm currently learning to play the guitar and watch a lot of tutorials.
I noticed that on long playing chords they often wave their guitars (slowly move neck up and down).
What is the reason?
Does it actually do anything to the sound?
Is it to help the guitar resonate, or maybe to change distance from listener so he hears more a interesting sound?
Comment: If you don't hear an effect on the sound, this could just be natural movement caused by the "flow state" of playing music. Similar to dancing, playing an instrument can cause your body to move in ways that you're not intending to do on purpose, as if your body and skills are just a passive vessel for the music itself. It feels good to move your instrument as you play it.
Answer: It's a way to produce a vibrato:

By bending the neck up and down you change the pitch a little bit.

Vibrato (Italian, from past participle of "vibrare", to vibrate) is a musical effect consisting of a regular, pulsating change of pitch.
Wikipedia

Keep in mind that you can actually break your guitar neck this way, so don't overdo it.
Comment: Good point about breaking the neck. I saw a video somewhere where some guy snapped his (guitar) neck on an acoustic doing that. Hard to watch once you know what's going to happen. I'll post it if I can find it.
Comment: @pro exactly, yes. I didn't use the word "wave". I used "shake" , but bend is  indeed the correct word. I'll edit my post
Comment: @pro again, read the answer. The word "wave" was used in the question, not in my answer. I specifically said *shaking the neck*
Comment: @pro exactly, where is the word wave?

Why do guitarists wave their guitars?


Using chinese character in drupal_mail_send() generates unreadable character

Question: I am using code similar to the following one.
$message = array(
  'id' => 'named_fund',
  'to' => $email,
  'subject' => '恭喜您!',
  'body' => '恭喜!',
  'headers' => array('From' => 'Company Name <firstname.lastname@example.com>', 'Content-Type' => 'text/html; charset="utf-8";'),
);

drupal_mail_send($message);

When I use Chinese characters for the "From" header, I get strange characters. Why?
Answer: Try this:
drupal_mail_send(utf8_encode($message));

As Oswald implied, this is a character encoding issue.
If any data is coming from, or going to be stored in, the database, also make sure your database character encoding is utf8_general_ci.
Answer: If you have configured drupal to use an SMTP library (by setting the smtp_library drupal variable), you are responsible for properly encoding the headers (unless the library does that for you).
If you did not configure drupal to use an SMTP library, make sure that the source code of the module is encoded in UTF 8.


Integrating over the region between a line tangent to a circle, and the circle itself

Question:
If I want to integrate a function between $x=$ $0$ to $R$ (the cricle radius) and $y = 0$ to the lower half of the circle how do I describe the upper limit on the dy integral (see picture)?
The function I need to integrate over this region is $e^{-b(x^{2}+y^{2})}$. If you can integrate this over the region that would be great, but describing the upper limit as a function of $x$ would do...
Answer: $y = R - \sqrt{R^2 - x^2}$. That's because the circle equation is $(y-R)^2 + x^2 = R^2$.
The choice of the negative square root is because you want the lower rather than the upper limb of the circle.
Comment: Ok so you can deal with the circle equation. Cheers!


How to have more instances of marked js in one project?

Question: My problem, which I cannot overcome is that I cannot achieve a state where I have 2 parallel instances oh markedjs which work fully independently in terms of configuration I am passing through.
In other words I need two different markdown handlers, as in one case I want to parse markdowned text using say algorithm X, and the other using alghorithm Z. But from what I read in official docs it's... impossible...? Really?

Calling marked.use() to override the same function multiple times will give priority to the version that was assigned last. Overriding functions can return false to fall back to the previous override in the sequence, or resume default behavior if all overrides return false. Returning any other value (including nothing) will prevent fallback behavior.

And indeed, that's how it works on my end - I use marked.parse() in two different parts of app and I see changes reflected in both places even though I try to conditionally change the configuration (using flag or something, doesn't matter, the result is always the same).
How to fix it? How can one develop such nonsensefull solution?
Comment: Have you tried importing it once and then assigning `marked` to `markedX`, and then importing again and assigning `marked` to `markedZ`?
Comment: @Pointy doesn't work. I either have X or Z options applied
Comment: Sounds like a question for https://github.com/markedjs/marked/issues, since this is an open source package, and if your use-case makes sense, the docs (and possibly the code) should probably address it.
Comment: @Mike'Pomax'Kamermans I will ask there then
Comment: Well I have definitely been able to reassign the global `marked` to something else. Maybe you have to import it, reassign, unset the global `marked` symbol, then repeat that.
Comment: @Pointy but are you able to apply 2 different tokenizers or renderers and use them respectively in two different places in app and see 2 different outputs? Because I can't
Comment: @MateuszOsuch well what *exactly* have you tried along those lines?
Answer: A little late on this answer.
To avoid this, you can instantiate a new Marked object and pass extensions to that, e.g.:
import {Marked} from 'marked';

const renderer: RendererObject = {
  heading(text, level) {
    // turn headers into plain text
    return `<p>text</p>`;
  }
}
const marked = new Marked({renderer});

Documentation: https://marked.js.org/using_advanced#instance


Optimizing MySQL for ALTER TABLE of InnoDB

Question: Sometime soon we will need to make schema changes to our production database. We need to minimize downtime for this effort, however, the ALTER TABLE statements are going to run for quite a while. Our largest tables have 150 million records, largest table file is 50G.
All tables are InnoDB, and it was set up as one big data file (instead of a file-per-table).
We're running MySQL 5.0.46 on an 8 core machine, 16G memory and a RAID10 config.
I have some experience with MySQL tuning, but this usually focusses on reads or writes from multiple clients. There is lots of info to be found on the Internet on this subject, however, there seems to be very little information available on best practices for (temporarily) tuning your MySQL server to speed up ALTER TABLE on InnoDB tables, or for INSERT INTO .. SELECT FROM (we will probably use this instead of ALTER TABLE to have some more opportunities to speed things up a bit).
The schema changes we are planning to do is adding a integer column to all tables and make it the primary key, instead of the current primary key. We need to keep the 'old' column as well so overwriting the existing values is not an option.
What would be the ideal settings to get this task done as quick as possible?
Comment: Related: http://stackoverflow.com/q/9524938/632951
Answer: You might want to look at pt-online-schema-change from Percona toolkit. Essentially what it does is:

Copies original table structure, runs ALTER.
Copies rows from old table to newly created one.
Uses triggers to track and sync changes while copying.
When everything is finished it swaps tables by renaming both.

Works very well for single instance databases, but might be quite tricky if you use replication and you can't afford stopping slaves and rebuilding them later.
There's also a nice webinar about this here.
PS: I know it's an old question, just answering in case someone hits this via search engine.
Comment: A note from the pt-online-schema-change docs: The use of triggers means that the tool will not work if any triggers are already defined on the table.
Answer: You need to think about your requirements a little more carefully.
At the simplest level, the "fastest" way to get the table changed is to do it in as few ALTER TABLE statements as possible, preferably one. This is because MySQL copies a table's data to change the schema and making fifteen changes whilst make a single copy is obviously (and really is) faster than copying the table fifteen times, making one change at a time.
But I suspect you're asking how to do this change with the least amount of downtime. The way I would do that, you basically synthesize the way a non-block ALTER TABLE would work. But it has some additional requirements:

you need a way to track added and changed data, such as with a "modified" date field for the latter, or an AUTO_INCREMENT field for the former.
you need space to have two copies of your table on the database.
you need a time period where alterations to the table won't get too far ahead of a snapshot

The basic technique is as you suggested, i.e. using an INSERT INTO ... SELECT .... At least you're in front because you're starting with an InnoDB table, so the SELECT won't block. I recommend doing the ALTER TABLE on the new, empty table, which will save MySQL copying all the data again, which will mean you need to list all the fields correctly in the INSERT INTO ... SELECT ... statement. Then you can do a simple RENAME statement to swap it over. Then you need to do another INSERT INTO ... SELECT ... WHERE ... and perhaps an UPDATE ... INNER JOIN ... WHERE ... to grab all the modified data. You need to do the INSERT and UPDATE quickly or your code will starting adding new rows and updates to your snapshot which will interfere with your update. (You won't have this problem if you can put your app into maintenence mode for a few minutes from before the RENAME.)
Apart from that, there are some key and buffer related settings you can change for just one session that may help the main data move. Things like read_rnd_buffer_size and read_buffer_size would be useful to increase.
Answer:
Setup slave
Stop replication.
Make ALTER on slave
Let slave catch up the master
swap master and slave, so slave becomes production server with changed structure and minimum downtime
Answer: Unfortunately, this is not always as simple as staticsan leads on in his answer. Creating the new table while online and moving the data over is easy enough, and doing the cleanup while in maintenance mode is also do-able enough, however, the Mysql RENAME operation automatically manipulates any foreign key references to your old table. What this means is that any foreign key references to the original table will still point to whatever you rename the table to.
So, if you have any foreign key references to the table you're trying to alter you're stuck either altering those tables to replace the reference to your new table, or worse if that table is large you have to repeat the process with large table number two.
Another approach that has worked for us in the past has been to juggle a set of Mysql replicas handling the alter. I'm not the best person to speak to the process, but it basically consists of breaking replication to one slave, running the patch on that instance, turning replication back on once the alter table is completed so that it catches up on replication. Once the replication catches up, you put the site into maintenance mode (if necessary) to switch from your master to this new patched slave as the new master database.
The only thing I can't remember is exactly when you point the other slaves at the new master so that they also get the alter applied. One caveat to this process, we typically use this to roll alter patches before the code needs the change, or after the code has changed to no longer reference the columns/keys.
Comment: I hadn't thought of foreign keys. Where I was a DBA where I used the technique I described, we didn't use foreign keys at all as the application believed it and only it handled all of that.
Answer: I tested various strategies to speed up one alter table. Eventually I got about 10x speed increase in my particular case. The results may or may not apply to your situation. However, based on this I would suggest experimenting with InnoDB log file/buffer size parameters.
In short, only increasing innodb_log_file_size and innodb_log_buffer_size had a measurable effect (Be careful! Changing innodb_log_file_size is risky. Look below for more info).
Based on the rough write data rate (iostat) and cpu activity the bottleneck was io based, but not data throughput. In the faster 500s runs the write throughput is at least in the same ballpark that you would expect from the hard disk.
Tried performance optimizations:

select * into outfile + load data infile as suggested in https://stackoverflow.com/a/5303805/1148030
"disable keys" I missed the fact that it has no effect in InnoDB which is true also based on the results. This was on in all cases except the final alter table.  How to disable index in innodb
innodb_buffer_pool_size, "Disk I/O Tips" at http://dev.mysql.com/doc/refman/5.1/en/innodb-tuning.html
foreign_key_checks=0 et al, "Bulk data loading tips" at http://dev.mysql.com/doc/refman/5.0/en/innodb-tuning.html
Bigger innodb_log_file_size (See warnings below) and innodb_log_buffer_size. See "Logging tips" at http://dev.mysql.com/doc/refman/5.0/en/innodb-tuning.html and http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_innodb_log_file_size and http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_log_buffer_size

Changing innodb_log_file_size can be dangerous. See http://www.mysqlperformanceblog.com/2011/07/09/how-to-change-innodb_log_file_size-safely/ The technique (file move) explained in the link worked nicely in my case.
Also see http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/ and http://www.mysqlperformanceblog.com/2008/11/21/how-to-calculate-a-good-innodb-log-file-size/ for information about innodb and tuning log sizes. One drawback of larger log files is longer recovery time after crash.
Test runs and rough timings:

The simple load data to a freshly createad table: 6500s
load data w. innodb_log_file_size=200M, innodb_log_buffer_size=8M, innodb_buffer_pool_size=2200M, autocommit= 0; unique_checks=0, foreign_key_checks=0: 500s
load data w. innodb_log_file_size=200M, innodb_log_buffer_size=8M: 500s
Equivalent straight alter table w. datainnodb_log_file_size=200M, innodb_log_buffer_size=8M: 500s

Testing details: Table: InnoDB, 6M rows, 2.8G on disk, single file (innodb_file_per_table  option), primary key is 1 integer, +2 unque constraints/indices, 8 columns, avg. row length 218 bytes. Server: Ubuntu 12.04, x86_64, virtual machine, 8 cores, 16GB, sata consumer grade disk, no raid, no database activity, minuscule other process activity, minuscule activity in other and much smaller virtual machines. Mysql 5.1.53. The initial server config is pretty default except for increased innodb_buffer_pool_size of 1400M. The alter table adds 2 small columns. I didn't clock the raw alter table, but instead experimented with equivalent load data infile statement, finally I did the straight alter table and got comparable result.
This question is related to at least following questions:

MySQL: add a field to a large table
Add a new column in a large mysql table
mysql slow query
MySQL "set unique_checks", "set foreign_key_checks" vs. "alter table disable keys"
Optimizing MySQL for ALTER TABLE of InnoDB
Comment: Fantastic answer, this deserves many more votes. Thank you for publishing your research and results.
Answer: I really don't know how to optimize that, but it's usually a good practice to put the site in offline mode before doing such updates.
Then, you can run your DB scripts at, say, 3 am, so it shouldn't matter much if downtime's a big longer than ideal.
Comment: Yes -- we will bring down the site during night hours to do this task. But with such big tables the ALTER TABLE may run for many hours, and that is probably not going to fit into one night :(
Comment: Maybe you'd want to run the script in a separate server with a dump of the production DB timing how much it takes. Hopefully, you'll get a nice surprise! :)
Comment: Yes, we're preparing a QA environment for this. Can't optimize without testing your optimizations ;)


Winding number of a curve contained in a triangle around a point outside the triangle

Question: I am considering the following problem:
If a closed curve $\gamma$ is contained in a triangle $\Delta$ and $a$ is a point outside of $\Delta$ then $I(\gamma, a) = 0$, where $I$ is the winding number of $\gamma$ around $a$.
Intuitively this makes sense, $\Delta$ is simply connected, and so since $a$ is outside $\Delta$, no curve in $\Delta$ can wind around $a$.
However, the solution I am given to this problem makes the following argument: By convexity of $\Delta$, for all $p \in \Delta$, the straight line homotopy from $\gamma$ to $p$ is contained in $\Delta$, and since $a \not \in \Delta$, defining $\Gamma_0(t) = \gamma(t)$ and $\Gamma_1(t) = p$ gives $I(\Gamma_0, a) = I(\Gamma_1, a) = 0$.
I do not understand how we can conclude that $I(\gamma, a) = 0$ from the fact that $I(\Gamma_1, a) = 0$. Is there some obvious fact about homotopies I am missing?
Comment: Homotopies that don't pass through $a$ preserve the winding number.
Answer: This bypasses part of your question, but because it is nearly a one-liner, it should be mentioned:
The closed set $\Delta$ is the intersection of three closed half-planes. As $a \notin \Delta,$ at least one of them doesn't contain $a.$ Any continuous choice of $\arg(z - a)$ for $z$ in such a half-plane is also valid for $z \in \Delta,$ so $I(\gamma, a) = 0$ if $[\gamma] \subseteq \Delta.$


Is Windows Disk Defragmenter good enough?

Question: I know this can be sound a little like an open question, but I want your thoughts here. Maybe if someone have make a benchmark test or based on your experience.
Do you think that Windows Defragmenter is good enough?
Are tools like Diskeeper or Raxco PerfectDisk worthwhile?
Do you really win I/O performance or are the optimizations not high enough to be noticed?
Answer: Unless you have a lot of tiny files being written/erased/written to the drive, basic defragmentation should be more than enough on Windows. You're going to have other factors that can't possibly be generalized...most servers you would have to factor in your RAID controller, caching, load, type of RAID, etc. to get performance. Your best bet would be to run a benchmark before and after running your defrag to get numbers for your particular circumstance on your particular hardware.
If defragging makes a big difference in your performance, I'd venture a bet that you should be looking at changing hardware a bit.
The bigger benchmark is whether your server is having performance issues and your users notice. Is that the case? Then start narrowing down the cause as to whether it's the disk subsystem and whether it's a problem with your drives being too slow, utilization is wrong for your RAID configuration, etc. rather than randomly poking whether a particular type of defrag will speed up access.
Comment: Thank you @Bart. Why Tiny files? What about defrag virtual hard disks?  http://www.raxco.com/business/hyper-v.aspx
Comment: Smaller files lead to more slack space scattered all over the drive. Larger files get chunked if there's not enough space to store them contiguously, and you can have random writes all over the drive for access that some defragmenters will try to "optimize" for better access times. Defragmenting virtual disks within a filesystem I've found useful when shrinking disks down.
Answer: I would also suggest that you do not use defrag on virtual disks. While it is theoretically possible for it to be beneficial there are also many times where it is not. For example, dynamically allocated disks will balloon in size when defragged with very little improvement in access times. I have also heard negative things about live migrations with defrag.
I have used the defrag to shrink a virtual disk suggestion as Bart outlines - I have found this to be the only particularly useful case.
I suggestion letting the hypervisor take care of the disk management in most cases.
Comment: Agreed, there's a lot of abstraction when you throw both newer controllers and their drivers, RAID, and virtualized disks into the mix. The results you get from what the software thinks is going on isn't necessarily what is actually going on; it's one of those things where you might want to leave the controllers to do their own magic.
Answer: The nice part about other defraggers is that they usually move most frequently accessed files around to the fastest parts of the disks (e.g. the beginning vs the end of the spindle).
Some can also measure boot time to figure out which files need to be defragged during next boot.
Some also have an automatic defragment mode that runs in the background and when your PC is idle it defragments.
The free selection of defragmenters for Windows is excellent with very nice features:

MyDefrag
Smart Defrag
Defraggler


Custom server control disappears from page when UpdatePanel updates

Question: I created a custom asp.net server control. It works fine on a regular asp.net page and  as a DOM object inside of the browser. But I've never used the UpdatePanel before and now I'm trying to make sure that this control works there as well. It doesn't. If I add my control to the page outside of an update panel and click some panel's inside button (trigger), everything works fine. But if I place my control inside of update panel and click that button again, the control "disappears" from the page completely. I still can see my control in javascript debugger and the update, meaning that the object itself is still in DOM. It looks like the panel just "hides" the outer div element of my control for some reason. I tried to call panel's Update() method on button click handler, set panel's UpdateMode to both Conditional and Always. All with the same result. How can I fix that? Thank you!
Answer: Posting from another computer, in response to TriLLi:
Here is my aspx code. Again, the UpdateMode can be Conditional or Always - the result is the same, the cc1:WebControl disappears but the lblInfo label is there, fully functional after the btnTest click.
<form id="form1" runat="server">
<asp:scriptmanager ID="scrManager" runat="server"></asp:scriptmanager>

    <asp:updatepanel ID="pnlTest" runat="server" UpdateMode="Conditional">
        <ContentTemplate>

                <cc1:WebControl ID="ceFirst" runat="server" decisionxmlfilevirtualpath="/Person.config" autolist="User" help="Off"></cc1:WebControl>

                <asp:label ID="lblInfo" runat="server" forecolor="red"></asp:label>

                <asp:button ID="btnTest" runat="server" width="100" text="Test" />

        </ContentTemplate>
        <Triggers>
            <asp:AsyncPostBackTrigger ControlID="btnTest" EventName="Click" />
        </Triggers>
    </asp:updatepanel>

</form>

Comment: I am not so sure about this, but if your control use javascript to show you need to recall javascript code when update panel ends callback, try to add in your code behind if update panel is in callback ScriptManager.RegisterStartupScript(....
Comment: Yes, I'm aware of that and do that. I'm sure I'm missing something very simple here. Thank you, though.
Comment: Guys. Anyone else with suggestions, please? :)


How to change the order of values in body

Question: I am using go-swagger. Is there any way to make attribute in generated swagger have the same order as defined. In generated swagger it is sorted in alphabetical order.
This is my definition

This is generated swagger
Comment: See https://goswagger.io/faq/faq_model.html#generated-code-changes-the-order-of-properties-in-struct and https://goswagger.io/use/models/schemas.html#custom-extensions
Comment: @jub0bs where should i put it, it seem doesn't work
Comment: https://github.com/go-swagger/go-swagger/issues/2202
Comment: @medasx wow, so this is an unresolved issue and there is nothing I can do right?
Answer: To get the order in generated schema you need to add x-order with number value within double quotes (") or single quotes (')
// swagger:parameters SaveAccountRequest
type SaveAccountRequestWrapper struct {
    // in:body
    Account SaveAccountRequest `json:"body"`
}

type SaveAccountRequest struct {
    // This text will appear as description of your request body.
    // in:body
    // Extensions:
    // x-order: "0"
    VendorID string `json:"vendorId"`
    // Extensions:
    // x-order: "1"
    VendorName string `json:"vendorName"`
    // Extensions:
    // x-order: "2"
    FullName string `json:"fullName"`
    // Extensions:
    // x-order: "3"
    Source string `json:"source"`
    // Extensions:
    // x-order: "4"
    Skill []string `json:"skill"`
    // Extensions:
    // x-order: "5"
    Phone string `json:"phone"`
    // Extensions:
    // x-order: "6"
    Email string `json:"email"`
    // Extensions:
    // x-order: "7"
    StartDate time.Time `json:"startDate"`
    // Extensions:
    // x-order: "8"
    EndDate time.Time `json:"endDate"`
}