Mac 10.6.8 Zend Server CE MySQL does not start — "ERROR! Manager of pid-file quit without updating file"

Question: MySQL does not start, I have previously installed zend server on the same Mac and MySQL worked. I have reinstalled after rm -r /usr/local/zend .  Are there any other directories I need to clear?
Or do you have any ideas?
/usr/local/zend/mysql/data
bash-3.2# echo $PATH
/usr/local/zend/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
bash-3.2# zendctl.sh start-mysql
Starting MySQL
. ERROR! Manager of pid-file quit without updating file.
bash-3.2#

This is the log file of the failed start-up
110817 13:20:08 mysqld_safe Starting mysqld daemon with databases from /usr/local/zend/mysql/data
110817 13:20:09 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
110817 13:20:09 [Warning] '--skip-locking' is deprecated and will be removed in a future release. Please use '--skip-external-locking' instead.
110817 13:20:09 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/zend/mysql/data/ is case insensitive
110817 13:20:09 [Note] Plugin 'FEDERATED' is disabled.
/usr/local/zend/mysql/bin/mysqld: Table 'mysql.plugin' doesn't exist
110817 13:20:09 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
110817 13:20:09  InnoDB: Initializing buffer pool, size = 8.0M
110817 13:20:09  InnoDB: Completed initialization of buffer pool
InnoDB: Log scan progressed past the checkpoint lsn 0 37356
110817 13:20:09  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 0 44233
110817 13:20:09  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110817 13:20:09  InnoDB: Started; log sequence number 0 44233
110817 13:20:09 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
110817 13:20:09 mysqld_safe mysqld from pid file /usr/local/zend/mysql/data/puidhcp-088-227.is.nottingham.ac.uk.pid ended

Answer:
[ERROR] Fatal error: Can't open and lock privilege tables: Table
  'mysql.host' doesn't exist

Run mysql_install_db command to initial MySQL data directory and creates the system tables.


Two-way fixed effects and collinearity

Question: I am modeling the relationship of an independent variable $(x_t/z_i),$ where $t$ indexes time and $i$ individual with an outcome $y_{it}$. The log transformation of $(x_t/z_i)$ is prohibitively collinear with fixed effects $(D_t,D_i)$ but neither of $D_t$ nor $D_i$ individually. The untransformed $(x_t/z_i)$ is not collinear with $(D_t,D_i)$, $D_t$, or $D_i$.
      (1) $\qquad y_{it} = \alpha + \beta(x_t/z_i) + D_t + D_i +
    \varepsilon_{it}$
      (2) $\qquad y_{it} = a + \beta \ln(x_t/z_i) + D_t
    + D_i + \varepsilon_{it}$
What's happening here? Is model (1) plausible?
Some values, as requested in comments.
Comment: Welcome. Could you share a small subset of the untransformed covariate?
Comment: Thanks, @ThomasBilach. I'm quite sure what you are asking for. Would you like to see table of values?
Comment: Yes. I would like to see the untransformed data values.
Comment: @ThomasBilach, I've added some values to the post.
Answer: # Reproducing the OP's panel
# N = 4
# T = 2
# 8 unit-time observations

library(tidyr)
library(dplyr)
library(tibble)
library(plm)

set.seed(2021)

df <- tibble(
  unit = c(1, 1, 2, 2, 3, 3, 4, 4),
  time = c(1, 2, 1, 2, 1, 2, 1, 2),
  y = runif(8, 0, 1),
  x = c(9454.5, 9368.55, 9454.5, 9368.55, 9454.5, 9368.55, 9454.5, 9368.55),
  z = c(11175.81, 11175.81, 8590.219, 8590.219, 11175.81, 11175.81, 8590.219, 8590.219),
  raw_xz = x / z,
  log_xz = log(x / z)
  )

It appears your model is estimable, so long as you drop the time fixed effects.
Assuming you acquired a panel with only two time periods, then your two-way fixed effects (TWFE) equation should produce results equivalent to a first difference (FD) model. Once we difference your covariate of interest across time, it is more obvious why your model is inestimable. For instance, your differenced raw covariate $\frac{x_t}{z_i}$ shows variation across units, whereas differencing $\ln(\frac{x_t}{z_i})$ results in the exact same change over time, resulting in no variation across units. Software should warn you when it cannot estimate a variance.
Let's see if I can demonstrate this using some sample data:
# A tibble: 8 x 7
   unit  time     y     x      z raw_xz  log_xz
  <dbl> <dbl> <dbl> <dbl>  <dbl>  <dbl>   <dbl>
1     1     1 0.451 9454. 11176.  0.846 -0.167
2     1     2 0.784 9369. 11176.  0.838 -0.176
3     2     1 0.710 9454.  8590.  1.10   0.0959
4     2     2 0.382 9369.  8590.  1.09   0.0867
5     3     1 0.636 9454. 11176.  0.846 -0.167
6     3     2 0.701 9369. 11176.  0.838 -0.176
7     4     1 0.640 9454.  8590.  1.10   0.0959
8     4     2 0.267 9369.  8590.  1.09   0.0867

The foregoing panel dataset observes four cross-sectional units ($N = 4$) over two time periods ($T = 2$). I simulated some $y$ values to approximate the subset of numerical quantities presented. Everything else is similar to the data values you presented.
# Comparing a TWFE vs. FD (T = 2)

mod_lm <- lm(y ~ raw_xz + as.factor(unit) + as.factor(time), data = df)  # two-way fixed effects (T = 2)
mod_plm <- plm(y ~ raw_xz, model = "fd", data = df)                      # first difference (T = 2)

mod_lm$coefficients['raw_xz']
  raw_xz
237.4307

mod_plm$coefficients['raw_xz']
  raw_xz
237.4307

As indicated above, your two models produce equivalent estimates of the untransformed covariate. But what variation is left over once you difference your model?
Let's run a couple of FD estimators comparing $\frac{x_t}{z_i}$ with $\ln(\frac{x_t}{z_i})$:
# Note: FD models estimated using the plm() function
# model = "fd"

plm_raw <- plm(y ~ raw_xz, model = "fd", data = df)
plm_log <- plm(y ~ log_xz, model = "fd", data = df)

plm_raw$coefficients['raw_xz']
  raw_xz
237.4307
plm_log$coefficients['log_xz']
<NA>
  NA

Software cannot estimate a variance. As a consequence, log_xz must be dropped.
Oftentimes a simulation isn't necessary. Instead, we can look at the data frame and see what redundancies exist, if any. Let's difference the periods and see what variation we can exploit. Here is the differenced data frame:
df %>%
  group_by(unit) %>%
  mutate_at(vars(y:log_xz), list(~ .x - lag(.x))
  )

# A tibble: 8 x 7
# Groups:   unit [4]
   unit  time       y     x     z   raw_xz   log_xz
  <dbl> <dbl>   <dbl> <dbl> <dbl>    <dbl>    <dbl>
1     1     1 NA       NA      NA NA       NA
2     1     2  0.333  -86.0     0 -0.00769 -0.00913
3     2     1 NA       NA      NA NA       NA
4     2     2 -0.328  -86.0     0 -0.0100  -0.00913
5     3     1 NA       NA      NA NA       NA
6     3     2  0.0650 -86.0     0 -0.00769 -0.00913
7     4     1 NA       NA      NA NA       NA
8     4     2 -0.374  -86.0     0 -0.0100  -0.00913

Note what $\ln(\frac{x_t}{z_i})$ does to your data values in the last column. Each unit undergoes the exact same change across time. Software should produce a warning(s) (e.g., cor(y, haty): the standard deviation is zero).

See Comments
In response to your comments, it appears you observe multiple units over more than two time periods. I erroneously assumed your truncated dataset preserved the size of $T$. My answer is still useful, assuming each $i$ has the same pattern.
The variable $x_t$ is unit-invariant. Normalizing by $z_i$, which is time-invariant, is valid. However, log-transforming this quotient produces values where each $i$ undergoes the same change from period-to-period. By the properties of logarithms: $\ln(\frac{x_t}{z_i}) = \ln(x_t) - \ln(z_i)$. Put more simply, each $x_t$-value is centered around $z_i$, where each $i$ now experiences the same deviation across time periods. Here is some sample data in tabular form where $N = 3$ and $T = 3$:
$$
\begin{array}{ccc}
unit & time & x_t & z_i & \ln(x_t) - \ln(z_i) \\
\hline
 1 & 1 & 9,454 & 11,176 & -0.17 \\
 1 & 2 & 9,369 & 11,176 & -0.18 \\
 1 & 3 & 6,419 & 11,176 & -0.55 \\
\hline
 2 & 1 & 9,454 & 23,190 & -0.90 \\
 2 & 2 & 9,369 & 23,190 & -0.91 \\
 2 & 3 & 6,419 & 23,190 & -1.28 \\
\hline
 3 & 1 & 9,454 & 62,176 & -1.88 \\
 3 & 2 & 9,369 & 62,176 & -1.89 \\
 3 & 3 & 6,419 & 62,176 & -2.27 \\
\end{array}
$$
Note, the within-unit change is the same for all $i$ as you move from period-to-period. Software will invariably drop a time period to achieve something estimable. According to my example, which is based upon your sample data, you can estimate individual-specific intercepts.
It will be difficult to reproduce the collinearity without seeing your data. In my estimation, it appears each shift in $\ln(\frac{x_t}{z_i})$ over the time periods is the same for all persons in your sample. Suppose your covariate is some intensity measure (i.e., treatment), then you can think about this as each unit receiving a similar dosage over time.
Discretized measures produce the same problem. Suppose some discrete jump in $x_t$ is observed. By construction, if $x$ is $t$-subscripted, then the change is the same for all units. The only way to avoid collinearity is if some individuals receive the shock (i.e., treatment) and others do not; discrete shocks experienced by all $i$ and in the same periods $t$ will be collinear with the time fixed effects.
In either case, any panel data estimator run in either R or Stata should balk. You might be receiving the unhelpful empty model message in your output. In particular, software is warning you that once you condition on unit and time fixed effects, you have nothing left over to exploit.
Comment: thanks so much for your help! I wasn't quite sure what you were after. I have a balanced panel with 7 time periods and 58 individuals. As you mention, the model runs if I drop the time FE works. Alternatively, the model runs if I drop the individual FE. The issue seems to be that the variance of ln(x_t/z_i) is constant across both time and individual, but not so for (x_t/z_i).
Comment: Does the lack of variance that I demonstrated above generalize to your setting? I assume your covariate of interest drops out of the model once you condition on unit and time fixed effects. If possible, show me a subset of values for two individuals over all 7 periods and I will adjust my answer.
Comment: Or more simply, does each individual experience the same deterministic change over time, like it does in your post?
Comment: I lost track of this question, @thomas, but wanted to thank you. the log transformation explanation nailed it for me. Thaks!

Two-way fixed effects and collinearity


Change primary key to auto increment

Question: I updated my database schema in Doctrine 2 / Sf2 and added some new tables to my database. Everything was going okay, but when I wanted to create the admin side of the application I realized that I need some IDs to be AUTO_INCREMENT.
Then, I wanted to change my schema. From this.
X\XBundle\Entity\Currency:
type: entity
table: currencies
repositoryClass: CurrencyRepository
id:
    id:
        type: integer
fields:
    name:
        type: string
        length: 255
    slug:
        type: string
        length: 64
manyToMany:
    brokers:
        targetEntity: X\XBundle\Entity\Broker
        mappedBy: currencies

To this:
X\XBundle\Entity\Currency:
type: entity
table: currencies
repositoryClass: CurrencyRepository
id:
    id:
        type: integer
        generator: { strategy: AUTO } # I changed on this row!
fields:
    name:
        type: string
        length: 255
    slug:
        type: string
        length: 64
manyToMany:
    brokers:
        targetEntity: X\XBundle\Entity\Broker
        mappedBy: currencies

Then I tried to run php app/console doctrine:schema:update --force in CLI and got this message:
 SQLSTATE[HY000]: General error: 1833 Cannot change column 'id': used in a foreign key constraint 'FK_706E39538248176' of table 'forexcbc.broker_currencies'

I tried everything now. I deleted the foreign keys manually and rerun the script, didn't work. I deleted the relations between the two tables, didn't work. And the funny thing is, there is nothing in the tables yet.
My Broker schema looks like this (I don't know if this is helping in any form)
X\XBundle\Entity\Broker:
type: entity
table: brokers
repositoryClass: BrokerRepository
id:
    id:
        type: integer
        generator: { strategy: AUTO }
fields:
    name:
        type: string
        length: 255
    slug:
        type: string
        length: 64
oneToMany:
    accountTypes:
        targetEntity: X\XBundle\Entity\AccountType
        mappedBy: broker
        cascade: ["persist"]
manyToMany:
    currencies:
        targetEntity: X\XBundle\Entity\Currency
        inversedBy: brokers
        joinTable:
            name: broker_currencies
            joinColumns:
                broker_id:
                    referencedColumnName: id
            inverseJoinColumns:
                currency_id:
                    referencedColumnName: id

My question is, how on Earth is it possible to set the ID field to auto increment after the tables were created. Again, there is nothing in the tables yet.
Thank you for your help in advance!
Adam
Answer: Instead of applying the update to the database, you can dump the SQL to be applied:
app/console doctrine:schema:update --dump-sql --complete

Open a mysql shell, select your DB and first disable foreign key checks:
SET FOREIGN_KEY_CHECKS = 0;

Then execute the generated SQL queries in the shell.
When you're done, close the shell, and run the doctrine:schema:update command again, to see if everything was successful.
Comment: Actually, the reason is that `SET FOREIGN_KEY_CHECKS` only affects the current connection. The moment you close your MySQL connection and open a new one, key checks are on again.
Comment: Thank you, that helped! Just for the record, if you only disable foreign keys with SET FOREIGN_KEY_CHECKS = 0 and then run schema:update it won't work. Because schema:update re-enables foreign keys automatically.

Change primary key to auto increment


OpenCV Slow camera framerate

Question: I just copied an program to detect faces through webcam but the video capture is really slow, and i dont know how to fix it!
Here is the code:
#include<stdio.h>
#include<math.h>
#include<opencv\cv.h>
#include<opencv\highgui.h>
#include<opencv2\objdetect\objdetect.hpp>
#include<opencv2\highgui\highgui.hpp>
#include<opencv2\imgproc\imgproc.hpp>
#include<vector>

using namespace cv;
using namespace std;

int main()
{
CascadeClassifier face_cascade;
if(!face_cascade.load("c:\\haar\\haarcascade_frontalface_alt2.xml")) {
    printf("Error loading cascade file for the face");
    return 1;
}
VideoCapture capture(0);
if(!capture.isOpened())
{
    printf("Error trying to start the Camera");
    return 1;
}
Mat cap_img,gray_img;
vector<Rect> faces;
while(1)
{
    capture >> cap_img;
    waitKey(10);
    cvtColor(cap_img, gray_img, CV_BGR2GRAY);
    cv::equalizeHist(gray_img,gray_img);
    face_cascade.detectMultiScale(gray_img, faces, 1.1, 10, CV_HAAR_SCALE_IMAGE | CV_HAAR_DO_CANNY_PRUNING, cvSize(0,0), cvSize(300,300));
    for(int i=0; i < faces.size();i++)
    {
        Point pt1(faces[i].x+faces[i].width, faces[i].y+faces[i].height);
        Point pt2(faces[i].x,faces[i].y);
        rectangle(cap_img, pt1, pt2, cvScalar(191,191,191), 2, 8, 0);
    }
    imshow("Result", cap_img);
    waitKey(3);
    char c = waitKey(3);
    if(c == 27)
        break;
}
return 0;
}

I am using Visual studio 2012 and this is the main.cpp file. Im using OpenCV 2.4.9!
Comment: Did you compile in debug or in release mode? OpenCv's runtime is greatly improved when compiled and run in release mode!
Comment: In release mode it is giving me an exe missing error! It's the first time im working with OpenCv..
Comment: Looks like the Libraries paths get messed up when i change it to release mode, i dont know why! Here is a print screen of some of the errors when i change it to release mode...
http://i.imgur.com/9aXf9GA.png
Comment: You will have to setup your project properties the same way as you did in debug mode. Otherwise, neither your header files nor the libs can be found. IMPORTANT: Make sure you specify the libraries whose filenames are not suffixed with a 'd'. E.g. `opencv_imgproc249.lib` instead of `opencv_imgproc249d.lib`
Comment: Ok, thanks man. There are no errors now but still the Exe missing error!
Comment: It's working now, i just had to change some stuff in the preprocessor to accept fopen! Thanks for everything man, it is working now :)
Comment: The Framerate is a lot better now, but do u know something more to increase it?
Comment: You should definitely get rid of all those `waitKey()` calls. It waits the given amount of time in milliseconds. Just keep the last three lines of your while loop and remove the other two calls.
Comment: Wow thanks man, its really faster now! You're a OpenCV professional xd
Comment: I'll post a summary as an answer then. Please accept it by ticking the green check mark next to that answer to mark that it solved your problem.
Comment: Yes man, of course :)
Answer: OpenCV comes with prebuild libraries. When you use them in an application you actually want to deploy make sure you use the release libraries. In debug mode you have lots of additional checks in the form of asserts as well as debug symbols allowing you to step into the libraries with a debugger that are removed in release mode.
Another more specific advise regarding the code you posted: Avoid calls to cv::waitKey() as every single call, who would have guessed, makes your main thread wait the specified amount of time in milliseconds. Do not drop calls to it entirely as e.g. cv::imshow() will only work appropriately with these in place.

Edit:
Reduce your while loop to:
while ( true ) {
    capture >> cap_img;
    imshow("Result", cap_img);

    if(waitKey(1) == 27)
        break;
}

When you know how much time you need for capturing the image and just displaying it, then you can compare how much of an impact whatever algorithm you are running has on your performance.
Comment: Sweet, glad I could help you out ;)
Comment: Just one fast question, OpenCV is normaly slow with webcams? because It is a lot faster now but still 1 sec delay or something.
Comment: If you want to check the impact of your tracking code on the runtime just comment everything but the first and the last 5 lines of the code you posted and set the argument of `cv::waitKey(x)` to x = 1. That will be as fast as you can get with your current hardware.
Comment: Which lines? can u tell me which lines should i not comment?
Comment: By the way, im working for school, one guy here said that it should be geting like +30fps and talked about changing the format to mpeg-4 or something like that, do you know something about this?
Comment: I'm definitely no expert myself. But since you're just reading from your webcam stream you should not be able to change the format of any kind as you would be when using a video as input. For now, please measure your fps and also list your current hardware setup. Otherwise I can't tell you if what you perceive as 'slow' is actually slow, or not. So, please read up a bit about profiling your current code and report back with your measurements.
Comment: Somehow now is giving me this error: http://i.imgur.com/MkcpX1Y.png
I dont know what this is, even trying to run other projects its giving me this error! I tried to create another project with same code and still, wtf!
Answer: Here is solution...everytime you read!!!..everytime you use vcap>>cap or cap.read(vcap)... you must set FPS ..25 is max..if i try to set 26 or higher it runs slow.
...this runs 30 fps for me
If you not set FPS...it runs slow too
if (cap.read(vcap)){
        imshow("Cam Feed", vcap);
        cap.set(CV_CAP_PROP_FPS, 25);
    }

Answer: On top of the other answers, Multiprocessing is what fixed my issue. I split my program into 3 processes. One for grabbing the image, one for processing the image, and one for displaying the image. Referencing this page helped me a lot. I also followed the tips of the other answers here.

OpenCV Slow camera framerate


Constrained, Supervised Classification

Question: I am playing around with a classification model, and I would like to know if there are any known methods to achieve what I am looking to do.
The data looks something like this:

Class
id
$x_1$
$x_2$
...
$x_n$

0
1

0
1

0
1

...

1
1

2
1

3
1

0
2

0
2

0
2

...

1
2

2
2

3
2

0
3

0
3

0
3

...

1
3

2
3

3
3

...

Where $\{x_1,\dots,x_n\}$ are the features and $id$ is a flag I would like to use, not a "feature".
There are four classes - $\{0,1,2,3\}$, with the 0 class representing a bit under 90% of the data. The thing is, within an ID tag, I know there to exist exactly one occurrence of class 1, class 2, and class 3, with the rest being class 0. I will know this on unclassified data too. The classes within these ID tags are thus not independent(?) in that if one observation is a 1, then no other observation can be a 1.
So my question is, is there a method I can use that allows me to incorporate my knowledge about these ID groups to force the model to assign exactly one observation to group 1, exactly one to group 2, and exactly one to group 3 within each ID flag?
Edit:
I should mention that I have tried undersampling the majority class, oversampling the minority classes, and modifying the loss function to more heavily penalise the misclassification of the minority classes. These are ideas I have seen used in fraud detection. But in the example of fraud detection, you don't know how many fraudulent transactions there are in a group, ahead of time.
My question is focused on if there is a way to incorporate my pre-existing knowledge of the count of each class (inside an id flag) into my prediction. Could I perhaps customise the loss function to heavily penalise classifying the incorrect number into each class?
Edit (2):
To provide more context, the id flag represents a game, and all observations with the same id flag are players in that game. The features are the 'stats' for each player in that particular game. After a game is played, a player is awarded 3 points if they are the best player, 2 points if the second best player, and 1 point if they are the third best player. All other players are awarded 0 points. This is the definition of a class.
Edit (3):
The classes are some sort of transformation of outcomes in the game. For example in a soccer match, if a player kicks more goals and touches the ball more times than a second player, the first player has most likely played better and will most likely be ranked higher. Also, I don't need to predict those game outcomes, they are known at the time of ranking, which occurs after the game has been played. I would like to predict what the ranking will be, given the outcomes from the game.
Comment: The answer to any  "would there be merit/does X work" question is usually "depends on your problem," unfortunately -- but details can help! The new context is extremely helpful, and makes me think this is a learning-to-rank problem -- I don't have the bandwidth to put together a full answer now but hopefully, this gives you a starting point to google.
Comment: Is the class (0, 1, 2, 3) a transformation of some specific outcome in the game (an outcome which you observed)? For example, if players are actively eliminating each other in the game, then no: the class is just the order of elimination (ignoring those outside the top 3). But if players are collecting rewards, then the answer is yes: the class is the rank of rewards. I'm asking b/c if the answer is yes, then the baseline to beat is to predict rewards (assuming they're independent, even if they're not) and then rank based on those predictions.
Comment: I have edited the question to clarify.
Comment: I'm not 100% that a standard supervised learning method is the right approach (what's the definition of a class? For most real-world applications, couldn't you check each example individually?). So if there's further problem assumptions you have, it would be helpful to hear them.
Comment: Circling back to class definition -- maybe this is an outlier detection problem, and you can pick the "top 3 most outlier-ish" (as defined by the method) points and enumerate all $3! = 6$ sets of predictions. If we are given no assumptions, then maybe your "heavily penalize classifying the incorrect number" idea would work as an optimization constraint/regularization term, but I'm not sure a priori.
Comment: I have edited my question to hopefully provide more context and information about the classes. The outlier detection idea sounds interesting. Do you think there would be merit in a model to detect the top three outliers as you suggest, and then feed in those outliers to another model that was trained on only classes 1, 2, and 3?
Comment: After some quick research, following @chang_trenton's comment about learning to rank, it seems like the ListMLE objective is applicable. It's part of a larger class of ranking approaches called the [listwise ranking approach](https://en.wikipedia.org/wiki/Learning_to_rank#Listwise_approach).
Answer: Firstly, you have to acknowledge that your dataset is heavily imbalanced (class sizes are disproportionate).
Therefore, you have to train differently by oversampling (the minority classes) or undersampling (majority classes) your training set.
Alternatively, you can also modify your loss function (e.g. focal loss) to penalize the errors made on minority classes more than on majority classes.
You have to do careful exploratory data analysis to identify those features that have high correlation in predicting the minority classes. Also, identify features that have high correlation in predicting the majority classes. Discard features that have high correlation in predicting both minority and majority classes.
You could use an indicator variable for each class and id as a input feature. However, this variable is dependent/conditioned on the output class from the previous input. i.e. the previous output. For this you need to create an augmented training set that includes this indicator variable.
Comment: -1 This seems not to address the problem at hand, preferring to focus on the likely-non-problem of imbalance (see the link), and also gives some poor advice that seems to be rooted in common statistical misconceptions, starting with the idea that class imbalance is an inherent problem in need of remedy, which [it almost certainly is not](https://stats.stackexchange.com/questions/357466/are-unbalanced-datasets-problematic-and-how-does-oversampling-purport-to-he).
Comment: @Dave , care to give a detailed explaination of the misconceptions?
Comment: Class imbalance and univariate variable screening are two topics that are discussed extensively on here.

Constrained, Supervised Classification


How do I create a .tpl file for a panel pane?

Question: So, I want to add some HTML to surround the output of a Panel pane. I know the existing pane is rendered with:
panels-pane--entity-field-extra.tpl.php
If I edit this file, of course, every pane using it will show the changes.
If I create a new .tpl, say panels-pane--entity-field-top.tpl.php how do I apply it to a particular pane?
Answer: You can refer Documentation on Panels Pane template, which provides details on how to define a template for Panels Pane. If you want to add your own template name you can also do that by implementing template_preprocess_panels_pane in your theme or module file. Example for same is also available on documentation page.


QTabWidget methods not available in QtDesigner

Question: One of the methods available in the QTabWidget class is setTabEnabled - I can't see of a way to set this in QtDesigner (I know I can set it in the code) - is there a specific reason why this method (and presumably others?) aren't settable in QtDesigner?
Comment: Only properties are visible in the property editor. This is currently not a property. You could submit a feature request [here](http://bugreports.qt-project.org/). [Read this](http://qt-project.org/wiki/ReportingBugsInQt#73ea5953d68b2c20a82bb94240f6fe17) if you want to know more about requesting features.
Answer: Just change enabled property of respective widget inside QTabWidget.

according to documentation this suppose to work. I've taken a look on QTabWidget code and it looks like either documentation is wrong or there is a bug in code. I will not explain why just provide workaround on it.
Create this method:
void tabBarWorkaround(QTabWidget *tabWidget) {
    for (int i=0; i<tabWidget->count(); ++i) {
        tabWidget->setTabEnabled(i, tabWidget->widget(i)->isEnabled());
    }
}

And use it in construction time:
MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent),
    ui(new Ui::MainWindow)
{
    ui->setupUi(this);

    tabBarWorkaround(ui->tabWidget);
}

From this moment changing enabled property of page in designer will work like you want.
I've test this with Qt 5.2.1 and it works.
Comment: As far as I can tell, the 'enabled' checkbox is for the parent `QWidget`, *not* each individual page widget; if you have a widget in the page, this has properties, but individual pages don't
Comment: @ChrisW you can select the tab pages in the object explorer on the right hand side
Comment: @ratchetfreak I believe he doesn't want to disable the tab widgets, but the tabs in the tab bar.
Comment: @thuga did you read the link doc in the OP? `setTabEnabled` sets the `enabled` property of the relevant tabWidgets.
Comment: @ratchetfreak It also disables the tab from the tab bar. `The page's tab is redrawn appropriately.`. Disabling the widget from the property editor does not do this.
Comment: @thuga you were correct in my assumptions about wanting the tab disabled on the tab bar.  Setting this manually in the code does work, and produces the effect I want.  Using the enabled checkbox in Qt Designer didn't achieve it - as you state, it does only disable the widget that the `page` is a child of
Answer: I've realised that there are two separate possible events:

unchecking the 'enabled' checkbox for a tab widget in QtDesigner actually invokes self.<tabname>.setEnabled(False), which seems to disable all child widgets within the tab, but does not disable the tab button itself(!)
Disabling the tab button so that it's not clickable is not possible in QtDesigner, but is possible in code (as discussed in the previous answer): self._uiform.<tab frame>.setTabEnabled(<tab index>,False)


Java - if statement failing when comparing an element of an array to a passed int to see if identical

Question: A bit confused here. The way I have it set, when called, it should cycle through each element of the array that's populated and check to see if that element's int accNum is equal to the passed int accNum and if so output a summary of the account.
Maybe my logic is wrong but in my mind it should work:
public void accountInfo(int accNum)
    {
        for (int i = 0; i < numAccounts; i++)
        {
            if (accNum == accounts[i].getAccNum())
            {
                accounts[i].summary();
            }
            else
            {
                System.out.println("No such account on record.");
            }
        }
    }

Account class:
public class Account
{
    private String ssn;
    private int accNum, accType;
    private double balance;
    private Customer accountHolder;

    public Account(Customer accountHolder, String ssn, int accNum, int accType, double balance)
    {
        this.accountHolder = accountHolder;
        this.ssn = ssn;
        this.accNum = accNum;
        this.accType = accType;
        this.balance = balance;
    }

    public String summary()
    {
        String fullType;
        switch (accType)
        {
            case 1:
                fullType = "Checking";
                break;
            case 2:
                fullType = "Savings";
                break;
            default:
                fullType = "Other";
        }

        return String.format(("\t- Number: %d\n\t- %s\n\t- Balance: $%.2f\n\t- Customer: %s"),
                               accNum, fullType, balance, accountHolder.getName());
    }

    public double getBalance()
    {
        return balance;
    }

    public int getAccNum()
    {
        return accNum;
    }

    public void deposit(double balance)
    {
        this.balance += balance;
    }

    public void withdraw(double balance)
    {
        this.balance -= balance;
    }

}

/*Note that an Account object can have only one Customer.
 */

And my output:
========== READ DATA ==========
========== DONE ==========

========== BANK INFORMATION ==========
Bank Name: CSUMB
Number of Customers: 4
    Tom Smith: 777-77-7777
    Alice Smith: 888-88-8888
    Joe Otter: 999-99-9999
    Monica Smith: 123-45-7777
Number of Accounts: 5
    1000: $10.00
    2000: $50.25
    3000: $100.00
    5000: $100.25
    6000: $500.25
Bank Total Balance: $760.75

========== ACCOUNT INFORMATION ==========
No such account on record.
No such account on record.
No such account on record.
No such account on record.

when it should be
========== ACCOUNT INFORMATION ==========
    - Number: 1000
    - Checking
    - Balance: $10.00
    - Customer: Tom Smith

========== ACCOUNT INFORMATION ==========
    - Number: 1000
    - Checking
    - Balance: $160.25
    - Customer: Tom Smith

========== ACCOUNT INFORMATION ==========
    - Number: 1000
    - Checking
    - Balance: $60.25
    - Customer: Tom Smith

========== ACCOUNT CLOSE ==========
Account closed.

Comment: Try to run the program with the debugger and check what you actually get back from `accounts[i].getAccNum()`. Alternatively, print the value before comparing it.
Comment: *if (accNum == accounts[i].getAccNum())* Where is the getAccNum method implementation?
Comment: Please delete code from your question that is not directly required to show the problem in your `accountInfo()` method (ie most of it, especially the entire Bank class). Also, you haven't shown the Account class, upon whose `getAccNum()` your method relies.
Comment: Sorry, should have specified, getAccNum is a method in Accounts, it just returns the integer accNum associated with it.
Comment: @NinStarRune it would still be interesting to see `getAccNum`.
Answer: i donno u figured it out or not , but there are some problem in your code:

When you wanna use Class variables in the methods you have to use this before them , like this.name that you used in Bank method.
You used a method called getAccNum() but there is no method with this name.

Check each method outputs then combine them together, it will make you think better while coding because you know the method you wanna use later are works fine and if you get any error you can easily find where is the problem.
hope i could help you <3
Comment: No so: You don't have to use the `this` keyword to access instance fields/methods. In fact, I delete `this` from all source code I touch, unless it is required to disambiguate a reference, becuase it's "code noise".


How to arm specific Reaper tracks using Behringer MIDI footswitch (FCB 1010)?

Question: I'd like to switch the record arm of specific tracks in Reaper using a Behringer FCB 1010, which is a MIDI footswitch.
As an example, let's say I have 4 tracks in Reaper:
3 for guitar (same input, but different FX);
1 for piano.
I'd like to arm only one guitar track at a time. How could I use the footswitch so that pedal 1 would only arm the first guitar track and disarm the other two? Likewise, when I press pedal 2 on the footswitch, then the second guitar track should be armed, and the other two guitar tracks disarmed. But it is important for me that the piano track stays armed all the time. I think that would be a rather usual usage of controlling guitar effects via MIDI in live situations with multiple players.
I tried creating custom actions and link them to the pedals:
Pedal 1: "Toggle record arm for last touched track" + "Toggle record arm for track 01"
Pedal 2: "Toggle record arm for last touched track" + "Toggle record arm for track 02"
Pedal 3: "Toggle record arm for last touched track" + "Toggle record arm for track 03"
In this case, pedal 1 does not seem to do anything, and pedals 2 and 3 change the arming of multiple channels.
Are there options I'm missing that could help me solve that? Any help is appreciated!
Comment: "Toggle record arm for last touched track" + "Toggle record arm for track 01" : at the first glance these two seem to be redundant, why not only the second one? (Note that I do not use Reaper). Note also (may prove useful, or not) that the FCB is not sending  momentary (nothing happens when you release): if you set it to send 127 on CC5, it will do each time you press, but will not return to zero. (This caused me troubles…)
Answer: The two "toggle" commands in series relies on a specific state for the toggles to work as intended. However, according to "https://wiki.cockos.com/wiki/index.php/Action_List_Reference", there is an action called "Track: Unarm all tracks for recording (40491)"
This would allow you to know for certain the armed state of all tracks.
If you use this command instead of the "last touched" command, and then send a "toggle" command to a specific track, the arming state will always be "true."
Try: "Unarm all + toggle track 1"
Comment: Works perfectly! It doesn't seem to be that great for live situations however; I noticed that the FX for each track takes too long to switch (around half a second). I didn't yet test whether that's the case for all FX or only the one I use (EZ Mix).


Escrever em nova coluna de um arquivo csv usando StringBuilder c#

Question: Vi um exemplo de um código aqui mesmo no stackoverflow onde utilizo o exemplo abaixo para escrever um arquivo CSV usando stringbuilder:
public void ExportarCsv(List<Result> erros)
        {
            string filename = "resultado.csv";
            string path = @"C:\Arquivo\";

            Console.WriteLine("Escrevendo no arquivo.");

            StringBuilder csvBuilder = new StringBuilder();

            //cria o header
            csvBuilder.AppendLine("Order;Motivo");

            foreach (var item in erros)
            {
                csvBuilder.AppendLine(item.CodigoPedido);

            }

            if (!Directory.Exists(path))
            {
                DirectoryInfo di = Directory.CreateDirectory(path);
            }

            File.WriteAllText(path + filename, csvBuilder.ToString(), Encoding.UTF8);
        }

Nesse exemplo ai tenho o cabeçalho Order e Motivo, porem consigo escrever somente abaixo do Order. Como faço para escrever os valores em cada colunas respectivamente ?
Answer: Simples... você está fazendo isso, só não está escrevendo o resto do conteúdo. Lembrando que CSV é um arquivo com delimitadores, não existem colunas... E no seu caso o separador é o padrão, ;.
Ao invés de ir concatenando a seu texto para compor a linha, eu recomendo utilizar o recurso de interpolação indicado pelo símbolo $ antes da sua string.
foreach (var item in erros)
{
    csvBuilder.AppendLine($"{item.CodigoPedido};{item.Motivo}");
}

Comment: boa muito obrigado. Simples mesmo, nao estava conseguindo enxergar, valeu pela luz kkkk


Do conductors scores ("partitur") ever differ greatly from the full score?

Question: I believe most conductor's scores are full scores, with all instruments appearing on them in a prescribed order, in the same notation the player will see
However, it seems like there would be occasions when the full score is not ideal in some way.  Perhaps in cases of polymeter/polytonality and other complexities/ambiguities, it may be easier for the conductor to transpose or change the meter of some parts.  I suppose maybe also the score might be customised by the conductor based on different players performance.
In such cases (or any others too), do conductors highly customise the full score?
Do some conductors have drastically differing partitur for the same piece?
Comment: Would discussion of condensed scores count? My recollection is that conductors sometimes conduct from condensed scores instead, and everything in a condensed score must be in concert pitch.
Comment: Conductors (as all musicians) do make annotations on their scores, in some specific cases they would even alter it for various reasons, but that's mostly due to performance requirements ("traditional cuts", removed repeats, etc). Besides that, there's usually no distinction: the conductor's score *is* the full score. Any "complexity" is normally written just as it is in the instrument part (including differing meters), with the possible exception of transposing instruments. The conductor doesn't need a score that is comfortable to her/him, but *does* need to know what the musician are reading.
Answer: There are often various scores for a piece that conductors can choose from:

a full score with a system for each instrument, even it they are not playing. Generally several similar instruments (e.g two flutes) will be combined on one system. Normally transposing instruments are transposed in the score, but sometimes they are in concert pitch in the score (except for octave-transposing instrument like piccolo or contrabass). This needs to have exactly the same rhythmic information the players have.
sometimes the full score is optimised: if a longer section involves only a few instruments, other systems will be left out to save pages.
This is an example (the beginning of Wagner's Siegfried):

sometimes a 'condensed' score is used that has two or three lines containing all the important musical information. The is common for band music.
An example:

a 'piano conductor' score is like a condensed score but designed for a conductor who is conducting from the piano (often found in older musicals).
sometimes a conductor might even conduct from a piano reduction of the score.

In any case the conductor will usually enter a lot of markings to make sure they don't miss any anything important.
This is an excerpt from Leonard Bernstein's marked up score for Mahler's First Symphony (of course another conductor would use totally different markings):
Comment: @ModalNest those markings certainly mean something to me.  They're quite standard.  The letters are just reminders of which instrument is playing (mostly based on their German names, interestingly).  The circles just mean "don't overlook this."   The double slashes at the repeat sign similarly make it easier to notice.  Another conductor would perhaps make different decisions about what to call attention to, but most musicians would understand most markings of most conductors.
Comment: @phoog Look at any of the John Williams film scores floating around on the net: none of them are optimised. Most larger scale musicals since about the seventies are conducted from a full score e.g. "Phantom of the Opera", "The Lion King", although some conductors prefer a reduction even if they have the full score. Also note that a piano-conductor score is not always the same thing as a vocal score.
Comment: Optimized scores save paper and page turns, but require a bit of preparation to get used to reading. In situations where there will not be much (or any!) rehearsal time, an unoptimized score is preferred. This includes film scores, as well as scores prepared for reading sessions.
Comment: Thank you.  The markings seem to be the closest thing to what I was imagining.  Do those markings mean anything to anyone other than Bernstein?  I.e. Each conductor has their own system of marking.
Comment: "sometimes the full score is optimised": I've never seen one that wasn't.  Have you?  "often found in older musicals": why not newer ones?  When did the practice change? I've never seen performance materials for a musical that didn't include a piano/conductor score.  Can you cite an example?  Furthermore, the piano/conductor score is in my experience the only conductor's score available, so the conductor uses it whether conducting from the piano or from a podium.
Answer:
In such cases (or any others too), do conductors highly customise the full score?

Conductors do not typically produce the score.  This is done by the publisher, or, for a new work, the composer or a professional copyist.  As noted in the other answer, the conductor typically makes notes in the score, but it's not usually possible to modify the grouping of multiple instruments on one staff, for example.
Older works may be available in multiple different editions in which editors have made different choices about how to present the piece.  For such works a conductor may choose a particular edition based perhaps on choices such as that.  But even so, most editions will follow the practice reflected in the composer's manuscript.  Generally, composers and editors all follow standard  conventions, and deviating from them would not generally have a lot of benefit.
In the present day, with computerized music notation, it is somewhat easier for a conductor to produce a customized full score or even a complete set of performance materials.  But it is still a huge job, and most conductors won't be likely to do it.  Furthermore, for works protected by copyright, the conductor would need permission to do so, which might be denied.

it may be easier for the conductor to transpose or change the meter of some parts.

This is tantamount to sacrilege in classical music (except for renaissance and earlier music, as note values have been inflated over the centuries and there have been other significant changes in rhythmic and metrical notation).  But if a conductor wanted to do this, it would be necessary not only to change the score but also to change all the performing parts.  Otherwise it would be too confusing to try to communicate with the orchestra during rehearsals, because the conductor would have to remember to translate from the meter in the full score to the meter in the players' parts.


Gradient And Hessian Of General 2-Norm

Question: Given $f(\mathbf{x}) = \|\mathbf{Ax}\|_2 = (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}$,
$\nabla f(\mathbf{x}) = \frac {\mathbf{A}^\mathrm{T} \mathbf{Ax}} {\|\mathbf{Ax}\|_2} = \frac {\mathbf{A}^\mathrm{T} \mathbf{Ax}} {(\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}}$
$\nabla^2 f(\mathbf{x}) = \frac { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2} \cdot \mathbf{A}^\mathrm{T} \mathbf{A} - (\mathbf{A}^\mathrm{T} \mathbf{Ax})^\mathrm{T} (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{-1/2} \mathbf{A}^\mathrm{T} \mathbf{Ax} } {(\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} ) } = \frac { \mathbf{A}^\mathrm{T} \mathbf{A} } { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{1/2}} - \frac {\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{A} \mathbf{A}^\mathrm{T} \mathbf{Ax} } { (\mathbf{x}^\mathrm{T} \mathbf{A}^\mathrm{T} \mathbf{Ax} )^{3/2} }$
I guess I am looking for confirmation that I have done the above correctly. The dimensions match up except for the second term of the Hessian is a scalar, which makes me think that something is missing.
Edit: Also, the last equality reduces to
$\nabla^2 f(\mathbf{x}) = \frac {\mathbf{A}^\mathrm{T} \mathbf{A} - \nabla f(\mathbf{x})^\mathrm{T} \nabla f(\mathbf{x})} {\|\mathbf{Ax}\|_2}$
Comment: My advice is to do it first without the square root. Given your $f,$ define $g = f^2$ and find the gradient and Hessian. Just to confirm your suspicions, the gradient really is a vector, the Hessian really is a square (symmetric) matrix.
Comment: Try squaring and expanding $f^2(x+\delta)$. The gradient and Hessian (of $f^2$) should be fairly clear.
Comment: It looks to me like everything is fine except that the numerator should be $A^TA - (\nabla f)(\nabla f)^T$. (Also, $\nabla^2$ usually denotes the Laplacian, which is the trace of the Hessian.)
Comment: Also, do this in agonizing detail for the case of just two variables, you could call them $x_1,x_2$ if you like, but you could also stick with $x,y.$ The gradient becomes a vector with just two functions as entries, the Hessian a 2 by 2 matrix. You are getting a bit lost in the formalism.
Answer: It is easier to work with $\phi(x) = \frac{1}{2} f^2(x)$. Just expand $\phi$ around $x$.
$\phi(x+\delta) = \frac{1}{2} (x + \delta)^T A^T A (x + \delta) = \phi(x) + x^TA^TA \delta + \frac{1}{2} \delta^T A^T A \delta$. It follows from this that the gradient $\nabla \phi(x) = A^T A x$, and the Hessian is $H = A^TA$.
To finish, let $g(x) = \sqrt{2x}$, and note that $f = g \circ \phi$. To get the first derivative, use the composition rule to get
$D f(x) = Dg(\phi(x)) D \phi(x)$, which gives $Df(x) = \frac{1}{\sqrt{2 \phi(x)}} x^T A^T A = \frac{1}{\|Ax\|} x^T A^T A$.
Let $\eta(x) = \frac{1}{\|Ax\|}$, and $\gamma(x) = x^T A^T A$, and note that $D f(x) = \eta(x) \cdot \gamma(x)$, so we can use the product rule. Let $h(x) = Df(x)$ then the product rule gives $D h(x) (\delta) = (D \eta(x) (\delta))  \gamma(x) + \eta(x) D \gamma(x) (\delta)$.
Expanding this yields: $Dh(x)(\delta) = (- \frac{1}{\|Ax\|^2} \frac{1}{\|Ax\|} x^T A^T A \delta) x^T A^T A + \frac{1}{\|Ax\|} \delta^T A^T A $. Noting that $x^T A^T A \delta = \delta^T A^T A x$, we can write this as:
$$Dh(x)(\delta) = \delta^T(\frac{1}{\|Ax\|} A^T A - \frac{1}{\|Ax\|^3 } A^T A x x^T A^T A),$$
or alternatively:
$$D^2 f(x) = \frac{1}{\|Ax\|} A^T A - \frac{1}{\|Ax\|^3 } A^T A x x^T A^T A .$$
The only difference with the formula given in the question is that the latter dyad was written incorrectly (instead of the dyad $g g^T$, you have the scalar $g^T g$).
Comment: This is helpful but not complete. To get to the answer, one now needs the chain rule to find the gradient and Hessian of $f(\mathbf{x})=2\sqrt{\phi(\mathbf{x})}$.
Comment: @AnonSubmitter85: I elaborated my answer a little more.
Comment: Thank you. This has helped me tremendously.
Comment: You're very welcome.

Gradient And Hessian Of General 2-Norm


Find aggregated values and relative values based on a value in a table using lookup

Question: Hello Power BI community. I have a challenge in powerbi. I searched in the forum but I could not find the answer. I have a table of costs of different categories in different years. I want to search categories and find the aggregated costs in each category in each year. But I want to be able for each to display the aggregated cost of previous year(year-1) of each selected category and year.
My output would be:
Cost of Category in year x(the year and the category would be input by user,) Cost of Category.. in previous year (it is year-1).
The table, I have looks like this:
<code> Year   CostItem 1  CostItem2   Category
 2010   1000         24            A
 2010   1250         3115          A
 2010   455          5115          A
 2010   5848.45      4904          B
 2010   34343        3352          B
 2010   4193         2899          B
 2010   824          3031          B
 2010   3944    3394    B
 2010   1546    3527    C
 2010   2175    1391    C
 2010   4942    2336    C
 2010   2988    5803    C
 2010   2423    3520    C
 2011   4132    2077    A
 2011   1666    834 A
 2011   2043    4419    A
 2011   4307    4030    A
 2011   4303    3339    A
 2011   4989    4619    A
 2011   5977    1655    A
 2011   4338    3698    A
 2011   2734    1321    B
 2011   1662    782     B
 2011   3036    3164    B
 2011   4488    5891    B
 2011   1679    5513    B
 2011   4169    4013    B
 2011   5647    4445    C
 2011   2728    5219    C
 2012   3155    2333    A
 2012   3893    1657    A
 2012   3626    1880    A
 2012   5805    796     A
 2012   753 4137        A
 2012   5537    4230    B
 2012   542 4244        B
 2012   2811    3300    B
 2012   866 3717    C
 2012   2356    2257    C
 2012   2712    2813    C
 2012   2250    4878    C
 2012   796 1610    C
 2012   4297    4241    C
2010    1250    3115    A
 2010   455 5115    A
 2010   5848.45 4904    B
 2010   34343   3352    B
 2010   4193    2899    B
 2010   824 3031    B
 2010   3944    3394    B
 2010   1546    3527    C
 2010   2175    1391    C
 2010   4942    2336    C
 2010   2988    5803    C
 2010   2423    3520    C
 2011   4132    2077    A
 2011   1666    834     A
 2011   2043    4419    A
 2011   4307    4030    A
 2011   4303    3339    A
 2011   4989    4619    A
 2011   5977    1655    A
 2011   4338    3698    A
 2011   2734    1321    B
 2011   1662    782     B
 2011   3036    3164    B
 2011   4488    5891    B
 2011   1679    5513    B
 2011   4169    4013    B
 2011   5647    4445    C
 2011   2728    5219    C
 2012   3155    2333    A
 2012   3893    1657    A
 2012   3626    1880    A
 2012   5805    796     A
 2012   753     4137    A
 2012   5537    4230    B
 2012   542     4244    B
 2012   2811    3300    B
 2012   866     3717    C
 2012   2356    2257    C
 2012   2712    2813    C
 2012   2250    4878    C
 2012   796     1610    C
 2012   4297    4241    C
</code>
I tried using lookup function and value functions but could not end up a solution.
Answer: You should model your data a bit different, go to query editor, you starting point:

Select the 2 columns CostItem1 and CostItem2. Go to Tab Transform and select: Unpivot, the result is as below:

Now you can use the matrix visual to create 2 levels, first category and then Attributes.

End result is overview per year:
Comment: Thanks @aldert, so with what formula I can extract the cost item if I do not write the year exactly?
Comment: What do you mean by: i do not write the year exactly?


$n$ degree polynomial $p(k)=\frac{k}{k+1}$ for integral $k=0$ to $n$.

Question: Here's the problem statement: Given an $n$ degree polynomial $p(k)$ such that:
$$p(k)=\frac{k}{k+1}$$
for all integer $k$ from $0$ to $n$, determine $p(n+1)$. Any ideas on how to solve it?
Comment: Duplicate of http://math.stackexchange.com/q/668108/115115
Comment: There's always Lagrange interpolation.
Comment: For specific cases, like for n=3 or 4, I can determine the polynomial by determining p(0), p(1), p(2) till p(n). I get n+1 simultaneous equations which I solve to get the coefficients of the polynomial and then plug in (n+1). But it's too long and time consuming.
Answer: The answer is $P(n+1)=1$.
Indeed, consider $Q(X)=(X+1)P(X)-X$. We have $\deg Q\leq n+1$ and $Q(k)=0$ for $k=0,1,2,\ldots,n$. Thus, there is a constant $\lambda$ such that $Q(X)=\lambda\prod\limits_{k=0}^n(X-k)$.
To determine lambda, we note that $Q(-1)=1=\lambda(-1)^{n+1}(n+1)!$, therefore
$$
XP(X+1)-X=\frac{1}{(n+1)!}\prod_{k=0}^n(k-X)
$$
Substituting $X=n+1$ we get $P(n+1)=1$.


From a mathematician's point of view, what is the purpose of '$dx$' in $\int f(x)\ dx$?

Question: I've done a bit of searching and found a fairly well written explanation, but at the end, the author noted that this explanation seems to work fine for a physicist's purposes - but a mathematician would groan at it due to oversimplifications or inaccuracies.

Since I first posted this paper, two different people have emailed me
  to tell me that Real Mathematicians don't do this. Playing with dx in
  the ways described in this paper is apparently one of those smarmy
  tricks that physicists use to give headaches to mathematicians.

http://www4.ncsu.edu/unity/lockers/users/f/felder/public/kenny/papers/dx.html
I was also confused when reading it because my Calculus prof last semester said that the chain rule
$$\frac{dy}{dz}=\frac{dy}{dx}\cdot \frac{dx}{dz}$$
cannot be treated as fractions, despite the fact that they look like it, and the $dx$ would not cancel out between the two since you can't do that with differentials. But this article said just the opposite.
Comment: You can think of it as just a symbol to say that your integrating function is a function of the variable $x$. So, you know what I'm saying if I write $\int x^2y+\cos(yx) dx$.
Comment: duplicate of http://math.stackexchange.com/q/200393/264
Comment: Your calculus professor is right. A lot of these little "tricks", however, are allowed in a branch of mathematics known as non-standard analysis.
Comment: @ZevChonoles: I disagree with closing this as a duplicate since this question seems to contrast the physicists and the mathematicians way of understanding $dx$. I think something useful can be said here.
Comment: http://meta.math.stackexchange.com/questions/6424/requests-for-reopen-votes/8243#8243
Comment: If you try to use this trick with **partial derivatives** you get demonstrably incorrect results. For plain derivatives, though, it shouldn't lead to any contradictions. And I think it is useful for intuition, although you should know it is a not a rigorous proof.
Answer: I have heard from several mathematicians that one thing that they didn't like about physics was the "abuse" of mathematics. Physicists are not necessarily bad mathematicians, they (IMO) just use math differently. For them, the precise nature isn't (always) at the core of what they are doing. Since the physicist works with the "real world", he/she doesn't often work with approximations and therefore is not passionately interested in the exact nature of equations and such (hopefully I didn't offend anyone).
I see on the site that you give reference to, that the person thinks about $dx$ as a finite quantity. He talks for example about $x + dx$ as being a real number. This is not necessarily a bad way to think about it, but again it isn't the exact definition.
About the integral: When we write for example
$$
\int_a^b f(x)\; dx
$$
we have defined this symbol as is. We haven't defined the symbol as made up of different individual component that can we written on their own. So even though $f(x)$ in the expression indeed does have meaning on its own, we are not allowed to remove it and just write:
$$
\int_a^b dx.
$$
Now, someone might actually use this notation from time to time, but in "standard basic calculus" this hasn't been defined, and so it doesn't have any meaning. (It doesn't mean that we can think about it in a certain way).
Now all that said, this doesn't mean that there isn't a rationale behind the choice of the different components of the symbol. So, of course, the $f(x)$ is the function. The $a$ and the $b$ are the limits of the integral and the $dx$ tells us to treat $x$ as the variable (we say that we integrate with respect to $x$). Why is this important? As mentioned in another answer, if you consider the integral:
$$
\int_{a}^{b} x^2y.
$$
you wouldn't know if $x$ is the variable of $y$ is the variable (or if both are constants). So that is why we like to "add" the $dx$ or $dy$ to the end.
This isn't the only reason. The definite integral
$$
\int_a^b f(x)\; dx
$$
is defined as a limit:
$$
\lim_{n\to \infty}\sum_{i=1}^{n} f(x_i)\Delta x.
$$
Here $\Delta x$ is a finite number defined as $\Delta x = \frac{b - a}{n}$. And taking the limit $n\to \infty$ we have $\Delta x \to 0$. So we "think" of $dx$ as this infinitely small number. We can "think" of the $\Delta$ as becomming the $d$, even though the $dx$ can't be written on its own.
The same applies to the derivative
$$
\frac{dx}{dy}.
$$
Here again we have just defined this one symbol where the parts of the symbol ($dx$ and $dy$) are not well defined on their own.
All this said, one can actually make a meaning (i.e. define) of $dx$, but when we are just taking about basic calculus, we don't usually do that.
Also note that some mathematicians will just write $\int f(x)$ when they mean $\int f(x) \; dx$ because it is "obvious" that we are integrating with respect to $x$.
You can find more things about the $dx$ in this question: What is $dx$ in integration?
Answer: A possibility is to see $dx$ as a differential form. Then, the integration for differential forms is well defined.
Answer: There are a number of ways to view the $dx$. As Seirios pointed out it might be a differential form, which is the usual way to have it "make sense.", but it's often simply defined as the notation of an integral. Integrating a function $f$ with some measure $\mu$ can be described by the notation $\int f d\mu$ and when the measure used is the Lebesgue measure it is customary to replace $d\mu$ with $dx$. In this way the notation "makes sense" because it is well defined, but the $dx$ itself has no meaning without the explicit context of the integral.
$\frac{dy}{dx}$ means something entirely different, that is the derivative of  $y$ with respect to $x$. The reuse of the symbols, and the fact that $\frac{dy}{dx}$ looks like a fraction is only because the symbols were in use long before the appropriate formalism was invented for calculus to make sense rigorously, and at that point there was no turning back.
Answer: Remember, that from a historical perspective, differential calculus was developed first from the "physics" point of view, that is, $\int f(x) dx$ was originally thought of as a sum (the $\int$ sign is actually an $s$):
$$\sum f(x_i)\Delta x$$
Similarly, the chain rule makes a lot of sense if used as a fraction, and that was it's original meaning. The problem is that when trying to set mathematics on a rigorous foundation, one could no longer use the original "intuitive" definitions. These "physical" methods help us think about what the definitions mean, but we cannot use them in mathematical proofs.


Removing the penalty on the quadratic term in a penalized cubic spline in MGCV?

Question: I'd like to fit a penalized cubic spline with the R package mgcv in which I don't apply any penalty to the intercept, linear, AND quadratic terms in the model. The penalty should only apply to the cubic and other terms in the spline basis. I'd like to fit my model this way because the standard in my field is to use a quadratic term to adjust for x in some code like lm(y~x+x^2). I believe that there could be moderate departures from this model in my data, so I would like to fix a more flexible (but not too wiggly) model, and hence use the penalized splines.
It is my current understanding that mgcv will automatically place no penalty on the intercept and linear term, but the quadratic term will be penalized.
So, if my working model can be fit with the following code
x <- seq(0,1, length = 100)
y <- 0.5*x + x^2 + rnorm(100)
mod1 <- gam(
    y~s(x, fx = F, k = 5, bs = "cr")
)

then calling mod1$coefficients yields a vector of length 5, representing the intercept, linear term, quadratic term, cubic term, and one cubic spline term.
Hence, it is my current understanding that mod1$coefficients[1:2] are not penalized and mod1$coefficients[3:5] are penalized. Is my understanding correct? If so, how might I modify the code above to remove the penalty in the estimation of mod1$coefficients[3]?
I have tried toying with the parameter m within the spline function s(), as the mgcv documentation indicates that this will alter the derivative of the spline function on which the penalty is placed. However, this doesn't seem to alter the fitted spline at all.
mod1 <- gam(
    y~s(x, fx = F, k = 10, bs = "cr")
)
mod2 <- gam(
    y~s(x, fx = F, k = 10, bs = "cr", m = c(3,3))
)
all(mod1$fitted.values == mod2$fitted.values) # this is always true

Comment: The proper way to do this in `lm` might be with `poly(x, 2)` and then to compare with `poly(x,3)` but that might not fit with the mgcv methods. You can force the quadratic and cubic terms into a formula with `I(x^2)` and `I(x^3)` but then you loose the orthogonality features of `poly`.  Does the "standard approach" in your field understand the statistical issues that arise when quadratic and cubic terms are entered into regression formulas? (Note: in R, you cannot do this with `x^2` and `x^3`.)
Answer: It's not a cubic regression spline, but it does clearly separate the two components of your problem. Here I show how to do this with a thin plate spline, the default basis in mgcv::gam().
Using your setup plus my package for working with GAMs
library("mgcv")
library("gratia")
library("ggplot2")

set.seed(1)
df <- data.frame(x = seq(0,1, length = 100),
                 y = 0.5*x + x^2 + rnorm(100))

we begin by looking at the basis for a thin plate spline with a 3rd derivative penalty:
basis(s(x, m = 3), data = df) |>
  draw() +
  facet_wrap(~ bf)

which results in

The last three basis functions (9, 10, & 11) are in the null space of the penalty; they are not affected by the penalty because they have 0 third derivative everywhere. Function 11 is a quadratic term. Function 9 is confounded with the model intercept and will be removed from the basis via the imposition of a sum-to-zero constraint on the basis; this is the default constraint and gam() does this by default when fitting a GAM.
The GAM you want to fit (assuming a thin plate spline is OK) then is
m <- gam(y ~ s(x, m = 3), data = df, method = "REML")
summary(m)

The model uses 2 EDF as we would expect given the way the data has been simulated
> summary(m)

Family: gaussian
Link function: identity

Formula:
y ~ s(x, m = 3)

Parametric coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.6939     0.0907   7.651 1.47e-11 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Approximate significance of smooth terms:
     edf Ref.df    F  p-value
s(x)   2      2 11.1 4.56e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

R-sq.(adj) =   0.17   Deviance explained = 18.6%
-REML = 134.45  Scale est. = 0.82259   n = 100

If you want a formal test of the spline bit over the order 2 polynomial terms, then we can refit the model above but including the required terms as parametric terms via poly(x, 2) but modify the basis of the thin plate spline to get rid of all the functions in the penalty null space. We remove the null space by setting m = c(3,0):
basis(s(x, m = c(3, 0)), data = df) |>
  draw() +
  facet_wrap(~ bf)

Notice how functions 9, 10, and 11 are no longer found in the basis. This is what will allow us to isolate the wiggliness-beyond-quadratic component in the spline, leaving the order-2-polynomial-wiggliness to the parametric terms in the model.
m0 <- gam(y ~ poly(x, 2) + s(x, m = c(3, 0)), data = df, method = "REML")
summary(m0)

The last line produces
> summary(m0)

Family: gaussian
Link function: identity

Formula:
y ~ poly(x, 2) + s(x, m = c(3, 0))

Parametric coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.6939     0.0907   7.651 1.47e-11 ***
poly(x, 2)1   4.2437     0.9113   4.657 1.02e-05 ***
poly(x, 2)2   0.5129     0.9074   0.565    0.573
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Approximate significance of smooth terms:
           edf Ref.df F p-value
s(x) 0.0001194      8 0    0.72

R-sq.(adj) =   0.17   Deviance explained = 18.6%
-REML = 130.47  Scale est. = 0.82259   n = 100

And as we'd already seen, the beyond-quadratic wiggliness is tiny and we fail to reject the null hypothesis that the f(x) === 0.

I think you can do this using the B spline basis, which allows you to control the order of the polynomial and the order of the penalty (unlike the cubic regression spline basis, which is fixed to be formed from piecewise cubic polynomials, with 2nd order derivative penalty):
# make it mildly cubic
set.seed(1)
df2 <- data.frame(x = seq(0,1, length = 100),
                  y = 0.5*x + x^2 + (0.1 * (x^3)) + rnorm(100))

m1 <- gam(y ~ s(x, bs = "bs", m = c(3,3)),
          data = df2, method = "REML")

This will penalise functions with non-zero third derivative so shouldn't be penalising the linear and quadratic functions in the span of the basis. It's just much less clear how it is doing this, and you certainly can't interpret the coefficients in the way you assumed for the CRS (you can't interpret them that way for the bs = "cr" basis either - plot the basis functions for s(x, bs = "cr") to see why). This is what the fitted basis functions look like for m1 with the B spline basis
basis(m1) |>
  draw()

The fitted functions have to look like because the estimated spline is:
draw(m1)

Answer: Here's an appoach to adding a cubic term to a quadratic model:
> x <- seq(0,1, length = 100)
> y <- 0.5*x + x^2 + rnorm(100)
> mod1 <- gam(
+     y~s(x, fx = F, k = 3, bs = "cr")+ I(x^3)
+ )
> mod1$coefficients
(Intercept)      I(x^3)      s(x).1      s(x).2
-1.70442708  9.32342373  0.03978659 -5.49355285
> summary(mod1)

Family: gaussian
Link function: identity

Formula:
y ~ s(x, fx = F, k = 3, bs = "cr") + I(x^3)

Parametric coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   -1.704      1.014  -1.680   0.0961 .
I(x^3)         9.323      3.999   2.331   0.0218 *
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Approximate significance of smooth terms:
     edf Ref.df     F p-value
s(x) 1.8   1.96 1.925   0.125

R-sq.(adj) =  0.255   Deviance explained = 27.6%
GCV = 0.94177  Scale est. = 0.90598   n = 100

Here's the output of plot:
png( ); plot(mod1) ; dev.off()

Compare with:
> mod2 <- gam(
+     y~s(x, fx = F, k = 3, bs = "cr")
+ )
> summary(mod2)

Family: gaussian
Link function: identity

Formula:
y ~ s(x, fx = F, k = 3, bs = "cr")

Parametric coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  0.64997    0.09781   6.646 1.74e-09 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Approximate significance of smooth terms:
       edf Ref.df     F  p-value
s(x) 1.433  1.679 14.62 1.35e-05 ***
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

R-sq.(adj) =  0.213   Deviance explained = 22.4%
GCV = 0.98046  Scale est. = 0.9566    n = 100
> png( ); plot(mod2) ; dev.off()
quartz
     2


Stored passwords and scheduled tasks in Windows Server 2003

Question: In general, if you are PC\Admin and you have a stored password for network share DOMAINSERVER\Share (done using the interface "Stored User Names and Passwords"), and you have a scheduled task that runs as you even when you are not logged in, will a script that uses, say, xcopy be able to copy over to that share drive using your credentials?
Answer: It should work.  When you schedule the Task, you can specify a user and password to use when the task runs,  use the Domain\user and password.   I prefer to create special accounts for this, as if you use a user's acct (your own) and the password changes, the tasks will fail.  SO I create a account just for scheduled tasks, set a really long password,  limit the rights to the bare minimum, and exempt it from the password expiration that my domain users have.
Answer: I reckon so.  Never tested it, but when the process impersonates you, it should be able to reference your stored credentials.  It's easy to test... why not try it.


Prepared default query with result to prepared query with bindings

Question: I have a query that is working but is does not have binded values:
$this->_db->query("SELECT * from posts WHERE post_title LIKE '%$this->search_keywords%' OR post_content LIKE '%$this->search_keywords%' LIMIT 100");
$this->rows_results_found = $this->_db->resultset();

$this->results_found_num = sizeof($this->rows_results_found);

I am rewriting it to this one but with no luck:
$this->_db->query('SELECT * from posts WHERE post_title LIKE :search_keywords OR post_content LIKE :search_keywords LIMIT 100');
            $this->_db->bind(':search_keywords', '%$this->search_keywords%');

            $this->rows_results_found = $this->_db->resultset();

            $this->results_found_num = sizeof($this->rows_results_found);

$this->results_found_num appears always to be an empty array.
This is what I have in the resultset() (It is from another outer class):
public function resultset() {
        $this->execute();
        return $this->stmt->fetchAll(PDO::FETCH_ASSOC);
    }

What I am missing here?
Thank you in advance!
Comment: Please RTFM how to use prepared statements: http://php.net/manual/en/pdo.prepared-statements.php
Comment: Uhm, just for the record, I wrote the Fk manual. Thank you! :)
Comment: ...then you should be more than qualified to answer your own question...?! :) \*confused\*
Comment: At the very least we're missing a ton of code here, there's not enough information on what the problem is or what exactly is going on.
Answer: Assuming you have a valid  connection adjust your function to the following:
public function get_posts($conn) {
    $query = 'SELECT *
              FROM posts
              WHERE post_title LIKE :search_keywords1 OR
                    post_content LIKE :search_keywords2 LIMIT 100';
    $stmt = $conn->prepare($query);
    $stmt->bindValue(':search_keywords1', '%'.$this->search_keywords.'%');
    $stmt->bindValue(':search_keywords2', '%'.$this->search_keywords.'%');
    $stmt->execute();

    return $stmt->fetchAll(PDO::FETCH_ASSOC);
}

Usage:
$posts = $this->_db->get_posts($conn);
//var_dump($post)
$this->rows_results_found = count($post);

Comment: Thanx a much! I just did not have my query variable and it was not prepared. Now everything is working! I am just using $this->_db-> instead of $stmt-> but I suppose it is the same because my connection is set in another class, that is why I am using _db->. Once again, thanx! :) This was what I needed!


Understanding the .binvox file format

Question: I have a file with a list of (x,y,z) coordinates of points which I am attempting to convert into a voxel file format readable by viewvox. I have come across the binvox file format which seems like it should be relatively simple but I just can't get my head around it. In particular when it mentions the use of a value byte and a count byte I must have misunderstood because it is unclear to me why the range for count must be 1<=count<=255.
Hopefully somebody can do one or more of the following: explain the format to me like I am five years old, provide a working example of a binvox file for a small model or suggest a better way of converting my coordinate data to a voxelised format.
Thanks in advance for any advice.
EDIT: After thinking some more I've understood the range of the count byte - 1 is the minimum because a count of 0 would be uninformative and the maximum is 255 because that is the maximum value of a byte.
Answer: Sorted it out in my head now. For anyone else who has similar comprehension issues here is the .binvox file format explained:

Header goes first and must contain

Version number
Dimensions of voxel grid
Translation and scale factors to go from model coordinates to unit cube.
The word "data" before the actual data begins

An example header would be:
<code>#binvox 1
dim 512 512 512
translate 0 0 0
scale 1
data
</code>
Then the rest of the file after this is composed of pairs of bytes giving the voxel value (1 for presence and 0 for absence) and the count of voxels (e.g. this byte would be 2 if there was a run of 2 coordinates with the same value). There are no separators between byte pairs. The order of these byte pairs is determined by running through your model coordinates; with y changing fastest, then z, then x.
Now I understand it, I can see that the documentation is actually already good and all I have done here is summarised it. Actual documentation can be found here.
Comment: @GregCawthorne AFAIK the binvox format does not specify colour at all, just the 3D structure.
Comment: Hi. Do you know if more than 255 colours can be represented with this format?

Understanding the .binvox file format


C# How to pass ARRAY from another Method

Question: I'm a beginner with C# and I need little help. I'm not sure how can I take 'studentArray' from CreateStudent() and use it at ViewAllStudents(). Any help is appreciated!
<code>        static void CreateStudent()
        {
            // User Inputs
            Console.WriteLine("Student Name: ");
            var name = Console.ReadLine();
            Console.WriteLine("Student Surname: ");
            var surname = Console.ReadLine();
            Console.WriteLine("Student ID: ");
            var id = int.Parse(Console.ReadLine());

            var student = new Students(name, surname, id); // Student Constructor

            // Create Array and Resize it
            Students[] studentArray = new Students[] { student };
            Array.Resize(ref studentArray, studentArray.Length + 1);
            studentArray[studentArray.GetUpperBound(0)] = student;

            Console.WriteLine($"Student {student.Name} {student.Surname} with ID {student.StudentId} is successfully created!");
        }
        static void ViewAllStudents()
        {
            foreach(var student in studentArray)
            {
                Console.WriteLine($"\t {student}");
            }
        }
    }
}
</code>
Answer: Change return type of CreateStudent from void to Students[] and return "studentArray" . Change the parameter type of ViewAllStudents() to "ViewAllStudents(Students[] studentArray )" .I hope the following will work.
<code>static Students[] CreateStudent()
{
    // User Inputs
    Console.WriteLine("Student Name: ");
    var name = Console.ReadLine();
    Console.WriteLine("Student Surname: ");
    var surname = Console.ReadLine();
    Console.WriteLine("Student ID: ");
    var id = int.Parse(Console.ReadLine());

    var student = new Students(name, surname, id); // Student Constructor

    // Create Array and Resize it
    Students[] studentArray = new Students[] { student };
    Array.Resize(ref studentArray, studentArray.Length + 1);
    studentArray[studentArray.GetUpperBound(0)] = student;

    Console.WriteLine($"Student {student.Name} {student.Surname} with ID {student.StudentId} is successfully created!");
     return studentArray;
}

static void ViewAllStudents(Student[] studentArray)
{
    foreach(var student in studentArray)
    {
        Console.WriteLine($"\t {student}");
    }
}
</code>
Comment: Demir...I'd take it one step further and show in `main()` (presumably) how you'd capture the return value of `CreateStudent()` and then pass that along to `ViewAllStudents()`.


How to transfer a Database on a weekly basis for analysis purposes

Question: Just wondering what kind of processes you guys have for transferring production data offsite for analysis purposes?
I have read some of the literature but I am not sure of the overheads involved and the setup processes.

Replication: With the production acting as both the distributor and publisher. I will employ a push system that updates my database on my analysis server weekly
SSIS: No experience but seems to be the 'standard' way?
Manual: Use windows scheduler to zip back my backup file and open up an FTP connection to my analysis server
Mirroring: No experience but used more for high availability?

FYI, I am using SQL Server 2008 with Windows Server 2008 R2. MY DB is about 36 gigs. I am only interested in the data.
Other methods welcome!
*EDIT
Additional information:
I don't mind the database being read only and I am looking to run this process once a week.
Comment: what's the change rate of the database in question?
Comment: not very much at the moment. you can assume the database is constantly aroud 36 gigs
Comment: How close in terms of bandwidth is your 'analysis server' to your database server?
How much down-time is permissible for your 'weekly load'?
Answer: You could look at log shipping, very similar to your 'manual' approach but uses built-in SQL components to do the work and works with transaction log backups.
A great explanation and comparision to db mirroring can be see at http://blogs.technet.com/josebda/archive/2009/04/02/sql-server-2008-log-shipping.aspx
Answer: As a mere suggestion, you say your db is 36 gigs. Depending on the compression + speed of transfer you might find it easier to dump it on a USB drive and snail mail it.
Furthermore, some ftp will never allow files of > 4GB. Incremental pushes might be your way, but again consider the worst case: what happens if your data changes faster than your db can push it to the replicator?
Answer: I have a client who needed this exact thing.  They couldn't accept the db being inaccessible when the logs restore so log shipping was out.  Replication was not possible because their schema would be too much of a hastle.  Mirroring wouldn't work because it's read-only, and they needed a writable database... but those writes did not have to go back to the main server.
I ended up writing a maintenance plan and a few scripts that does a full backup of the db in question (120 GB), then uses xp_cmdshell to copy the file to a network share on the other SQL server, and then lastly executes a job remotely (you can set what server to execute on in the plan step) to do a restore of the database.
You may want to check out Tara Kizer's isp_Restore script which can help you with this.  http://weblogs.sqlteam.com/tarad/archive/2005/11/08/8262.aspx  I ended up writing my own script because hers didn't do exactly what I needed, but it should at least get you started.
Comment: Keep in mind, this job I created only runs once a night, however it can easily accommodate more frequent updates with diff/trn backups, but again the client couldn't accept the database being unavailable during the restores.  If you don't mind the DB being inaccessible, log shipping might be perfect for you depending on how heavy your writes are cause you will get big TRN's that will take longer to restore.  Just set the restore frequency to once an hour.  I have also done this for clients who can take the inaccessible DB during restores and didn't need writable databases.
Comment: Shouldn't there be somesort of 'RECOVERY' option? I mean afterall log shipping is also meant to provide a backup of your database? How else would you restore it anyway? It sounds like log shipping might be for me considering I don't mind it being read only. However, there probably will be a need to create bespoke indices which is not available unless the DB can be written to.
Comment: If the database restores nightly, the database will be offline, which is the same effect as log shipping. Why not just use Log Shipping but set it to only run once a day at whatever time the script runs. This way it's just one restoration a night, and you're not copying the entire database, and would most likely execute quicker than a full backup/restore...
Comment: For me anyway, the client needed the full backup/restore because of how they were working with the data.  I did it this way because I wanted to have complete control of what was happening, also log shipping would require the LSN to be maintained at all times which there was no guarantee of which would break log shipping.  With my script I can configure it to do the full backup/restores and get diffs or trns worked into the mix as well if they wanted more up to date information throughout the day.  Log shipping would have to be reconfigured every time they wanted to change things around.
Answer: Log shipping is great for analysis, but be prepared for high overhead with the initial backup and transfer of the data to another server/instance.  Additionally, log shipping implies a delay between live data and analysis data.  After that, you're only moving transaction log backups, which are theoretically smaller.
In our environment, we log ship nightly, because management runs reports when the come in to the office in the morning, and those reports typically require a significant amount of pivots.
Answer: We had a 4GB database, where we infrequently needed a backup for analyzing issues. The database was too big to take a backup during the day while production was running. So our solution was to run a backup at night when our production system was down and keep the most recent backups (I think we kept one week).
One of our requirements was that the database needed to be shipped to different departments depending on the issue. Another reason was, that we could make the backup files available to Project Managers, who  did not have direct access to any SQL server and they were in charge of sending the backups to the appropriate recipients. They didn't needed to involve Admins to get the backups.
The daily backups were scheduled using the standard SQL server tools.
Comment: How did you actually go about 'sending' the DBs to the respective departments? It sounds like you used log shipping too?
Comment: No we just took a full offline backup. Than we send the backup to the person that needed it (which is sometimes tricky, since the zipped backup file was around 80 MB). The person receiving the Backup had his/her own SQL server and imported (or whatever that function was called) the database. You need to change the paths of the database files when importing.
Answer: Assuming your target analytic database is the same platform as your production database, the easiest solution is to do a weekly incremental backup and then apply it to your analytic database.  If your database isn't growing very quickly, an incremental file should be transmittable in a reasonable period of time.
If you are transmitting to some other database type than what you are currently running in production, an ETL tool like Pentaho Data Integration might be a better fit.  An ETL tool is an even better solution if you only care about some of the tables and not all of them.


Tunneling a public IP to a remote machine

Question: I have a Linux server A with a block of 5 public IP addresses, 220.127.116.11/29.
Currently, 126.96.36.199 is assigned to eth0, and 220.127.116.11 is assigned to eth0:1.
I have another Linux machine B in a remote location, behind NAT.  I would like to set up an tunnel between the two so that B can use the IP address 126.96.36.199 as its primary IP address.
OpenVPN is probably the answer, but I can't quite figure out how to set things up (topology subnet or topology p2p might be appropriate.  Or should I be using Ethernet bridging?).  Security and encryption is not a big concern at this point, so GRE would be fine too -- machine B will be coming from a known IP address and can be authenticated based on that.
How can I do this?  Can anyone suggest an OpenVPN config, or some other approach, that could work in this situation?  Ideally, it would also be able to handle multiple clients (e.g. share all four of spare IPs with other machines), without letting those clients use IPs to which they are not entitled.
Comment: I hope you just made up those addresses, rather than that you actually work at Google. If you don't, you aren't going to be able to use their address space.
Comment: For future reference, [we have an RFC for that](http://tools.ietf.org/html/rfc5737).
Comment: What firewalls are at both locations?
Comment: Robert: **A** is a Linux server with some simple `iptables` rules.  **B** is behind a NAT that is another Linux server running `shorewall`.
Comment: Michael: Yes, I changed the first three octets to 8 to obfuscate them, but still indicate that they're public.  Sorry, Google.
Answer: I ended up going with the Ethernet bridging.  Lots of extremely verbose examples to wade through online, but it turns out to be pretty easy:
First, on A, /etc/network/interfaces was changed from:
auto eth0
iface eth0 inet static
    address 184.108.40.206
    netmask 255.255.255.248
    gateway 22.214.171.124

to:
auto br0
iface br0 inet static
    address 126.96.36.199
    netmask 255.255.255.248
    gateway 188.8.131.52
    pre-up openvpn --mktun --dev tap0
    bridge_ports eth0 tap0
    bridge_fd 3

in order to bridge eth0 (the real WAN interface) with tap0 (a new tunnel interface) at boot.
Then, on A, run the openvpn server with:
openvpn --dev tap0

On B, connect to it with:
openvpn --remote 126.96.36.199 --dev tap0 --route-gateway 22.214.171.124 \
        --redirect-gateway def1 --ifconfig 126.96.36.199 255.255.255.248

That's the super simple config I was looking for, and it works -- B is now publicly accessible at 188.8.131.52, and outgoing connections originate from the same address.
Add security (--secret, --tls-server, etc) as needed, of course.
Comment: Yeah.  Add another `pre-up openvpn` line to create `tap1` too, add `tap1` to `bridge_ports`, and run another instance of openvpn with `openvpn --dev tap1`.
Comment: Nice! I'm going to try that. Did you find a way to configure that: "without letting those clients use IPs to which they are not entitled"?
Comment: I didn't bother in my setup (which was temporary), but I imagine you could do it with ebtables.
Comment: Very useful. A question: what I need to change in A configuration if I need to route two IP from A: A => B  and  A => C (where C is another host)? Do I need to configure another bridge ?
Comment: @JimParis If you don't assign IP's at OpenVPN and do it later manually you can assign multiple IPs to the same `tap` device. I also managed to get this to work in a way you can bind only some programs to those IPs and have everything else use the local gateway: https://unix.stackexchange.com/questions/527677/debian-2-interfaces-2-gateways
Comment: How about if you wanted to make A's gateway local via B so any system on the LAN can use B and assign the remote default gateway and use public IPs?
Answer: You're going to have a hard time I think. Most firewalls will have difficulty routing OpenVPN traffic if both sides of the VPN are in the same subnet.
If you are trying to route for public access, I'd move both servers to different subnets from your public addresses and then use Virtual IPs (1 to 1 Nat) to connect them. To connect the two sites, OpenVPN would work or an IP-Sec tunnel.
Virtual IPs:
http://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses%3F
Site to site:
http://doc.pfsense.org/index.php/VPN_Capability_IPsec
Edit based on comments:
I'd personally install pfSense on the A box and give it the port you wanted for its WAN. Then setup an OpenVPN server on a local subnet (which is all ready to go in the pfSense web interface) and setup the other machine with a Virtual IP pointed to its local OpenVPN ip. This would give you room for expansion later (add more machines with Virtual IPs, logically forward specific ports to different servers, really have a full blown LAN/WAN/DMZ setup with OpenVPN for virtual access. Not to mention that you'd have a full blown router so it'd likely be more secure.
Comment: But when computer A wants to route to computer B it won't know if it should use the WAN (with your public IPs), the LAN (with its static IP) or the OpenVPN (also with your public IPs) because they are all same subnet. B to A should work though.
Comment: Also there is this, I've had it working but not with public IPs. I think virtual ip's will be much better either way. http://openvpn.net/index.php/open-source/documentation/miscellaneous/76-ethernet-bridging.html
Comment: I don't understand how intermediate firewalls are involved; they certainly won't be looking inside the OpenVPN packets between **A** and **B**.  For the OpenVPN config itself, I was expecting that something like `push "route 22.214.171.124 255.255.255.255 net_gateway"` would ensure that the VPN data is still pushed over the normal network.
Comment: To be clear, I want to create tunnel directly between **A** and **B**, not on separate firewalls at each end.
Comment: "To be clear, I want to create tunnel directly between A and B, not on separate firewalls at each end." You are going to need to open a port somewhere for an OpenVPN server
Comment: "it won't know if it should use the WAN, the LAN, or the OpenVPN" -- The OpenVPN server at **A** should be sending its packets back to their source address and port, which would be the NAT firewall at **B** (not in 22.214.171.124/29), so I don't think there's any confusion there.  Note there's no LAN at **A**, just the WAN.
Comment: "You are going to need to open a port somewhere for an OpenVPN server".  All ports are open.  **A** is directly on the Internet.
Comment: "Also there is this..." Yeah, I mentioned Ethernet bridging in my question.  It seems like this might be the only way to get things working without pulling my hair out, although I'd really prefer a point-to-point IP tunnel if possible.
Comment: I've dealt with an older version of OpenVPN.  And I agree with Robert, if your addresses are in the same subnet, you'll have trouble with anything other than bridging.  The subnet mode is best for setting up routes between two different subnets, i.e. 184.108.40.206/24 and 188.8.131.52/24; when it does so, the version I'm using dynamically adds/removes the routing rules needed, and points the route for each "remote" subnet to the VPN's internal address.


Statistical test for two samples and unreplicated blocked data

Question: I used the Friedman test with more than two samples and unreplicated blocked data.
However, in my situation, I have only two samples and unreplicated blocked data.
What statistical test you think would be suitable for such samples. The column is for regression method and rows are for different data sets. SAD =sum absolute deviations. I do not care about normality.
Comment: What is the hypothesis you want to test? If it is about agreement, maybe add the tag [tag:agreement-statistics].  And also, add the example data as text, so it is readable ...
Answer: It isn`t clear what your hypothesis is, you could clarify that by editing your post, it could possible be about agreement? But anyhow, I would start by plotting the data, and since this is paired data a Tukey mean-difference plot could be of use (also called Bland-Altman plot).
I use logs for the x scale of means:
Comment: I want to investigate and check if there is a significant difference between algorithm BR and IRLS (first and second column).
Comment: The plots shows an obvious outlier. Maybe start with investigating that! The outlier must be *PPPTS. What is that?


RecyclerView need to scroll along the activity

Question: I have a situation like this

 <RecylerView>

 <RecylerView>

Everything works fine - here is one thing I want to achieve.
When I scroll on the page, I need to scroll it as one list.
But now as there are two list I can scroll inside the recycler view.
Comment: Why don't you use one `RecyclerView` and make you adapter handle the views depending on whether it's a header or a data item?
you can try [this](https://stackoverflow.com/a/26245463/886001) to handle different view types
Comment: Its complex inside as I have drag and drop inside each Recycle view. And drag and drop is recycle view dependent.
Answer: If you want to use 2 RecyclerViews, you should use NestedScrollView as parent view, also add android:nestedScrollingEnabled="false" to both the RecyclerViews and they will scroll along the page as a single list:
<android.support.v4.widget.NestedScrollView>

  <RecylerView
   android:nestedScrollingEnabled="false"   <-- P.S this tag is very important or else each recyclerview will scroll within them selves.
   />

  <RecylerView
   android:nestedScrollingEnabled="false" />
</android.support.v4.widget.NestedScrollView>

Comment: Glad to know :)


How can I run a rails server virtually?

Question: I want to be able to work on my app but I won't have internet. So how can I run the app? Would regular rails s do the trick or is a network required for that?
Comment: How will you be able to work on the app without google and stackoverflow? You need internet at least for that :)
Comment: Have a look here (`rails server`): http://guides.rubyonrails.org/getting_started.html#starting-up-the-web-server
Answer: rails s runs the server locally.  Furthermore, I think you misunderstand the term virtual.
rails s will get the job done.  Just run that command, then go to http://localhost:4000  (or whatever port it runs on)
Comment: Well I assumed that it required a network because `rails s` runs on a local network (which is why I used the term virtual), if it doesn't that's great, thanks!
Comment: ah, yea. no it doesn't run on a local network, it simply opens a port on your local computer. that's it :)
Comment: Are you sure? Because when I run it at work I can access it from other computers on the network, is that because there's a plugin hat does that?

How can I run a rails server virtually?