What is the difference between Z-scores and p-values?

Question: In network motif algorithms, it seems quite common to return both a p-value and a Z-score for a statistic: "Input network contains X copies of subgraph G".  A subgraph is deemed a motif if it satisfies

p-value < A,
Z-score > B and
X > C,
for some user-defined (or community-defined) A, B and C.

This motivates the question:

Question: What are the differences between p-value and Z-score?

And the subquestion:

Question: Are there situations where the p-value and Z-score of the same statistic might suggest opposite hypotheses?  Are the first and second conditions listed above essentially the same?
Answer: I would say, based on your question, that there is no difference between the three tests.  This is in the sense that you can always choose A, B, and C such that the same decision is arrived at regardless of what criterion you are using.  Although you need to have the p-value be based on the same statistic (i.e. the Z-score)
To Use the Z-score, both the mean $\mu$ and variance $\sigma^2$ are assumed to be known, and the distribution is assumed normal (or asymptotically/approximately normal).  Suppose the p-value criterion is the usual 5%.  Then we have:
$$p=Pr(Z>z)<0.05\rightarrow Z>1.645\rightarrow \frac{X-\mu}{\sigma}>1.645\rightarrow X > \mu+1.645\sigma$$
So we have the triple $(0.05, 1.645, \mu+1.645\sigma)$ which all represent the same cut-offs.
Note that the same correspondence will apply to the t-test, although the numbers will be different.  The two tails test will also have a similar correspondence, but with different numbers.
Comment: Thanks for that!  (and thanks to the other answerers too).
Answer: A $Z$-score describes your deviation from the mean in units of standard deviation. It is not explicit as to whether you accept or reject your null hypothesis.
A $p$-value is the probability that under the null hypothesis we could observe a point that is as extreme as your statistic. This explicitly tells you whether you reject or accept your null hypothesis given a test size $\alpha$.
Consider an example where $X\sim \mathcal{N}(\mu,1)$ and the null hypothesis is $\mu=0$. Then you observe $x_1=5$. Your $Z$-score is 5 (which only tells you how far you deviate from your null hypothesis in terms of $\sigma$) and your $p$-value is 5.733e-7. For 95% confidence, you will have a test size $\alpha=0.05$ and since $p<\alpha$ then you reject the null hypothesis. But for any given statistic, there should be some equivalent $A$ and $B$ such that the tests are the same.
Comment: @Gary - a p-value doesn't tell you to reject or not any more than a Z-score.  They are just numbers.  It is only the decision rule which determine accepting or rejecting.  This decision rule could equally well be defined in terms of a Z-score (e.g the $2\sigma$ or $3\sigma$ rule)
Comment: @Gary - the p-value is no more rigorous just because it is a probability.  It is a monotonic 1-to-1 transformation of the Z-score.  any "rigor" that is possessed by the p-value is also possessed by the Z-score.  Although if you are using a two sided test then the equivalent is the absolute value of the Z-score.  And in order to compare $H_1:\mu\neq 0$ to the null, you have to take a "minimax" approach: which is to pick the sharp hypothesis which is most supported by the data and consistent with $H_1$.  Unless you can demonstrate how to calculate $P(X|\mu\neq 1)$
Comment: @probabilityislogic I agree with you. Indeed, you could construct some test based on $Z$ score threshold but it does not allow you to explicitly define a test size in the classical sense (i.e. in terms of probability). This sort of criteria might be trouble some if your distribution has thick tails. When you construct a test, you explicitly define a test size and thus the $p$-value immediately tells you whether you accept or reject, which is the point I was trying to make.
Comment: @gary - not really, for the p-value makes no reference to alternatives.  So it can't be used to directly compare alternatives.  For example, take $H_0:\mu=0$ vs $H_A:\mu=-1$.  The the p-value for $H_0$ stays the same $5\times 10^{-7}$.  So you say "reject the null" which means "accept the alternative" and declare $\mu=-1$.  But this is absurd, nobody would do this, but the p-value rule you use here does this.  Put another way, the p-value rule you described is not invariant with respect to what is called the "null hypothesis" (resolution coming)
Comment: (cont'd)  The resolution of the apparent absurdity is note that the p-value is not an "absolute" test, but a relative one, defined with an implicit alternative hypothesis.  In this case, the implicit alternative is $H_{imp}:\mu=5$.  You can see this by noting that if I calculate the p-value of $H_A$ I get $1\times 10^{-9}$, which is smaller than the p-value for $H_0$.  Now in this example, the "implicit alternative" is easy to find by intuition, but it is much harder to find in more complex problems, where nuisance parameters or no sufficient statistic.
Comment: @probabilityislogic what you seem to be describing is the power of the test. I've made no suggestions to "accepting the alternative" which is not the same as "rejecting the null." In fact, the implicit alternative is not what you described but rather the two sided test $H_1:\mu\neq0$. The only point I was making is that the $p$-value seems more rigorous and a choice of $Z$ score threshold, which may or may not come from a normal distribution, seems more arbitrary.
Answer: $p$-value indicates how unlikely the statistic is. $z$-score indicates how far away from the mean it is. There may be a difference between them, depending on the sample size.
For large samples, even small deviations from the mean become unlikely. I.e. the $p$-value may be very small even for a low $z$-score. Conversely, for small samples even large deviations are not unlikely. I.e. a large $z$-score will not necessarily mean a small $p$-value.
Comment: Not really. Suppose you sample from N(0, 1). Then your std will be about 1 regardless of sample size. What will get smaller is the standard error of the mean, not standard deviation. p-values are based on SEM, not on std.
Comment: if the sample size is large, then the standard deviation will be small, hence the Z-score will be high.  I think you may discover this if you tried a numerical example.
Comment: The Z-score is (observed-mean)/(standard deviation).  But the mean and standard deviation are of the observed statistic, not of the population from which components of it were drawn.  My slack terminology has been caught here.  However, if you are testing the mean, then the appropriate standard deviation in the Z-score is the standard error, which gets smaller at the same rate as the p-value.


Cox PH model and regression with log-log link function: is there a connection?

Question: In this post: interpreting estimates of cloglog logistic regression, I read that when interpreting the results of a regression with a log-log link function, the exponent of the estimated coefficient is equal to the hazard (= probability of mortality) per unit time. So to arrive at the probability of mortality per unit time, I take the exponent of the intercept and then the exponent of each coefficient*variable product, and add the results together?
On the other hand, to interpret a Cox proportional hazards model as a hazard per unit time, you multiply coefficient*variable for each coefficient, add everything up, exponentiate it, and multiply the result by the baseline baseline hazard function (if you can work out what the baseline hazard is).
These two models sound fairly different to me. However, I've been experimenting with them and I get suspiciously similar coefficients. Can anyone explain why?
Here is the code in R. Variables are continuous except for cop99, a factor with 7 levels; surv is a survival model built from survival time and status (1=event, 0=survived):
coxmodel <- cph(surv ~ cost_mean + elev_mean + popn_mean + cop99 + PAs_mean,
             data = grid3@data, x = TRUE, y = TRUE, surv = TRUE, time.inc=1)
glmmodel <- glmer(status ~ cost_mean + elev_mean + popn_mean + PAs_mean + (1|cop99),
    family=binomial(link="cloglog"), nAGQ=7, data=grid3@data)

Here is the output for the Cox PH model:
                         Model Tests    Discrimination
                                               Indexes
 Obs     36918    LR chi2    1525.11    R2       0.056
 Events   2222    d.f.            11    Dxy      0.462
 Center 1.6031    Pr(> chi2)  0.0000    g        1.268
                  Score chi2 1268.99    gr       3.552
                  Pr(> chi2)  0.0000

           Coef    S.E.   Wald Z Pr(>|Z|)
 cost_mean -0.5778 0.0272 -21.21 <0.0001
 elev_mean -0.5786 0.0345 -16.77 <0.0001
 popn_mean  0.0729 0.0083   8.83 <0.0001
 cop99=30   1.8168 0.3802   4.78 <0.0001
 cop99=40   2.3828 0.4017   5.93 <0.0001
 cop99=50   2.0343 0.3775   5.39 <0.0001
 cop99=110  0.5465 0.4989   1.10 0.2733
 cop99=160  0.0572 0.4474   0.13 0.8982
 cop99=170 -0.2884 0.4563  -0.63 0.5273
 cop99=190  2.0738 1.0607   1.96 0.0506
 PAs_mean  -1.5656 0.1502 -10.42 <0.0001

and here is the summary of the glmer model:
Random effects:
     Groups Name        Variance Std.Dev.
     cop99  (Intercept) 0.9941   0.997
    Number of obs: 36918, groups:  cop99, 8

    Fixed effects:
                Estimate Std. Error z value Pr(>|z|)
    (Intercept) -3.84387    0.37838 -10.159  < 2e-16 ***
    cost_mean   -0.56986    0.02713 -21.002  < 2e-16 ***
    elev_mean   -0.57268    0.03444 -16.630  < 2e-16 ***
    popn_mean    0.06933    0.00851   8.147 3.74e-16 ***
    PAs_mean    -1.57468    0.15089 -10.436  < 2e-16 ***

..surely the coefficients for the 4 continuous variables are too similar for this to be a coincidence?
If the baseline hazard in the Cox model is 1, it'll have no effect on the hazard. Equally, exp(-3.84387) is insignificantly small.
One final puzzle: this all looks to me like the continuous variables should have a big effect, not just a 'significant' one. But the r-squared is about 0.06 in both models. Doesn't that mean that, on the contrary, the model doesn't explain much?
P.P.S. bearing in mind @EdM's advice in the answer below, I tried a binomial approach with R-INLA as follows:
lattice_temp <- poly2nb(grid3, row.names = grid3@data$id2)
nb2INLA(paste(getwd(), "/lattice.graph", sep = ""), lattice_temp)
lattice.adj <- paste(getwd(), "/lattice.graph", sep = "")
# INLA requires a dataframe and spatial lattice as input
grid10 <- grid3@data
survinla <- inla.surv(grid3@data$def_mean, grid3@data$status)
coxinla <- inla(survinla ~ 1 + cost_mean + elev_mean + popn_mean + cop99 +
                  f(id2, model = "bym", graph = lattice.adj, scale.model = TRUE),
                data = grid10, family = "coxph",
                control.compute=list(dic=TRUE, mlik=TRUE, waic=TRUE),
                control.hazard = list(hyper = list(prec = list(param = c(0.001, 0.001)))))

...I ran this without the spatial term too. Adding the spatial term improves the marginal log-likelihood from -2167.46 to 890.66  and the WAIC score improves from 4376 to 3651. Maybe that's the way to go.
Having said that, some of the Cox PH model coefficients do have a low standard error so I wonder if it would still have some predictive value? Not that I know how to extrapolate to future years - which I should read up on (this is really a predictive and not an explanatory exercise) - but I do have yearly data for 2 of the covariates, so the prospect of a Cox model with time-dependent covariates is still a temptation...unless I can work them into a more standard model.
Comment: I suspect that the low standard errors in the Cox PH model come from the (probably unwarranted) assumption of independence among events. Your point estimates of survival might still be OK, but their standard errors will be too low. A potential problem with predictions involving time-varying covariates is [survivorship bias](https://en.wikipedia.org/wiki/Survivorship_bias). You'll have to judge the risk of that for yourself, based on your understanding of the subject matter.
Answer: There is, as you suspect, a fundamental connection between a Cox model and a binomial generalized linear model with a complementary log-log (cloglog) link.* That is described exactly for interval-censored data, as explained on this page with a link to the literature for further reading. The binomial regression in that circumstance with cloglog link is actually called a "grouped proportional hazards model." The intercepts in that case are related to the differences in cumulative baseline hazard between the endpoints of the time intervals.
The above is for a binomial model that explicitly includes (discrete) times in the model. With data organized so that censored cases are omitted beyond the last observation time, a discrete-time binomial model handles censoring like a continuous-time survival model: cases only contribute to calculations at times when they are in the risk set.
It's not clear that your binomial model handled times to events. Perhaps there was a fixed study duration with all participants starting at the same time, so that each individual either had an event during the study or was censored at a survival time equal to the full study duration. That would be OK: you then have interval censoring with a single interval equal to study duration, and the explanation above for the cloglog link would hold. In general, though, if you don't model time properly in a binomial model you risk having problems if there's censoring.
Finally, don't perseverate over the apparently small pseudo-$R^2$ value of the Cox model. As Frank Harrell explains:

The only thing going against pseudo $R^2$ is the difficulty in interpreting its absolute value. But it is excellent for comparing two or more models, even though examining increases in LR [likelihood ratio] is better.

With respect to your survival model, the $D_{xy}$ value shows how well you can distinguish between cases. You can calculate the concordance $C$ from that, $C= 0.5 + (D_{xy}/2)$ or about 0.73 in your Cox cph model. That means for any pair of cases whose survival times can be compared, the model puts them in the correct order 73% of the time. As a next step, use the calibrate() function for cph objects to get an estimate of how well predicted and actual survival probabilities align at any specified survival time.

*Technically you shouldn't call that a "logistic regression," as that term should be reserved for a binomial regression with a logit link.
Comment: Many thanks @EdM. I got into trouble with the function 'calibrate' and asked a StackOverflow question: [link](https://stackoverflow.com/questions/68141106/calibrating-a-cox-ph-model-with-r-packages-survival-and-rms-time-unit-confu) ...the subject of the model is a bit unusual. I have 2km grids, and the event 'status = 1' is where deforestation in 30m pixels (over 2000-20) reaches 10% of the maximum seen in my dataset. The 'time to event' is the mean year of deforestation. For all grid squares with status = 0, I set the time to event as 21 (but maybe that wasn't necessary?)
Comment: @MichaelSmith you might need to reconsider your approach. Your "events" are unlikely to be independent among neighboring grid squares (as your current Cox and binomial models implicitly assume). There are ways to take geospatial correlations into account in regression modeling, but I don't have any experience with them. Take care of that problem before you worry much more about improving the current, independence-assuming, models.
Comment: This is definitely a concern. I'm looking at models with a lattice-based spatial element using R-INLA (hence the interest in log-log models) to see if the AIC value is better with/without a spatial element. Some of the variables (cost-distance as a measure of accessibility, and a categorical variable of vegetation type) will have information on space but I'm not sure if that deals with the problem. I was hoping to do a cross-validation exercise ...perhaps that's easier without using a coxph approach? It's really a predictive (not explanatory) project.


Add value to field using Entity Metadata Wrapper

Question: I'm trying to accomplish the seemingly simple task to add a value to a multi-value file field in Drupal 7. I have been trying for hours, but none of the answers I found would solve my problem.
My field is called field_video_ext. Using the Remote Stream Wrapper module, I would like to add values like this (using some of the module's code):
<?php

// My videos and their respective nodes.
$videos = array(
  array(
    'nid' => '1903',
    'uri' => 'http://example.com/movie.flv',
  ),
  array(
    'nid' => '3172',
    'uri' => 'http://example.com/movie2.flv',
  ),
);

foreach ($videos as $video) {
  // From the Remote Stream Wrapper module.
  try {
    $file = remote_stream_wrapper_file_load_by_uri($video['uri']);
    if (!$file) {
      $file = remote_stream_wrapper_file_create_by_uri($video['uri']);
      file_save($file);
    }
  }
  catch (Exception $e) {
    drupal_set_message($e->getMessage());
  }

  if (empty($file->fid)) {
    drupal_set_message(t('Unable to add file from URL %file.', array('%file' => $video['uri'])));
    exit;
  }
  // Display needs to be set.
  $file->display = 1;

  // Get Entity Metadata Wrapper
  $wrapper = entity_metadata_wrapper('node', $video['nid']);

  // Write information into fields.
  $wrapper->field_video_ext->set(get_object_vars($file));

  // Save wrapper.
  $wrapper->save();

}

?>

Now the problem is that this doesn't work if the field accepts multiple values. It throws the error

EntityMetadataWrapperException: Invalid data value given for wrapper
  chain

I then tried using $wrapper->field_video_ext[] = get_object_vars($file);, but that doesn't work either. I think it was this error I got:

Fatal error: Cannot use object of type Project as array

I also tried the following instead of get_object_vars($file):
$file = (array) $file;
$items = array($file);

And then saving that array using set() or the [] way. But to no avail. Can anybody tell me what I am doing wrong?
Comment: What does your $file contain?
Comment: $file is a full file object returned by remote_stream_wrapper_file_create_by_uri(). I checked it against other files and it looks exactly the same.
Answer: One issue may be that get_object_vars() returns an array with keys that the file field does not understand.  The EntityMetadataWrapper documentation provides two options for setting file reference values, though it appears to use single value fields in the examples:
Setting the file directly:
$w_containing_node->field_attachment_content->file->set( $file_obj );

Providing an array with just fid:
$w_containing_node->field_attachment_content->set( array('fid' => $fid) )

One of the comments provides the following examples from the entity tests which apply to a multi-value field, though I'm not sure if setting the full file object will work appending a new item:
$wrapper->field_file[0] = array('fid' => $file->fid, 'display' => TRUE);
$wrapper->field_file[1]->file = $file;

Comment: Using an array with just fid I get the mentioned "Invalid data value given" error. Setting the file directly through file->set(), I get "Call to a member function set() on a non-object" (both with $file being an object or an array). But ´$wrapper->field_video_ext[0] = array('fid' => $file->fid, 'display' => TRUE);´ seems to work. I'll re-check that and get back.
Comment: Looks good, so I guess $wrapper->field_video_ext[0] = array('fid' => $file->fid, 'display' => TRUE); is the right answer. Thanks!


Deploying ova/ovf in Google Cloud

Question: Can we deploy an OVA/OVF file in Google Compute Engine? Example, one of IBM's virtual appliance
https://www.ibm.com/support/knowledgecenter/SS9H2Y_6.0.1/com.ibm.dp.xi.doc/virtual_installationimage.html
Comment: Does this answer your question? [VM Instance is not accessible The project you requested is unavailable](https://stackoverflow.com/questions/27285638/vm-instance-is-not-accessible-the-project-you-requested-is-unavailable)
Answer: I understand you want to exports VMs in OVA or OVF format into GCE. As per GCP documentation and this post compute engine support only RAW format. You may want to try with converting OVA file into RAW format using some free tools like VirtualBox or extracting files in RAW format.

Deploying ova/ovf in Google Cloud


How to make persistent exposed filters in views

Question: I'm aware of the "remember" option of views, but it is stored in session data (not persistent if you logout) and I want to keep the selections saved forever (so the users only have to do the selections once when they sign up).
I've done quite a lot of research regarding different options that maybe could do the trick, but haven't found out yet a viable solution to it. The options I've researched are:
1) With the module "views saved searches" (works but for usability reasons is not an option in this case)
2) Coding a way to remember the exposed filters selected via cookies or database (I haven't found anything about this and I don't know how to code a solution myself).
3) Making the selection in a node or profile page (with CCK, Nodereference/Entity Reference or Relation fields) and applying the selection of each user to "personalize" their view accordingly (maybe following one of the two options briefly mentioned in point 3 of this answer: https://stackoverflow.com/a/1409382 or using advanced features of relationships and/or arguments)
Any help would be appreciated.
Thanks!
Comment: You should make an answer for yourself, and then answer it so that others know you found an answer :)
Comment: I reply to myself: The easiest solution is the Views Global Filter module
Comment: is it possible to keep selection when changing language in a multilingual website?
Answer: Take a look at Views Global Filters module. That will solve your problem.
The project page reads

Use Views Global Filter on sites that feature multiple Views. You want
  the views to be driven by the same filter selection, for instance a
  country, a date range, a search term or even proximity to a post code.


What goodness-of-fit tests avoids the ambiguity in chi-square statistic?

Question: The chi-square statistic is a fairly standard goodness-of-fit test defined as:
$$\chi^2 = \sum_k^N \left ( \frac{d_k-m_k}{\Delta d_k}\right )^2$$
where $\mathbf{d}$ is a vector of data with error $\Delta \mathbf{d}$, and $\mathbf{m}$ is some modelled data. All vectors have length $N$.
This definition ensures that if, on average, the modelled data are within the error bars, then $\chi^2 < N$ and if the modelled data are outside the error bars then $\chi^2 > N$.
The problem with this test is that there is an inherent ambiguity to the test. See this example:

In the above figure, the data are shown as black dots with errors bars and there are two different models shown as red and blue dots. As you can see, both of these models will return $\chi^2 = N$.
What is another type of goodness-of-fit test which will show that both of these models fit the data equally well while simultaneously showing that the models themselves are different?
Any help or ideas are appreciated.
Comment: You would need a statistic that returned more than one value at once. COnsider two further models one which alternated being above and below and the other which alternated the other way. There are may ways to both differ from the data and from each other.
Answer: You need to decide what exactly are you measuring. $\chi^2$ test measures the squared errors. Obviously, after squaring it loses the information about the direction of an error. If the direction was important then you shouldn't be squaring and using this particular test. For instance, you could simply measure the mean error, and in both cases it would be different than zero. For one model it would negative, and positive for another. You could meausue whether the biases are significant, then make a conclusion about it.
However, you seem to pose the question in a general setting, using the biases as an example. I don't think it is possible to have a general test like you want.
Comment: If I don't square the residuals, what test would I use to test if "the biases are significant"?
Comment: look at ANOVA comparing the errors


How can I stop my slider from snapping back with isMoveToPointEnabled = true. (WPF C#)

Question: I am doing a small project for fun where I'm making a local music application (kinda like Spotify), and I would like for the slider I use for the music track (MusicSlider) to snap to where I click my mouse. I know you can do this with isMoveToPointEnabled = true, but since I have a really messy timer_tick (Update) function and whatnot, I haven't found a way of making it work properly.
I have tried for so long, and nothing my small brain comes up with works.
The problem is: Whenever I click on the slider like I should do, It immediately just snaps back.
The code is really messy because of the fact i'm not good at making clean code, and that the code has undergone a lot of brainstorming, and I haven't taken the time to fix it up yet.
Anyways, here is the code:
using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Music_Application.MVVM.View;
using System.Globalization;
using Music_Application.MVVM.ViewModel;
using Microsoft.Win32;
using System.Windows.Controls.Primitives;
using System.Windows.Media.Animation;
using System.Threading;
using System.Data;
using System.Data.SqlClient;
using System.Drawing;

namespace Music_Application
{
    public partial class MainWindow : Window
    {
        bool songProgressDrag;

        public bool fadeIn;
        public bool fadeOut;

        public double i;

        StreamWriter? sw;
        StreamReader? sr;

        System.Windows.Threading.DispatcherTimer timer = new System.Windows.Threading.DispatcherTimer();

        public MainWindow()
        {
            InitializeComponent();

            VolumeSlider.Value = Read(VolumeSlider.Value, "./Data/VolumeSave.txt");

            MusicPlayer.mediaPlayer.Open(new Uri("./Example Items/Music/NeverGonnaGiveYouUp.mp3", UriKind.RelativeOrAbsolute));

            MusicSlider.AddHandler(MouseLeftButtonDownEvent, new MouseButtonEventHandler(WhenMovingMusicSlider), true);
            MusicSlider.AddHandler(MouseLeftButtonUpEvent, new MouseButtonEventHandler(SyncMusicProgressToSlider), true);

            timer.Interval = TimeSpan.FromSeconds(0.01);
            timer.Tick += Update;
            timer.Start();

        }

        private void SongProgress_ValueChanged(object sender, RoutedPropertyChangedEventArgs<double>? e)
        {
            UpdateSongProgress();
        }

        private void ChooseSong(object sender, RoutedEventArgs e)
        {
            OpenFileDialog openFileDialog = new OpenFileDialog();
            openFileDialog.Filter = "MP3 files (*.mp3)|*.mp3|All files (*.*)|*.*";
            openFileDialog.Multiselect = false;
            if (openFileDialog.ShowDialog() == true)
            {
                MusicPlayer.mediaPlayer.Open(new Uri(openFileDialog.FileName));
                TagLib.File tagFile = TagLib.File.Create(openFileDialog.FileName);

                TagLib.IPicture coverArt = tagFile.Tag.Pictures[0];

                if(coverArt != null)
                {
                    MemoryStream ms = new MemoryStream(coverArt.Data.Data);
                    ms.Seek(0, SeekOrigin.Begin);

                    BitmapImage bitmap = new BitmapImage();
                    bitmap.BeginInit();
                    bitmap.StreamSource = ms;
                    bitmap.EndInit();

                    SongCover.ImageSource = bitmap;
                }

                SongName.Text = tagFile.Tag.Title;
                ArtistName.Text = tagFile.Tag.FirstAlbumArtist;

                MusicPlayer.mediaPlayer.Open(new Uri(openFileDialog.FileName, UriKind.RelativeOrAbsolute));

                if (MusicPlayer.isPlaying)
                {
                    MusicPlayer.mediaPlayer.Play();
                }

                MusicSlider.Value = 0;

                while(!MusicPlayer.mediaPlayer.NaturalDuration.HasTimeSpan)
                {

                }

            }
        }

        bool nisse;

        private void SyncMusicProgressToSlider(object sender, MouseButtonEventArgs e)
        {
            while(MusicPlayer.mediaPlayer.Position != TimeSpan.FromSeconds(MusicSlider.Value))
            {
                MusicPlayer.mediaPlayer.Position = TimeSpan.FromSeconds(MusicSlider.Value);
            }

            songProgressDrag = false;
        }

        private void WhenMovingMusicSlider(object sender, MouseButtonEventArgs e)
        {
            songProgressDrag = true;
        }

        void Update(object? sender, EventArgs e)
        {
            if (MusicPlayer.mediaPlayer.NaturalDuration.HasTimeSpan)
            {

                if (!songProgressDrag && MusicPlayer.isPlaying)
                {
                    MusicSlider.Value = MathF.Round((float)MusicPlayer.mediaPlayer.Position.TotalSeconds);
                }
                if (MusicPlayer.repeat && MusicSlider.Value == MusicSlider.Maximum && !songProgressDrag)
                {
                    MusicPlayer.mediaPlayer.Position -= MusicPlayer.mediaPlayer.Position;
                    MusicSlider.Value = 0;
                }
                if (!MusicPlayer.repeat && MusicSlider.Value == MusicSlider.Maximum && !songProgressDrag)
                {
                    MusicPlayer.mediaPlayer.Position -= MusicPlayer.mediaPlayer.Position;
                    MusicSlider.Value = 0;
                    PlayButton.IsChecked = false;
                }
            }

            PlayButton.IsChecked = MusicPlayer.isPlaying;

            if (MusicPlayer.isPlaying)
            {
                PauseIcon.Visibility = Visibility.Visible;
                PlayIcon.Visibility = Visibility.Hidden;
            }
            else
            {
                PauseIcon.Visibility = Visibility.Hidden;
                PlayIcon.Visibility = Visibility.Visible;
            }

            if (fadeIn)
            {
                if (MusicPlayer.mediaPlayer.Volume < VolumeSlider.Value / 100)
                {
                    i += VolumeSlider.Value / 100 / 12;
                    MusicPlayer.mediaPlayer.Volume = i;
                }
                else
                {
                    MusicPlayer.mediaPlayer.Volume = VolumeSlider.Value / 100;

                    fadeIn = false;
                }
            }

            if (fadeOut)
            {
                if (MusicPlayer.mediaPlayer.Volume > 0)
                {
                    i -= VolumeSlider.Value / 100 / 12;
                    MusicPlayer.mediaPlayer.Volume = i;
                }
                else
                {
                    MusicPlayer.mediaPlayer.Volume = 0;

                    MusicPlayer.Pause();

                    MusicSlider.Value = MathF.Round((float)MusicPlayer.mediaPlayer.Position.TotalSeconds);

                    fadeOut = false;
                }
            }

            UpdateSongProgress();
        }

        private void UpdateSongProgress()
        {
            MusicSlider.Maximum = Convert.ToDouble(MathF.Round((float)MusicPlayer.mediaPlayer.NaturalDuration.TimeSpan.TotalSeconds));

            TimeSpan length = TimeSpan.FromSeconds(MathF.Round((float)Convert.ToDouble(MusicPlayer.mediaPlayer.NaturalDuration.TimeSpan.TotalSeconds)));
            TimeSpan progress = TimeSpan.FromSeconds(MusicSlider.Value);

            string songLength = string.Format("{0:D2}:{1:D2}",
                    length.Minutes,
                    length.Seconds);
            string songProgress = string.Format("{0:D2}:{1:D2}",
                progress.Minutes,
                progress.Seconds);

            if (songLength.Length == 5 && songLength.StartsWith("0"))
            {
                SongValueTotal.Text = songLength.Remove(0, 1);

            }
            else if (songLength.Length == 5 && !songLength.StartsWith("0"))
            {
                SongValueTotal.Text = songLength;
            }
            else if (songLength.Length == 8 && songLength.StartsWith("0"))
            {
                SongValueTotal.Text = songLength.Remove(0, 1);
            }
            else if (songLength.Length == 8 && !songLength.StartsWith("0"))
            {
                SongValueTotal.Text = songLength;
            }

            if (songProgress.Length == 5 && songProgress.StartsWith("0"))
            {
                SongValueProgress.Text = songProgress.Remove(0, 1);
            }

            else if (songProgress.Length == 5 && !songProgress.StartsWith("0"))
            {
                SongValueProgress.Text = songProgress;
            }
            else if (songProgress.Length == 8 && songProgress.StartsWith("0"))
            {
                SongValueTotal.Text = songLength.Remove(0, 1);
            }
            else if (songProgress.Length == 8 && !songProgress.StartsWith("0"))
            {
                SongValueTotal.Text = songLength;
            }
        }

        private void VolumeChanged(object sender, RoutedPropertyChangedEventArgs<double> e)
        {
            MusicPlayer.mediaPlayer.Volume = VolumeSlider.Value / 100;
        }

        public void Pause(object sender, RoutedEventArgs e)
        {
            PauseIcon.Visibility = Visibility.Hidden;
            PlayIcon.Visibility = Visibility.Visible;

            MusicPlayer.isPlaying = false;

            fadeOut = true;
            fadeIn = false;
            i = VolumeSlider.Value / 100;
        }

        public void Play(object sender, RoutedEventArgs e)
        {
            PauseIcon.Visibility = Visibility.Visible;
            PlayIcon.Visibility = Visibility.Hidden;
            if (MusicPlayer.mediaPlayer.NaturalDuration.HasTimeSpan)
            {
                MusicPlayer.Play();
            }

            MusicPlayer.isPlaying = true;

            fadeIn = true;
            fadeOut = false;
            i = 0;
        }

        public void RepeatOn(object sender, RoutedEventArgs e)
        {
            RepeatButtonDot.Visibility = Visibility.Visible;

            MusicPlayer.repeat = true;
        }

        public void RepeatOff(object sender, RoutedEventArgs e)
        {
            RepeatButtonDot.Visibility = Visibility.Hidden;

            MusicPlayer.repeat = false;
        }

        protected override void OnClosed(EventArgs e)
        {
            Write(VolumeSlider.Value, "./Data/VolumeSave.txt");

            base.OnClosed(e);

            Application.Current.Shutdown();
        }

        private void Write(object sender, string url)
        {
            sw = new StreamWriter(url);

            sw.WriteLine(sender.ToString());

            sw.Close();
        }

        private double Read(object sender, string url)
        {
            sr = new StreamReader(url);

            sender = Convert.ToDouble(sr.ReadLine());

            sr.Close();

            return (double)sender;
        }

        private void FillHomeButton(object sender, RoutedEventArgs e)
        {
            HomeButtonSelected.Visibility = Visibility.Visible;
            HomeButtonUnselected.Visibility = Visibility.Hidden;

            HomeButtonText.Foreground = System.Windows.Media.Brushes.White;
        }

        private void UnfillHomeButton(object sender, RoutedEventArgs e)
        {
            HomeButtonUnselected.Visibility = Visibility.Visible;
            HomeButtonSelected.Visibility = Visibility.Hidden;

            HomeButtonText.Foreground = new SolidColorBrush(System.Windows.Media.Color.FromRgb(70,70,70)); /*(Brush) new BrushConverter().ConvertFrom("#b3b3b3");*/
        }

        private void ExitPlaylistCustomizer(object sender, RoutedEventArgs e)
        {
            CustomizePlaylistPopup.Visibility = Visibility.Hidden;
        }
    }

    public static partial class MusicPlayer
    {
        public static bool isPlaying;

        public static bool repeat;

        public static MediaPlayer mediaPlayer = new MediaPlayer();

        public static void Play()
        {
            var mainWindow = (MainWindow)Application.Current.MainWindow;

            if (mediaPlayer.NaturalDuration.HasTimeSpan)
            {
                mediaPlayer.Play();
            }

            isPlaying = true;

        }

        public static void Pause()
        {
            mediaPlayer.Pause();
            isPlaying = false;
        }
    }
}

And also, here is my slider in xaml in case you need it:
<Slider IsManipulationEnabled="True" TickFrequency="1" x:Name="MusicSlider" ValueChanged="SongProgress_ValueChanged" IsSnapToTickEnabled="True" IsMoveToPointEnabled="True" Style="{StaticResource Horizontal_Slider}"
                    VerticalAlignment="Center" Margin="50,0,50,0"/>

Answer: The movement of slider thumb by clicking slider track seems not fire MouseLeftButtonDownEvent. As a result, you timer logic could revert slider's value before MouseLeftButtonUpEvent is fired.
I have no idea on direct solution for this issue. Perhaps it would be better to sync slider value and media player by ValueChanged event.


What purpose do chords serve in music?

Question: I'm a complete beginner to music theory and have recently been learning a few basics.
One thing I can't wrap my mind around are chords, and to a certain extent, chord progressions. I know how they are defined and how you play them - however I can't tell how one applies them to a piece of music.
In a similar sense that one can consider the bassline/melody/drums "separate pieces" of a piece of music, can you think of chords and chord progressions as their own "building block" of what makes a song? Or are chords and chord progressions simply "part" of these mentioned "pieces", with some melodies having an occasional chord playing within them? Basically viewing chords as "special tones" you can play when you feel like it - another part of your tonal palette.
For example, I've been listening to one of my favorite tracks as of late and it sounds like the bassline consists of nothing but a very low sounding chord progression. Yet in another track, it seems as if the chord progression was just another "building piece" of the song, accompanying the melody in its own, more separate fashion.
Comment: The exact same question occurs on Quora. This happens rather often, especially with somewhat general questions.
Answer: Basically what you're asking is "what is tonality." And the first disclaimer is that hasn't always been "a thing" in Western music, and isn't always in other music-cultures. Just as a "bassline/melody/drums" construct doesn't make sense for Gregorian chant,* neither do "chords" or "chord progressions."
First of all, I used the word "tonality." Let me unpack that; it includes the idea of "being in a key," and the idea that that key uses certain chords, and uses them in certain ways. It's more than a tool in your toolkit; it is the toolkit, the workshop in which you use it, and the way we understand the concept of "tool." It's the way we've "done business" in music from somewhere around the late Renaissance all the way through Justin Bieber. You can point to individual exception songs, or to entire branches that revolt against the idea of being "in a key" (atonalism), but it's so ingrained that the only way to exit the construct is by explicit choice.
Analyzing the music by contrasting "bassline, melody, and drums" is a useful approach, but it breaks the music into different roles. By analogy, language can be broken into nouns, verbs, and adjectives... but the thing that matches "tonality" in that example is "grammar." It's the assumptions and conventions about how you use your musical tools, and the way we're used to making sense of what we hear.
Now, you used the word "chords." That could mean "stacks of 3 or more notes that play simultaneously," or it could refer to the imaginary or implied chords that are suggested even if not actually heard. If I sing "Happy Birthday to you," with only myself singing, then you're not hearing chords as in stacks of notes; that would take at least three voices. But the melody suggests (and we're all used to) a "V chord" on the first "you," a "I" on the next "you," and finally a IV, V, and I at the end of the song, just like millions of symphonies, sonatas, doowop songs, rock ballads, and techno drops. If you're listening to your favorite track and hear the bass changing notes, chances are those notes are implying full chords and their tonal functions, just as the 7 bass notes here....

... contain the seeds for the chords that arrive by 1:18.
At the same time, I could imagine a work in which bass and melody weren't related by a tonal relationship, but developed their own patterns independently. (It might sound rather edgy, though, like this 1673 depiction of a lot of people singing different tunes at once:

* (Insert red herring about unspecified instrumental forces in medieval genres, ultimately getting into Hildegard von Bingen and devolving into critiquing various recordings.)
Comment: Should've been "… from Heinrich Biber all the way through to Justin Bieber."
Comment: That's an interesting passage in the Biber piece. Here's a similar example, harpsichord using tone clusters to imitate canons https://youtu.be/41AuDJBhQO4?t=160
Comment: @M.Vinay You know, I spent a little time trying to say "from Palestrina to..." but couldn't come up with a pop star with a "Pal__" name. From di Lasso to Dua Lipa? Really, I'd prefer to find a late-renaissance choral composer to pair with BTS.
Comment: Wow, I didn't know Biber at all. That's some seriously far ahead-of-his-time music there.
Comment: @leftaroundabout Yeah, Biber's hardcore. He has one piece where he wants you to take the two middle strings of the violin and switch their position on the bridge so they make an "X" below the bridge (more for symbolic purposes than sonic). But I did my dissertation on the "Capriccio Stravagante" by Carlo Farina, 50 years earlier, with this cat-and-dog fight (as well as like 5 "extended techniques" like col legno and ponticello): https://youtu.be/h8npmLIWRXo?t=702
Comment: @AndyBonner Paloma Faith
Answer: To understand chords, I find some history helpful.
Pre-chord history
The earliest Western music we know about is chant. ("Earliest" in this case meaning the first music where we can "connect the dots" to modern music.) In its simplest form, chant is one person singing one pitch at a time: melody, but only melody.
Eventually, musicians discovered that one could create a very compelling musical effect by accompanying a melody with a drone, a single sustained pitch. In the loosest sense, this is the "invention" of chords: more than one simultaneous sound, or put another way, more than one simultaneous voice.
From there, an evolution occurred in which a second drone was added, and then  a second voice which moved in parallel with the melody.
This general practice — of having one or more drone or parallel voices — is called organum (Wikipedia: Organum).
The history described here is illustrated in the video "Early Organum" by David Goodall.
The "invention" of chords
Eventually, musicians began to make these various voices truly independent of each other. That is, rather than singing a single pitch, or a melody that moved in lock step with another, it was more like two people each singing a different song, but where those two songs blended together in a pleasing way.
One of the masters of this was the composer Palestrina (Wikipedia:
Palestrina), whose music was considered the model for the next century or so. One of his most important works was his Pope Marcellus Mass, which can be heard here, performed by the Tallis Scholars.
Composers and theorists began to study these independent voices to understand how certain simultaneous notes moved most effectively to another set of simultaneous notes. This, essentially, is the "invention" of chords: a theoretical construct to describe how one set of pitches moved to another set of pitches (a "chord progression"). The composer/theorist most credited with our modern way of thinking about chords, is Jean-Philippe Rameau (Wikipedia: Rameau.
Chords as musical tools
As composers shifted from considering independent melodies to thinking in terms of the movement of chords, chords took on a life of their own. They are both a set of notes played simultaneously, but also a descriptive tool for describing the sound of a piece of music.
Especially in jazz and popular music, songs are often written as a melody with a set of chord changes describing the overall sound desired. It's then up to the various musicians to "realize" those sounds, those chords.
In Classical music, chords are more "built in", in that the composer indicates more precisely what should be played and when.
Comment: One thing Leonard Bernstein said about chords/harmony is that it's like adjectives and adverbs that modify the nouns and verbs of the melody. The same melody can sound different with a different harmony. Just something to maybe add to this otherwise comprehensive answer.
Comment: Or, putting different spices and herbs into what's being cooked?
Answer: In the broadest sense chords and other musical elements serve one main purpose: to express thoughts and feelings. Talking about "purpose" in art is a tricky business. If you try to stay objective about it, you can talk about stability/instability and consonance/dissonance as the dynamic forces in chords and harmony. Those elements give music a basic emotional sense of tension and release and sense of forward direction to start, continue, or stop.
When you mention that parts like melody and bass can seem like independent elements or work together, and ask whether chords can be treated like independent and singular elements, you get into the technical concept of musical texture.
When parts seem to be independent but work together that is usually called polyphony. Depending on the style of composition such independent parts can be constructed so that the combined parts make chords. That's a common way of describing harmony: multiple voices moving independently, the combination of parts at different points in time form chords, the flow of chords is the art of harmony. When each of those parts is a finely developed melody - like in a fugue by Bach - specific composition methods are used with the techniques of counterpoint.
But, chords can be treated in a more "generic" way in regard to the combination of parts. ("Block chords" is one way to describe what I mean by generic.) In that generic treatment chords do indeed become like singular elements as you suggested in your question. When chords are treated that way and combined with a single melody it is called homophony. A huge amount of pop music is homophonic: melody and chords with bass.
As you might expect there is a spectrum of styles between polyphony and homophony. It is not black and white, one or the other. In the long history of "classical" music polyphonic music tends to have a continuous rhythmic flow while homophonic music allows for more boldly demarked rhythms. That is a huge generalization, but it gives a general sense of why a composer might choose a polyphonic form over a homophonic form.

What purpose do chords serve in music?


Estimate accuracy of an estimation on Poisson binomial distribution

Question: I manage a website that charges its customers using payment cards. Some transactions area approved, others are declined. I compute the approval rate of transactions for a interval (a calendar day) as the number of approved transactions (A) divided by the total number of transactions (T) processed during the specified time interval.
Example:
          | Approved | Declined | Total |       Approval Rate (R)
          |    (A)   |    (D)   |  (T)  |   (A/T) || A/(T+3) | (A+3)/(T+3)
----------+----------+----------+-------+---------++---------+-------------
yesterday |    50    |    29    |   79  |  0.6329 ||  0.6097 |   0.6463
today     |     7    |     4    |   11  |  0.6363 ||  0.5000 |   0.7142

The approval rates (R=A/T, the 4th column) are very close but it's obvious that the second one is not very reliable. Three more transactions make the today's rate change a lot (0.5000 .. 0.7142, the last two columns) but they have much smaller impact on yesterday's rate (because yesterday's rate is computed using a larger set of transactions).
What I need: based on the above facts I need a way (algorithm, formula) to compute or estimate the accuracy of the approval rate computed as above. A confidence level or something like this. Or a range where the approval rate is expected to be in the next hours of the day (with a certain probability, of course). Or a different way to estimate the approval rate and get a result that makes it clear that the 0.6329 of yesterday is more reliable than the 0.6363 of today.
Or something else, I don't know...
I have a very limited knowledge and understanding of statistics; I tried to express the facts as well as I could using plain English. After reading several articles on Wikipedia I found out that my problem is a Poisson binomial distribution (I hope I identified it correctly). I think the approval rate I compute is an estimation of the probability of the distribution but I don't have any idea what parameter models the information I need and how to compute it.
Answer: If your sole need is to express your relative uncertainty in the probability of a declined transaction each day, it would seem you could fit a simple beta-binomial model. In this model, each of $n$ total transactions has the same probability $p$ of success, and there are $k$ total successes.
If you took the cookie-cutter Bayesian approach and assumed this value was equally likely to be any value between $0$ and $1$ at the start of each day, then the uncertain estimate you're looking for could be expressed as a $Beta(n + 1, k + 1)$ distribution. You could then use summary statistics, say the 95% highest-density interval, to express this uncertainty.
For your data, these distributions would produce plots like this (Python code to produce them below):

from matplotlib import pylab as plt
from scipy.stats import beta
x = np.linspace(0,1,100)
y1 = beta.pdf(x, 80, 51)
y2 = beta.pdf(x, 12, 8)
plt.plot(x, y1)
plt.plot(x, y2)

Comment: Thank you for your answer. While it puts me on the right track from the statistical point of view, it still leaves me in the dark on the practical side (coding). This is because I don't know how to translate the concepts into formulas (and I don't even know what these concept mean, in the first place). Could you please add some references where I could find how to compute the values? *(I use PHP which is not one of the favorite languages of the statisticians; therefore there are not many libraries for statistic computations written in PHP. I have to implement the formulas myself.)*
Comment: I have found this [PHP library](https://github.com/mcordingley/PHPStats) for statistics. Is it possible to compute the uncertainty using the methods it provides in [class Beta](https://github.com/mcordingley/PHPStats/blob/master/lib/ProbabilityDistribution/Beta.php)? How?
Comment: @axiac Ah, now that's another challenge entirely :) From the looks of the source code you linked, `getPdf()` will return the density, which is what I graphed above. If you'd like a quick way to get credible ranges, I would run something like `getCdf(0.0275)` and `getCdf(0.0975)`. (This is *not* a highest-density interval, since the beta distribution has some skew to it, but should be a suitable approximation in many applications.)
Comment: Why `0.0275` and `0.0975`? Are these values fixed or they depend on the input data? If they are fixed, shouldn't it be `0.9725` (i.e. `1-0.0275`)? And what values to use as object's constructor arguments (`$alpha`, `$beta`)?
Comment: Hmm... Or maybe I have to compute `ppf(0.0275)` and `ppf(0.9725)`? It says on `ppf()`: [*"returns the percent-point function, the inverse of the cdf"*](https://github.com/mcordingley/PHPStats/blob/master/lib/ProbabilityDistribution/Beta.php#L97).
Comment: @axiac You've caught two errors on my part. First, yes, use `ppf`. Choose the two inputs based on what you want percentage of the posterior distribution you want captured within the two values. For example, it's common to use a 95% interval, in which case you'll want $\frac{1 - .95}{2} = 0.025$ in each tail of the distribution. Thus, you'll want to use `ppf(0.025)` and `ppf(0.975)`. For the parameters, `$alpha` should be `approved + 1`; for `$beta` , `declined + 1`.
Answer: 1. If you can assume that, on the same day, all cards have equal probability of approval
You try to estimate an unknown appoval rate for each day, let us call this unknown rate $p$.  On one day you observe a number of events $N$ (79 yesterday and 11 today). If you assume that all events are independent and that at each individual event the unknown rate (a probability) is the same then you have a Binomial random variable.  For a Binomial random variable the expected value is $Np$ and the variance is $Np(1-p)$. So $X \sim Bin(N;p)$ X is the number of approvals you observe in a day and $\mu_X=Np,\sigma_X^2=Np(1-p)$.
The $p$ is unknown but can be estimates as the number of approvals divided by the total number of events in a day, so $\hat{p}=\frac{A}{T}$, which is what you compute.
So what you compute is $\frac{X}{N}$ where $\mu_X=Np,\sigma_X^2=Np(1-p)$. But if $\mu_x=N.p$ then the mean of $\frac{X}{N}$ is the mean of X divided by N, so $\mu_{(\frac{X}{N})}=\frac{\mu_X}{N}=\frac{Np}{N}=p$ therefore you have an unbiased estimator of the unknown $p$.
The variance of $\frac{X}{N}$ is the variance of $X$ divided by $N^2$, so it is $\frac{Np(1-p)}{N^2}=\frac{p(1-p)}{N}$.  The square root of the variance is the standard deviation. This standard deviation of your estimator is a measure of imprecision: the larger it is, the more imprecise your estimate.
So in your case, for day 1, $\hat{p}=\frac{50}{79}=0.63$ and the standard deviation on it is $\sqrt{\frac{p(1-p)}{N}}$ which can be approximated by replacinig $p$ with $\hat{p}$. So the standard deviation is $\sqrt{\frac{0.63\times0.37}{79}}=0.054$, the larger this standard deviation, the more imprecise your estimate is.
On the second day you find $\hat{p}=0.64$ with a 'imprecision' of $\sqrt{\frac{0.64\times0.36}{11}}=0.14$, so a larger standard deviation and thus less precise.
2. If you can assume that, on the same day, all cards have equal probability of approval, AND this probability is the same every day
If you can reasonably assume that on all days this rate will be the same, then you can increase the precision of your estimate by ''pooling''; indeed, on day 1 you have a binomial with success probability p and size 79, on day two these are respectively p and 11, and the sum of two independent binomials is again binomial, so in that case you have Bin(p;90) and you observe 57 approvals, so $\hat{p}=\frac{57}{90}=0.63$ and the 'imprecision' is $\sqrt{ \frac{0.63 \times 0.37}{90}}=0.050$
3. If you can NOT assume that, on the same day, all cards have equal probability of approval
If you can not assume that, within each day, all events have the same $p$, then you have sums of Bernouilli variables with different succes probability and your random variable is no longer Binomial but so-called Poisson-Binomial.  The idea is similar, the estimate for the rate is the same, but your variance of Poisson-Binomial has a correction term ( -N times the variance of the p's, this variance of the p's must be estimated from your data, as you subtract this amount, the variance will be smaller: $Np(1-p) - N \times Var(p)$ in stead of $Np(1-p)$ for the Binomial).
So the only difference is the formula for the variance, the rest is similar, but I don't see how to estimate Var(p) with your data.
4. Responses to the comment(s)
@axiac: If you say that "no day is like any other day" then it can still be that the probabilities $p$ above are different between the days BUT within one day an individual transaction has the same probability of approval $p_{day}$.  If that is a reasonable assumption then you are in case 1.
If the probability of approval is realy different from transaction to transaction, even for transactions on the same day, then the only difference between case 3 and case 1 is the formula for the variance:  your estimate $\hat{p}$ is then an estimator of the average of all (different) $p_{transaction}$ and the variance is computed as $\frac{\hat{p}(1-\hat{p}) - Var(p_{transaction})}{N}$
So if the variance of the probability of approval over all transactions is not too large, then you can 'assume away' the second term and you are back in case 1. If You think that there is considerable $Var(p_{transaction})$ (i.e. large compared to $\hat{p}(1-\hat{p})$) then you have to find an estimate of the variance of the p's. The only one I can think of is that you compute the proportions on several days, and then use the variance of these proportions as your estimate of $Var(p_{transction})$.
Comment: Thank you for your detailed answer. Obviously, the 3rd case applies to me; the events are completely independent, their probabilities are different and not related in any measurable way, no day is like any other day.
Comment: @axiac: you're welcome :-). Because of the limit on the number of characters in a comment, I added a section 4 in my answer. This sections contains a reaction to you're comment, please take a look at it if you want.
Comment: Using the values from previous days is a good idea. In fact, 24/7 it is the normal way of running an online business, our human behaviour is the one that imposes splitting the continuous flux of data into days.


Deadlink (404) editing post

Question: If question or answered post have included dead(404) link and we update that  with current working link for example in magento github repo in version 2.0 there is develop branch currently not so. that type of link should be editable..
Can user edit that post or not.
If editable then why community members are rejecting that edit ?
Here is the screenshot i have edited post with working link
Answer: Can user edit that post or not.
yes, you can edit those dead(404) links .

If editable then why community members are rejecting that edit?

If any community member rejects then they did the mistake.
Please share the edit links where your edit is got the reject,We moderator check those Post.

Also, request to all community member, Please do a proper  check
  before Reject any edit.
As a community member, we should have do a proper check on all edits and
  reviews before taking any action.

Himanshu, we encourage your contributions over those 404 link.!:)
Comment: Thanks a lot for your answer...
Comment: Here is the link https://magento.stackexchange.com/review/suggested-edits/144563
Comment: this is also https://magento.stackexchange.com/review/suggested-edits/144519

Deadlink (404) editing post


node API hook that can be used to alter fields before loaded by Views?

Question: I need to use a node API hook to alter some node content before it's displayed, in Drupal 7.
It just occurred to me though that if I used hook_node_view() it will not effect the output of a View that display's those nodes fields (right?).
I need to alter the fields in both the display of the node and the node's display in any views. Hopefull this can be done in one function, and not have duplicate code in both hook_node_view() and hook_views_pre_render() or function hook_views_data()whichever hook is used.
Answer: There are multiple ways in which this could be achieved:

Override the field itself in your theme with template_preprocess_field().
If the customization is more towards the markup, you can directly achieve that by overriding the field template.
Or you could just write a field formatter for this field, and select it in the field settings of the views and the view-mode (manage display) of your content type.
Comment: @DanMantyla Yep! In the preprocess field you should be able to find the node with something like `$variables['element']['#object']` and then use any of its properties (type, status, published, etc) or even field values.
Comment: Thanks! I didn't consider #1 because I don't have any experience doing that. I already have several custom module hooks so I thought I would just make another. I'll look into `template_preprocess_field` though, that might be just what I need. #2 won't work for me because, what I'm doing is I'm not displaying a field at all if certain conditionals are met. I did consider #3 and that would be very clean but maybe a little more effort. And if a field formatter is already selected for the field then can I have two field formatters for one field??
Comment: BTW, I need to alter the field for only certain node types and sometimes only when certain tags are chosen for that node type. Is it still possible to use template_preprocess_field for this?


Group Homomorphism: Is it required to have the inverse preservation to prove Identity Mappings?

Question: Consider two Groups $\left(G,\cdot\right)$ and $\left(H,\times\right)$
And let $f$ be a function $f:G\rightarrow H$
Then $f$ is a Group Homomorphism if $\forall g_{1}, g_{2} \in G$ we have
$f\left(g_{1} \cdot g_{2}\right) = f\left(g_{1}\right) \times f\left(g_{2}\right)$
and that the inverse of $G$ is preserved, i.e.
$f\left(g_{1}^{-1}\right) = f\left(g_{1}\right)^{-1}$
In the textbook I'm reading this is essential so that the identity for $G$ here $1_{G}$ maps onto the identity of $H$ here $1_{H}$ .
This is illustrated simply with
$f\left(g_{1} \cdot g_{1}^{-1}\right) = f\left(1_{G}\right) =  f\left(g_{1}\right) \times f\left(g_{1}^{-1}\right) = f\left(g_{1}\right) \times f\left(g_{1}\right)^{-1} = 1_{H}$
And so,
$f\left(1_{G}\right) = 1_{H}$
I however went about it differently (I'm sure I'm missing something) though and I was wondering if people could have a look over and see if what I've done is valid and if not where I went wrong.
So my method was
$f\left(1_{G}\cdot 1_{G}\right) = f\left(1_{G}\right) = f\left(1_{G}\right) \times f\left(1_{G}\right)$
And so
$f\left(1_{G}\right)^{-1} \times f\left(1_{G}\right) = f\left(1_{G}\right)^{-1} \times f\left(1_{G}\right) \times f\left(1_{G}\right)$
$1_{H} = 1_{H} \times f\left(1_{G}\right) = f\left(1_{G}\right)$
And so
$1_{H} = f\left(1_{G}\right)$
Is this valid? as in can we show that the identity of $G$ maps to the identity of $H$ without having to impose the inverse requirement as before?
Thanks
Comment: The basic property $f(ab) = f(a)f(b)$ also implies $f(a^{-1}) = f(a)^{-1}$. In theory development, it can be cleaner to adopt a minimal definition, and use lemmas to get useful additional properties. In textbooks, though, there is often redundancy, for clarity's sake.
Comment: As an aside, the notion of "group" is more naturally described as having three structure operations: a constant (the identity), a unary operation (inverses), and a binary operation (the product). $(G, \cdot)$ is better described as a *semigroup*, and that you are considering a *semigroup* with the extra property of having identity and inverses. A $\cdot$-preserving map is a *semigroup* homomorphism, but a group homomorphism should preserve identities and inverses too. It is a strange feature that these two different structures turn out to be basically the same.
Comment: For an example of how things go awry, consider the relation between monoids and semigroups; monoids are not merely semigroups with identity; e.g. not all semigroup homomorphisms between semigroups with identity are actually monoid homomorphisms.
Comment: @Hurkyl - I didn't realise that a unary operator to defined inverses was required. In all definitions I've read so far (and it's limited I'm very new to this) it merely states that inverse elements exists as opposed to required a unary function to map each element to it's inverse.

Can you then have say two groups that are identical in their definitions outside of the inverse unary function? i.e. can there be more than one inverse function?
Answer: Your method looks correct. It's a fact that a group homomorphism preserves the identity element.
Comment: Indeed, it was just that all the proofs of identity mapping that I observed used the preservation of inverses to illustrate it.
I suppose doing it with the preservation of identity mapping does kill two birds with one stone, but still nice it can be shown without it.

Also gives it more 'strength' so to speak :-)
Answer: Yes. You are right and you proved correctly that if $f\colon G\longrightarrow H$ is such that $(\forall g_1,g_2\in G):f(g_1g_2)=f(g_1)f(g_2)$, then $f(1_G)=1_H$.
Comment: Thank you! I was worried as most of the illustrations I'd seen online required the preservation of the inverse in order for the identity mapping to be shown. I'm very happy I was able to do it a different way!
Thanks for looking over.


Can't get max result of spot price history - US-EAST region

Question: When I retrieve the history price of spot for "us-east-f1" or any region in "us-east-1", the result always less than 200 price, I need for single region and single instance type.
How can I retrieve a huge number of results?
EX:
ec2 = boto3.client('ec2')
t=datetime.datetime.now() - datetime.timedelta(0)
f=datetime.datetime.now() - datetime.timedelta(90)
response= ec2.describe_spot_price_history(InstanceTypes =['c3.4xlarge'],ProductDescriptions = ['Linux/UNIX'], AvailabilityZone = 'us-east-1a', StartTime= f, EndTime = t, MaxResults=1000)
response =response['SpotPriceHistory']

I mean for a single region and instance type, I need the max result to be larger than this.
Edit:
I use paginator to get all result for all available pages:
paginator = ec2.get_paginator('describe_spot_price_history')
page_iterator = paginator.paginate(StartTime= t, EndTime = f, MaxResults=2000 )
for page in page_iterator:
   output = page['SpotPriceHistory']

However, I still get the same number of results! When I fetch the results of 90 days, I still got have the same amount of results?
How to get all results or to get the max amount of price values?
Comment: How do you know that more results exist?
Comment: You mean it return what history is there
Comment: How to access another region while I have permission only for us-east region. to get more result
Comment: It can happens that there is little price spike for particular AZ.
Comment: @mootmoot, How to access another region?
Comment: http://boto3.readthedocs.io/en/latest/guide/ec2-example-regions-avail-zones.html
Comment: all of them us-east-1 and for a single region and type I can't retrieve large amount of result :(
Comment: @Hana90 describe_spot_price_history is paginated.  You *might* need to capture the `NextToken` and repeat the request.  I don't believe this call is expected to always return `MaxResults` items per response, even if `MaxResults` items are available -- subsequent requests may be required in order to get more data.
Comment: @Hana90  I just try out, it seems boto3 `describe_availability_zone` either contains bugs or untold features, I cannot list the AZ outside my default region.  however,  the aws cli looks fine, e.g. `aws ec2 describe-availability-zones --region us-west-1` .  Please post your describe-availability-zones issue here  : https://github.com/boto/boto3/issues  .
Answer: Your code appears to be working just fine, but the start and end timestamps were backwards.
I ran this code:
import boto3
import datetime

start_date = datetime.datetime.now() - datetime.timedelta(90)
end_date = datetime.datetime.now() - datetime.timedelta(0)

ec2 = boto3.client('ec2', region_name='us-east-1')
paginator = ec2.get_paginator('describe_spot_price_history')
page_iterator = paginator.paginate(InstanceTypes =['c3.4xlarge'],ProductDescriptions = ['Linux/UNIX'], AvailabilityZone = 'us-east-1a', StartTime= start_date, EndTime = end_date, MaxResults=2000 )

for page in page_iterator:
   print page['SpotPriceHistory']

I got back one page that had 122 entries.
The first entry was for 2018-04-01:
{u'Timestamp': datetime.datetime(2018,
    4,
    1,
    4,
    40,
    44, tzinfo=tzutc()), u'AvailabilityZone': 'us-east-1a', u'InstanceType': 'c3.4xlarge', u'ProductDescription': 'Linux/UNIX', u'SpotPrice': '0.840000'
},

The last entry was for 2018-01-02:
{u'Timestamp': datetime.datetime(2018,
    1,
    2,
    0,
    28,
    35, tzinfo=tzutc()), u'AvailabilityZone': 'us-east-1a', u'InstanceType': 'c3.4xlarge', u'ProductDescription': 'Linux/UNIX', u'SpotPrice': '0.840000'
}

This cover the maximum 90 days of data that is available.
Comment: Correct. The [DescribeSpotPriceHistory documentation](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSpotPriceHistory.html) says that the **StartTime** can be **up to the past 90 days**.
Comment: Yes, I got the similar number, can I get more than this? Is it impossible to access longer than 90 days :( ?
Answer: To make an API call to another region, specify the region_name:
ec2 = boto3.client('ec2', region_name='ap-southeast-2')

When MaxResults is exceeded, obtain the next set of results by making the same call, but set NextToken to the value returned by NextToken in the previous result set.
Comment: How to access previous 12 months?
Comment: Spot Price history is only available for the previous 90 days.
Comment: For more result, I use the following: paginator = ec2.get_paginator('describe_spot_price_history')
page_iterator = paginator.paginate(StartTime= t, EndTime =  f, MaxResults=2000 )
for page in page_iterator: output = page['SpotPriceHistory']
Comment: is that will access nextoken? or I need to access it with another way? could u plz help me in this way?
Comment: You can either use a paginator, or you can call the normal method again, passing a `NextToken` value.
Comment: HOw? from where i can get the NextToken
Comment: So, paginator is a right way to access all pages? But still i have a same number of result as i call the same history method
Comment: I'm sorry, but I don't understand what you are asking. Feel free to Edit your question to add details (eg what you are receiving vs what you are expecting).


PHP Request To Fetch Data From WSO2 Data Services Server

Question: I have successfully set up the WSO2 Data Services Server and have created to some procedures which fetch the data as XML. I can see them in the DSS admin panel. In order to do that I have completed DSS wizard which allows me to add new data sources and created some procedures as well. That's fine.
What I want to know is how can I connect PHP with the DDS ? In other words How can I make a PHP request so that data are fetch to my PHP script from the DSS ? Is it possible to data to be fetched as JSON ? If so how ?
Answer: This simple script might give you an idea how to connect to your server:
<?php
$client = new SoapClient("http://*yourDSserver*:*yourPort*/services/yourDSservice?wsdl",array('trace' => 1));

try {
  $info = $client->__soapCall("op_name",array(*--your request data goes here--*));

} catch (SoapFault $fault) {
  print($fault);
}

print_r($info);

change yourDSserver, yourDSport,yourDSservice, as well as op_name to values suitable for your installation.
A few more lines that might be usefull for debugging:
echo "\nRequest:\n" . $client->__getLastRequest() . "\n";
echo "\nResponse:\n" . $client->__getLastResponse() . "\n";

Answer: To access the data service from PHP, there are many approaches. The first one would be to use a SOAP client of PHP to access the data services. The data service endpoint URLs are mentioned in the service dashboard of the service, for you to use as the SOAP endpoint. Also, if you want to access it RESTfully, you can create data service "resources", where you can use access the created HTTP paths, as configured. JSON is also supported, and you can use this as a reference on how to do that. Also, basically if that approach is not suitable, you can use WSO2 ESB to mediate the JSON requests to remove the XML namespaces and so on and get the JSON payload in the way you want. A sample on how to do that is shown here.
Cheers,
Anjana.


hook_user_login being called twice?

Question: I'm storing the session details from Drupal to share with another system, so users don't have to log in twice for each system.
It appears that my hook_user_login is being called twice.  Right now I'm just calling:
mymodule_user_login(&$edit, $account) {
  debug(session_id());
}

to see if the session I'm storing is the same as the session being used.
I'm getting the debug statement twice, with two different session_id values.
Am I missing something here?
Comment: Are you sure there is just a logged-in user when you are executing that code?
Comment: Yes, it's my own dev machine, and I'm the only one connected, and the only one with the test credentials.  I am authenticating against my own user db however, but I can't see how that would trigger two login hooks
Comment: If you are already logged in when executing that code, and there isn't code that impersonates another user, that hook should not be more than once. It is rather difficult to say what it is going on, though.
Comment: I'm starting from anonymous, and then logging in.  Is the anonymous session playing somehow in this?
Comment: the session_name() isn't changing, just the session_id().  Does calling session_id() change its value?
Comment: I tried saving data in the session, logging in and out, with a custom module that implements `hook_user_login()` (and calling `session_id()`) installed on my test site, but I could not get that hook to be called twice; it is just called when the current user is an authenticated user.
Comment: Just curious, are you using `user_external_login_register`? Adding that to the login form validation can trigger two login events.
Answer: I know this is not an answer (I don't yet have enough reputation to post a comment); I am also uncertain if you are still looking for answer. It is possible sometimes for distributed modules to call user_login_submit.
For example, the D7 LDAP module's _ldap_authentication_user_login_authenticate_validate calls user_login_submit, which causes the user login form to fork two calls to user_login_finalize, both of which triggers hook_user_login. If you are still experiencing this problem, one possible way to debug your application is to use PHP XDebug, and dsm function traces.
After you have installed XDebug, you can use xdebug_call_function() on user_login_submit, user_login_finalize, user_module_invoke to see where the two calls of hook_user_login are coming from. I hope this helps.

hook_user_login being called twice?


Role based authentication in Vue using JWT

Question: I'm working on a project with a RESTful Java backend and a Vue SPA front-end. Whilst figuring out how to do user authentication I came across JWT-tokens and since it (sorta) was what I was looking for I recklessly implemented it.
Few weeks later I realize that because the content being shown on the client side, depends on the users role, the client of course needs to know the users role. For obvious reasons I don't want to store the users role inside my client.
My question: I could create a request on the server that looks at the Authentication header and returns the role, but would this be save? If not are there any common strategies when it comes to roles and JWT-tokens? Or should I forget the JWT way of doing things and implement another kind of authentication entirely?
Answer: JWT is the common way to Authenticate users with SPA as Frontend + REST Api as backend.

You definitely should not store your Token Secret in frontend app.
You definitely can do kind of <code>/user/roles</code> endpoint in your API which will return the list of user's role.

Point 2 solution is 100% safe. Why? Even if someone will hack your frontend app to show the content which they should not see, your backend is checking Authorization at endpoint, so they will not get/put/change any data which they have not privilege to access in their JWT.
Comment: Thank you for your Crystal clear anwser. It helps me a lot getting a general understanding.


Prevent Apache from deleting files on 500 error

Question: I have the following problem when using Apache 2.2.16 running on Windows Server 2008 R2: I have the server configured as a WebDAV server for automated client software to upload files to. However if the network connection between the client and the server is lost then it takes a while for the connection to timeout and for Apache to generate an HTTP 500 error, once this error is generated Apache deletes the partially uploaded file that caused the error. This would be fine but the automated software will retry the upload, sometimes before the network connection has timed out. This has the effect that a correctly fully uploaded file will be deleted by Apache when the previous failed upload of the same file timed out.
Obviously this is highly undesirable behaviour because as far as the client is concerned the file was uploaded correctly. Is there a simple config change to prevent Apache from deleting uploaded files in the event of an upload failure or a 500 error occurring?
Thanks in advance.
Answer: In the end we found out that this issue was because Apache was rejecting the same message sent within five minutes of the failed message. We solved this in our upload client by putting in a configurable wait.
Hopefully that might help someone who has had the same issue. i admit it's a niche problem!


Streaming Video from Linux to Windows

Question: I have been searching on the Internet but find no straight answer on what to use in this case. Basically, I have a client that has an Icecast radio streaming server running on Linux. People are able to listen to his radio by connecting on his website. This specific client would like to have a video streaming service where he can stream either a webcam stream or a straight video file to his linux server. People would then connect to a specific webpage to see the live video stream.
First, I would like to know if it is possible to stream video from VLC to Icecast and then connect to the Icecast mountpoint on the web in order to see the video.
Second, I would like to know if Red5 would be a good server software in case Icecast is not a good one.
Also with all those questions, what would be the protocol used and what would be the file format for the video files that he would like to stream.
Regards and thank you for the answer :)
Answer: This question is borderline off-topic as you are essentially asking for a full solution.
Regardless, Icecast does provide both audio and video streaming and would work for a small number of clients.  Video streaming requires a LOT of bandwidth however and overall my recommendation would be to utilize a web host that specializes in video streaming to provide the client end-points.

Streaming Video from Linux to Windows


mandar dado json do ajax para o php

Question: Arquivo JS:
var jsonsave = [];

var dado = new Object();
dado.nome= "João";
dado.cpf = "000";
dado.teste = "teste";
var jason = JSON.stringify(dado);
jsonsave.push(jason);

$.ajax({
               type: "POST",
               dataType: "JSON",
               url: "http://localhost/moodle/my/index.php",
               data: "data="+jsonsave+"&ENVIO_RESPOSTAS=true",
               success: function(data){
                   console.log('foi');
               },
               error: function(data)
               {
                    console.log(data);

               }
           });

Arquivo php:
$arquivo = $_POST['data'];
$value= json_decode($arquivo, TRUE);

Eu chamo a página php pelo ajax, porém não consigo pegar os valores do json, vem null, alguem tem alguma ideia?
Comment: Vi agora que você já tinha postado essa pergunta a pouco. E postou novamente. Não é recomendável fazer isso. Sua pergunta será respondida assim que alguém com conhecimento no assunto e com disponibilidade de tempo, visualizar ela. Postar a mesma pergunta novamente, só vai fazer com que ela seja excluída.
Comment: ah eu exclui aquela, porque eu ja tinha postado, achei que não tinha ido, foi eu mesmo que exclui aquela, desculpe
Comment: Ok, nesse caso, desconsidere o que eu disse.
Answer: Tenta enviar os dados no formato JSON puro:
 ...
 dataType:"json",
 url: "http://localhost/moodle/my/index.php",
 data: jason,
 ...

E no PHP, para obter os valores:
$json = json_encode($_POST);
var_dump($json);

Observação:
Para que você consiga checar se o PHP recebeu os dados, você deve mostrar o retorno no terminal, ou de alguma outra forma. Pois fiz um teste e constatei que mesmo da forma que está fazendo, o PHP recebeu os dados. Então recomendo a seguinte alteração para poder checar isso:
 success: function(data){
     console.log(data);
 },

Resposta obtida ao executar essa alteração no código que você forneceu:
//string(62) ""{\"nome\":\"Jo\u00e3o\",\"cpf\":\"000\",\"teste\":\"teste  //\"}""

Ter os dados nesse formato, no PHP, não faz muito sentido, talvez o que você queira é obter um array/object. Nesse caso, está utilizando a função inversa. O correto seria json_decode:
$json = json_decode($_POST['data']);
var_dump($json);

E a saída:
//object(stdClass)#1 (3) {
//  ["nome"]=>
//  string(5) "João"
//  ["cpf"]=>
//  string(3) "000"
//  ["teste"]=>
//  string(5) "teste"
//}

Comment: não deu certo mesmo assim
Comment: vou fazer um teste, ver o que está ocorrendo então.
Comment: eu preciso manter o ENVIO_RESPOSTAS também no caso
Comment: Teu código está funcionado, assim como a forma que sugeri. Tem certeza que está checando da forma correta? Da uma olhada na edição da resposta.
Comment: eu alterei para:
    type: "POST",
                   dataType:"JSON",
                   url: "http://localhost/moodle/my/index.php",
                   data: "data="+jsonsave+"&ENVIO_RESPOSTAS=true",
como você sugeriu mas ele cai no error do ajax, se eu tiro o dataType ele vai porém vai nulo
Comment: qual a mensagem de erro?
Comment: object {readystate: 0, responseJSON: undefined, status: 0, statusText: "error"}
Comment: "url: "localhost/moodle/my/index.php";, ", o código é esse mesmo? pois tem um ";" onde não deveria.
Comment: não, no codigo ta virgula mesmo, acho que foi edição do site
Comment: Ta retornando null? No seu código original você não estava exibido a resposta do servidor. Apenas a mensagem "foi". Explica melhor isso.
Comment: ele cai no sucess se eu tiro o datatype: "json" , agora se eu coloco vai pro error

↵↵↵↵↵↵", status: 200, statusText: "OK"}
Comment: edita o código da pergunta, da forma que está fazendo agora.
Comment: alterei la o codigo
Answer: Uso muito isto:
var PrimeiroNome=$("#InputPrimeiroNome").val();

$.post("pagina2.php",{nome:PrimeiroNome}).done(function(resultado){
   $("div #recebeValor").html(resultado);
});

Basicamente dentro do javascript vai abrir uma pagina2.php em segundo plano onde vai enviar o valor da input com o id ="InputPrimeiroNome" atribuilo a um POST para O PHP com o nome de "nome" e vai devolver o echo dessa pagina php e atribuilo a div #recebeValor
Com algumas brincadeiras assim pode tornar o site bem dinamico
Answer: Tente utilizar da seguinte forma:
    $.ajax({
                   type: "POST",
                   url: "http://localhost/moodle/my/index.php",
                   data: function (data){
                      data.data = jason;
                      data.envio_resposta= true;
                   },
                   success: function(data){
                       console.log('foi');
                   },
                   error: function(data)
                   {
                        console.log('n foi');

                   }
               });

Abraço,
Answer: No meu caso deu certo o usado pelo Bruno Almeida:
var PrimeiroNome=$("#InputPrimeiroNome").val();
$.post("pagina2.php",{nome:PrimeiroNome}).done(function(resultado){
   $("div #recebeValor").html(resultado);
});
e no php usei:
$arr = json_decode($_POST['nome']);
echo $arr
Answer: Utilizo dessa forma:
$postdata = file_get_contents('php://input');
$json = json_decode($postdata);

mandar dado json do ajax para o php


How does one design a custom loss function? What features make a loss function "good"?

Question: I have a custom situation for which I am trying to design a cost function.
The idea is that you have a stack of LSTMs doing something slightly unconventional. Each LSTM$_l$ computes a linear transformation of its hidden layer $V_{l-1}h^t_l$ to predict the hidden layer vector $h^{t+1}_{l-1}$ at the level below; at the same time, it should not stray far from the value it was predicted to have from the layer above (i.e., we want $h^t_l \approx V_lh^{t-1}_{l+1}$). That is, the goal at each layer is that:
$V_{l-1}h^t_l \approx h^{t+1}_{l-1}$
and
$h^t_l \approx V_lh^{t-1}_{l+1}$
I would like to design a cost function that will bring about this situation in a system of $L$ stacked LSTMs. I introduced an error variable at each layer $E^{t}_{l} = V_lh^{t-1}_{l+1} - h^{t}_{l}$, where $l$ ranges from $0$ to $L-1$, such that $h^t_0 \equiv x^t$, and the goal is to model a sequence $x^{1:T}$. The loss function is then:
$\Large \mathcal{L}_{\text{subtract}} = \sum_{t=1}^T\sum_{i=0}^{L-1}|E^t_i|$
And the sequence of steps is the following:

However, this simple subtractive loss function is not very good, in the sense that the model could not even overfit a minibatch (I did this as a test). I tried a divisive loss:
$\Large E^t_l  = \frac{V_{l}h_{l+1}^{t-1}}{\max(\epsilon, h_{l}^t)} + \frac{h_{l}^t}{\max(\epsilon,V_{l}h_{l+1}^{t-1})}$
But the model also did not learn anything (probably because division led to numerical instability). How do I go about designing a stable loss function that I can use at all layers?
NB My goal for this project is to build a language model based on predictive coding principles, to test a hypothesis about brain activity (i.e., the goal is not to beat SoTA).
Comment: I'll ask the same question I asked when you brought up this 'divisive loss', because it seems this may be an XY problem—why do you think it makes sense at all as a function worth computing?
Comment: The divisive loss works if the goal is to reconstruct a one-hot vector: https://imgur.com/a/u3EboOp ; when the entries corresponding to $0$ have a value greater than $\epsilon$, the loss will be large; and when the entries corresponding to $1$ are small, the loss will also be large. And when the reconstruction vector entries corresponding to $0$ are $< \epsilon$ and those corresponding to $1$ are $\approx 1$, the loss will be small. So its minimum is achieved when the "correct" vector is reconstructed; that's the rationale. But now I realize that it will not work for arbitrary embeddings.
Answer: To answer the titular question, a key characteristic of a loss function is that the loss is minimized at the target values $y$. In other words, if you're estimating a quantity, the least loss should be assigned to the estimates that are exactly correct.
Using $\mathcal{L}_\text{subtract}$ and divisive losses are not loss functions because they do not involve the target variables. In other words, you can achieve the absolute minimum value of 0 without learning anything about what you want to model.
Consider this loss function $L(\hat{y}) = (\hat{y} - c)^2$ for model outputs $\hat{y}$, a constant $c$ and model targets $y$. This function is clearly convex in $\hat{y},c$, but it has nothing to do with the model targets $y$. If $c$ is a fixed target and our model outputs $\hat{y}$, then we achieve a minimum at $\hat{y}=c$, and this is true no matter what the targets $y$ happen to be.
My suggestion is to use $\mathcal{L}_\text{subtract}$ as a regularization to augment a typical loss function. So if you're using a binomial cross entropy loss, you could write down a total loss as
$$
\text{total loss} = \text{BCE}(y,\hat{y}) + \lambda \mathcal{L}_\text{subtract}$$
for $\lambda >0$ some tuning parameter controlling how much deviation can occur from one layer to the next. Likewise, you can use an alternative loss in place of $\text{BCE}$ to model other types of data.

This seems related to, but not exactly the same as "Regularizing RNNs by Stabilizing Activations" by David Krueger & Roland Memisevic. Perhaps their work and bibliography is of interest.
Comment: (+1) It has not been mentioned but making a loss-function differentiable (preferably twice) will help the optimisers quite a bit. Smoother surface to explore / easier convergence.
Comment: Ah, so it's not that you're computing the error only using the *hidden* layers, but using *all* layers, hidden, input or output. I still think you should use a loss function of the type that I describe at the end: apply the regularization to the hidden layers, but compute the model loss using an appropriate loss. MAE for binary targets isn't a good loss because it penalizes all errors proportionally to misfit, instead of assigning much larger loss the further you are from the target.
Comment: Yep, that's a standard language model. Given $k$ words, predict the $(k+1)$th word. Use a cross-entropy loss.
Comment: Note that the LSTM at the bottom of the stack is directly predicting the target input sequence; although the rest of the loss is concerned with predicting the hidden state of the LSTM below, it is all eventually coupled to the target sequence. Maybe I misunderstood your answer - am I missing something?
Comment: Where does your model compare the target $y$ to the model predictions $\hat{y}$? Is $h$ a hidden sequence or the target vector?
Comment: The target at each time step $t$ is the $t$'th item in the input sequence; that is, the target is $x^t$, which is defined in the model as $h^{t}_{0}$. The relevant error term is $E^{t}_{0} = V_0h^{t-1}_{1} - h^{t}_{0} = V_0h^{t-1}_{1} - x^{t}$. This will be large when the prediction $V_0h^{t-1}_{1}$ doesn't match the target.
Comment: That's right - I am using the hidden layers to predict the input sequence. Note that there is no explicit output term.
Comment: When you say things like "there is not explicit output term," it makes it sound like your model doesn't make a prediction of  the target, which is what my answer responds to. But you said in your previous comment that the last layer of the network is the prediction of the input. Is the goal to learn the function $f(x)=x$ or the function $f(x)=y$ for inputs $x$ and targets $y$?
Comment: The targets are the future inputs given some past input - the goal is to learn $f(x^{1:t-1}) = x^t$. I don't want to constrain myself to inputs that are necessarily one-hot vectors; I want to use this model with, e.g., a sequence of pre-trained word embeddings.


Prove that a function $g$ is differentiable in 0

Question: Let $f : \mathbb{R} \to \mathbb{R}$ continu in 0 and let $g : \mathbb{R} \to \mathbb{R}$ defined by $g(x) = x f(x)$, for $x \in \mathbb{R}$. Prove that $g$ is differentiable at 0 and determine $g'(0)$.
This is an assignment for my class, but I'm having trouble finding the right argumentation. My thoughts were the following:
If I can prove that the function f is differentiable at 0, and I can prove that a function $h : \mathbb{R} \to \mathbb{R}$ defined by $h(x) = x$ is differentiable at 0, then also the function $g$ is differentiable at 0?
Well, I know how to prove that the function $h$ is differentiable in 0 and what the derivative is. But how do I prove that the function $f$ is differentiable?
Comment: The problem with your approach is that continuity does not imply differentiability. For example $f(x) = |x|$ is continuous at $0$ but not differentiable.
Comment: You can not know that $f$ is differentiable, try using the definitions of both differentiability and use the continuity of $f$.
Answer: $f$ may not be differentiable at $0$, but the theorem still holds, as
$$g'(0) = \lim_{h\to 0} \frac{g(h)-g(0)}{h} = \lim_{h\to 0} \frac{hf(h) - 0f(0)}{h} = \lim_{h\to 0} f(h) = f(0)$$
where the last equality holds because $f$ is continuous at $0$.
Answer: We want to evaluate $$\lim_{h\to0} \frac{(0+h)f(0+h)-0\cdot f(0)}{h} = \lim_{h\to0} \frac{hf(h)}{h} = \lim_{h\to0} f(h).$$
Since $f$ is continuous at $x=0$, the limit exists and equals $f(0)$. Therefore, $g$ is differentiable at $x=0$ and $g'(0) = f(0)$.
Answer: Hint:
$$\lim \frac{g(t)- g(0)}{t}=\lim\frac{tf(t) - 0\cdot f(0)}{t}=\lim f(t)$$
Where the limit is taken as $t\to 0$


python ............. delete

Question:
Possible Duplicate:
Python delete in a string

I have the following list which has 3 elements:
YELLOW,SMALL,STRETCH,ADULT,T
YELLOW,SMALL,STRETCH,ADULT,T
YELLOW,SMALL,STRETCH,CHILD,F
I would like to remove everything after the last comma.. Note that I could have more than one character after the last comma. How can I do that?
Thanks
Comment: @hssss: You just duplicated a question you yourself asked (and received good answers for) less than 30 minutes ago. http://stackoverflow.com/questions/4357499/python-delete-in-a-string
Comment: +1 that this is a duplicate question.
Comment: @sukhbir.  You may be right.  Never the less... `bad questions beget bad answers` (see below for confirmation)
Comment: @sukbbir (I just noted your [deleted] answer).  That is/was the only correct answer, if we assume that this 2nd question of hssss is about deleting part of each of strings elements in an a list.
Comment: mjv, hehe I realize now after reading the last line 'more than one character'. But anyways since the question is closed, doesn't really matter :)
Answer: If you name your string s it would be:
s = s[:s.rfind(",")+ 1]

remove the one if don't want the comma at the end.
Comment: typo: "+ 1" should be inside the braces;... s[:s.rfind(",")+1]
Answer: If you refer to string elements, you can utilize
str.rsplit() to separate each string, setting maxsplit to 1.

str.rsplit([sep[, maxsplit]])
Return a list of the words in the string, using sep as the delimiter string. If maxsplit is given, at most maxsplit splits are done, the rightmost ones. If sep is not specified or None, any whitespace string is a separator. Except for splitting from the right, rsplit() behaves like split() which is described in detail below.

>>> lst = "YELLOW,SMALL,STRETCH,ADULT,T"
>>> lst.rsplit(',',1)[0]
'YELLOW,SMALL,STRETCH,ADULT'
>>>

Answer: Try this:
In [84]: s = 'YELLOW,SMALL,STRETCH,ADULT,Tkj'

In [85]: re.search('(.*,)(.*)$', s).groups()
Out[85]: ('YELLOW,SMALL,STRETCH,ADULT,', 'Tkj')
In [86]: (a, b) = re.search('(.*,)(.*)$', s).groups()

a will contain the part you want and b will contain the part you want to delete

python ............. delete


Where is the refreshToken endpoint implementation?

Question: I am using springboot-security-jwt because have good recomendation, and it is running... But when I was testing refreshToken, where the implementation?  How to use it?
Perhaps it is so obvious for a "Senior Developer Spring", but it is not for me, I not see it there. Where the /auth/token endpoint implementation?
There are some examples or documentation about it and how to (parameters) call it?
... Where the springboot-security-jwt /token endpoint implementation? to check it (or a kind of "health endpoint test")...
Answer: The primary configuration in the project springboot-security-jwt is in the WebSecurityConfig.java: (see https://github.com/svlada/springboot-security-jwt/blob/master/src/main/java/com/svlada/security/config/WebSecurityConfig.java).
In this class you will see a bean created of type AjaxLoginProcessingFilter configured with to intercept requests matching "/api/auth/login".  This will process the login and generate the JWT tokens.
You can then follow to the next bean configured - JwtTokenAuthenticationProcessingFilter to see what it is intercepting and authenticating using the JWTToken provided on the api requests
refreshToken is a standard spring controller - see RefreshTokenEndpoint class (https://github.com/svlada/springboot-security-jwt/blob/master/src/main/java/com/svlada/security/endpoint/RefreshTokenEndpoint.java)
The author also provides a detailed explanation in the Blog.md under the etc folder - check it out! there are lots of useful links to get up to speed on using JWTs