Send email through Gmail API Google App Engine

Question: Trying to send email using the Gmail API into GAE. Previously I created a Service account key into Credentials page, it generates a .P12 file which is in setServiceAccountPrivateKeyFromP12File parameter. It has a ID key joined to the account firstname.lastname@example.com into Service account page. The code:
    /* Application name. */
    private static final String APPLICATION_NAME = "appnamefromappengine";

    String emailAddress = "firstname.lastname@example.com";
    JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();
    try {
        HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();

        Set<String> scopes = new HashSet<String>();
        scopes.add(GmailScopes.GMAIL_SEND);
        scopes.add(GmailScopes.GMAIL_COMPOSE);
        scopes.add(GmailScopes.MAIL_GOOGLE_COM);
        scopes.add("https://www.googleapis.com/auth/admin.directory.user");

        GoogleCredential credential = new GoogleCredential.Builder()
            .setTransport(httpTransport)
            .setJsonFactory(JSON_FACTORY)
            .setServiceAccountId(emailAddress)
            .setServiceAccountPrivateKeyFromP12File(new File("/home/myuser/Test/src/main/webapp/resources/**somename**cd30e7118ad5.p12"))
            .setServiceAccountScopes(scopes)
            .setServiceAccountUser("firstname.lastname@example.com")
            .build();
        Gmail gmail = new Gmail
            .Builder(httpTransport, JSON_FACTORY, credential)
            .setApplicationName(APPLICATION_NAME)
            .build();

        Properties props = new Properties();
        Session session = Session.getDefaultInstance(props, null);
        MimeMessage message = new MimeMessage(session);
        message.setFrom(new InternetAddress("email@example.com"));
        message.addRecipient(javax.mail.Message.RecipientType.TO, new InternetAddress("email@example.com"));
        message.setSubject("Test Mail");
        message.setText("Test Mail");

        Message msg = createMessageWithEmail(message); //createMessageWithEmail function from Gmail API
        msg = gmail.users().messages().send(emailAddress, msg).execute();

        System.out.println("Mail was sent. Message id: " + msg.getId());

    } catch (GeneralSecurityException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (MessagingException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

It returns me this error:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 400
  Bad Request
  {   "code" : 400,
  "errors" : [ {
      "domain" : "global",
      "message" : "Bad Request",
      "reason" : "failedPrecondition"   } ],
  "message" : "Bad Request" }

I'm not sure which parameter I'm setting wrong here in code or into Google cloud console. What else can I try ?
Comment: Check this SO question [Gmail REST API : 400 Bad Request + Failed Precondition](http://stackoverflow.com/questions/29327846/gmail-rest-api-400-bad-request-failed-precondition) if it can help you :)
Comment: It was @Nicholas. The problem was that I didn't configure Delegate domain-wide authority to the service account into admin console. Either way all the code is completely working if someone wants to send an email using Gmail API. The question could be complemented answer how to use Email API authorized senders into the code. Once someone registered an email in that section, how to reference that email. Thanks for all.
Comment: Thanks for clarifying the issue.  Feel free to post it as an answer and I'll upvote it.
Comment: Was the link provided by @KENdi helpful?  Have you been able to narrow down or address the failing precondition?
Answer: The problem was that I didn't configure Delegate domain-wide authority to the service account into Admin console.

Go to your Google Apps domain's Admin console.
Select Security from the list of controls. If you don't see Security
  listed, select More controls from the gray bar at the
      bottom of the page, then select Security from the list of controls.
      If you can't see the controls, make sure you're signed in as an
      administrator for the domain.
Select Show more and then Advanced settings from the list of
  options.
Select Manage API client access in the Authentication section.
In the Client Name field enter the service account's Client ID. You
  can find your service account's client ID
      in the
          Service
                  accounts section of the Developers Console's Permissions page.
In the One or More API Scopes field enter the list of scopes that
  your application should be granted access to.
      For
              example, if
                      your application needs domain-wide access to the Google Drive API
                      and the Google Calendar API, enter:
                      https://www.googleapis.com/auth/drive,
                      https://www.googleapis.com/auth/calendar.
Click Authorize.

All the documentation:
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
Comment: the problem with this is that it gives the service account access to all accounts in such domain within the specified scope, which might not be ideal. Google doesn't provide a way to only limit it to a single or a number of specific accounts within that domain. Also, it is only available to G Suite customers. Normal Gmail accounts don't have this.
Answer: Optionally you could use a library like this one, who will do all of the set up for you:
https://github.com/3wks/spring-gae-gmail


Return data from promise in nodejs

Question: i have an issue with a promise where I am unable to return the data from that promise...
I have tried different solutions so far, but I can't get it to work; I am new to promises and I am trying to understand what I am doing wrong:
 try {
      const classList = options.hash.class;
      const filePath = path.resolve(__dirname, path.join(PATH.documentation, options.hash.startPath, options.hash.icon + ".svg"));
      let data = fs.readFileSync(filePath, "utf-8");
      let transformedSVG;

 // Promise
      const transformable = () => {

        // another promise... apparently
        svgo.optimize(data, { path: filePath }).then(function(result) {
          transformedSVG = result.data;
          transformedSVG = transformedSVG.replace(
            /^<svg/u,
            '<svg class=" ' + classList + '" '
          );
          return transformedSVG;
        });
      };

      // wait for Promise to be finished
      async function getResult() {
        let result = transformable;
        return result;
      }
      // wait for transformable to receive data and get the result
      async function doTask() {
        return await getResult();
      }

      // store svg data returned from doTask
      transformedSVG = doTask();
      console.log(transformedSVG);
} catch (err) {
      console.error(err);
}

any help with this?
Answer: You're doing several things wrong. I fixed issues and added comment:
  const transformable = () => {
    // another promise... apparently
    return svgo.optimize(data, { path: filePath }).then(function(result) { // <== add return
      transformedSVG = result.data;
      transformedSVG = transformedSVG.replace(
        /^<svg/u,
        '<svg class=" ' + classList + '" '
      );
      return transformedSVG;
    });
  };

  // wait for Promise to be finished
  async function getResult() {
    let result = await transformable(); // <== call the function with `await`
    return result;
  }
  // wait for transformable to receive data and get the result
  async function doTask() {
    return await getResult();
  }

  // store svg data returned from doTask
  transformedSVG = await doTask(); // <== add `await`

One more thing to note is your doTask() method is doing nothing, so just remove it and call await getResult() directly.
Comment: `getResult()` doesn't *do* anything either, you could omit it and just call `transformable()` directly :-)
Comment: Thank you I appreciate that alot, problem is, that I can't use the await there, as I get the error message that I can use await only within an async function. I do get the result within the promise and I can console.log completely fine, but I cant return it so I get the actual data within my template q.q

Return data from promise in nodejs


Why isn't fuel ux working in my MVC5 app (wizard, and general render issue)

Question: I'm trying to get fuelUX working in my MVC5 app. I've created a simple view to sandbox in, but it isn't behaving as I expect.
There are two (so far) issues I'm getting.
1) Everything is rendering to the full width of the page.
2) The wizard renders appropriately (other than issue 1) but the tabs are all disabled, and the next button doesn't do anything.

Here is the page source that gets rendered out of asp.net
<!DOCTYPE html>

    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Home Page - My ASP.NET Application</title>

    <link href="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css" rel="stylesheet">
    <link href="//www.fuelcdn.com/fuelux/3.6.3/css/fuelux.min.css" rel="stylesheet">

            <input class="sr-only" data-toggle="#hereKittyKitty" type="checkbox" value="option1">
            I love kittens!

Great. Meow, too!

        <input type="text" class="form-control">

            <button type="button" class="btn btn-default dropdown-toggle" data-toggle="dropdown"></button>

One

Two

Three

Four

1Campaign

2Recipients

3Template

            <button type="button" class="btn btn-default btn-prev">Prev</button>
            <button type="button" class="btn btn-default btn-next" data-last="Complete">Next</button>

Setup Campaign

Veggies es bonus vobis, proinde vos postulo essum magis kohlrabi welsh onion daikon amaranth tatsoi tomatillo melon azuki bean garlic. Beetroot water spinach okra water chestnut ricebean pea catsear courgette.

Choose Recipients

Celery quandong swiss chard chicory earthnut pea potato. Salsify taro catsear garlic gram celery bitterleaf wattle seed collard greens nori. Grape wattle seed kombu beetroot horseradish carrot squash brussels sprout chard.

Design Template

Nori grape silver beet broccoli kombu beet greens fava bean potato quandong celery. Bunya nuts black-eyed pea prairie turnip leek lentil turnip greens parsnip. Sea lettuce lettuce water chestnut eggplant winter purslane fennel azuki bean earthnut pea sierra leone bologi leek soko chicory celtuce parsley jícama salsify.

Answer: You'll need to initialize the wizard.
Either add data-initialize="wizard" or use JavaScript via $('.wizard').wizard();
See Wizard Usage.


.BAT File to Execute SQL Query (Backup)

Question: I have an SQL script which is as follows:-
   BACKUP DATABASE [**********] TO  DISK = N'G:\Super Sacred Databases\Orderwise Backup\20-04-2017-13-05.bak' WITH NOFORMAT, NOINIT,  NAME = N'**********-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
GO

On the server I wish to run a .bat file that i can schedule on windows task scheduler.
My confusion is, how do i write a .bat file to execute this query on the SQL Server?
I have so far tried to use this:
sqlcmd -U myLogin -P myPassword -S MyServerName -d MyDatabaseName
    -Q ?

The problem is that is doesnt work at all for me, furthermore i have no idea what the -Q ? would be to run a specific query in SQL Server.
Furthermore im not sure exactly how to specify the Servername or Databasename in the fields required.
I know im new to this, but im trying hard.
Thanks in advance for your help!
CONTINUED -
Now i am working with this
sqlcmd -U s******a -P 0****! -S LPSERVER\ORDERWISE
-Q "BACKUP DATABASE [********Live] TO  DISK = N'G:\Super Sacred Databases\Orderwise Backup\20-04-2017-13-05.bak' WITH NOFORMAT, NOINIT,  NAME = N'******estLive-Full Database Backup', SKIP, NOREWIND, NOUNLOAD,  STATS = 10
GO
"

But it just doesnt seem to do anything just shows the cmd line in the CMD interface and does nothing more.
Any suggestions?
Comment: Possible duplicate of [Execute SQL script from command line](http://stackoverflow.com/questions/6737064/execute-sql-script-from-command-line)
Comment: Thanks, ill check that out now and remove my question if it solves the same problem.
Comment: No need to remove your question, SO does not mind duplicates: If you didn't find the other question, others might not find it as well. They might, however, find your question and the link there.
Comment: -Q is the query. so for instance -Q "select from DB where NAME='Frank'"
Comment: anyway, you say command is not working for you, what errors are you getting?
Comment: Do you not use SQL Agent? An Agent Job is the usual way of doing this...
Comment: I tried using SQL Agent but i think if im not mistaken Express Versions of SQL dont have an agent, i only think this is true because of utterly failing to activate the agent.
Answer: Since BACKUP DATABASE can be executed in the context of any database, you can just skip the -d parameter.
The "server name" is usually the Windows name of the PC running the SQL server instance, optionally extended by \InstanceName, if you are not running the default instance of SQL Server. For the local PC, you can simply use . as the server name.
-Q contains the query you are executing:
sqlcmd -U myLogin -P myPassword -S MyServerName -Q "BACKUP DATABASE ..."

Comment: Does it Go "BACKUP DATABASE *DATABASENAME*"?
Comment: (1) Make sure your SQL script works when run in SQL Server Management Studio. If you need help with that, feel free to ask a new question. (2) Once your SQL script works, plug it in here: `... -Q "...my sql script..."`.
Comment: Also does the BACKUP DATABASE cmdlet allow me to choose the destination of the backup save or will it use the queries default?
Comment: @LewisMCT333: I don't understand your question. You write `TO  DISK = N'G:\Super Sacred Databases\Orderwise Backup\20-04-2017-13-05.bak'`... is this not where you want your backup to be saved?
Comment: The SQL script i have in my initial post works perfectly. I guess that should do the trick then.
Comment: I didnt realise you could explicitly run SQL code in CMD using cmdlet, dumb me. That makes life easier. Is there a way to add some conversation between the backup SQL function to show that it is processing so you can see that on the CMD screen so it doesnt look like its just hanging?

-Correction it does just hang and doesnt appear to work... No errors though, just shows the script in the CMD line.
Comment: @LewisMCT333: (1) The "GO" shouldn't be necessary. (2) Please, no line breaks in your batch file!

.BAT File to Execute SQL Query (Backup)


Diffeomorphisms of $S^1$ is path connected.

Question: Let $X=\text{Diff}^+(S^1)=$ set of all orientation preserving diffeomorphism of $S^1$. I want to prove that this is path connected.
I tried with the following argument:
Let us consider the natural action of $O(2,\mathbb{R})$ on $S^1$. Now define the map from $O(2)$ to  $X$ defined as $A\mapsto \phi $ where $\phi$ is the action i.e. $\phi(x)=Ax$. Now, I am having problem in showing the above map is onto (that is my guess).
Thanks.
Comment: Not all diffeomorphisms of $S^1$ are rotations.
Comment: Not only is it not onto, but its codomain can't be taken to be $\text{Diff}^{+}$ because the reflections in $O(2)$ reverse orientation. You want $SO(2)$.
Comment: How will we prove it is path connected?
Answer: $\text{Diff}^+(S^1)$ path connected means that for each $f \in \text{Diff}^+(S^1)$ there exists a homotopy $H : S^1 \times I \to S^1$ such that $H_t \in \text{Diff}^+(S^1)$ for all $t \in I$, where $H_t(x) = H(x,t)$, and $H_0 = f$, $H_1 = id$. Here $I = [0,1]$.
Consider the covering $e : \mathbb{R} \to S^1, e(x) = e^{2\pi i x}$. The map $f \circ e : \mathbb{R} \to S^1$ has a lift
$F : \mathbb{R} \to \mathbb{R}$ (i.e. $e \circ F = f \circ e$). It is an orientation preserving diffeomorphism such that $F(x + 1) = F(x) + 1$ for all $x$.
Define
$$\Gamma : \mathbb{R} \times I \to \mathbb{R}, \Gamma(x,t) = (1-t)F(x) + tx .$$
We have $\Gamma_t'(x) = (1-t)F'(x) + t > 0$. Therefore each $\Gamma_t$ is an orientation preserving diffeomorphisms such that $\Gamma_t(x + 1) = \Gamma_t(x) + 1$ for all $x$. Therefore $\Gamma$ induces a unique homotopy $H : S^1 \times I \to S^1$ such that $e \circ \Gamma = H \circ (e \times id_I)$. We  have $H_0 = f, H_1 = id$ and all $H_t$ are orientation preserving diffeomorphisms.
Added on request:
$e : \mathbb{R} \to S^1$ is a smooth covering, $F$ is a lift of the smooth map $f \circ e$, hence $F$ is a smooth map. Consider any $x \in \mathbb{R}$. We have $e(F((x,x+1))) = f(e((x,x+1)) = f(S^1 \backslash \{ e(x) \}) = S^1 \backslash \{ f(e(x)) \}$ because $f$ is a bijection. $F((x,x+1))$ is a connected subset of $\mathbb{R}$, i.e. an interval. It is mapped by $e$ onto $S^1 \backslash \{ f(e(x)) \}$. This is only possible if $F((x,x+1)))$ is itself an open interval of length $1$. Therefore
(1) $F((x,x+1)) = (x',x'+1)$ for some $x' \in \mathbb{R}$. Note that necessarily $e(x') = f(e(x))$.
Using the charts $e_{(x,x+1)}^{-1}$ and $e_{(x',x'+1)}^{-1}$ for $S^1$, we see that
(2) $F \mid_{(x,x+1)} = e_{(x',x'+1)}^{-1} \circ f \mid_{e(x,x+1)} \circ  \phantom{.} e_{(x,x+1)} : (x,x+1) \to (x',x'+1)$ is a smooth bijection having everywhere a positive derivative since $f$ is an orientation preserving diffeomorphism. In particular, $F \mid_{(x,x+1)}$ is a strictly increasing smooth bijection.
(1), (2) and continuity show us
(3) $F(x) = x', F(x+1) = x'+1$
This implies
(4) $F(x+1) = F(x) + 1$ and $F((x,x+1)) = (F(x),F(x)+1)$.
This suffices to see that $f$ is bijective. Since $F$ has a positive derivative on each open interval of length $1$, it has a positive derivative on all of $\mathbb{R}$. Therefore
(5) $F$ is an orientation preserving diffeomorphism.
Final remark:
Although not needed her, let us consider an arbitrary continuous map $f : S^1 \to S^1$. As above we find $F : \mathbb{R} \to \mathbb{R}$ such that $e \circ F = f \circ e$. We infer $e(F(x+1)) = e(F(x))$ which is equivalent to $F(x+1) - F(x) \in \mathbb{Z}$. But now the function $g(x) = F(x+1) - F(x)$ is continuous on $\mathbb{R}$ with values in $ \mathbb{Z}$, and this is possible only if $g$ is constant. Therefore there exists $k \in \mathbb{Z}$ such that $F(x+1) = F(x) + k$ for all $x$. This shows that $F$ is completey determined by $F \mid_{[0,1]}$, but that's irrelevant here. We can define a homotopy $\Gamma : \mathbb{R} \times I \times \mathbb{R}, \Gamma(x,t) = (1-t)F(x) + tkx$. Then $\Gamma_0 = F$, $\Gamma_1(x) = kx$ and each $\Gamma_t$ has the property that $\Gamma_t(x+1) = \Gamma_t(x) + k$. Hence $\Gamma$ induces a unique homotopy $H : S^1 \times I \to S^1$ such that $H_0 = f$ and $H_1(z) = z^k$.
Comment: Yes. Being smooth is a local property, and $e$ is a local diffeomorphism.
Comment: @SachchidanandPrasad This is impossible in the range $t \in [0,1]$. $\Gamma'_t(x)$ is a convex combination of the two numbers $F'(x) >0$ and $1$. Alternatively look at your equation.  It has no solution $t$ if $F'(x) = 1$, a negative solution $t$ if $F'(x) < 1$ and a solution $t > 1$ if $F'(x) > 1$.
Comment: Let $B^A$  be the set of all continuous functions $A \to B$ endowed with the compact-open topology. Then the *exponential correspondence* establishes a bijection $e : Z^{X \times Y} \to (Z^Y)^X $, where $f' = e(f)$ is given by  $f'(x)(y) = f(x,y)$, provided $Y$ is locally compact. It is even a homeomorphism, but that is irrelevant here. In particular, for $X = I$ and locally compact $Y$ this yields a bijection between homotopies $H : I \times Y \to Z$ and paths $\omega : I \to Z^Y$. Apply this for $Y = Z = S^1$, noting that $\text{Diff}^+(S^1)$ is subspace of $(S^1)^{S^1}$.
Comment: If $A$ is compact and $B$ is a metric space, then the compact-open topology on $B^A$ agrees with the topology induced by the supremum-metric on the set $B^A$. This is the topology of *uniform convergence*.
Comment: Will you please tell me why $F$is an orientation preserving diffeomorphism and why $F(x+1)=F(x)+1$ ?
Comment: I added some material to my answer. I did it in great detail so that it is longer than my original answer.
Comment: If you understand the covering $e : \mathbb{R} \to S^1$, then all this is fairly obvious.
Comment: Using the charts..., that will be $e|_{(x.x+1)}$ not the inverse.
Comment: @SachchidanandPrasad A chart for an $n$-manifold $M$ is a homeomorphism $\varphi$ from an open subset $U \subset M$ to an open subset $V \subset \mathbb{R}^n$. For  $M = S^1$ we take $U = e((x,x+1)), V = (x,x+1)$ and $\varphi = e^{-1} \mid_{(x,x+1)}$.
Comment: Okay, sorry that was my mistake.
Comment: Will you please tell me why $\Gamma$ induces a unique homotopy?
Comment: $\Gamma$ has the property $\Gamma(x+1,t) = \Gamma(x,t) +1$. Therefore $\Gamma' = e \circ \Gamma$ satisfies $\Gamma'(x+1,t) =  \Gamma'(x,t)$. This means that it factors through the quotient map $e \times id_I : \mathbb{R} \times I \to S^1 \times I$ - we have $\Gamma' = H \circ (e \times id_I)$.
Comment: Why $\Gamma_t(x)>0$? I know it is $\ge 0$. Also,  $\Gamma_t(x)=0\iff F'(x)+t(1-F'(x))=0$, so there is no cintradiction that I am seeing.
Comment: @PaulFrost      Is it obvious that lift of a smooth map is smooth?
Comment: @SachchidanandPrasad We know that $F'(x) > 0$ for all $x$ because $F$ is an orientation preserving diffeomorphism (as a lift of an orientation preserving diffeomorphism on $S^1$). Hence $\Gamma'_t(x) = (1-t)F'(x) + t > 0$ for all $x,t$.
Comment: @PaulFrost  But what if $(F'(x)-1)t=F'(x)$? As, if $(F'(x)-1)t=F'(x)\implies (1-t)F'(x)-t=0$
Comment: @PaulFrost $t\mapsto H_t:S^1\to S^1$ is the path. But why this path is continuous in the Diff$^+(S^1)$, i.e. you have not used the topology of Diff$^+(S^1)$ which is compact-open topology or any other.


Jquery cycle - auto resizing images on changing browser window

Question: Afternoon all..
I have a site with an intro page including a jquery cycle slideshow of three images.
I have set up the CSS so that the images resize in accordance with the browser window.
This all works fine when the images are on their own, however when they are part of the slideshow I have to refresh the screen to make the images fit properly. They do actually still resize properly, just not automatically with a manual change in window size as they do when on their own.
The Url is - http://subzerostudio.com/Clients/ex1/index.html -
You can see that if you bring the window size down and leave it during the slideshow the images are the correct size, and the same if the window size is full, the images still fit - it is just when resizing the window they remain the previous height and width that was initially loaded.
They are also set up to be flexible in terms of position in order to fit different resolutions to a certain extent however this aspect is fine.
Once this is sorted it's ready to go out :) any ideas are much appreciated!
Chris
Comment: Check out the resize method of jQuery: http://api.jquery.com/resize/
Comment: Just add an answer to your own question instead.
Comment: Hi Baszz thanks for this! I've looked through and am not totally sure how to integrate this, can you point out how I'd get this talking to the slideshow?
Comment: I don't know the slideshow code so I cannot help you with that. I think you should implement the resize method and set some variables or something that are being used by the slideshow. Or maybe re-initialize the slideshow on resize...something like that.
Comment: Thanks a lot Baszz - I've found a solution in case anyone else gets stuck on this - http://stackoverflow.com/questions/2183869/problem-trying-to-do-window-resize-with-jquery-cycle-slideshow - this post gives a simple bit of JS and CSS which seems to have taken care of the problem very efficiently :)
Comment: Embarrasingly I can't work out how to go to a new line on these comments without pressing return (which posts the comment) otherwise I'd write out the solution here :)
Answer: In case anyone else gets stuck on this - stackoverflow.com/questions/2183869/… - this post gives a simple bit of JS and CSS which seems to have taken care of the problem very efficiently :)


Why is the Confidence Interval Changing for this Time-Series

Question: I have some R code (which I did not write) and which performs some state space analysis on some time-series. The data itself is shown as dots (scatter plot) and the Kalman filtered and smoothed state is the solid line.

My question is regarding the confidence intervals shown in this plot. I calculate my own confidence intervals using the standard method (my C# code is below)
public static double ConfidenceInterval(
    IEnumerable<double> samples, double interval)
{
    Contract.Requires(interval > 0 && interval < 1.0);

    double theta = (interval + 1.0) / 2;
    int sampleSize = samples.Count();
    double alpha = 1.0 - interval;
    double mean = samples.Mean();
    double sd = samples.StandardDeviation();

    var student = new StudentT(0, 1, samples.Count() - 1);
    double T = student.InverseCumulativeDistribution(theta);
    return T * (sd / Math.Sqrt(samples.Count()));
}

Now this will return a single interval (and it does it correctly) which I will add/subtract from each point on the series I have applied the calculation to to give me my confidence interval. But this is a constant and the R implementation seems to change over the time-series.
My question is why is the confidence interval changing for the R implementation? Should I be implementing my confidence levels/intervals differently?
Thanks for your time.

For reference the R code that produces this plot is below:
install.packages('KFAS')
require(KFAS)

# Example of local level model for Nile series
modelNile<-SSModel(Nile~SSMtrend(1,Q=list(matrix(NA))),H=matrix(NA))
modelNile
modelNile<-fitSSM(inits=c(log(var(Nile)),log(var(Nile))),model=modelNile,
method='BFGS',control=list(REPORT=1,trace=1))$model

# Can use different optimisation:
# should be one of "Nelder-Mead", "BFGS", "CG", "L-BFGS-B", "SANN", "Brent"
modelNile<-SSModel(Nile~SSMtrend(1,Q=list(matrix(NA))),H=matrix(NA))
modelNile
modelNile<-fitSSM(inits=c(log(var(Nile)),log(var(Nile))),model=modelNile,
method='L-BFGS-B',control=list(REPORT=1,trace=1))$model

# Filtering and state smoothing
out<-KFS(modelNile,filtering='state',smoothing='state')
out$model$H
out$model$Q
out

# Confidence and prediction intervals for the expected value and the observations.
# Note that predict uses original model object, not the output from KFS.
conf<-predict(modelNile,interval='confidence')
pred<-predict(modelNile,interval='prediction')
ts.plot(cbind(Nile,pred,conf[,-1]),col=c(1:2,3,3,4,4),
ylab='Predicted Annual flow', main='River Nile')
KFAS 13

# Missing observations, using same parameter estimates
y<-Nile
y[c(21:40,61:80)]<-NA
modelNile<-SSModel(y~SSMtrend(1,Q=list(modelNile$Q)),H=modelNile$H)
out<-KFS(modelNile,filtering='mean',smoothing='mean')

# Filtered and smoothed states
plot.ts(cbind(y,fitted(out,filtered=TRUE),fitted(out)), plot.type='single',
col=1:3, ylab='Predicted Annual flow', main='River Nile')

# Example of multivariate local level model with only one state
# Two series of average global temperature deviations for years 1880-1987
# See Shumway and Stoffer (2006), p. 327 for details
data(GlobalTemp)
model<-SSModel(GlobalTemp~SSMtrend(1,Q=NA,type='common'),H=matrix(NA,2,2))

# Estimating the variance parameters
inits<-chol(cov(GlobalTemp))[c(1,4,3)]
inits[1:2]<-log(inits[1:2])
fit<-fitSSM(inits=c(0.5*log(.1),inits),model=model,method='BFGS')
out<-KFS(fit$model)
ts.plot(cbind(model$y,coef(out)),col=1:3)
legend('bottomright',legend=c(colnames(GlobalTemp), 'Smoothed signal'), col=1:3, lty=1)

# Seatbelts data
## Not run:
model<-SSModel(log(drivers)~SSMtrend(1,Q=list(NA))+
SSMseasonal(period=12,sea.type='trigonometric',Q=NA)+
log(PetrolPrice)+law,data=Seatbelts,H=NA)

# As trigonometric seasonal contains several disturbances which are all
# identically distributed, default behaviour of fitSSM is not enough,
# as we have constrained Q. We can either provide our own
# model updating function with fitSSM, or just use optim directly:
# option 1:
ownupdatefn<-function(pars,model,...){
model$H[]<-exp(pars[1])
diag(model$Q[,,1])<-exp(c(pars[2],rep(pars[3],11)))
model #for option 2, replace this with -logLik(model) and call optim directly
}
14 KFAS
fit<-fitSSM(inits=log(c(var(log(Seatbelts[,'drivers'])),0.001,0.0001)),
model=model,updatefn=ownupdatefn,method='BFGS')
out<-KFS(fit$model,smoothing=c('state','mean'))
out
ts.plot(cbind(out$model$y,fitted(out)),lty=1:2,col=1:2,
main='Observations and smoothed signal with and without seasonal component')
lines(signal(out,states=c("regression","trend"))$signal,col=4,lty=1)
legend('bottomleft',
legend=c('Observations', 'Smoothed signal','Smoothed level'),
col=c(1,2,4), lty=c(1,2,1))

# Multivariate model with constant seasonal pattern,
# using the the seat belt law dummy only for the front seat passangers,
# and restricting the rank of the level component by using custom component
# note the small inconvinience in regression component,
# you must remove the intercept from the additional regression parts manually
model<-SSModel(log(cbind(front,rear))~ -1 + log(PetrolPrice) + log(kms)
+ SSMregression(~-1+law,data=Seatbelts,index=1)
+ SSMcustom(Z=diag(2),T=diag(2),R=matrix(1,2,1),
Q=matrix(1),P1inf=diag(2))
+ SSMseasonal(period=12,sea.type='trigonometric'),
data=Seatbelts,H=matrix(NA,2,2))
likfn<-function(pars,model,estimate=TRUE){
model$H[,,1]<-exp(0.5*pars[1:2])
model$H[1,2,1]<-model$H[2,1,1]<-tanh(pars[3])*prod(sqrt(exp(0.5*pars[1:2])))
model$R[28:29]<-exp(pars[4:5])
if(estimate) return(-logLik(model))
model
}
fit<-optim(f=likfn,p=c(-7,-7,1,-1,-3),method='BFGS',model=model)
model<-likfn(fit$p,model,estimate=FALSE)
model$R[28:29,,1]%*%t(model$R[28:29,,1])
model$H
out<-KFS(model)
out
ts.plot(cbind(signal(out,states=c('custom','regression'))$signal,model$y),col=1:4)

# For confidence or prediction intervals, use predict on the original model
pred <- predict(model,states=c('custom','regression'),interval='prediction')
ts.plot(pred$front,pred$rear,model$y,col=c(1,2,2,3,4,4,5,6),lty=c(1,2,2,1,2,2,1,1))

## End(Not run)
## Not run:
# Poisson model
model<-SSModel(VanKilled~law+SSMtrend(1,Q=list(matrix(NA)))+
SSMseasonal(period=12,sea.type='dummy',Q=NA),
KFAS 15
data=Seatbelts, distribution='poisson')

# Estimate variance parameters
fit<-fitSSM(inits=c(-4,-7,2), model=model,method='BFGS')
model<-fit$model

# use approximating model, gives posterior mode of the signal and the linear predictor
out_nosim<-KFS(model,smoothing=c('signal','mean'),nsim=0)

# State smoothing via importance sampling
out_sim<-KFS(model,smoothing=c('signal','mean'),nsim=1000)
out_nosim
out_sim

## End(Not run)
# Example of generalized linear modelling with KFS
# Same example as in ?glm
counts <- c(18,17,15,20,10,20,25,13,12)
outcome <- gl(3,1,9)
treatment <- gl(3,3)
print(d.AD <- data.frame(treatment, outcome, counts))
glm.D93 <- glm(counts ~ outcome + treatment, family = poisson())
model<-SSModel(counts ~ outcome + treatment, data=d.AD,
distribution = 'poisson')
out<-KFS(model)
coef(out,start=1,end=1)
coef(glm.D93)
summary(glm.D93)$cov.s
out$V[,,1]
outnosim<-KFS(model,smoothing=c('state','signal','mean'))
set.seed(1)
outsim<-KFS(model,smoothing=c('state','signal','mean'),nsim=1000)

## linear
# GLM
glm.D93$linear.predictor

# approximate model, this is the posterior mode of p(theta|y)
c(outnosim$thetahat)

# importance sampling on theta, gives E(theta|y)
c(outsim$thetahat)

## predictions on response scale
16 KFAS

# GLM
fitted(glm.D93)

# approximate model with backtransform, equals GLM
c(fitted(outnosim))

# importance sampling on exp(theta)
fitted(outsim)

# prediction variances on link scale
# GLM
as.numeric(predict(glm.D93,type='link',se.fit=TRUE)$se.fit^2)

# approx, equals to GLM results
c(outnosim$V_theta)
# importance sampling on theta
c(outsim$V_theta)
# prediction variances on response scale
# GLM
as.numeric(predict(glm.D93,type='response',se.fit=TRUE)$se.fit^2)
# approx, equals to GLM results
c(outnosim$V_mu)
# importance sampling on theta
c(outsim$V_mu)
## Not run:
data(sexratio)
model<-SSModel(Male~SSMtrend(1,Q=list(NA)),u=sexratio[,'Total'],data=sexratio,
distribution='binomial')
fit<-fitSSM(model,inits=-15,method='BFGS',control=list(trace=1,REPORT=1))
fit$model$Q #1.107652e-06

# Computing confidence intervals in response scale
# Uses importance sampling on response scale (4000 samples with antithetics)
pred<-predict(fit$model,type='response',interval='conf',nsim=1000)
ts.plot(cbind(model$y/model$u,pred),col=c(1,2,3,3),lty=c(1,1,2,2))

# Now with sex ratio instead of the probabilities:
imp<-importanceSSM(fit$model,nsim=1000,antithetics=TRUE)
sexratio.smooth<-numeric(length(model$y))
sexratio.ci<-matrix(0,length(model$y),2)
w<-imp$w/sum(imp$w)
for(i in 1:length(model$y)){
sexr<-exp(imp$sample[i,1,])
sexratio.smooth[i]<-sum(sexr*w)
oo<-order(sexr)
sexratio.ci[i,]<-c(sexr[oo][which.min(abs(cumsum(w[oo]) - 0.05))],
+ sexr[oo][which.min(abs(cumsum(w[oo]) - 0.95))])
}

# Same by direct transformation:
out<-KFS(fit$model,smoothing='signal',nsim=1000)
KFS 17
sexratio.smooth2 <- exp(out$thetahat)
sexratio.ci2<-exp(c(out$thetahat)
+ qnorm(0.025) * sqrt(drop(out$V_theta))%o%c(1, -1))
ts.plot(cbind(sexratio.smooth,sexratio.ci,sexratio.smooth2,sexratio.ci2),
col=c(1,1,1,2,2,2),lty=c(1,2,2,1,2,2))

## End(Not run)
# Example of Cubic spline smoothing
## Not run:
require(MASS)
data(mcycle)
model<-SSModel(accel~-1+SSMcustom(Z=matrix(c(1,0),1,2),
T=array(diag(2),c(2,2,nrow(mcycle))),
Q=array(0,c(2,2,nrow(mcycle))),
P1inf=diag(2),P1=diag(0,2)),data=mcycle)
model$T[1,2,]<-c(diff(mcycle$times),1)
model$Q[1,1,]<-c(diff(mcycle$times),1)^3/3
model$Q[1,2,]<-model$Q[2,1,]<-c(diff(mcycle$times),1)^2/2
model$Q[2,2,]<-c(diff(mcycle$times),1)
updatefn<-function(pars,model,...){
model$H[]<-exp(pars[1])
model$Q[]<-model$Q[]*exp(pars[2])
model
}
fit<-fitSSM(model,inits=c(4,4),updatefn=updatefn,method="BFGS")
pred<-predict(fit$model,interval="conf",level=0.95)
plot(x=mcycle$times,y=mcycle$accel,pch=19)
lines(x=mcycle$times,y=pred[,1])
lines(x=mcycle$times,y=pred[,2],lty=2)
lines(x=mcycle$times,y=pred[,3],lty=2)
## End(Not run)

The time-series data is:
Time, 2.4, 2.6, 3.2, 3.6, 4, 6.2, 6.6, 6.8, 7.8, 8.2, 8.8, 8.8, 9.6, 10, 10.2, 10.6, 11, 11.4, 13.2, 13.6, 13.8, 14.6, 14.6, 14.6, 14.6, 14.6, 14.6, 14.8, 15.4, 15.4, 15.4, 15.4, 15.6, 15.6, 15.8, 15.8, 16, 16, 16.2, 16.2, 16.2, 16.4, 16.4, 16.6, 16.8, 16.8, 16.8, 17.6, 17.6, 17.6, 17.6, 17.8, 17.8, 18.6, 18.6, 19.2, 19.4, 19.4, 19.6, 20.2, 20.4, 21.2, 21.4, 21.8, 22, 23.2, 23.4, 24, 24.2, 24.2, 24.6, 25, 25, 25.4, 25.4, 25.6, 26, 26.2, 26.2, 26.4, 27, 27.2, 27.2, 27.2, 27.6, 28.2, 28.4, 28.4, 28.6, 29.4, 30.2, 31, 31.2, 32, 32, 32.8, 33.4, 33.8, 34.4, 34.8, 35.2, 35.2, 35.4, 35.6, 35.6, 36.2, 36.2, 38, 38, 39.2, 39.4, 40, 40.4, 41.6, 41.6, 42.4, 42.8, 42.8, 43, 44, 44.4, 45, 46.6, 47.8, 47.8, 48.8, 50.6, 52, 53.2, 55, 55, 55.4, 57.6
mcycle, 0, -1.3, -2.7, 0, -2.7, -2.7, -2.7, -1.3, -2.7, -2.7, -1.3, -2.7, -2.7, -2.7, -5.4, -2.7, -5.4, 0, -2.7, -2.7, 0, -13.3, -5.4, -5.4, -9.3, -16, -22.8, -2.7, -22.8, -32.1, -53.5, -54.9, -40.2, -21.5, -21.5, -50.8, -42.9, -26.8, -21.5, -50.8, -61.7, -5.4, -80.4, -59, -71, -91.1, -77.7, -37.5, -85.6, -123.1, -101.9, -99.1, -104.4, -112.5, -50.8, -123.1, -85.6, -72.3, -127.2, -123.1, -117.9, -134, -101.9, -108.4, -123.1, -123.1, -128.5, -112.5, -95.1, -81.8, -53.5, -64.4, -57.6, -72.3, -44.3, -26.8, -5.4, -107.1, -21.5, -65.6, -16, -45.6, -24.2, 9.5, 4, 12, -21.5, 37.5, 46.9, -17.4, 36.2, 75, 8.1, 54.9, 48.2, 46.9, 16, 45.6, 1.3, 75, -16, -54.9, 69.6, 34.8, 32.1, -37.5, 22.8, 46.9, 10.7, 5.4, -1.3, -21.5, -13.3, 30.8, -10.7, 29.4, 0, -10.7, 14.7, -1.3, 0, 10.7, 10.7, -26.8, -14.7, -13.3, 0, 10.7, -14.7, -2.7, 10.7, -2.7, 10.7

Comment: Your "confidence interval" procedure is not recognizable as a "standard method" for regression. It would seem you are trying to co-opt a procedure (Student t confidence limits for univariate data) and apply it where it is inappropriate.  Perhaps the [Wikipedia article on regression](http://en.wikipedia.org/wiki/Simple_linear_regression#Confidence_intervals) would help clarify the distinction for you.
Comment: (+1) 1) I have added the data and the time points. These data are recorded at irregular time periods, so the times at which they are recorded is relevant. I posted it in commas-separated format since someone may want to load the data in some other software instead of getting it from the MASS package. 2) I think you can stick to show the C# code chunk and the last chunk of R code that reproduces the plot. The rest of the code is not even related to this data set. If it is relevant to this question, please explain.
Comment: I am not sure I understand you. I have some series of observations and the formula I have implemented for these should provide the Confidence Interval for the specified level (95% or whatever). Why am I interested in regression here? Sorry I am not a statistician, thanks very much for your time...
Answer: These are irregularly spaced data, there are even more than one observations at
a certain time points (e.g. 2 observations at times 8.8, 15.6, 15.8, 6 observations at time 14.6, 4 observations at time 15.4). These are the number of observations at each time:
table(mcycle[,1])
#  2.4  2.6  3.2  3.6    4  6.2  6.6  6.8  7.8  8.2  8.8  9.6   10 10.2 10.6   11
#    1    1    1    1    1    1    1    1    1    1    2    1    1    1    1    1
# 11.4 13.2 13.6 13.8 14.6 14.8 15.4 15.6 15.8   16 16.2 16.4 16.6 16.8 17.6 17.8
#    1    1    1    1    6    1    4    2    2    2    3    2    1    3    4    2
# 18.6 19.2 19.4 19.6 20.2 20.4 21.2 21.4 21.8   22 23.2 23.4   24 24.2 24.6   25
#    2    1    2    1    1    1    1    1    1    1    1    1    1    2    1    2
# 25.4 25.6   26 26.2 26.4   27 27.2 27.6 28.2 28.4 28.6 29.4 30.2   31 31.2   32
#    2    1    1    2    1    1    3    1    1    2    1    1    1    1    1    2
# 32.8 33.4 33.8 34.4 34.8 35.2 35.4 35.6 36.2   38 39.2 39.4   40 40.4 41.6 42.4
#    1    1    1    1    1    2    1    2    2    2    1    1    1    1    2    1
# 42.8   43   44 44.4   45 46.6 47.8 48.8 50.6   52 53.2   55 55.4 57.6
#    2    1    1    1    1    1    2    1    1    1    1    2    1    1

The asymmetry of the confidence interval may reflect different degrees of uncertainty depending on the amount of information around a time point. We may expect narrower confidence intervals at periods where more data are available, and vice versa. The following plot suggests that at periods with higher concentration of observations (e.g. around times 14-15 and 26-27) the confidence interval is narrower.
plot(mcycle$times, pred[,3] - pred[,2])
mtext(side = 3, text = "width of confidence interval in the plot posted by the OP", adj = 0)

Also, be aware that even with evenly spaced data we would observe asymmetric
confidence intervals at the beginning and the end of the sample. This is due to
the initialization of the Kalman filter (forwards recursions) which typically involves a large variance attached to the initial state vector and the initialization of the Kalman smoother, which is usually initialized with zeros at the end of the sample (backwards recursions). The width of the confidence interval converges to a fixed width.
For illustration, we can take this example where the local-level plus seasonal component model is fitted to a series recorded regularly spaced times. As mentioned before, the confidence interval is symmetric except at the beginning and the end of the sample.
require("stsm")
data("llmseas")
m <- stsm.model(model = "llm+seas", y = llmseas)
res <- maxlik.fd.scoring(m = m, step = NULL,
  information = "expected", control = list(maxit = 100, tol = 0.001))
comps <- tsSmooth(res)
sse1 <- comps$states[,1] + 1.96 * comps$sse[,1]
sse2 <- comps$states[,1] - 1.96 * comps$sse[,1]
par(mfrow = c(2, 1), mar = c(3,3,3,3))
plot(ts.union(comps$states[,1], sse1, sse2), ylab = "", plot.type = "single", type = "n")
polygon(x = c(time(comps$states), rev(time(comps$states))), y = c(sse2, rev(sse1)), col = "lightgray", border = NA)
lines(comps$states[,1])
mtext(side = 3, text = "fitted level and 95% confidence interval", adj = 0)
plot(sse2 - sse1, plot.type = "single", ylab = "", type = "b", lty = 2, pch = 16)
mtext(side = 1, line = 2, text = "Time")
mtext(side = 3, text = "width of confidence interval in the top plot", adj = 0)

Comment: See the functions `dnorm`, `pnorm` and `qnorm` for the density, distribution function and quantile function for the normal distribution. For questions not related to the original post please make a new post, if the question is related to the usage of R functions you may post it at [Stack Overflow](http://stackoverflow.com/).
Comment: Hi, I have asked a new question here. It is a simple one, but getting the information is proving difficult... http://stats.stackexchange.com/questions/131482/how-to-map-desired-confidence-interval-to-a-quantile-value
Comment: This is fantastic, thanks for the clarification. I did not realize that the data was irregularly spaced so this make much more sense now! I appreciate that the larger error variances at the start and end of the state series (the state output from the Kalman filter and the smoother state) would lead to larger confidence intervals, but how are these intervals calculated? Specifically, can you provide a reference (or anything) that will show me how to implement the calculation of these varying confidence intervals?
Comment: Clearly the implementation for the Confidence Intervals I have above is not correct for such cases where the Kalman Filter Smoother is adopted. But what is the correct formula/algo. to use for the calculation of such intervals? Can the same formula take account of the case of irregularly spaced data (as in the case of the motorcycle data above)? If the same formula that is used to calculate the confidence intervals for the Kalman Filtered state cannot be used for the irregularly spaced data, what other formula can?
Comment: I really appreciate your time in helping me here, if you can expand your answer to encapsulate the question on the formulation of the calculation of the confidence intervals I will award the biggest bounty I can to you (which isn't much at this point!). Again, thanks.
Comment: Once you have your model written in state space form and given a set of parameters, the state smoothing recursions provide an estimate of the state vector given the observed data. The variance of the smoothed vector can also be obtained recursively. Upon these elements, it is straightforward to compute confidence intervals as the smoothed state vector plus and minus two times the corresponding standard deviations at each time point. This is what I did in the example above for the local level plus seasonal model.
Comment: To get an idea of how equations the smoother look like, you may see Appendix A in [this document](http://www.jalobe.com:8080/doc/stsm-em.pdf) available online. But I recommend you to study some textbook on this issue, e.g. Durbin and Koopman (2001) _Time Series Analysis by State Space Methods_ or Harvey (1989) _Forecasting, Structural Time Series Models and the Kalman Filter_. Chapter 2 of the first reference is a good place to start, the Kalman filter and smoother is derived for the local-level model, which is easier to handle as the state space form involves scalars instead of matrices.
Comment: Thank you. I have a Kalman Filter and Disturbance Smoother implemented ("filter/smoother"). This is done for the most generic state space model and can handle N-Dimensional state vectors. My problem come my understanding of the more simple statistical methods which are assumed to be known, whereas the more complex stuff is explicitly stated [hence I can implement]. Because I do not have a statistics background, easy things like confidence interval calculations for such state space models are unknown to me. Why is it 2x the stddev at each time point and what interval is given by 2x stddev?
Comment: When you say "smoothed state vector plus and minus two times the corresponding standard deviations at each time point" do you mean the stddev of the smoothed state series? Why have you use a factor of 1.96 in your R code above? Thanks very much for your time, it is most appreciated. I have just realized that you have updated your answer, please forgive me if you have already answered my questions...
Comment: I meant the _standard errors_ of the smoothed state series, which are obtained by the recursions of the smoother. I used the factor $\pm1.96$ to obtain the 95% confidence interval because it is the quantile 0.975 of the Gaussian distribution, the distribution assumed for the innovation terms in the model that I used as example. In practice there shouldn't be much difference rounding the factor to 2.
Comment: Thanks again, very helpful. I understand I can use the Inverse Cumulative Distribution function (quantile function) at some value q to get the q^(th) quantile of a distribution. But when I do this for the Normal distribution and calculate InvCDF(0.975) I get the value ~ 1.96; and for InvCDF(0.95) ~ 1.65, this is part way there... My question now is, how can I map my probability input values (i.e. the 0.975 and 0.95 which I input to the InvCDF) to my desired confidence intervals without the need for table lookup? I apologies for the basic questions here, but Google is failing me...


Can I transform the Viterbi score from tensorflow into a probability?

Question: I understand the Viterbi algorithm as it is explained in Wikipedia
However, the TensorFlow implementation is different:
  def viterbi_decode(score, transition_params):
  """Decode the highest scoring sequence of tags outside of TensorFlow.
  This should only be used at test time.
  Args:
    score: A [seq_len, num_tags] matrix of unary potentials.
    transition_params: A [num_tags, num_tags] matrix of binary potentials.
  Returns:
    viterbi: A [seq_len] list of integers containing the highest scoring tag
        indices.
    viterbi_score: A float containing the score for the Viterbi sequence.
  """
  trellis = np.zeros_like(score)
  backpointers = np.zeros_like(score, dtype=np.int32)
  trellis[0] = score[0]

  for t in range(1, score.shape[0]):
    v = np.expand_dims(trellis[t - 1], 1) + transition_params
    trellis[t] = score[t] + np.max(v, 0)
    backpointers[t] = np.argmax(v, 0)

  viterbi = [np.argmax(trellis[-1])]
  for bp in reversed(backpointers[1:]):
    viterbi.append(bp[viterbi[-1]])
  viterbi.reverse()

  viterbi_score = np.max(trellis[-1])
  return viterbi, viterbi_score

In the "traditional" Viterbi algorithm we basically keep multiplying probabilities and take the maximum. But the Viterbi score of Tensorflow is consistently greater than 1 and increases with sequence length.

Where is the Tensorflow implementation coming from?
How can the Viterbi score retrieved by Tensorflow be consistently >1 and increase with the sequence length?
Is there a way of transforming the TensorFlow Viterbi score into a probability? I want to compare the confidence of several sequence predictions in a Deep Learning architecture whose final layer is a CRF.
Comment: Sorry I dropped on this for a couple of months. Your answer is excellent. I learnt a lot! I appreciate it.
Answer: This is doable, and fortunately the code to do it is already written.
Bear in mind that the Viterbi algorithm on a CRF uses potentials, not probabilities. These are not required to sum to 1. The Wikipedia article's language is tailored to Viterbi on an HMM, so it keeps talking about probabilities.

Where is the Tensorflow implementation coming from?

It's doing the familiar Viterbi algorithm, but working with log-potentials instead of potentials. Normally the two key operations in the Viterbi algorithm are $(\max, \times)$. In log-space, these become $(\max, +)$.

How can the Viterbi score retrieved by Tensorflow be consistently >1 and increase with the sequence length?

This ties back to the log-space idea. If we were working with probabilities, they would be consistently < 1 and decrease with the sequence length. But log-potentials can be any real number, and they keep getting added together. It seems that yours are non-negative.

Is there a way of transforming the TensorFlow Viterbi score into a probability?

Yes, certainly! You'll have to do a bit of extra work to get it, using the crf_log_norm function.
Remember that we use the Viterbi algorithm to find the highest-probability sequence $y^* = \arg\min_{y} p(y \mid x)$. But it does this by finding the sequence with the highest unnormalized probability $u(y \mid x)$—the 'Viterbi score' you got is the log of this. The relationship is that $p(y \mid x) = \frac{u(y \mid x)}{Z(x)}$. The Viterbi algorithm doesn't bother computing the normalizing constant $Z(x) = \sum_{y'} u(y' \mid x)$.
You're interested in $p(y^* \mid x)$. You already have $\log u(y^* \mid x)$. And there's a function in that library called crf_log_norm to compute $\log Z(x)$. All that remains is to compute the difference and then exponentiate.
\begin{align}
\log p(y^* \mid x) &= \log u(y^* \mid x) - \log Z(x) \\
p(y^* \mid x) &= \frac{u(y^* \mid x)}{Z(x)}
\end{align}

Just a reminder, though, computing $Z(x)$ with the forward algorithm is somewhat expensive: $O(nk^2)$ for $n$ tokens and $k$ possible states at each position. If you're comparing multiple possible taggings for one sequence, try to only compute $Z(x)$ once.


Criar uma variável global no Delphi

Question: Preciso executar o seguinte código, porém o formulário em questão nem sempre é UniForm1. Então penso em armazenar o nome do form numa variável e substituir o UniForm1 por ela.
Em vez de usar:
UniForm1.Parent := UniPanel1;
UniForm1.Show;
UniForm1.SetFocus;

UniForm1.Width := UniPanel1.Width;
UniForm1.Height := UniPanel1.Height;

Preciso de algo como:
VariavelForm.Parent := UniPanel1;
VariavelForm.Show;
VariavelForm.SetFocus;

VariavelForm.Width := UniPanel1.Width;
VariavelForm.Height := UniPanel1.Height;

Como fazer isso?
Conforme a resposta de @Tmc, fiz algumas alterações.
Porém ocorre o erro:

[dcc32 Error] Main.pas(74): E2010 Incompatible types: 'TUniForm' and
  'string'

procedure TMainForm.UniTreeView1Change(Sender: TObject; Node: TUniTreeNode);
var nome : string;
begin
   nome := Node.Text;

   Vforms := nome;

   Vforms.Parent := UniPanel1;
   Vforms.Show;
   Vforms.SetFocus;

   Vforms.Width := UniPanel1.Width;
   Vforms.Height := UniPanel1.Height;
end;

Answer: Pode fazer da seguinte maneira:
unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms, Dialogs, StdCtrls;

type
  TForm1 = class(TForm)
    Edit1: TEdit;
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
    procedure FormCreate(Sender: TObject);
  private
    procedure ChangeSetForms;
  public
    { Public declarations }
  end;

var
  Form1: TForm1;
    //declarar variavel global
    Vforms: TComponent;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
begin
  Edit1.Text := 'Form1';
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  // atribui valor de form a variavel
  Vforms := Application.FindComponent(Edit1.Text);

  //chama a procedure
  ChangeSetForms;
end;

procedure TForm1.ChangeSetForms;
Begin
  //faz as alterãções no form escolhido
  TForm(VForms).Parent := UniPanel1;
  TForm(VForms).Show;
  TForm(VForms).SetFocus;

  TForm(VForms).Width := UniPanel1.Width;
  TForm(VForms).Height := UniPanel1.Height;
End;

end.

Tentei detalhar todos os passos mas alguma dúvida estou disponível.
Outra hipótese é não criar uma variável chamando o procedure e passando a variável por ele, deixo também o exemplo:
procedure TForm1.FormCreate(Sender: TObject);
begin
  Edit1.Text := 'Form1';
end;

procedure TForm1.Button1Click(Sender: TObject);
begin
  //chama a procedure
  ChangeSetForms(Application.FindComponent(Edit1.Text));
end;

procedure TForm1.ChangeSetForms(VForm: TComponent);
Begin
  TForm(VForm).Width := 500;
End;

Comment: Por favor evitem longas discussões nos comentários; a conversa de vocês foi [movida para o chat](http://chat.stackexchange.com/rooms/66127/discussion-on-answer-by-tmc-criar-uma-variavel-global-no-delphi)

Criar uma variável global no Delphi


How to write the permission of a instance's user is the request.user?

Question: How can I write the IsThePhysicalServerUser?
you see my UpdateAPIView:
class PhysicalServerCustomerUpdateAPIView(UpdateAPIView):
    """
    Customer's update PhysicalServer
    """
    serializer_class = CustomerUpdatePhysicalServerSerializer
    permission_classes = [IsThePhysicalServerUser]
    queryset = PhysicalServer.objects.all()

There I want to write a permission which is IsThePhysicalServerUser.
the PhysicalServer model is bellow:
class PhysicalServer(models.Model):
    """
    实体服务器
    """
    name = models.CharField(max_length=32)
    desc = models.CharField(max_length=256, null=True, blank=True)
    ...
    user = models.ForeignKey(to=User, related_name="physical_servers", null=True, blank=True, on_delete=models.SET_NULL)

I want the IsThePhysicalServerUser permission realize the PhysicalServer instance's use is the request user. How to write the permission?
Answer: Just check the instance's user equals request.user:
class IsSuperAdminOrPhysicalServerCustomer(permissions.BasePermission):
    """
    SuperUser or instance.user equals request.user
    """
    def has_object_permission(self, request, view, obj):
        # if you want everyone can see user info
        if request.method in permissions.SAFE_METHODS:
            return True
        # if you use Django2.0 is_authenticated(request.user) should be changed to request.user.is_authenticated
        if request.user and request.user.is_authenticated:
            # instance.user equal self or is superuser
            return obj.user == request.user or request.user.is_superuser
        else:
            return False


Does the developer payload persist when RESTORE_TRANSACTIONS is called in Google in-app billing?

Question: We are trying to find a way to identify specific purchases when using Google in-app billing via Google Play—specifically, to identify a specific purchased product on a new device, or after a memory wipe.  I've considered trying to retrieve the Google ID used in purchase, as in this question, but it's proven problematic.
However, the developer payload might provide a solution, but this depends on how it behaves.  Basically what I am hoping is that, when a purchase is restored using RESTORE_TRANSACTIONS, the developer payload put into the original purchase request is returned.  If so we can use this payload to identify a user: when they install the application on a new device, we restore transactions, and get the developer payload to identify them; otherwise we assume they are a new user and generate a new developer payload.
Can this work?  Does the developer payload persist for managed products?
Answer: a late reply :)
I just tried in-app billing, and i can confirm you that the developer payload is returned also with RESTORE_TRANSACTION command.
I think your solution should work. Hope this late reply help you.
Comment: Thanks for your helpful answer.


About a representation on cardinals

Question: If k is a limit cardinal, necessarly one have that  $ ZFC ^  {R_{K}}  $ ?
And, if k is only ordinal, does it have any sense?
Answer: If $\kappa$ is not a strong limit cardinal, then the answer is clearly not. Because in that case there is some $\alpha<\kappa$ such that $R_\alpha\geq\kappa$, in which case $R_\kappa\models R_\alpha\text{ cannot be put in bijection with an ordinal}$.
If $\kappa$ is a strong limit cardinal, but not a fixed point of the $\beth$ function then $\kappa=\beth_\alpha$ and $\alpha<\kappa$. In that case we have a definable function whose domain is $\alpha$, but its range is unbounded in the ordinals of $R_\kappa$. So replacement fails.
If $\kappa$ is not a limit cardinal (ordinal or otherwise), then $R_\kappa$ thinks that there is a maximal cardinal, so it doesn't satisfy $\sf ZFC$.
More specifically if $\sf ZFC$ can prove $\kappa$ with a certain property exists, then it doesn't prove that $R_\kappa$ necessarily satisfies $\sf ZFC$. This is true for limit ordinals, cardinals, limit cardinals, strong limit cardinals, and so on. If $\sf ZFC$ can prove it exists, it can't prove that $R_\kappa$ is a model of $\sf ZFC$.
Comment: The general statement in your last paragraph is not quite true.  Let $\varphi(\kappa)$ say "$\kappa$ is the least inaccessible cardinal if it exists, and 0 otherwise."  Then $\mathsf{ZFC} \vdash \exists \kappa\,\varphi(\kappa)$, but (assuming the consistency of inaccessibles) $\mathsf{ZFC} \not\vdash \exists \kappa \,(\varphi(\kappa) \wedge R_\kappa \not\vDash \mathsf{ZFC})$.
Comment: We need to coin a term for properties which avoid this smartassery. I'll fix this later, thanks :-)
Comment: I think it works for $\Sigma_2$ statements.  Alternatively, you could weaken the conclusion to "it _can't_ prove that $R_\kappa$ _does_ have to be a model of $\mathsf{ZFC}$."

About a representation on cardinals


PROBLEMA AO CLICAR COM UM ROBÔ USANDO PUPPETEER

Question: Eu estou criando este robô para automatizar algumas tarefas do eu dia a dia, eu peço para ele acessar o site, clicar no input, e depois logar. Ele acessa a pagina, Mas ele não está clicando, não consigo achar o problema. este é o código

const puppeteer = require('puppeteer');

(async () =>{
    const browser = await puppeteer.launch({ headless: false });
    const site = 'http://www.goolsystem.com.br';
    const page = await browser.newPage();
    await page.goto('http://www.goolsystem.com.br/4/Login.aspx?ReturnUrl=%2f4');
    await page.waitFor('#ucTrocarModulo_btnIconeUrbano > input');
    await page.click('#ucTrocarModulo_btnIconeUrbano > input');
    await page.type('input[name="ucLogarUsuario$txtLogin"]','@@@@@', {delay: 100});
    await page.type('input[name="ucLogarUsuario$txtSenha"]', '@@@@@', {delay: 100});
    await page.keyboard.press('Enter');

    await browser.close();
})();

quando eu executo mostra o seguinte erro: "(node:5920) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:5920) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will
terminate the Node.js process with a non-zero exit code."
Answer: O seletor do input que deseja clicar esta errado, o id do seletor pai é ucTrocarModulo_moduloUrbano e você esta utilizando o id do seletor filho que é ucTrocarModulo_btnIconeUrbano (o input) e tentando buscar um input dentro do mesmo, ou seja, você esta tentando clicar um input dentro do input que você quer clicar, como não existe ele apresenta o erro.
Opções corretas:
1 - #ucTrocarModulo_moduloUrbano > input
2 - #ucTrocarModulo_moduloUrbano #ucTrocarModulo_btnIconeUrbano
3 - #ucTrocarModulo_btnIconeUrbano
Você pode utilizar o ID do elemento diretamente (como na opção 3). Tente isso:
const puppeteer = require('puppeteer');

(async () =>{
    const browser = await puppeteer.launch({ headless: false });
    const site = 'http://www.goolsystem.com.br';
    const page = await browser.newPage();
    await page.goto('http://www.goolsystem.com.br/4/Login.aspx?ReturnUrl=%2f4');
    await page.waitFor('#ucTrocarModulo_btnIconeUrbano'); // ID do input
    await page.click('#ucTrocarModulo_btnIconeUrbano'); // ID do input
    await page.waitFor('#ucLogarUsuario_txtLogin');
    await page.type('#ucLogarUsuario_txtLogin','@@@@@', {delay: 100}); // ID do input Login
    await page.type('#ucLogarUsuario_txtSenha', '@@@@@', {delay: 100}); // ID do senha
    await page.keyboard.press('Enter');

    await browser.close();
})();


How to properly authorize nested microservice calls

Question: I want to understand the best practice for authorizing nested service calls.  The scenario is a client calls some service A to perform some operation.  Service A then needs to call another service B on behalf of the user, so it calls service B.  Service A then calls another service C.  What I want is:

service B to be able to authenticate and authorize the request made by service A
service C to be able to authenticate and authorize the request made by service A
service C to be able to prevent it being called by service B on the user's behalf

Assuming I want to use OpenID Connect tokens, what is the appropriate flow and token content to support the above 3 requirements?
Answer: Good question and the answer is related to areas of data and OAuth scopes.
Access tokens are designed to be passed around between microservices and you should not (by default) need to run new OAuth flows every time APIs call each other, which adds complexity, leading to multiple problems.
Instead each API should validate the JWT and check scopes and claims that makes sense to that API's area of data.
A good design is to ensure the following:

Tokens used by internet clients are confidential
Tokens used by APIs can contain rich claims and be forwarded

See these 2 articles for some further info:

Scope Best Practices
Phantom Token Pattern for Confidential Tokens

On your point 3 there is a privilege there and it depends on the scenario - eg it could be a different subdivision of your company - in which case a different token might make sense.
Most dynamic authorization should use claims though - there is a Claims Best Practices article following on from the scopes article above.


How to use effects loop of the amp if your mfx has only input and output... No send and return unlike boss me

Question: My mfx is zoom g1xon... I just wonder how to bypass preamp and use my built in amp simulation..
It only has input and output. If i send it directly to d input it sounds terrible.
Comment: See if you can disable the **speaker** simulation on your effects unit. Usually the speaker simulation sound that is sent through a real speaker sounds very bad.
Answer: Without an fx loop in your amp you cannot bypass the preamp and so the main input is the only place you can plug the pedal into. Having said that you can still use the amp models if you want to. If you set your amp to completely clean with a flat Eq, you could use the zoom like it is running into the Fx return. Otherwise you could just use the effects on the zoom with your amp.
Answer: The G1Xon is designed to go between guitar and amplifier.   In what way does it sound terrible?  Try setting each effect in the patch to Off.  Does it now sound as if it isn't there at all (it should!).   Maybe you're just trying to make it add too many effects, or ones you didn't really want.  It CAN be a 'multi' effect.  It doesn't HAVE to be!


Paginação Dinâmica em tabela com AngularJs

Question: Tenho o seguinte cenário:

Recupero esses dados de uma API Rest com Vraptor. Gostaria de exibir apenas dois itens por página, porém não sei por onde começar. Implementei essa paginação utilizando o ui-bootstrap.
Como faço para fazer uma consulta dinâmica e exibir apenas dois itens por página?
Back-end:
Controller:
@Get
@Path("/colaboradores")
public void listarTodos() {
        result.use(Results.json())
        .withoutRoot()
        .from(colaboradorDAO.listarColaboradores())
        .serialize();
}

DAO:
@SuppressWarnings("unchecked")
    public List<Colaborador> listarColaboradores() {
        List<Colaborador> lista = new ArrayList<>();
        Session sessao = HibernateUtil.getSessionFactory().openSession();
        Transaction transacao = null;
        try {
            transacao = sessao.beginTransaction();
            Query consulta = sessao.getNamedQuery("Colaborador.listar");
            lista = consulta.list();
            transacao.commit();
        } catch (RuntimeException ex) {
            ex.printStackTrace();
            throw ex;
        } finally {
            sessao.close();
        }
        return lista;
    }

Front-end:
Controller:
angular.module("oraculo").controller("colaboradorController", function($scope, $routeParams, $location, colaboradorAPI){

 $scope.tamanhoMaximo = 6;
 $scope.totalItems = 175;
 $scope.currentPage = 1;

 ...//Código Omitido

Página:
...//Trecho omitido

<pagination total-items="totalItems" ng-model="currentPage" max-size="tamanhoMaximo" class="pagination-sm" boundary-links="true" rotate="false" num-pages="numPages"></pagination>

Page: {{currentPage}} / {{numPages}}

Answer: Faça assim:
public List<Colaborador> listarColaboradores(Integer paginaInicio, Integer paginaFim) {
    List<Colaborador> lista = new ArrayList<>();
    Session sessao = HibernateUtil.getSessionFactory().openSession();
    Transaction transacao = null;
    try {
        transacao = sessao.beginTransaction();

        Query consulta = sessao.getNamedQuery("Colaborador.listar");
        consulta.setFirstResult(paginaInicio);
        consulta.setMaxResults(paginaFim);

        lista = consulta.list();
        transacao.commit();
    } catch (RuntimeException ex) {
        ex.printStackTrace();
        throw ex;
    } finally {
        sessao.close();
    }
    return lista;
}

No teu serviço pode passar dois parâmetros definindo o inicio e final da pagina. No teu caso, por exemplo, seria na primeira consulta: 0 e 2, na segunda consulta: 2 e 4 e assim por diante.
Comment: eu passaria esses parâmetros por GET mesmo ou por POST?
Comment: Vc pode passar via GET
Comment: Entendi, eu to passando apenas a currentPage, esta quase dando certo porém ele ainda não está paginando da forma correta. O que eu passo nesse `MaxResults` ?
Comment: E quando pulo pra próxima página ele repete o ultimo registro da pagina anterior, além do registro com código = 1 não aparecer na consulta
Comment: MaxResults vc define até qual pagina será retornada. Imagine a seguinte situação. Vc tem 100 registros no banco. Passando FirstResult=10 e MaxResults=20 o hibernate irá retornar apenas os registros entre essa faixa, ou seja, registros entre 10 e 20, entendeu?
Comment: Ele não repete o ultimo registro, passa para o próximo
Comment: Entendi, como não exibo ele na próxima página?
Comment: No retorno do teu serviço vc seta $scope. =


OPENPYXL: How can I get the maximum number of columns with data in a specific row?

Question: I have my data in an excel file and I want to python to get the number of occupied cells in a specific row. How can I do that?
For instance, if I want to know how many columns are occupied with data in row 5, how would I achieve that?
PS: The cells are occupied with strings Notice how some rows have more data than some (click here to see image)
Comment: This question has been asked a lot.
Answer: you can iterate on the row:
def get_filled_cols(row_obj):
    counter = 0
    for c in row_obj:
        if c.value != None:
            counter += 1
    return counter

now you can check for row, 5 for example:
get_filled_cols(ws[5])


Install Drupal 8 on MAMP: can't detect version of Apache

Question: I'm trying to install Drupal 8 on my mac using MAMP Pro. I get this error message when Drupal tries to verify I have the proper configuration:
Apache Version:
Due to the settings for ServerTokens in httpd.conf, it is impossible to accurately
determine the version of Apache running on this server. The reported value is
Apache, to run Drupal without mod_rewrite, a minimum version of 2.2.16 is needed.

I have the latest version of MAMP Pro which is running 2.2.29.
Answer: Drupal uses Apache's ServerTokens in system_requirements() to determine the version number. You currently have that set to "Prod", which just reveals "Apache" as the token, without any version number.
For Drupal to detect the major and minor versions, plus revision, you need to update the config setting to one of:

Full - Server sends (e.g.): Server: Apache/2.4.2 (Unix) PHP/4.2.2 MyMod/1.2.
Min - Server sends (e.g.): Server: Apache/2.4.2.
OS - Server sends (e.g.): Server: Apache/2.4.2 (Unix).
Comment: Located '/applications/MAMP/config/apache/extra/httpd-default.conf' if you use MAMP on a Mac. It's set to Full and I still get the error message.
Comment: Changing "ServerTokens Prod" to "ServerTokens Full" in httpd.conf solved my problem.
Comment: It's hard to overstate how assinine this entire situation is.  ServerTokens should be "prod" in production, to give away the least amount of info to attackers.  Fingerprinting the apache version through the HTTP headers will not work in properly configured environments.  Moreover, reducing the security so that you can get past an installer is obviously not best practices.  And final insult to injury, the instructions it prints are bunk, it sends you to edit httpd.conf which only is the right file if you compiled apache from source. Nonetheless this is the only solution to get it to install.


Add +1 to span and displaying am I doing this the hard way?

Question: I have a feeling there is a much easier way to do this, but even this took me a bit to figure out. I need to remember that javascript and php are have similar syntax. Can someone reduce this?
I even tried $('span#'+venue).html(thisvenuecount++); but that didn't work.
var thisvenuecount = $('span#'+venue).html();
thisvenuecount++;
$('span#'+venue).html(thisvenuecount);

HTML
1

Comment: Im just trying to add +1 to the count. Its a number that I get from the value. This works but I feel likes it more than I need.
Answer: You shoud parse it to an int first.
var venue = 69;
$('span#'+venue).html(parseInt($('span#'+venue).html()) + 1);

Or even:
var el = $('#69');
el.html(parseInt(el.html()) + 1);


When add new model to database has error Cannot insert duplicate key

Question: When i want add new model to DB
Violation of PRIMARY KEY constraint 'PK_dbo.Factors'. Cannot insert duplicate key in object 'dbo.Factors'. The duplicate key value is (1001).
The statement has been terminated.

My code :
var factor = new Factor();
factor.UserId = UserId;
Db.Factors.Add(factor);
Db.SaveChanges();

And model map:

In my design factor is must start whit 1000 and my code for this :
protected override void Seed(BreDbContext context)
{
    context.Database.ExecuteSqlCommand("DBCC CHECKIDENT('Factors', RESEED, 1000);");
}

Comment: On insert you do not need to set the id, this will be set by the database
Comment: this is becuase same id is assign to event modal that why db throw error
Comment: you mean Baskets?
Comment: I know ,i don't set
Answer: I think it's because u insert the value that has already in the PRIMARY KEY field which mean PRIMARY KEY value must be a unique value
For example u have a table with value :
id name
1 john
2 serah
3 crish

But after that you insert a new value:
3 victoria

in above case will throw an error because in the table already has a value with id = 3 (crish) // id field is unique (PRIMARY KEY)
OR
In other problem case if ur table has a relation with other table, if u want to insert a new value into FOREIGN KEY Table, u need to inserted first in PRIMARY KEY Table because FOREIGN KEY need to reference to PRIMARY KEY value.
I hope my answer helping u
Comment: Tnx for answering. But id set by the database
Comment: what table u insert a value ? into PK table or FK table ? @HadiJahangiri
Comment: in the code section u post " factor.UserId = UserId; ", what value in "UserId" variable ? and please "SELECT * FROM ur table" to ensure u not insert the same key
Comment: UserId = post data user id, Generate key by database not me
Comment: Whether UserId is PK in ur table?
Answer: I Find solution.
Description for problem
When run below code in Migration Seed
context.Database.ExecuteSqlCommand("DBCC CHECKIDENT('Factors', RESEED, 1000);");

this code run per request and this code destroy database.
Remove this code and insert below code
protected override void Seed(BreDbContext context)
{
    //context.Database.ExecuteSqlCommand("DBCC CHECKIDENT('Factors', RESEED, 1000);");
    for (int i = 0; i < 999; i++)
    {
        context.factors.add(new factor());
    }
    context.savechanges();
    var rang = context.factors.tolist();
    context.factors.removerange(rang);
    context.savechanges();
}

after create database comment all code in seed method.
best reference https://weblog.west-wind.com/posts/2013/Mar/26/Firing-an-Entity-Framework-Database-Initializer-from-within-DbContext


ImportError SciKit-learn

Question: I am trying to get started with machine learning, so I have installed the packages: numpy, Scikit-learn, matplotlib, scipy. Some I have installed directly from pip with:
python -m pip install "package name"

and and others i have downloaded the binary files and then installed with pip. It shows no errors when I import matplotlib, numpy and sklearn, but when I write:
from sklearn import svm

it gives me the error:
ImportError: cannot import name 'svm'

I am on Python 3.5.1 and on Windows 10. Does anyone have any solutions?
Comment: Have you tried simply `import sklearn` from your other comment below it doesn't seem that you installed it properly.
Comment: @Leb Yes i have tried that and it works just fine, that is why i am so confused :/

I might just reinstall python or something... Now even if i open a new file and just write `import sklearn` it gives me no errors but for some reason it writes: hello world.. I am so confused
Comment: Try this then: `from sklearn.svm import SVC`. Does that work?
Comment: No it still does not work, I guess I must have installed them incorrectly like Chris said
Answer: import sklearn.svm as svm

model = svm.SVC()
....

http://scikit-learn.org/stable/modules/classes.html#module-sklearn.svm
Comment: It still gives me an error.

`ImportError: No module named 'sklearn.svm'; 'sklearn' is not a package`
Comment: There must be an installation issue. Given you say it writes hello world when you import the module, something must be incorrect. I recommend anaconda if you have the flexibility to do so, it makes managing these things much easier.
Comment: @Chis Should i just delete the packages and try with anaconda then? Also would it make a difference that i installed some of them through pip and others i downloaded from [here](http://www.lfd.uci.edu/~gohlke/pythonlibs)?
Comment: uninstall python then install anaconda, it has easy installation for a tonne of packages. you just do conda install sklearn and it will get the correct version for your python, OS and architecture. you can also use pip and binary installs for some less popular modules, but 90% of the stuff you need is a one line install, no need to worry about selecting the correct dist etc. In fact it already comes with sklearn installed. http://docs.continuum.io/anaconda/pkg-docs
Answer: It does seem that you didn't install it properly. Since you're on windows I would recommend using the Unofficial Windows Binaries for Python Extension Packages website to install future packages.
Make sure you also install the proper binaries as I pointed out in this post Installing scipy in Python 3.5 on 32-bit Windows 7 Machine. The windows version doesn't matter just make sure you're downloading Visual C++ 2015 redistributable package.

ImportError SciKit-learn


Excel cross referencing multiple data sets across multiple indices

Question: I'm trying to semi-automate identifying when members of a distributed squad have public holidays (or other events) that fall between two specific dates.  I have several datasets in use:

I have a list of public holidays with DATE and LOCATION in two columns, which looks something like:

A
B

2
DATE
LOCATION

3
19/09/2023
INDIA

4
29/09/2023
AUSTRALIA

5
02/10/2023
INDIA

6
23/10/2023
NEW ZEALAND

7
etc
etc

I have a list of squad members across columns with their location in the second row, something like:

A
B
C
D
E
F

1
Name
Rob
Sam
Tim
Grace
Elinor

2
Location
Australia
India
Australia
New Zealand
India

I also have a start and end date:

A
B

1
Start Date
End Date

2
02/09/2023
16/10/2023

What I'd like to do is have a row underneath each team member that will automatically indicate if there is a public holiday in country that falls on a date between the start and end date, and if there are multiple then providing a count.
I have a couple of pretty hacky ideas that may work involving a bunch of "helper" columns, but I'd like to find a more tidy solution (hopefully in a single formula) if there is one.  Any ideas?
Answer: =SUM((FILTER($A$2:$A$5,$B$2:$B$5=B14)>=$A$8)*(FILTER($A$2:$A$5,$B$2:$B$5=B14)<=$B$8))

or
=LET(z,FILTER($A$2:$A$5,$B$2:$B$5=B14),SUM((z>=$A$8)*(z<=$B$8)))

or to spill:
=MAP(B14:F14,LAMBDA(a,LET(z,FILTER($A$2:$A$5,$B$2:$B$5=a),SUM((z>=$A$8)*(z<=$B$8)))))

Result:
Comment: Brilliant again!  Thanks so much for the reply.  Lots of cool options to try.
Answer: Try using the following formula with MMULT()

• Formula used in cell E3
=LET(
     α, --REPT(PublicHolidays[DATE],N(E2:I2=PublicHolidays[LOCATION])),
     φ, IFERROR((α>=Dates[Start Date])*(α<=Dates[End Date]),0),
     MMULT( SEQUENCE(,ROWS(φ),,0),φ))

Or, using LAMBDA() helper function BYCOL()

• Formula used in cell E3
=LET(
     α, --REPT(PublicHolidays[DATE],N(E2:I2=PublicHolidays[LOCATION])),
     BYCOL((α>=Dates[Start Date])*(α<=Dates[End Date]),LAMBDA(φ, SUM(IFERROR(φ,0)))))

Note that with recent update in MS365 for Beta Channel users can avoid the LAMBDA() construct to use BYCOL() just like shown below.

=LET(
     α, --REPT(PublicHolidays[DATE],N(E2:I2=PublicHolidays[LOCATION])),
     BYCOL(IFERROR((α>=Dates[Start Date])*(α<=Dates[End Date]),0),SUM))

Also note:
Where: PublicHolidays refers to table structured range: A1:B5
Where: Dates refers to table structured range: A8:B9

BTW, you can also use an IF() in place of REPT()

=LET(
     α, IF(E2:I2=PublicHolidays[LOCATION],PublicHolidays[DATE],""),
     BYCOL(IFERROR((α>=Dates[Start Date])*(α<=Dates[End Date]),0),SUM))

Comment: Brilliant!, Thanks so much for the prompt reply.  I will give these a go.
Answer: For something like this, you could also just keep things simple and use the COUNTIFS function. For example:
=COUNTIFS(A2:A5, ">="&D2, A2:A5, "<="&E2, B2:B5, E5:I5)

Note: only one formula was required in cell E6 because all locations in the range E5:I5 were passed to the 3rd criteria parameter, causing the results to spill dynamically across. Cheers!


Is there a way to declare a controller in AngularJS and only specify its dependencies once?

Question: In order to keep dependency injection working after minification I'm using this style to declare a controller:
myModule.controller('myController', ['$rootScope', 'Service',
  function($rootScope, Service) {
    $scope.foo = 'bar';
}]);

Let's say you want to add another service. You'd have to add it in 2 places, in the array preceding the controller function, and as a parameter of the controller function itself. My question is, is there a way to make this DRY-er? Could you declare a dependency array and use that to build the outer array and the inner parameters, something like this?:
var dependencies = ['$rootScope', 'Service'];

var myProtoController = function() {
  $scope.foo = 'bar';
};

var myController = dependencies.push(myProtoController.bind.apply(this, [this].concat(dependencies)));

myModule.controller('myController', myController);

Answer: Can you use Ngmin in your tool chain?  It will convert your angular functions into the minify-able style you're currently using.
It integrates with grunt and rails, so it should be simple to use in your current pipeline.
Answer: Using concat ?
var deps = ['$rootScope', 'Service'];

myModule.controller('myCtrl', deps.concat(function ($scope, $rs, S) {
  $scope.foo = 'bar';
}));

Comment: I like it.  But there is a syntax error.  Your missing a closing paren.
Comment: It looks like you still need to write the dependencies twice in this one.
Comment: How so? But +1 for ngmin anyway, I just usually try to suggest from scratch-working solutions.


Free object in the category of fields

Question: I've seen this stated on this site as well as elsewhere that there is no free object in the category of fields on a set $X$ of any size including the empty set. I'm wondering why this is the case?
Comment: what have you tried? https://math.stackexchange.com/questions/81458/category-of-field-has-no-initial-object might be of some use
Comment: You should play around with the concept, as @PraphullaKoushik hints.
Answer: Hint: Assume $F$ is a free field on the set $X$. Then try to extend any map $X\to K$ to a homomorphism where $K$ has different characteristic.
Comment: What about the category of fields of a given characteristic?

Free object in the category of fields


Choose two basis $A,B$ from $\mathbb{R}^3$ such that $T^{A}_{B}(F)=\left(\begin{smallmatrix}1&0&0\\1&1&0\\0&0&0\end{smallmatrix}\right)$

Question:
$M= \begin{pmatrix}  1 & 0 & 1\\   2 & 1 & 1\\
-1 & 1 & -2 \end{pmatrix}$ is a real matrix and $F: \left\{\begin{matrix} \mathbb{R}^{3} \mapsto \mathbb{R}^{3} \\  x
\mapsto M \cdot x \end{matrix}\right.$ is the linear mapping given by
  $M$
Choose two basis $A= (v_{1},v_{2},v_{3})$ and $B= (w_{1},w_{2},w_{3})$
  from $\mathbb{R}^3$ such that
  $T^{A}_{B}(F)=\begin{pmatrix}1&0&0\\1&1&0\\0&0&0\end{pmatrix}$

I know how to do it the other way around, I mean when basis are given and I'm supposed to determine the tansformation matrix. But how to solve it this way? I found that task on the internet and I hope the things given above are even enough. I could try to write a solution but I barely have an idea :(
Comment: If you know how to determine the transformation matrix, you could write the transformation formula in general, for unknown basis with coefficients $b_{íj}$ and then trying to compute these coefficients.
Answer: Note that you can choose both bases, which is rather uncommon for maps from a vector space to itself (where one usually would want to use the same basis twice). Given that you have this freedom, the only constraint that might form an obstruction to your task is the rank of the matrix (which will be the same whatever bases you use to express things on). It is not hard to see that $A$ has rank$~2$, and so has the prescribed form $T^A_B(F)$. In order to get a last column entirely zero, you must choose $v_3\in\ker(F)$, and nonzero. With this done, choose any $v_1,v_2$ to complete to a basis. Now the first column column of $T^A_B(F)$ is saying $T(v_1)=w_1+w_2$ and the second is saying $T(v_2)=w_2$. It is not hard to solve $w_1,w_2$ from this, and you can choose $w_3$ in any way to complete to a basis. For another pair of bases, just make other choices.
Comment: You can answer your own question (rather than edit it) using my suggestions; that is fine, and I'll look at it.
Comment: Yes, you need to do that.
Comment: Do you know about the [extension from an independent set to a basis](https://en.wikipedia.org/wiki/Basis_(linear_algebra)#Extending_to_a_basis)? Just take any basis of your space (for instance the standard basis), and keep adding elements from it to your independent set (initially in single vector in your case) provided the result is still linearly independent (if not just drop that vector).
Comment: Not a lot of time. You already had $v_3=(-1,1,1)$. Complete with $v_1=(1,0,0),v_2=(0,1,0)$ from a standard basis. Get $w_2=F(v_2)=(0,1,1)$ and then $w_1=(w_1+w_2)-w_2=F(v_1)-F(v_2) = (1,2,-1)-(0,1,1)=(1,1,-2)$ and complete with (for instance) $w_3=(1,0,0)$. That was only long because I needed to keyboard it correctly.
Comment: Thank you very much! If I solved that task and editted my answer later, could you please take a look at it?
Comment: A question before I start, $\text{ker}(F)$, so I have to calculate the kernel of matrix $M$, right?
Comment: Sorry for several questions  already, but how "choose any $v_1,v_2$ to complete to a basis"? I have that  $\text{ker}(F)= \left\{\begin{pmatrix}
-z\\
z\\
z
\end{pmatrix} \mid z \in \mathbb{R}\right\}$, I set $z=1$ and get $v_3= \begin{pmatrix}
-1\\
1\\
1
\end{pmatrix}$.
Comment: Ok never mind I cannot do it I thought it was easy but it isn't. But thank you anyway for your answer and time. If it's asked in the exam, we will get low points on it anyway even though it seems to require a lot time to solve it...
Comment: I cannot believe it but now I really understood how its done. I could even solve another task and everything was correct. Thank you so much again. I made another topic another day ago and I put a bounty on it (200). I'd like to give you the bounty so it's not wasted. So maybe just write an answer there saying some words (linking to this topic which helped me, or something like that). But don't waste your time writing another complete answer for me because I really understood because of you now! :)


How to replace all instances of text between two characters in a huge JSON string using T-SQL in an Azure SQL database?

Question: How to replace text between 2 characters MULTIPLE TIMES in a huge string (50,000 characters) in T-SQL?
I have a JSON output from my Azure Pipeline which we drop into a single cell in a table ready for the stored procedure to parse and process the data.  However, one of the lines in the JSON looks like this
"body":"bxncucbuc "what is happening" cbibciuc",
"subject:"whatever" etc etc. .........
"body":"bxncucbuc "what more things are happening" cbibciuc",
"subject:"whatever" etc etc. .........
"body":"bxncucbuc "what else is happening" cbibciuc",
"subject:"whatever" etc etc.

The openjson cannot parse the script as the "what is happening" (or similar) looks like another column as it has the " 's in.  So, I am looking to replace everything BETWEEN "body":" and "subject":, as actually I don't need this data anyway.
I can get a single instance replaced using
select
    stuff(jsonresponse, CHARINDEX('"body":"', jsonresponse), CHARINDEX('"subject":', jsonresponse) - CHARINDEX('"body":"', jsonresponse) + 1, '"bodyX":"replaced with a random piece of text","')
from
    stage.JsonResponse

However
while (select CHARINDEX('"body":"<', jsonresponse) from stage.JsonResponse) != 0
begin
    update stage.JsonResponse
    set JsonResponse =  stuff(jsonresponse, CHARINDEX('"body":"', jsonresponse), CHARINDEX('"subject":', jsonresponse) - CHARINDEX('"body":"', jsonresponse) + 1, '"bodyX":"replaced with a random piece of text","')
    from stage.Mimecast_JsonResponse

    if (select CHARINDEX('"body":"<', jsonresponse)
        from stage.JsonResponse) = 0
        BREAK
    else
        CONTINUE
END

just runs forever. I have changed it to look for "body":"< in case it was inadvertently changing some script and then looking for itself, and changing that again etc.
Similarly, I have tried a CTE:
;WITH Cte(jsonresponse) AS
(
    SELECT jsonresponse FROM #a
    UNION ALL
    SELECT STUFF(jsonresponse, CHARINDEX('"body":"', jsonresponse), CHARINDEX('"subject":"', jsonresponse) - CHARINDEX('"body":"', jsonresponse) + 1, '"body":"replaced with a random piece of text","')
    FROM Cte
    WHERE CHARINDEX('"body":"<', jsonresponse) != 0
)
SELECT jsonresponse
FROM Cte
WHERE CHARINDEX('"body":"<', jsonresponse) = 0
OPTION (MAXRECURSION 0); `

but this too seems to run and run, but actually only changes the first instance.
My JSON is 50,000 characters long, and there are multiple instances in the json that I need to replace ALL OF THEM, and this is where I have struggled, how to change EVERY INSTANCE of the offending script.
Any help will be appreciated.
I am using T-SQL on an Azure SQL database.
Comment: I appreciate this isn't answering your question, but.... If you can dump your JSON into a SQL table, could you dump the JSON file to another storage system (BLOB Container?) where you can just read the JSON file? You've got all the overhead of SQL without any of the benefits by putting the data into one cell.
Comment: The other option is to use [ISJSON](https://learn.microsoft.com/en-us/sql/t-sql/functions/isjson-transact-sql?view=sql-server-ver16) before you insert your data. That way you should be able to use `OPENJSON`
Comment: Thankyou @PGHE for your consideration, certainly that could have been an option for us - good suggestion.  Thankyou
Answer: Looking at the option of using a recursive CTE, I believe this should work - I did some experimenting and it appears (probably due to how the data gets amended and referenced or passed around internally) the most reliable way is to not affect the length of the string but just to replace characters in-situ.
The following example works to find the sections between each body/subject element and replaces the content with a single repeating character without changing the length.
Since you don't need this content anyway you can then use replace to remove it completely.
Does this help for your scenario?
Demo DBFiddle
with p as (
  select Id, jsonCol,
    CharIndex('"body"', jsonCol) as Bpos,
    CharIndex('"subject"', jsonCol) as Spos
  from t
  union all
  select Id, j,
    CharIndex('"body"', j, Spos) as Bpos,
    CharIndex('"subject"', j, Spos + 1) as Spos
  from p
  cross apply(values( Stuff(jsoncol, bpos + 8, spos - bpos - 10, Replicate('#', spos - bpos - 10))))n(j)
  where Bpos > 0
)
select Replace(jsoncol, '#','')
from p
where bpos = 0;

Comment: Thankyou @Stu this works awesomely, and is so fast on my dataset.  This is an awesome script.  Really appreciate the efforts.


.bind( ) Javascript not working in Firefox & IE?

Question: I have a div that on click moves to the right, and then on click again and it goes back to the left (its a newsletter pop-out). Everything is working perfectly in Safari and Chrome, however in Firefox and IE it will only open, it won't close when you click again.  I am using .bind() and .unbind()
Any suggestions?  I am using jQuery version 1.3.2.
FF Error : NS_ERROR_XPC_BAD_CONVERT_JS: Could not convert JavaScript argument arg 0 [nsIDOMWindow.getComputedStyle]
IE Error: SCRIPT16386: No such interface supported
Javascript:
$(function () {
    $('.newsletter').bind('click.open', open)
    $(window).resize(function () {
        var wWth = $(window).innerWidth(),
            wHgt = $(window).innerHeight();
        $('#overflow').fadeIn('slow').css({
            width: wWth,
            height: wHgt
        });
    });
});

function open() {
    var overflow = $('

');
    $(this).stop().animate({
        left: '-35'
    }, 650);
    if ($('#overflow').length < 1) {
        $('body').append(overflow);
    }
    var wWth = $(window).innerWidth(),
        wHgt = $(window).innerHeight();
    $('#overflow').fadeIn('slow').css({
        width: wWth,
        height: wHgt
    });
    $('.newsletter').unbind('click.open').bind('click.close', close)
}

function close(e) {
    if ($(e.target).is('input')) return;
    $('#overflow').fadeOut('slow')
    $('.newsletter').stop().animate({
        left: '-390px'
    }, 650);
    $('.newsletter').unbind('click.close').bind('click.open', open)
}

HTML

    Newsletter bestellen

    Lassen Sie sich per Newsletter über Neuheiten und

    Aktionen von Tempur informieren. Jetzt anmelden

    <form id="leftnewsletter" action="" method="get">
    <input type="email" name="email" id="emailsignup" placeholder="E-MAIL ADDRESS" style="margin-left:76px; margin-top:16px; width:187px;">
    <input type="submit" name="signupbtn" id="signupbtn" value="SIGN UP" class="signupbutton">
</form>

    *Sie können jederzeit Ihre Einwilligung zum Erhalt des

Newsletters zurückziehen und sich abmelden.

Comment: It would help to specify the jQuery version that you're using.
Comment: can you try to recreate the problem in [this fiddle](http://jsfiddle.net/arunpjohny/bWbxT/1/) it seems to be working fine in FF
Comment: can you share the related html also
Comment: The cause might be in broken HTML layout (incorrectly closed tags). This is the most common cause of browsers behaving differently in similar situations.
Comment: @ArunPJohny i also created a fiddle which i had been using in ff and it was working fine too, its only in my actual code? [fiddle](http://jsfiddle.net/FpEZg/2/)
Comment: add few console logging statements in the handlers and see whether they are getting executed
Comment: First thing to do is to open the FF debugger to see if any js error or warning is thrown.
Comment: I am getting the error **NS_ERROR_XPC_BAD_CONVERT_JS: Could not convert JavaScript argument arg 0 [nsIDOMWindow.getComputedStyle]** in FF and IE
Comment: Works fine here here with jQuery 1.3.2 http://jsfiddle.net/aneSm/; FF and IE version?
Comment: Yes, it just seems to be in my actual coding, could there be a conflict or something?
Comment: It's possible; can you provide more context, other code plugins, and so on?
Comment: Theres quite a few plugins used, like **jquery.jcarousel.pack.fixed.js**, **jquery.validate.js** and so on.... it tells me the problem is with **jquery-1.3.2.min.fixed.js** when i go into firebug > console?
Comment: Can you update your old 1.3.2 jQuery core version? The validate min version is 1.6.4, for carousel I don't know
Comment: I will check with my collegue first as i know he told me before there was a reason they haven't updated jQuery version, but i can't remember, thank you for your help!
Comment: Ok let me know if the issue is related to the jQuery versione, so I'll add the comment as answer
Answer: As of jQuery 1.7, the .on() method is the preferred method for attaching event handlers to a document.
There is detailed post in the past discussing it.
In short, .bind() is kept just for the compatibility purpose, we should forget it and use the more flexible .on() instead.
For your problem, change your bind() statement from:
$('.newsletter').bind('click.open', open)

into:
$(document).on('click', '.newsletter', open);

UPDATE:
Since you are using a very early version of jQuery, I guess the best choice you have is to use .noConflict to load the two versions together. jQuery has changed a LOT since 1.3. It is understandable that you might hesitate to upgrade, since your old code might have used some old functions like browser detection, etc.
So here is how you do it:
jqNew = jQuery.noConflict( true );

Then you will use jqNew(document).on(event, selector, function); or whatever to bind your event. The old code will continue to use $.xxx.
There is an example here. Scroll that page down to the section: Example: Load two versions of jQuery (not recommended). Then, restore jQuery's globally scoped variables to the first loaded jQuery.
Comment: I was using the on/off method before bind, however i was experiencing the problem of the newsletter not opening at all? And thats why i tried using bind
Comment: Just updated the answer. You might want to consider loading a new version of jQuery beside the old one.
Answer: Got it working by simplifying the code down to:
$('.newsletter_signup').click(function(event) {
    if(!$(event.target).is('input')) {
        if (parseInt($(this).css('left')) == -35) {
            $(this).stop().animate({ left: '-390px' }, 650);
        } else {
            $(this).stop().animate({ left: '-35px' }, 650);
        }
    }
});

Works perfectly now, thank you all for your help! :)
Answer: Is difficult to say where the problem is, probably is somewhere in your HTML code and its manipulation.
Your sample code works on jQuery 1.3.2: http://jsfiddle.net/aneSm/
Some suggestions:

update your jQuery core version, your plugin can't run on 1.3.2; the validate min version is 1.6.4, for carousel I don't know (...)
check to not have any comment near a not wrapped node on which you
use a plugin (ref. 1)
check if you have a (direct or not) getComputedStyle on a DOM node
which can't have styles (such as a text node)

ref. 1

    ...


Strange out-of-key chords in Blow by Beyonce?

Question: Can someone explain to me what is going on tonally in Beyonce's song Blow? The bass notes deceptively seem to suggest the chords in the verse are G major and F major. But these chords do not sound right. It seems that the actual chords are FM7/G and EbM7/F. And then later in the song there is a C to D7, which suggests to me that the song is in G major, But then I don't understand how FM7/G and EbM7/F are fitting into this key?
I am used to understanding chord progressions as degrees in a key/scale (for example I'm Yours by Jason Mraz is I, V, vi, IV in B Ionian). I cannot wrap my head about what is going on in Blow in terms of key/mode! Any insight is much appreciated.
Answer: Yes, chords in the verse are G and F, with G being the tonic. So it's a major key, and F is indeed out of key. In this case it could be considered a modal interchange from the key of G minor. You could notate the chord progression as I bVII.
Interestingly, while in the intro G and F are clearly outlined by the melody, in the verses the melody sticks to G minor exclusively, against the mode of underlying G. Such coexistence of minor and major probably shows influence of blues genre in this case.
There is no rule that says that a song, or even a section must stick to a single scale. In fact changing scales can be used to make music more interesting. There are many techniques, including secondary dominants, modal interchange, modulation, and they are used in many styles. Pick any jazz standard and you may find scale changing in every measure!
Comment: So you're saying that the chords in the verse are G and F and not FM7/G and EbM7/F like the OP suggests?
Comment: @Dekkadeci yes. The chords are not simple triads, and various instruments add various components (consider the guitar in particular). Interestingly there seem to be both 11 and 3. Fmaj7/G and Ebmaj7/F could be valid voicings, but if we speak of harmonic analysis I hear the roots as F and G. This is particularly clear in the chorus (?) consisting of chords C (IV) and D (V) which then resolves to this "weird" G (I). Perhaps this is all subjective, but that's what I hear there.
Comment: So why don't you call those chords G13(no3) and F13(no3), then?
Comment: @Dekkadeci guitar does play the thirds. Yes, there are rich upper structures. But OP asked about functional analysis, so the key information is what the roots of the chords are.
Answer: The tonal centre is, indeed G, and the chords are indeed Fmaj7/G and E♭maj7/F.
Fmaj7/G is a specific voicing.  It's rooted on G, but it isn't based on a G triad.   There's no B, there's no D.  We'd have to call it G13(sus4)(omit5) or something just as ugly.   No, Fmaj7/G is fine!
There's no functional 'cycle of 5ths' stuff going on here (I seem to be writing this rather a lot lately!).  There's a flavour of G7, F7.
Here's something else I've been saying a lot when looking at today's pop music.   The ♭VII chord is so commonly used that it almost has to be considered diatonic.  (The ♭VII NOTE has been unremarkable for a lot longer, ever since the Blues crept into Western popular music.)