Integrate a Windows service with aspnetboilerplate

Question: Goal
Ok, so my aim here is to have a windows service run alongside my aspnetboilerplate web application. The reason for the windows service is that it will run the Hangfire background jobs, from a completely separate virtual server.
Configuration
This is the template i used to get things going

FYI, everything is working from from the web application.
Here is the Service program file (i am using Topshelf but i don't think that matters).
HostFactory.Run(x =>
{
    x.Service<PortalService>(sc =>
    {
        sc.ConstructUsing(() => new PortalService());

        sc.WhenStarted((tc, hostControl) => tc.Start(hostControl));
        sc.WhenStopped((tc, hostControl) => tc.Stop(hostControl));
    });

    x.UseLog4Net();
    x.RunAsLocalSystem();
    x.SetServiceName(PortalService.ServiceName);
    x.SetDisplayName(PortalService.ServiceDisplayName);
    x.SetDescription(PortalService.ServiceDescription);
    x.StartAutomatically();
});

Here is the Service Module
[DependsOn(typeof(PortalCoreModule), typeof(PortalEntityFrameworkModule), typeof(PortalApplicationModule), typeof(AbpHangfireModule))]
public class PortalServiceModule : AbpModule
{
    public override void Initialize()
    {
        IocManager.RegisterAssemblyByConvention(typeof(PortalServiceModule).GetAssembly());
    }

    public override void PreInitialize()
    {
        Configuration.BackgroundJobs.UseHangfire(config =>
        {
            config.GlobalConfiguration.UseSqlServerStorage(PortalConsts.ConnectionStringName, new SqlServerStorageOptions
            {
                PrepareSchemaIfNecessary = true
            });

            config.Server = new BackgroundJobServer(new BackgroundJobServerOptions
            {
                Queues = new[] {BackgroundJobQueueNames.Critical, BackgroundJobQueueNames.Autotask, BackgroundJobQueueNames.Default}
            });

            config.GlobalConfiguration.UseConsole();
        });

        GlobalJobFilters.Filters.Add(new DisableMultipleQueuedItemsFilter());
    }
}

And then here is my Service code
public class PortalService : ServiceControl
{
    public const string ServiceName = "PortalWorker";
    public const string ServiceDisplayName = "Portal Worker";
    public const string ServiceDescription = "Process the background jobs for the Portal.";

    private AbpBootstrapper _bootstrapper;

    public bool Start(HostControl hostControl)
    {
        _bootstrapper = AbpBootstrapper.Create<PortalServiceModule>();

        _bootstrapper.IocManager
            .IocContainer
            .AddFacility<LoggingFacility>(f => f.UseAbpLog4Net().WithConfig("log4net.config"));

        _bootstrapper.Initialize();
        return true;
    }

    public bool Stop(HostControl hostControl)
    {
        _bootstrapper.Dispose();
        return true;
    }
}

Yet every time i run the service, i get the following exception

Castle.MicroKernel.ComponentNotFoundException: 'No component for supporting the service Portal.BackupMonitoring.IBackupMonitoringManager was found'

However, if i was to run the Web application, it works fine, so to me its not an issue with IBackupMonitoringManager. It must be something to do with the Service not being able to register against the DI.
I have had a look in this repo for examples https://github.com/aspnetboilerplate/aspnetboilerplate-samples
But the examples are either massively out of date, or i cannot see an example of a console application with a dot net core web application.
Any help on where to get started with this would be greatly appreciated.
Update
Here is the new exception when a job tries to execute.

Castle.MicroKernel.Handlers.HandlerException: 'Can't create component
  'Portal.Authorization.Users.UserManager' as it has dependencies to be
  satisfied.
'Portal.Authorization.Users.UserManager' is waiting for the following
  dependencies:
  - Service 'Portal.Authorization.Roles.RoleManager' which was registered but is also waiting for dependencies.
  'Portal.Authorization.Roles.RoleManager' is waiting for the following
  dependencies:
  - Service 'System.Collections.Generic.IEnumerable1[[Microsoft.AspNetCore.Identity.IRoleValidator1[[Portal.Authorization.Roles.Role,
  Portal.Core, Version=220.127.116.11, Culture=neutral, PublicKeyToken=null]],
  Microsoft.Extensions.Identity.Core, Version=220.127.116.11, Culture=neutral,
  PublicKeyToken=adb9793829ddae60]]' which was not registered.
  - Service 'Microsoft.AspNetCore.Identity.ILookupNormalizer' which was not registered.
  - Service 'Microsoft.AspNetCore.Identity.IdentityErrorDescriber' which was not registered.
  - Service 'Microsoft.Extensions.Logging.ILogger1[[Abp.Authorization.Roles.AbpRoleManager2[[Portal.Authorization.Roles.Role,
  Portal.Core, Version=22.214.171.124, Culture=neutral,
  PublicKeyToken=null],[Portal.Authorization.Users.User, Portal.Core,
  Version=126.96.36.199, Culture=neutral, PublicKeyToken=null]], Abp.ZeroCore,
  Version=220.127.116.11, Culture=neutral, PublicKeyToken=null]]' which was not
  registered.
  - Service 'Microsoft.Extensions.Options.IOptions1[[Microsoft.AspNetCore.Identity.IdentityOptions,
  Microsoft.Extensions.Identity.Core, Version=184.108.40.206, Culture=neutral,
  PublicKeyToken=adb9793829ddae60]]' which was not registered.
  - Service 'Microsoft.AspNetCore.Identity.IPasswordHasher1[[Portal.Authorization.Users.User,
  Portal.Core, Version=188.8.131.52, Culture=neutral, PublicKeyToken=null]]'
  which was not registered.
  - Service 'System.Collections.Generic.IEnumerable1[[Microsoft.AspNetCore.Identity.IUserValidator1[[Portal.Authorization.Users.User,
  Portal.Core, Version=184.108.40.206, Culture=neutral, PublicKeyToken=null]],
  Microsoft.Extensions.Identity.Core, Version=22.214.171.124, Culture=neutral,
  PublicKeyToken=adb9793829ddae60]]' which was not registered.
  - Service 'System.Collections.Generic.IEnumerable1[[Microsoft.AspNetCore.Identity.IPasswordValidator1[[Portal.Authorization.Users.User,
  Portal.Core, Version=220.127.116.11, Culture=neutral, PublicKeyToken=null]],
  Microsoft.Extensions.Identity.Core, Version=126.96.36.199, Culture=neutral,
  PublicKeyToken=adb9793829ddae60]]' which was not registered.
  - Service 'Microsoft.AspNetCore.Identity.ILookupNormalizer' which was not registered.
  - Service 'Microsoft.AspNetCore.Identity.IdentityErrorDescriber' which was not registered.
  - Service 'System.IServiceProvider' which was not registered.
  - Service 'Microsoft.Extensions.Logging.ILogger1[[Microsoft.AspNetCore.Identity.UserManager1[[Portal.Authorization.Users.User,
  Portal.Core, Version=22.214.171.124, Culture=neutral, PublicKeyToken=null]],
  Microsoft.Extensions.Identity.Core, Version=126.96.36.199, Culture=neutral,
  PublicKeyToken=adb9793829ddae60]]' which was not registered. '
Comment: Where is `IBackupMonitoringManager` injected?
Comment: @aaron ok so i managed to resolve that to a certain degree. The Service project was using an old connection string and thus a different database. Correcting this got rid of this particular error. However i am now getting this error `Castle.MicroKernel.Handlers.HandlerException: Can't create component 'Portal.Authorization.Users.UserManager' as it has dependencies to be satisfied.`    The problem i've got is that the `BackupMonitoringManager` implementation does not reference `Portal.Authorization.Users.UserManager`
Comment: Infact let me correct that i've just had a further look into it and the `BackupMonitoringManager` references `GroupManager` which itself depends on `UserManager`.
Comment: It's injected all over the application. However in this instance i believe it is trying to resolve it when it's executing the background job like so `manager.AddOrUpdate("Get.Backup.Results", Job.FromExpression(j => j.GetEmails(null)), "*/15 * * * *", TimeZoneInfo.Local, queue: BackgroundJobQueueNames.Critical);`
Comment: The Service starts up ok and Hangfire starts polling for changes. When that job is executed it throws the exception in the update i've posted. If the code is executed from the web application, it works fine.
Comment: I've added the DependOns to the ServiceModule but it is still giving the same exception. I noticed that this code is called from the startup of the webapplication `IdentityRegistrar.Register(services);`, i'm thinking maybe i need to call this from the Console application as well?
Comment: @aaron do you know how to get a reference to `IServiceCollection services` from the console app?
Comment: Just `new` it. See https://andrewlock.net/using-dependency-injection-in-a-net-core-console-application/
Answer: I think i have got this now, so i'm going to post it as the answer, but let me know if there is anything wrong with it.
First of all i had to install Abp.AspNetCore package. This allowed access to the IServiceCollection.AddAbp() extension method.
So my Program.cs changed to this
static void Main(string[] args)
{
    Clock.Provider = ClockProviders.Utc;

    var services = new ServiceCollection();
    IdentityRegistrar.Register(services);

    services.AddAbp<PortalServiceModule>(options =>
    {
        //Configure Log4Net logging
        options.IocManager.IocContainer.AddFacility<LoggingFacility>(
            f => f.UseAbpLog4Net().WithConfig("log4net.config")
        );
    });

    HostFactory.Run(x =>
    {
        x.Service<PortalService>(sc =>
        {
            sc.ConstructUsing(() => new PortalService());

            sc.WhenStarted((tc, hostControl) => tc.Start(hostControl));
            sc.WhenStopped((tc, hostControl) => tc.Stop(hostControl));
        });

        x.UseLog4Net();
        x.RunAsLocalSystem();
        x.SetServiceName(PortalService.ServiceName);
        x.SetDisplayName(PortalService.ServiceDisplayName);
        x.SetDescription(PortalService.ServiceDescription);
        x.StartAutomatically();
    });
}

I had some problems with the PortalServiceModule. The automapper configuration is already in the PortalApplicationModule so i may have to revisit this to remove the duplicate code. For the moment, this is what i got working.
[DependsOn(typeof(PortalCoreModule), typeof(PortalEntityFrameworkModule), typeof(PortalApplicationModule), typeof(AbpHangfireModule), typeof(AbpZeroCoreModule), typeof(AbpAutoMapperModule))]
public class PortalServiceModule : AbpModule
{
    public override void Initialize()
    {
        IocManager.RegisterAssemblyByConvention(typeof(PortalServiceModule).GetAssembly());

        Configuration.Modules.AbpAutoMapper().Configurators.Add(cfg =>
        {
            cfg.CreateMap<object, int?>().ConvertUsing<NullableIntTypeConverter>();

            // Scan the assembly for classes which inherit from AutoMapper.Profile
            cfg.AddProfiles(typeof(PortalApplicationModule).GetAssembly());
        });

        Configuration.BackgroundJobs.UseHangfire(config =>
        {
            config.GlobalConfiguration.UseSqlServerStorage(PortalConsts.ConnectionStringName, new SqlServerStorageOptions
            {
                PrepareSchemaIfNecessary = true
            });

            config.Server = new BackgroundJobServer(new BackgroundJobServerOptions
            {
                Queues = new[] { BackgroundJobQueueNames.Critical, BackgroundJobQueueNames.Autotask, BackgroundJobQueueNames.Default }
            });

            config.GlobalConfiguration.UseConsole();
        });

        GlobalJobFilters.Filters.Add(new DisableMultipleQueuedItemsFilter());
    }

    public override void PostInitialize()
    {
        HangfireJobsConfigurer.Configure();
    }
}

Then my actual service code became this
public class PortalService : ServiceControl
{
    public const string ServiceName = "PortalWorker";
    public const string ServiceDisplayName = "Portal Worker";
    public const string ServiceDescription = "Process the background jobs for the Portal.";

    private AbpBootstrapper _bootstrapper;

    public bool Start(HostControl hostControl)
    {
        _bootstrapper = AbpBootstrapper.Create<PortalServiceModule>();

        _bootstrapper.Initialize();

        return true;
    }

    public bool Stop(HostControl hostControl)
    {
        _bootstrapper.Dispose();
        return true;
    }
}

Now i can run the background jobs from the windows service, although i will try and reduce the dependencies and duplicate code.


How do I plot this dataframes first and second columns as coordinates and the third as the size of the points?

Question: The dataframe is as follows:

col1
col2
col3

0.005
0.004
359

0.007
0.004
320

0.009
0.004
316

0.005
0.008
359

0.007
0.008
320

0.009
0.008
316

I would also like to label each point on the graph with its corresponding size from column three.
Comment: https://stackoverflow.com/questions/67353268/how-do-i-plot-the-first-column-of-a-dataframe-as-the-x-axis-the-second-column-a
Answer: With ggplot2:
df <- read.table(text="
col1    col2    col3
0.005   0.004   359
0.007   0.004   320
0.009   0.004   316
0.005   0.008   359
0.007   0.008   320
0.009   0.008   316",header=T)

library(ggplot2)
ggplot(df,aes(x=col1,y=col2,label=col3)) + geom_point(aes(size=col3)) +
                                           geom_text(vjust=2)


Deploy pivotal cloud foundry on AWS

Question: i got the following error when i tried to create the stack. have no idea hot to fix this.
enter image description here
what's the problem
Comment: Please click the link to open the details
Answer: You should file a Pivotal Support help ticket, they are pretty quick and there would be a support engineer who can guide you through the installation.
I have done this myself in the past and I am quite satisfied with the support that I received
Answer: You can raise a ticket in Pivotal Support

Deploy pivotal cloud foundry on AWS


Rank of block matrix structure

Question: Given matrix $B$ of size $n\times n$ with rank $r>1$.
Denote $S_r$ as $n\times n$ permutation matrix that shifts each row one step down while elevating last row to first.
Denote $S_c$ as $n\times n$ permutation matrix that shifts each column one step right while elevating last column to first.
Form $n^2\times n^2$ block matrix $W$ with $i,j$th block of matrix being $B_{ij}=S_r^{j-1}BS_c^{i-1}$. (Note $j-1$ is for row rotations).
Total number of matrices $B_{ij}$ is $n^2$.
$W$ satisfies row sum = column sum.
Is $\mathsf{rank}(W)\geq rn$?
Comment: Is $B_{ij}$ the element at row $i$ and column $j$, or the other way around?
Comment: ah i see there is a confusion
Comment: Your edit didn't help. Does $B_{ij}$ go in the $i$-th block row of $W$ or the $j$-th block row of $W$? The usual convention is that the first index indicates the row.
Comment: @VictorLiu $i$th.
Answer: No. Let $B$ be the matrix of all $1$'s. Then $B_{ij}$ is always the matrix of all $1$'s, and so is $W$. In this case, $r=1$, but $\operatorname{rank}(W) = 1$.
Comment: Is example obvious for $r>1$?

Rank of block matrix structure


media library thumbnails missing for txt and pdf

Question: The http://drupal.org/project/media module cannot display the thumbnails of txt and pdf files in my installation. I'm not expecting a thumbnail of the actual txt or pdf content of course but a generic placeholder.
Display as list:

Display as thumbnail:

Note that:

no missing file (404) requests are being tracked in firebug
this also affects the media popup file chooser from within content editing
changing the theme doesn't solve this problem
inspecting the "empty thumbnail" with firebug doesn't show anything useful, the  is empty.
Answer: Using 7.x-2.x-dev instead of 7.x-1.1 solves this.
Answer: I think we've had this problem before when people have the 'Styles' module enabled (and it's sub-module File styles). The use of those modules is now highly discouraged and they should be disabled if possible. I don't know if this is the case however here.
Comment: I do have `styles` and `file_styles` in my path but they are both disabled.
Answer: You have to set the file type settings under configuration. If you click on the "File types" under the Media section you will see a list of default media types. The one you want to update is "Application (mulitpurpose)" or "Other". Click "Manage file display" and check the box next to Large filetype icon. You should do this at least for the "Preview" display. Save and reload and your problem should be cleared.


try() in R not continuing execution on exception

Question: I am new to R, and learning it online by trying out some simple code
I read try() allows to ignore any error and continue. So I tried this:

When I put log() inside try(), I was still getting the same output;

I was expecting something like:
Error in log(x) : non-numeric argument to mathematical function
[1] 10

But that 10 is not appearing in the output. Why is this so?
Answer: You need source("C:/Mahesh/workspaces/r/temp.r", echo = TRUE)


Statistical anecdotes

Question: I was inspired by this question on statistics jokes and wanted to start a similar thread on statistical anecdotes.
These are famous stories from the past where statistics have been used to solve some problem in an interesting way.
I like to tell my students these stories in between, as I feel that they sometime spark interest in people that are less motivated to learn about statistics.
I'll start with three of my favorites:
Poincaré and the baker
The German tank problem
The lady tasting tea
Comment: I don't feel qualified to judge whether this is in the scope / norms of CV, but the question certainly sounded useful to me, e.g. for teaching stats.
Comment: This comment by Glen_b is on the mark: "It's probably better if it stays around in any case, so that people who are inspired as you were and try do the same again will be likely to turn this one up in a search." I came across this thread because I intended to post something similar and wanted to avoid duplicating any existing threads. I would've put out  a call for interesting historical anecdotes, in the same vein as the one about Wald and the bullet holes in the Allied planes. Because this was left up, I now know that it could be considered off-topic for StackExchange.
Comment: I'm voting to close this question as off-topic because it does not fit the SE model of posting questions that ought to have objectively recognizable good answers.
Comment: @whuber Ok, I will take the question down. But what makes this any different from the statistics joke question? Why was that allowed in the first place? Also what would be the correct SE venue for such a question? Academia perhaps?
Comment: It's not at all different from the joke question--and that's precisely why it was closed.  Please see my comment to that question at http://stats.stackexchange.com/questions/1337/statistics-jokes#comment39045_1337 . I believe this kind of question is not on-topic anywhere on SE.
Comment: The standards across the entirety of the SE network are changed from what they were in the early days. Many SE sites have old posts that break the present rules or in some cases the presently accepted interpretation of what were already existing rules back then (and such posts would normally close retrospectively on that basis) but some are retained for essentially "historical reasons". New posts that break the same rules cannot survive. It would be nice if we could add a "Post Notice" to that effect on those old posts (to stop them from encouraging others), but I don't see a way to achieve it
Comment: Thanks for the clarification @Glen_b . Is the correct action now for me to delete the post or should it remain on hold indefinitely?
Comment: It shouldn't be necessary for you to do anything; but you couldn't even if you wished to because there's an upvoted answer. It's probably better if it stays around in any case, so that people who are inspired as you were and try do the same again will be likely to turn this one up in a search. If there's reason to delete it, that can be done later. Since it's on hold, some users can vote to delete it after 2 days, but the existence of upvotes on both question and answer means none of the autodelete processes will apply.
Comment: Thanks @Glen_b I just noticed that I couldn't delete it. I am fairly new on SE w.r.t. answering and submitting questions. Hope this serves it's purpose.
Comment: I would add that I think it's an excellent question and it would be likely to draw some interesting and useful answers, but it doesn't fit SE's model.
Comment: Ok thanks again @Glen_b I'll leave a comment here if I find another venue for this question!
Comment: @GuðmundurEinarsson try reddit! The statistics and ML communities there are very active and have some amazing people. See https://www.reddit.com/r/statistics and https://www.reddit.com/r/machinelearning
Comment: Thanks @PedroTabacof I will consider that. I just felt that the crowd here is more relevant and this community is much more active.
Answer: Not sure if this is too serious, but there is the Berkeley gender bias case illustrating Simpson's paradox. A very recent nearly identical case illustrating the problem seems a PNAS study on gender bias in research funding, discussed, e.g., in this Science news article.
Comment: This is completely relevant! This is a very interesting example.

Statistical anecdotes


Convert CSV file contents to Markdown

Question: Background
The objective is to read from a CSV file and write the contents in a Markdown table format.
The application uses the R engine Renjin, which does not support knitr, kable, or pandoc.
Problem
The write.table command has an eol option, but no corresponding sol option. Thus for the following:
f <- read.csv('planning.csv')
write.table(
   format(f, digits=2), "",
   sep="|", row.names=F, col.names=F, quote=F, eol="|\n")

The output appears as follows:
Geothermal|1250.0|Electricity|0.0|
Houses|  13.7|Shelter|4.2|
Compostor|   1.2|Recycling|0.2|

But each line should appear with a | prefix, as follows:
|Geothermal|1250.0|Electricity|0.0|
|Houses|  13.7|Shelter|4.2|
|Compostor|   1.2|Recycling|0.2|

It should be possible to do something like (note the extra eol pipe):
write.table(
       format(f, digits=2), "",
       sep="|", row.names=F, col.names=F, quote=F, eol="|\n|")

Then capture everything as a string, concatenate a leading pipe, and finally trim the extraneous ending pipe. That is, fix the problems with the output that'll resemble:
Geothermal|1250.0|Electricity|0.0|
|Houses|  13.7|Shelter|4.2|
|Compostor|   1.2|Recycling|0.2|
|Fire Station|  -9.6|Protection|0.5|
|Roads|   0.0|Transport|0.9|
|

Such string manipulation doesn't seem very R-like, though.
Question
What is the most efficient way to transmogrify a CSV file into a Markdown format without relying on third-party libraries?
The Markdown flavour in question looks like:
|Header|Header|Header|
|---|---|---|
|Data|Data|Data|
|Data|Data|Data|

Hints for how to write only the header data and the table header separator are also welcome.
Comment: There's [significant discussion](https://groups.google.com/forum/?nomobile=true#!searchin/pandoc-discuss/csv%7Csort:relevance/pandoc-discuss/znGQ62WpWrg/lFpgTPZzGgAJ) about CSV Tables on the pandoc google group, including a pandoc filter. I don't know the state of it, but you might check it out.
Comment: apply(), lapply(), sapply(), mapply(), vapply() are all functions from the base package, which is always available from within Renjin.
Answer: You can write your own version of kable if you like; it's mostly just paste.
x <- read.csv(system.file('misc', 'exDIF.csv', package = 'utils'))

md_table <- function(df){
    paste0('|', paste(names(df), collapse = '|'), '|\n|',
           paste(rep('---', length(df)), collapse = '|'), '|\n|',
           paste(Reduce(function(x, y){paste(x, y, sep = '|')}, df), collapse = '|\n|'), '|')
}

cat(md_table(x))
#> |Var1|Var2|
#> |---|---|
#> |2.7|A|
#> |3.14|B|
#> |10|A|
#> |-7|A|

cat(md_table(head(mtcars)))
#> |mpg|cyl|disp|hp|drat|wt|qsec|vs|am|gear|carb|
#> |---|---|---|---|---|---|---|---|---|---|---|
#> |21|6|160|110|3.9|2.62|16.46|0|1|4|4|
#> |21|6|160|110|3.9|2.875|17.02|0|1|4|4|
#> |22.8|4|108|93|3.85|2.32|18.61|1|1|4|1|
#> |21.4|6|258|110|3.08|3.215|19.44|1|0|3|1|
#> |18.7|8|360|175|3.15|3.44|17.02|0|0|3|2|
#> |18.1|6|225|105|2.76|3.46|20.22|1|0|3|1|

Rewrite the second line to handle justification based on type if you like.
Answer: Since you want to put it into markdown, I think it's safe to say that the table size is manageable, so performance is not a factor. (Edit #3: I had some small bugs relating to presence of row names, so to simplify things I'm going to remove them completely from the sample data.)
mtcars$rowname <- rownames(mtcars)
rownames(mtcars) <- NULL
mtcars <- mtcars[,c(ncol(mtcars), 1:(ncol(mtcars)-1))]
head(mtcars)
#             rowname  mpg cyl disp  hp drat    wt  qsec vs am gear carb
# 1         Mazda RX4 21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
# 2     Mazda RX4 Wag 21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
# 3        Datsun 710 22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
# 4    Hornet 4 Drive 21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
# 5 Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
# 6           Valiant 18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

Now the work:
dashes <- paste(rep("---", ncol(mtcars)), collapse = "|")
txt <- capture.output(
  write.table(mtcars, stdout(), quote = FALSE, sep = "|", row.names = FALSE)
)
txt2 <- sprintf("|%s|", c(txt[1], dashes, txt[-1]))
head(txt2)
# [1] "|rowname|mpg|cyl|disp|hp|drat|wt|qsec|vs|am|gear|carb|"
# [2] "|---|---|---|---|---|---|---|---|---|---|---|---|"
# [3] "|Mazda RX4|21|6|160|110|3.9|2.62|16.46|0|1|4|4|"
# [4] "|Mazda RX4 Wag|21|6|160|110|3.9|2.875|17.02|0|1|4|4|"
# [5] "|Datsun 710|22.8|4|108|93|3.85|2.32|18.61|1|1|4|1|"
# [6] "|Hornet 4 Drive|21.4|6|258|110|3.08|3.215|19.44|1|0|3|1|"

If you are concerned about alignment, you can check for characters (and perhaps others, over to you). This uses the alignment row of the markdown table format:
(ischar <- vapply(mtcars, is.character, logical(1)))
# rowname     mpg     cyl    disp      hp    drat      wt    qsec      vs      am    gear    carb
#    TRUE   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE   FALSE
dashes <- paste(ifelse(ischar, ":--", "--:"), collapse = "|")
txt <- capture.output(write.table(mtcars, stdout(), quote = FALSE, sep = "|", row.names = FALSE))
txt2 <- sprintf("|%s|", c(txt[1], dashes, txt[-1]))
head(txt2)
# [1] "|rowname|mpg|cyl|disp|hp|drat|wt|qsec|vs|am|gear|carb|"
# [2] "|:--|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|"
# [3] "|Mazda RX4|21|6|160|110|3.9|2.62|16.46|0|1|4|4|"
# [4] "|Mazda RX4 Wag|21|6|160|110|3.9|2.875|17.02|0|1|4|4|"
# [5] "|Datsun 710|22.8|4|108|93|3.85|2.32|18.61|1|1|4|1|"
# [6] "|Hornet 4 Drive|21.4|6|258|110|3.08|3.215|19.44|1|0|3|1|"

And when you're finally ready to save, use cat(txt2, file = "sometable.md") (or writeLines).
Edit #1: note that the other suggested answers (including mine above) do not address pipe symbols within the content:
mtcars$mpg[1] <- "2|1.0"
ischar <- vapply(mtcars, is.character, logical(1))
dashes <- paste(ifelse(ischar, ":--", "--:"), collapse = "|")
txt <- capture.output(write.table(mtcars, stdout(), quote = FALSE, sep = "|", row.names = FALSE))
txt2 <- sprintf("|%s|", c(txt[1], dashes, txt[-1]))
head(txt2, n = 3)
# [1] "|rowname|mpg|cyl|disp|hp|drat|wt|qsec|vs|am|gear|carb|"
# [2] "|:--|:--|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|"
# [3] "|Mazda RX4|2|1.0|6|160|110|3.9|2.62|16.46|0|1|4|4|"
###                ^ this is the problem

You can escape it manually on all character (or add in factors, too) columns:
ischar <- vapply(mtcars, is.character, logical(1))
mtcars[ischar] <- lapply(mtcars[ischar], function(x) gsub("\\|", "|", x))
dashes <- paste(ifelse(ischar, ":--", "--:"), collapse = "|")
txt <- capture.output(write.table(mtcars, stdout(), quote = FALSE, sep = "|", row.names = FALSE))
txt2 <- sprintf("|%s|", c(txt[1], dashes, txt[-1]))
head(txt2, n = 3)
# [1] "|rowname|mpg|cyl|disp|hp|drat|wt|qsec|vs|am|gear|carb|"
# [2] "|:--|:--|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|"
# [3] "|Mazda RX4|2|1.0|6|160|110|3.9|2.62|16.46|0|1|4|4|"
###                ^^^^^^ this is the pipe, interpreted correctly in markdown

This doesn't work well when the pipe is within a code block, though a workaround was suggested here: https://stackoverflow.com/a/17320389/3358272
At this point, as @alistaire suggested, you're somewhat reimplementing knitr::kable. For that matter, just grab knitr/R/table.R) and use kable_markdown which does the pipe-escaping for you. It takes a character matrix, not a data.frame, so kable_markdown(as.matrix(mtcars)). You can't just grab the single function as it uses several helper functions also in that file. You can certainly prune some functions, including kable itself which requires functions in other files.
Edit #2: since you said renjin doesn't support *apply functions (a comment suggests that is incorrect, but I'll continue for the sake of argument), here's a for-loop implementation that includes alignment and |-escaping:
mtcars$mpg[1] <- "2|1.0" # just a reminder that it's here
dashes <- rep("--:", length(mtcars))
for (i in seq_along(mtcars)) {
  if (is.character(mtcars[[i]]) || is.factor(mtcars[[i]])) {
    mtcars[[i]] <- gsub("\\|", "|", mtcars[[i]])
    dashes[i] <- ":--"
  }
}
txt <- capture.output(write.table(mtcars, stdout(), quote = FALSE, sep = "|", row.names = FALSE))
txt2 <- sprintf("|%s|", c(txt[1], paste(dashes, collapse = "|"), txt[-1]))
head(txt2, n = 3)
# [1] "|rowname|mpg|cyl|disp|hp|drat|wt|qsec|vs|am|gear|carb|"
# [2] "|:--|:--|--:|--:|--:|--:|--:|--:|--:|--:|--:|--:|"
# [3] "|Mazda RX4|2|1.0|6|160|110|3.9|2.62|16.46|0|1|4|4|"

For the record, my *apply and for-loop implementations are effectively the same performance, while @alistaire's solution is over twice as fast (with mtcars):
Unit: microseconds
              expr      min        lq      mean    median        uq      max neval
     apply_noalign  917.881  947.9665 1031.9288  971.3060 1041.5050 1999.499   100
       apply_align  945.960  975.1350 1083.2856  995.7390 1063.7500 3523.101   100
 apply_align_pipes 1110.429 1148.5360 1255.5460 1176.9815 1275.2600 1905.778   100
           forloop 1188.104 1217.0950 1309.2549 1261.2205 1342.3600 2979.010   100
         alistaire  451.830  473.7105  511.5778  496.1370  518.5645  827.443   100
   alistaire_pipes  593.687  626.6900  718.6898  652.7645  700.5360 5460.970   100

I used his original function for alistaire and added a simple gsub for alistaire_pipes. There may be a more efficient way to do it, but (a) simple/straight-forward is good, and (b) I think your tables will be small enough where true performance will not be a driving force.
Comment: I did not initially see the restrictions against `*apply` functions, sorry about that. I'm glad the first works. (It's actually quite straight forward to replace the `apply`s with `for` loops, if necessary.)
Comment: Your statement in **Edit #2** is incorrect: Renjin fully supports `*apply` functions. The answer provided by @alistaire, which uses the functional `Reduce()`, also works fine in Renjin. Finally, the `mtcars` data set doesn't have character columns so your use of `ischar` is superfluous here.
Comment: The OP said it, I trusted that and did not verify, my bad (I don't use renjin so do not know). I changed one column to character (3rd code block), so `ischar` is *not* superfluous overall, though I see I missed a bug in the 2nd block that would have made that clearer.
Comment: Dave Jarvis, I fixed some minor bugs (prodded by @mjkallen's comment), they may or may not be a factor in your use. Also, based on his comment, if the `*apply` functions work for you, I suggest you actually base off of one of my `apply` solutions or alistaire's function.

Convert CSV file contents to Markdown


Como pegar o CSS que foi defido em outra página?

Question: Olá, sou novo em desenvolvimento front-end, desculpa se a minha pergunta é meio tosca, mas o que eu gostaria de saber é se é possível pegar o arquivo de CSS que foi definido em uma página e usa esse mesmo CSS nas demais. Esse projeto utiliza Velocity, portanto estava pensando em algo do tipo:
Um arquivo chamado style.vm por exemplo

       #IF(cliente == 1)
       <link rel="stylesheet" type="text/css" href="cliente1.css">
       #ELSE IF (cliente == 2)
       <link rel="stylesheet" type="text/css" href="cliente2.css">
       #ELSE IF (cliente == 3)
       <link rel="stylesheet" type="text/css" href="cliente3.css">
       #ELSE IF (cliente == 4)
       <link rel="stylesheet" type="text/css" href="cliente4.css">
       .
       .
       .
       #END

e nos outros .VMs e .JSPs pegar o css que foi definido nesse arquivo, pois não queria copiar todos esses IFs para cada arquivo do projeto.
Eu tenho a opção de usar JQuery caso seja necessário, pois todas as páginas desse projeto importam JQuery.
Comment: A resposta do Daniel Omine respondeu a sua dúvida? Se sim, marque a reposta dele como a solução. Saiba como no [tour](http://pt.stackoverflow.com/tour).
Answer: Poderia fazer do seguinte modo
<link rel="stylesheet" type="text/css" href="cliente[$cliente].css">

Só isso basta, não precisaria da condicional if.
obs: no lugar de [$cliente], coloque a variável cliente do "velocity"
Answer: Geralmente quando um cliente esta logado no sistema ele pega essas informações na session, o que você poderia fazer era definir diferentes temas para o sistemas, e atribuir um tema que chamaria o arquivo css. Para isso acho que você deveria utilizar de alguma linguagem para o back-end da aplicação caso não use.
Armazenando assim os dados em um banco de dados você poderia imprimir no link do estilo o tema do usuário que esta acessando.


Powershell Get-ODBCDSN error - Timeout when waiting for another concurrent operation to complete

Question: After some database issues the night before, multiple servers (separate from the database servers) are getting the following error when just running the 'Get-ODBCDSN' function from a powershell console:

Get-OdbcDsn : Timeout when waiting for another concurrent operation to complete.
  At line:1 char:1
  + Get-OdbcDsn
  + ~~~~~~~~~~~
      + CategoryInfo          : ResourceBusy: (MSFT_OdbcDsnTask:Root/Microsoft/...SFT_OdbcDsnTask) [Get-OdbcDsn], CimExc
     eption
      + FullyQualifiedErrorId : MI RESULT 27,Get-OdbcDsn

I have found nothing online about this error. I've tried tweaking the one switch that has any correlation '-ThrottleLimit' but using various values doesn't help.
I believe, that a reboot will resolve this error.  However, this function is customer-impacting and needs to 'just work'.  I use the module to get all current User DSNs then delete them.  Then I rebuild with the New-ODBCDSN function. All of these functions are called in the login script.  Some 20 to 50 users may login to a server at once during peak times. Both Get and New-ODBCDSN commands exhibit the same error.
In order to work around the issue in the meantime, I have stopped using the function and switched to setting up DSNs through the registry HKCU:\Software\ODBC.INI\ODBC\etc,etc..  This is not my preferred option.
I'm looking to understand the problem, identify resolutions, or find another workaround.
Comment: Out of curiosity, why are you doing this in the first place? It seems an odd thing to delete and recreate the DSN's on every user log on.
Comment: Customer DSN names are created based on some custom factors, version of the software, customer id, etc to specifically identify their database.  When a customer changes versions they get a new db and DSN name. Also they may have created their own DSNs (which we don't want).  In order to keep their DSNs set correctly, we wipe all User DSNs out and recreate them to their currently assigned settings.
Answer: I know i am a year late by a year, however hopefully this will help somebody else out. I was just getting the same error "Get-OdbcDsn : Timeout when waiting for another concurrent operation to complete. At line:1" mine was because MSFT_OdbcDsnTask was in use by trying to add a dsn through PowerShell and got locked up, and killing the command did nothing. You probably noticed that you can still use "get-obdcdriver", at least i did.
I was able to fix this by going to start > resource Monitor > CPU > Associated handles and typing ODBC in the search. I then was able to see wmiprvse.exe in the search result. After killing one of the wmiprvse.exe then my list was blank and i was able to run the command. Note i would make sure you kill all if they do not clear, you are basic pulling the rug out from that exe. Also anything that it is trying to be completed will be lost.
Comment: That solution worked for me. It seems that the underlying server the ODBC connection pointed to got locked up.  Then this Get-ODBCDSN fails even after the DB recovers, and will not work again until a reboot or by performing this wmiprvse.exe kill.  Thanks again~


Number of passcodes with primes and >= 1 two

Question:
A passcode is either 4, 5, 6 digits and has primes from {$2, 3, 5, 7$}. How many passcodes are there such that they are from the primes above and have at least one 2?

Confusion: When I have something like how many possible codes that are 4 digits long and are from {0->10}

Is something like $2222$ counted 4 times?
Comment: To avoid double counting you might create a tree of dependent choices which begins with the position of the leftmost `2`.  For each position to the left, the options are limited to `3,5,7`, but four options are available for each position to the right of this "first" `2`.
Answer: If you ignore the constraint then there are $$4^4+4^5+4^6=5376$$
The number of  passwords which don't use $2$ is $$3^4+3^5+3^6=1053$$
You want the complement, hence $$5376-1053=\boxed {4323}$$
To your side question:  No.  You shouldn't count $2222$ four times.
Comment: If I have _ _ _ _. each has say 4 options, then you have $4^4$ passcodes, but does this count $5555$ 4 times as well?
Comment: No.  It only counts it once.  I get $4^4$ by saying "you have $4$ choices for the first slot, $4$ for the second, and so on".   What makes you think we should count $5555$ more than once?
Comment: Try two slots.  Then I claim there are $4^2=16$.  Indeed, they are $22,23,25,27,32,33,35,37,52,53,55,57,72,73,75,77$.  Note that we do not count $55$ twice.


Problema com Proxy (corporativo) utilizando o dev_appserver.py - GAE

Question: Olá,
Estou com um problema na utilização do Google App Engine e gostaria de saber se alguém pode me ajudar.
Na minha máquina tenho instalado:

Windows XP
Python 2.7.6
Google App Engine 1.9.17 (é o mais novo)

Fiz uma aplicação exemplo e não estou conseguindo fazer com que ela execute localmente na minha máquina. Pesquisando no Google identifiquei que o problema é o Proxy da empresa. Veja o mesmo problema no artigo https://www.marshut.net/iwuhqp/dev-appserver-py-cannot-connect-to-some-http-endpoint.html
Consigo chamar o dev_appserver.py [aplicação], mas quando acesso http://localhost:8080 o erro é apresentado. Consigo realizar o Deploy (push) para o a nuvem sem erro.
Na minha casa tenho a mesma versão do Python e Google App Engine, somente o Windows que é o Vista. Consigo realizar os meus testes sem erros, tanto ambiente de desenvolvimento quanto o deploy.
Não consigo desabilitar o Proxy na empresa, pois questões da política de segurança.
O que posso fazer para ter o ambiente de desenvolvimento funcional na minha estação de trabalho? Existe algum workaround?
Abraço e obrigado pela ajuda!
Comment: Podes adicionar o traceback do erro?
Answer: O grande problema eh que seu CMD nao esta com proxy ativo, voce precisa levantar o http_proxy dentro do seu CMS:

Acesse o painel de controle e clique no Sistemas
Vai em avancado (tab), click em variável de ambiente
Click em nova
Adicione http_proxy com as informacoes do seu proxy

Isso eu achei em um blogpost para Windows XP, segue o link de referencia:
http://kaamka.blogspot.ca/2009/06/httpproxy-environment-variable.html
Comment: Obrigado pelo apoio. Realizei as configurações informadas, mas mesmo assim o erro persiste.


Markov Decision Process - Discounted return

Question: I was reading this article about Discounted return (in the context of MDP): http://deeplizard.com/learn/video/a-SnJtmBtyA
I got the section:
Now, check out this relationship below showing how returns at
successive time steps are related to each other.
We'll make use of this relationship later.

[please use the image url (below) if it doesn't appear here][1]

[1]: https://i.stack.imgur.com/KgOML.png

The extract shows 3 maths formulas. I have 3 questions on this:
1- I noticed that Rt + 3 is occurring twice, in the time steps. Could this be a typo error? I.e. shouldn't the next time step be Rt + 4? Or is this correct? If so, then it doesn't make sense to me.
2- I didn't understand how in the third formula at the bottom, we changed it to yGt+1?
3- Why would the discount y be increased exponentially (i.e. its power increases by one) with every time step? Doesn't that seem like a dramatic increase in the discount with every time step (as opposed to maybe multiply the the discount by an increasing co-efficient that is equal to the time step value)?
Many thanks in advance for any help.
Answer:
this is a typo, it should be $(t+4)$
The definition of $G_t$ is
$$
G_t = \sum_{k=1}^{\infty} \gamma^k R_{t+k+1}
$$
this is a function of $t$. So if we wanted to find $G_{t+a}$ for some number $a$, we would have:
$$
G_{t+a} = \sum_{k=1}^{\infty} \gamma^k R_{(t+a) +k+1} = R_{t+a+1} + \gamma R_{t+a+2} + \gamma^2 R_{t+a+3} + \dots
$$
The choice of gamma determines how much you want to care about actions in the past/future. If you take $\gamma=0.5$, then you are saying that you only care about rewards that are coming 1 step in the future half as much as you care about a reward right now. If you want to care more about future rewards, you increase $\gamma \to 1$. It's not that dramatic depending on the choice of $\gamma$, and in most applications you don't want to give too much weight to very distant returns, using an exponential discounting gives us that.
Comment: Many thanks @dimebucker for the answer. Makes sense. Also for the 3rd question, I forgot to take into account that gamma is between 0 and 1. Meaning adding exponential to it would reduce the discount value rather than increase it (e.g.  0.5 squared = 0.25). I couldn't upvote your answer because I don't have enough reputation score. Thanks again :)
Comment: yes, but for example $\gamma=0.99$ describes our desire to consider future rewards as being as important as today, and as $\gamma \to 0$ then we care much more about rewards right now (note that $\gamma=0$ then we are just looking at a greedy approach)


How can I make this script better?

Question: I was wondering if there is a better way to create a file other than appending each line to the file. I did it this way so I can preserve readability, however, there is no indentation. Is there a way to create a file and input multiple lines at once?
if [ -d "/srv/www/$1" ]; then
 echo "Domain name already exists!"
else
 mkdir -p /srv/www/$1/public_html;
 mkdir -p /srv/www/$1/logs;
 echo "<VirtualHost>" > /etc/apache2/sites-available/$1
    echo "ServerAdmin support@$1" >> /etc/apache2/sites-available/$1
    echo "ServerName $1" >> /etc/apache2/sites-available/$1
    echo "ServerAlias www.$1" >> /etc/apache2/sites-available/$1
    echo "DocumentRoot /srv/www/$1/public_html/" >> /etc/apache2/sites-available/$1
    echo "ErrorLog /srv/www/$1/logs/error.log" >> /etc/apache2/sites-available/$1
    echo "CustomLog /srv/www/$1/logs/access.log combined" >> /etc/apache2/sites-available/$1
 echo "</VirtualHost>" >> /etc/apache2/sites-available/$1
 a2ensite $1

Answer: Use a heredoc.
cat > /etc/apache2/sites-available/"$1" << EOF
<VirtualHost>
ServerAdmin support@$1
...
EOF

Comment: 8 minutes until you get best answer
Answer: Heredoc as stated in answer #1 or just have your echo go across multiple lines
echo "line 1
line2
line3" > file

Answer: If it's readability you're after, have you considered splitting it out into multiple files? Have a "template" file that you can edit that is copied over with your shellscript.
## /path/to/vhtemplate
<VirtualHost>
    ServerAdmin support@#1
    ServerName #1
    ServerAlias www.#1
    DocumentRoot /srv/www/#1/public_html/
    ErrorLog /srv/www/#1/logs/error.log
    CustomLog /srv/www/#1/logs/access.log combined
</VirtualHost>

Example script:
if [ -d "/srv/www/$1" ]; then
 echo "Domain name already exists!"
else
 mkdir -p /srv/www/$1/public_html;
 mkdir -p /srv/www/$1/logs;
 cp /path/to/vhtemplate /etc/apache2/sites-available/$1
 sed -i -e 's/#1/$1/' /etc/apache2/sites-available/$1
 a2ensite $1

Comment: I like this a lot, and I upvoted your answer. Using templates is a great idea too.

How can I make this script better?


MozyPro Reseller Alternative for Backing up SQL Servers (and files)

Question: We have a MozyPro reseller account and have been testing the service with around 10 customers. Our biggest requirement is to backup a SQL Server (2000/2005/2008/2008r2), personal/express, standard or enterprise; basically our customers have different SQL Server versions.
Requirements:

Be able to automatically (i.e. no SQL Agent involved) backup SQL Server (any version/edition) >= 2000, while the database is in use.
Have a reseller program where we can create users and configure the program for them, assign space (if needed) and bill them as we please.
Support any OS that the above SQL Servers support (Windows XP —> Windows Server 2003/8 -> W7)
Possibility to add other folders to the backup in case the user needs to backup an Images folder.

Truth is MozyPro (reseller) does all of the above, you can buy "licenses" and "gigabytes". Then assign them to any "group" (which they recommend you create per customer).
So you can have 5 customers, which will be "5 Groups". These "Groups" can have "MACHINES". Each machine is a License (can be desktop -cheaper, less features- or server).
Then to these licenses you assign Gigabytes of storage 1 by 1. (minimum 1gb).
Pros of MozyPro:

Licencing is very easy to handle, you purchase "resources" (licenses/GBs) and you assign them to your customers (groups). If a customer leaves, you can transfer those "resources" to another customer or even return them to Mozy (and they won't charge you the monthly fee anymore).
Pricing is "ok" if you have a small user base, because you only pay "as you go".
The software is "ok", kind of easy to set up, and you can have software profiles where you preconfigure the settings and save it as a profile. When setting up a customer you can assign a profile and the "client" will read those settings and be configured. That's nice.
The Control Panel for the reseller is also ok, you can do everything from there and it's more or less intuitive once you learn to use it, they will also share screen with you to teach you if needed.
When it works, it works fine, even restore process (which is a little bit weird). (see cons).

Cons of MozyPro:

Customer Service is very "friendly" (at least with us), but we have three customers (out of our 10 customers pilot-test) that have been unable to backup the SQL Server for 180 days (that's how much they have been trying to make it work). This means that 3 out of 10 have failed. These are three separate customers, with different computers in different places. Mozy Customer support have been unable to fix it (but have tried). They remotely connect there, install, touch, download logs, etc. Nothing works.
Restore works by a request: I need to restore X,Y,Z. And the server "prepares" a zip file (sometimes multiples if the data is too big) which you can download. It will contain your information. This is ok, but can be a tedious process to put the data back if it's a lot (paths are honored within the zip, so the C drive becomes a C folder, then a Program Files folder, etc.).

So Given the above, We're looking for an alternative. if it would have been one customer of 10, one can say: ok, the customer has something weird on his server… but 3 is too much. I can't imagine we have 100 customers where 30 are failing. This is critical backup medical information.
I've created a reseller program for JungleDisk, only to "see" what it was about, but I'm not convinced yet, I haven't seen the program but they don't mention SQL Backup in any product, and backing up a .BAK is not acceptable, because it requires SQL Agent or additional setup to create the BAK in the first place, we don't want that.
What other experiences/alternatives can you share?
Answer: No matter what system you use, you can't backup the SQL Server database files while the SQL Service is running and get a usable backup.  The data files will be out of sync and corrupt when you try and restore.  Even if the restore works some times there is no guarantee that it'll when when you need it to.
The only ways to backup the SQL Server database are:

Stop the SQL Server service and backup the files.
Use the BACKUP DATABASE job within the SQL Server to backup the database to a file that can then be uploaded to the backup service.

If the database is in full recovery then the transaction log needs to be backed up using the BACKUP LOG command or eventually the hard drive will fill up.  If the database is in simple recovery then the transaction log backups are not needed.
Comment: Can't VSS handle this? Obviously you can't just back up the database like a normal file, but shouldn't a "SQL aware" backup agent be able to take a proper online backup?
Comment: @minamhere MozyPro does that, it uses VSS (but apparently it has problems in some systems that they have been unable to identify and solve)
Comment: The transaction log backups still need to be done separately. VSS tells the database to flush all writes to the disk, so that a block level backup can be taken.  However SQL Server will still maintain the locks on the files.  VSS is used by Storage Array's like EMC, IBM, etc to take consistent block level copies.  It can't really be used by a file system backup tool however.
Answer: For people having the same requirements as the original poster, Zmanda Cloud Backup (ZCB) would neatly fit in. It offers hot/live backup of SQL server and supports both full and differential backups. For more information, please visit Zmanda Cloud Backup website
Disclaimer: I am the product manager of ZCB  at Zmanda.
Answer: We used Iron Mountain's LiveVault a few years ago. It was very good. It was able to do proper "Online" backups of our Exchange and SQL databases. We stopped using it because on issue on our server, unrelated to LiveVault.
It was (much?) more expensive than the alternatives, but worked very well. We didn't need to resell the service, so I'm not sure if/how that works.
Backing up online SQL and Exchange databases seems to be a feature that most services do not offer. The databases require their transaction logs to be properly backed up in order to correctly restore. Most "file level" backup tools, don't capture these correctly, so JungleDisk, etc are not able to handle this correctly.
Answer: Are you set on outsourcing to someone or would an inhouse hosted option be viable?
Comment: Outsourcing, we don't have/want the complexity of maintaining such structure.
Answer: You might want to look into eFolder. I don't use them personally, but they have a lot of cred within the managed services provider community. It'll backup SQL and the enterprise level software will probably kick the pants off of Mozy. I just recently dropped Mozy from our offerings to clients. I'm a fan of using something designed for business use rather than a consumer level product modified to run on servers.
Jungledisk is good, but the recovery times are not. If you have to recover 60GB worth of data, you're going to be waiting a while. Also, there's no native SQL backup in Jungledisk.
Check out Xilocore as well. I'm going through all of the stuff I got from them. I'm not 100% sure that they support SQL. It's something to ask them. The thing I like about them is that they offer a 100% restore guarantee. That's important.
If you want an AWESOME solution, look at Acxient. They have a really nice disaster recovery solution. It does local server failover with an offsite copy in the cloud as well. This isn't cheap but it's an amazing backup strategy.


Which distribution to choose when modeling variance of a normal distribution?

Question: I have a simple time series model where there is a single hidden variable $\lambda_t$ which changes over time: $\lambda_{t+1} \sim \mathcal{N}(\lambda_t,\sigma)$. The $\lambda_t$ is then used as a parameter for a Poisson distribution from which observations are made.
Since $\sigma$ here is also unknown, what distribution should be used to model it?
Comment: Why would it make sense for $\lambda$ be normal? $\lambda$ is necessarily non-negative, while a normal (necessarily) has some of its probability below 0. If you model a Poisson with small $\lambda$ you're going to be dealing with negative $\lambda$.
Comment: The way you wrote it $\sigma$ seems to be fixed, so why would you care about its distribution?
Comment: Forllowing @Glen_b, you should use a truncated-normal or a truncated Cauchy or a log-normal on the $\lambda_t$'s. And I would opt for an improper prior $\pi(\sigma)=1/\sigma$. But note that there is no unique choice for a prior distribution and thus you are entitled to any prior you feel reflects your prior information or lack thereof.
Comment: ... or a gamma, perhaps
Comment: @RichardHardy I don't care about the distribution of $\sigma$, but it is an unknown parameter, so I thought it needs to be modelled.
Comment: This model makes no sense. Maybe you better describe your process, then folks can help you come up with a more sensible model.
Comment: I'm totally confused on the setup of this problem, but you don't need to assume a distribution to estimate the value of $\sigma$. What's wrong with just using the sample standard deviation? Also keep in mind that the mean and variance are the same for the Poisson RV.
Comment: @wcampbell: $\lambda$ is a hidden variable (as stated in the OP) so I am not sure whether its sample standard variation can be observed...
Answer: If I understand correctly, $\lambda$ is fed to Poisson as its mean. I do not quite understand why you use a Gaussian to model its dynamics as a Gaussian includes negative support. Why not, say, pass $\lambda$ through some link function e.g., $\exp()$ and feed $\exp(\lambda)$ to Poisson ?
You said

I don't care about the distribution of σ, but it is an unknown parameter, so I thought it needs to be modelled.

So why don't you treat $\sigma$ as a parameter and determine it in M-step of EM algorithm ? No prior distribution is needed. If you really want a prior distribution over it, a Gamma distribution might be a good choice as its domain is positive real.


Optimizing Equation involving CDF and PDF

Question: Assume you have two random variables, $v_b \sim N(\mu_b,\sigma_b^2)$ and $v_s \sim N(\mu_s,\sigma_s^2)$ and let $\rho$ be the correlation between them. Denote by $F$ and $f$ the CDF and PDF of the variables respectively, and assume I have the following optimization problem: $$\max_{\pi}F_{v_s}(\pi)(\mu_b-\pi)-\rho\sigma_b\sigma_sf_{v_s}(\pi)$$ where $\pi$ is the variable of interest. I am stuck on the approach - first, the second order conditions are of ambiguous sign. I'm also not convinced there exists a closed-form expression for $\pi$ after trying for several hours to solve it. How would I go about approximating solutions numerically? Can this even be done?
Comment: I would ask this question as: Consider a random variable $Y \sim N(\nu,\tau^2)$ with CDF $F$ and PDF $f$. Let $\mu$ and $c$ be other real numbers, which can be thought of as the mean of another normal variable $X$, and its covariance with $Y$. How can one maximize
$$\max_z[(\mu-z)F(z)-c f(z)]?$$There are indeed no closed-form solutions, but from this presentation it should be easier to apply standard numerical techniques.
Answer: If getting more familiar with the (numeric rather than exact) solutions is the objective, then looking at some dynamic graphics would be helpful.  This can certainly be done in R and Mathematica (and probably several other languages).  Consider the following Mathematica code for looking at where the maximum occurs and the shape of the function being maximized:
<code>Manipulate[sol = Quiet[FindMaximum[g[zz, \[Mu]b, \[Sigma]b, \[Mu]s, \[Sigma]s, \[Rho]], {{zz, zz0}}]];
 Show[Plot[g[z, \[Mu]b, \[Sigma]b, \[Mu]s, \[Sigma]s, \[Rho]], {z, -10, 10},
   PlotRange -> All, PlotLabel -> "z = " <> ToString[zz /. sol[[2]]] <> ", Max = " <>   ToString[sol[[1]]]],
  ListPlot[{{zz /. sol[[2]], 0}, {zz /. sol[[2]], sol[[1]]}}, Joined -> True, PlotStyle -> Red]],
 {{\[Mu]b, 7}, -10, 10, Appearance -> "Labeled"},
 {{\[Sigma]b, 6}, 0.001, 10, Appearance -> "Labeled"},
 {{\[Mu]s, 0}, -10, 10, Appearance -> "Labeled"},
 {{\[Sigma]s, 1}, 0.001, 10, Appearance -> "Labeled"},
 {{\[Rho], 0}, -1, 1, Appearance -> "Labeled"},
 {{zz0, 0, "Starting value"}, -10, 10, Appearance -> "Labeled"},
 TrackedSymbols :> {\[Mu]b, \[Sigma]b, \[Mu]s, \[Sigma]s, \[Rho], zz0},
 Initialization :> (g[z_?NumericQ, \[Mu]b_?NumericQ, \[Sigma]b_?NumericQ, \[Mu]s_?NumericQ, \[Sigma]s_?NumericQ, \[Rho]_?NumericQ] :=
    CDF[NormalDistribution[\[Mu]s, \[Sigma]s], z] (\[Mu]b - z) - \[Rho] \[Sigma]b \[Sigma]s PDF[NormalDistribution[\[Mu]s, \[Sigma]s], z])]
</code>
Comment: Thank you, this is amazing


Can the official fastest lap of a race be set in the first lap in F1?

Question: Actually there are many reasons that make the cars slower in the first lap of a race.

Cars accelerate from 0 kph
Traffic is at maximum
Tyres are not at ideal temperature (still soft tyres can perform at peak)
Fuel tanks are full

On the other hand, the first lap is shorter than other laps on most circuits, because the start line is usually ahead of the finish line. In addition, if it starts raining in the 1st or 2nd lap, the fastest lap can be easily set in the first lap.
I'm questioning whether this lap can be officially recorded as the fastest lap. Since the distance is different from other laps and also different for each car, I would expect the first lap to be excluded from fastest lap records.
I checked the 2019 Formula One Sporting Regulations, but couldn't find anything relevant. The following part in Article 6.4 is about the 1-point award for the fastest lap, again nothing relevant.

In addition to the above, one point will be awarded to the driver who
  achieved the fastest lap of the race and to the constructor whose car
  he was driving, provided the lap time was achieved without incurring a
  penalty and he was in the top ten positions of the final race
  classification (see Article 45). No point will be awarded if the
  fastest lap is achieved by a driver who was classified outside the top
  ten positions.
Answer: Yes, the first lap is valid for setting the fastest lap.
Before anything else, it's worth mentioning that the starting line and the finishing line are not at the same position because the finishing line must be in front of the race control tower. Since the full grid takes quite some space (specially with these ridiculously long cars we currently have), it normally ends up with the pole position car sitting way ahead of the race control tower, hence the starting line ahead of the finishing line. But if the main straight is long enough both lines can be at the same position, as in Interlagos. In some rare occasions the finishing line can be ahead of the starting line, as in the old Silverstone.
Back to the question, the fact that one finds no mention of the first lap in the regulations cannot be viewed as it being invalid: quite the opposite, if the first lap is not mentioned it should be valid. What I mean is if the first lap was not a valid lap for setting the fastest lap, then it should be explicitly mentioned as invalid in the regulations.
However, for avoiding an opinion based answer, let's back up this assumption with an official record: all we need is a FIA official record of a driver setting his/her fastest lap on the first lap. It's not easy to find one, but for instance Martin Brundle set his fastest lap in the first lap of the South African GP in 1992 (columns are: Pos/No/Driver/Car/Lap/Time):

Source: https://www.formula1.com/en/results.html, in the "fastest laps" table
This shows us that the first lap is a valid one for setting the fastest lap.
Therefore, when you ask "can the official fastest lap of a race be set in the first lap?" the answer is yes, it can. What's the real chance of this happening some day? Well, except for a downpour starting on the second lap, as you mentioned, the chance is pretty much zero.


Save body of a streaming future object to disk

Question: I'm using the PollyClient from rusoto, where I'm attemping to handle what I believe is a Future response of some type from the method synthesize_speech().
Here is a snippet of what I have that compiles and runs as-is:
extern crate rusoto_core;
extern crate rusoto_polly;

use std::default::Default;
use tokio::{io};

use rusoto_core::credential::{ProfileProvider, ProvideAwsCredentials};
use rusoto_core::Region;
use rusoto_core::ByteStream;
use rusoto_polly::{PollyClient, SynthesizeSpeechInput, Polly};
use std::fs::File;
use futures::AsyncBufRead;
use std::pin::Pin;
use futures::Future;

fn main() {

    let region = Region::UsEast1;
    let polly_client = PollyClient::new(region); // using default credenetials

    let speech_input = SynthesizeSpeechInput{
        engine: Option::from(String::from("neural")),
        language_code: None,
        lexicon_names: None,
        output_format: "mp3".to_string(),
        sample_rate: None,
        speech_mark_types: None,
        text: String::from("hello world!"),
        text_type: None,
        voice_id: String::from("Amy")
    };
    let mut response = polly_client.synthesize_speech(speech_input);

    println!("Now how to we handle this respones object...");

}

My goal is to save what should be MP3 binary data in the response.  I have not shared any of the compiler errors, as I've gone through quite a few different ones.  I'm brand new to Rust, and while was progressing with non-async code, this was my first encounter with it.
Do I need to wrap that synthesize_speech() method in a closure, such that I could wrap that in some kind of await block?
Any help or suggestions would be much appreciated.
Answer: I realize this is just a toy example, but after mimicing some of the tokio async pattern on main() from the rusoto documentation, was able to get this to work:
extern crate rusoto_core;
extern crate rusoto_polly;

// use rusoto_core::credential::{ProfileProvider, ProvideAwsCredentials};
use rusoto_core::{Region};
use rusoto_polly::{PollyClient, SynthesizeSpeechInput, Polly};
use std::fs;

#[tokio::main] // https://docs.rs/tokio/0.2.2/tokio/attr.main.html
async fn main() {

    let region = Region::UsEast1;
    let polly_client = PollyClient::new(region); // using default credentials

    let speech_input = SynthesizeSpeechInput{
        engine: Option::from(String::from("neural")),
        language_code: None,
        lexicon_names: None,
        output_format: "mp3".to_string(),
        sample_rate: None,
        speech_mark_types: None,
        text: String::from("Hello world, from Rust!"),
        text_type: None,
        voice_id: String::from("Amy")
    };

    match polly_client.synthesize_speech(speech_input).await {
        Ok(output) => match output.audio_stream{
            Some(audio_stream) => {
                println!("{}", audio_stream.len());
                fs::write("mp3/rstts.mp3", audio_stream).expect("Unable to write file");
            },
            None => println!("audio stream not found"),
        },
        Err(error) => {
            println!("error: {:?}", error);
        }
    }
}


Determining cause of high NFS/IO utilization without iotop

Question: I have a server that is doing an NFSv4 export for user's home directories.  There are roughly 25 users (mostly developers/analysts) and about 40 servers mounting the home directory export.  Performance is miserable, with users often seeing multi-second lags for simple commands (like ls, or writing a small text file).  Sometimes the home directory mount completely hangs for minutes, with users getting "permission denied" errors.
The hardware is a Dell R510 with dual E5620 CPUs and 8 GB RAM.  There are eight 15k 2.5" 600 GB drives (Seagate ST3600057SS) configured in hardware RAID-6 with a single hot spare.  RAID controller is a Dell PERC H700 w/512MB cache (Linux sees this as a LSI MegaSAS 9260).  OS is CentOS 5.6, home directory partition is ext3, with options "rw,data=journal,usrquota".
I have the HW RAID configured to present two virtual disks to the OS: /dev/sda for the OS (boot, root and swap partitions), and /dev/sdb for the home directories.
What I find curious, and suspicious, is that the sda device often has very high utilization, even though it only contains the OS.  I would expect this virtual drive to be idle almost all the time.  The system is not swapping, according to "free" and "vmstat".  Why would there be major load on this device?
Here is a 30-second snapshot from iostat:
Time: 09:37:28 AM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await svctm  %util
sda               0.00    44.09  0.03 107.76     0.13   607.40    11.27     0.89    8.27   7.27  78.35
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00    44.09  0.03 107.76     0.13   607.40    11.27     0.89    8.27   7.27  78.35
sdb               0.00  2616.53  0.67 157.88     2.80 11098.83   140.04     8.57   54.08   4.21  66.68
sdb1              0.00  2616.53  0.67 157.88     2.80 11098.83   140.04     8.57   54.08   4.21  66.68
dm-0              0.00     0.00  0.03 151.82     0.13   607.26     8.00     1.25    8.23   5.16  78.35
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00  0.67 2774.84     2.80 11099.37     8.00   474.30  170.89   0.24  66.84
dm-3              0.00     0.00  0.67 2774.84     2.80 11099.37     8.00   474.30  170.89   0.24  66.84

Looks like iotop is the ideal tool to use to sniff out these kinds of issues.  But I'm on CentOS 5.6, which doesn't have a new enough kernel to support that program.
I looked at Determining which process is causing heavy disk I/O?, and besides iotop, one of the suggestions said to do a "echo 1 > /proc/sys/vm/block_dump".  I did that (after directing kernel messages to tempfs).  In about 13 minutes I had about 700k reads or writes, roughly half from kjournald and the other half from nfsd:
# egrep " kernel: .*(READ|WRITE)" messages | wc -l
768439
# egrep " kernel: kjournald.*(READ|WRITE)" messages | wc -l
403615
# egrep " kernel: nfsd.*(READ|WRITE)" messages | wc -l
314028

For what it's worth, for the last hour, utilization has constantly been over 90% for the home directory drive.  My 30-second iostat keeps showing output like this:
Time: 09:36:30 PM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     6.46  0.20 11.33     0.80    71.71    12.58     0.24   20.53  14.37  16.56
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     6.46  0.20 11.33     0.80    71.71    12.58     0.24   20.53  14.37  16.56
sdb             137.29     7.00 549.92  3.80 22817.19    43.19    82.57     3.02    5.45   1.74  96.32
sdb1            137.29     7.00 549.92  3.80 22817.19    43.19    82.57     3.02    5.45   1.74  96.32
dm-0              0.00     0.00  0.20 17.76     0.80    71.04     8.00     0.38   21.21   9.22  16.57
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00 687.47 10.80 22817.19    43.19    65.48     4.62    6.61   1.43  99.81
dm-3              0.00     0.00 687.47 10.80 22817.19    43.19    65.48     4.62    6.61   1.43  99.82

Comment: maybe it's being used for swap? logs?
Answer: The easier approach is to upgrade your OS packages.
CentOS 5.7 can definitely use iotop. Red Hat backported per process IO accounting into the 2.6.18-144 kernels, and I began to see iotop working somewhere in 2011 via RPMForge packages. Red Hat made iotop part of the standard OS in 2012. On a 5.7 system...
[root@Tantalalicious ~]# cat /etc/issue
CentOS release 5.7 (Final)
Kernel \r on an \m

[root@Tantalalicious ~]# uname -a
Linux Tantalalicious 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@Tantalalicious ~]# iotop
Total DISK READ: 25.54 M/s | Total DISK WRITE: 87.03 K/s
  TID  PRIO USER<     DISK READ  DISK WRITE  SWAPIN      IO    COMMAND
31441 be/4 465         0.00 B/s    0.00 B/s  0.00 %  0.00 % -bash
31540 be/4 465         0.00 B/s    0.00 B/s  0.00 %  0.00 % dbc
22587 be/4 admin       0.00 B/s    0.00 B/s  0.00 %  0.00 % sh
22588 be/4 admin       0.00 B/s    0.00 B/s  0.00 %  0.00 % sh

Don't think this is a cop-out of an answer... But at this point, there's no reason to be running the older OS. EL 5.8 has stabilized, fixed a ton of bugs and gives you access to the profiling tool that you need (iotop). I'm assuming you've already modified your Linux I/O elevators and tuned the hardware RAID controller by now.
Comment: If you're using EL6.3, please also look at the tuned and tuned-adm parameters I mention [here](http://serverfault.com/questions/430955/openfiler-iscsi-performance/431112#431112) and [here](http://serverfault.com/questions/458339/overriding-default-scheduler-for-blkio-requests-in-cgroups/458342#458342). They will help tremendously.
Comment: Went ahead an upgraded to EL 6.3, which gives me all the tools I need an magically improves performance dramatically as well.  Too many variables changed with the OS upgrade to know exactly where the problem was, but it's all good now.
Answer: You can use lsof to list open files, it might help determine what is being used.
Look into general performance tuning, e.g., make sure write-back caching is enabled on the RAID controller, mount ext3 and nfs shares with noatime, tune rsize/wsize, don't mount nfs with noac unless you have to.


Oracle não exporta todas as tabelas

Question: Estou realizando um export pelo banco de dados em uma instancia 126.96.36.199.0 full, porém ao verificar o log percebo que as tabelas que não tem registro não são exportadas.
comando que estou usando para exportar o banco de dados
exp system/senha owner=freitechnew file=freitechnew.dmp log=freitechnew.log compress=N
Alguém já passou por esse problema?
Comment: Poderia adicionar a o comando **FULL=Y** ao seu comando. Porque na documentação da Oracle, eles dão um exemplo básico assim: `exp user@DB FULL=Y FILE=PATH_TO_DUMPFILE`
Comment: Ao exportar com o usuario ele retorna o erro `EXP-00008: Erro Oracle: 1406 encontrado` e não exporta, vou tentar resolver esse ora-1406 pra ver se consigo exportar assim, obrigado por enquanto.
Answer: se voce especifica o owner ele só pega este owner, para fazer do banco todo usar conforme abaixo.
exp system/senha file=freitechnew.dmp log=freitechnew.log  full=y compress=N

Oracle não exporta todas as tabelas


How to search Nagios by service?

Question: I'm trying to search for Current Load across all my hosts but Nagios Quick Search returns nothing.  I've found this thread about it possibly just searching host names or aliases.  Is there another way?
Answer: After reading the source, I found that you can call status directly with a service filter to get Nagios search by service:
/status.cgi?servicefilter=Current%20Load

Comment: See https://raw.githubusercontent.com/lemenkov/nagios/master/cgi/status.c for other possibilities.
Answer: Add to nagios/share/side.php next block after your navbarsearch(Quick Search):

<form method="get" action="<?php echo $cfg["cgi_base_url"];?>/status.cgi" target="<?php echo $link_target;?>">
 <fieldset>
   <legend>Service Search:</legend>
   <input type='text'   name='servicefilter' size='15' class="NavBarSearchItem">
 </fieldset>
</form>

How to search Nagios by service?


Datagrip, create custom template

Question: I was wondering if it is possible to create my custom SQL template in Datagrip.
 As users of Datagrip know, if you type sel and then wait or press Ctrl+Space, a panel is shown with options. One of them is the template for:
Select * from ;

How could I create my own?
Answer: I have found out that if you type in a new file, for example:
sel and then hit CTRL+space

it will show:

Click the yellow lamp in the right top corner of the popup. It will show another menu with "Edit live template settings" option inside it where you can create your own templates.
Answer: Go to Settings → Editor → Live Templates

Also, follow official custom live template creation guidance.

Datagrip, create custom template


Port redirection with iptables to localhost / blocking the destination port

Question: I like to forward all traffic coming to 80 to be redirected to 8000.
I used
iptables -tnat -A PREROUTING -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 8000

But then port 8000 is still open to public.
I tried -DNAT:
iptables -t nat -A PREROUTING  -p tcp --dport 80 -j DNAT --to-destination 127.0.0.1:8000

But this does not work at all.
/proc/sys/net/ipv4/ip_forward is set to 1.
There are no other rules configured.
Please help me out :)
--
edit:
In answer to the question below: Here is exaclty, what I want to achieve:
I have a service running at port 8000 which can be seen as a webserver.
It should be accessible via port 80 from outside, but not via port 8000. The internal network does not matter: may be restricted or not.
I understand, that REDIRECT does not drop packets directly addressed to port 8000, but if I would drop all packets to 8000 in INPUT chain, then the redirected packets are dropped also.
iptables -A INPUT -p tcp -m tcp --dport 8000 -j DROP

Cannot work for me.
Is there a way to drop packets with destination port 8000, that have not been redirected via my PREROUTING chain?
Adding a source ip via -s does not help, right? because the source IP does not change at redirection.
Answer
Based on the accepted answer I could solve it this way:
iptables -tmangle -A PREROUTING -p tcp -m tcp --dport 8000 -j MARK --set-mark 1
iptables -A INPUT -m mark --mark 1 -j DROP

Comment: You do a redirect, so the port 80 will be redirected to 8000. You are not filtering the port 8000. So add a filter rule to 8000 if you want, or add more details on your question, please !
Comment: What are you trying to achieve - it's not very clear from your question?
Answer: Set mark in the packet you redirect. Then allow the marked packs before the blocked packs. see iptables -j MARK -h to mark a packet and iptables -m mark -h to match a pack
Comment: I know how to block a port. I need the condition on detecting that is had been redirected or not.
I will clearify my question once more.
Comment: This was a good hint, I added the complete answer above. Thanks.


What do these two same/duplicate notes on the opposite sides of the same stem mean in a piano score? Like the two D4 notes here. How are they played

Question: This is from carol of the bells.
Comment: That sheet music wasn't created very well. I wouldn't worry too much about anything strange you find in it - it probably shouldn't be there. In this case, there's no point in having those duplicate notes and they are probably a mistake. Other mistakes include the use of the bass clef on the lower staff and usually "RH" and "LH" are not written in because it's understood and instead there would be a brace joining the staves.
Comment: @Terry I respectfully recommend you try and find another score to use. This notation is, frankly, a bit of a nightmare.
Comment: Where did you get that from? It doesn't make much sense for piano. Also those notes are E5 not D4.
Comment: Just leaving the note, if this *were* for a different instrument, it could mean an interval of a unison. On violin, for instance, you can play an E on two different strings at the same time, as a "double stop."
Comment: That E could be a poorly notated third voice: the descending dotted-half notes that are added to the ostinato and harmonic background.
Comment: And just to answer the second question, *on piano* it can have no impact on how it's played (aside from any hypothetical slight impact on articulation to bring out a moving line). On an instrument capable of playing two copies of the same note simultaneously (including, I imagine, organ notated in grand staff), one might have to decide whether to take it literally or simply as an indication of voices (but then it would be notated more clearly anyway).
Comment: Thank you everyone for your helpful kind answers I have added a bit more of the music. Found it on line. Please forgive my ignorance of the notes.
Comment: @Richard - indeed.  I don't think I've ever seen six ledger lines in bass clef before.
Answer: Thank you for adding more context (it make it much easier to diagnose).
When looking at this excerpt I notice:

"LH" and "RH".  Piano music with two staves implies left hand for the bottom stave and right hand for the top.  I've never seen this written out before, very strange.  Also, there should be a brace contenting these staves into a grand staff.

Very high notes in the bottom staff. The left hand can play notes this high, but it really should be using a treble clef. Three leger lines above bass clef is fine for piano, but five is simply too many without a good reason (there is not a good reason here).

The "double stops" (what your actual question was about).
Some instruments can play the same note twice at the same time (on different strings of a violin for example). But obviously on a piano, a note/key can only be pressed once at a time.  However these can occasionally be seen in piano/keyboard music. They are for when different musical voices playing different melodic lines and both "need" to certain note to make musical sense (if they were played in isolation). Looking at the context, this is not what is happening here.
As written it's it doesn't mean anything; it's a mistake.  (Perhaps one of the Es got placed in the wrong octave somehow?)

Overall it's clearly engraved quite poorly, and you really should find another version of this music.
Answer: That first note isn't even joined to the stem! So a four beat note in 3/4 time sgnature? The whole thing is pretty disastrous, so the best idea is to scrap it, and find a different, better, non-amateurly written out version.
Answer: That's an accepted way of writing a unison.  It might make sense in e.g. harp music, where two strings can play the same pitch.  Or other stringed instruments (though most of them couldn't manage all 4 notes!).  Yes, the first notehead is slightly distanced from the stem.  But there are the right number of dots.  The notation, as such, isn't too bad. Sibelius does it much the same (see below).
But, as a piano score, no need to overthink beyond 'it's a rubbish one'.   I suspect this is a MIDI file or similar, imported to a notation program by someone not very experienced in notation or piano playing.


Prove that the following is vector space

Question: Prove that for any $n\ge 1$, $(K^n/K,+,\cdot)$ is a vector space, where $(K^n,+)$ is an abelian group and $(K,+,\cdot)$ is a commutative field.
I've been having trouble proving this lately. Any help is welcome!
Comment: For another answer see also the [wikipedia article](https://en.wikipedia.org/wiki/Examples_of_vector_spaces), section "Coordinate space $K^n$".
Comment: @DietrichBurde I'm looking for a more formal proof.
Comment: The wikipedia proof is formal. You can easily fill in more details, if you want.
Answer: Abstract: yes, it is.
In order to prove it, we need to prove the following:

Distributivity of $v \in K^n $ over $t_1, t_2 \in K $. This is often called "Pseudo-Distributivity".
Distributivity of $t \in K $ over $v_1, v_2 \in K^n $.
Associativity of $v \in K^n $ over $t_1, t_2 \in K $.
Normalization. $1\cdot v = v$ for all $v \in K^n$.

I assume $K$ is a numerical set.
I believe the definition of summation and product by scalar in $K^n$ is already given, so I'll assume those too.

Let $v \in K^n$, $t_1, t_2, t_3 \in K$.  Let $t_3 = t_1, t_2$. Then:
$$ v(t_1 + t_2) = t_3v $$
$$ t_1v + t_2v = t_3v $$
$$ \implies v(t_1 + t_2) = t_1v + t_2 v $$
Let $v_1, v_2, v_3 \in K^n$, $t \in K$.  Let $v_3 = v_1 + v_2$. Then:
$$ t(v_1 + v_2) = tv_3 $$
$$ tv_1 + tv_2 = tv_3 $$
$$ \implies t(v_1 + v_2) = tv_1 + tv_2 $$
Let $v \in K^n$, $t, s, p  \in K$. Let $p = ts$. Then:
$$ (ts)v = (p)v = pv $$
$$ t(sv) = pv $$
$$ \implies (ts)v = t(pv) $$
Let $v \in K^n$. Then: $1\cdot v = v$.

These properties follow from how operations on your vector set are defined. To put it briefly, summation should be the usual memberwise summation of vectors, while multiplication by scalar should be the usual memberwise product.
You will find out that under these circumstances, your vector space $(\mathbb{K}^n, \mathbb{K}, +, \cdot )$ is the typical coordinate vector space over $\mathbb{K}$.
Of course, this works for every $n \in \mathbb{N}$. An interesting insight about this is that a commutative field can always be considered a vector space over itself.
You could also prove this by induction, given the above insight.


How do I fix a cloned machine that wasn't sysprepped?

Question: My Scenario
I made a bunch of Hyper-V VM (Windows Server 2003) clones, without using sysprep or anything of the sort. All I did was clone, remove the computer from the domain, rename the computer, and re-join it to the domain. And everything was perfectly fine.
Until I was told I had to install the SCCM client on every clone (and the original). So I did that, manually and very painfully.
But then the SCCM client of course failed to install any updates which is what I went through the whole process of installing it for. Upon investigation, it appears I needed to create a new SID for each VM, since they all had the same one. Apparently, the de-facto application for this, NewSID, has been removed in favour of sysprep.
What I did Bad and Need Help with
Following some instructions found in the Tome of Google, I ran sysprep and just clicked the "Reseal" button and the VM shut down. When I turned it back on, I was greeted with the message:
An error has been encountered that prevents setup from continuing.

Windows is unable to start because the registry could not be updated. To address this problem, please contact your
computer manufacturer. Windows must now shut down.

Press OK to view the Setup log file.

The Setup log file that then appears in a message box contains the same message in the middle (Windows is unable to...).
Safe mode doesn't work so I'm completely stuck.
Answer: Sorry to be the bearer of bad news, but you already know what you need to do:

Sysprep the existing VMs. Yes, data will be lost unless you back it up.
Recreate the original W2K3 server, sysprep and re-clone. Yep, what you should have done the fist time.

There is a lesson here, something to do with best practices. I've made dumb mistakes too, don't think I'm 'beating you up' about it, but when it doubt, play it safe in production.
Comment: sysprep is a ***must*** when cloning a Windows box! there are some exceptions, but that is the general rule.
Comment: You're right, and especially since these were VMs I could've made snapshots. Really dumb. Anyways I got lucky and was able to just replace the C-drives on the VMs with the one from the original. The important data was on other drives. I don't think I'll touch sysprep again though.


A quantitative orthogonality argument

Question: Let $S={\rm span}(v_1,\ldots,v_k)$ be a subspace of $R^n$, and let $P_S$ be the orthogonal projection matrix onto $S$. If for some $x$ we have $x^T v_i = 0$ for all $i=1,\ldots,k$, then we can conclude that $P_S x=0$.
I would like a quantitative version of this statement which I'll now describe. Suppose that instead we just know that $|x^T v_i| \leq \epsilon$ for all $i=1,\ldots,k$; and moreover $||x||_2=1$, $||v_i||_2=1$ for all $i=1,\ldots,k$. I'd like an argument that concludes $||P_S x||_2 \leq f(\epsilon, v_1, \ldots, v_n)$ where $f(\epsilon, v_1, \ldots, v_n)$ is some function which approaches $0$ as $\epsilon \rightarrow 0$ and the other arguments are kept fixed. Naturally, the faster rate of decay as $\epsilon \rightarrow 0$, the better.
Comment: You're going to need additional assumptions on how close the $v_i$ are allowed to be to each other. Otherwise, you could have $v_1 = (1,0,0)$, $v_2 = (\sqrt{1-\epsilon^2},\epsilon,0)$, and $x = (0,1,0)$ with $\lVert P_S x\rVert_2 = 1$.
Comment: There are many projection matrices onto a given subspace.  Is $P_S$ the orthogonal projection onto $S$?
Comment: Yes; thanks - I edited this into the question.
Comment: Good point; I've reworded the question to allow the function $f$ to depend on the $v_i$.
Answer: Intuitively, the statement is obvious. Here is a proof.
Let $A=(v_1,v_2,...,v_k)\in\mathbb{R}^{n\times k}$. Then the orthogonal projection matrix $P_S$ is $$P_S=A(A^TA)^{-1}A^T\in\mathbb{R}^{n\times n}$$
Denote $y=A^Tx\in\mathbb{R}^k$. Then $|v_i^Tx|<\epsilon$ implies $|y_i|<\epsilon$ and $$\|y\|^2<k\epsilon^2$$
Moreover,
$$\|P_Sx\|^2=x^TP_Sx=y^T(A^TA)^{-1}y$$
Let $(A^TA)^{-1}=U^T\Sigma U$ be an SVD of $(A^TA)^{-1}$ with $\Sigma=\mathrm{diag}(\sigma_1,...,\sigma_k)$. Denote $z=Uy$. Then
$$\|P_Sx\|^2=y^T(A^TA)^{-1}y=z^T\Sigma z=\sum_{i=1}^k \sigma_i z_i^2\le \sigma_1\sum_{i=1}^k z_i^2=\sigma_1\|z\|^2=\sigma_1\|Uy\|^2=\sigma_1\|y\|^2<\sigma_1k\epsilon^2$$
where $\sigma_1$ is the largest singular value of $(A^TA)^{-1}$.

A quantitative orthogonality argument


Conceptual question on Cauchy's Theorem on Groups (abelian case)

Question: The theorem states that let G be a finite abelian group and p be a prime. If p divides the order of G, then G has an element of order p.
The proof given is this:
We first prove the special case that where G is abelian, and then the general case; both proofs are by induction on n = |G|, and have as starting case n = p which is trivial because any non-identity element now has order p. Suppose first that G is abelian. Take any non-identity element a, and let H be the cyclic group it generates. If p divides |H|, then a|H|/p is an element of order p. If p does not divide |H|, then it divides the order [G:H] of the quotient group G/H, which therefore contains an element of order p by the inductive hypothesis. That element is a class xH for some x in G, and if m is the order of x in G, then xm = e in G gives (xH)m = eH in G/H, so p divides m; as before xm/p is now an element of order p in G, completing the proof for the abelian case.
I can't see why we need the condition of prime here. Which part of the proof uses that property. From what I see, it applies to all numbers. Thanks
Comment: The base case does not hold if $p$ is not prime, for instance $\mathbb{Z}_6$ has an element 3 of order 2.
Comment: it's used to prove the induction base, where you say "... which is trivial since any non-identity element now has order $p$."
Answer: Already in "...which is trivial because any non-identity element now has order p" because for example $\mathbb{Z}/2 \times \mathbb{Z}/2$ no element has order $4$.
Comment: my lecturer gives me another proof that uses this base case:
n = 1 is vacuously true, because there is no prime divisor of 1. Does that mean this this cannot be a correct base case since i need base case of order p?


Ideals: $I\cdot(J+K)=I\cdot J+ I \cdot K$

Question: For a commutative ring $R$ and ideals $I,J,K$ it is the case that $I\cdot(J+K)=I\cdot J+ I \cdot K$.
I am trying to prove this but I do not understand this step:
We start off by definition $I\cdot(J+K)=\langle i\cdot(j+k)| i \in I ,j \in J, k \in K\rangle$. We have distrubitivity in the ring so $i\cdot(j+k)=ij+ik$. This gives $I\cdot(J+K)=\langle ij+ik| i \in I ,j \in J, k \in K\rangle$. Now we want this to be equal to $\langle ij| i \in I ,j \in \rangle+\langle ik| i \in I , k \in K\rangle$. But how do we justify this?
Comment: Every element of the form $ij$ or $i'k$ lies in the set of elements of the form $ij+ik$ (by taking $k=0$ or $j=0$), giving $IJ+IK\subseteq I(J+K)$. Converseley, every element of the form $ij+ik$ lies in the ideal $IJ+IK$, hence the ideal these elements generate is contained in the ideal $IJ+IK$.
Comment: ah yes! If you would like add your comment as an answer I will accept it to close this question.
Answer: Note that every element of the form $ij$ or $i'k$ lies in $I(J+K)$ (by taking $k=0$ or $j=0$). So both $IJ$ and $IK$ are contained in $I(J+K)$, hence so is $IJ+IK$.
Conversely, every element of the form $ij+ik$ lies in $IJ+IK$, so the ideal they generate, namely $I(J+K)$, is contained in $IJ+IK$, giving equality.


Como ordenar colunas do sql no alter table

Question: Tenho uma alteração para fazer na tabela Inclusão de colunas a duvida é tem como eu definir a ordem quando estou fazendo a alteração no sql server 2008 ?
ALTER TABLE dbo.Arquivos
ADD IdTipoDocumento int NULL, -- order 3
    NomeInterno nvarchar(200) NULL, -- order 4
    DirArquivo nvarchar(200) NULL,-- order 5
    DescrArquivo nvarchar(max) NULL, -- order 6
    LarguraArquivo int NULL, -- order 12
    AlturaArquivo int NULL, --  order 13
    FlagAtivo bit NULL, -- order 14
    IdUsuarioUpload int NULL, -- order 15
    DtInclusao datetime NULL -- order 16

Comment: Não faço ideia se tenha como fazer isso, mas uma maneira seria você criar uma tabela temporária e recriar sua tabela na ordem correta, e fazer o insert dos registros da tabela dbo.arquivos na temporária, algo assim.
Comment: sei como funciona... mas dar um trabalho, achava que teria algo mais fácil...
Answer: Sim, apenas pelo SQL Server Management Studio. Pelo Transact-SQL isto não é possível.
Pro banco de dados, a ordem das colunas é irrelevante. Essa funcionalidade pode ser considerada até cosmética, visto que não afeta em nada o desempenho ou o funcionamento da tabela.
Comment: hauhauha eu ia falar para criar uma tabela virtual, exportar pra ela os dados, dar um drop da tabela e recriar ela, depois inserir os dados da tabela virtual ;P deve ser isso que o management faz hauauha
Comment: @CiganoMorrisonMendez, esperimente fazer o seguinte: `Ferramentas` -> `Opções` -> `Designers`, então marque a opção `Evitar salvar alterações que exijam recriação de tabela`... Se fizer isto, você não vai conseguir salvar alterações no Design da tabela como alterar a ordem das colunas, tornar um tipo NULL em NOT NULL, etc... Se fizer isto e tentar salvar, o `Management Studio` irá lhe informar que não pode prosseguir.
Comment: Caramba. Sério isso? Só preciso da fonte.
Comment: @Rod, é justamente isto que o `Management Studio`, você pode verificar isto, abrindo o Design para uma tabela, re-ordenando as colunas e ao invés de salvar a alteração use a opção para `Gerar Script de Alteração`, ele vai lhe entregar um script que cria uma tabela temporaria, copia os dados para esta tabela, apaga a anterior junto com os relacionamentos e renomeia a temporaria e recria os relacionamentos.
Comment: talvez seja interessante mencionar que o management Studio irá recriar a tabela e caso alguma coisa dê errado, a tabela pode acaba no limbo ou corrompida.
Comment: @CiganoMorrisonMendez Sim, em termo de desempenho não afeta mesmo, era algo mais apresentável para minha aplicação mesmo. Só não sabia se tinha algo que fizesse isso no próprio `alter table`. Vlw.
Comment: @Rod é possível sim fazer dessa forma, mas é um pouco trabalhoso. O que o SQL Server Management Studio faz por traz para ordenar... só falando com o Bill Gates
Comment: @TobyMosque Dica sensacional. Não vou nem complementar a resposta para te dar o devido crédito.


What does the Argument "type" in VAR() - function do?

Question: Right now I am working with vector autoregressive models in order to make 3 months forecasts for a commodity good (sawlogs) y. I have several time-series of "follow-up-products" of sawlogs that should work as "predictors" for saw-log prices from a logical point of view.
I encountered within the VAR-function from package "vars" (http://cran.r-project.org/web/packages/vars/vars.pdf), that one attribute called "type" has the following expressions: "const", "both", "trend", "none". I really don't know what this means from a statistical point of view.
Since neither the package-description nor other literature I've screened so far can give me an answer I actually understand I'd like to ask you guys the following:
How should I interpret/understand and use the argument "type" in R's VAR() Function?
What do those 4 different arguments really mean? "both", "none", "trend", "constant"?
Could anyone explain this in a simple way and probably provide an example as well?
Does this mean that I can directly use non-stationary time series for my VAR-model since I can consider trend/season afterwards by setting the "type-argument" to both, or am I wrong here?
Answer: constant means there is an intercept included in each equation in the model. trend means there is a linear time trend included. both means both of them are included. none means neither of them is included.
Type ?VAR in R to get an explanation. There you will find the algebraic form of the model provided. Pay attention to the CD_t term. You may also see the vignette for the vars package, perhaps it could help.
Comment: If the non-stationarity is caused by a deterministic time trend (for example, a linear time trend), it can be treated by including a time trend regressor in the VAR model (option `type="trend"`). If, on the other hand, non-stationarity is due to the process being integrated (e.g. I(1) process), this cannot be cured in such a simple way. By the way, `both` is **not** about seasonal component. There is another option for that – it has a name `season`.
Comment: Thanks for your answer Richard. I read the vignette and didn't really get smarter afterwards. Sometimes it is stated that the data has to be stationary BEFORE usig a VAR model, then I ask myself how there even could be a trend?

Does this "type"argument imply that you can directly use non-stationary data within a VARestimateion and then set "type" to "both" in order to consider trend and season?