How to list files in a file system based on the limit : java

Question: How to list the files available in file system with the starting number and ending number?
like if there are 500 files in C:\Test\ then how to list files starting from 1 to 20 like
give start number and end number based on this list the files available for particular file path.
i am trying this in java
I tried some thing like this and it gives me all the files available for the given path
public static List<String> loadAllFiles(String filesLocation) {

        //find OS
        //String osName = System.getProperty("os.name");

        //replace file path based on OS
        filesLocation = filesLocation.replaceAll("\\\\|/", "\\"+System.getProperty("file.separator"));

        List<String> pdfFiles = new ArrayList<String>();
        if (log.isDebugEnabled()) {
            log.debug("In loadAllFiles execute start");
        }

        File directoryList = new File(filesLocation);
        File[] filesList = directoryList.listFiles();

        try {
            for (int count = 0; count < filesList.length; count++) {
                if (!filesList[count].isDirectory() && filesList[count].getName().endsWith(SPLIT_AND_SAVE_WORKING_FILE_EXTENSION.trim())) {
                    // load only PDF  files
                    pdfFiles.add(filesList[count].getName().replace(SPLIT_AND_SAVE_WORKING_FILE_EXTENSION.trim(), ""));
                }
            }

        } catch (Exception filesException) {
            filesException.printStackTrace();
            //TODO : Log the exception
        } finally {

            if (filesList != null)
                filesList = null;

            if (directoryList != null)
                directoryList = null;
        }

        log.debug("In loadAllFiles execute end");

        return pdfFiles;
    }

I think the question is misunderstood, Say if i have 1000 files[file names can be anything] and i want to restrict getting the files name like i will give starting Number and ending number. like 1 to 20 and i want to load those 20 files alone.

Comment: After reading your question again. Do you want to list only the first 20 files from a list of 500? Or do you want to list the files which are starting with the numb2er 1 till 20? What about a file starting with 100?
Comment: yes, I want to list only the first 20 files from a list of 500. Thanks for understanding
Comment: I add two new examples to list the first 20 files in the directory.
Answer: An example without external libraries using plain Java 7
import java.io.IOException;
import java.nio.file.DirectoryStream;
import static java.nio.file.DirectoryStream.Filter;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

// list files starting with 1 till 20 "-.*"
public class FileNameFilter {

    private static final Filter<Path> fileNameFilter = new Filter<Path>() {
        @Override
        public boolean accept(Path entry) throws IOException {
            if (!Files.isRegularFile(entry)) {
                return false;
            }
            return entry.getFileName().toString().matches("^([1][0-9]{0,1}|2[0]{0,1})-.*");
        }
    };

    public static void main(String[] args) {
        final String filesLocation = "resources/";

        Path path = Paths.get(filesLocation);
        try (DirectoryStream<Path> dirStream = Files.newDirectoryStream(path, fileNameFilter)) {
            for (Path entry : dirStream) {
                System.out.printf("%-5s: %s%n", "entry", entry.getFileName());
            }
        } catch (IOException e) {
            // add your exception handling here
            e.printStackTrace(System.err);
        }
    }
}

edit
a Java 8 version
// list files starting with 1 till 20 "-.*"
public class FileNameFilter {
    public static void main(String[] args) {
        final String filesLocation = "resources/";

        try {
            Files.walk(Paths.get(filesLocation))
                    .filter(p -> p.getFileName().toString().matches("^([1][0-9]{0,1}|2[0]{0,1})-.*"))
                    .forEach(entry -> {System.out.printf("%-5s: %s%n", "entry", entry.getFileName());});
        } catch (IOException e) {
            // add your exception handling here
            e.printStackTrace(System.err);
        }
    }
}

edit 2
Examples to list the first 20 files in a directory.
note The order of the files is the same as you would run ls or dir in the directory.
Java 7 example
import java.io.IOException;
import java.nio.file.DirectoryStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class FileListLimiter {

private static final int MAX_FILES_TO_LIST = 20;

    public static void main(String[] args) {
        final String filesLocation = "resources/";

        Path path = Paths.get(filesLocation);
        try (DirectoryStream<Path> dirStream = Files.newDirectoryStream(path)) {
            int fileCounter = 1;
            for (Path entry : dirStream) {
                System.out.printf("%-5s %2d: %s%n", "entry", fileCounter++, entry.getFileName());
                if (fileCounter > MAX_FILES_TO_LIST) {
                    break;
                }
            }
        } catch (IOException e) {
            // add your exception handling here
            e.printStackTrace(System.err);
        }
    }
}

Java 8 example
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class FileListLimiter {

private static final int MAX_FILES_TO_LIST = 20;

    public static void main(String[] args) {
        final String filesLocation = "resources/";

        try {
            Files.walk(Paths.get(filesLocation))
                    .filter(p -> p.toFile().isFile())
                    .limit(MAX_FILES_TO_LIST)
                    .forEach(entry -> {System.out.printf("%-5s: %s%n", "entry", entry.getFileName());});
        } catch (IOException e) {
            // add your exception handling here
            e.printStackTrace(System.err);
        }
    }
}

Comment: `[1-20]` matches a `0`, `1` or `2`, not numbers 1 through 20.
Comment: @JakeCobb Thanks. Fixed the pattern and ask Anto to make his question more precise. As a file starting with 200 would also start with 20.
Answer: Try something like
final String pattern = "^[1-20].*";
final FileFilter filter = new RegexFileFilter(pattern);
final File[] files = directory.listFiles(filter);

Comment: NO @almas shaikh, it is giving me the zero files and i am not looking for text in the file but i have to list the files which are from 1 to 20 for example..
Comment: What are your file names? it would work for 1abc.txt, 20abc.txt etc.
Comment: I think the question is misunderstood, Say if i have 1000 files[file names can be anything] and i want to restrict getting the files name like i will give starting Number and ending number. like 1 to 20 and i want to load those 20 files alone.
Answer: why not use this logic. its pretty straight forward and you will not need any substitution or replacement.i assume that you will be able to convert this into code very easily into Java. i have not tested the regex but you get the basic idea.
1.use scanner(or any other way) to get the input ranges for the start and the end limit.
2.create a directory listing scanning every file in the directory. use a for loop here.
3.define a regular expression to fetch only the starting data for the file .
      start pattern = (^[0-9]{0,2})[a-zA-Z].+[0-9]{0,2}$ use grouping in regular expression to fetch this group1

4.When you get the values from group1 check the file name if it starts with group1 data.
5 if this data falls in between the ranges that you want to search then create a arraylist and store the matched file in the arraylist.
6.carry on the process till end of the directory. The final arraylist would contain your files in the given ranges.
hope this helps
Answer: Starting from Java 7 you can take advantage of Files#walkFileTree . During visitFile you can use a regex to look for you specific file.
Something like this
Path start = Paths.get(filesLocation);

Final Pattern pattern = Pattern.compile("^([1]?[0-9]|[2][0])\\.pdf");

Files.walkFileTree(start, new SimpleFileVisitor<>()
{
   @Override
   public FileVisitResult visitFile(Path filePath, BasicFileAttributes attrs)
      throws IOException
   {

    File f = filePath.toFile();
    if (pattern.matcher().matches(f.getName())) {
            System.out.println("Valid match found " + f.getName())
    }
      return FileVisitResult.CONTINUE;
   }
});

Answer: I played around with it and came up with something that works for me. Perhaps you can find the parts you need. If you don't want the full path, just remove the getCanonicalPath().
    import java.io.File;
    import java.io.FilenameFilter;
    import java.io.IOException;
    import java.util.ArrayList;

    public class GetFirst_20_pdf_files {

        // Creating an empty array list with 10 entries (the default)
        ArrayList<String> pdfFiles = new ArrayList<String>();

        public ArrayList<String> myList() {
            // here you can get start and end from user
            int start = 2;
            int end   = 4;
            // and set the directory
            String filesLocation = ".";

            File directoryList = new File(filesLocation);
            File[] files = directoryList.listFiles();
            int count = 0;
            for (File file : files) {
                // don't want directories
                if (file.isDirectory()) { continue; }
                // want ".pdf" files only
                if (!file.getName().toLowerCase().endsWith(".pdf")) { continue; }
                // only want between start and end
                count++;
                if (count >= start && count <= end) {
                try {
                     pdfFiles.add(file.getCanonicalPath());
                 } catch (IOException e) {
                     System.out.println(e);
                 }

                 if (count >= end) { return pdfFiles; }
                }
            }
            return pdfFiles;
        }

        public static void main(String[] args) throws IOException {

            GetFirst_20_pdf_files L = new GetFirst_20_pdf_files();
            ArrayList<String> aList = L.myList();

            for (String p : aList) {
                System.out.println(p);
            }
        }
    }

Answer: Not sure i fully understood your question, but here is a Java 8 example that iterates over files of a directory sorted by name, starting at a given offset and limiting the results to a given size. It also allows you to include/exclude directories from results.
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Collection;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class DirectoryWalkerTest {

    public static void main(final String... args) throws IOException {
        final String filesLocation = "/path/to/files";

        System.out.println(getFiles(filesLocation, true, 0, 10));
        System.out.println(getFiles(filesLocation, true, 10, 10));

        System.out.println(getFiles(filesLocation, false, 0, 10));
        System.out.println(getFiles(filesLocation, false, 10, 10));
    }

    /**
     * Returns a limited list of filenames starting at the given offset.
     *
     * @param filesLocation     the directory to scan
     * @param filterDirectories should we include directories in results
     * @param offset            the starting offset
     * @param limit             the maximum number of results
     * @return a collection of file names
     */
    private static Collection<String> getFiles(final String filesLocation, final boolean filterDirectories,
            final int offset, final int limit) throws IOException {
        Stream<Path> stream = Files.list(Paths.get(filesLocation));

        if (filterDirectories) {
            stream = stream.filter(path -> !Files.isDirectory(path));
        }

        return stream
                .map(Path::toString)
                .sorted()
                .skip(offset)
                .limit(limit)
                .collect(Collectors.toList());
    }

}

Answer: Based on your code, you could try this :
int begin = 0;
int limit = 10;
for (int count = 0; count < filesList.length; count++)
{
    if (!filesList[count].isDirectory() && filesList[count].getName().endsWith(SPLIT_AND_SAVE_WORKING_FILE_EXTENSION.trim()))
    {
        // moving to begin index, only if the file is not a directory and is matching your criteria, whatever it is
        if (count < begin) continue;

        // load only PDF  files
        pdfFiles.add(filesList[count].getName().replace(SPLIT_AND_SAVE_WORKING_FILE_EXTENSION.trim(), ""));
    }

    // Stopping when limit is reached
    if (pdfFiles.size() == limit) break;
}

It will move to begin index and get limit items, based on your criteria.


Maximum number of points you can put on grid $ n\times m$ with no equidistant?

Question: Assume we have a grid of $n\times m$ points. and the distance between two rows or two columns is 1 ( unit ). I have a couple of questions related to this grid:-

What is the list of possible length between points?
What is the maximum number of points that they have no equidistant between any pairs?

For question number one , I know if we have we have $n \times m$ so we have length of size $1,2 \dots \mbox{max}\{m,n\}$ in addiction of this we have $\sqrt{2},\sqrt{5}$ and some others. but I am interested in all of these distance.
For question number 2 intuitionally it look like for $n\times n$ case we the maximum point with no equidistant is n.

I have feeling this has been studied in some books but I am not sure where or how to answer the general case of this problem. any hints and/or answers would be appreciated.
There are solution of $5 \times 4$ for example this is two of the solution of it ( there is 5 non-equidistant points):-
Reference:- I found this problem from the twitter and you can find it here.
Comment: +1 for just being the most interesting question I've read today.
Comment: It's not hard to describe the *set* of possible lengths: $\{\sqrt{i^2 + j^2}: i =0, 1, \ldots, n,\text{ and } j = 0, \ldots, m\}$, but its size in terms of $m$ and $n$ isn't obvious. Primarily due to the existence of Pythagorean triples; for example $5 = \sqrt{3^2 + 4^2}$. (We could get rid of a lot of repeats here by requiring $j > i$, but since it isn't good for counting as it is, I won't bother). It's not hard to bound the number of unique lengths above by $m(m+1)/2 + m(n - m)$, but that's all I've got.
Comment: Perhaps I'm miscounting, but in a 5×5 grid I can't seem to find 5 points such that no two pairs of them are equidistant -- contrary to your intuition about Question 2 in case m=n. (?)
Comment: @r.e.s. actually there is solution for $5 \times 4$ ( I know 2 of them but I hear there are 7) .. so of course there is solution for $5\times 5$.and about the Taxicab geometry I agree with you , this is the only related thing I founded which the relation to this problem but it is not so obvious to me. thanks
Comment: OK, I must have overlooked something -- would you mind posting any one of those $5\times 4$ solutions?  (Sorry, before I saw your reply I edited out my comment about Taxicab geometry as a variant.)
Comment: @r.e.s. thanks for you recommendation. I added two of the solution. which have been found by brute force.
Answer: The squared lengths are of the form $i^2 + j^2$ for $i \in [0,n]$ and $j \in [0,m]$; assuming (with no loss of generality) that $n \ge m$, we can consider only the cases where $i \ge j$, of which there are $$(m+1)(n-m)+\frac{1}{2}(m+1)(m+2)=\frac{1}{2}(m+1)\left(m+2(n-m+1)\right).$$
On the other hand, placing $k$ points creates $k(k+1)/2$ pairs, so if all of these pairs must have distinct distances,
$$
k(k+1) \le (m+1)\left(m + 2(n-m+1)\right).
$$
So for $n=m$ this implies $k \le m+1$, and more generally
$$
k \le \sqrt{(m+1)\left(m + 2(n-m+1)\right)}.
$$
A tighter bound, at least in the limit of large $n$, will come from improving the estimate of the number of distinct distances on the grid.  We know that the number of positive integers less than $2n^2+1$ that are the sum of two squares is $O\left(n^2 / \sqrt{\log n}\right)$ (cf. the Landau-Ramanujan constant); i.e., it grows more slowly than $n^2$.  For $n=m$, then, the growth rate of $k$ is necessarily in $O\left(n / \sqrt[4]{\log n}\right)$.  So $k=n$ is not correct: in fact, $k/n \rightarrow 0$ for large $n$.
Comment: The paper "A note on distinct distance subsets" (Charalambides 2013) gives this same upper bound, and also gives the asymptotic lower bound $k \in \Omega\left(n^{2/3} / \sqrt[3]{\log n}\right)$.
Comment: Your closing sentence naturally leads to the question: *What is the least $n$ such that in an $n\times n$ grid, there is no set of $n$ points among which all point-to-point distances are distinct?*
Comment: @r.e.s.: Yes, it does :).  Maybe you should ask that question separately... it may not be very large, but then again, $(\log n)^{1/4}$ grows pretty slowly.
Comment: @mjqxxxx that was little surprised to me to see $k=n$ is not correct. thanks for your answer. Just wonder do you think there is a mechanism to know the location of these points ? thanks


Corrected AIC (AICC) for k-means

Question: I want to calculate the $AIC_c$ (corrected $AIC$) for k-means to decide on the number of clusters, but there is an overfitting problem that I don't know how to solve. Let's say that I have $n$ data points of $d$ dimensions each, and I want to cluster those $n$ points into $c$ clusters. The Akaike Information Criterion ($AIC$) is $-2ln(L)+ 2k$ where $k$ is the number of free parameters, and for k-means, it is $c(d+1)$. And for $AIC_c$, the formula becomes $-2ln(L)+ 2kn/(n-k-1)$.
Now, let's say $n=1000$, $d=200$, $c=10$, then the number of parameters is $k=2010$. Then the denominator in the penalty term of $AIC_c$ becomes negative, which means a negative penalty in the formula. And when $c$ increases, the penalty decreases, resulting in $AIC_c$'s supporting the models with the maximum number of clusters (when $c=n$).
I think I am missing or I have misunderstood a point about $AIC_c$. What is that? Thanks in advance.
Comment: The point is that they don't use AICc for k-means. AICc was designed mostly for regression modeling. K-means clustering is not regression problem and is not a modeling at all, in the straightforward sense. You may still use AIC, though.
Comment: Kmeans is Gaussian mixtures with assumed equal covariances across all clusters, so the covariances are excluded from the parameters.  So this means that each cluster now only has a weight (its a pdf, so it has to sum to 1.0) and a mean.  I would think this means that k = count_of_weights + count_of_means*number_of_dimensions.
Answer: The form of $AICc$ of
$$
AICc = AIC + \frac{2k(k+1)}{n-k-1}
$$
was proposed by
Hurvich, C. M.; Tsai, C.-L. (1989), "Regression and time series model selection in small samples", Biometrika 76: 297–307
specifically for a linear regression model with normally distributed errors. For different models, a different correction will need to be derived.
These derivations are often difficult and the resulting correction may be challenging to calculate. For instance
Hurvich, Clifford M., Jeffrey S. Simonoff, and Chih‐Ling Tsai. "Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 60, no. 2 (1998): 271-293.
propose a correction to be used in the case of nonparametric regression models which takes the form
$$
AICc = -2ln(L) + n^2\int_0^1(1-t)^{r/2-2}\prod_{j=1}^{r}(1-t+2d_j)^{-1/2}dt+n\int_0^{\infty}\sum_{i=1}^n\frac{c_{ii}}{1+2d_it}\prod_{i=1}^n(1+2d_it)^{-1/2}dt
$$
I will not go into the details here as they are largely irrelevant but I wanted to illustrate the complexity involved. Actual calculation of this value involves eigen-analysis and numerical integration.
For reasons like this, many authors such as
Burnham, K. P.; Anderson, D. R. (2002), Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach (2nd ed.), Springer-Verlag, ISBN 0-387-95364-7
suggest to use the form
$$
AICc = AIC + \frac{2k(k+1)}{n-k-1}
$$
regardless of model. Even Hurvich et al. (1998) despite deriving their complicated $AICc$ for nonparametric regression ultimately conclude that you might as well use the much simpler version for linear regression.
Generally this advice seems to work well, giving practically useful results. However there are circumstances, such as the one you've highlighted where it doesn't work. You would need to find an appropriate $AICc$ for k-means, or derive one yourself, or simply use $AIC$ which is more generally applicable.
Comment: Thanks a lot for the explanation. $AIC$ selects a higher value for $c$ than I expected (and needed). And since $AIC_c$ is suggested when $n<<k$, I wanted to use $AIC_c$ hoping that it would penalize the models with high number of parameters more than $AIC$ does. But learning that it would not work for k-means is very good to know. I also tried $BIC$, but opposite to $AIC$, it selects a smaller $c$ than I expected. Are there any other criterion that you suggest using? (I would not prefer cross-validation since my data is much bigger than the one I gave as the example.)
Comment: @user5054 It sounds like you have certain expectations about the "true" number of clusters in the data set. Is there some reason you need to use a model selection criterion at all?
Comment: I am doing a biological application (clustering genes into pathways), and biologically, 40-50 genes in a pathway is reasonable. However, I cannot explain this in terms of application purposes to a machine learning or statistics person (paper reviewer). In order to convince them that I chose the cluster count in a fair way, I need to use some statistical criterion to choose the model order. That's why I need a criterion that would choose the "reasonable" cluster count :)
Comment: @user5054 There are other approaches you could consider such as scree/elbow plot or gap statistic. However, I think you should be able to successfully argue that "We chose the number of clusters which yielded the most biologically consistent results". Especially if the ideal number of clusters by your interpretation has almost the optimal AIC...
Comment: ...It is not always the best idea to blindly take the model which has the optimal AIC - other considerations are important including interpretability. To help argue your case, I would definitely point out that AIC only holds asymptotically and your sample size is small. If you show an appreciation for these statistical nuances then you should be able to satisfy the reviewer.

Corrected AIC (AICC) for k-means


How can I overcome '8va' for a few notes?

Question: Let's say I have a whole passage marked with 8va, like below:

I want to play just the circled notes at their normal octave. Is there a way to denote this, instead of ending the 8va bracket and starting it again afterwards, which would be tiring if the passage with the 8va is long and I had to do this over and over again?
Comment: Since you're up in leger lines with those notes, why not just write them an octave lower than they are currently written? In this case move the G# to the second line from the bottom and the A to the second space, and then keep the 8va and they will be played at the correct pitches.
Comment: Just writing them an octave down will be the easiest for a performer to read. Anything else would be cumbersome. When I'm reading and I come across an 8va I go into "8va mode" and I just play an octave higher and my brain transposes automagically. If you break that for two notes it would really mess up my brain and I would have to stop in my sight reading. I can see how it will break the line that you have but it's the best way for the reader.
Comment: That's ok! I was sort of joking about that. What don't you like about writing those notes an octave lower? I'm brainstorming but not coming up with much else that would work. Uhh.. you could change the clef real quick - maybe not to bass clef (too many leger lines) but like tenor or something.
Comment: @ToddWilcox - looked up 'automagically'. Couldn't find it and think it really should be in the dictionary!
Comment: I honestly believed the response to my comment would either be, "Of course I thought of that Todd but here's why that won't work for me", or possibly, "Ok now I will delete this question in shame for having not seen such a straighforward solution", which is why I only made it a comment in the first place. Now I'm curious as to whether it's really a workable solution for you to just write them an octave lower.
Comment: Of course I thought of that Todd but I was hoping there was something else. Also I cannot delete the question if it has an upvoted answer. So it's too late for that
Comment: I was hoping for answer like 'here shev, use these letters and the 8va will be overcome
Comment: It's not that I don't like that solution, but since I'm already writing the notes up there, I was wondering if I didn't have to move down
Comment: You've got a point there. Maybe that's why there isn't something like what I'm looking for
Comment: I very strongly agree with Todd & Tim that it will be clearer if you change the visual position of the actual notes.
Comment: Ah @Tim, you beat me to it! (By six years…)
Answer: Just write them where they will be in 8va, as in an octave lower than they are right now. A reader will have more of a clue as to what's happening like that. So the G# will sit on the second line up, followed by the A on the second space up. Stems the other way - going up!


Two runners on same base

Question: October 17, 2017 Chicago Cubs v Los Angeles Dodgers, two Cubs runners occupied third base at the same time in the ninth inning. The Dodgers' third baseman tagged the second runner first, and then the first runner. The umpires called neither out. It did not seem that play was dead but apparently the umpires thought it was.
Please clarify the rule on this?
Answer: As you can see from the full play-by-play of the game, the hit by Albert Almora Jr. was a ground-rule double. This is due to the ground rule at Wrigley Field that any ball which lodges in the ivy is automatically a double, so the play was dead. The umpires were (as is almost always the case) completely correct.

Two runners on same base


Create alternate "View Modes" for Webform Submissions?

Question: In a View of type "Webform Submissions", I have added a field called "Webform submission: Rendered entity". This displays the entire webform submission (content from all fields in the webform). My problem is that I have a few field elements in the webform that have create/view/update access permissions set to the Administrator role only. This is important because the webform is used by the Commerce Webform Order module to collect extended info for orders. The submitted results of those restricted fields are being displayed in the View.
The purpose of the View is to display the submitted contents of the webform in the Commerce receipt email. I do so with the following code in my order receipt template:
{{ drupal_view('my_view_name', 'block_1', order_entity.getOrderId) }}

As I stated, the restricted field results are being included in the Views output. In the "Webform submission: Rendered entity" field settings in the View, there is a selector called "View Mode" in which the only option is "Default". Is there a way to create a new View Mode for webform submissions, which would include only those fields that I choose? If so, I would be able to add the name, phone, etc fields and exclude the Admin-specific fields, and then select that alternate View Mode in the Views field settings. I have scoured the Webform settings in the back-end of the website, and I cannot find any opportunity to create that new View Mode for Webform Submissions.
Answer: WebformSubmission entities have only a few 'view modes', as defined in \Drupal\webform\WebformSubmissionViewBuilder::buildComponents method. Which means that if you want to have another one, you need to override WebformSubmission entity type view builder handler. These view modes are not exposed to views, so you need to provide your own views field handler as well.


PowerBI reports and dashboard on-premise hosting

Question: As per available documentation, power bi reports and dashboards will be published to microsoft cloud. Can we configure on-premise server to host power bi reports just like SSRS reports?
Comment: What part of  the answer in your exact previous question did you not understand? It's not yet available, said the Microsoft Product Manager.
Comment: possible duplicate of [Power BI Reports hosting](http://stackoverflow.com/questions/30210638/power-bi-reports-hosting)
Comment: @SandeepYadav: Please use SO responsibly, do not post duplicates.
Answer: Not yet, it's something we're looking at. Please use support.powerbi.com to get notified when we have that capability.
HTH,
-Lukasz
http://dev.powerbi.com
http://blogs.msdn.com/powerbidev
Answer: This feature was in the Idea site of Power BI and they have just rolled out a preview.
https://ideas.powerbi.com/forums/265200-power-bi-ideas/suggestions/7948020-powerbi-reports-and-dashboard-on-premise-hosting
You can access the preview in the form of a Azure VM here: https://blogs.msdn.microsoft.com/sqlrsteamblog/2016/10/25/announcing-a-technical-preview-of-power-bi-reports-in-sql-server-reporting-services


WooCommerce: Get custom post type IDs or specific order status email notification

Question: Attempting to receive and display custom post type wcfa_attachment ID to only be displayed on a order status change notification email with the id of wc_order_status_email_731.
For example:
$attachments = get_posts( array( 'numberposts' => -1, 'post_parent' => $order_id, 'post_type'=> 'wcfa_attachment', 'order' => 'ASC', 'orderby' => 'ID', 'suppress_filters' => false/*,  'meta_key' => 'wcaf_is_active', 'meta_value' => true */ ));

I am able to target wc_order_status_email_731 and pull the order_id then display if for testing but not the custom post type post id's for wcfa_attachment associated with the order.
function add_content_specific_email( $order, $sent_to_admin, $plain_text, $email ) {
    if ( $email->id == 'wc_order_status_email_731' ) {

        $order_id = $order->get_id();

        /// works to print the order id so we know its being received
        print_r ($order_id);

        function get_attachments($order_id)
        {
            $result = array();

            $attachments = get_posts( array( 'numberposts' => -1, 'post_parent' => $order_id, 'post_type'=> 'wcfa_attachment', 'order' => 'ASC', 'orderby' => 'ID', 'suppress_filters' => false/*,  'meta_key' => 'wcaf_is_active', 'meta_value' => true */ ));
            foreach((array)$attachments as $attachment)
            {
                $result[$attachment->ID] = $this->get_attachment($attachment->ID);
            }
            return $result;
            print_r ($result);
        }
    }
}

Answer: The post ID  that you are looking for your custom post type, is simply $attachment->ID.
Now you should never embed a function in another function. Just add them apart.
Then you can call your function in the other function like:
function get_attachments( $order_id ) {
    $result = array();

    $attachments = get_posts( array( 'numberposts' => -1, 'post_parent' => $order_id, 'post_type'=> 'wcfa_attachment', 'order' => 'ASC', 'orderby' => 'ID', 'suppress_filters' => false/*,  'meta_key' => 'wcaf_is_active', 'meta_value' => true */ ));
    foreach((array)$attachments as $attachment){
        $result[$attachment->ID] = $this->get_attachment($attachment->ID);
    }
    return $result;
}

function add_content_specific_email( $order, $sent_to_admin, $plain_text, $email ) {
    if ( $email->id == 'wc_order_status_email_731' ) {

       $order_id = $order->get_id();

        /// works to print the order id so we know its being received
        print_r ($order_id);

        $attachments = get_attachments( $order_id ); // Call the function

        print_r ($attachments);
    }
}

It should better work.


CKEditor: Como deixar as imagens responsivas?

Question: Pessoal criei um site do zero e integrei ao CKEditor, meu site está em Bootstrap mas as imagens que envio pelo CKEditor não estão responsivas, já instalei um plugin (imageresponsive) mas não sei o que colocar no inputs sourceset e sizes:
Comment: Que plugin é esse? Sem saber qual é o plugin, nem que código ele gera, fica difícil de saber o que pode estar acontecendo. O nome *"sourceset"* me remete ao [elemento `picture`](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/picture) (que não é padronizado, e nem todo *browser* suporta), mas não sei se é isso mesmo que o plugin está usando ou se é outra coisa. Não há nada na documentação desse plugin que diga como usar?
Comment: @mgibsonbr é esse plugin, ele requer outros plugin para funcionar... http://ckeditor.com/addon/imageresponsive
Comment: Pelo jeito esse `srcset` não tem nada a ver com o `picture`, [ele é um atributo do `img` mesmo...](http://caniuse.com/#feat=srcset) O mesmo vale pro `sizes`. Nunca tinha visto esses atributos antes, então ainda não sei bem o que eles fazem, mas assim que der vou ler mais sobre o assunto (achei [uma explicação](https://ericportis.com/posts/2014/srcset-sizes/) - em inglês, e um bocado longa - depois se ninguém der uma resposta melhor vou tentar resumi-lo aqui).
Answer: Esse plugin parece estar utilizando os novos atributos srcset e sizes do elemento img. Aqui está a especificação, e o conjunto de browsers que já dão suporte a eles.
Sobre o que colocar nos campos, assumindo que cada campo se refere ao atributo correspondente, eu diria o seguinte:

URL: a URL da imagem "padrão", caso o browser não dê suporte ao srcset;
Sourceset: uma lista com duas ou mais imagens "candidatas", cada imagem acompanhada de sua largura em pixels ou seu device-pixel-ratio. Exemplo:
imagem1.jpg 100w, imagem2.jpg 200w, imagem3.jpg 400w

ou
 imagemA.jpg 1x, imagemB.jpg 1.5x, imagemC.jpg 2x

Sizes: uma lista com zero ou uma media query e a largura que a imagem deve ocupar naquelas condições. Exemplo:
(max-width: 500px) 95vw, (max-width: 1000px) 50vw, 500px

(Nota: eu não entendo nada sobre o funcionamento de device-pixel-ratio, mas encontrei um artigo detalhando os conceitos envolvidos.)
Um exemplo concreto seria o seguinte (veja o exemplo em "página toda" e redimensione a janela do browser para notar o efeito):

Aqui eu dou uma lista de imagens e suas larguras originais (cada uma 4 vezes maior que a anterior) e deixo o browser decidir qual delas é a mais apropriada pra ser exibida. Além disso, eu estabeleço as larguras que essas imagens ocuparão de fato na página sob diversas condições (expressas como media queries). Por fim, uma URL alternativa pro caso do browser não dar suporte a nada disso.
Qual o critério que o browser vai usar para escolher uma imagem da lista, não sei dizer (no meu teste com o Chrome e Firefox em uma tela de PC ambos exibiram o 1600x400 não importa o quanto eu redimensionasse a tela). Ao observar o exemplo em várias condições eu cheguei à conclusão que ele vai escolher a melhor imagem no momento da carga, e continuar usando-a depois não importa o que aconteça. Não sei se isso é por design ou alguma limitação na implementação.
Comment: Consegui, usei esse codigo...


Why does the coefficients of the expansion of (2x+1)^n produce the elements of the hypercube?

Question: Why does the coefficients of the  expansion of $(2x+1)^n$ produce the elements of the hypercube?
For elements I mean the number of vertices, edges, square faces, cubes, hypercubes, etc that the next hypercube will have. For example for the tesseract:
$(2x+1)^4=16 x^4+32 x^3+24 x^2+8 x+1$
Meaning that the Tesseract has: 16 vertices, 32 edges, 24 square faces, 8 cubes and 1 tesseract as its elements.
Comment: Could you add some more detail, what do you mean by components of the hypercube?
Comment: We understand hypercubes, your term "components" was non-standard...
Answer: The $k$-th coefficient of $(2x+1)^n$ is $2^k\binom nk$. The $n$-cube can be modeled as the graph with vertices the elements of $\{0,1\}^n$ connected iff they differ by one digit. Using this, can you count the number of $k$-cubes an $n$-cube contains? Note it has $2^n$ vertices (the $0$-cubes) and $n2^{n-1}$ edges (the $1$-cubes).

Spoiler The formula is $2^{n-k}\binom nk$. We can create a $k$-cube inside an $n$-cube as follows. First, pick a vertex. This can be done in $2^n$ ways. Now, pick $k$-spots in that vertex. Then start changing them. This gives a $k$-cube. But this $k$-cube has $2^k$ vertices, and any of those would have worked in obtaining it. Thus, the final result amounts to $2^n\binom nk/2^k=2^{n-k}\binom nk$.
Answer: There is no need to appeal to the binomial theorem; aim to see the result directly. The point is that we can think of an $n$-dimensional cube as the cartesian product $[0, 1]^n$ of $n$ copies of the $1$-dimensional cube, that we can break up $[0, 1]$ into two vertices $\{ 0 \}, \{ 1 \}$ and an open edge $(0, 1)$, and that cartesian product distributes over disjoint union. For example, when $n = 2$,
$$\left( \{ 0 \} + (0, 1) + \{ 1 \} \right)^2 = \{ 0 \} \times \{ 0 \} + \{ 0 \} \times (0, 1) + \{ 0 \} \times \{ 1 \} + (0, 1) \times \{ 0 \} + \dots$$
This is best explained by a picture but I can't find an existing picture that does what I want. In any case it's better to think of the result as an expansion of $(2 + e)^n$ where $e$ stands for the open edge so that the exponent of $e$ matches with the dimension of the corresponding face.
Answer: The number of $m$-dimensional cubes on a $n$-dimensional cube is given by
$$
2^{n-m}\binom{n}{m}
$$
(see Wikipedia).
Expanding your polynomial (via the Binomial theorem) gives the same numbers as above for the coefficients.


Can we find the GCD of a positive and negative number?

Question: I'm trying to evaluate whether or not $\gcd(p,q) = \gcd(-p,q)$ for non-zero integers $p$ and $q$.
I was wondering if it's possible to find $\gcd(-p,q)$.
If so, this statement should be true, correct?
Comment: If we define $\gcd(a,b)$, where $a$ and/or $b$ may be negative, in the usual way (*greatest* common divisor), then it is not hard to show directly from the definition that $\gcd(a,b)=\gcd(|a|,|b|)$.
Comment: The prescription Andre gives is also consistent with Bezout's theorem, since if $ap+bq=gcd(p,q)$ then $(-a)(-p)+bq=gcd(p,q)$ as well.
Comment: [Check my animated answer](http://meta.stackoverflow.com/a/270017/2917832).
Comment: Oh! yes. The word "greatest" .
Comment: Easy answer.  Sign doesn't matter $\gcd(\pm a, \pm b) = \gcd(a,b)$.  If you feel weird about that just keep in mind if $n|m$ so $m = kn$ then $-n|m$ ans $m = (-k)(-n)$ and $n|-m$ as $-m = (-k)n$.
Comment: See my answer below, if there is no notion of "greatest" in your ring, then do as @fleablood says.  Say that your ring "has $\gcd$'s" which means each $\gcd(a,b)$ is only unique up to mulplication by a unit.
Comment: Does this answer your question? [How to prove $\gcd(m, n) = \gcd(-m, n)?$](https://math.stackexchange.com/questions/92434/how-to-prove-gcdm-n-gcd-m-n)
Answer: Somebody has a good proof on proofwiki which makes sense to me.
It shows that
$$
\gcd\{a,b\}=\gcd\{|a|,b\}=\gcd\{a,|b|\}=\gcd\{|a|,|b|\} .
$$
Answer: Define the $\gcd$ like a categorial universal property.  This avoids having to make use of an ordering $\leq$ on the ring in which you desire a $\gcd$.
Let $R$ be a commutative ring with $1$.
Define $\gcd(a,b)$ to be any element $d \in R$ such that $d \mid a, b$ and if $e \mid a,b$ then $e \mid d$.
Then when $R = \Bbb{Z}$, $\gcd(-n, m),$ for $n,m \gt 0$ always has two elements, but they are "isomorphic" meaning they're the same upto a unit factor: $d = (-1)d'$.
So simply define $\gcd$ as you do "product" in a category i.e. there can be many products of objects $A, B$ but they are all isomorphic.


Unable to use AMQP protocol with network of brokers

Question: Am trying to use AMQP instead of Openwire for network broker connection but it doesn't work for me. The broker A connects with broker B and C in duplex mode.
Broker A configuration:

<beans
        xmlns="http://www.springframework.org/schema/beans"
        xmlns:amq="http://activemq.apache.org/schema/core"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
                  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd
                  http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">

                  <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
                        <property name="locations">
                          <value>file:${activemq.base}/conf/credentials.properties</value>
                        </property>
                  </bean>

                  <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker-A" networkConnectorStartAsync="true"
                                  dataDirectory="${activemq.base}/data" schedulePeriodForDestinationPurge="60000" persistent="false">

                        <managementContext>
                          <managementContext createConnector="false"/>
                        </managementContext>

                        <destinationPolicy>
                          <policyMap>
                                <policyEntries>
                                  <policyEntry queue=">" producerFlowControl="false" gcInactiveDestinations="true"
                                                           inactiveTimoutBeforeGC="300000" optimizedDispatch="true" useCache="false"/>
                                  <policyEntry topic=">" producerFlowControl="false" optimizedDispatch="true"/>
                                </policyEntries>
                          </policyMap>
                        </destinationPolicy>

                        <networkConnectors>

                          <networkConnector
                                          name="broker-B-topics"
                                          uri="static:(tcp://broker-B:61616)"
                                          duplex="true"
                                          decreaseNetworkConsumerPriority="true"
                                          dynamicOnly="true">
                                <excludedDestinations>
                                  <queue physicalName=">"/>
                                </excludedDestinations>
                                <dynamicallyIncludedDestinations>
                                  <topic physicalName="ActiveMQ.>"/>
                                  <topic physicalName="testing.>"/>
                                </dynamicallyIncludedDestinations>
                          </networkConnector>

                          <networkConnector
                                          name="broker-B-queues"
                                          uri="static:(tcp://broker-B:61616)"
                                          duplex="true"
                                          decreaseNetworkConsumerPriority="true"
                                          dynamicOnly="true"
                                          conduitSubscriptions="false">
                                <excludedDestinations>
                                  <topic physicalName=">"/>
                                </excludedDestinations>
                                <dynamicallyIncludedDestinations>
                                  <queue physicalName="testing.>"/>
                                </dynamicallyIncludedDestinations>
                          </networkConnector>

                          <networkConnector
                                          name="broker-C-topics"
                                          uri="static:(tcp://broker-C:61616)"
                                          duplex="true"
                                          decreaseNetworkConsumerPriority="true"
                                          dynamicOnly="true">
                                <excludedDestinations>
                                  <queue physicalName=">"/>
                                </excludedDestinations>
                                <dynamicallyIncludedDestinations>
                                  <topic physicalName="ActiveMQ.>"/>
                                  <topic physicalName="testing.>"/>
                                </dynamicallyIncludedDestinations>
                          </networkConnector>

                          <networkConnector
                                          name="broker-C-queues"
                                          uri="static:(tcp://broker-C:61616)"
                                          duplex="true"
                                          decreaseNetworkConsumerPriority="true"
                                          dynamicOnly="true"
                                          conduitSubscriptions="false">
                                <excludedDestinations>
                                  <topic physicalName=">"/>
                                </excludedDestinations>
                                <dynamicallyIncludedDestinations>
                                  <queue physicalName="testing.>"/>
                                </dynamicallyIncludedDestinations>
                          </networkConnector>

                        </networkConnectors>

                        <systemUsage>
                          <systemUsage>
                                <memoryUsage>
                                  <memoryUsage percentOfJvmHeap="70"/>
                                </memoryUsage>
                                <storeUsage>
                                  <storeUsage limit="1 gb"/>
                                </storeUsage>
                                <tempUsage>
                                  <tempUsage limit="1 gb"/>
                                </tempUsage>
                          </systemUsage>
                        </systemUsage>

                        <transportConnectors>
                          <transportConnector name="amqp" uri="amqp://0.0.0.0:61616"/>
                          <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>
                        </transportConnectors>

                  </broker>

                  <import resource="jetty.xml"/>

                </beans>

Broker B configuration:

<beans
                                xmlns="http://www.springframework.org/schema/beans"
                                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

                  <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
                        <property name="locations">
                          <value>file:${activemq.conf}/credentials.properties</value>
                        </property>
                  </bean>

                  <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
                                lazy-init="false" scope="singleton"
                                init-method="start" destroy-method="stop">
                  </bean>

                  <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker-B" dataDirectory="${activemq.data}"
                                  networkConnectorStartAsync="true" schedulePeriodForDestinationPurge="60000" persistent="false">

                        <destinationPolicy>
                          <policyMap>
                                <policyEntries>
                                  <policyEntry queue=">" producerFlowControl="false" gcInactiveDestinations="true"
                                                           inactiveTimoutBeforeGC="300000" optimizedDispatch="true" useCache="false"/>
                                  <policyEntry topic=">" producerFlowControl="false" optimizedDispatch="true"/>
                                </policyEntries>
                          </policyMap>
                        </destinationPolicy>

                        <plugins>
                          <timeStampingBrokerPlugin ttlCeiling="0" zeroExpirationOverride="0" futureOnly="true"/>
                        </plugins>

                        <managementContext>
                          <managementContext createConnector="false"/>
                        </managementContext>

                        <systemUsage>
                          <systemUsage>
                                <memoryUsage>
                                  <memoryUsage percentOfJvmHeap="70"/>
                                </memoryUsage>
                                <storeUsage>
                                  <storeUsage limit="1 gb"/>
                                </storeUsage>
                                <tempUsage>
                                  <tempUsage limit="1 gb"/>
                                </tempUsage>
                          </systemUsage>
                        </systemUsage>

                        <transportConnectors>
                          <transportConnector name="amqp" uri="amqp://0.0.0.0:61616"/>
                          <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61615"/>
                        </transportConnectors>
                  </broker>

                </beans>

Broker C configuration:
<beans
                                xmlns="http://www.springframework.org/schema/beans"
                                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                                xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
                  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

                  <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
                        <property name="locations">
                          <value>file:${activemq.conf}/credentials.properties</value>
                        </property>
                  </bean>

                  <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
                                lazy-init="false" scope="singleton"
                                init-method="start" destroy-method="stop">
                  </bean>

                  <broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker-C" dataDirectory="${activemq.data}"
                                  networkConnectorStartAsync="true" schedulePeriodForDestinationPurge="60000" persistent="false">

                        <destinationPolicy>
                          <policyMap>
                                <policyEntries>
                                  <policyEntry queue=">" producerFlowControl="false" gcInactiveDestinations="true"
                                                           inactiveTimoutBeforeGC="300000" optimizedDispatch="true" useCache="false"/>
                                  <policyEntry topic=">" producerFlowControl="false" optimizedDispatch="true"/>
                                </policyEntries>
                          </policyMap>
                        </destinationPolicy>

                        <plugins>
                          <timeStampingBrokerPlugin ttlCeiling="0" zeroExpirationOverride="0" futureOnly="true"/>
                        </plugins>

                        <managementContext>
                          <managementContext createConnector="false"/>
                        </managementContext>

                        <systemUsage>
                          <systemUsage>
                                <memoryUsage>
                                  <memoryUsage percentOfJvmHeap="70"/>
                                </memoryUsage>
                                <storeUsage>
                                  <storeUsage limit="1 gb"/>
                                </storeUsage>
                                <tempUsage>
                                  <tempUsage limit="1 gb"/>
                                </tempUsage>
                          </systemUsage>
                        </systemUsage>

                        <transportConnectors>
                          <transportConnector name="amqp" uri="amqp://0.0.0.0:61616"/>
                          <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61615"/>
                        </transportConnectors>
                  </broker>

                </beans>

Error what am getting in broker-A:

INFO   | jvm 1    | 2018/01/18 22:25:45 |  INFO | error with pending remote brokerInfo on: tcp://broker-B/xxx.xxx.xxx.xxx:61616@14953
INFO   | jvm 1    | 2018/01/18 22:25:45 | java.io.IOException: Unknown data type: 80
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:471)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.v12.BaseDataStreamMarshaller.looseUnmarsalCachedObject(BaseDataStreamMarshaller.java:487)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.v12.ConnectionInfoMarshaller.looseUnmarshal(ConnectionInfoMarshaller.java:154)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:367)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:278)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |  INFO | error with pending remote brokerInfo on: tcp://broker-B/xxx.xxx.xxx.xxx:61616@14954
INFO   | jvm 1    | 2018/01/18 22:25:45 | java.io.IOException: Unknown data type: 80
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:471)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.v12.BaseDataStreamMarshaller.looseUnmarsalCachedObject(BaseDataStreamMarshaller.java:487)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.v12.ConnectionInfoMarshaller.looseUnmarshal(ConnectionInfoMarshaller.java:154)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:367)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:278)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)[activemq-client-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |  WARN | Network connection between vm://broker-A#0 and tcp://broker-B/xxx.xxx.xxx.xxx:61616@14953 shutdown due to a remote error: {}
INFO   | jvm 1    | 2018/01/18 22:25:45 | java.util.concurrent.TimeoutException
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport$FutureBrokerInfo.get(DemandForwardingBridgeSupport.java:1828)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport.collectBrokerInfos(DemandForwardingBridgeSupport.java:388)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:113)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:366)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |  INFO | Network Connector DiscoveryNetworkConnector:broker-C-topics:BrokerService[broker-A] started
INFO   | jvm 1    | 2018/01/18 22:25:45 |  WARN | Network connection between vm://broker-A#6 and tcp://broker-B/xxx.xxx.xxx.xxx:61616@14954 shutdown due to a remote error: {}
INFO   | jvm 1    | 2018/01/18 22:25:45 | java.util.concurrent.TimeoutException
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport$FutureBrokerInfo.get(DemandForwardingBridgeSupport.java:1828)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport.collectBrokerInfos(DemandForwardingBridgeSupport.java:388)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:113)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:366)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |  INFO | Network Connector DiscoveryNetworkConnector:broker-C-queues:BrokerService[broker-A] started
INFO   | jvm 1    | 2018/01/18 22:25:45 |  INFO | broker-A bridge to Unknown stopped
INFO   | jvm 1    | 2018/01/18 22:25:45 |  INFO | broker-A bridge to Unknown stopped
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport.collectBrokerInfos(DemandForwardingBridgeSupport.java:388)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport.access$500(DemandForwardingBridgeSupport.java:113)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at org.apache.activemq.network.DemandForwardingBridgeSupport$5.run(DemandForwardingBridgeSupport.java:366)[activemq-broker-5.15.2.jar:5.15.2]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_91]
INFO   | jvm 1    | 2018/01/18 22:25:45 |       at java.lang.Thread.run(Thread.java:745)[:1.8.0_91]

Not sure why its not connecting properly. Based on another thread http://activemq.2283324.n4.nabble.com/Unable-to-configure-Network-of-brokers-successfully-with-AMQP-td4689623.html need configure Openwire protocol's only for Network connections.
Answer: You can't do this, you need to use an OpenWire based connection for the network connection between brokers.  ActiveMQ supports AMQP clients but for networks you must use the ActiveMQ OpenWire connections, no other network connections are supported.


Chord-spelling change in Chopin's Mazurka Op. 24 n.4

Question: In the manuscript of measure 138 (marked mancando) of his Mazurka, Op. 24 n.4, Chopin crosses out the (same) chord on the third beat and respells it—I assume in preparation for the key change that begins in the next measure):

(Note that after m.1, Chopin does not repeat the 5 flats at the beginning of every line of this Mazurka.)
I believe that most editions subsequent to the Frist French Edition stay true to this:

However, Klindworth (1880) in his Complete Edition ignores Chopin's correction and keeps the chord unchanged:

I have two questions:
(1) To my ears, the change to the major actually starts on the second, rather than the first, beat of the next measure (139), so why does Chopin anticipate it in m. 138?
(2) If I am correct, is this Klindworth's reasoning, or is he making some other formal/notational point by keeping the spelling unchanged?
(Klindworth also starts the mancando a measure earlier.)
All quotes from IMSLP.
Answer:
Even though the change to major isn't confirmed until beat 2, it does happen (abstractly) on beat 1, with a clear phrase break beforehand. Therefore, the change is signaled in measure 138, but only on the final beat.

I suspect Klindworth just found it easier to read to keep the chord the same, and that the signal of the tonal shift wasn't needed.


SharePoint 2016 Templates in VS2015

Question: New to StackOverflow so sorry if I'm doing this wrong.  Trying to get set up with SharePoint 2016 Development.  Moving from SP2010 and need to look into getting some things updated.  Have a 2016 single server farm set up for evaluation/development and have VS2015 installed on machine, but cannot for the life of me get SP2016 development templates to show up in VS.  I had them briefly after installing the Office Development Tools preview, but after upgrading it to update 2 they're gone again!  Please help.
Also, looking for good information on converting SP2010 projects to SP2016 if that's possible.
Comment: This is not a question for a recommendation
Comment: Sorry mate: "_Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam._"
Answer: You need to install: Microsoft Office Developer Tools Preview 2 for Visual Studio 2015

SharePoint 2016 Templates in VS2015


Is this THE MOST comprehensive commands to completely reset your iptables?

Question: I'm running a VPS, and would like to reset the iptables' rules to its fresh-out-of-the-box
default state. These are the commands I've come up with:
#!/bin/sh
echo "Resetting all iptables rules..."

#Reset default table policies
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

#Reset nat table policies
iptables -t nat -P PREROUTING ACCEPT
iptables -t nat -P POSTROUTING ACCEPT
iptables -t nat -P INPUT ACCEPT
iptables -t nat -P OUTPUT ACCEPT

#Reset mangle table policies
iptables -t mangle -P PREROUTING ACCEPT
iptables -t mangle -P POSTROUTING ACCEPT
iptables -t mangle -P INPUT ACCEPT
iptables -t mangle -P OUTPUT ACCEPT
iptables -t mangle -P FORWARD ACCEPT

#Reset raw table policies
iptables -t raw -P PREROUTING ACCEPT
iptables -t raw -P OUTPUT ACCEPT

#Flush all rules and delete empty chains
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X

QUESTIONS:

Are these rules comprehensive enough? I've messed with my iptables and I just want to start from a clean slate.
Will I be locked out of my VPS if I reboot?
Do I need to use the -Z command on every table to zero the packet and byte counters on all rules in a chain? E.g. "iptables -t nat -Z" (and repeat the same for all other tables)?

Thanks!
Comment: "Will I be locked out of my VPS if I reboot?" <- that
Comment: I suggest you install Linux on a virtual machine on your desktop computer, and learn how to use IPTables there. Doing remote firewall work is _not_ safe unless you have a second connection for out-of-band management.
Comment: By the way, HB, you might want to have a small answer-acceptance festival at some point; you're beginning to pile up questions with answers, and no acceptances.
Comment: @pauska, could you please elaborate on the "not safe" part? Thanks!
Answer:
I think those are pretty good; I can offhand think of no rule that would escape that pruning.
It depends.  What are your current arrangements for rules on reboot?  If they amount to DROPping everything, then yes, you'll lock yourself out.  The script you've shown above is lovely (see 1.), but it's not going to magically be run on reboot.
It depends whether you want the counters to be zeroed or not.

As for not getting locked out, I agree with pauska that doing remote firewall work can be tricky.  That's why, before I commit any change I've made to a remote system's firewalls, I do
# at now+5min
at> service iptables stop
at> ^D

If the changes I then commit lock me out, well, I'll be OK to get back in in five minutes.  If they don't, I can use atrm to remove the job I just submitted.  It's saved my bum a few times!  (NB: that service command is good for Red Hat Linuces, you may need to find an equivalent for other unices.)
Comment: For safely iptables apply, you can use iptables-apply, who will try to apply your rules and then ask you if you are ok. If you say yes (then you have connection) the rules will be apply, if not the work will be undo.
Comment: I love UNIX, there's always more than one *good* way to do the job!  Thanks, Brigo.


Dual optimization problem for standard linear programming

Question: Let $K$ be a proper convex cone. Let's have a cone problem
$$
\text{min } \textbf{c}^T\textbf{x} \\ \text{s.t. } A\textbf{x} =\textbf{b}\\\textbf{x}\succeq_K \textbf{0}
$$
How to derive the dual program for this one? Thx
I know that when solving duals one works with the Lagrange dual function - but in this case, I have troubles because of the cone inequality.
Answer: Notice:

If $K$ is a proper cone, we have:
$$\mathbf{x} \succeq_{K} \mathbf{y} \text{ if and only if }
\boldsymbol{\lambda}^T \mathbf{x} \geq \boldsymbol{\lambda}^T
\mathbf{y} \text{ for all } \boldsymbol{\lambda} \in K^*,$$ where
  $K^*$ is the dual cone of $K$.

Suppose the generalized inequality is the component wise inequality defined by $K = \mathbb{R}_+^n$. For the standard linear programming (LP) problem we have:
\begin{aligned}
    & \min_{\mathbf{x} \in \mathbb{R}^n} && \hspace{-1mm} \mathbf{c}^T \mathbf{x} \\
    & \hspace{2mm} \text{s.t.}  && \hspace{-4mm} \mathbf{A}\mathbf{x} = \mathbf{b} \\
    & \text{}       && \hspace{-2mm} \mathbf{x}  \succeq \mathbf{0}
\end{aligned}
The Lagrangian can be written as
\begin{aligned}
\mathcal{L}(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\nu}) &= \mathbf{c}^T \mathbf{x} - \boldsymbol{\lambda}^T \mathbf{x}+ \boldsymbol{\nu}^T(\mathbf{A}\mathbf{x}-\mathbf{b}) \\
&= (\mathbf{c}-\boldsymbol{\lambda}+\mathbf{A}^T\boldsymbol{\nu})^T \mathbf{x} - \mathbf{b}^T \boldsymbol{\nu}.
\end{aligned}
As it is clear $\mathcal{L}$ is an affine function over $\mathbf{x}$ and hence it is unbounded below except for the case $\mathbf{c} \boldsymbol{\lambda}+\mathbf{A}^T\boldsymbol{\nu} = \mathbf{0}$. Thus, the dual function is
\begin{aligned}
g(\boldsymbol{\lambda},\boldsymbol{\nu}) &= \inf_{\mathbf{x}} \mathcal{L}(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\nu}) \\
&= \begin{cases}
     - \mathbf{b}^T \boldsymbol{\nu}  & \quad \text{if:} \;\mathbf{c}-\boldsymbol{\lambda}+\mathbf{A}^T\boldsymbol{\nu} = \mathbf{0}\\
    -\infty  & \quad \text{otherwise}
  \end{cases},
\end{aligned}
where $\text{dom } g = \{(\boldsymbol{\lambda},\boldsymbol{\nu})| \; \mathbf{c}-\boldsymbol{\lambda}+\mathbf{A}^T\boldsymbol{\nu} = \mathbf{0}\}$.
Then, the Lagrange dual problem of the LP problem can be obtained as
\begin{aligned}
    & \hspace{-10mm} \max_{(\boldsymbol{\lambda},\boldsymbol{\nu}) \in \text{dom } g} && \hspace{-1mm} g(\boldsymbol{\lambda},\boldsymbol{\nu}) \\
    & \hspace{-3mm} \text{s.t.}  && \hspace{-1mm} \boldsymbol{\lambda}  \succeq \mathbf{0},
\end{aligned}
and hence the dual problem can be described as
\begin{aligned}
   & \hspace{6mm}  \max_{\boldsymbol{\lambda},\boldsymbol{\nu}} && \hspace{-5mm}  - \mathbf{b}^T \boldsymbol{\nu}  \\
    & \hspace{8mm} \text{s.t.}  && \hspace{-3mm} \boldsymbol{\lambda}  \succeq \mathbf{0} \\
& {} && \hspace{-3mm} \mathbf{c}-\boldsymbol{\lambda}+\mathbf{A}^T\boldsymbol{\nu} = \mathbf{0}.
\end{aligned}
Comment: I would suggest that you take this a bit further and move the domain constraint for $g$ out of the objective and make it an explicit constraint, as is standard practice.
Comment: Dear @MichaelGrant: Thanks for your comment. It is edited.
Comment: Bingo! Nicely done.


Invalid JSON inside .ajax() function error

Question: In my $.ajax I have
error: function (xhr, ajaxOptions, thrownError) {
            var result = $.parseJSON(xhr.responseText);
            alert(result.ErrorMessage);
  }

Even though this has worked for all my previous apps, this time this choked and gave me an invalid JSON error. I am using jQuery 1.5, asp.net MVC 3.
PS:  I looked at the jQuery documentation but I didn't see the example of what JSON consumed by error should be.
Answer: What is the value in xhr.reponseText? Just print it out using console.log(). Unless you are formatting your error pages as JSON, you're probably getting some standard 500 HTML from the server, that's why you can't parse it as JSON.
jQuery does not dictate the responseText that is returned by the server in case of an error
Comment: My bad, I neglected to add a handler to the controller action. The handler is the routine that formats the JSON properly.


Why KeyPair.getPrivate().getEncoded() is null?

Question: public KeyPair generateKeyPair(@NonNull Context context, @NonNull String alias) {
    Calendar startDate = Calendar.getInstance();
    Calendar endDate = Calendar.getInstance();
    endDate.add(Calendar.YEAR, 1);

    KeyPairGenerator keyPairGenerator;
    try {
        keyPairGenerator = KeyPairGenerator.getInstance("RSA", "AndroidKeyStore");
        AlgorithmParameterSpec spec;
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.M) {
            spec = new KeyGenParameterSpec.Builder(alias, KeyProperties.PURPOSE_SIGN | KeyProperties.PURPOSE_VERIFY)
                    .setCertificateNotBefore(startDate.getTime())
                    .setCertificateNotAfter(endDate.getTime())
                    .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_RSA_PKCS1)
                    .setDigests(KeyProperties.DIGEST_SHA256, KeyProperties.DIGEST_SHA512)
                    .build();
        } else {
            spec = new KeyPairGeneratorSpec.Builder(context)
                    .setAlias(alias)
                    .setSubject(new X500Principal("CN=fake"))
                    .setSerialNumber(BigInteger.ONE)
                    .setStartDate(startDate.getTime())
                    .setEndDate(endDate.getTime())
                    .build();
        }
        keyPairGenerator.initialize(spec);
    } catch (Exception e) {
        e.printStackTrace();
        return null;
    }
    return keyPairGenerator.generateKeyPair();
}

i know privateKey.getEncoded can be null, but how to get the privateKey base64 value?
Comment: Were you able to found a solution for this issue?
Answer: If you were using a provider based on a hardware cryptographic device AndroidKeyStore it has been set up not to publish private keys outside of it. Therefore, Key.getEncoded() on the private key might actually return null. More details here


Easiest way to store NSCache full of NSImages

Question: What is the simplest way to persistently store a NSCache filled with NSImages? I'm not looking for Core Data, which seems like overkill for just storing a dictionary with images.
Answer: If all you have is a set of pictures, I'd simply get a path from the user and use NSImage's TIFFRepresentation to get instances of NSData for each picture, which can then be written to the path using NSFileManager and NSFileHandle.
If you have info accompanying the images, like tags, I'd put that, along with identifying titles for the images, in an array of dictionaries. Then you can use NSArray's writeToFile:atomically: method to write out a plist.
Comment: `NSCache` doesn't have a `writeToFile`-like method, and as far as I can see there's no way to convert it to a `NSDictionary`.
Comment: Your archival routine would get the objects out of the cache and put them into dictionaries, which you would put into an array, upon which you would then call `writeToFile`.
Comment: The problem is as far as I can see there's no method that gets the objects out of a NSCache apart from the method to get an object for a given key. Therefore the only way would be to keep a separate array of the NSCache keys, but I asked this question because i thought there would be a better way...


Bridge on a bonded interface with systemd-networkd

Question: So I recently changed my server from a Debian based distribution that used the old
ifupdown style configs over to systemd-networkd.
What I want is, my two primary interfaces, bonded, and then a bridge attached to that bonded interface.
There will be an interface on the bridge, for the server, in the below example this is bridge0.2. The intention is that the host can communicate with guests operating on bridge0. VMs, and containers might be attached to bridge0. They may be on whatever VLAN they choose.
With that I had this configuration:
/etc/network/interfaces
auto lo
iface lo inet loopback

iface eth0 inet manual
iface eth1 inet manual

auto bond0
iface bond0 inet manual
  bond-slaves eth0 eth1
  bond-mode 802.3ad
  bond-xmit-hash-policy layer2+3

auto bridge0
iface bridge0 inet manual
  bridge-ports bond0
  bridge-stp off
  bridge-fd 0
  bridge-vlan-aware yes
  bridge-vids 2-4094

auto bridge0.2
iface bridge0.2 inet static
  address 192.168.2.253/24
  gateway 192.168.2.1

I've tried to repeat the same thing with systemd-networkd. I can get VLAN 2 up on a bonded interface, but I can't seem to get the VLAN on the bridge up.
If I do DHCP=true on the bridge0 interface, I can an address via VLAN=1. This makes me think there's some issue with the VLAN exiting the bridge.
I've tried for far too long to get this to work, and I really don't know what other combination to try. Here is what I have:
10-all-nic-to-bond0.network
[Match]
Name=eno1 eno2
Type=ether

[Network]
Bond=bond0

20-bond0.netdev
[NetDev]
Name=bond0
Kind=bond

[Bond]
Mode=802.3ad
MIIMonitorSec=1s
LACPTransmitRate=fast
UpDelaySec=2s
DownDelay=8s
TransmitHashPolicy=layer2+3

20-bond0.network
[Match]
Name=bond0

[Network]
BindCarrier=eno1 eno2
Description=Unconfigured bond interface
Bridge=bridge0

30-bridge0.netdev
[NetDev]
Name=bridge0
Kind=bridge

[Bridge]
VLANFiltering=false
STP=false

30-bridge0.network
[Match]
Name=bridge0

[Network]
VLAN=vlan2

[BridgeVLAN]
VLAN=2-4094

40-vlan2.netdev
[NetDev]
Name=vlan2
Kind=vlan

[VLAN]
Id=2

40-vlan2.network
[Match]
Name=vlan2

[Network]
Description=VLAN for direct access to gateway
DHCP=true

#[Address]
#Address=192.168.253/24
#DNS=192.168.2.1

#[Route]
#Gateway=192.168.2.1

Answer: I think I got this working, it was VLANFiltering=true that was causing the issues, changing this to false allowed me to exit the bridge on VLAN2.


Can additive boosting be used in case a classification?

Question: In case of a regression we can apply a boosting approach as follows:

Train a very simple model using the data set.
Find a difference between the predictions and targets and use this difference as a new target.
Train a new model using a new target.
Repeat it as long as it make sense.

Can this idea be generalised on the case of a classification?
For example, we train a very simple model to predict probabilities of two exclusive classes. Now, we need some how define a difference between the targets and predictions. However, it is not straightforward anymore. Is there a way to do it?
Comment: You are talking about residuals right? Training new models on the residuals of the previous models, such as what XGBOOST does? It does so on both classification and regression.
Answer: Yes, it works perfectly for the case of classification because we use the deviance residuals. Deviance residuals are directly related with the contribution each point has to the overall likelihood score. The deviance residuals are commonly used to check the model fit at each observation for generalized linear models; a gradient boosting machine is no different (see Elements of Statistical Learning, Hastie et al. (2009) Ch. 10.2 "Boosting Fits an Additive Model" for the relation between boosting and additive models).
When dealing with a binary classification task, we often assume that we are concerned with a binomial distribution. Because the outcome variable has only two possible types it naturally extends to classification tasks and to binomial (or binary) logistic regression. Basing our boosting procedure to the binomial likelihood and the subsequent residuals allows us to directly move from standard regression tasks to classification tasks; CV.SE as a short but to-the-point exposition on the relation between deviance residuals and binomial (log)-likelihood here.


Logit Model for panel data [r]

Question: I would like to ask preferably preferably econometrics and statistics community about logit model for panel data.
When dealing with panel data one must take into account individual effects (individual heterogenity) of each cross-sectional unit,
when we suppose that individual effects are correlated with indipendent variable(s) we use fixed effect approach, that is each cross sectional unit gets its own intercept, or within transformation, where we eliminate individual heterogenity from equation.
For panel data in R, very good package is plm that does pretty much of panel data models,
when having binary dependent variable we can do something like linear probability model, when we would use within model from plm on dummy dependent variable as a baseline model.
What are your best packages that you can recommend for panel data models with binary dependent variable,
I found bife package and paper for it :fixed effect logit. Are there another packages and papers for it in R for panel data that account for individual heterogenity in models?
I know about lme4, lmerTest but I do not know if these models account for it.
Also does anyone know about package and documentation for Spatial panel data logit?, I know
there is package splm for spatial panel data
Answer: Knowing from Panel data Econometrics with R , written by Yves Croissant & Giovanni Millo.
for binomial panel data, random effect you can use pglm,
fixed effect can use clogit from survival package.

Logit Model for panel data [r]


how to use the last id SQLite

Question: I am new to the android world and have a problem with an id. What i need is that when the user clicks on new match it will insert a new row into the db. This is working and i get the lastId but now i need this id in the next activities. How can i store that id so i can use it elsewhere?
This is how i insert the new match:
public void newMatch(WedstrijdenGeschiedenis wedstrijd){

// 1.
SQLiteDatabase db = this.getWritableDatabase();

// 2.
DateFormat dateFormat = new SimpleDateFormat("dd-MM-yyyy");
Date date = new Date();
ContentValues values = new ContentValues();
values.put(KEY_DATUM, dateFormat.format(date)); // get datum

// 3.
long lastId = db.insert(TABLE_WEDSTRIJD, // table
null, //nullColumnHack
values); // key/value -> keys = column names/ values = column values
Log.d("New Match","ID ="+lastId);

// 4. close
db.close();

}

so i see the lastId in LogCat but i don't know how to store it for further use. I tried void but offcourse that is not possible on void. Sorry for the dummy question
Comment: I'm not sure I understand your question. The id you get back on `lastId` is the id that was automatically created as your primary key for that record. It is already stored in the database.
Comment: Why? The database knows that id and if you insert from another activity it will still know it and use another id (usually just 1 higher).
Answer: change void to long and add a return statement that returns the lastId
public long newMatch(WedstrijdenGeschiedenis wedstrijd){
  // Your other code
  return lastId;
}

Access it with:
long lastId= db.newMatch(new WedstrijdenGeschiedenis());

Comment: In my activity i do this:
        MySQLiteHelper db = new MySQLiteHelper(this);
        db.newMatch(new WedstrijdenGeschiedenis());
So how do i get the id there?
Comment: yes it is. sorry for my ignorance and thank you for your help! This is what i needed

how to use the last id SQLite


Convolutional neural network: why would training accuacy and well as validation accuracy fluctuate wildly?

Question: I am training a convnet on a binary classification problem using medical images. I;m doing a preliminary evaluation of various shallow nets to get a sense of what the best hyperparameters are likely to be. My dataset is small - about 2500 images in each of the 2 classes. I'm getting wild fluctuations in training accuracy as well as validation accuracy. Most of the questions online discuss this in the context of validation accuracy. But what does in mean when training accuracy is also fluctuating - to give an example:
<code> Epoch 1/40
 11/11 [==============================] - 108s 10s/step - loss: 0.9150 - acc: 0.6183 - val_loss: 3.7535 - val_acc: 0.7344
 Epoch 2/40
 11/11 [==============================] - 144s 13s/step - loss: 1.1297 - acc: 0.5153 - val_loss: 11.2487 - val_acc: 0.2656
 Epoch 3/40
 11/11 [==============================] - 101s 9s/step - loss: 1.3976 - acc: 0.1474 - val_loss: 2.3932 - val_acc: 0.8516
 Epoch 4/40
 11/11 [==============================] - 149s 14s/step - loss: 0.7517 - acc:   0.4707 - val_loss: 3.1836 - val_acc: 0.7266
 Epoch 5/40
 11/11 [==============================] - 101s 9s/step - loss: 0.7319 - acc: 0.4734 - val_loss: 0.8679 - val_acc: 0.8750
</code>
My intution was that the learning rate was too high and I was bouncing around a local minimum but I lowered the learning rate from 0.01 to 1e-3 and it had no effect. I also wondered about the possibility of the network throwing NaN variables but I have no experience of that or what to do about it. Incidentally I am using the Adam optimizer which seems to across the board result in better test accuracies. I should add though that by best shallow network for 20 Epoch reached 62% accuracy on a binary problem (not much good).
If someone could give me some intuition on this pattern of behavior it would be great - I am pretty sure this is a symptom of one thing....I think....
If it were lack of data then why would I not see a gradual overfit with a really bad validation accuracy?
Comment: Considering that your loss goes up between epochs, it sounds like this is due to small batch size. This isn't *necessarily* bad, but you can try to increase the batch size to see if it makes things smoother.
Comment: Did you wait for more epochs? It might indeed take about 5 epochs for the optimizer to find a way.
Comment: I'm trying with a longer training cycle now but would you expect these kinds of fluctuations over 10 or 20 epochs? If not, I guess I am looking for some kind of intuition as to why this is happening.
Comment: Wht kind of network are you using ? It could be that it's too deep or wide for your dataset. Try to reduce the number of parameters. I shall add that this question would be more appropriate on the Cross Validated SE and i'm voting for closing.
Comment: Something I have noticed (and I'll admit, this is a rookie error) but my classes are in order in datasets and I'm using hdf5 files. I haven't been shuffling the data at all. I would imagine that would cause the effects above....if all batches for a while are one class and then another. Correct me if I'm wrong.
Answer: OP's network not only exhibits large fluctuations in accuracy, but also large fluctuations in the loss. This is a classic symptom of using too large a learning rate, too small a batch size, or both.
Even though OP reduced the learning rate from $10^{-2}$ to $10^{-3}$, this smaller learning rate might not be a small enough learning rate for the problem. Finding a good configuration of neural network parameters is like lock picking, and involves a lot of trial and error; trying one thing probably isn't enough experiments to find a good configuration of model parameters. Some optimizers, like SGD, can be very picky about the learning rate used.
More information:

What should I do when my neural network doesn't learn?
What could be the reasons that making validation loss jumping up and down?
Answer: One scenario where you see fluctuating training accuracy and loss is when you print out mini-batch accuracy and mini-batch loss. If that is the case, then try printing out train accuracy and loss for the whole training set. Accuracy should be increasing and loss should be decreasing if the network is getting trained properly.


SLURM with "partial" head node

Question: I am trying to install SLURM with NFS on a small ubuntu 18.04 HPC cluster, in a typical fashion, e.g. configure controller (slurmctld) and clients (slurmd) and shared directory, etc. What I am curious about is, is there a way to set it up such that there is a controller on a portion of the head node, and other drives in the head node are used by the resource partitioning routine, like the other nodes? Is there a way to accomplish this using the SLURM configuration file?
I am essentially asking how to maximise the resources if the controller will only be doing light work.
Thank you, cheers!
Comment: I can't fully understand your question. NFS has nothing to do with SLURM. What you're trying to achieve? Do you want to use your headnode (where you run slurmctld) as a computing node so you can run both slurmctld and slurmd?
Comment: The NFS was a possibly irrelevant detail, except for maybe if the shared home drive was impacted by the set up. Yes, this is what I wanted to do, computations on head node instead of wasting its resources, since it is a small cluster and I want to maximize everything. In any case, this is no longer my project, so I have no interest in it anymore. But, if you have tips for the future I would gladly hear them
Comment: I will write an answer, but yes you can run slurmd on the headnode, we usually do this on small machines.
Answer: You're trying to consume the headnode as a compute node. This is perfectly normal on small clusters and even on workstations that have SLURM as a queue system to consume it as an easier way to enqueue jobs or to share the compute power among a group of users that have access to this workstation.
To do this just enable slurmd on the same machine that runs slurmctld. Remember to add the respective node and partition entry on /etc/slurm/slurm.conf with the compute specifications. As example, you should have something like this:
ClusterName=Cloyster
ControlMachine=charizard.cluster.example.com
SlurmUser=slurm
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm/ctld
SlurmdSpoolDir=/var/spool/slurm/d
SwitchType=switch/none
MpiDefault=none
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmdPidFile=/var/run/slurmd.pid
ProctrackType=proctrack/pgid
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log
JobCompType=jobcomp/none
TaskPlugin=task/affinity
PropagateResourceLimitsExcept=MEMLOCK
AccountingStorageType=accounting_storage/filetxt
Epilog=/etc/slurm/slurm.epilog.clean
SlurmctldParameters=enable_configless

ReturnToService=2
NodeName=charizard Sockets=2 CoresPerSocket=64 ThreadsPerCore=2 State=UNKNOWN
PartitionName=execution Nodes=charizard Default=YES MaxTime=720:00:00 State=UP Oversubscribe=EXCLUSIVE

Observe that NodeName have the hostname of the control machine.
Comment: Assigning a slurmctld management daemon to the head looks like exactly what would be needed. Thanks!

SLURM with "partial" head node


Git: How to check if the local and remote branches point to the same commit (i.e. in sync)?

Question: How could I check if feature_branch and origin/feature_branch point to the same commit?
Answer: You can compare the output of git-rev-parse, such as in this bit of shell script:
export BRANCH_A=feature_branch
export BRANCH_B=origin/feature_branch

if [ x"$(git rev-parse $BRANCH_A)" = x"$(git rev-parse $BRANCH_B)" ]
then
    echo $BRANCH_A and $BRANCH_B are the same
fi

If you're not interested in using this in a script, but just want some indication of where feature_branch and origin/feature_branch are relative to each other, then the top of the output of git status will tell you their relative positions if they are not the same, e.g.:
$ git status
# On branch foo
# Your branch is behind 'origin/foo' by 187 commits, and can be fast-forwarded.

However, note that this only works if your git config indicates that origin/feature_branch is "upstream" for feature_branch.  If you've created the local branch from a pre-existing remote-tracking branch, this will typically be the case.  If, instead, feature_branch was a new branch you created locally, you can set up that association in your config by pushing the branch with -u, e.g. git push -u origin feature_branch.)
Comment: Why you use `x` prefix?
Comment: It's an old habit, from trying to write autoconf tests that are as portable as possible. Some old shells (or implementations of `[` I guess?) could fail if the interpolated command result was the empty string or they could be interpreted as an option). For example, here's a question about that: https://stackoverflow.com/q/11265847
Answer: git rev-parse origin/foo_branch will not get you the latest commit hash on the remote branch. It will give you the latest commit hash that your local git repository knows about on the remote branch.
If you want to compare whether your local foo_branch is up to date with the current remote, you likely want:
# remote commit hash.
# will do a network request to get latest.
git ls-remote --head --exit-code origin foo_branch | cut -f 1

# local commit hash.
git rev-parse foo_branch

Alternatively, if you want to use git rev-parse, you should fetch the latest changes first before running rev-parse:
git fetch origin foo_branch
git rev-parse origin/foo_branch

I like the git ls-remote approach because it leaves your local repository untouched.
Answer: You can use git rev-list. Simply call git rev-list feature_branch...origin/feature_branch (symmetric difference). If there is no output, both commits are the same.
The cleanest way is obviously by the way of git rev-parse and comparing the results, as pointed out by Mark Longair
Comment: I don't think that's quite right: `a..b` gives you every commit in `b` which isn't also in `a`.  So, if `a` is ahead of `b`, they won't be at the same point, but the output will still be empty.
Comment: That's interesting, but it works only if `feature_branch` is **behind** `origin/feature_branch`.
Comment: @MarkLongair: oh… you are totally right! I've edited my answer to use symmetric difference. It should then work for all cases.
Answer: You can use simple command like this:
git diff feature_branch origin/feature_branch

which will shows something only if branches will points to something else.

git log -1 --decorate feature_branch

if it show you something like this
commit c1e77c1....896b (origin/feature_branch, feature_branch)

than both branches points to this same commit.
commit c1e77c1....896b (feature_branch)

means that branches aren't synced.

You also can use
git merge-base feature_branch origin/feature_branch

which will point last common commit. And next
git merge-base feature_branch origin/feature_branch | git show --decorate

As Mark Longair said, git status can be also a solution (sometime the most convenient), however this depends on your working repository, and speed of git status (what can be speed up by git gc used once a while)
Comment: Git diff will also show no difference if the commits are different but point to the same tree, so this might not be what you want
Answer: If you do git log feature_branch it shows you history. Just check if the commit SHA id is the same as on the git log origin/feature_branch history.
Answer: Simple git status should show how many commits your current branch is ahead/behind the remote. Make sure all the remotes are configured properly.


regular expression for numbers with many spaces and special characters?

Question: Hi I am new to the regular expression.
Can someone tell how to format a regular expression pattern for validating Numbers with many spaces and special characters? here the space position are not defined.
I have tried like this ^[0-9]{0,12}$ but I don't know how to place the spaces in-between ?
Ex:'2356 85 568#','5875 #2545','#2525','4567'

Comment: Have you already tried a regex tester? Like this: http://www.regexplanet.com/advanced/dotnet/index.html Examples would be useful.
Comment: Please give a couple of examples for the string to parse and the expected result.
Comment: please add an example of a possible input, and your expected output
Comment: Are the spaces and # include in the count of 12? Then see solutions bellow. Or do you want tu have upto 12 numbers with maybe 345 Whitespaces in between?
Comment: No ,spaces and # shouldn't include in the count of 12 and it should allow the # only once @dognose
Answer: To allow any number of spaces, dashes and #-signs between up to 12 numbers, use this:
^[ #-]*([0-9][ #-]*){0,12}$

initially 0 or more specials
then 0-12 groups of

one digit
followed by 0 or more specials
Comment: Thanks sir,If suppose i wants to limit occurrence of the spaces and dashes and # then how should i modify the above one? @Hans Kesting
Comment: @user1863261 - I don't think it's possible to limit both the digits and the specials, other than like this: `^[ #-]*([0-9][ #-]{0,2}){0,12}$`. You will need another test to check for "max 4 specials in total".
Answer: If you want at most twelve digits, and an unlimited amount of whitespace and hash signs, you can do this:
^[\s#]*([0-9][\s#]*){0,12}$

Answer: To allow "spaces and special characters" to include any non-digit characters, use:
^\D*(\d\D*){0,12}$

To explain:

^\D* matches any non-digit characters at the start of the string
(\d\D*) matches a single digit followed by any non-digits
{0,12} allow the previous group to occur up to 12 times