emmeans pairwise contrasts result in same output values for all?

Question: First of all, I am a beginner at mixed models, so I beg your patience and advice if this post could be improved.
The structure of the data:
> str(data)
'data.frame':   44 obs. of  5 variables:
$ colony   : Factor w/ 3 levels "1","2","3": 1 1 1 1 1 1 1 1 1 1 ...
$ condition: Factor w/ 2 levels "filter","pre": 2 1 2 1 2 1 2 1 2 1 ...
$ location : Factor w/ 3 levels "AN","AM","AS": 1 1 1 1 1 1 2 2 2 2 ...
$ trial    : Factor w/ 4 levels "1","2","3","4": 2 2 3 3 4 4 1 1 2 2 ...
$ ninety   : int  3658 616 797 824 580 405 915 476 1445 620 ...

There were three ant colonies in the study, and the measurements were repeated on each colony, treatment, and location multiple times (but in the end were unbalanced, some colonies had fewer repeat trials than others). The response variable, "ninety", is in elapsed seconds.
The mixed-effects model:
I used the lme4 package in R: > model <- lmer(log(ninety) ~ condition + location + (1 |colony), data = data, REML = FALSE)
Mixed Model output
> summary(finallogninetybuffermodel)
## Linear mixed model fit by maximum likelihood . t-tests use
##   Satterthwaite's method [lmerModLmerTest]
## Formula: log(ninety) ~ condition + location + (1 | colony)
##    Data: buffer
##
##      AIC      BIC   logLik deviance df.resid
##     78.0     88.7    -33.0     66.0       38
##
## Scaled residuals:
##     Min      1Q  Median      3Q     Max
## -1.5606 -0.6441 -0.2184  0.5658  3.3769
##
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  colony   (Intercept) 0.2360   0.4858
##  Residual             0.2174   0.4662
## Number of obs: 44, groups:  colony, 3
##
## Fixed effects:
##                 Estimate Std. Error      df t value Pr(>|t|)
## (Intercept)       6.0443     0.3093  4.0027  19.539 4.03e-05 ***
## conditionfilter  -0.5491     0.1406 41.1101  -3.906 0.000342 ***
## locationAM        0.3927     0.1699 41.4570   2.312 0.025831 *
## locationAS        0.2638     0.1844 41.9729   1.431 0.159922
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Correlation of Fixed Effects:
##             (Intr) cndtnf lctnAM
## conditnfltr -0.227
## locationAM  -0.230  0.000
## locationAS  -0.212  0.000  0.455

Emmeans post-hoc pairwise contrasts:
posthoccondloc <- emmeans(finallogninetybuffermodel, lmer.df =
"satterthwaite", specs = "condition", by = "location")

The locations in the summary (below) show different standard errors and confidence limits, but for the pairwise contrasts by location, all the output values, for each location, are the same, but they look different on the bar-arrow plot. Why might that be? I've read everything I could find about lme4 and emmeans online, but I still can't figure it out.
> summary(posthoccondloc, adjust = "bonferroni")
## location = AN:
##  condition   emmean        SE   df lower.CL upper.CL
##  pre       6.044318 0.3093493 4.00 4.963388 7.125248
##  filter    5.495213 0.3093493 4.00 4.414283 6.576143
##
## location = AM:
##  condition   emmean        SE   df lower.CL upper.CL
##  pre       6.436980 0.3168496 4.38 5.377601 7.496360
##  filter    5.887875 0.3168496 4.38 4.828496 6.947255
##
## location = AS:
##  condition   emmean        SE   df lower.CL upper.CL
##  pre       6.308155 0.3248875 4.81 5.264069 7.352240
##  filter    5.759050 0.3248875 4.81 4.714964 6.803135
##
## Degrees-of-freedom method: satterthwaite
## Results are given on the log (not the response) scale.
## Confidence level used: 0.95
## Conf-level adjustment: bonferroni method for 2 estimates

> contrast(posthoccondloc, pairwise = TRUE, adjust = "bonferroni")
## location = AN:
##  contrast        estimate         SE    df t.ratio p.value
##  pre effect     0.2745525 0.07028776 41.11   3.906  0.0007
##  filter effect -0.2745525 0.07028776 41.11  -3.906  0.0007
##
## location = AM:
##  contrast        estimate         SE    df t.ratio p.value
##  pre effect     0.2745525 0.07028776 41.11   3.906  0.0007
##  filter effect -0.2745525 0.07028776 41.11  -3.906  0.0007
##
## location = AS:
##  contrast        estimate         SE    df t.ratio p.value
##  pre effect     0.2745525 0.07028776 41.11   3.906  0.0007
##  filter effect -0.2745525 0.07028776 41.11  -3.906  0.0007
##
## Results are given on the log (not the response) scale.
## P value adjustment: bonferroni method for 2 tests

emmeans Graphs
 > emmip(posthoccondloc, condition ~ location)

 > plot(posthoccondloc, comparisons = TRUE)

[
Comment: I am not sure what null hypotheses are corresponding to the p values generated by "contrast(posthoccondloc, pairwise = TRUE, adjust = "bonferroni")". Could you figure them out?
Comment: My null hypothesis was that the "filter" and "pre" conditions would be the same at each location along the trail. As rvl points out below, I should have included the interaction term for condition:location, since that was the case. Thanks for that clarifying question.
Answer: You have fitted an additive model - the fixed-effects part is condition + location. Therefore you have in fact specified that the differences for one factor are exactly the same at each level of the other factor. Since emmeans() summarizes a model, then, lo and behold, the results reflect what is specified.
If instead you include the interaction between condition and location in the model, then the emmeans() results will reflect the possibility that factor levels compare differently at levels of the other factor.
I recommend that people think more carefully about the models they are fitting. I think there is a tendency to rush forward without realizing what s crucial thing it is to get the model right.
Comment: rvl - thanks for pointing out my mistake. Your comment has helped me create the model I meant to create, and those results make a lot of sense, given the bad model input I had. I will keep working on examples to work on my model building skills, but thanks for helping me move forward with this one.
Comment: @Russ Lenth , but what if the best model is indeed an additive model? That's my case. I have a very clear lineplot for data in year 1 and Year 2 for two groups which shows almost perfect paralell lines. In addition to it, the anova() output comparing the models also indicates, as suggested, that adding the interaction term does not improve the model. I'm also getting the same results for all contrasts, because it's additive. Is there a way to turn things around in this scenario?
Comment: @LarissaCury Sure, it's fine if the best model is additive. That is not a barrier, in fact, it simplifies things. In particular, you don't need to do "by" comparisons, you can just compare the overall EMMs of `condition` and `location`.


Implementar uma API para regras de negócio de um site web: é um exagero? Ou é uma boa prática?

Question: Em alguns lugares que trabalhei com C# MVC vi duas formas de se programar o modelo de negócio de um sistema web, sendo elas: codificando dentro do Controller da aplicação web; criando uma web API rest e criando a regra de negócio na API, apenas chamando essa API dentro do Controller da aplicação web.
Só para ficar claro, um exemplo:
Uma aplicação que faz um cadastro de um cliente, no MVC existirá uma View chamada 'Cadastrar.cshtml' com código html e razor criando um formulário de cadastro. Ao clicar em cadastrar, os dados do formulário serão enviados ao Controller Cadastrar.cs, e é aí que entram as diferenças.
Eu posso simplesmente já cadastrar o cliente ali mesmo no Controller ou eu posso criar uma Web API rest que faça essa ação, então no Controller eu só chamaria essa API. Desta maneira, meu aplicativo web teria basicamente as views, mas sem acesso a banco de dados ou regras de negócio, e essas regras ficariam todas na API.
A questão é: é melhor fazer assim? Porquê?
Penso que possa ser um exagero e que até prejudique a performance, mas já ouvi dizer que é uma boa prática, justamente porque a regra de negócio fica na API.
Comment: Primeiro, a regra de negócio não deve estar na `Controller`. Segundo, a questão da WebAPI seria muito útil se você quer disponibilizar as funcionalidades para outros clientes que não estejam dentro da sua solution  ou que não sejam .net. Aí vem a vantagem da regra de negócio não estar implemantada na `Controller` e sim em uma camada isolada, pois assim você pode expor e acessá-la conforme for a sua necessidade
Answer: O correto seria ter uma segunda camada de "Serviços" que terá suas regras de negócios (nunca direto na controller).
O uso da API faz sentido para projetos médios/grandes (facilita a manutenção), e para reutilização de código (para outros sistemas).


how to export Hbase data to elastic search using nutch?

Question: i have followed https://gist.github.com/xrstf/b48a970098a8e76943b9  to integrate nutch and elastic-search. everything is working fine data is stored in Hbase 'webpage' table but i am not able to fetch data in elastic search.i want to know how to fetch data in elastic search.
Answer: below is my code
<code>package com.process;
/*
  import package will be here
*/

public class HbaseToElastic extends Configured implements
    org.apache.hadoop.util.Tool {

static class Mapper extends TableMapper<Text, IndexWritable> {

    public static String CLUSTER;
    public static String SEARCH_HOST;
    public static String SEARCH_PORT;
    public static String SEARCH_INDEX_NAME;
    public static String SEARCHtYPE;
    public static int BULKSIZE;
    public static String TABLENAME;
    public static String FAMILY;
    private static List<String> SPORTS_KEYWORDS;
    private static List<String> BUSINESS_KEYWORDS;
    private static List<String> GOSSIP_KEYWORDS;
    private static List<String> CRIME_KEYWORDS;
    private static Map<String, Map<String, String>> STATE_MAP = new HashMap<String, Map<String, String>>();
    private static Map<String, String> CITY_MAP = new HashMap<String, String>();

    private static Mapper mapper = new Mapper();

    static {
        try {
            System.out.println("done1");
            DetectorFactory.loadProfile("./profiles");
            System.out.println("done2");
        } catch (final LangDetectException e) {
            System.out.println("done3");
            e.printStackTrace();
        }

    }

    Configuration hbaseConf = null;
    HTable table = null;

    List<Put> hbasePutErrorList = new ArrayList<Put>();

    /**
     * Clean up the hbase table object
     */
    @Override
    protected void cleanup(final Context context) throws IOException,
            InterruptedException {
        super.cleanup(context);

        table.put(hbasePutErrorList);
        table.close();
        hbasePutErrorList.clear();
    }

    /**
     * Initialize various variables
     */
    @Override
    protected void setup(
            final org.apache.hadoop.mapreduce.Mapper<ImmutableBytesWritable, Result, Text, IndexWritable>.Context context)
            throws IOException, InterruptedException {
        final Configuration conf = context.getConfiguration();

        CLUSTER = conf.get("cluster");
        SEARCH_HOST = conf.get("search_host");
        SEARCH_PORT = conf.get("search_port");
        SEARCH_INDEX_NAME = conf.get("search_index_name");
        SEARCHtYPE = conf.get("search_type");
        BULKSIZE = conf.getInt("search_bulk_size", 500);
        TABLENAME = conf.get("table_name");
        FAMILY = conf.get("family");

        hbaseConf = HBaseConfiguration.create();
        hbaseConf.set("hbase.zookeeper.quorum",
                conf.get("hbase.zookeeper.quorum"));
        hbaseConf.set("hbase.zookeeper.property.clientPort",
                conf.get("hbase.zookeeper.property.clientPort"));
        hbaseConf.set("hbase.rpc.timeout", conf.get("hbase.rpc.timeout"));
        hbaseConf.set("hbase.regionserver.lease.period",
                conf.get("hbase.regionserver.lease.period"));
        hbaseConf.set("hbase.master", conf.get("hbase.master"));
        table = new HTable(hbaseConf, conf.get("table_name"));

        SPORTS_KEYWORDS = new ArrayList<String>();
        BUSINESS_KEYWORDS = new ArrayList<String>();
        GOSSIP_KEYWORDS = new ArrayList<String>();
        CRIME_KEYWORDS = new ArrayList<String>();
        String keywrods = conf.get("sportskeywords");
        String[] keyarr = keywrods.split(",");
        for (final String key : keyarr) {
            SPORTS_KEYWORDS.add(key.trim());
        }
        keywrods = conf.get("businesskeywords");
        keyarr = keywrods.split(",");
        for (final String key : keyarr) {
            BUSINESS_KEYWORDS.add(key.trim());
        }
        keywrods = conf.get("gossipkeywords");
        keyarr = keywrods.split(",");
        for (final String key : keyarr) {
            GOSSIP_KEYWORDS.add(key.trim());
        }
        keywrods = conf.get("crimekeywords");
        keyarr = keywrods.split(",");
        for (final String key : keyarr) {
            CRIME_KEYWORDS.add(key.trim());
        }

        final String stateMap = conf.get("statemap");
        final Gson g = new Gson();

        STATE_MAP = g.fromJson(stateMap, Map.class);

    }

    /**
     * map function
     */
    @Override
    public void map(final ImmutableBytesWritable row, final Result result,
            final Context context) throws IOException, InterruptedException {
        try {
            final byte b = 0;
            int deleteFlag = 0;
            final String keyString = Bytes.toString(row.get());
            final Map<String, Object> mapobject = new HashMap<String, Object>();
            for (final KeyValue kv : result.raw()) {
                final String key = (new String(kv.getQualifier()));
                final String value = (new String(kv.getValue()));
                mapobject.put(key, value);
            }
            final Gson g = new Gson();
            if (checkValidType(mapobject)) {
                refineMetaTags(mapobject);
                if (refineDescription(mapobject)) {
                    assignCity(mapobject);
                    if (checkTitleImage(mapobject)) {
                        if (setLang(mapobject)) {
                            setCorrectCategory(mapobject);
                            correctDuplicateTitle(mapobject);
                            final String json = g.toJson(mapobject);
                            context.write(new Text(keyString),
                                    new IndexWritable(json, b));
                            deleteFlag = 1;
                        }
                    }
                }
            }
            if (deleteFlag == 0) {
                final Put put = new Put(Bytes.toBytes(keyString));
                put.add(Bytes.toBytes("cf"), Bytes.toBytes("ErrorFlag"),
                        Bytes.toBytes("1"));
                hbasePutErrorList.add(put);
            }
        } catch (final Exception e) {
            e.printStackTrace();
        }
    }

    /**
     * Remove duplicate statement in the title
     *
     * @param mapobject
     */
    private void correctDuplicateTitle(final Map<String, Object> mapobject) {
        final String duplicateTitle = mapobject.get("title").toString();
        final String stripedTitleArr[] = duplicateTitle.split(" ", 4);
        if (stripedTitleArr.length == 4) {
            final String subString = stripedTitleArr[0] + " "
                    + stripedTitleArr[1] + " " + stripedTitleArr[2];
            if (stripedTitleArr[3].contains(subString)) {

                mapobject.put("title", duplicateTitle
                        .substring(duplicateTitle.indexOf(subString,
                                subString.length() - 1)));

                mapobject.put("title", stripedTitleArr[3]
                        .substring(stripedTitleArr[3].indexOf(subString)));
            }
        }
    }

    /**
     * Set category based on the various category specific keyword
     *
     * @param mapobject
     */
    private void setCorrectCategory(final Map<String, Object> mapobject) {
        final String url = mapobject.get("url") + "";
        final String cat = mapobject.get("tags") + "";
        if ("sports".equalsIgnoreCase(cat)
                || "cricket".equalsIgnoreCase(cat)) {
            if (!(url.toLowerCase().contains("sport")
                    || url.toLowerCase().contains("खेल")
                    || url.toLowerCase().contains("cric") || url
                    .toLowerCase().contains("क्रिकेट"))) {
                final String desc = mapobject.get("description").toString();
                boolean isSports = false;
                int count = 0;
                for (final String keyword : SPORTS_KEYWORDS) {
                    if (desc.contains(keyword)) {
                        count++;
                    }
                }
                if (count > 1) {
                    isSports = true;
                }
                if (!isSports) {
                    mapobject.put("tags", "national");
                }
                if (isSports

                        && (desc.contains("क्रिकेट")
                                || url.toLowerCase().contains("cric")
                                || desc.contains("टॉस")
                                || desc.contains("वनडे") || desc
                                    .contains("बल्लेबाज"))) {
                    mapobject.put("tags", "cricket");
                }
            }
        } else if ("business".equalsIgnoreCase(cat)) {
            if ((url.toLowerCase().contains("sport") || url.toLowerCase()
                    .contains("खेल"))) {
                mapobject.put("tags", "sports");
            } else if (url.toLowerCase().contains("cric")
                    || url.toLowerCase().contains("क्रिकेट")) {
                mapobject.put("tags", "cricket");
            } else if (!(url.toLowerCase().contains("busines")
                    || url.toLowerCase().contains("व्यापार")
                    || url.toLowerCase().contains("economy")
                    || url.toLowerCase().contains("finance")
                    || url.toLowerCase().contains("बिजनेस")
                    || url.toLowerCase().contains("market")
                    || url.toLowerCase().contains("karobar") || url
                        .contains("कारोबार"))) {
                final String desc = mapobject.get("description").toString();
                int count = 0;
                for (final String keyword : BUSINESS_KEYWORDS) {
                    if (desc.contains(keyword)) {
                        count++;
                    }
                }
                if (count < 2) {
                    mapobject.put("tags", "national");
                }
            }
        } else if ("gossip".equalsIgnoreCase(cat)) {
            if ((url.toLowerCase().contains("sport") || url.toLowerCase()
                    .contains("खेल"))) {
                mapobject.put("tags", "sports");
            } else if (url.toLowerCase().contains("cric")
                    || url.toLowerCase().contains("क्रिकेट")) {
                mapobject.put("tags", "cricket");
            } else if (url.toLowerCase().contains("busines")) {
                mapobject.put("tags", "business");
            } else if (!(url.toLowerCase().contains("masala")
                    || url.toLowerCase().contains("gossip")
                    || url.toLowerCase().contains("gupshup") || url
                    .toLowerCase().contains("garam"))) {
                final String desc = mapobject.get("description").toString();
                int count = 0;
                for (final String keyword : GOSSIP_KEYWORDS) {
                    if (desc.contains(keyword)) {
                        count++;
                    }
                }
                if (count < 2) {
                    mapobject.put("tags", "national");
                }
            }
        } else if ("crime".equalsIgnoreCase(cat)) {
            if ((url.toLowerCase().contains("sport") || url.toLowerCase()
                    .contains("खेल"))) {
                mapobject.put("tags", "sports");
            } else if (url.toLowerCase().contains("cric")
                    || url.toLowerCase().contains("क्रिकेट")) {
                mapobject.put("tags", "cricket");
            } else if (url.toLowerCase().contains("busines")) {
                mapobject.put("tags", "business");
            } else if (!(url.toLowerCase().contains("crime")
                    || url.toLowerCase().contains("terrorist")
                    || url.toLowerCase().contains("abuse")
                    || url.toLowerCase().contains("forgery")
                    || url.toLowerCase().contains("assault")
                    || url.toLowerCase().contains("violence")
                    || url.toLowerCase().contains("rape")
                    || url.toLowerCase().contains("teasing")
                    || url.toLowerCase().contains("molestation")
                    || url.toLowerCase().contains("scandal") || url
                    .toLowerCase().contains("murder"))) {
                final String desc = mapobject.get("description").toString();
                int count = 0;
                for (final String keyword : CRIME_KEYWORDS) {
                    if (desc.contains(keyword)) {
                        count++;
                    }
                }
                if (count < 2) {
                    mapobject.put("tags", "national");
                }
            }
        } else if (cat != null && cat.startsWith("local")) {

        }
    }

    /**
     * Check valid type of the HTML pages
     *
     * @param mapobject
     * @return
     */
    private boolean checkValidType(final Map<String, Object> mapobject) {
        if (mapobject.containsKey("type")
                && !(mapobject.get("type").toString().contains("image") || mapobject
                        .get("type").toString().contains("rss"))) {
            return true;
        }
        return false;
    }

    /**
     * refine the description according to its length and must starting with
     * english and it the description is not present get the description
     * from the metatags description
     *
     * @param mapobject
     * @return {@link Boolean}
     */
    private boolean refineDescription(final Map<String, Object> mapobject) {

        if (mapobject.containsKey("description")
                && mapobject.get("description").toString().length() > 75
                && !mapobject.get("description").toString().contains(";}")
                && !mapobject.get("description").toString()
                        .contains("<cite>")
                && !mapobject.get("description").toString()
                        .contains("href=")
                && !mapobject.get("description").toString()
                        .contains("All rights reserved")) {
            return true;
        } else if (mapobject.containsKey("metatag.description")
                && mapobject.get("metatag.description").toString().length() > 75
                && !mapobject.get("metatag.description").toString()
                        .contains(";}")
                && !mapobject.get("metatag.description").toString()
                        .contains("<cite>")) {
            mapobject.put("description",
                    mapobject.get("metatag.description"));
            return true;
        }
        return false;
    }

    /**
     * refine metatags by refining meta keyword to only include the English
     * keyword only that has at most three keyword and if not present then
     * create the keyword with title field of the html and if none of the
     * keyword found then form it using the help of the url and exclude the
     * number from the keywords
     *
     * @param mapobject
     */
    private void refineMetaTags(final Map<String, Object> mapobject) {
        String metaTag = "";
        int tagFlag = 0;
        if (mapobject.containsKey("metatag.keywords")) {

            final String metaTags[] = mapobject.get("metatag.keywords")
                    .toString().replaceAll("\\|", ",").split(",");
            String domain = null;
            StringBuilder temp = null;
            for (final String metaTag2 : metaTags) {
                if (mapobject.containsKey("host")) {
                    domain = mapobject.get("host") + "";
                    if (domain.split("\\.").length > 1
                            && (metaTag2
                                    .contains(domain.split("\\.")[domain
                                            .split("\\.").length - 2]) || metaTag2
                                    .contains(domain.split("\\.")[0])))

                    {
                        continue;
                    }
                }
                String[] arr = metaTag2.split(" ");
                arr = removeUnicodeWords(arr);
                if (arr.length > 0 && arr.length < 5) {
                    temp = new StringBuilder();
                    for (final String str : arr) {

                        temp.append(str);
                        temp.append(" ");
                    }
                    if (metaTag.length() + temp.length() < 70) {
                        metaTag = metaTag + "," + temp.toString();
                    }
                }

            }
            if (metaTag.startsWith(",")) {
                metaTag = metaTag.trim();
                metaTag = metaTag.substring(1, metaTag.length());
            }
        }

        if (metaTag.length() < 1 && mapobject.containsKey("title")) {
            /**
             * Extracting tags from the title tag if the length of the
             * keyword is greater than 4
             */
            final String title = (String) mapobject.get("title");
            final String splitTitle[] = title.split(" ");
            int count = 0;
            for (int i = 0; i < splitTitle.length; i++) {
                if (splitTitle[i].length() > 4
                        && !splitTitle[i].matches("^[\\u0900-\\u097F].*")) {
                    metaTag = metaTag + splitTitle[i] + ",";
                    count++;
                    if (count == 5) {
                        break;
                    }
                }
            }
            if (metaTag.split(",").length > 3) {
                if (metaTag.endsWith(",")) {
                    metaTag = metaTag.trim();
                    metaTag = metaTag.substring(0, metaTag.length() - 1);
                }
            } else {
                metaTag = "";
            }
        }
        if (metaTag.length() < 1) {
            /**
             * Extracting the tags from the url if the length of the keyword
             * is greater than 4
             */
            final String splitUrl[] = mapobject.get("url").toString()
                    .split("/");
            final String lastSplitValue = splitUrl[splitUrl.length - 1];

            final String tagList[] = generateTokens(lastSplitValue);
            if (tagList != null) {
                int count = 0;
                for (int i = 0; i < tagList.length; i++) {
                    if (tagList[i].length() > 4
                            && !tagList[i].matches("^[\\u0900-\\u097F].*")) {
                        metaTag = metaTag + tagList[i] + ",";
                        count++;
                        if (count == 5) {
                            break;
                        }
                    }
                }
            }
            if (metaTag.endsWith(",")) {
                metaTag = metaTag.trim();
                metaTag = metaTag.substring(0, metaTag.length() - 1);
            }
        }
        if (metaTag.length() > 0) {
            metaTag = metaTag.replaceAll("\\[", "");
            metaTag = metaTag.replaceAll("\"", "");
            metaTag = metaTag.replaceAll(";", "");
            metaTag = metaTag.replaceAll(":", "");
            metaTag = metaTag.replaceAll("\u0027", "");
            metaTag = metaTag.replaceAll("\u003d", "");
            metaTag = metaTag.replaceAll("\u0026", "");
            tagFlag = 1;
        }
        mapobject.put("TagFlag", tagFlag);
        mapobject.put("metatag.keywords", metaTag);
    }

    /**
     * Remove unicode character
     *
     * @param arr
     * @return
     */
    private String[] removeUnicodeWords(final String[] arr) {
        final List<String> returnArr = new ArrayList<String>();
        for (final String str : arr) {
            if (str != null && str.trim().length() > 3
                    && !str.matches("^[\\u0900-\\u097F].*")
                    && !(str.matches("^[0-9].*"))) {
                returnArr.add(str.trim());
            }
        }
        final String[] retrnArr = new String[returnArr.size()];
        returnArr.toArray(retrnArr);
        return retrnArr;
    }

    /**
     * Generate Token list with the help of the lucene analyzer
     *
     * @param lastSplitValue
     * @return {@link ArrayIndexOutOfBoundsException} of the list of the
     *         keywords
     */
    private String[] generateTokens(String lastSplitValue) {
        final List<String> list = new ArrayList<String>();
        lastSplitValue = lastSplitValue.replace("\\.", " ").replace("%20",
                " ");
        try {
            final Version matchVersion = Version.LUCENE_45;
            final Analyzer analyzer = new HindiAnalyzer(matchVersion);
            final TokenStream ts = analyzer.tokenStream("field",
                    new StringReader(lastSplitValue));
            ts.reset();
            while (ts.incrementToken()) {
                final CharTermAttribute cta = ts
                        .getAttribute(CharTermAttribute.class);
                if (cta.toString().length() > 4
                        && !cta.toString().matches("^[0-9].*")) {
                    list.add(cta.toString());
                }
            }
            ts.end();
            ts.close();
            analyzer.close();
        } catch (final Exception e) {
            e.printStackTrace();
        }
        if (list.size() > 3) {
            return list.toArray(new String[list.size()]);
        } else {
            return null;
        }
    }

    /**
     * Checks title and assign their language based on their first character
     * of the title
     *
     * @param mapobject
     * @return {@link Map}
     */
    private boolean setLang(final Map<String, Object> mapobject) {
        final String title = mapobject.get("title").toString();
        final String description = mapobject.get("title").toString();
        String language = "";
        try {
            language = mapper.detect(title);
            mapper.detect(description);
        } catch (final LangDetectException e) {
            System.out.println("\n title with error is - " + title);
            System.out.println("\n description with error is - "
                    + description);
            e.printStackTrace();
            /*
             * String title = mapobject.get("title").toString(); language =
             * mapobject.get("lang") + ""; language = language.trim(); if
             * (language.trim().equalsIgnoreCase("hi") ||
             * language.trim().startsWith("en") ||
             * language.trim().equalsIgnoreCase("lt")) { String[] titleArr =
             * title.trim().split(" "); int i = 0; for (String titlePart :
             * titleArr) { if
             * (titlePart.trim().matches("^[\\u0900-\\u097F].*")) { i++; } }
             * if (i >= titleArr.length * 0.5) { mapobject.put("lang",
             * "hi"); } else { mapobject.put("lang", "lt"); } return true; }
             */
            return false;
        }

        if (language.trim().equalsIgnoreCase("hi")
                || language.trim().startsWith("en")
                || language.trim().equalsIgnoreCase("lt")) {
            mapobject.put("lang", language);
            return true;
        }

        return false;
    }

    private String detect(final String text) throws LangDetectException {
        final Detector detector = DetectorFactory.create();
        detector.append(text);
        return detector.detect();
    }

    /**
     * Checks whether to include the doc based on their title and get the
     * title from anchor tag title to choose the title that has largest
     * number of the words and in hindi and it also gets the image from
     * anchor tag href attribute
     *
     * @param mapobject
     *            of the key value pair
     * @return {@link Boolean}
     */
    private boolean checkTitleImage(final Map<String, Object> mapobject) {
        final TreeSet<String> set = new TreeSet<String>(new SetSort());

        final Gson gson = new Gson();
        JsonArray array = null;
        JsonObject object2 = null;

        if (mapobject.containsKey("anchor")
                && mapobject.get("anchor") != null) {
            final String arr = (String) mapobject.get("anchor");

            try {

                array = gson.fromJson(arr, JsonArray.class);

                for (final JsonElement jsonElement : array) {

                    try {
                        object2 = gson.fromJson(jsonElement.getAsString(),
                                JsonObject.class);
                    } catch (final Exception e) {

                        if (object2 == null) {
                            object2 = new JsonObject();
                            object2.addProperty("title",
                                    jsonElement.getAsString());
                            object2.addProperty("href", "");
                            object2.addProperty("alt", "");
                        }
                    }
                    if (object2 != null) {
                        assignTitleImage(mapobject, set, object2);
                    }
                    object2 = null;
                }
            } catch (final ClassCastException e) {

                object2 = gson.fromJson(arr, JsonObject.class);
                assignTitleImage(mapobject, set, object2);

            } catch (final Exception e) {
                e.printStackTrace();
            }

            if (!set.isEmpty()) {
                int loop = 0;
                final List<String> tempList = new LinkedList<String>();
                for (final String string : set) {
                    final String title = string;
                    tempList.add(title.trim());
                    loop++;
                    if (loop == 2) {
                        break;
                    }
                }
                if (!tempList.isEmpty()) {
                    if (tempList.get(0).matches("^[\\u0900-\\u097F].*")) {
                        mapobject.put("title", tempList.get(0));
                    } else if (tempList.size() > 1
                            && !(tempList.get(0)
                                    .matches("^[\\u0900-\\u097F].*"))
                            && tempList.get(1).matches(
                                    "^[\\u0900-\\u097F].*")) {
                        mapobject.put("title", tempList.get(1));
                    } else {
                        mapobject.put("title", tempList.get(0));
                    }
                }
            }

        }
        if (mapobject.containsKey("title")
                && mapobject.get("title").toString().length() > 0
                && mapobject.get("title").toString().split(" ").length > 2
                && mapobject.get("title").toString().split(" ").length < 20
                && !mapobject.get("title").toString().contains("<")) {
            if (set.isEmpty()) {
                mapobject.put("title",
                        getTitleRefined(mapobject.get("title") + ""));
            }
            return true;
        }
        return false;
    }

    /**
     * @param mapobject
     * @param set
     * @param object2
     */
    private void assignTitleImage(final Map<String, Object> mapobject,
            final TreeSet<String> set, final JsonObject object2) {
        if (!mapobject.containsKey("ImgH1")
                && !mapobject.containsKey("ImgH2")) {
            if (object2.get("href") != null
                    && object2.get("href").getAsString().length() > 0
                    && (object2.get("href").getAsString().toLowerCase()
                            .contains(".jpg")
                            || object2.get("href").getAsString()
                                    .toLowerCase().contains(".jpeg") || object2
                            .get("href").getAsString().toLowerCase()
                            .contains(".gif"))) {
                putImages(mapobject, object2.get("href").getAsString()
                        .trim(), mapobject.get("tags").toString().trim()
                        .toLowerCase());
            }
        }

        if (object2.get("title") != null
                && object2.get("title").getAsString().length() > 0
                && object2.get("title").getAsString().split(" ").length > 2
                && object2.get("title").getAsString().split(" ").length < 20
                && !object2.get("title").getAsString().contains("<")) {

            final String newTitle = getTitleRefined(object2.get("title")
                    .getAsString());
            set.add(newTitle.trim());
        }
    }

    /**
     * This function used to refine the title based on specific bad keyword
     * during observation
     *
     * @param title
     * @return refined title
     */
    private String getTitleRefined(String title) {
        title = title.replaceAll("\u0027", "");
        title = title.replaceAll("\u0026", "");
        title = title.replaceAll("\u003d", "");
        if (title.contains("-")) {
            if (title.trim().split("-").length > 1
                    && !title.trim().split("-")[1].trim().matches(
                            "^[\\u0900-\\u097F].*")) {
                return title.trim().split("-")[0].trim();
            }
        } else if (title.contains(":")) {
            if (!title.trim().split(":")[0].trim().matches(
                    "^[\\u0900-\\u097F].*")
                    && title.trim().split(":").length > 1) {
                return title.trim().split(":")[1].trim();
            }
        }
        return title;
    }

    /**
     * Creates the path for the images
     *
     * @param map
     *            of the key value pair
     * @param imageUrl
     * @param category
     */
    private void putImages(final Map<String, Object> map2,
            final String imageUrl, final String category) {
        try {
            map2.put("ImgSrc", StringEscapeUtils.unescapeHtml(imageUrl)
                    .trim());
            if (map2.containsKey("ImgSrc") && map2.get("ImgSrc") != null
                    && map2.get("ImgSrc").toString().length() > 0) {
                map2.put(
                        "ImgSrc",
                        StringEscapeUtils.unescapeHtml(map2.get("ImgSrc")
                                .toString())
                                + "##RAFTAAR##"
                                + imageUrl.trim());
            } else {
                return;
            }
            String imgNamearr[] = null;
            try {
                imgNamearr = imageUrl.split("/");
            } catch (final Exception e) {
                e.printStackTrace();
            }
            String imgName = null;
            try {
                imgName = imgNamearr[imgNamearr.length - 1];
            } catch (final Exception e) {
                e.printStackTrace();
            }
            final String imagePath = "/"
                    + String.valueOf(imgName.charAt(0));
            imgName = imgName.replaceAll(" ", "_").replaceAll("%20", "_");
            if (imgName.split(".jpg").length > 0) {
                imgName = imgName.split(".jpg")[0];
                imgName = imgName + ".jpg";
            }

            map2.put("ImgH1", "h1/" + category + imagePath + "/" + imgName);
            map2.put("ImgH2", "h2/" + category + imagePath + "/" + imgName);
        } catch (final Exception e) {
            e.printStackTrace();
        }
    }

    /**
     * Inserts the data to the elasticsearch
     *
     * @param mapobject
     * @param key
     *            unique id generally it is the unique url
     */
    public static void insertToElastic(final Map<String, Object> mapobject,
            final String key) {

        final Settings settings = ImmutableSettings.settingsBuilder()
                .put("cluster.name", CLUSTER).build();/*
                                                     * change ccluster.name
                                                     * to cluster
                                                     */
        final Client client = new TransportClient(settings)
                .addTransportAddress(new InetSocketTransportAddress(
                        SEARCH_HOST, Integer.parseInt(SEARCH_PORT)));
        client.prepareIndex(SEARCH_INDEX_NAME, SEARCHtYPE, key)
                .setSource(mapobject).execute().actionGet();
        client.close();
    }

    /**
     * Assign the city to the news without city
     *
     * @param category
     * @param description
     * @return update category with city
     */
    private static void assignCity(final Map<String, Object> mapobject) {

        String category = mapobject.get("tags").toString();
        if (category.endsWith("/")) {
            boolean flag = true;
            final String catArr[] = category.split("/");
            if (catArr.length == 2) {
                final String state = catArr[1];
                CITY_MAP = STATE_MAP.get(state);
                for (final Entry<String, String> e : CITY_MAP.entrySet()) {
                    final String description = mapobject.get("description")
                            .toString();
                    if (description.contains(e.getValue())) {
                        category = category + e.getKey();
                        mapobject.put("tags", category);
                        flag = false;
                        break;
                    }
                }
            }
            if (flag) {
                mapobject.put("tags", "national");
            }
        }
    }
}

/**
 * Update the data to hbase
 *
 * @param tableName
 * @param rowKey
 * @param family
 * @param qualifier
 * @param value
 * @param conf
 */
public static void updateIntoHbase(final String tableName,
        final String rowKey, final String family, final String qualifier,
        final String value, final Configuration conf) {
    HTable table = null;
    try {
        table = new HTable(conf, tableName);
    } catch (final IOException e) {
        e.printStackTrace();
    }
    final Put put = new Put(Bytes.toBytes(rowKey));
    put.add(Bytes.toBytes(family), Bytes.toBytes(qualifier),
            Bytes.toBytes(value));
    try {
        table.put(put);
        table.close();
    } catch (final IOException e) {
        e.printStackTrace();
    }
}

/**
 * Return the map of the all states and city
 *
 * @param fileName
 * @return
 */
private static Map<String, Map<String, String>> returnMap(
        final String fileName) {
    final Map<String, Map<String, String>> map = new HashMap<String, Map<String, String>>();
    BufferedReader br = null;
    try {
        br = new BufferedReader(new FileReader(fileName));
        String line;
        while ((line = br.readLine()) != null) {

            final String arr[] = line.split("\t", 3);
            if (arr.length == 3) {
                if (map.containsKey(arr[0])) {
                    Map<String, String> m = new HashMap<String, String>();
                    m = map.get(arr[0]);
                    m.put(arr[1], arr[2]);
                } else {
                    final Map<String, String> m = new HashMap<String, String>();
                    m.put(arr[1], arr[2]);
                    map.put(arr[0], m);
                }
            }
        }
    } catch (final FileNotFoundException e) {
        e.printStackTrace();
    } catch (final IOException e) {
        e.printStackTrace();
    } finally {
        if (br != null) {
            try {
                br.close();
            } catch (final Exception e) {
                e.printStackTrace();
            }
        }
    }
    return map;
}

public static void main(final String[] args) throws Exception {

    int c = 0;
    c = ToolRunner.run(new Configuration(), new HbaseToElastic(), args);
    System.exit(c);
}
}
</code>
Comment: by integrate nutch and elastic-search. everything is working fine data is stored in Hbase 'webpage' table but i am not able to fetch data in elastic search.i want to know how to fetch data in elastic search.


Dummies tutorial for Websocket in tomcat8 server and client

Question: My first post.
I have experience of basic java servlet and jsp and have got my webpage implemented in tomcat7.
Regarding Websocket, I am finding difficult to build understanding of how to implement it, i want to use tomcat8 implementation of websocket api and uplift my webpage (jsp, java, jquery, tomcat7) to use the websocket features, have not been able to find the the tutorial that can guide me through, something like hello world example. any pointers (sample codes, tutorials)?
have tried to understand tomcat8 examples but not understanding them at all
Answer: You need to understand that you don't just "add" WebSocket to an existing web application.. to make it faster, better, cheaper, more scalable, etc. Instead, you have to completely re-architect the web-layer of the application to take advantage of its capabilities.
I suggest that you first read a lot about WebSocket and what the whole idea is before you try to write any code, using Tomcat or any other WebSocket-capable server.
Nick Williams has a forthcoming book that appears to cover everything in the web application world, and as I understand it, will have a great deal of information about WebSocket-based code. You will unfortunately have to wait until March 2013 (at least) to use that particular book.
I'm sure there are similar books available, or even online tutorials to help get you started using WebSocket. Just be aware that switching to WebSocket isn't some simple configuration option: it's a very disruptive change to any existing web application.


Take value from SQL server and delete relevant folders

Question: Im new to powershell and would like to
-delete all rows in a sql server DB that have a date older than 10 years
-for every row that is deleted also delete a folder or a hard disk
So for example if I run the query
DELETE FROM [RMS].[dbo].[requests] where date_logged < DATEADD(year, -10, GetDate())

I then thought I could get the lowest request_id and just delete any folders under that number.
So for example if I delete 10 rows with my delete query and then do a select
It would say that the lowest request_id is 11.
I've started below but I'm not sure how to capture that the oldest request_id is?
The SQL would be this ...
SELECT TOP 1 request_id FROM [RMS].[dbo].[requests] order by request_id asc

And also how I would delete any folder "less" than that value.
So if request_id = 11 then I'd need to delete
C:\temp\1
C:\temp\2
C:\temp\3
...
C:\temp\10

Thanks
P
$connectionString = "Data Source=server;Initial Catalog=RMS;Integrated Security=SSPI";
$connection = New-Object System.Data.SqlClient.SqlConnection($connectionString);
$commandR = New-Object System.Data.SqlClient.SqlCommand("DELETE FROM dbo.requests WHERE request_id= 1", $connection);
$commandCount = New-Object System.Data.SqlClient.SqlCommand("select count(*) from requests", $connection);
$connection.Open();
$rowsDeletedR = $commandR.ExecuteNonQuery();
Write-Host "$rowsDeletedR rows deleted";
$rowsCountR = $commandCount.ExecuteScalar();
Write-Host "$rowsCountR rows in requests table";
$connection.Close();

Comment: Deleting rows from an SQL table and deleting files from a folder are two different operations. One requires an SMO object for connection while the other uses the physical I/O system.
I would say you should look for connections between them. What I mean to say is if the name of the folder in the I/O path is same as that in one of the columns (eg Name column) hen you could use this criteria to do both the operations simultaneously.
Comment: If the `request_id` is a field in the record containing the directory name, I would SELECT it into a file, then delete dir directory based on the name in the file. After that I would DELETE ... WHERE... Depending on your database, you might be able to wrap these in an all-or-nothing transaction.
Answer: Your task is broad. I intentionally splitted it into smaller pieces. Take a look at this demo and comments.

Since Invoke-SqlCmd is considered harmful (SQL Injection), I use my own function to invoke SQL
function Invoke-Sql(
    $ConnectionString,
    $Query,
    $Parameters
) {
    $conn = New-Object System.Data.SqlClient.SqlConnection -ArgumentList $ConnectionString
    $cmd = New-Object System.Data.SqlClient.SqlCommand -ArgumentList $Query,$conn
    $conn.Open()
    if ($Parameters) {
      foreach ($arg in $Parameters.GetEnumerator()){
        $cmd.Parameters.AddWithValue($arg.Key, $arg.Value) | Out-Null;
      }
    }
    $reader = $cmd.ExecuteReader()
    if ($reader.Read()) {
      [string[]]$columns = 0..($reader.FieldCount-1) |
          % { if ($reader.GetName($_)) { $reader.GetName($_) } else { "(no name $_)" } }
      do {
        $obj = @{}
        0..($reader.FieldCount-1) | % { $obj[$columns[$_]] = $reader[$_] }
        [PSCustomObject]$obj
      } while ($reader.Read())
    }
    $reader.Dispose()
    $cmd.Dispose()
    $conn.Dispose()
}

You need a database table. Since there is no strict schema in question, I assume following, minimal:
$conn = 'Data Source=.;Initial Catalog=Test;Integrated Security=SSPI'

$createTestTable = @'
CREATE TABLE MyRequests
(
    RequestId int,
    DateLogged datetime
)
'@

Invoke-Sql $conn $createTestTable

There is no sample data, I assume folders named 1, 2, 3, .. 7 and matching records in SQL database:
1..7 | % {
    Invoke-Sql $conn 'INSERT MyRequests VALUES (@id,@value)' @{id=$_;value=[DateTime]::Now.AddDays(-$_)}
    mkdir $_
}

Table should contain following records (dates may differ):
RequestId   DateLogged
----------- -----------------------
1           2018-09-23 14:47:49.113
2           2018-09-22 14:47:49.130
3           2018-09-21 14:47:49.137
4           2018-09-20 14:47:49.140
5           2018-09-19 14:47:49.140
6           2018-09-18 14:47:49.143
7           2018-09-17 14:47:49.147

Then, final solution:
#get deleted id's using OUTPUT clause
$older = Invoke-Sql $conn 'DELETE FROM MyRequests OUTPUT deleted.RequestId WHERE DateLogged<@date' @{date=[DATETime]::Now.AddDays(-4)}

#foreach id in returned set, delete corresponding folder
$older | select -ExpandProperty RequestId | % { rm $_ }


Authentication with oidc-client.js and Identityserver4 in a React frontend

Question: Lately I'm trying to set-up authentication using IdentityServer4 with a React client. I followed the Adding a JavaScript client tutorial (partly) of the IdentityServer documentation: https://media.readthedocs.org/pdf/identityserver4/release/identityserver4.pdf also using the Quickstart7_JavaScriptClient file.
The downside is that I'm using React as my front-end and my knowledge of React is not good enough to implement the same functionality used in the tutorial using React.
Nevertheless, I start reading up and tried to get started with it anyway. My IdentityServer project and API are set-up and seem to be working correctly (also tested with other clients).
I started by adding the oidc-client.js to my Visual Code project. Next I created a page which get's rendered at the start (named it Authentication.js) and this is the place where the Login, Call API and Logout buttons are included. This page (Authentication.js) looks as follows:
import React, { Component } from 'react';
import {login, logout, api, log} from '../../testoidc'
import {Route, Link} from 'react-router';

export default class Authentication extends Component {
    constructor(props) {
      super(props);
    }

    render() {
      return (

                <button id="login" onClick={() => {login()}}>Login</button>
                <button id="api" onClick={() => {api()}}>Call API</button>
                <button id="logout" onClick={() => {logout()}}>Logout</button>

                <Route exact path="/callback" render={() => {window.location.href="callback.html"}} />

                {/* {<Route path='/callback' component={callback}>callback</Route>} */}

      );
    }
  }

In the testoidc.js file (which get's imported above) I added all the oidc functions which are used (app.js in the example projects). The route part should make the callback.html available, I have left that file as is (which is probably wrong).
The testoidc.js file contains the functions as follow:
import Oidc from 'oidc-client'

export function log() {
  document.getElementById('results').innerText = '';

  Array.prototype.forEach.call(arguments, function (msg) {
      if (msg instanceof Error) {
          msg = "Error: " + msg.message;
      }
      else if (typeof msg !== 'string') {
          msg = JSON.stringify(msg, null, 2);
      }
      document.getElementById('results').innerHTML += msg + '\r\n';
  });
}

var config = {
  authority: "http://localhost:5000",
  client_id: "js",
  redirect_uri: "http://localhost:3000/callback.html",
  response_type: "id_token token",
  scope:"openid profile api1",
  post_logout_redirect_uri : "http://localhost:3000/index.html",
};
var mgr = new Oidc.UserManager(config);

mgr.getUser().then(function (user) {
  if (user) {
      log("User logged in", user.profile);
  }
  else {
      log("User not logged in");
  }
});

export function login() {
  mgr.signinRedirect();
}

export function api() {
  mgr.getUser().then(function (user) {
      var url = "http://localhost:5001/identity";

      var xhr = new XMLHttpRequest();
      xhr.open("GET", url);
      xhr.onload = function () {
          log(xhr.status, JSON.parse(xhr.responseText));
      }
      xhr.setRequestHeader("Authorization", "Bearer " + user.access_token);
      xhr.send();
  });
}

export function logout() {
  mgr.signoutRedirect();
}

There are multiple things going wrong. When I click the login button, I get redirected to the login page of the identityServer (which is good). When I log in with valid credentials I'm getting redirected to my React app: http://localhost:3000/callback.html#id_token=Token
This client in the Identity project is defined as follows:
new Client
                {
                    ClientId = "js",
                    ClientName = "JavaScript Client",
                    AllowedGrantTypes = GrantTypes.Implicit,
                    AllowAccessTokensViaBrowser = true,

                    // where to redirect to after login
                    RedirectUris = { "http://localhost:3000/callback.html" },

                    // where to redirect to after logout
                    PostLogoutRedirectUris = { "http://localhost:3000/index.html" },

                    AllowedCorsOrigins = { "http://localhost:3000" },
                    AllowedScopes =
                    {
                        IdentityServerConstants.StandardScopes.OpenId,
                        IdentityServerConstants.StandardScopes.Profile,
                        "api1"
                    }

                }

Though, it seems the callback function is never called, it just stays on the callback url with a very long token behind it..
Also the getUser function keeps displaying 'User not logged in' after logging in and the Call API button keeps saying that there is no token. So obviously things are not working correctly. I just don't know on which points it goes wrong.
When inspecting I can see there is a token generated in the local storage:

Also when I click the logout button, I get redirected to the logout page of the Identity Host, but when I click logout there I don't get redirected to my client.
My questions are:

Am I on the right track implementing the oidc-client in combination with IdentityServer4?
Am I using the correct libraries or does react require different libraries than the oidc-client.js one.
Is there any tutorial where a react front-end is used in combination with IdentityServer4 and the oidc-client (without redux), I couldn't find any.
How / where to add the callback.html, should it be rewritten?

Could someone point me in the right direction, there are most likely more things going wrong here but at the moment I am just stuck in where to even begin.
Comment: Hi Nicolas, it'd help if you can put a minimal repro code somewhere so that it's easy for folks to debug and find out what's broken. I have successfully integrated ID4 with React. You don't need a separate `callback.html`, all of this can be handled easily using browser routes in a single page application. But having access to your code will help get to the bottom of this quicker.
Comment: @hazardous Thank you for your reply and sorry for my late reaction. Due to vacation and all it was a bit hectic. My knowledge about React and ID4 has fortunately increased since I posted this, In the coming weeks I will try to implement it again. Should I fail then I'll try upload a repro as soon as possible. May I succeed then I will post the results back here as well.
Comment: What is your back-end for this project? I ask because often times there's no reason to implement the OIDC flow completely in the front-end, and it's trivial to implement the whole flow outside of react. For example, I've done many react projects where my entire OIDC functionality is handled by the .net OIDC middleware, and it's often the better way to go.
Comment: @DanielTabuenca if your client js app be it react or angular is running all on the browser then you just can't implement the whole flow outside of react.
Answer: IdentityServer4 is just a backend implementation of OIDC; so, all you need to do is implement the flow in the client using the given APIs. I don't know what oidc-client.js file is but it is most likely doing the same thing that you could have implemented yourself. The flow itself is very simple:

React app prepares the request and redirects the user to the Auth server with client_id and redirect_uri (and state, nonce)
IdentityServer checks if the client_id and redirect_uri match.

If the user is not logged in, show a login box
If a consent form is necessary (similar to when you login via Facebook/Google in some apps), show the necessary interactions
If user is authenticated and authorized, redirect the page to the redirect_uri with new parameters. In your case, you the URL will look like this: https://example.com/cb#access_token=...&id_token=...&stuff-like-nonce-and-state

Now, the React app needs to parse the URL, access the values, and store the token somewhere to be used in future requests:

Easiest way to achieve the logic is to first set a route in the router that resolves into a component that will do the logic. This component can be "invisible." It doesn't even need to render anything. You can set the route like this:
<Route path="/cb" component={AuthorizeCallback} />

Then, implement OIDC client logic in AuthorizeCallback component. In the component, you just need to parse the URL. You can use location.hash to access #access_token=...&id_token=...&stuff-like-nonce-and-state part of the URL. You can use URLSearchParams or a 3rd party library like qs. Then, just store the value in somewhere (sessionStorage, localStorage, and if possible, cookies). Anything else you do is just implementation details. For example, in one of my apps, in order to remember the active page that user was on in the app, I store the value in sessionStorage and then use the value from that storage in AuthorizeCallback to redirect the user to the proper page. So, Auth server redirects to "/cb" that resolves to AuthorizeCallback and this component redirects to the desired location (or "/" if no location was set) based on where the user is.
Also, remember that if the Authorization server's session cookie is not expired, you will not need to relogin if the token is expired or deleted. This is useful if the token is expired but it can be problematic when you log out. That's why when you log out, you need to send a request to Authorization server to delete / expire the token immediately before deleting the token from your storage.


How can I get the form validated with AJAX?

Question: I'm building pretty simple forms loaded with AJAX and displayed with Foundation Reveal. I can get the form submission to work with AJAX, and update the markup inside the reveal, but with the AJAX-powered submission button, the validation method is not executed.
Why? How can I get the form validated with AJAX?
I just want the normal form validation process, where I can add my own validation to ::validateForm and then pass control to my AJAX form submission handler.
$form['submit'] = [
  '#type' => 'submit',
  '#value' => 'Tallenna',
  '#ajax' => [
    'callback' => '::ajaxFormSubmitHandler'
  ]
];

I tried with $form['#validate'] = array('::validateForm');, but it doesn't work. I also tried calling Form::validateForm($form, $form_state) in the submission handler with the following code.
Form::validateForm($form, $form_state);
if ($form_state->hasAnyErrors()) {
  $form_state->setRebuild();
  $errors = $form_state->getErrors();
  return($form);
}
else {
  $form_state->setRebuild(FALSE);
}

I can set up the callback for a single element.
$form['field_asiakkaan_nimi'] = [
  '#type' => 'textfield',
  '#title' => $this->t('Asiakkaan nimi'),
  '#required' => TRUE,
  '#ajax' => [
    'callback' => array($this, 'validateNimiAjax'),
    'event' => 'change',
  ],
  '#suffix' => ''
];

I can use that callback for the field processing.
  public function validateNimiAjax(array &$form, FormStateInterface $form_state) {
    $response = new AjaxResponse();
    $message = 'Nimi validoitu!';
    $response->addCommand(new HtmlCommand('.nimi-valid-message', $message));
    return $response;
  }

This doesn't seem to be the proper way to go about this.
I am also getting this error:

The form has become outdated. Copy any unsaved work in the form below
  and then reload this page.

How can I get AJAX delivered forms validated and processed?
Answer: Here is a very basic example that will validate your data. It should give you an idea of how to handle it:
class myForm extends FormBase {
  public function getFormId() {
    return 'my_form';
  }

  public function buildForm(array &$form, FormStateInterface $form_state) {
    $form['#prefix'] = '
';
    $form['#suffix'] = '
';
    $form['some_text'] = [
      '#type' => 'textfield',
      '#title' => $this->t('Enter a value'),
    ];

    $form['submit'] = [
      '#type' => 'submit',
      '#value' => $this->t('Submit'),
      '#ajax' => [
        'wrapper' => 'my_form_wrapper',
        'callback' => '::ajaxCallback',
      ],
    ];

    return parent::buildForm($form, $form_state);
  }

  public function validateForm(array &$form, FormStateInterface $form_state) {
    if($form_state->getValue('some_text') != 'Awesome')  {
      $form_state->setError($form['some_text'], $this->t('Please enter "Awesome"'));
    }
    parent::validateForm($form, $form_state);
  }

  public function ajaxCallback(array &$form, FormStateInterface $form_state) {
    return $form;
  }

  public function submitForm(array &$form, FormStateInterface $form_state) {
    drupal_set_message($this->t('The form was submitted'));

    parent::submitForm($form, $form_state);
  }
}

With the above code, the form is built, and the ajax is set. Note that no validation needs to be set, as the validateForm() method will be called automatically if it exists. When the ajax is executed, it will always call the validate form method first. If validation is passed, then the submitForm() method will be called, but if validation is failed, it will not. Finally the ajax callback is called and the form is returned.
Comment: $form is an array and has no setValue method
Comment: Oops, typo. Fixed.
Answer: This look like correct, just try to use an ID instead of a class
  '#ajax' => [
    'callback' => '::ajaxFormSubmitHandler'
  ]

and
//$response->addCommand(new HtmlCommand('.nimi-valid-message', $message));
$response->addCommand(new HtmlCommand('#nimi-valid-message', $message));

Example:
http://www.drupal8.ovh/en/tutoriels/26/ajax-form
Comment: And again correcting myself for someone else struggling with this..

I had to check manually for the errors in my ajax submit and then return the whole form, like so:

`    if ($form_state->hasAnyErrors()) {
      $form_state->setRebuild();
      $errors = $form_state->getErrors();
      return($form);
    }
    else {
      $form_state->setRebuild(FALSE);
    }`
Comment: Just to return to this older topic.. (Oh, automatic submit)

I'm still struggling with form validation in ajax submissions.

I have pretty much the same form as previously, and everything apart from validation works as expected. My validateForm method is called, but I cannot raise an error and form submits no matter what.


Ajuda no cálculo de subtração com jquery

Question: Tenho o seguinte campo:

O objetivo é conforme for incluindo o adulto, ele vai somando os valores e quando remover campo, ele subtrai. Levando em consideração que o pacote é 150,00 e cada adulto é 50,00, o cálculo de adição está funcionando corretamente. O problema está na subtração. Quando tento retirar um adulto, o cálculo não está saindo da forma correta. Estou fazendo da seguinte forma:
<input type="hidden" name="SubTotal" id="subTotalForm" value="">
  <input type="hidden" name="ValorTotal" id="totalForm" value="">

              <select name="TipoPassageiro[]" class="form-control TipoPassageiro">
                <option value="">Tipo de Passageiro</option>
                <option data-tipo="1" value="Adulto">Adulto</option>
                <option data-tipo="2" value="Crianca6Anos">Criança de colo de até 6 anos</option>
                <option data-tipo="3" value="Crianca6a12Anos">Criança entre 6 e 12 anos</option>
                <option data-tipo="4" value="Adolescentes">Adolescente entre 12 e 18 anos</option>
              </select>

          <input type="text" name="NomePAX[]" class="form-control" placeholder="Nome do pax" value="">

            <select id="TipoDocumento" name="TipoDocumento[]" class="form-control">
              <option>Tipo de documento</option>
              <option value="Carteira de Identidade">Carteira de Identidade</option>
              <option value="Carteira Nacional de Habilitação">Carteira Nacional de Habilitação</option>
              <option value="Carteira de Trabalho">Carteira de Trabalho</option>
              <option value="Certidão de Nascimento">Certidão de Nascimento</option>
              <option value="Passaporte">Passaporte</option>
              <option value="Cédula de Identidade Estrangeira">Cédula de Identidade Estrangeira</option>
              <option value="Outros">Outros</option>
            </select>

             <input type="text" name="Documento[]" class="form-control" placeholder="RG da pessoa autorizada" value="">

             <input type="text" name="OrgaoEmissor[]" class="form-control" placeholder="Órgão Emissor" value="">

          <button type="button" class="removerCampo btn btn-danger" title="Remover linha"> Remover</button>

          <button type="button" class="adicionarCampo btn btn-primary" title="Adicionar item"> Adicionar mais passageiros</button>

      Valor do Pacote:  R$ 150,00

      Valor Subtotal:

         R$ 0,00

      Valor Total:

         R$ 150,00

      <button type="submit" class="btn btn-primary">FINALIZAR </button>

e o Jquery
     $(function () {
       function removeCampo() {
             $(".removerCampo").unbind("click");
             $(".removerCampo").bind("click", function () {
                if($("tr.linhas").length > 1){
                   var campo = $(".TipoPassageiro:last").val();
                   //var valor_subtotal = 0;
                   var valor_subtotal = $("#subTotalForm").val();
                   var valor_total = $("#valortotal").val();

                   if(campo == 'Adulto'){
                    valor = <?php echo $visualizar[1]->ValorAdulto; ?>.toFixed(2).replace(" ",".");
                    var valorSubTotal = valor_subtotal - valor;
                    document.getElementById("subTotalForm").value = valorSubTotal.toFixed(2).replace(" ",".");
                    $("#valorsubtotal").text(valorSubTotal.toFixed(2).replace(".", ","));
                    if(valor_total == 150.00){
                       var valor_total = 150.00;
                       var valor_total = valor_total.toFixed(2).replace(" ",".");
                       var valorTotal = valor_total - valor;
                    }else{
                      var valor_total = $("#valortotal").val();
                      var valor_total = valor_total.toFixed(2).replace(" ",".");
                      var valorTotal = valor_total - valor;
                    }
                    $("#valortotal").text(valorTotal.toFixed(2).replace(".", ","));
                  }
                    $(this).parent().parent().remove();
                 }
             });
       }
       $(".adicionarCampo").click(function () {
          novoCampo = $("tr.linhas:first").clone();
          novoCampo.find('input[type="text"]').val("");
          novoCampo.find('select').val("");
          novoCampo.insertAfter("tr.linhas:last");
          removeCampo();
});
$(document).on("change", ".TipoPassageiro", function(){
   // objeto com os preços separados por "tipo"
    var precos = {
       tipo1: <?php echo $visualizar[1]->ValorAdulto; ?>,
       tipo2: <?php echo $visualizar[1]->ValorCrianca6Anos; ?>,
       tipo3: <?php echo $visualizar[1]->ValorCrianca6a12Anos; ?>,
       tipo4: <?php echo $visualizar[1]->ValorAdolescentes; ?>
    }
    // inicia o valor total com 0
    var valor_subtotal = 0;
    var valor_total = 150.00;
    // laço que irá percorrer os selects e somar de acordo com o tipo selecionado
    $(".TipoPassageiro").each(function(){
       // pega o valor do tipo (1, 2, 3 ou 4)
       var tipo = $("option:selected", this).data("tipo");
       // só inclui na soma o select que tiver um tipo selecionado
       if(tipo) valor_subtotal += parseFloat(precos['tipo'+tipo]);
    });
    // formata a soma final com duas casas decimais separadas por vírgula
    $("#valorsubtotal").text(valor_subtotal.toFixed(2).replace(".", ","));
    var valorTotal = valor_subtotal + valor_total;
    $("#valortotal").text(valorTotal.toFixed(2).replace(".", ","));
    document.getElementById("subTotalForm").value = valor_subtotal.toFixed(2).replace(" ",".");
    document.getElementById("totalForm").value = valorTotal.toFixed(2).replace(" ",".");
  });
});

Answer: O código tem vários problemas. Em vários pontos você redeclara as variáveis, como por exemplo:
var valor_total = 150.00;
var valor_total = valor_total.toFixed(2).replace(" ",".");
var valorTotal = valor_total - valor;

Se você já tinha declarado a variável com var antes, não coloque novamente o var:
var valor_total = $("#valortotal").val(); // AQUI A VARIÁVEL FOI DECLARADA
...
valor_total = 150.00; // AQUI VOCÊ APENAS ALTERA O SEU VALOR (SEM var)
valor_total = valor_total.toFixed(2).replace(" ","."); // E AQUI TAMBÉM
var valorTotal = valor_total - valor;

Nesta linha, o seletor está errado:
var campo = $(".TipoPassageiro:last").val();

Vai sempre buscar o último select, que não corresponde ao do botão clicado. O correto seria:
var campo = $(this).closest("tr").find(".TipoPassageiro").val();

Irá buscar o select que está na mesma linha do botão clicado.
E aqui não está achando nenhum valor porque #valortotal é uma div:
var valor_total = $("#valortotal").val();

O correto seria pegar o texto e converter para número com parseFloat() (sem o var, como dito antes):
valor_total = parseFloat($("#valortotal").text());

Vai ficar assim:
$(function () {
       function removeCampo() {
             $(".removerCampo").unbind("click");
             $(".removerCampo").bind("click", function () {
                if($("tr.linhas").length > 1){
                   var campo = $(this).closest("tr").find(".TipoPassageiro").val();
                   //var valor_subtotal = 0;
                   var valor_subtotal = $("#subTotalForm").val();
                   var valor_total = $("#valortotal").val();

                   if(campo == 'Adulto'){
                    valor = <?php echo $visualizar[1]->ValorAdulto; ?>.toFixed(2).replace(" ",".");
                    var valorSubTotal = valor_subtotal - valor;
                    document.getElementById("subTotalForm").value = valorSubTotal.toFixed(2).replace(" ",".");
                    $("#valorsubtotal").text(valorSubTotal.toFixed(2).replace(".", ","));
                    if(valor_total == 150.00){
                     valor_total = 150.00;
                     valor_total = valor_total.toFixed(2).replace(" ",".");
                    }else{
                      valor_total = parseFloat($("#valortotal").text());
                      valor_total = valor_total.toFixed(2).replace(" ",".");
                    }
                    var valorTotal = valor_total - valor;
                    $("#valortotal").text(valorTotal.toFixed(2).replace(".", ","));
                  }
                    $(this).parent().parent().remove();
                 }
             });
       }
       $(".adicionarCampo").click(function () {
          novoCampo = $("tr.linhas:first").clone();
          novoCampo.find('input[type="text"]').val("");
          novoCampo.find('select').val("");
          novoCampo.insertAfter("tr.linhas:last");
          removeCampo();
});
$(document).on("change", ".TipoPassageiro", function(){
   // objeto com os preços separados por "tipo"
   var precos = {
       tipo1: <?php echo $visualizar[1]->ValorAdulto; ?>,
       tipo2: <?php echo $visualizar[1]->ValorCrianca6Anos; ?>,
       tipo3: <?php echo $visualizar[1]->ValorCrianca6a12Anos; ?>,
       tipo4: <?php echo $visualizar[1]->ValorAdolescentes; ?>
    }
    // inicia o valor total com 0
    var valor_subtotal = 0;
    var valor_total = 150.00;
    // laço que irá percorrer os selects e somar de acordo com o tipo selecionado
    $(".TipoPassageiro").each(function(){
       // pega o valor do tipo (1, 2, 3 ou 4)
       var tipo = $("option:selected", this).data("tipo");
       // só inclui na soma o select que tiver um tipo selecionado
       if(tipo) valor_subtotal += parseFloat(precos['tipo'+tipo]);
    });
    // formata a soma final com duas casas decimais separadas por vírgula
    $("#valorsubtotal").text(valor_subtotal.toFixed(2).replace(".", ","));
    valorTotal = valor_subtotal + valor_total;
    $("#valortotal").text(valorTotal.toFixed(2).replace(".", ","));
    document.getElementById("subTotalForm").value = valor_subtotal.toFixed(2).replace(" ",".");
    document.getElementById("totalForm").value = valorTotal.toFixed(2).replace(" ",".");
  });
});

Exemplo de funcionamento sem os códigos PHP:

$(function () {
       function removeCampo() {
             $(".removerCampo").unbind("click");
             $(".removerCampo").bind("click", function () {
                if($("tr.linhas").length > 1){
                   var campo = $(this).closest("tr").find(".TipoPassageiro").val();
                   //var valor_subtotal = 0;
                   var valor_subtotal = $("#subTotalForm").val();
                   var valor_total = $("#valortotal").val();

                   if(campo == 'Adulto'){
                    valor = (50).toFixed(2).replace(" ",".");
                    var valorSubTotal = valor_subtotal - valor;
                    document.getElementById("subTotalForm").value = valorSubTotal.toFixed(2).replace(" ",".");
                    $("#valorsubtotal").text(valorSubTotal.toFixed(2).replace(".", ","));
                    if(valor_total == 150.00){
                     valor_total = 150.00;
                     valor_total = valor_total.toFixed(2).replace(" ",".");
                    }else{
                      valor_total = parseFloat($("#valortotal").text());
                      valor_total = valor_total.toFixed(2).replace(" ",".");
                    }
                    var valorTotal = valor_total - valor;
                    $("#valortotal").text(valorTotal.toFixed(2).replace(".", ","));
                  }
                    $(this).parent().parent().remove();
                 }
             });
       }
       $(".adicionarCampo").click(function () {
          novoCampo = $("tr.linhas:first").clone();
          novoCampo.find('input[type="text"]').val("");
          novoCampo.find('select').val("");
          novoCampo.insertAfter("tr.linhas:last");
          removeCampo();
});
$(document).on("change", ".TipoPassageiro", function(){
   // objeto com os preços separados por "tipo"
   var precos = {
       tipo1: 50,
       tipo2: 20,
       tipo3: 10,
       tipo4: 30
    }

    // inicia o valor total com 0
    var valor_subtotal = 0;
    var valor_total = 150.00;
    // laço que irá percorrer os selects e somar de acordo com o tipo selecionado
    $(".TipoPassageiro").each(function(){
       // pega o valor do tipo (1, 2, 3 ou 4)
       var tipo = $("option:selected", this).data("tipo");
       // só inclui na soma o select que tiver um tipo selecionado
       if(tipo) valor_subtotal += parseFloat(precos['tipo'+tipo]);
    });
    // formata a soma final com duas casas decimais separadas por vírgula
    $("#valorsubtotal").text(valor_subtotal.toFixed(2).replace(".", ","));
    valorTotal = valor_subtotal + valor_total;
    $("#valortotal").text(valorTotal.toFixed(2).replace(".", ","));
    document.getElementById("subTotalForm").value = valor_subtotal.toFixed(2).replace(" ",".");
    document.getElementById("totalForm").value = valorTotal.toFixed(2).replace(" ",".");
  });
});

<input type="hidden" name="SubTotal" id="subTotalForm" value="">
  <input type="hidden" name="ValorTotal" id="totalForm" value="">

              <select name="TipoPassageiro[]" class="form-control TipoPassageiro">
                <option value="">Tipo de Passageiro</option>
                <option data-tipo="1" value="Adulto">Adulto</option>
                <option data-tipo="2" value="Crianca6Anos">Criança de colo de até 6 anos</option>
                <option data-tipo="3" value="Crianca6a12Anos">Criança entre 6 e 12 anos</option>
                <option data-tipo="4" value="Adolescentes">Adolescente entre 12 e 18 anos</option>
              </select>

          <input type="text" name="NomePAX[]" class="form-control" placeholder="Nome do pax" value="">

            <select id="TipoDocumento" name="TipoDocumento[]" class="form-control">
              <option>Tipo de documento</option>
              <option value="Carteira de Identidade">Carteira de Identidade</option>
              <option value="Carteira Nacional de Habilitação">Carteira Nacional de Habilitação</option>
              <option value="Carteira de Trabalho">Carteira de Trabalho</option>
              <option value="Certidão de Nascimento">Certidão de Nascimento</option>
              <option value="Passaporte">Passaporte</option>
              <option value="Cédula de Identidade Estrangeira">Cédula de Identidade Estrangeira</option>
              <option value="Outros">Outros</option>
            </select>

             <input type="text" name="Documento[]" class="form-control" placeholder="RG da pessoa autorizada" value="">

             <input type="text" name="OrgaoEmissor[]" class="form-control" placeholder="Órgão Emissor" value="">

          <button type="button" class="removerCampo btn btn-danger" title="Remover linha"> Remover</button>

          <button type="button" class="adicionarCampo btn btn-primary" title="Adicionar item"> Adicionar mais passageiros</button>

      Valor do Pacote:  R$ 150,00

      Valor Subtotal:

         R$ 0,00

      Valor Total:

         R$ 150,00

      <button type="submit" class="btn btn-primary">FINALIZAR </button>

Edição
Adicione um value="" na primeira option do select:
<option value="">Tipo de documento</option>

Comment: @Fox.11 Adicionei um detalhe no final da resposta. Abs!
Comment: Show de bola Sam. Obrigado mais uma vez pela força!
Comment: Certo. Obrigado.


How do you use the Android Volley API?

Question: I am thinking of implementing the Android Volley library in my next projects (Google IO presentation about Volley).
However, I haven't found any serious API for that library.
How do I upload files, do POST/GET requests, and add a Gson parser as a JSON parser using Volley?
Source code
Comment: Go to this link http://java.dzone.com/articles/android-%E2%80%93-volley-library written by Stack Overflow User named Paresh Mayani.
Comment: Check this out: http://afzaln.com/volley/overview-summary.html
Comment: I've written about this here -> https://goo.gl/nl2DfN
Answer: Edit: finally here it is an official training about "Volley library"
I found some examples about Volley library

6 examples by Ognyan Bankov :

Simple request
JSON request
Gson request
Image loading
with newer external HttpClient (4.2.3)
With Self-Signed SSL Certificate.

one good simple example by Paresh Mayani

other example by Hardik Trivedi

(NEW) Android working with Volley Library by Ravi Tamada
Comment: Interesting library and great links!! Thanks @Abdrahmn_msi ^^
Comment: I need help for PATCH requests with volley, can anyone help?
Answer: Unfortunately there is no documentation for a Volley library like JavaDocs until now. Only repo on github and several tutorials across the Internet. So the only good docs is source code :) . When I played with Volley I read this tutorial.
About post/get you can read this : Volley - POST/GET parameters Hope this helps
Answer: This is an illustration for making a POST request using Volley.  StringRequest is used to get response in the form of String.
Assuming your rest API returns a JSON. The JSON response from your API is  received as String here, which you can covert again to JSON and process  it further. Added comments in code.
StringRequest postRequest = new StringRequest(Request.Method.POST, "PUT_YOUR_REST_API_URL_HERE",
            new Response.Listener<String>() {
                @Override
                public void onResponse(String response) {
                    try {
                        final JSONObject jsonObject = new JSONObject(response);
                        // Process your json here as required
                    } catch (JSONException e) {
                        // Handle json exception as needed
                    }
                }
            },
            new Response.ErrorListener() {
                @Override
                public void onErrorResponse(VolleyError error) {
                    String json = null;
                    NetworkResponse response = error.networkResponse;
                    if(response != null && response.data != null){
                        switch(response.statusCode) {
                            default:
                                String value = null;
                                try {
                                    // It is important to put UTF-8 to receive proper data else you will get byte[] parsing error.
                                    value = new String(response.data, "UTF-8");
                                } catch (UnsupportedEncodingException e) {
                                    e.printStackTrace();
                                }
                                json = trimMessage(value, "message");
                                // Use it for displaying error message to user
                                break;
                        }
                    }
                    loginError(json);
                    progressDialog.dismiss();
                    error.printStackTrace();
                }
                public String trimMessage(String json, String key){
                    String trimmedString = null;
                    try{
                        JSONObject obj = new JSONObject(json);
                        trimmedString = obj.getString(key);
                    } catch(JSONException e){
                        e.printStackTrace();
                        return null;
                    }
                    return trimmedString;
                }
            }
    ) {
        @Override
        protected Map<String, String> getParams() {
            Map<String, String> params = new HashMap<>();
            params.put("abc", "pass abc");
            params.put("xyz", "pass xyz");
            // Pass more params as needed in your rest API
    // Example you may want to pass user input from EditText as a parameter
    // editText.getText().toString().trim()
            return params;
        }
        @Override
        public String getBodyContentType() {
            // This is where you specify the content type
            return "application/x-www-form-urlencoded; charset=UTF-8";
        }
    };

    // This adds the request to the request queue
    MySingleton.getInstance(YourActivity.this)
.addToRequestQueue(postRequest);

// Below is MySingleton class
public class MySingleton {
    private static MySingleton mInstance;
    private RequestQueue mRequestQueue;
    private static Context mCtx;
    private MySingleton(Context context) {
        mCtx = context;
        mRequestQueue = getRequestQueue();
    }

    public static synchronized MySingleton getInstance(Context context) {
        if (mInstance == null) {
            mInstance = new MySingleton(context);
        }
        return mInstance;
    }

    public RequestQueue getRequestQueue() {
        if (mRequestQueue == null) {
            // getApplicationContext() is key, it keeps you from leaking the
            // Activity or BroadcastReceiver if someone passes one in.
            mRequestQueue = Volley.newRequestQueue(mCtx.getApplicationContext());
        }
        return mRequestQueue;
    }

    public <T> void addToRequestQueue(Request<T> req) {
        getRequestQueue().add(req);
    }
}

Answer: Just add volley.jar library to your project.
and then
As per Android documentation :
// Instantiate the RequestQueue.
RequestQueue queue = Volley.newRequestQueue(this);
String url ="http://www.google.com";

// Request a string response from the provided URL.
StringRequest stringRequest = new StringRequest(Request.Method.GET, url,
            new Response.Listener<String>() {
    @Override
    public void onResponse(String response) {
        // process your response here

    }
}, new Response.ErrorListener() {
    @Override
    public void onErrorResponse(VolleyError error) {
        //perform operation here after getting error
    }
});
// Add the request to the RequestQueue.
queue.add(stringRequest);

For more help refer How to user Volley
Answer: Use this class. It provides you an easy way to connect to the database.
public class WebRequest {

    private Context mContext;
    private String mUrl;
    private int mMethod;
    private VolleyListener mVolleyListener;

    public WebRequest(Context context) {
        mContext = context;
    }

    public WebRequest setURL(String url) {
        mUrl = url;
        return this;
    }

    public WebRequest setMethod(int method) {
        mMethod = method;
        return this;
    }

    public WebRequest readFromURL() {
        RequestQueue requestQueue = Volley.newRequestQueue(mContext);
        StringRequest stringRequest = new StringRequest(mMethod, mUrl, new Response.Listener<String>() {
            @Override
            public void onResponse(String s) {
                mVolleyListener.onRecieve(s);
            }
        }, new Response.ErrorListener() {
            @Override
            public void onErrorResponse(VolleyError volleyError) {
                mVolleyListener.onFail(volleyError);
            }
        });

        requestQueue.add(stringRequest);
        return this;
    }

    public WebRequest onListener(VolleyListener volleyListener) {
        mVolleyListener = volleyListener;
        return this;
    }

    public interface VolleyListener {
        public void onRecieve(String data);

        public void onFail(VolleyError volleyError);
    }
}

Example usage:
new WebRequest(mContext)
     .setURL("http://google.com")
     .setMethod(Request.Method.POST)
     .readFromURL()
     .onListener(new WebRequest.VolleyListener() {
         @Override
         public void onRecieve(String data) {
         }

         @Override
         public void onFail(VolleyError volleyError) {
         }
     });

Comment: @Piyush use this link:
https://www.computerworld.com/article/2702452/how-to-send-a-post-request-with-google-volley-on-android.html
Comment: @Piyush Just add getParams method.
Comment: I don't think just adding the getParam method solves the whole purpose. Anyways I have figured out and did the desired changes in class
Answer: In simple way
 private void load() {
    JsonArrayRequest arrayreq = new JsonArrayRequest(ip.ip+"loadcollege.php",
            new Response.Listener<JSONArray>() {

                @Override
                public void onResponse(JSONArray response) {

                    Album a;

                    try {
                        JSONArray data = new JSONArray(response.toString());
                        for (int i = 0; i < data.length(); i++) {
                            JSONObject c = data.getJSONObject(i);
                            one = c.getString("cname").split(",");
                            two=c.getString("caddress").split(",");
                            three = c.getString("image").split(",");
                            four = c.getString("cid").split(",");
                            five = c.getString("logo").split(",");

                            a = new Album(one[0].toString(),two[0].toString(),ip.ip+"images/"+ three[0].toString(),four[0].toString(),ip.ip+"images/"+ five[0].toString());
                            albumList.add(a);

                        }
                        adapter.notifyDataSetChanged();

                    } catch (JSONException e) {
                        e.printStackTrace();
                    }
                }
            },
            // The final parameter overrides the method onErrorResponse() and passes VolleyError
            //as a parameter
            new Response.ErrorListener() {
                @Override
                // Handles errors that occur due to Volley
                public void onErrorResponse(VolleyError error) {
                    Log.e("Volley", "Error");
                }
            }
    );
    // Adds the JSON array request "arrayreq" to the request queue
    requestQueue.add(arrayreq);
}

Answer: Before testing all of the above answers, include
compile 'com.android.volley:volley:1.0.0'
in your gradle file and don't forgot to add the Internet permission to your Manifest file.
Answer: private void userregister() {

  final  ProgressDialog pDialog = new ProgressDialog(this);
    pDialog.setMessage("Loading...");
    pDialog.show();

    RequestQueue queue = Volley.newRequestQueue(SignupActivity.this);
    String url = "you";
    StringRequest stringRequest = new StringRequest(Request.Method.POST, url,
            new Response.Listener<String>() {
                @Override
                public void onResponse(String response) {
                    pDialog.cancel();

                    try {
                        JSONObject jsonObject= new JSONObject(response.toString());
                        Log.e("status", ""+jsonObject.getString("status"));
                        if(jsonObject.getString("status").equals("success"))
                        {
                            String studentid=jsonObject.getString("id");
                            Intent intent=new Intent(SignupActivity.this, OTPVerificationActivity.class);
                            startActivity(intent);
                            finish();
                        }

                    } catch (JSONException e) {
                        e.printStackTrace();
                    }
                    Log.e("String ", ""+response);

                }
            }, new Response.ErrorListener() {
        @Override
        public void onErrorResponse(VolleyError error) {

        }
    }) {

        @Override
        protected Map<String, String> getParams() throws AuthFailureError {
            Map<String, String> params = new HashMap<>();
            params.put("password", input_password.getText().toString());
            params.put("cpassword", input_reEnterPassword.getText().toString());
            params.put("email", input_email.getText().toString());
            params.put("status", "1");
            params.put("last_name", input_lastname.getText().toString());
            params.put("phone", input_mobile.getText().toString());
            params.put("standard", input_reStandard.getText().toString());
            params.put("first_name", input_name.getText().toString());
            params.put("refcode", input_reReferal.getText().toString());
            params.put("created_at","");
            params.put("update_at", "");
            params.put("address", input_address.getText().toString());
            return params;
        }
    };
    // Add the request to the RequestQueue.
    queue.add(stringRequest);

Get full code here
Comment: Never answer a question with only code. Explain what your code does and why it's better than the all other answers.

How do you use the Android Volley API?


simple arduino code exhibiting unexpected results

Question: can anyone see why this simpple arduino code isnt working?
<code>void loop() {
  if (Serial.available() > 0) {
    char num = Serial.read();  // Read a single character from serial input
    if (num == '5') {
      Serial.println("hooray!");
    }
    else {
      Serial.println("booo");
    }
  }
}
</code>
it correctly prints hooray when i enter a 5 but also a boo and when i enter a non 5 char it prints boo twice. Thanks
I expected it to print hooray once if i entered a 5 and boo if i entered anything else.
Answer: It was the line ending/ carriage return. i turned the line ending off my monitor and its working!


What is the exact, rigorous, full statement of Divergence (Gauss') Theorem in $\mathbb{R}^3$ (without being too complicated)?

Question: The wolfram page http://mathworld.wolfram.com/DivergenceTheorem.html states the formula
$$
\int_{V} \nabla \cdot \mathbf{F} dS = \int_{\partial V} \mathbf{F} \cdot d\mathbf{S}
$$
but it does not speak much of what kind of conditions should be imposed on $\mathbf{F}, V$ and so on.
I think it is enough for $\mathbf{F}$ to be continuously differentiable over $V$ (is it?). But what should be on $V$?
Q1) Is it enough for $V$ to have $\partial V$ as a parametrized (smooth) surface (even piecewise)?
Q2) (It may be a topological one.) But my textbook says a parametrized surface is the image of a continuously differentiable mapping $\mathbf{r} : \mathcal{R} \to \mathbb{R}^3$ where $\mathcal{R}$ is a region (i.e., open, bounded, its boundary having Jordan content 0) in $\mathbb{R}^2$. Then can a sphere have a parametrization?
Q3) What should be the exact imposition on $\mathbf{F}$ including how to specify its domain?
(I hope you'd not talk about manifolds and forms and other complex definitions...)
Answer:
Yes, Divergence theorem is good for "piecewise smooth" domain, namely Lipschitz domain, for example, a regular polyhedron.
The unit $2$-sphere's most common parametrization:
$$
\mathbf{\Phi}: [0,\pi]\times [0,2\pi] \to \mathbb{R}^3\\
(u,v)\mapsto(\sin v \cos u, \sin v \sin u, \cos v)
$$
has two artificial singularities at two poles $(0,0,\pm 1)$ (in $xyz$-coordinates).
Normally we require $\mathbf{F}$ to be smooth on $V$ and continuous up to the boundary. However if we are allowed to go to the realm of weak derivatives, the weak divergence of certain $\mathbf{F}$ has the exact form of Divergence theorem as well. A sufficient requirement for this is that $\nabla \cdot \mathbf{F} \in L^1(V)$, $\mathbf{F}\cdot \mathbf{n}\in L^1(\partial V)$, and $\mathbf{F}\cdot \mathbf{n}$ is continuous on any surface lying within $V$.
Comment: An extra question. Do we have to (or had better) assume $V$ is an open set so that the derivative of $\mathbf{F}$ (including its partial ones) can be properly defined on its domain?
Comment: @julypraise Yes we have to, not just had better.
Comment: @ShuhaoCao Can you point me to a reference in the literature about your third point? I need exactly a proof of that claim!