Get height of div after content change in vanilla javascript

Question: How can I get the new height of div after the content of the div changes? Using JQuery, this is easy, but I am having problems with this in vanilla javascript.
I tried to attach an event listener to the div, but it never gets triggered.
document.getElementById('test').addEventListener('DOMContentLoaded', function() {
      console.log(document.getElementById('cookies-popup').getBoundingClientRect().height);
    });

The content of the div is dynamically changed after retrieved from the server. I tried to get the height with a callback after the http request is done, but the element.offsetHeight or the above command still return the old height when the div has no content inside.
The code that changes the content of the div:
this.staticPageService.getData('test').subscribe(payload => {
    this.divContent = payload.content;
});

Any ideas?
Thanks
Comment: Where is the code who change the height of your div ?
Comment: The code changes the content of the div and consequently the height, it doesn't set a new fixed height.
Comment: Sure but I think the event "DOMContentLoaded" is not appropriate here. You maybe need to get the "height" after the change of the div. Can you post the code who change the div content ?
Comment: Done. this.divContent is the new content of the div which is then connected to the div using angular's [innerHtml].
Answer: After change the content of your div, you can call a function to get the new height of your div.
this.staticPageService.getData('test').subscribe(payload => {
  this.divContent = payload.content;
  getHeightDiv('cookies-popup');
});

function getHeightDiv(idElem) {
  console.log(document.getElementById(idElem).getBoundingClientRect().height);
  return document.getElementById(idElem).getBoundingClientRect().height
}

Update
I don't now what is .divContent but I think you can use element.innerHTML instead.
If you need to use .divContent try to use setTimeout.
this.staticPageService.getData('test').subscribe(payload => {
  this.divContent = payload.content;
  setTimeout(() => getHeightDiv('cookies-popup'), 100);
});

function getHeightDiv(idElem) {
  console.log(document.getElementById(idElem).getBoundingClientRect().height);
  return document.getElementById(idElem).getBoundingClientRect().height
}

Comment: It returns 0 as height.
Answer: Obvious way for addEventListener('DOMContentLoaded', () => {}) using it as
document.addEventListener('DOMContentLoaded', () => {})

not on single element.
Answer: Try using getComputedStyle
let elem = document.getElementById('cookies-popup')
let ht = window.getComputedStyle(elem ).getPropertyValue('height')

Comment: I have tried that and it returns the old height when the div has no content. Maybe I have not placed it in the correct place.


How to have dashed border in Jetpack Compose?

Question: I can easily create a normal border using the Modifier.border() but how to create a dashed border as shown in the image below.
Answer: There isn't a parameter in Modifier.border() to achieve a dashed path.
However you can use a DrawScope to draw a dashed Path using PathEffect.dashPathEffect.
Something like:
val stroke = Stroke(width = 2f,
    pathEffect = PathEffect.dashPathEffect(floatArrayOf(10f, 10f), 0f)
)

You can draw it using the drawBehind modifier:
Box(
    Modifier
        .size(250.dp,60.dp)
        .drawBehind {
            drawRoundRect(color = Color.Red, style = stroke)
        },
    contentAlignment = Alignment.Center
) {
    Text(textAlign = TextAlign.Center,text = "Tap here to introduce yourseft")
}

If you want rounded corner just use the cornerRadius attribute in the drawRoundRect method:
drawRoundRect(color = Color.Red,style = stroke, cornerRadius = CornerRadius(8.dp.toPx()))

If you prefer you can build your custom Modifier with the same code above. Something like:
fun Modifier.dashedBorder(strokeWidth: Dp, color: Color, cornerRadiusDp: Dp) = composed(
    factory = {
        val density = LocalDensity.current
        val strokeWidthPx = density.run { strokeWidth.toPx() }
        val cornerRadiusPx = density.run { cornerRadiusDp.toPx() }

        this.then(
            Modifier.drawWithCache {
                onDrawBehind {
                    val stroke = Stroke(
                        width = strokeWidthPx,
                        pathEffect = PathEffect.dashPathEffect(floatArrayOf(10f, 10f), 0f)
                    )

                    drawRoundRect(
                        color = color,
                        style = stroke,
                        cornerRadius = CornerRadius(cornerRadiusPx)
                    )
                }
            }
        )
    }
)

and then just apply it:
Box(
    Modifier
        .size(250.dp,60.dp)
        .dashedBorder(1.dp, Red, 8.dp),
    contentAlignment = Alignment.Center
) {
    Text(
        text = "Tap here to introduce yourself",
        textAlign = TextAlign.Center,
    )
}

Comment: @CôngHải check the updated answer. Just use the cornerRadius attribute in the drawRoundRect method.
Comment: I don't understand why there is no longer `Modifier.border()`?  Actually I am using android beta 04 and it's not deprecated
Comment: @DavidIbrahim The Modifier.border() exists and it is not deprecated.
Comment: Ohh sorry I misunderstood, I thought you are providing a different solution for this problem because of my code going deprecated lol.
Comment: @GabrieleMariotti if I need add radius what I need to do?
Answer: After some digging in the normal border modifier, I found out that it uses Stroke object which can take a parameter PathEffect that can make it dashed, here is a modified version of the normal border function that takes this parameter.
https://gist.github.com/DavidIbrahim/236dadbccd99c4fd328e53587df35a21
Comment: After the question above I started digging into this and was stuck at this error: "Type mismatch: inferred type is DashPathEffect but PathEffect was expected." This worked before alpha-09 but not on beta. After your answer I got it working. Thank you!
Answer: I wrote this extension for the Modifier you can simply use it or modify it.
fun Modifier.dashedBorder(width: Dp, radius: Dp, color: Color) =
    drawBehind {
        drawIntoCanvas {
            val paint = Paint()
                .apply {
                    strokeWidth = width.toPx()
                    this.color = color
                    style = PaintingStyle.Stroke
                    pathEffect = PathEffect.dashPathEffect(floatArrayOf(10f, 10f), 0f)
        }
        it.drawRoundRect(
            width.toPx(),
            width.toPx(),
            size.width - width.toPx(),
            size.height - width.toPx(),
            radius.toPx(),
            radius.toPx(),
            paint
        )
    }
}

Comment: Using this as is my border had a bit of a gap from the actual edge of the view.
Updating to this fixed that

    it.drawRoundRect(
                (width.toPx() / 2),
                (width.toPx() / 2),
                size.width - (width.toPx() / 2),
                size.height - (width.toPx() / 2),
                radius.toPx(),
                radius.toPx(),
                paint
            )
Answer: @Composable
private fun AddNewFileButton(
    modifier: Modifier = Modifier,
    onClick: () -> Unit,
) {
    val stroke = Stroke(
        width = 4f,
        pathEffect = PathEffect.dashPathEffect(floatArrayOf(10f, 10f), 0f)
    )

    Box(
        modifier = modifier
            .drawBehind {
                drawRoundRect(
                    color = Color.Blue,
                    style = stroke,
                    cornerRadius = CornerRadius(16.dp.toPx())
                )
            }
            .clip(RoundedCornerShape(16.dp))
            .clickable { onClick() },
    ) {
        Row(
            modifier = modifier.padding(16.dp),
            horizontalArrangement = Arrangement.Center
        ) {
            Text(
                text = "Add new item"
            )
        }
    }
}


Abrir Relatório Access com Critério

Question: Sou iniciante em C# e estou abrindo um relatório do Access, porém este relatório possui critério, ou seja, depende de uma numeração para abrir.
Esta numeração possuo em uma textbox e minha dúvida é como passar o valor desta textbox direto para o relatório do Access quando ele for aberto.
Segue o código que utilizo para abrir o relatório:
Microsoft.Office.Interop.Access.Application accessApp = new Microsoft.Office.Interop.Access.Application();         accessApp.OpenCurrentDatabase(@"E:\Desenvolvimento\C#\ReciboFácil\ReciboFácil\bin\Debug\ReciboFacil.mdb");
accessApp.DoCmd.OpenReport("ReciboServiço");
accessApp.CloseCurrentDatabase();
accessApp.DoCmd.Quit();

Answer: Consegui resolver. Alterei o código da seguinte forma:

Microsoft.Office.Interop.Access.Application accessApp = new Microsoft.Office.Interop.Access.Application();         accessApp.OpenCurrentDatabase(@"E:\Desenvolvimento\C#\ReciboFácil\ReciboFácil\bin\Debug\ReciboFacil.mdb");
accessApp.DoCmd.OpenReport("ReciboServiço", AcView.acViewNormal,"", "[codrecibo]=" + codrecibo);
accessApp.CloseCurrentDatabase();
accessApp.DoCmd.Quit();

Comment: Marca como respondida sua questão.

Abrir Relatório Access com Critério


Shiny selectize-dropdown menu open in upward direction

Question: In my shiny dashboard I have a couple of dropdown menus of type selectizeInput. They are located at the bottom of the page, so instead of opening the dropdown menus downward I would like to open them upward.
I did find a solution for the shinyWidgets dropdown menu called pickerInput. The solution here was to add a css tag:
.dropdown-menu{bottom: 100%; top: auto;}

However, this tag isn't working for selectizeInput. Any idea which css I have to add to my script?
Edit (answer by maartenzam with example)
library(shiny)

ui <- fluidPage(
  # selectize style
  tags$head(tags$style(type = "text/css", paste0(".selectize-dropdown {
                                                     bottom: 100% !important;
                                                     top:auto!important;
                                                 }}"))),
  div(style='height:200px'),
  selectizeInput('id', 'test', 1:10, selected = NULL, multiple = FALSE,
                 options = NULL)
)

server <- function(input, output, session) {

}

shinyApp(ui, server)

Comment: By googling, I find there's an option `dropdownDirection` of `selectize.js`. However `options = list(dropdownDirection = "up")` does not work. Perhaps `selectize.js` is not up-to-date in shiny.
Answer: You can do somethink like
.selectize-dropdown {
  top: -200px !important;
}

Comment: That is indeed the tag that I was looking for! I added a minimal reproducible example for future readers in my question. Instead of using `top: -200px`, it is better to use `bottom: 100% !important; top:auto!important;`. Thanks for pointing me in the right direction!
Answer: Thanks for this, very useful!
Leaving this here just in case someone is interested in changing the behaviour only for some selectizeInput's and leave the others default (just as I was):
library(shiny)

ui <- fluidPage(
  tags$head(tags$style(HTML('#upwardId+ div>.selectize-dropdown{bottom: 100% !important; top:auto!important;}'))),
  selectizeInput(inputId = 'downwardId', label='open downward', choices = 1:10, selected = NULL, multiple = FALSE),
  div(HTML("

")),
  selectizeInput(inputId = 'upwardId', label='open upward', choices = 1:10, selected = NULL, multiple = FALSE)
)

server <- function(input, output, session){}

shinyApp(ui, server)

Answer: You can process this in onDropdownOpen event.
$('select[multiple]').selectize({
  onDropdownOpen: function($dropdown) {
    $dropdown.css({
      bottom: '100%',
      top: ''
    }).width(this.$control.outerWidth());
  }
});

In my project I used data-dropdown-direction attribute to specify which element should dropdown in up direction.
In template:
<select multiple data-dropdown-direction="up"></select>

In script:
$('select[multiple]').selectize({
  onDropdownOpen: function($dropdown) {
    if (this.$input.data('dropdownDirection') === 'up') {
      $dropdown.css({
        bottom: '100%',
        top: ''
      }).width(this.$control.outerWidth());
    }
  }
});

Answer: It is possible to do this via Selectize options.
$('#selectize').selectize({
    dropdownDirection: 'up',
    plugins: [
        'dropdown_direction'
    ],
});

Answer: Similar to @ismirsehregal, dropping a helpful variation on this using the new virtualSelectInput function from shinyWidgets:
library(shiny)
library(shinyWidgets)

ui <- fluidPage(
  # modify virtual select css https://github.com/sa-si-dev/virtual-select/blob/master/dist/virtual-select.min.css
  tags$head(tags$style(type = "text/css", paste0(".vscomp-dropbox {
                                                        position: absolute !important;
                                                        bottom: 100% !important;
                                                        top: auto !important;
                                                     }}"))),
  div(style='height:200px'),
  virtualSelectInput('id', 'test', 1:10, selected = NULL, multiple = TRUE,
                     options = NULL)
)

server <- function(input, output, session) {

}

shinyApp(ui, server)


Como passar/receber dados para os componentes através do routing

Question: No meu app-routing-module eu vou reutilizar o mesmo componente para quatro rotas diferentes, porém preciso saber uma forma de diferenciar esses componentes.
Atualmente meu app-routing passa uma propriedade title no data:
  {
    path: 'x',
    loadChildren:'./pages/x/x-x/x-x.module#xModule',
    data: {
      title: 'x'
    }
  }

Como eu recupero esse valor no meu componente para fazer a lógica que preciso?
Answer: constructor(
    private route: ActivatedRoute,
    ) { }

ngOnInit(){
  let lastRoute=this.route;
  while(lastRoute.firstChild){
   lastRoute=firstChild
  }
  this.lastRoute.data.subscribe(data=>console.log(data)
 }
}

Comment: Você está pegando os dados pela URL?
Comment: Não é pela url, é pela propriedade data da rota
Comment: E como fica o código no arquivo routing? Como ele recebe e envia os valores?
Comment: fica o igual da pergunta por exemplo. Os valores são estaticos.
Comment: Fiz algo parecido, vou adicionar minha resposta
Answer: Resolvi da seguinte forma:
No meu app-routing.module:
{
    path: 'xxx',
    loadChildren:'./pages/xxx/xxx-xxx/xxx-xxx.module#xxxModule',
    data: {
      title: 'Título que vou passar'
    }
}

No meu componente:
public rotaAtual: Subscription;

constructor(
public route: ActivatedRoute){}

public getTitulo(): void {
    this.rotaAtual = this.route.data.subscribe((res) => {console.log(res)}
)}

public ngOnDestroy(): void {
    this.rotaAtual.unsubscribe();
}

Answer: A forma que nosso amigo Edward mostrou é com parametros na rota, você também consegue passar query parametros que acredito ser da forma que você deseja obter.
Pelo seu exemplo não ficou claro em qual momento você vai mudar de uma rota para outra, mas vou dar meu exemplo que seria um click em uma tag de ancora, ficaria assim:

  Clique Aqui

Desta forma através do seu this.router você consegue pegar o parâmetro passado. A sua URL irá ficar desta forma: rotautilizada?data="Titulo que passou".
Se quiser passar query parameter pelo ts é só fazer como o exmeplo que nosso amigo deu passando a propriedade queryParams.
Nete artigo explica exatamente o que você está procurando: Angular Router
Answer: Passando pelo data eu não aprendi ainda, eu sei fazer desse jeito:
{ path: 'artigo/edit/:id', component: RhFormArtigoComponent},
{ path: 'artigo/edit/:setor/:id', component: RhFormArtigoComponent},

No componente recebe assim:
import { Router, ActivatedRoute } from "@angular/router";

constructor(
    private route: ActivatedRoute,
    ) { }

this.id_setor = this.route.snapshot.params.setor;
this.id = this.route.snapshot.params.id;

Comment: No caso eu não gostaria de deixar esses dados no link, eu fiz de uma forma um pouco diferente, mas obrigado de qualquer forma
Comment: Qual foi a forma que você fez? @VeroneseComS
Comment: adicionei uma resposta com a forma que eu resolvi


Unable to click on filter on grid in selenium

Question: I am giving the code below where I need to click on the filter icon. please help me out
I have tried using the code as
WebElement element1= driver.findElement(By.xpath("//*[@id=\"grdAutoLaborOps_active_cell\"]/a[1]/span"));
JavascriptExecutor executor = (JavascriptExecutor) driver;
executor.executeScript("arguments[0].click();", element1);

    Status

Comment:

Status
Comment: Welcome to stack! Please put relevant code to your question in the actual question and not the comments. Formatting it as code would also help to get an answer.
Answer: You can try to click  element directly or  class preceding  element.
SPAN:
driver.findElement(By.cssSelector("span[class*='filter']")).click()
It basically means "Find  element which attribute class CONTAINS filter
TH:
Your code is almost correct. The line //*[@id=\"grdAutoLaborOps_active_cell\"] contains backslashes which tells java NOT to parse ". Which leads to following xpath @id=grdAutoLaborOps_active_cell. You should use single quote like this:
//*[@id='grdAutoLaborOps_active_cell']
or use cssSelector
driver.findElement(By.id("grdAutoLaborOps_active_cell")).click();
Comment: Hi Rafal, thanks for the response .........this works  driver.findElement(By.cssSelector("span[class*='filter']")).click(); but it is clicking on the first column filter of the grid  .The filter that i wanted selenuim code to click is on the 10 th column of the grid ...
Comment: Hi @Latha
Can you post some more code to your question? We can figure this out. Add HTML code for 1st and 10th column


Reshape operator

Question: I am reshaping matrices into a single vector. This is the function that I am using. Is there a conventional symbol for this operation (latex command)?
Comment: The APL programming language called this the "ravel" operator and just used a comma.
Answer:
This is typically denoted by the operator $\text{vec}(\cdot)$.
In MATLAB, you could also just use the built-in colon operator (e.g. b = A(:)), which will stack entries of a matrix in column-major order.

Reshape operator


How to view assembly code in codeblocks?

Question: I want to see the assembly version of my C code and I've learned that -S option can show it but I get this error when using that option:
cannot specify -o with -c, -S or -E with multiple files

The only compiler options I have used besides that one are: -m64 and -Wfatal-errors. I use the Codebloc::Blocks IDE and mingw64 compiler. The source file is a43.c and I inserted the option in this place:

Am I doing it right? I prefer using the GUI instead of command line.
Comment: I don't think that's going to work.  Try invoking the compiler directly from the console.
Comment: You want to see it when debugging? or after
Comment: Never mind, I got it working using command line.
Comment: Related: [How to remove "noise" from GCC/clang assembly output?](https://stackoverflow.com/questions/38552116/how-to-remove-noise-from-gcc-clang-assembly-output) for compiler options to give nicer asm (and don't forget `-O3` if you want non-braindead code)
Answer: The compiler options:
-c  # compile only (don't link)
-S  # assemble only (don't compile or link)
-E  # preprocess only (don't assemble, compile or link)

are mutually exclusive. Furthermore, you cannot specify:
-o foo.o    # write output to `foo.o`

if you are specifying multiple input source files.
You are adding -S a43.c to the Other compiler options. That means you
are adding them to the existing compilation options for your project. Those
options already include -c by default, because that's the option to compile,
so adding -S conflicts with -c.
-S does not take any argument, specifically no filename argument. It tells
gcc only to assemble (not compile or link) the input files, whatever they are. So adding:
-S a43.c

as well as adding -S to the existing options, adds an input file,
a43.c, to the project's compiler options. But of course Code::Blocks will add the
name of an input file to the compiler options whenever it compiles one of the
files in the project. So there will be two source filenames in the commandline.
Hence the error:
cannot specify -o with -c, -S or -E with multiple files

If you want to modify your Code::Blocks compilation options so that they will also
produce the assembly listing of each file compiled, then remove
-S a43.c

from Other compiler options and replace it with
-save-temps

Then, whenever gcc compiles a file foo.c in the project, it will save the
intermediate files as:
foo.s   # The assembly
foo.i   # The preprocessed source

These files will be saved in the same directory as foo.c itself; not in the
directory where the object file foo.o is output.
See the documentation of -save-temps in the GCC manual
Answer: If you are on Linux, You can make use of the gcc to compile the code,
and objdump -M intel -D a.out where a.out is your compile code, and the -M intel is to output into intel syntax or remove the -M intel.
But you can also make use of gcc -S code.c -o asm_code in other to see the assembly language output.
Comment: Of course, mingw comes with `objdump` and (I think) the rest of binutils so this general idea is applicable.
Comment: `mingw64` is a windows compiler


Python get list of csv files in public GitHub repository

Question: I am trying to pull some csv files from a public repository using Python. I have the code to process the data after I have the files' URL. Is there some sort of equivalent to ls for GitHub? I'm not seeing anything in GitHub's API, also it seems like it's possible using PyCurl, but then I'll need to parse through the html. Is there any prebuilt way to do this?
Comment: Git trees API? https://developer.github.com/v3/git/trees/
Comment: Does this answer your question? [Get all file names from a Github repo through the Github API](https://stackoverflow.com/questions/25022016/get-all-file-names-from-a-github-repo-through-the-github-api)
Answer: BeautifulSoup (hacky and probably very inefficient) solution:
# Import the required packages:
from bs4 import BeautifulSoup
import requests
import pandas as pd
import re

# Store the url as a string scalar: url => str
url = "https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_daily_reports"

# Issue request: r => requests.models.Response
r = requests.get(url)

# Extract text: html_doc => str
html_doc = r.text

# Parse the HTML: soup => bs4.BeautifulSoup
soup = BeautifulSoup(html_doc)

# Find all 'a' tags (which define hyperlinks): a_tags => bs4.element.ResultSet
a_tags = soup.find_all('a')

# Store a list of urls ending in .csv: urls => list
urls = ['https://raw.githubusercontent.com'+re.sub('/blob', '', link.get('href'))
        for link in a_tags  if '.csv' in link.get('href')]

# Store a list of Data Frame names to be assigned to the list: df_list_names => list
df_list_names = [url.split('.csv')[0].split('/')[url.count('/')] for url in urls]

# Initialise an empty list the same length as the urls list: df_list => list
df_list = [pd.DataFrame([None]) for i in range(len(urls))]

# Store an empty list of dataframes: df_list => list
df_list = [pd.read_csv(url, sep = ',') for url in urls]

# Name the dataframes in the list, coerce to a dictionary: df_dict => dict
df_dict = dict(zip(df_list_names, df_list))


How to update value of dictionary in iOS Swift?

Question: var request: [String: Any] = [

    "Token": "Token",
    "Request": [
        "CityID": "CityID",
        "Filters": [
            "IsRecommendedOnly": "0",
            "IsShowRooms": "0"
        ]
    ]
]
//
print(request)

Console output:
["Token": "Token", "Request": ["CityID": "CityID", "Filters": ["IsRecommendedOnly": "0", "IsShowRooms": "0"]]]

Here I want to update value of "IsShowRooms" key from value "0" to value "1", I was tried different steps but I am unable to do so, can anyone help me out in this?
Answer: You can get the value by typecasting Any to its type and store its value back
if var requestVal = request["Request"] as? [String: Any], var filters = requestVal["Filters"] as? [String: String] {
    filters["IsShowRooms"] = "1"
    requestVal["Filters"] = filters
    request["Request"] = requestVal
}

Output

["Token": "Token", "Request": ["CityID": "CityID", "Filters": ["IsRecommendedOnly": "0", "IsShowRooms": "1"]]]

OR
Instead of storing values in Dictionary create a struct. It will be easier to update its properties
Answer: You can do this following way.(Dictionary in swift is treated as value type)
if var reqObj = request["Request"] as? [String: Any] {

  if var obj = reqObj["Filters"] as? [String: Any] {

    obj["IsShowRooms"] = "1"
    reqObj["Filters"] = obj
  }
  request["Request"] = reqObj
}
print(request)

OUTPUT
["Request": ["CityID": "CityID", "Filters": ["IsShowRooms": "1", "IsRecommendedOnly": "0"]], "Token": "Token"]

Comment: It doesn't change IsShowRooms's value to 1. It adds a new IsShowRooms with value 1
Comment: agree with comment of @RajeshKumar R


Thymeleaf expression objects: #session and ${session....}

Question: There are two ways to obtain a session attribute in Thymeleaf:

${#session.getAttribute('attr')}
${session.attr}

What is the difference between both session "objects"? Is there a situation in which one would work and not the other?
Answer: #session usage only works in Web Context ie. it is a helper to directly access to the javax.servlet.http.HttpSession object associated with the current request. This is clearly stated in documentation. So you are directly accessing the Context object here not the variable that is defined by the thymeleaf.
$session is a shortcut for accessing session attributes. This is not a context object, but it is a map added to the context as a variable, which is added by thymeleaf explicitly. You can find the relevant information here.
Answer: Some of the session related object has been removed in new version of the thymeleaf.
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).


T versus Mann-Whitney U test

Question: I am working on an online learning problem, in which I need to detect whether two independent samples (possibly of different sizes) come from the same underlying distribution. Let us call the samples $S_{1}$ and $S_{2}$, such that $100 \leq |S_{1}|, |S_{2}| \leq 1000$ and both can come from any kind of distributions (i.e., I can make no assumptions on their shape or form).
After going over multiple online resources, I found out that there exist two main tests that can serve my goal:

paired T test: with the assumption that $|S_{1}| = |S_{2}|$ and that both samples come from normal distribution.
Mann-Whitney U test: with no assumptions on the samples' sizes and that both need to come from normal distributions.

Which of the two tests should I use, and what is the accuracy of Mann-Whitney U test if both samples come from normal distributions**(question 1)? Is there a rule of thumb on which test one should pick based on sample size **(question 2)?
Comment: 1. What you have not stated is your hypothesis that is to be tested. What are you trying to find out? 2. Why would you use a paired t-test on independent samples?? What is the justification for this? 3. Also note that there is no requirement whatever in the Mann-Whitney in relation to the normal distribution.
Comment: 1. The distributions can differ in all sorts of ways even with identical medians; if you want to test for differences in distribution, *then test that*. 2. Neither the Mann Whitney nor the  two sample t-test is specifically a test of medians.  3. In relation to my earlier comment 3. then I don't understand what you mean when you wrote "Mann-Whitney U test: with no assumptions on the samples' sizes and that both need to come from normal distributions." in your question
Comment: Could you please explain, what you mean by $100 \leq |S_{1}|, |S_{2}| \leq 1000$ ?
Comment: @Bernhard I mean that each sample size will contain a number of elements between $100$ and a $1000$. For instance, $S_{1}$ might contain $111$ elements and $S_{2}$ $527$ elements.
Comment: @Glen_b the hypothesis I want to test is whether the distributions, from which $S_{1}$ and $S_{2}$ are the same (or at least similar). That can be translated as similarity in medians (or even deviation from median values). Bernhard explained why I don't need to use a paired T-test, so this is a mistake on my end. Turning to comment 3, I am aware of that. Thank you for your comment.
Comment: @Glen_b thank you for the additional comments. To answer your question: when I researched about non-parametric tests, I saw people comment that the samples' sizes can affect the correctness of the statistical test. This is why I added this comment. For my particular use-case, what would you suggest is a sound way to test for differences in distribution?
Comment: 1. I don't follow what you're saying about the size of samples affecting correctness (maybe you can offer a direct quote of what you saw?), nor what it has to do with your statement about normal distributions. 2. A two sample goodness of fit test would be my first thought (perhaps a two sample Kolmogorov-Smirnov, for example), unless you're esepcially looking for sensitivity to particular kinds of difference in distribution, in which case it suggests a hypothesis about that difference.
Answer: Please note, that the non-paired t-test as well as the U-test both tell you, whether $S_1$ or $S_2$ leads to systematically larger or smaller values. They will not detect different distributions, as long if not one produces larger or smaller values.
t.test(rnorm(100), runif(100, -1, +1))

In R this will perform a t-test comparing values from a standard normal distribution with values from a uniform distribution between -1 and +1 and there is no significant result. The same is true for the U test.
As there are no assumptions about the underlying distribution, I would tend towards a U-test, but the t-test is quite robust against deviation from normality when you compare large numbers (and yours are large in this respect). It will however eventually loose it's advantages with respect to statistical power in non-normally distributed data.
So you are basically free to choose between both tests with only a slight and subjective tendency towards the nonparametric test.
As you have independent data, you do not want a paired t-test.
Comment: @Bernhard the two sample t-test has a degree of level-robustness for the reasons you mention (depending on your tolerance for impact on type I error rate). However, it doesn't necessarily have very good *power* in all such situations, and in fact the power may sometimes be fairly poor. Since people care about more than type I error, this may often be an important consideration, particularly if it's large sample sizes and small effect sizes
Comment: @nick.katsip Yes, the Central Limit Theorem allows us to regard the means as normally distributed where the distribution of the data does not matter. In my personal taste, if I may add that, the t-test is preferable if you need a nice effect size measure and thus if you need to do power analysis. Also, it seems to be easier to understand a confidence interval of the difference of means then that of a quasi-median. If you just need a p-value, have high enough numbers and no strong distribution assumptions, go with the rank-sum test/U-test.
Comment: @nick.katsip $S_1$ ist over 100, which is a large number in this respect und $S_2$ will probably also be "large", if you describe it as "less then 1000". If you feel uncomfortable with the whole "large, whatever that is" -thing, then go for nonparametric. Your whole discussion with Glen_b here is of no importance for the U-test.   If you want to stick with t-tests but are unshure about unequal sample sizes and unequal variances you might consider a modification of the t-test known as the Welch-test (or Welch correction of the t-test).
Comment: thank you very much for your answer. Just a clarification: Is the reason that you mention t-test's robustness when comparing large samples due to the Central Limit Theorem?
Comment: @Glen_b I had the power included in my text but probably understated it. Thanks for making this more prominent.
Comment: @Bernhard thank you very much for the advice. I would go for the t-test, but I am skeptical of the following points: (a) what should be the minimal sample size I need to have in order to assume that data are normally distributed (i.e., fairly apply CLT), (b) my samples might not be equal in size and I don't have a robust approach on what to do in this case.
Comment: @nick the *data* are not increasingly normally distributed no matter how large the sample size. The CLT relates to the distribution (in the limit as n goes to infinity) of a standardized sample mean.
Comment: @Glen_b I apologize for wrongly stating my question. I meant that as the sample size goes to infinity, the sample will resemble a normal distribution.
Comment: You're saying the same thing again, and again, no. This would only be true if the *population* were to be normally distributed. This would require an assumption about exactly what you said in your question you were not making assumptions about.  There are two facts you need to be clear on: 1. Under random sampling, the sample cdf approaches the population cdf as sample sizes increase.  2. The distribution of the population doesn't change when you take a larger sample from it. Clearly  then, as n becomes larger, the distribution of the sample will not 'resemble a normal distribution' but ...
Comment: ... instead it will increasingly resemble whatever the distribution of the population is.
Comment: @Bernhard sounds good. I will look into the Welch test as well. Thank you very much for your advice.

T versus Mann-Whitney U test


All the solutions to tile an 8 by 8 square with tetrominoes

Question: I want to fully tile an 8x8 board using the 19 fixed Tetrominoes, allowing repeats of Tetrominoes. For example, this is a valid solution:

Here are the 19 fixed Tetrominoes:

I am looking for a list of all solutions or an algorithm that can find all the solutions.  Or a link to one of the known solvers
Comment: It seems you are allowing repeats, so the tetrominoes being "fixed" is not really relevant. There are several billion solutions, cause an exhaustive search shows that a 4x8 rectangle has $40899$ solutions, giving at least $40899^2$ solutions for the square already.
Comment: If you don't allow any repeats, then the square has $1113112$ solutions, or about $1113112/8=139139$ if rotations/reflections are removed.There are many polyomino solvers out there with which you can find such solutions.
Comment: I used my own, which you can find here: [PolyForm Puzzle Solver](https://www.jaapsch.net/puzzles/polysolver.htm)
Comment: @JaapScherphuis, can you mention the solver that you use to get those answers?
Comment: I think that this is exactly what I was looking for, if you make it an answer I will accept it.
Answer: Since you allow repeats of the tetrominoes it does not really matter whether you consider them fixed or not. You are simply filling the 8x8 square with tetrominoes, regardless of their shape and orientation.
I think there are probably several billion solutions. If you fill exactly half the board, a 4x8 rectangle, then there are $40899$ solutions. Putting any two such solutions together as an 8x8 square we already get $40899^2=1,672,728,201$ solutions for the square. There will be many more solutions without a fault line along the centre. Of course, if you reduce by symmetry, i.e. considering solutions that differ by rotation/reflection the same, then there will be about one eighth as many solutions (slightly more due to solutions with symmetry).
There are many polyomino solvers out there to explore these problems. I use my own, which you can download here: Polyform Puzzle Solver. It is not the fastest, but quite flexible.


Why do we want low autocorrelation for MCMC convergence?

Question: Usually, autocorrelation is one diagnostical tool for judging the convergence of a MCMC trail. Low autocorrelation is desired as this would mean that the parameter space is well explored.
I have a real struggle with this. Assume that we have a lot of data and the highest density interval of the posterior is very narrow. Thus, most of the density would fall onto a small range of parameters. If we derive this from a MCMC chain, this would mean that the parameters of the steps of the chain would not differ much. From my understanding, this would also imply that the autocorrelation is high.
What am I misunderstanding here?
Answer: There's two things going on here that both lead to wanting to low autocorrelation, but from slightly different angles.
First is that if you have some sort of sampler (i.e. Metropolis Hasting, Gibbs sampler, whatever), you would like to have very little autocorrelation in the samples. This can be explained very easily by thinking of the MCMC error: for the posterior mean, for example, the MCMC error will be lower if you have weakly correlated samples than if you have strongly correlated samples. That's the easy explanation, but it's worth thinking about more too. In general, it is easy to see that a sampler that produces weakly correlated samples is preferred over one that produces strongly correlated samples.
Second, which I think is more what you are getting, is using the empirical autocorrelation to determine what "burn-in" to remove. The reasoning behind this is that typically, we don't have great starting points; we often begin far from the mode. Including these starting points in a finite sample can add bias, as we are over representing these start points by beginning there and potentially slowly drifting toward the mode. Different samplers may "drift" faster than others. Once we get close to the mode, we should essentially bouncing around the mode at that point. But here in is the important note: when we are far from the mode, many samplers will have strong 'tug' back toward the mode, as moving toward the mode will likely greatly increase the posterior probability. However, when we are close to the mode, our sampler should be taking random jumps, with much less 'tug' toward the mode (because the samples are much closer). The stronger the tug, the higher the autocorrelation. Thus, if we observe that there is heavy autocorrelation early in our chain, which then levels off after awhile, this is indicative of being very far from the mode early on.
Comment: @ManuHaq: The easiest way to see this is looking at the variance of the sum of two random variables: $V[X + Y] = V[X] + V[Y] + Cov[X, Y]$. Note that the variance is higher if they are positively correlated, the variance of the sum is higher than if they were not. Since we use the average (which is the sum times a constant) of all the MCMC samples, if they are positively correlated, the average will have a higher variance than if they were independent.
Comment: @ManuHaq: you are correct that this is the variance of the mean. But (a) the mean of the chain is unbiased for the true posterior mean (remember that a random mcmc sample is asymptotically equivalent to a sample from the posterior) and (b) we generally use the mean of the chain as our point estimate for the parameter. There's also a closer to the point reason we want uncorrelated samples. Basically, often each hop in a highly correlated chain only moves very little across the posterior, so we run a higher risk of not fully exploring the posterior with the same number of samples.
Comment: I still don't quite get it. I understand everything regarding burn-in etc. But if I am very far in my chain (many iterations passed) and I am converging, then all my samples are very close to each other, if the posterior is very concentrated. Does that not imply strong autocorrelation then?
Comment: No, at least not when the auto-correlation is estimated locally. Consider that if you have a large drift parameter, if you don't remove this effect from your series, your process will have a very high autocorrelation. On the other hand, if your drift parameter is very small, all other things equal, you will have a lower autocorrelation.
Comment: Suppose my posterior is very concentrated on $\theta=0.5$, then my samples drift towards this parameter and don't change much anylonger. Maybe I am really misunderstanding something completely, but the auotocrrelation for such a sequence would be high instead of low.
Comment: The emphasis is about the *local* autocorrelation. If you are random bouncing around $\theta = 0.5$, then how much of your current variation *over a given window in which you are close to 0.5* is explained by your previous value is very little. However, you are much further away from $\theta = 0.5$, how much of your variation is explained by your previous value *over a given window in which you are drifting toward 0.5* is explained by your previous value is very high.
Comment: @CliffAB, could you please elaborate on the reasoning behind this sentence: "_... it is easy to see that a sampler that produces weakly correlated samples is preferred over one that produces strongly correlated samples_"?
Comment: Thanks, @CliffAB. That is a neat way of looking at it. But if I am understanding your comment correctly, it concerns the variance *of the mean*. (Is it the variance if we have, let's say, a moving mean of the chain?). But how does it bias the confidence interval of the estimates (which I see it related to the variance *around* the mean)?

If we reduce the autocorrelation by thinning, isn't it comparable to sampling an AR(1) process (which, to my intuition, shouldn't change the marginal mean/variance of the time series)?


Force save an xml file to xls format in Python

Question: I have this code here, that downloads this fund data in Excel 2004 xml format:
import urllib2
url = 'https://www.ishares.com/us/258100/fund-download.dl'
s = urllib2.urlopen(url)
contents = s.read()
file = open("export.xml", 'w')
file.write(contents)
file.close()

My goal is to, programmatically, convert this file to .xls by which I can then read it into a pandas DataFrame. I am aware I can parse this file using python's xml libraries however, I did notice that if I open the xml file and manually save it with the xls file extension, it can be read by pandas and I get my desired result.
I have also attempted using the following code which renames the file extension, however this method does not "force" save the file and it remains as an underlying xml doc with an xls file ext..
import os
import sys
folder = '~/models'
for filename in os.listdir(folder):
    if filename.startswith('export'):
        infilename = filename
        newname = infilename.replace('newfile.xls', 'f.xls')
        output = os.rename(infilename, newname)

https://www.ishares.com/us/258100/fund-download.dl
Comment: "... if I open the XMl file and manually save it" - with what application? Excel?
If it is Excel and if you do not care about the performance you can do the same conversion that you now perform manually with OLE scripted from python.
Comment: @sophros Yes, manually saving it with Excel. Thanks I'll look into oletools
Answer: With Excel for Windows, consider using Python to COM connect to the Excel object library using the win32com module. Specifically, save the downloaded xml as csv using Excel's Workbooks.OpenXML and SaveAs methods:
import os
import win32com.client as win32
import requests as r
import pandas as pd

cd = os.path.dirname(os.path.abspath(__file__))

url = "http://www.ishares.com/us/258100/fund-download.dl"
xmlfile = os.path.join(cd, 'iSharesDownload.xml')
csvfile = os.path.join(cd, 'iSharesDownload.csv')

# DOWNLOAD FILE
try:
    rqpage = r.get(url)
    with open(xmlfile, 'wb') as f:
        f.write(rqpage.content)
except Exception as e:
    print(e)
finally:
    rqpage = None

# EXCEL COM TO SAVE EXCEL XML AS CSV
if os.path.exists(csvfile):
    os.remove(csvfile)
try:
    excel = win32.gencache.EnsureDispatch('Excel.Application')
    wb = excel.Workbooks.OpenXML(xmlfile)
    wb.SaveAs(csvfile, 6)
    wb.Close(True)
except Exception as e:
    print(e)
finally:
    # RELEASES RESOURCES
    wb = None
    excel = None

# IMPORT CSV INTO PANDAS DATAFRAME
df = pd.read_csv(csvfile, skiprows=8)
print(df.describe())

#        Weight (%)       Price  Coupon (%)     YTM (%)  Yield to Worst (%)    Duration
# count  625.000000  625.000000  625.000000  625.000000          625.000000  625.000000
# mean     0.159888  101.298768    6.500256    5.881168            5.313760    2.128688
# std      0.126833   10.469460    1.932744    4.059226            4.224268    1.283360
# min     -0.110000    0.000000    0.000000    0.000000           -8.030000    0.000000
# 25%      0.090000  100.380000    5.130000    3.430000            3.070000    0.970000
# 50%      0.130000  102.940000    6.380000    4.930000            3.910000    2.240000
# 75%      0.190000  105.000000    7.630000    6.820000            6.070000    3.260000
# max      1.750000  128.750000   12.500000   40.900000           40.900000    5.060000

Comment: I should have specified prior, I am running this script on mac os so can't use the win32com client
Comment: I actually knew that since you cited *Excel 2004* as there is no such Windows version. For future readers this can be helpful. Consider building a macro version of the same and have Python call it command line.
Answer: With Excel for MAC, consider a VBA solution as VBA is the most common language to interface to the Excel object library. Below downloads the iShares xml then saves it as csv for pandas import using OpenXML and SaveAs methods.
Note: this is untested on Mac but hopefully the Microsoft.XMLHTTP object is available.
VBA (save in a macro-enabled workbook)
Option Explicit

Sub DownloadXML()
On Error GoTo ErrHandle
    Dim wb As Workbook
    Dim xmlDoc As Object
    Dim xmlfile As String, csvfile As String

    xmlfile = ActiveWorkbook.Path & "\file.xml"
    csvfile = ActiveWorkbook.Path & "\file.csv"

    Call DownloadFile("https://www.ishares.com/us/258100/fund-download.dl", xmlfile)

    Set wb = Excel.Workbooks.OpenXML(xmlfile)

    wb.SaveAs csvfile, 6
    wb.Close True

ExitHandle:
    Set wb = Nothing
    Set xmlDoc = Nothing
    Exit Sub

ErrHandle:
    MsgBox Err.Number & " - " & Err.Description, vbCritical
    Resume ExitHandle
End Sub

Function DownloadFile(url As String, filePath As String)
    Dim WinHttpReq As Object, oStream As Object

    Set WinHttpReq = CreateObject("Microsoft.XMLHTTP")
    WinHttpReq.Open "GET", url, False
    WinHttpReq.send

    If WinHttpReq.Status = 200 Then
        Set oStream = CreateObject("ADODB.Stream")
        oStream.Open
        oStream.Type = 1
        oStream.Write WinHttpReq.responseBody
        oStream.SaveToFile filePath, 2 ' 1 = no overwrite, 2 = overwrite
        oStream.Close
    End If

    Set WinHttpReq = Nothing
    Set oStream = Nothing
End Function

Python
import pandas as pd

csvfile = "/path/to/file.csv"

# IMPORT CSV INTO PANDAS DATAFRAME
df = pd.read_csv(csvfile, skiprows=8)
print(df.describe())

#        Weight (%)       Price  Coupon (%)     YTM (%)  Yield to Worst (%)    Duration
# count  625.000000  625.000000  625.000000  625.000000          625.000000  625.000000
# mean     0.159888  101.298768    6.500256    5.881168            5.313760    2.128688
# std      0.126833   10.469460    1.932744    4.059226            4.224268    1.283360
# min     -0.110000    0.000000    0.000000    0.000000           -8.030000    0.000000
# 25%      0.090000  100.380000    5.130000    3.430000            3.070000    0.970000
# 50%      0.130000  102.940000    6.380000    4.930000            3.910000    2.240000
# 75%      0.190000  105.000000    7.630000    6.820000            6.070000    3.260000
# max      1.750000  128.750000   12.500000   40.900000           40.900000    5.060000

Answer: I was able to circumvent web–scraping by finding that the site I was working with had developed an api. Then using python's requests module.
url = "https://www.blackrock.com/tools/hackathon/performance
for ticker in tickers:
    params = {'identifiers': ticker ,
              'returnsType':'MONTHLY'}
    request = requests.get(url, params=params)
    json = request.json()


Moving the a view Contextual Filter default value("PHP Code") to a module code

Question: I am really wondered to know how I can move my designed view's Contextual Filter default value which is being provided by my written code as PHP Code, to a module code or a hook.
Is there any possibilities?
Moreover, I am not really interested in exporting view and putting it inside some hooks and ... (as it's again using eval for that PHP Code).
I really like to move the Contextual Filter default value which is being provided by my written code as PHP Code, to a module code or a hook.
Even if you can provide me a way to access&manipulate the designed view inside the Drupal, by a module hook/code, it can be really helpful for me.
Comment: Yes, you'll need to write a custom views filter handler. If you install the advanced help module and look under views, you should find code samples
Comment: Thank you, there was some notes about making a view and ... but I will try to figure out how the PHP filter module works beside the Views module, and write my own code for the View.
Comment: Do you know any ways to manipulate designed View inside the Drupal by a module hook/code? (I'll add it to my post either)
Comment: Yes Views has [loads of hooks](https://api.drupal.org/api/views/views.api.php/group/views_hooks/7) you can use. Exactly which one you'd need and the exact code you need to write will obviously depend on your needs
Answer: This tutorial shows goes through exactly what you're wanting to do http://www.undpaul.de/en/blog/2013/04/04/contextual-content-flag-module-and-custom-views-default-argument-handler
You're definitely going to want to look at the module code in its entirety on his github linked in the article because in the blog post he left out some crucial code (like how to define the views_argument_default_flag handler as class that extends from views_plugin_argument_default.
In my own experience, I needed a custom argument validator instead of a default argument handler, so I followed this tutorial http://www.joetsuihk.com/en/en/custom_views_argument_validator_views_plugin_argument_validate and came up with the following working result.
The need for this custom argument validator arose from needing nested or hierarchical taxonomy term names in the URL instead of the taxonomy term IDs, like /parent-category/child-category/grandchild-category.  I tried to use the contextual filter validation method of converting the term name in the URL into its corresponding term id and replace spaces with dashes, but my category names have special characters like /, &, +, and -, which meant decoding the URL didn't always match the original term name.
So, the following custom views contextual filter (argument) validator correctly parses the term name out of the URL and figures out the corresponding term id by querying the url_alias table.  After querying a particular path once, the result is cached for better performance.  To get these hierarchical paths stored in the url_alias table, I set up a pathauto pattern for my taxonomy vocabulary, like store/[term:parents:join-path]/[term:name]
The info file:
name = Example Module
package = "Example"
description = Example Module custom views argument validator
core = 7.x

files[] = plugins/example_module_plugin_argument_validate_url.inc

The module:
/**
 * @file example_module/example_module.module
 * Implement hook_views_api - load up the custom views include files
 */
function example_module_views_api() {
  return array(
    'api' => 3.0,
    'path' => drupal_get_path('module', 'example_module'),
  );
}

The views include file:
/**
 * @file example_module/example_module.views.inc
 * Defines a custom contextual filter validator
 */
function example_module_views_plugins() {
  return array(
    'argument validator' => array(
      'taxonomy_url_alias' => array(
        'title' => t('Taxonomy URL Alias'),
        'handler' => 'example_module_plugin_argument_validate_url',
        'path' => drupal_get_path('module', 'example_module') . '/plugins',
      ),
    ),
  );
}

The contextual filter handler:
/**
 * @file example_module/plugins/example_module_plugin_argument_validate_url.inc
 * Contains the taxonomy url alias contextual filter validation plugin.
 */

/**
 * Validate whether an argument (contextual filter) is a valid
 * @ingroup views_argument_validate_plugins
 */
class example_module_plugin_argument_validate_url extends views_plugin_argument_validate {

  private $intAddUnixTime = 604800;
  private $error = FALSE;

  function validate_argument($argument) {
    $view = &$this->view;
    $handler = &$this->argument;

    if (!empty($view->args)) {
      //reconstruct the url from the views argument array
      $strPath = implode("/", $view->args);

      //get the taxonomy term id based on the url, not the term name
      $objAlias = $this->getTermId($strPath);

      //split the url_alias's source column up into an array.
      //the source column contains the term id at the end of a path like 'taxonomy/term/123'.
      $arrTermData = explode("/", $objAlias->source);
      $tid = end($arrTermData);
      $term = taxonomy_term_load($tid);

      //manually assign the term id to be the view's argument
      $handler->argument = $term->tid;
      return true;
    }
    else {
      return false;
    }
  }

  /**
   * Get the taxonomy term id from either the url_alias table or from persistent cache
   * @param string $strPath like 'parent-category/child-category/grandchild-category'
   * @return object containing the result of the url_alias table or persistent cache lookup
   */
  private function getTermId($strPath) {
    $objReturn = NULL;
    //attempt to get the taxonomy term id from cached url_alias table result object
    $objReturn = $this->getCache($strPath);

    //if cache is not set, do a select query
    if ($objReturn == NULL) {
      try {
        if(!$this->error) {
          $objReturn = $this->_dbSelect($strPath);
          //cache the response
          $this->setCache($strPath, $objReturn);
        }
        else {
          //log some error because $this->error was true
        }
      }
      catch (Exception $ex) {
        $this->error = TRUE;
        //log an exception
      }
    }

    return $objReturn;
  }

  /**
   * Run a query to look up the taxonomy term id based on the url alias path
   * @param string $strPath url alias like 'parent-category/child-category/grandchild-category'
   * @return object results of the query
   */
  private function _dbSelect($strPath) {
    $result = NULL;
    try {
      //The alias stored in the url_alias table is going to be prepended with 'store/' because of my pathauto taxonomy pattern
      $result = db_select("url_alias", "ua")
        ->fields('ua', array('pid','source','alias'))
        ->condition('alias', 'store/'.$strPath, '=')
        ->condition('source', 'taxonomy/term%', 'LIKE')
        ->range(0,1)
        ->execute()
        ->fetchObject();
    }
    catch(Exception $ex) {
      //log an exception
    }

    //$result is an object like:
    //$result->pid //'12345'
    //$result->source //'taxonomy/term/123'
    //$result->alias //'store/parent-category/child-category/grandchild-category'
    //$result->uid //1
    return $result;
  }

   /**
   * Get an object from cache
   * @param  string $strPath used as part of the cache id
   * 'cache_taxonomy_url_alias' is a custom cache bin
   * @return object data from the persistent cache
   */
  private function getCache($strPath) {
    $objCache = cache_get('example_module_plugin_argument_validate_url' . '::' . 'store/'.$strPath, 'cache_taxonomy_url_alias');
    if (isset($objCache->data)) {
      return $objCache->data;
    }
    else {
      return NULL;
    }
  }

  /**
   * Store an object in cache
   * @param  string $strPath used as part of the cache id
   * @param  object $objData contains the url alias table results
   * @return void
   */
  private function setCache($strPath, $objData) {
    cache_set('example_module_plugin_argument_validate_url' . '::' . 'store/'.$strPath, $objData, 'cache_taxonomy_url_alias', time() + $this->intAddUnixTime);
  }
}

After that I selected my custom contextual filter validator:


Interop.word serviço

Question: Boa tarde, estou utilizando a microsoft.office.interop.word para converter documentos word em pdf em WCF, contudo a interopr simplesmente para funcionar o WCF fica online só a interopr não funciona, aonde é necessário restartar o serviço  de WCF para ele voltar a funcionar.
Alguém tem ideia o que pode ser?
Comment: Mas tens algum erro? "não funciona" o que significa?
Comment: Significa que o serviço fica online mais a interop não funciona a rotina que convert word para pdf atravéns da interop, não retorna erro e quando restar o  wcf volta a funcionar.
Comment: Para ajudar seria interessante que você colocasse o que está tentando fazer e qual mensagem de erro ocorre. Podem ter centenas de motivos.
Comment: E o código de onde vc está utilizando o interop? Melhora essa explicação ai, senão fica difícil né!
Answer: Talvez possa ser um conflito da versão do site, usando Interop você deve rodar sua aplicação como 32bits.  Configure no pool de aplicação do IIS pra executar a aplicação como 32 bits. O servidor deve ter instalado a versão do office que você esta usando o Interop também.
Após usar o Interop, você deve liberar o objeto.
<code>try
{
    System.Runtime.InteropServices.Marshal.ReleaseComObject(msWordDoc);
    msWordDoc = null;
}

catch (Exception exReleaseObject)
{
    msWordDoc = null;
    //   Console.WriteLine(CMSResourceFile.REALESE_FAILED+ exReleaseObject);

}
finally
{
    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();
    GC.WaitForPendingFinalizers();
}
</code>
Talvez essa resposta te ajude.
https://stackoverflow.com/questions/44544295/keeping-one-interop-office-application-alive-until-iis-restart-for-all-users-i
Comment: Então aplicação web chama um serviço WCF que fica responsável pela interop.

Interop.word serviço


Reflexivity of Relations

Question: Explain why the relation
$$T=\{(a,b)\in\mathbb{R}\times\mathbb{R}:a≤b\}$$
is reflexive, but the relation
$$ Q=\{(a,b)\in \mathbb{R}\times\mathbb{R}:a<b\}$$
is not reflexive.
Is $T$ reflexive because $a$ can equal $b$? While in $Q$, $a$ can only be smaller than $b$ and not equal to be reflexive as in $T$.
Comment: Yeah, you got it.
Answer: You've got it precisely right. We know that an equivalence relation $\sim$ on a set $X$ is reflexive if $x \in X$ satisfies $x\sim x$. Is it possible for a real number $x$ to satisfy $x<x?$ No. However, any real number can (and does) satisfy $x\le x$. So, $"\le"$ is a reflexive relation on $\mathbb{R}$, while $"<"$ is not.
Answer: If every element in the set is related to itself, then the relation is Reflexive.
every $a \in \mathbb{R}$ satisfies $a \leq a$. So $T=\{(a,b)\in\mathbb{R}\times\mathbb{R}:a≤b\}$ is reflexive.
No real number satisfies $a < a$. So $Q=\{(a,b)\in \mathbb{R}\times\mathbb{R}:a<b\}$ is not reflexive.

Reflexivity of Relations


Como pegar o retorno deste metodo

Question: Estou utilizando a lib betterpickers, tentei fazer uma classe para pegar sua informação de forma OO, porem o método não tem um retorno, ele é um void, existe uma maneira de pegar esse retorno?
public class Inflate implements CalendarDatePickerDialogFragment.OnDateSetListener{

private static final String FRAG_TAG_DATE_PICKER = "fragment_date_picker_name";

public void piker(FragmentManager supportFragmentManager){
    CalendarDatePickerDialogFragment cdp = new CalendarDatePickerDialogFragment()
            .setOnDateSetListener(this);
    cdp.show(supportFragmentManager, FRAG_TAG_DATE_PICKER);
}

@Override
public void onDateSet(CalendarDatePickerDialogFragment dialog, int year, int monthOfYear, int dayOfMonth) {
   //Esse é o metodo
}
}

Eu o chamo na MainActivity assim:
   Inflate inflate = new Inflate();

   tes.setOnClickListener(new View.OnClickListener() {
        @Override
        public void onClick(View v) {
           inflate.piker(getSupportFragmentManager());
        }
    });

Eu já tentei algo como onDateSet chamar outro método:
@Override
public void onDateSet(CalendarDatePickerDialogFragment dialog, int year, int monthOfYear, int dayOfMonth) {
    getData(year + "/" + monthOfYear + "/" + dayOfMonth);
}

public String getData(String data){
    return data;
}

Porem, como eu pegaria esse retorno se eu estou chamando o piker? Esse é o problema, existe alguma maneira de o fazer? Porque nessa activity tenho 4 lugares onde deve ser definido uma data, se para cada um eu tiver que fazer um método diferente o código fica enorme.
Documentação: https://github.com/code-troopers/android-betterpickers
Comment: Posta a documentação para vermos.
Comment: @RBZ pronto, adicionei
Answer: Finalmente consegui, eu mudei o método e a classe, ficou assim:
public class Inflate{

    private static final String FRAG_TAG_DATE_PICKER = "fragment_date_picker_name";

    public void piker(FragmentManager supportFragmentManager, final TextView textView){
        CalendarDatePickerDialogFragment cdp = new CalendarDatePickerDialogFragment();
                cdp.setOnDateSetListener(new CalendarDatePickerDialogFragment.OnDateSetListener() {
                    @Override
                    public void onDateSet(CalendarDatePickerDialogFragment dialog, int year, int monthOfYear, int dayOfMonth) {
                        String data = year + "/" + monthOfYear + "/" + dayOfMonth;                 //Após setar a data eu chamo o método na MainActivity para passar a data para o TextView
                        MainActivity.setData(data, textView);
                    }
                });

        cdp.show(supportFragmentManager, FRAG_TAG_DATE_PICKER);
    }
}

Eu o chamo assim agora:
Inflate inflate = new Inflate();
tes.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View v) {
             //Agora além do FragmentManager, também passo o TextView que irá receber o valor.

               inflate.piker(getSupportFragmentManager(), tes);
            }
 });

Na MainActivity, este é o método chamado após setar a data, ele recebe a data em String e o TextView que irá receber o valor:
public static void setData(String data, TextView textView) {
       //Então é só setar a data
        textView.setText(data);
}

Como pegar o retorno deste metodo


Copy all files to another location

Question: I have a folder called "employee" and within that folder there are sub-folders with each employee name.
employee
>> amar
>> akbar
>> anthony

Each of these folders contain 1 or 2 files. There are around 50 employees. I want to copy all those files into a new folder "/home/employee_files" without the sub-folders. In other words, all the files should be available when I list "ls /home/employee_files"
Answer: $ cp employee/*/* /home/employee_files/

Answer: find /home/attachments/ -type f -exec cp {} /home/attachments_all/ \;

Beware that if two files have the same name, you'll only end up with one of them.
An alternate approach is to copy files to a name that is a hash of their contents.
#!/bin/bash
cp "$1" /home/attachments_all/`sha1sum "$1" | cut -f 1 -d ' '`

That will give you a unique copy of each file regardless of what it was named before. You can alter the script to provide you an index file.
Answer: find /home/attachments/ -name "*.*" -exec scp {} /home/attachments_all/ \;

Comment: Because, of course, all files have a period in them, but a directory *never* would...

Copy all files to another location


Bounties affecting privileges

Question: If you offer a bounty and the resulting point drop in reputation would put you below a threshold for a certain privilege, do you lose that privilege or does it just delay acquisition of new privileges?
Answer: See this. It is explained pretty clear in the second bullet point.

If your new reputation brings you below the requirement for any
  privileges, you will lose access to those privileges.

Bounties affecting privileges


Tomcat service does not see $JAVA_HOME

Question: I'm trying to setup a Tomcat service with Systemd on CentOS 7.
I've installed Oracle Java 1.8u74 to /usr/java/jdk1.8.0_74 and set the environment variable $JAVA_HOME at boot like so:
# echo "export JAVA_HOME=/usr/java/jdk1.8.0_74" > /etc/profile.d/setenv.sh

When I login to the system I can run echo $JAVA_HOME and see the correct path. I have installed Tomcat and the tomcat.service file has the following:
# Systemd unit file for Tomcat

[Unit]
Description=Apache Tomcat
After=syslog.target network.target

[Service]
Type=forking

Environment=JAVA_HOME=/usr/java/jdk1.8.0_74
Environment=CATALINA_PID=/opt/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/opt/tomcat
Environment=CATALINA_BASE=/opt/tomcat
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'

ExecStart=/opt/tomcat/bin/startup.sh
ExecStop=/opt/tomcat/bin/shutdown.sh

User=tomcat
Group=www

[Install]
WantedBy=multi-user.target

The problem is that if I omit the line that reads Environment=JAVA_HOME=/usr/java/jdk1.8.0_74, Tomcat does not find $JAVA_HOME, but I expect it to find $JAVA_HOME because it is set in /etc/profile.d/setenv.sh.
My questions

Is it not found due to boot sequence (i.e. setenv.sh runs after the Systemd start)?
Did I put setenv.sh in the wrong place?
What's the best way to handle this?
Answer: This is the expected behavior with systemd.
To understand what the environment that systemd services run in, you can refer to man systemd.exec, specifically the section ENVIRONMENT VARIABLES IN SPAWNED PROCESSES. As it says, only very few variables are set, and you have to set anything else yourself.
It just happens that the files in /etc/profile.d are sourced by interactive shells, which is why you can see the variable when you log in. What you have done is exactly what the tutorial recommends, so you can just stick to it. If you are worried that a Java upgrade would break your unit file, you can make a symlink to your java installation, or even source the file at startup:
ExecStart=/bin/sh -c '. /etc/profile.d/setenv.sh; /opt/tomcat/bin/startup.sh'
ExecStop=/bin/sh -c '. /etc/profile.d/setenv.sh; /opt/tomcat/bin/shutdown.sh'

I would personally stick with what you have already.

Tomcat service does not see $JAVA_HOME


Running plugins with NRPE as root: NRPE unable to read output

Question: I have a problem with nagios and NRPE and I can't resolve it.
I've already tried.
# grep "nrpe_user=" /etc/nagios/nrpe.cfg
nrpe_user=nrpe

NRPE is running as the user nrpe. So I added this lines to visudo:
Defaults:nrpe !requiretty
nrpe ALL=(ALL) NOPASSWD: /usr/lib64/nagios/plugins/check_jboss_files

I edited my nagios nrpe.cfg:
command[check_jboss_files]=sudo /usr/lib64/nagios/plugins/check_jboss_files

But if I try to run the command from the Nagios server I get the following:
# /usr/lib64/nagios/plugins/check_nrpe -H 18.104.22.168 -c check_jboss_files
NRPE: Unable to read output

The script was running as expected when launched locally, even when running as user nrpe:
# sudo sudo -s -u nrpe
bash-4.2$ sudo /usr/lib64/nagios/plugins/check_jboss_files
JBOSS Files: 2049
WARNING - JBOSS Files: 2049

Without sudo the script is working on nagios server, but must work with sudo to give good results. What I missed? I've probably tried everything and I will be grateful for any help.
Comment: does NRPE log anything useful to syslog?
Comment: Nothing special @Keith :(

`Sep 19 20:18:08 idontknow nrpe[10286]: Host is asking for command 'check_jboss_files' to be run...`

`Sep 19 20:18:08 idontknow nrpe[10286]: Running command: sudo usr/lib64/nagios/plugins/check_jboss_files`

`Sep 19 20:18:08 idontknow nrpe[10286]: Command completed with return code 3 and output:`

`Sep 19 20:18:08 idontknow nrpe[10286]: Return Code: 3, Output: NRPE: Unable to read output`

`Sep 19 20:18:08 idontknow nrpe[10286]: Connection from  closed.`
Answer: I faced with a very similar situation recently and the solution was to configure selinux to allow sudo access.
Follow this article and you'll find out what to do.
For me this worked on CentOS 7.4:

yum install -y setroubleshoot-server
Search the violation in audit log
Use audit2why
Follow its suggestion

My violation:
$ grep -m 1 type=AVC.*sudo /var/log/audit/audit.log | audit2why
type=AVC msg=audit(1525359549.889:211763): avc:  denied  { getattr } for  pid=127488 comm="check_d" path="/usr/bin/sudo" dev="dm-0" ino=67120340 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:sudo_exec_t:s0 tclass=file
Was caused by:
The boolean nagios_run_sudo was set incorrectly.
Description:
Allow nagios to run sudo

Allow access by executing:
# setsebool -P nagios_run_sudo 1

Running the suggested command solved my problem.


Is a function that maps every connected set to a connected set continuous?

Question: A continuous function maps a connected set to a connected set. Is the converse of this true? That is, is a function that maps every connected set to a connected set necessarily continuous? How about a real-valued function?
Comment: See: http://math.stackexchange.com/questions/252031/continuity-of-function-mapping-connected-set-to-connected-set
Answer: The converse is not true. The function $f : \mathbb{R} \to \mathbb{R}$ defined by $f(x) = \begin{cases}\sin \dfrac{1}{x} & \text{if} \ x \neq 0 \\ 0 & \text{if} \ x = 0\end{cases}$ is discontinuous at $x = 0$, but every connected set is mapped to a connected set.
Answer: No. Let $i: X \to X$ be the identity, where the codomain has the cofinite topology and and the domain has the trivial (indiscrete) topology. If $X$ is infinite then this function maps connected to connected but is not continuous.
Answer: Yet another example - there are lots of topologies (such as the usual topology on the rationals, or the $p$-adics, or the Cantor set) which are nontrivial and yet totally disconnected - e.g., have no nontrivial connected subsets https://en.wikipedia.org/wiki/Totally_disconnected_space. Any function $f$ from such a space will send connected sets to connected sets, since the only connected sets in such a space are single points (fine, or the emptyset :P).
Answer: No. There is a function $f:\mathbb R\to\mathbb R$ which takes all real values in every interval. You can get such a function by constructing a sequence of pairwise disjoint Cantor sets (nonempty closed nowhere dense sets with no isolated points, i.e., homeomorphs of the standard Cantor set) so that every rational interval contains one of them, and then defining a function which maps each of those Cantor sets onto the whole line.
Clearly such a function is everywhere discontinuous, and the image of any connected set is either a single point or the whole line.


How to concat parameter of a function within a variable name

Question: I may have not asked the question correctly but let say I want to use the name of whatever parameter i pass in to my function, to have my variable within that function to be named ;
def myFunc(dfName: DataFrame): Unit = {

  val "{dfName}_concatenated" = dfName

}

so if i passed in myFunc(testDf) the variable inside should be named testDf_concatenated
Comment: Using variables within variable names is not a good practice. Use maps for this purpose.
Comment: Sound like you are trying to write **Python** in **Scala**, thus this seems to be an XY-question. - It would be better if you explain what you want to do, what is the meta-problem you are trying to solve instead on focusing on this specific solution you have in mind.
Comment: I would advice not use variable names as some parameters it would bring you difficulties while refactoring. Why do you want to use it?
Comment: Hello and welcome to SO! Please read the [tour](https://stackoverflow.com/tour), and [How do I ask a good question?](https://stackoverflow.com/help/how-to-ask). Please also read [How to create a Minimal, Reproducible Example](https://stackoverflow.com/help/minimal-reproducible-example)
Comment: What exactly do you expect to find at `"{dfName}_concatenated"`?
Answer: Values and variables have to be named at compile time and this process you're looking for needs to be done at runtime, so you can not do it this way, try using a simple name and the value would be a Map("{whatever}_concatenated" -> ...).
Answer: I am really not sure what is your question, will that work for you?
def myFunc(dfName: String, anotherParam: Int): Unit = {
  val dfNameConcatenated = s"${dfName}_concatenated"
}

Comment: I think he/she wants to name the variable dynamically! like:
    `val $someNameInjectedAtRuntime = "SomeStringOrWhatever"`


onClick RecyclerView Android

Question: Estou seguindo esse tutorial de implementação de TabLayout e ViewPager com RecyclerView:
TabLayout e ViewPager com RecyclerView
Estou usando o referido projeto para testes de como pegar a ação do click no RecyclerView. Para complementar, estou seguindo esse post que trata desse assunto e do mesmo projeto no StackOverFlow:
Post do StackOverFlow
O que eu fiz:

Criei a Interface ClickListener;
Instanciei a interface no adaptador do RecyclerView;
Usei a função de interface implementando o onClickListener no
ViewHolder; e
Implementei a interface no fragment.

Minha interface
public interface ClickListener {
    void itemlistener(int position);
}

RecyclerViewAdapter
public class RecyclerViewAdapter extends RecyclerView.Adapter<TextItemViewHolder> {

String[] items;
ClickListener clicklistener;

public RecyclerViewAdapter(String[] items, ClickListener clicklistener) {
    this.items = items;
    this.clicklistener = clicklistener;
}

@Override
public TextItemViewHolder onCreateViewHolder(ViewGroup parent, int viewType) {
    View view = LayoutInflater.from(parent.getContext()).inflate(R.layout.recycler_view_list_item, parent, false);
    return new TextItemViewHolder(view);
}

@Override
public void onBindViewHolder(TextItemViewHolder holder, int position) {
    holder.bind(items[position]);
}

@Override
public long getItemId(int position) {
    return position;
}

@Override
public int getItemCount() {
    return items.length;
}
}

TextItemViewHolder
public class TextItemViewHolder extends RecyclerView.ViewHolder implements View.OnClickListener{
private TextView textView;

public TextItemViewHolder(View itemView) {
    super(itemView);
    textView = (TextView) itemView.findViewById(R.id.list_item);
    itemView.setOnClickListener(this);
}

@Override
public void onClick(View view) {
    clicklistener.itemlistener(getAdapterPosition());
}

public void bind(String text) {
    textView.setText(text);
}

}

FragmentA
public class FragmentA extends Fragment implements ClickListener{

RecyclerView recyclerView;

@Nullable
@Override
public View onCreateView(LayoutInflater inflater, @Nullable ViewGroup container, @Nullable Bundle savedInstanceState) {
    View rootView = inflater.inflate(
            R.layout.fragment, container, false);
    return rootView;
}

@Override
public void onViewCreated(View view, @Nullable Bundle savedInstanceState) {
    super.onViewCreated(view, savedInstanceState);

    String[] items = getResources().getStringArray(R.array.tab_A);
    RecyclerViewAdapter adapter = new RecyclerViewAdapter(items, this);
    recyclerView = (RecyclerView) view.findViewById(R.id.recycler_view);
    LinearLayoutManager layoutManager = new LinearLayoutManager(getContext());
    recyclerView.setLayoutManager(layoutManager);
    recyclerView.setAdapter(adapter);
}

@Override
public void itemlistener(int position) {
    Toast.makeText(getContext(), "RecyclerView position is: " + position, Toast.LENGTH_SHORT).show();
}
}

Não consegue achar a referência

Porque está ocorrendo esse erro na implementação para o clique no RecyclerView?
Comment: Relacionado: [Qual é a abordagem certa para pegar o click/posicao no RecyclerView?](https://pt.stackoverflow.com/q/159830/2541)
Answer: Não acha porque você não tem nenhum atributo do tipo ClickListener declarado na classe TextItemViewHolder.
Duas possíveis soluções:

Torne a classe TextItemViewHolder uma inner class de RecyclerViewAdapter.
TextItemViewHolder usará o atributo declarado na RecyclerViewAdapter.
Declare o atributo na classe TextItemViewHolder e passe o listener quando a instanciar.
Comment: Ramaral , obrigado pelo vasto conteúdo disponibilizado, me ajudou muito. Vou me aprofundar mais nesses conteúdos.

onClick RecyclerView Android


Selenium2 phpunit: How to run tests on Opera browser (new one like chrome)

Question: My tests working normally on other browsers (Safari, Chrome, FF ...) but I cant do it on Opera. Im using grid, this is my node:
java -jar C:\Users\netr\Desktop\selenium-server-standalone-2.37.0.jar
     -role node -port 5222 -hub http://ip:4444/grid/register -maxSession 5
     -browser "browserName=opera,maxInstances=3,platform=WINDOWS,
               opera_binary=C:\Program Files (x86)\Opera\launcher.exe"

When I'm staring my cases the Opera browser opens and loaded this URL http://127.0.0.1:12486
for every case the port is different. Then there is the error:

Could not start Opera: launcher unable to start binary

I tried to change path to the binary (opera.exe or to launcher.exe) but still I cant to open my URL from the cases.
My Versions:

Opera version is 17.0.1241.53 - Opera is up to date
PHPUnit 3.7.24 by Sebastian Bergmann.
PHPUnit_Selenium 1.3.1 (stable)
Answer: There is a note on OperaDriver info page http://code.google.com/p/selenium/wiki/OperaDriver:

Current implementation of OperaDriver supports only 12.x and older versions of Opera browser (based on Presto engine) and does not support newer versions (based on Blink engine).

So the only solution seems to be waiting for ChromeDriver developers to fix this issue or use the 12.x version of Opera.


What's the easiest way to get a list from a Flowable in java?

Question: My database returns a Flowable like so:
Flowable<List<Users>> users = MyActivity.myDatabase.myDAO().getUsers();

I would like to quickly and easily convert it to a list so I can count it, iterate through it and perhaps pass it as an ArrayList. What's the best way to do that using Java?
Answer: You will have to use a blocking call to actually retrieve the values. However, it's not clear from the question whether you are expecting a single list of users or a list of user lists.
If you are expecting a single List<Users>, you can do this:
List<Users> usersList = users.blockingFirst();

If you are expecting a list of user lists, you would need to do something like this:
List<List<Users>> userLists = users.toList().blockingGet();

In either case, if you specifically need an ArrayList, you will have to construct one and copy over the elements, as there is no telling exactly what kind of List implementation the above codes return. (You can just do, for instance, new ArrayList<>(users.blockingFirst()) in the first case and something similar in the second.)


Python cv2 not matching template with main image

Question: I have my main image:

And my template:

However, the cv2 code that i am using is not generating the rectangle to show that there is a match, but I am also receiving 0 errors.
Here is the code:
# Read the main image
img_rgb = cv2.imread('main.png')

# Convert it to grayscale
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)

# Read the template
template = cv2.imread('temp.png',0)
template = np.array(template[:,::-1])

# Store width and height of template in w and h
w, h = template.shape[::-1]

# Perform match operations.
res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)

# Specify a threshold
threshold = 0.8

# Store the coordinates of matched area in a numpy array
loc = np.where( res >= threshold)

# Draw a rectangle around the matched region.
for pt in zip(*loc[::-1]):
    cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0,255,255), 2)

# Show the final image with the matched area.
cv2.imshow('Detected',img_rgb)
cv2.waitKey()
cv2.destroyAllWindows()

My initial thoughts are that the template image is too large to match with the main, however this is my first go with cv2 and I am unsure how to fix it.
Answer: The template matching code I have always matches the dimensions of the template to the dimensions of the image. In your case, where you've already converted the source images to grayscale, change
template = np.array(template[:,::-1])
w, h = template.shape[::-1]

(which might not be doing what you intend anyway) to
template = cv2.cvtColor(template, cv2.COLOR_BGR2GRAY)
w, h = template.shape


How do I filter based on if two column values "straddle" any of the values in another table?

Question: An example to illustrate what I'm trying to do:
Table A

Index
FirstValue
SecondValue

1
340
700

2
20
250

3
60
150

4
180
540

5
590
1000

Table B

StraddleValue

190

830

300

Desired output:

Index
FirstValue
SecondValue

1
340
700

3
60
150

I want to filter out any rows where, for any StraddleValue, FirstValue < StraddleValue AND SecondValue > StraddleValue.
I'm not sure how to do this without hardcoding the StraddleValues..which obviously isn't scalable.
Thanks.
Comment: confusing.  which stradleValue do you want the results for? why desired row 3? which one makes that a result?
Comment: If ANY of the StraddleValues fall in between the FirstValue and SecondValue, I want to exclude that row from my result. Row 3 is desired because non of the StraddleValues fall in between 60 and 150.
Answer: You can use not exists:
select a.*
from a
where not exists (select 1
                  from b
                  where a.FirstValue  b.StraddleValue
                 );


How do I stop a simple DDOS attack?

Question: There is a hacker who is mad at us, and has started a DDOS attack against us. I think he is a script-kiddie, using tools like DarkComet; the attack is probably simple.
I host a server on my PC. The server is on my public IP. I am using a router. The server does have a port forwarded for people to connect. There is a proxy and stuff, but I dont know how to set it up
The server is popular, and I do not want to shut it down.
Is there a way for me to stop the attack, or even just delay it?
Comment: @DragonLord, it's also about dealing with DDOS, which isn't necessarily the same as maintenance of a website. And the poster didn't state that the sever is a web server; it could be another type of server.
Comment: Do you have any idea how many source IP addresses he is using? If it's a comparatively small number or from an identifable IP address range, you could block all access to the server from those IP addresses. If it's thousands and you have no way to limit based on IP address, do you have any details about the attack, such as the type of traffic being used for the [DDOS](https://en.wikipedia.org/wiki/Denial-of-service_attack#Distributed_attack)?  Is there some way to distinguish the illegitimate traffic from legitimate traffic? What operating system is your server using?
Comment: My server is using windows 8.1 hacker is using windows and kali linux he is ddosing with all types of ip he use proxy and i think 5 vpn to hide his ip. And all of his hacker friends are helping him too soo that would be like 15 attacks. 15 different ips.
Comment: possible duplicate of [Does tcpdump bypass iptables?](http://superuser.com/questions/925286/does-tcpdump-bypass-iptables)
Comment: No, it is not a duplicate of that.
Answer: The best way to deal with a DDoS is to start as far away from your connection as possible; you want to cut off such an attack as close to the source as possible.
First, find out the IP addresses being used, whois them, and talk to the ISP on the far end. They might not be able to help if they're in a foreign country, or spread all over the world, but if it's an isolated region, you might be in luck.
If that is a dead-end, you can ask your ISP to provide a black hole for a range of IPs. Most consumer-grade accounts won't have this option, but it doesn't hurt to ask. Unless, of course, you're violating the TOU (Terms of Use) for that provider by running a server at all. You'd be admitting to using their connection in a way they don't like. They could decide to shut you down.
If you have a dynamic IP address, you can try pulling out the battery and power cord of your modem/router (the ISP-connected one, not any internal one you may have), and let it sit a few minutes. Hopefully, you'll renew with a new IP address and you can just update your DNS. It'll take a while for the DDoS to realize you've moved, although your legitimate customers will have the same problem. It will sort itself out over a few minutes/hours, but you might be better off in the end.
If you can't get down to getting ISP support or changing your IP address, you'll need a firewall. Go to your local store and buy a nice hardware firewall if you can. One with drop rules specific to traffic types/IP addresses/etc. It'll cost you dearly, but you can configure the firewall to drop all suspicious traffic. The benefit here is that most connections have a much smaller upstream cap than a downstream cap, so by dropping packets, you save your upstream bandwidth (going to your visitors) for actual page visits and not pings, naks, 404s, and whatever else might be going on.
You can try a software firewall but it has to be a good one. Windows Firewall just isn't going to cut it here. You need something like Linux, where you can configure drop rules on the IPs that are bombarding you. You'll still be using a lot of downstream bandwidth to your network, but your server will be able to take a breather while your other system takes the heat.
You might also just set up another virtual server on your server, with a single file that redirects (302) from the default port 80 or 443 to a non-standard port, and set up your services there. Normal browsers will get redirected to the new service without too much hassle on next page load, while the bots will probably continue hammering away on the default port. A simple HTTP redirect costs a lot less bandwidth than full-blown page loads, so it might reduce the effects of the attack. It really depends on how intelligent the DDoS attack scripts are. They might simply harmlessly bounce of the 302, which means they'd have to massively scale up their attack.
Finally, if everything else fails, just turn off your server for 10 minutes and see what happens. The DDoS scripts might get bored and wander off (not likely, but it's worth a shot). If you are hosting your own server on a consumer-grade connection, by the way, it's time to upgrade. Consumer-grade packages are not suitable for hosting, because they have very small upstreams (relatively) and little to no support included for DDoS, DNS, mail servers, and other usual problems.
Answer: Fifteen IP addresses is a relatively small number. You could block those using firewall software or hardware. If he is using a VPN or proxy server service that makes it look like the source IP is from address space belonging to an entity in Belgium and you don't expect any legitimate traffic to come from there, you could block the entire address range allocated to that entity at least for a period of time. There's always a risk that you might block some legitimate traffic as well, but that risk may be outweighed by the need to make your server accessible to other legitimate users.
E.g., suppose I see 188.8.131.52 as an attacking IP address, which is actually an address from which I've seen someone try to break into one of my own servers by attempting to guess a password. I could first go to the American Registry for Internet Numbers (ARIN) website at http://arin.net. In the "Search whois" field, I'd put 188.8.131.52, which would reveal that the address range 126.96.36.199 - 18.104.22.168 is assigned by another Regional Internet Registry (RIR), in this case the RIR is the Asia Pacific Network Information Centre (APNIC), the organization that assigns IP address blocks for that region. So then I could go to the APNIC site at http://www.apnic.net and put 126.96.36.199 in the "Whois search" field there. That tells me that the 188.8.131.52 - 220.127.116.11 block is assigned to China Telecom. If I don't expect any legitimate traffic to the server from China, I can just block the entire address range 220.127.116.11 - 184.108.40.206, so if the attacker comes from 18.104.22.168, instead of 220.127.116.11, he still is blocked. That technique may not work for you depending on the VPN service provider or proxy server service he is using, but it is something you could consider, if you know, for instance, that all of your legitimate traffic is from Canada. Even then it might become like a whack-a-mole game for awhile depending on how global a network the proxy/VPN service provider may have, if the attackers can switch to a VPN or proxy server operated by the provider in another country and realize quickly that they are being blocked if they use a server in a particular country, but if they are script kiddies they may not realize that.
Alternatively, if you are operating a website, but the DDOS traffic is some other type of traffic, i.e., not HTTP/HTTPS traffic to port 80 or 443, then you might be able to block that traffic prior to your server at a firewall you manage, if there is clearly no legitimate need for any external system to communicate with your server on the ports or using the protocols that the attackers are using.
I'd also suggest looking up what organization/company has been assigned IP addresses being used by the attackers; you can use the technique I mentioned above or just go to a website that provides whois services. E.g., you could use the Whois Lookup. It is possible that the contact information you find there won't identify the VPN or proxy server service provider, since the organization assigned the address space by the RIR may have doled out smaller chunks of the address space to others, such as a company providing the VPN or proxy server service, but you could attempt to contact the organization and notify them of the issue. If you can identify the proxy server/VPN provider and can find contact information from their website, a legitimate provider should look with disfavor on any customer using their service for such attacks. Many will state in their terms of service that using their service for such nefarious activities is a violation of their terms of service subjecting the malefactor to an immediate termination of his account and loss of any funds he has already spent for the service.
If he is using the DarkComet RAT to control PCs belonging to individual users, looking up the IP addresses through a whois search should yield the ISP. Contacting the ISP may result in the ISP blocking all network access from the infected system and would, hopefully, result in the user learning someone else can remotely control his or her system and taking measures to have the system disinfected.

How do I stop a simple DDOS attack?


Finding a number knowing the modular multiplicative inverse

Question: Let a = 44 and m = 67
The modular multiplicative inverse of 'a' and 'm' is 32
i = inverseModFunction( a, m )
i = inverseModFunction( 44, 67 )
i = 32

It can be calculated here:
https://www.xarg.org/tools/modular-inverse-calculator/
Is there a way to find 'a' knowing the inverse modulo 'i' and 'm'?
i = 32
m = 67

a = function( i , m )
a = function( 32 , 67 )
a = ?

Comment: If $i$ is inverse to $a$, then $a$ is inverse to $i$. (It seems that this is what you're asking about.) Just try plugging 32 and 67 into the calculator you linked.
Comment: https://en.m.wikipedia.org/wiki/Extended_Euclidean_algorithm is what you are looking for.
Comment: \begin{eqnarray*}
ai \equiv 1 \pmod{m}
\end{eqnarray*}
...
Comment: https://math.stackexchange.com/questions/3447954/by-knowing-1-1-1-mod-p-for-a-prime-p-why-can-we-obtain-multiplicative/3456525#3456525
Comment: **Hint** $ $ just like $\,-(-a) \equiv a\,$ we have $\color{#c00}{(a^{-1})^{-1}}\equiv a,\,$ by $\ i \equiv a^{-1}\!\!\iff ia \equiv 1\iff \color{#c00}{i^{-1}}\equiv a\ \ $
Answer: Modular inverse relation is symmetric, so if i is the modular inverse of a, then a is the modular inverse of i and can be found by inverseModFunction(i,m).


Problem with repeat sub-node to transform a XML to a CSV with XSL

Question: I'm working on a transformation of a XML file to  a CSV file.
See a piece of the XML :
<bota_form_v_20200610 xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:orx="http://openrosa.org/xforms" xmlns:odk="http://www.opendatakit.org/xforms" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:xsd="http://www.w3.org/2001/XMLSchema" id="fm_bota_2020" version="2020061000">
<start>2021-04-20T09:27:24.295+02:00</start>
<end>2021-04-20T10:18:51.038+02:00</end>
<deviceid>collect:JZUeuhzIlUe38qIl</deviceid>
<parcelle>
<identifier>Ab</identifier>
<sample_date>2021-04-20</sample_date>
<proto>farmland</proto>
<yes_no_parcelle_hza>no</yes_no_parcelle_hza>
<md_parcelle>18710002</md_parcelle>
<md_carre/>
<md_hauteur_vegetation>25</md_hauteur_vegetation>
<search_text_culture>Blé</search_text_culture>
<md_culture>Blé</md_culture>
<ocs>41</ocs>
</parcelle>
<quadrat_repeat>
<yes_no_md_numero_waypoint>no</yes_no_md_numero_waypoint>
<nom_gps>T12</nom_gps>
<md_numero_waypoint>264</md_numero_waypoint>
<md_sampling>in</md_sampling>
<point_sampling>in1</point_sampling>
<type_bordure>haie</type_bordure>
<zone>C1-H0-N0</zone>
<md_densite>12</md_densite>
<sous_quadrat_repeat_count>4</sous_quadrat_repeat_count>
<sous_quadrat_repeat>
<md_subquadrat>a</md_subquadrat>
<yes_no_diff_esp_plt>yes</yes_no_diff_esp_plt>
<esp_plt_rpt>
<search_text_plante_latin_rpt>Vero</search_text_plante_latin_rpt>
<md_plante_latin_rpt>Veronica-hederifolia</md_plante_latin_rpt>
<md_code_bayer_rpt>VERHE</md_code_bayer_rpt>
<md_abondance>2</md_abondance>
<md_stade_pheno>v</md_stade_pheno>
</esp_plt_rpt>

but with many 'quadrat_repeat' node which has each one many 'sous_quadrat_repeat' node and each of this node has some 'esp_plt_rpt' node.
I need one line for each 'esp_plt_rpt' sub node. And each of this line need to have all the attribute of their father (attribute of 'sous_quadrat_repeat', of 'quadra_repeat' and of parcelle"). like this :
Attr_Parc1    Attr_Parc2    Quadra_repeat1 ...   quadra_repeatN ... sous_quadrat_repeatN

And here my XSL (1.0) :
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="text" encoding="UTF-8" omit-xml-declaration="yes"/>
  <xsl:strip-space elements="*"/>

  <xsl:variable name="sep" select="';'" />
  <xsl:variable name="nl" select="'
'" />

  <xsl:template match="bota_form_v_20200610">

<xsl:apply-templates select="quadrat_repeat"/>
 </xsl:template>

  <xsl:template match="esp_plt_rpt">
    <xsl:value-of select="md_abondance" />
    <xsl:value-of select="$nl" />
  </xsl:template>

  <xsl:template match="sous_quadrat_repeat">
    <xsl:variable name="tmp">
      <xsl:value-of select="ancestor::bota_form_v_20200610/@id" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="ancestor::bota_form_v_20200610/quadrat_repeat/nom_gps" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="ancestor::bota_form_v_20200610/quadrat_repeat/md_sampling" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="ancestor::bota_form_v_20200610/quadrat_repeat/zone" />
      <xsl:value-of select="$sep" />

      <xsl:value-of select="ancestor::bota_form_v_20200610/quadrat_repeat/md_densite"/>
      <xsl:value-of select="$sep" />
    </xsl:variable>

    <xsl:for-each select="ancestor::bota_form_v_20200610/quadrat_repeat/sous_quadrat_repeat/esp_plt_rpt">
      <xsl:copy-of select="$tmp"/>

      <xsl:value-of select="ancestor::bota_form_v_20200610/quadrat_repeat/sous_quadrat_repeat/md_subquadrat"/>
      <xsl:value-of select="$sep" />
      <xsl:value-of select="md_plante_latin_rpt" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="md_stade_pheno" />
      <xsl:value-of select="$sep" />

      <xsl:apply-templates select="." />
    </xsl:for-each>
  </xsl:template>
</xsl:stylesheet>

This code don't take a new 'sous_quadrat_repeat' when he go to a new esp_pt_rpt.
So all my line have the attribute of the first sous_quadrat_repeat node.
Answer: Your xslt was not far off. Good approach to store that repeatable data into a variable. I adjusted it a little to this:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="text" encoding="UTF-8" omit-xml-declaration="yes"/>
  <xsl:strip-space elements="*"/>

  <xsl:variable name="sep" select="';'" />
  <xsl:variable name="nl" select="'
'" />

  <xsl:template match="bota_form_v_20200610">
    <xsl:apply-templates select="quadrat_repeat/sous_quadrat_repeat"/>
  </xsl:template>

  <xsl:template match="sous_quadrat_repeat">
    <xsl:variable name="tmp">
      <xsl:value-of select="ancestor::bota_form_v_20200610/@id" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="parent::quadrat_repeat/nom_gps" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="parent::quadrat_repeat/md_sampling" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="parent::quadrat_repeat/zone" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="parent::quadrat_repeat/md_densite"/>
      <xsl:value-of select="$sep" />
      <xsl:value-of select="md_subquadrat"/>
      <xsl:value-of select="$sep" />
    </xsl:variable>

    <xsl:for-each select="esp_plt_rpt">
      <xsl:value-of select="$tmp"/>
      <xsl:value-of select="md_plante_latin_rpt" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="md_stade_pheno" />
      <xsl:value-of select="$sep" />
      <xsl:value-of select="md_abondance" />
      <xsl:value-of select="$nl" />
    </xsl:for-each>
  </xsl:template>
</xsl:stylesheet>