Cant find Java VM to add in Eclipse

Question: I am following a tutorial to learn Java and it uses Eclipse. When starting a new Java project in the Eclipse IDE, after adding the project name the tutorial goes to Configure JREs->Add...->Standard VM and then browses to a folder which seems to be the VM home. However, in the tutorial when the browse window opens, it already opens in a directory which contains all VM folders, named jdk-14, jdk-15, etc.. I cannot by any means find any similar JVM folder in my computer, although I'm pretty sure I have Java installed.
I'm running in a Debian 11. if I run

java -version

I get

java version "17.0.2" 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86) Java  HotSpot(TM) 64-Bit Server VM (build
17.0.2+8-LTS-86, mixed mode,  sharing)

Any suggestions on where to look for folders named like jdk-xxx.xx to add to Eclipse?
Comment: Does this answer your question? [Find out what JVM Eclipse is running on](https://stackoverflow.com/questions/557169/find-out-what-jvm-eclipse-is-running-on)
Comment: What says `which java`? Please note, current Eclipse Java IDEs are shipped with an embedded Java 17 (or when using the Eclipse Installer you can choose to install it with or without an embedded Java).
Comment: which java returns /usr/bin/java while readlink -f $(which java) returns /usr/lib/jvm/jdk-oracle/bin/java
i also noted that trying to start a project with eclipse pops the error "Errors occurred during the build.
Errors running builder 'Java Builder' on project 'myfirstproject'.
For input string: "H" under radix 16"
Comment: @pringi it helps: eclipse.jn points to /usr/bin/java and if i go to /usr/bin and type java -version it tells me its the 17.0.2
However, when i try to start a new project in eclipse its interface seems to point at JavaSE-13
Answer: You can find where your Java installation lives with this command:
$ readlink -f $(which java)

On my Fedora Linux workstation it results in:
/usr/lib/jvm/java-11-openjdk-184.108.40.206.8-2.fc35.x86_64/bin/java

So the directory is /usr/lib/jvm/java-11-openjdk-188.8.131.52.8-2.fc35.x86_64
However you will notice that directory has a very specific version in the path. This is undesirable because it will change when java is updated and might break your Eclipse configuration.
But good news: When I look in my /usr/lib/jvm directory there is a versionless symlink on my system which points (eventually) to the same place: /usr/lib/jvm/java-11-openjdk
I would use the versionless symlink because then my Eclipse configuration will not break when java gets updated. Hopefully it is similar on Debian.
Comment: It kinda worked: readlink -f $(which java) showed me where is java and there is a default-java symlink. However, inside the symlink there is no folders named "jdk-XX.Y.Z" as in the tutorial. There is only "bin", "conf", etc.
Does that mean i have no jdk installed?
Answer: For Linux try command: readlink -f $(which java)
If you are in Windows try these steps
Open command prompt and execute  c:\> for %i in (java.exe) do @echo.   %~$PATH:i
This should print the path of your jdk.
Alternatively if java path is defined try where java in command prompt.
Comment: User explicitly states that they are on Linux.
Comment: I missed that , for Linux readlink -f $(which java) should work

Cant find Java VM to add in Eclipse


PHP - check if csv line is empty

Question: In my code I'm reading csv file with fgetcsv function
while($line = fgetcsv($handle, 4096)){
    .....
}

How can check when the $line is empty, I mean $line = ',,,,,,,,' ?
note: I'm not always know the number of columns.
Comment: `if (trim($line, ',') == '') {`?
Comment: I've linked a very similar question that was asked a few days ago, the accepted answer will easily handle your problem with some edits
Comment: You can simply check if(!empty($line[0])){ //your logic }
$line[0] for first column, $line[1] for second column same for other
Answer: One way to do it would be:
if (str_replace(array(',', ' '), '', $line) != '') {
  // do something
}

Basically, it will compare the line after you remove all comma's and spaces from the string.
Answer: You can take the contents of file in an array and check which index has empty value.
like this -
$handle = fopen("file.csv","r");
$data = fgetcsv($handle,",");
while($data = fgetcsv($handle))
{
$array = explode(",",$data[0]);
print_r($array);
}
fclose($handle);

PHP - check if csv line is empty


Data Fusion/Statistical Matching: Match Keys & Algorithms?

Question: I need to fuse multiple surveys using statistical matching, and would like to comply with any best practices.  I have studied multiple documents but -- despite the plethora of information -- it's unclear what the best practices are. Specifically...
First, what are the best match keys? I'm considering using region, gender and age for the critical variables (i.e.: MUST match) and persons in household, children, language and income for the non-critical ones.
Second, what's the best algorithm? I'm considering k-Nearest Neighbor.
Third, what should I do about mis-matching match keys? For instance, income brackets may differ in the two surveys.
Finally, what free tools exist?  I have Weka and R (and will look into StatMatch).  An ideal would be a tool that will produce a fused CSV from two (or more) CSVs and a set of match keys.
Also, I'm fine being pointed to a book. I'm a developer (not a statistician), so I'll happily read something geared to my level.
Answer: So happy to see someone is facing the same problems as me!
I agree that a pletora of documentation exists about statistical matching, but still it is unclear what solution is the best. I think the approach depends a lot on the field of study, that is: official statistics, market research ... The references, tools & approaches  are quite different as long as I know. What is exactly your field?
You ask mainly  for three things: (i) best practices, (ii) tools, (iii) books- references. I'll try to respond to those three in order:
Best practices
Independently of the field where you want to apply statistical matching, a unique best approach ("no free lunch for statistical mathing ") does not exist. I think the solution will depend highly on what type of problem you have, I mean:  if you have many common information or variables between the files but few specific variables (columns that you must fill); or if you have few common variables but a lot of specific ones ... I think the most useful way to think about statistical matching is that the final fused file (that is, a synthetic complete file that you will obtain, containing all the information for the original files) should correctly reflect the true relations on the underlying population – be it composed of individuals, houses, schools …  But the point is that, since you don't usually have any record with values for all the variables in the files you want to match, you can never check if your algorithm is reproducing the 'true' values (i.e. techniques like crossvalidation in machine learning algorithms are not applicable here). That means that the validity is much, much harder to assess.
Bayesian techniques can overcome this problem because they are ideal to deal with uncertainty, and I think they are the most promising approach, but I haven`t found any clear guide to proceed this way yet.  For the beginner (just as I am), I think the best way to start will be to try to use as much common information between the files in order to reduce uncertainty in the fused file. This is because the more the common variables (say Z) you use to determine the specific ones (say X, Y in the case of two files), the more you will reduce uncertainty. So you must convince yourself that a big part of the available time in your project must be spent in choosing good common variables. That means you need to compare distributions between the files (variables must measure the same thing in both files), and probably you will need to aggregate categories in order to make variables comparable. Also, you will probably need to distinguish between strata variables (matching is performed within those strata) and variables that will be used to search  for similar recipient/donor pairs. Also, for numerical variables, consider taking z-scores to make distributions comparable. And put a lot of imagination trying to construct new variables – for example using factor scores (constructed by factorial analyses).
Then you should use a good (robust) algorithm, and finally validate the fused file at least to make sure that the observed distributions in the original files (univariate-bivariate or higher) are correctly reproduced in the fused file (this is easy to do, and must be a minimum requirement). Ideally, you should also check that the fused file reflects correctly the true correlations in the populations (but this is much harder to do). For this purpose you could:

Perform a simulation study: this means trying to simulate your statistical matching situation splitting one of the files randomly in two files, deleting some variables, applying your algorithm and checking if you obtain good results. This is a good thing to do in order to understand your algorithm better, and to ensure it is robust, but it doesn't make sure the fused file reflects the true correlations, because you don't really have that information anywhere.
Try to use algorithms in a bayesian approach. That is: as long as you can, don't use k-nearest neighbour or similars (frequentist approach) because although they can be good at reproducing marginal distributions, they are too limited to assess validity.

Tools
I only know about tools in R:

StatMatch package for official statistics developed by ISTAT, based on frequentist techniques, but an interesting one because of its documentation and good hot-deck procedures (you mention this). It deals with survey weights, which is a serious problem with no easy solution
mice package: implements multiple imputation algorithms applicable to matching. You can start by reading this.

References
The ones I said before plus:

Statistical matching by S.Rässler. A very inspiring book (I think it is a must).
Some good guidelines + case studies for official statistics by Eurostat here.
Comment: You should have a close look at StatMatch package and its manual: http://cran.r-project.org/web/packages/StatMatch/vignettes/Statistical_Matching_with_StatMatch.pdf. The authors also have a book in Wiley but I find it quite hard to read. Your matching problem sounds like a typical one, it sounds have enough information to do something interesting. Take into account that your historical data can be used as 'auxiliary information' -statistical matching terminology-
Comment: I agree; StatMatch looks like my best bet, and the documentation even states that it does weight calibration.  Off to learn how to use it :)  Thanks again!
Comment: Thank you for your very helpful and detailed answer!  My field is market research, and I'm likely to fuse different data sources and split questionnaires we conduct.  I was planning to stratify on age, gender and region, and was aiming for income, family size, children, marital status, education & occupation type (eg: white collar workers) for common variables.  Thanks for the nearest neighbor warning, as I was looking at KNN in R and Weka.  Also, is there such a thing as too many common variables?  As for weighting, was hoping to split/merge to balance.
Comment: Also, we have quite a bit of historical data (full surveys and some simulated ones) so I may have more options for cross-validation.  The simulated ones would be a rough guide, as I'm not very confident of the quality of that data.  So one method I had in mind was to delete some of the specific variables in a historical full survey, apply the matching, then cross-validate.


R Shiny Data Table is not showing fully as per width

Question: I am trying to display the data table is R shiny and passing the width argument in the syntax. But, the table is not displaying to the full width. So, I am trying to reduce the gap between table and input slider. Any help appreciated. Below is the syntax and image from the output.
#### Required Packages

Pckg_to_Inst = c("dashboardthemes", "shiny", "shinythemes",
                 "shinydashboard", "shinyWidgets", "DT", "bslib", "thematic", "shinydashboardPlus", "summaryBox")

##### Defining Negation

'%not_in%' = Negate('%in%')

for(i in 1:length(Pckg_to_Inst)){

  if(Pckg_to_Inst[i] %not_in% installed.packages()) install.packages(Pckg_to_Inst[i],
                                                                     dependencies = TRUE)
  library(Pckg_to_Inst[i], character.only = TRUE)
}

#######################################################

ui <- fluidPage(

  titlePanel("Dashboard"),
  br(),
  navbarPage("",

             ###### Here : insert shinydashboard dependencies ######
             header = tagList(
               useShinydashboard()
             ),
             #######################################################

             navbarMenu("Data Tracker", icon = icon("database"),

                        tabPanel("Test", fluid = TRUE,

                                 hr(),

                                 fluidRow(
                                   column(12,
                                          uiOutput("summarybox"))
                                 ),

                                 hr(),

                                 fluidRow(

                                   column(8,
                                          column(width = 12,
                                                 mainPanel(DT::dataTableOutput("table1")))),

                                   column(4,
                                          box(
                                            width = 12,
                                            title = "Data Secription - ",
                                            selectInput(inputId = "Data_Type",
                                                        label = "Select Time Frame/Data Type",
                                                        choices = c("A", "B", "C", "D"),
                                                        selected = "A",
                                                        width = "220px"),
                                            background = "black",
                                            "A box with a solid black background"
                                          )))

                        ))

  ))

server <- function(input, output) {

  output$table1 <- DT::renderDataTable({

    DT::datatable(iris, class = 'cell-border stripe',
                  options = list(
                    initComplete = JS(
                      "function(settings, json) {","$(this.api().table().header()).css({'background-color': 'teal', 'color': 'black'});","}"),
                    scrollX = TRUE,
                    scrollY = TRUE,
                    pageLength = 5),
                  extensions = 'Buttons',
                  selection = 'single',       ## enable selection of a single row
                  filter = 'top',              ## include column filters at the bottom
    )

  })

}

shinyApp(ui, server)

And attached is the output image.
Answer: Your DT::dataTableOutput("table1") is inside a mainPanel and by default, mainPanel has a width of 8 (the sidebar is taking the other 4).
from ?mainPanel:

Usage
sidebarLayout(
sidebarPanel,
mainPanel,
position = c("left", "right"),
fluid = TRUE
)
sidebarPanel(..., width = 4)
mainPanel(..., width = 8)
Comment: Ok. but in that case blank space should be not produced in between, right ?
Comment: The `width = 8` in `mainPanel` means "take up 8 out of 12 columns of the parent container". Or take up 2/3 of the parent container width. Removing the `mainPanel` and placing the data table in the column directly, should solve the problem.
Comment: Yes, I did the same thing and removed the width as well as mainpanel. It solved the problem. Thanks
Answer: You can give a custom height and width to dataTableOutput.
library(shiny)

ui <- fluidPage(

    titlePanel("Dashboard"),
    br(),
    navbarPage("",

               ###### Here : insert shinydashboard dependencies ######
               header = tagList(
                 useShinydashboard()
               ),
               #######################################################

               navbarMenu("Data Tracker", icon = icon("database"),

                          tabPanel("Test", fluid = TRUE,

                                   hr(),

                                   fluidRow(
                                     column(12,
                                            uiOutput("summarybox"))
                                   ),

                                   hr(),

                                   fluidRow(

                                     column(8,
                                            column(width = 12,                                                    mainPanel(DT::dataTableOutput("table1", width = '800px')))),

                                     column(4,
                                            box(
                                              width = 12,
                                              title = "Data Secription - ",
                                              selectInput(inputId = "Data_Type",
                                                          label = "Select Time Frame/Data Type",
                                                          choices = c("A", "B", "C", "D"),
                                                          selected = "A",
                                                          width = "220px"),
                                              background = "black",
                                              "A box with a solid black background"
                                            )))

                          ))

    ))

  server <- function(input, output) {

    output$table1 <- DT::renderDataTable({

      DT::datatable(iris, class = 'cell-border stripe',
                    options = list(
                      initComplete = JS(
                        "function(settings, json) {","$(this.api().table().header()).css({'background-color': 'teal', 'color': 'black'});","}"),
                      scrollX = TRUE,
                      scrollY = TRUE,
                      pageLength = 5),
                    extensions = 'Buttons',
                    selection = 'single',       ## enable selection of a single row
                    filter = 'top',              ## include column filters at the bottom
      )

    })

  }

  shinyApp(ui, server)

Comment: Actually, i have tried this solution before posting the question but not sure somehow it did not worked for me. But, removing mainPanel argument works for me. Thanks for your response.
Comment: Thanks for the solution but it is still showing the white space in between.
Comment: You can change the `width` in `DT::dataTableOutput("table1", width = '800px')` to anything of your choice. It works for me.


Font Awesome aparece como "blocos" no nginx sob FreeBSD 10.1

Question: Tenho um site que transportei do CentOS para o FreeBSD usando rsync + mysqldump.
No CentOS + Apache funcionava normal no FreeBSD + nginx deu esse erro:

Acredito que seja relacionado a este código que "converte" o \f086 em um ícone.
<code>#primary_nav #nav_app_forums a:before {
    content: "\f086";
}
</code>
Ao invés dos ícones aparecem "blocos", estou usando o IP.Board 3.4.6 com o tema Dispersion e o ProMenu.
Alguem tem idéia de como resolver?
Obrigado.
Comment: Pode inspecionar o elemento utilizando as ferramentas de desenvolvedor do navegador e ver a fonte que está realmente sendo utilizada nesses quadrados?
Comment: Aparece algum erro no console ao abrir a página?
Comment: Tem certeza de que o CSS do Font Awesome está sendo importado para a página?
Comment: Sim, sem erros no console do Chrome e chequei os paths.
Comment: Pra fazer um ícone do Font Awesome aparecer, é necessário que você coloque as classes em uma tag *i*, dessa forma: ``. Sendo *fa-bed* a classe do ícone desejado. Veja se isso resolve seu problema. Caso não queira usar o `i`, vai ter que copiar todas as propriedades  da classe `fa` e da classe do ícone pra sua regra CSS.
Comment: Na verdade eu não editei nenhum código, é tudo do IP.Board e ProMenu: https://www.invisionpower.com/apps/board/ http://community.invisionpower.com/files/file/5435-promenu-basic/ o estranho é que funcionava normal no CentOS.
Comment: Aparentemente está faltando a fonte do **Font Awesome**. Verifique se os arquivos de fonte foram migrados corretamente, se não corromperam e tudo o mais. Teste também alterando o CSS para usar de outro lugar, tipo isto: ``
Comment: Continua o mesmo erro.
Comment: Nenhum erro aparecia no console não. Essa pergunta é um pouco antiga, infelizmente não tenho o site mais...
Comment: Votei para fechar essa pergunta como não reproduzível. O texto da pergunta não tem informação suficiente para se saber qual é a causa do problema ou reproduzi-lo, vez que há inúmeras causas possíveis. O autor da pergunta (@sistematico) já não tem mais o site porque a pergunta é antiga. Desta forma, tentativas de produzir uma resposta (tal como a minha ou as duas respostas deletadas) serão tentativas de adivinhação que não tem como serem averiguadas. Logo, não há como se dar uma resposta correta a essa pergunta, apenas chutes, e por isso estou votando pelo fechamento.
Answer: Às vezes eu tenho esse problema com o uBlock Origin, que acaba bloqueando o download de fontes do Font Awesome.
Caso seja esta a causa, a solução é simplesmente desbloquear a fonte no uBlock Origin ou outro tipo de adblock que você esteja usando. É verdade que você não tem como forçar o seu usuário a desbloquear, e portanto orientá-lo em casos como esse acaba sendo a melhor saída.


Standardizing dummy variables for variable importance in glmnet

Question: I've used glmnet to build a binomial logistic regression model and I'm now trying to determine the importance of the variables in the model. I've read a few posts about how to get the standardised coefficient values from the model, for example:

Variable importance from GLMNET
Standardized coefficients & glmnet

Initially I've approached it by standardizing all variables in my model matrix, both continuous and dummy variables, and setting standardize = FALSE.
As noted in this (pdf) free example chapter on dummy variables in regression  (in section 7.4), there is no value in standardizing coefficients of dummy variables if the aim is interpretation.
My question is: Is standardizing dummy variables (as well as continuous) and examining the magnitude of the standardized coefficients a valid approach for determining variable importance? Can I say that if a standardized coefficient for a given dummy variable is larger in magnitude than a standardized coefficient for a given continuous variable, the dummy variable is more important?
Answer: One option, advocated by Andrew Gelman, is to scale the continuous variables by two standard deviations and leaving the binary variables in their raw format. In this case, the comparison between the two is moving from -1SD to +1SD on the continuous variable is equivalent to going from 0 to 1 (i.e., Category A to Category B) in the binary variable. You can his paper here.
Answer: The problem with Gelman (2008)'s approach of scaling continuous variables by two SD is that if dummy variables are class-imbalanced, then their SD can be < 0.5. Dummy variables only have SD = 0.5 when the proportion of 1's and 0's is equal.
Variable importance for dummy variables vs. continuous variables would seem most comparable if both have been standardized, although the dummy variables would no longer retain their usual intuitive interpretation.


Liferay overriding JSONAction using hook

Question: Liferay has feature to modify Struts Action using hook which is straight-forward. However I am trying to override getJSON method of RateEntryAction.
The use case is I need total positive score and total negative score instead of average score that Liferay's OOTB rating component provides.
I already have necessary methods readily available in Liferay API to get the total positive and negative scores. I created a custom type just like "stars" and "votes" to write my necessary logic in hook. I was then planning to override struts action represented by /portal/rate_enty to override the method and add negative and positive votes in the json that is already being returned.
But RateEntryAction (which is the original class) is extending JSONAction and overriding getJSON method. Is it possible to override just getJSON method using struts action hook?
Answer: I can see this action class in struts-config.xml, so it's a Struts Action.
This is the path you could try to override.
<code><action path="/portal/rate_entry" type="com.liferay.portal.action.RateEntryAction" />
</code>
I would try to override this path, and see if can put your logic there.
RateEntryAction extends from JSONAction , and this one from org.apache.struts.action.Action  , which is the same class that PortletAction extends.
Hope this help
Comment: That's what I had earlier done. But getJson cannot be overrided. I had to override execute method
Answer: As per my research we can only hook com.liferay.portal.struts.PortletAction  sub classes, Yeah JSONAction also extend org.apache.struts.action.Action class same as PortletAction, but liferay allow only those classes hookable which comes under PortletAction class not except that.
Answer: As RateEntryAction is not a Struts Action, you can't overload it with a Struts Action Hook. I'm not aware of any hookable way to override it, thus my best guess is that you'll have to either go for ext or introduce a whole new JSON API function and hook all places in Liferay that refer to the original implementation to now go to your implementation.
I suspect that in this case an ext plugin will be easier to maintain. Someone correct me if I'm wrong.
Comment: Whoops, sorry. I missed Action's package name and got distracted by the indirection, secretly assuming the wrong thing. My bad, sorry for misleading you. Looks like @Joaco1977 found the answer, upvoting that one.
Comment: Actual RateEntryAction extends JSONAction which extends StrutsAction, so technically RateEntryAction is a struts action


Regrading 2018 Community Moderator Election

Question: How will others vote for moderators who stands for election recently ?
Answer: As mentioned here : https://magento.stackexchange.com/election?cb=1
Every election has three phases:
Nomination
Primary
Election
once Nomination completed , than you can see an option to vote....


How to treat a grouping variable in linear mixed modeling

Question: I am trying to learn how to analyze data using linear mixed effects. Specifically, I'm not sure how to handle a grouping variable that's part of my data set.
Here's a sample scenario: Let's say I am interested in the effect difficulty has on the time to read short paragraphs. There are three versions of each paragraph: Easy, Moderate and Hard and difficulty is repeated-measures. So, each subject will read 6 short paragraphs, 2 at each level of difficulty.
I will also construct a grouping variable to control for order of condition. Subjects will be assigned to one group (so essentially, I have an independent groups variable). In this case, each group will contain a specific sequence of the paragraphs in the conditions. For example, Group 1 will get Paragraphs 1 & 5 in the easy condition, paragraphs 2 & 6 in the moderate condition and 3 & 4 in the difficult condition. Group 2 will get paragraphs 3 & 4 in the easy condition, 1 & 5 in the moderate condition and 2 & 6 in the difficult condition. Group 3 will receive Paragraphs 2 & 6 in the easy condition, Paragraphs 3 & 4 in the moderate condition and paragraphs 1 & 5 in the difficult condition.
So, when I lay out the data in long format, I have something like:
https://docs.google.com/document/d/1HFYzAKnK1g3X_-V6ouRiArlqpNo9IRjSiPmAS-WwcT4/edit
My fixed effect is difficulty and random effects are subject and items. However, I'm not sure how to treat the grouping variable. Can I just eliminate the grouping variable -- when items are included, the grouping variable seems redundant. If I can't eliminate the grouping variable, should I treat it as a fixed effect or random effect? The grouping variable is not of interest -- it's just a sequence of items that move together across conditions.
I apologize if I have not been clear -- as I indicated I am learning LMM now. But thank you for your help and advice you can give.
Answer: From what I understand of the study design, the grouping variable is simply based on the paragraphs read. Therefore, if you include paragraph as a grouping variable for random intercepts, for example using:
time ~ difficulty + (1 | paragraph) + (1 | subject)

then you will not need to include group in the model at all.


Should we reinsert trend after doing forecast using detrended data?

Question: Basically, when we detrend a signal, we detect and remove a linear component of that signal. This produces a stationary version of that signal. And we can use various forecasting algorithms to forecast if the signal is stationary.
I checked a fair amount of notebooks/tutorials and I noticed that all forecasting tutorials finished after making forecasts from detrended data. I want to understand what I am missing or maybe these tutorials leave out the final (reinserting the trend) part since it is simple.
Answer: Since you presumably want forecasts for the original (trended) series, yes, you should insert the extrapolated trend after forecasting the detrended series. "Insert" here can be additively or multiplicatively, depending on how you detrended the series in the first place. And note that in extrapolating trends, it is often useful to dampen it, rather than let the trend grow without bounds in either the positive or the negative direction.
Comment: [Here](https://otexts.com/fpp3/forecasting-decomposition.html) is an example for a multiplicative *seasonal* decomposition. A multiplicative trend is rarely used, and I just mentioned it for completeness. It can go out of control rather quickly.
Comment: Thank you very much, Stephan. I am having a hard time finding a guide/tutorial on this which is weird since forecasting is such a popular topic.   I am using differencing to detrend the signal. Can you give me example of where multiplicatively insert is needed?
Comment: Thank you a lot Stephan. I have a better understanding and confirmation after your answer. I will use that to ask a more comprehensive question


Status message "The changes have been saved" displays when updating user data using AJAX and Services module

Question: I am using navigator.geolocation.getCurrentPosition to get the user's current location, and then using the Services module to update the user's location (part of the user object) via AJAX.
Here is the AJAX call to the service:
jQuery.ajax({
  type: "PUT",
  url: "/api/internal/user/" + uid,
  data: JSON.stringify(geolocation),
  dataType: "json",
  contentType: "application/json",
  success: function(r) {
    console.log(r);
  }
});

The update works just fine, but when I navigate to another page afterwards, I get the status message: "The changes have been saved". After some digging, I found that this status message gets displayed any time user data is updated, which is fine usually, but will be confusing in this scenario since data is being updated behind the scenes. I'm assuming that since the user update is taking place via AJAX, the status message doesn't get displayed until next page load.
Is there any way to prevent Drupal from displaying this status message only when user data is updated using Services?
Comment: what's your Drupal version?
Comment: Running Drupal 7.16
Answer: On your menu callback function ( at /api/internal/user/%user ) you can call drupal_get_messages('status');. This will clear the status messages displayed to the user. A better approach would be to check the message first:
$message = drupal_get_messages('status', false);
if( $message == t('The changes have been saved.') ) {
  drupal_get_messages('status');
}

If this is not a custom module, you should build one and use hook_page_alter() like this:
function yourmodule_page_alter(&$page){
  if(arg(0) == 'api' && arg(1) == 'internal' && arg(2) == 'user') {
    $message = drupal_get_messages('status', false);
    if( $message == t('The changes have been saved.') ) {
      drupal_get_messages('status');
    }
  }
}

Comment: Pardon my ignorance; should this menu callback be in a module, or somewhere else?
Comment: yeah. I thought this is in a custom module you built. I updated my answer with another solution.
Comment: Thanks, I was on my way to implementing a form of this, and your code helped me finish it. Appreciated!
Answer: If you want to use JavaScript/ jQuery then simply use it:
$(document).ready(function(){
 if($("h2.element-invisible").text() === "Status message") {
    $("h2.element-invisible").parent().css('display','none');
 }
});

Answer: In one of our recent projects, the client wanted to block users from changing some of their own profile values based on some run-time logic. However, even if nothing was changed, the user core module would send the message "The changes have been saved." which would be confusing for the user.
The accepted answer here will delete ALL pending Drupal messages if a match is found for one of the messages--perhaps not a desired outcome.
We decided to implement a more general-case message interception scheme. Given the restrictions not to upset the Drupal 7 core, it ends up being somewhat convoluted, but the run-time performance hit is mitigated given the extra conditionals.
Given that this code is accessing only the $_SESSION variable, it can be placed in a template or module, and the '...clear...' function should be called from within a late-stage preprocess-page hook/theme/template function (e.g., template_preprocess_page(), hook_page_build(), etc.).
Anyway, here's the code we used, in case it helps others:
 /**
 * Adds message to intercept and delete upon next page load.
 *
 * intercepts system message so that it's not displayed to users
 * (usually to avoid confusing Drupal messages)
 *
 * @param string $message
 * The exact text that will be intercepted.
 *
 * @param string $type
 * The Drupal message type that will be intercepted.
 */
function mymodule_intercept_system_message($message = NULL, $type = 'status') {
    if (NULL !== $message) {
       $_SESSION['mymodule_intercepted_message'][$type][] = $message;
    }
}

/**
 * Clears system messages that have been flagged for interception.
 *
 * See: mymodule_intercept_system_message()
 * Admin will still see the intercepted message with added prefix.
 *
 * @param bool $clear_flagged_messages
 * boolean: true = All flagged messages are cleared for next page load, defaults
 * to true
 *
 * @return
 * bool TRUE if message was intercepted, FALSE otherwise
 */
function _mymodule_clear_intercepted_system_messages($clear_flagged_messages = true) {
/**
 * Interact directly with $_SESSION since drupal core function drupal_get_messages()
 * only allows for all or no messages to be deleted, not specific messages
 *
 * drupal messages:
 *      $_SESSION['messages'][type][idx] = message-text
 * intercept messages:
 *      $_SESSION['mymodule_intercepted_message'][type][idx] = message-text;
 */
    $retval = false;
    if (isset($_SESSION['messages'])) {
        if (isset($_SESSION['mymodule_intercepted_message'])) {

            foreach ($_SESSION['messages'] as $type => $message_array) {
                foreach ($message_array as $key => $message) {

                    if (isset($_SESSION['mymodule_intercepted_message'][$type])) {
                        foreach ($_SESSION['mymodule_intercepted_message'][$type] as $int_key => $int_message) {

                            if ($int_message === $message) {
                                //message found, clear it so it doesn't display
                                unset($_SESSION['messages'][$type][$key]);

                                //unset entire type if empty, so Drupal doesn't ouput blank message block
                                if (0 == count($_SESSION['messages'][$type])) {
                                    unset($_SESSION['messages'][$type]);
                                    // unset messages if empty,
                                    // because Drupal core expects array not to exist if empty
                                    if (0 == count($_SESSION['messages'])) {
                                        unset($_SESSION['messages']);
                                    }
                                }

                                $retval = true;
                                if (user_access('administer users')) {
                                    //let admin know of message
                                    drupal_set_message('Intercepted message was: ' . $message, $type);
                                }
                            }
                        }
                    }

                }

            }
        }
    }

    if ($clear_flagged_messages) {
        //clear out all flagged messages for next page load
        unset($_SESSION['mymodule_intercepted_message']);
    }

    return $retval;
}


Coefficients of power series involving binomial expansion

Question: I want to expand $\frac{e^z}{z+3}$ around $z=1$.
For it's power series around $z=0$, I computed
$$
\sum_{n=0}^\infty a_n z^n \qquad \mbox{with} \ a_n = \sum_{k=0}^n \frac{(-1)^n 3^{-(n+1)}}{n!(k-n)!} .
$$
Using the binomial theorem, I now want to expand this power series around $z=1$. To this end, I use the substitution $u:=z-1$, to simply the notation.
$$
(z-1+1)^n = (u+1)^n \\
= \sum_{k=0}^n \binom nk u^k \ .
$$
Now, I'm trying to find the new coefficients $b_n$ such that
$$
\sum_{n=0}^\infty a_n z^n = \sum_{n=0}^\infty \sum_{k=0}^n \binom nk u^k = \sum_{n=0}^\infty b_n u^n .
$$
Answer: We can work directly. For $u=z-1$, $$f(z)=\frac{e^z}{z+3}=\frac{e^{u+1}}{u+4}=\frac{e}{4}e^u\frac{1}{1+\dfrac{u}{4}}=\frac{e}{4}\left(\sum_{n\ge 0}\frac{1}{n!}u^n\right)\left(\sum_{n\ge 0}\frac{(-1)^n}{4^n}u^n\right)\; (|u|<4),$$ and now, we can use the Cauchy product of series, to obtain $$f(z)=\sum_{n\ge 0}b_nu^n.$$
Answer: Expanding on Alone's suggestion,
$$
b_n = \sum_{k=0}^n a_n c_{k-n} = \sum_{k=0}^n  \frac{(-1)^n 4^{-n}}{(k-n)!} \ .
$$
Hence, the expansion around $z=1$ is given by, $$\frac{e^z}{z+3} = \sum_{n=0}^\infty b_n(z-1)^n \ .$$


No access control allow origin in rails

Question: I am getting this error while I am trying to hit on my api deployed on Heroku in rails application using Ruby on rails :
 Failed to load https://**: my API link deployed on Heroku No
'Access-Control-Allow-Origin' header is present on the requested resource.
 Origin 'http://localhost:3000' is therefore not allowed access.

Comment: You have not provided much info but check this articles on how to handle ssl on heroku https://devcenter.heroku.com/articles/ssl-endpoint#provision-the-add-on && https://readysteadycode.com/howto-setup-ssl-with-rails-and-heroku
Answer: http://localhost:3000 is not permitted via CORS. You need to set it up in your rails application deployed on heroku.
Add the below snippet to your config/application.rb file and then redeploy to heroku
  config.middleware.insert_before 0, Rack::Cors do
    allow do
      origins 'http://localhost:3000'
      resource '*', :headers => :any, :methods => [:get, :post, :options]
    end
  end

No access control allow origin in rails


Como verificar se data esta cadastrada através do MontheCalendar ? C#

Question: Fala galera bom dia!
Estou com uma pequena dificuldade em relação uso do MontheCalendar...
Eu tenho um sistema de agendamento por DATA, que após o cadastramento do determinado evento, ele é apresentado no DATAGRIDVIEW,com seguinte codigo
`public void GetData()
        {
            try
            {
                cg.con = new SqlConnection(cn.DBconn);
                cg.con.Open();
                cg.cmd = new SqlCommand("SELECT RTRIM(Id),RTRIM(Nome) ,RTRIM(Endereco), RTRIM(Quadra), RTRIM(Lote), RTRIM(Telefone), RTRIM(Celular),(Data), RTRIM(Hora), RTRIM(Email), RTRIM(Observacao), RTRIM(Locacao), RTRIM(Evento),  RTRIM(Estado) from Eventos order by Data", cg.con);
                cg.rdr = cg.cmd.ExecuteReader(CommandBehavior.CloseConnection);
                DGW_Agenda.Rows.Clear();

                while (cg.rdr.Read() == true)
                {
                    DGW_Agenda.Rows.Add(cg.rdr[0], cg.rdr[1], cg.rdr[2], cg.rdr[3], cg.rdr[4], cg.rdr[5], cg.rdr[6], cg.rdr[7], cg.rdr[8], cg.rdr[9], cg.rdr[10], cg.rdr[11], cg.rdr[12], cg.rdr[13]);
                }

                cg.con.Close();
            }
            catch (Exception ex)
            {
                MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
            }
        }`

O que eu gostaria é que quando eu clicasse em alguma data no monthercalendar ele checasse ao meu datagridview  na coluna 'DATA DE AGENDAMENTO' se a data esta agendada, ou data disponivel para cadastramento.
Segue o print da tela do sistema para melhor entendimento.
Comment: E o código onde está? Você não consegue checar no `DataSource`, você quer que ao selecionar uma data ele avise que não está disponível ou desabilitar as datas indisponíveis na seleção? O sistema vai ser utilizado por vários usuários? Está fazendo algum controle de consistência? Elabore mais a sua pergunta apresentando um [MCVE]
Comment: Olá @LeandroAngelo eu não consigo checar ele no meu datasource, pesquisei alguns foruns como fazer essa checagem, mas estou um pouco perdido. A ideia é somente verificar se a data esta cadastrada de acordo com o evento cadastrado! fazendo a verificação na coluna do datasource.
Comment: não entendi porque você não consegue consultar o datasource... mas de qualquer forma se que apresente o código, não tem como ajudar
Answer: Sem que apresente o seu código e estrutura não como detalhar mais, mas veja o exemplo abaixo de uma abordagem que pode lhe ajudar...
Estrutura hipotética da sua model para os agendamentos:
public class Agendamento
{
    public string Local { get; set; }
    public DateTime DataAgendamento { get; set; }
}

Estrutra do formulário popuplando a DataGridView e já com o bind do evento de seleção para o MonthCalendar, compartilhando a mesma source para validação...
public partial class Main : Form
{

    private List<Agendamento> agendamentosDataSource { get; set; } = new List<Agendamento>();

    public Main()
    {
        InitializeComponent();

    }

    private void Main_Load(object sender, EventArgs e)
    {
        //Simulando uma consulta para popular a lista de agendamentos
        agendamentosDataSource.AddRange(
            new List<Agendamento>() {
                new Agendamento{ Local = "Teste 1", DataAgendamento = DateTime.Parse("18/04/2019") },
                new Agendamento{ Local = "Teste 2", DataAgendamento = DateTime.Parse("19/04/2019") },
            });

        //Apontando o DataSource da Grid para a lista
        grdPessoas.DataSource = agendamentosDataSource;

    }

    private void mcSelecionaData_DateChanged(object sender, DateRangeEventArgs e)
    {
        var dataSelecionada = (sender as MonthCalendar).SelectionStart;

        //Utilizando a mesma lista do DataSource da Grid para recuperar as datas reservadas.
        var datasReservadas = agendamentosDataSource.Select(x => x.DataAgendamento);

        //Validação e feedback simples para o usuário...
        if (datasReservadas.Contains(dataSelecionada))
            MessageBox.Show("Essa Data já foi reservada");

    }

Comment: Fala Leandro Angelo Obrigado pela sua ajuda, eu reformulei a pergunta para melhor entendimento do problema srsrs. eu ja tenho a forma de apresentação dos dados do datagridview so nao sei como fazer essa checagem pelo monthercalendar


How do you programmatically change the tab order in a Win32 dialog?

Question: Often time I need to add a control to a dialog after the dialog has been generated via dialog template and CreateDialogIndirect.  In these cases the tab order is set by the dialog template and there is no obvious way to change the tab order by including a newly created control.
Answer: I recently discovered that you can use SetWindowPos to accomplish this.  Determine which control after which you want to insert the new control in the tab order then use SetWindowPos like this:
<code>SetWindowPos(hNewControl, hOldControl, 0, 0, 0, 0, SWP_NOMOVE|SWP_NOSIZE);
</code>
This changes the z-order of controls which, in turn, establishes the tab order.
Comment: This works well. There's an MFC equivalent in CWnd:  newControl.SetWindowPos(oldControl, 0, 0, 0, 0, SWP_NOMOVE | SWP_NOSIZE);
Comment: Works for me. Just a note: here `hNewControl` is the `HANDLE` to the control you want to change the tab order of. `hOldControl` is the `HANDLE` of the control you want to put `hNewControl` _after_ as far as tab order is concerned.
Answer: I know this is an old question but here is how to do it at compile time (which is preferable in the vast majority of cases):
http://msdn.microsoft.com/en-us/library/7039hzb0(v=vs.80).aspx
My favourite method:

From the View menu, choose Tab Order.
Choose Assign Interactively.
Double-click the tab order box beside the control you want to be the
first control in the tab order.
Click the tab order box for each of the other controls.
Click anywhere on the form to save your changes and exit Tab Order
mode, or press ESC to exit Tab Order mode without saving your
changes.
Comment: The question very clearly says that it can't be done at design time.


C# readonly access to List<> or other collection, or ToArray() magic

Question: 1st question is about way of protecting my List from changing(remove/add/clear etc...) from outside
there is my way:
class Foo
{
    public int[] MyCollection
    {
        get{ return (_myCollection==null)?null:_myCollection.ToArray();
    }
    protected List<int> _myCollection;
}

Is it good? Or is there any better ideas, or may be patterns?
2nd:
When iam testing this solution with stopwatch, i was very surprised.
List -enumeration was slower than List.ToArray() enumeration with cast time:
List<int> myList = new List<int>();
for (int j = 0; j < 10000; j++)
{
    myList.Add(j);
}
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 10000; i++)
{
    //casting every iteration:
    var ROC = myList.ToArray();
    int count = 0;
    foreach (var a in ROC)
    {
        count += a;
    }
}
sw.Stop();
Console.WriteLine(sw.Elapsed);

It shows me 700 msec, and
List<int> myList = new List<int>();
for (int j = 0; j < 10000; j++)
{
    myList.Add(j);
}
Stopwatch sw = new Stopwatch();
sw.Start();
for (int i = 0; i < 10000; i++)
{
    int count = 0;
    //No casting at all
    foreach (var a in myList)
    {
        count += a;
    }
}
sw.Stop();
Console.WriteLine(sw.Elapsed);

Shows me 843 msec... why is it so?
Comment: If you have two separate questions then ask two separate questions; don't include two different questions (that aren't closely related, and these aren't) in one question.
Comment: He means just that. **Never try to do performance timing with the debugger running**. The customer won't be running the debugger, and running the debugger **massively** changes the performance characteristics of a program. I am in the middle of a series on tech.pro about how to do this correctly; you might want to read it. http://tech.pro/blog/1293/c-performance-benchmark-mistakes-part-one
Comment: Concerning the stopwatch timing aspect: Looking at the IL code in ILSpy (or an equivalent tool) is usually good for these types of questions.
Comment: Did you time a release build from outside the debugger?
Comment: it is not separated 4 me, cause if it is realy fast - then there is no reasons to use any other accessToList.
Comment: @Matthew Watson  mmm whats means "time a release build from outside the debugger", i'am never heard about it
Comment: @Eric Lippert thx! U make my day!
Answer: First off, if you have two questions then post two questions.  I'll answer your first question.
Second, good for you for thinking about how to protect your list from mutation. A lot of people forget about this and expose their internal state to callers.
There are a number of things you can do.
1) Don't use a mutable list in the first place. Use an immutable list. The Add method of an immutable list returns a different list rather than mutating the current list. There are new immutable list classes in the BCL now. These can be very efficient in both space and time.
2) Use a mutable list and make a copy every time.  You're already doing this. The problem is of course that it is slow and uses a lot of memory.
3) Return AsReadOnly as Claudio suggests.  Note that this just makes a read-only façade around the list; if the list changes, the read-only façade changes too. "Read only" means just that: the user cannot write it. It does not mean that it never changes. Also, there's an enumeration issue, which I'll address in (4):
4) Make LINQ do the work. return _myCollection.Select(x=>x); and have the property be of type IEnumerable<int>.  This has two downsides. First, the caller only gets forward enumeration. Second, suppose the caller is doing a foreach over the property, and then inside the loop does something which changes the list. This will cause an exception; you can't change a list while you're enumerating it. If you need to support that scenario then (1) or (2) are your best bet; in those the enumeration will be over a snapshot, not over the mutating list.
Comment: Eric, can you elaborate a bit on " new immutable list classes in the BCL", please ? Don't see anything like that in Collections namespace. I know there are immutable lists in F#, which we can use, but if there are immutable lists in core libraries I'd obviously prefer to use these.
Comment: return _myCollection.Select(x=>x); That is! Thx a lot. My perfectionist are dancing inside of me :)
Comment: @Eugene: http://blogs.msdn.com/b/bclteam/archive/2012/12/18/preview-of-immutable-collections-released-on-nuget.aspx
Answer: For read only collection you could use List<T>.AsReadOnly()
public IList<int> MyCollection
{
   get{ return _myCollection==null  ? null : _myCollection.AsReadOnly();
}

If necessary and in order to make more clear that we're talking about a read only collection you could define your property like this
public IReadOnlyList<int> MyCollection

Comment: I'd probably make the return type `ReadOnlyCollection` to make it plain what is happening ehre.
Comment: @Claudio Redi  - but in this case i can write something like this: SpiderMan.MyCollection.Add(123); and it will provoke an exception. so we are losing any syntax-help at this way
Comment: @secunda, changed a little bit my answer.
Comment: +1 Good answer! @secunda Isn't that what you are after? "way of protecting my List from changing" sounds to me like you want it to be ReadOnly.
Comment: @Claudio Redi -looks like perfect, but i don't see any benefits except "ReadOnly" - word.
Comment: @Jesse Webb may be i am too effeminate with C# sugar, but I was hoping to find some one-word-soolution, or something-like a syntax pattern, cause i see this task every day. So, as i can see, there is no rule and invisible problems, and each can write as he whant, but no IEnumerable casting :)
Answer: Answer to the Second Question in the Question
Iterating over an int array vs iterating over a list of integers produces very different outputs at the IL code level.
Iterating over an Integer Array
  IL_0015:  ldloc.3
  IL_0016:  ldloc.s    CS$7$0001
  IL_0018:  ldelem.i4
  IL_0019:  stloc.2
  IL_001a:  ldloc.1
  IL_001b:  ldloc.2
  IL_001c:  add
  IL_001d:  stloc.1
  IL_001e:  ldloc.s    CS$7$0001
  IL_0020:  ldc.i4.1
  IL_0021:  add
  IL_0022:  stloc.s    CS$7$0001
  IL_0024:  ldloc.s    CS$7$0001
  IL_0026:  ldloc.3
  IL_0027:  ldlen
  IL_0028:  conv.i4
  IL_0029:  blt.s      IL_0015

Iterating over an Integer List
IL_0010:  ldloca.s   CS$5$0000
IL_0012:  call       instance !0 valuetype [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::get_Current()
IL_0017:  stloc.1
IL_0018:  ldloc.0
IL_0019:  ldloc.1
IL_001a:  add
IL_001b:  stloc.0
IL_001c:  ldloca.s   CS$5$0000
IL_001e:  call       instance bool valuetype [mscorlib]System.Collections.Generic.List`1/Enumerator<int32>::MoveNext()
IL_0023:  brtrue.s   IL_0010

The key here is that when using a list, the CLR is performing a type each iteration ( get_Current() object type to int). This may be causing performance issues.
Comment: Basically, `foreach` over an array is generated the same as the `for` loop you'd normally generate.


Django - Show date and time of appointment

Question: I'm currently showing all dates that appointments are not booked.
from datetime import date, timedelta

num_days = 5
start_date = date.today()
timeframe = [start_date + timedelta(days=d) for d in range(num_days)]
exclude = list(Appointment.objects.values_list("start_appointment__date", flat=True).distinct())

set(timeframe) - set(exclude)

Model of appointments
class Appointment(models.Model):

    user = models.ForeignKey(Account, on_delete=models.CASCADE)
    seat = models.ForeignKey(Seat, on_delete=models.SET_NULL, null=True)
    work = models.ManyToManyField(Therapy, related_name='work', verbose_name=_('Work'), blank=True)
    start_appointment = models.DateTimeField(default=timezone.now, blank=True)
    end_appointment = models.DateTimeField(default=timezone.now, blank=True)

For example i have appointments booked for:
[datetime.date(2021, 12, 8), datetime.date(2021, 12, 7), datetime.date(2021, 12, 7), datetime.date(2021, 12, 7)]

And the dates not booked:
{datetime.date(2021, 12, 10), datetime.date(2021, 12, 9), datetime.date(2021, 12, 11)}

I would like to show also the available times for the not booked dates with a time interval between 11:00AM to 21:00PM:.
Is that possible with the current implementation? If yes how can i achieve that? If not could you please suggest an alternative way?
Thank you
Answer: If you need to show available times, i suggest to separate your response by dates, so if you have one day like datetime(2021, 12, 8) and you want to show times between 11AM and 21PM you can use something like:
from datetime import date, datetime, time

date_of_reference = datetime(2021, 12, 8, 0, 0) #your the object of database
initial_time = datetime.combine(date_of_reference.date(), time(11, 0))
final_time = datetime.combine(date_of_reference.date(), time(21,0))

reserved_times = [datetime(2021, 12, 8, 11, 0), datetime(2021, 12, 8, 15, 0)] #reserved_times from database here

hours_between = [datetime.combine(date_of_reference.date(), time(hour + initial_time.hour, 0)) for hour in range(((final_time - initial_time).seconds // 3600) + 1)]
avaliable_hours = [hour for hour in hours_between if hour not in reserved_times]

In your code will be something like
from datetime import datetime, timedelta
from .models import Appointment

# views
def check_available_hours(request):
    day_request = request.GET.get('day','') #day in format yyyy-MM-dd
    date_of_reference = datetime.strptime(day_request, "%Y-%m-%d")
    initial_date_time = date_of_reference
    final_date_time = date_of_reference + timedelta(hours=23)
    appointments_of_day = Appointment.objects.filter(
        start_appointment__gte=initial_date_time,
        start_appointment__lte=final_date_time
    )
    reserved_times = [
    appointment.start_appointment + timedelta(minutes=10*i) for appointment in appointments_of_day for i in range((appointment.end_appointment - appointment.start_appointment).seconds//600)]
    # same as above but more undestandeble
# for appointment in appointments_of_day:
#     for i in range((appointment.end_appointment - appointment.start_appointment)//600):
#         reserved_times.append(appointment.start_appointment + timedelta(minutes=10*i))
    hours_between = [initial_date_time + timedelta(minutes=10*i) for i in range(((final_date_time - initial_date_time).seconds // 600) + 1)]
avaliable_hours = [hour.time() for hour in hours_between if hour not in reserved_times]

    return render(request, "name_template.html", {"hours": avaliable_hours})

If your appointments do not have a duration of a multiple of 10, you have to change the code to controll for every minute instead of 10 minutes as in the example above
Comment: Thank you for help. Is very close to what i want. If you could please help me to write based on my models i would highly appreciate it! Thank you once more.
Comment: @paniklas I edited my answer, if you have any problems if it i will be glad to reply you
Comment: Thank you for replying based on my models. I tried the code you provide me and here the results. I changed `day_request` to `day_request = date.today()
    day_request1 = str(day_request)` when i print i got `2021-12-08` and
`2021-12-08 00:00:00`. Then i get ìnitial_date_time 2021-12-08 00:00:00` and
`final_date_time 2021-12-08 23:00:00`. Then i correctly getting booked appointments `, ]>`. However the reserved_times does not return anything. Any ideas?


How can I turn the dot product of two normalized vectors into a percentage?

Question: When my vectors are perpendicular (90 degrees off) the dot product is 0, and when they are parallel (0 degrees off) the dot product is 1, but half way between these two states (45 degrees off) the dot product is around 0.7.
I'd like to manipulate the dot product to get a new value that represents "percentage aligned", so in the case above, half way between perpendicular and parallel, I would get the value 0.5, and the value would proceed in a linear fashion as you approach 0 or 1.
Thanks!
Comment: And if anyone feels like explaining why you get 0.7 when you're halfway aligned, I'd love to hear it! I feel like I have some ideas, but I'm not comfortable enough with the math to figure out what the relationship actually is.
Answer: If these are unit vectors, their dot product is the cosine of the angle between them.
In your examples, $\cos 0^{\circ}=1$, $\cos 90^{\circ}=0$, and $\cos 60^{\circ}=0.5$.
The angle $\theta$ producing the number $t\in[0,1]$ is $\theta = \cos^{-1} t$. That's the arccosine or inverse cosine. It's defined by $\cos \theta =t$.
And observe that $\cos 45^{\circ}=1/\sqrt{2}\approx0.7071$.
Comment: Ok, so if I'm following you, is my best option to convert the dot to an angle in degrees and take it as a percentage out of 90? I.e. `percentage = (arcos(dot)*(180/pi))/90`
Comment: Er, I supposed I could just do: `percentage = arcos(dot)/1.5708`
Comment: Yes. If your arccos function returns radians, then the factor $180/\pi$ converts that to degrees, and dividing by $90$ produces a number in the range 0-1 (which I presume you interpret as 0%-100%).
Comment: Awesome, thanks so much!
Answer: Observe that $\frac{a \cdot b}{|a||b|}=\cos{\theta}$ is the geometric definition of a dot product. Therefore $\theta = \arccos{\frac{a \cdot b}{|a||b|}}$ is the angle between the two vectors. A percentage is just a portion of a total yes? Here your total is $90^\circ$
I would then think that $\dfrac{\theta}{90^\circ}$ is your "percentage aligned". Let's test it.
Take 2 vectors at a $45^\circ$ angle say, $a = (0,1)$ and $b = (1,1)$ then
$\frac{a \cdot b}{|a||b|} = \frac{(0,1) \cdot (1,1)}{1\cdot\sqrt{2}} = \frac{1}{\sqrt{2}}
$
So $\theta = \arccos{\frac{1}{\sqrt{2}}} = 45^\circ$ and therefore $\frac{\theta}{90^\circ}=\frac{45^\circ}{90^\circ}=0.5$
Now you may ask why I calculated $\theta$ when I already knew it was $45^\circ$, it was just to show the full working, in general you would have to find the angle first.
Random Example
Take 2 vectors say, $a = (0,5)$ and $b = (1,5)$ then
$\frac{a \cdot b}{|a||b|} = \frac{(0,5) \cdot (1,5)}{5\cdot\sqrt{26}} = \frac{25}{5\sqrt{26}}
$
So $\theta = \arccos{ \frac{25}{5\sqrt{26}}} = 11.31^\circ$ and therefore $\frac{\theta}{90^\circ}=\frac{11.31^\circ}{90^\circ}=0.125$
This looks like a good "percentage aligned" as $b$ is close to the x-axis and 0.125 is small.
Random Example 2
Take 2 vectors say, $a = (0,5)$ and $b = (5,1)$ then
$\frac{a \cdot b}{|a||b|} = \frac{(0,5) \cdot (5,1)}{5\cdot\sqrt{26}} = \frac{5}{5\sqrt{26}}
$
So $\theta = \arccos{ \frac{1}{\sqrt{26}}} = 78.69^\circ$ and therefore $\frac{\theta}{90^\circ}=\frac{78.69^\circ}{90^\circ}=0.874$
This looks like a good "percentage aligned" as $b$ is close to the y-axis and 0.874 is large.
Here (If this is what you are asking) tending to 0 would mean parallel and tending to 1 would be perpendicular.
Comment: Marking this as the answer for providing the "percentage aligned" aspect as well as the proofs. Thanks for everyone's help!


Airflow with KubernetesExecutor: Recorded pid does not match the current pid.airflow.Getting AirflowException: Task received SIGTERM signal

Question: I am running Airflow:2.1.4 on a  KubernetesExecutor with Kubernetes: 1.21.0
Getting AirflowException: Task received SIGTERM signal
Stack trace:
[2021-10-11 06:22:52,543] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 17
[2021-10-11 06:22:52,543] {taskinstance.py:1236} **ERROR - Received SIGTERM. Terminating subprocesses.**
[2021-10-11 06:22:52,560] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1165, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1283, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1313, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 150, in execute
    return_value = self.execute_callable()
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/python.py", line 161, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/usr/local/airflow/dags/a.py", line 12, in test
    time.sleep(5)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1238, in signal_handler
    raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal

When a task receives heartbeat_callback(according to job_heartbeat_sec in airflow configuration), the recorded pid and current pid do not match because this part of the code is executed. https://github.com/apache/airflow/blob/main/airflow/jobs/local_task_job.py#L201-L203
if ti.run_as_user or self.task_runner.run_as_user:
            recorded_pid = psutil.Process(ti.pid).ppid()
            same_process = recorded_pid == current_pid

I tried printing the values of run_as_user here by changing source code:
ti.run_as_user = None
self.task_runner.run_as_user = airflow
I have tried changing the job_heartbeat_sec value and the task always fails at that point. I am running the scheduler, webserver as airflow user. The DAGs are also triggered by the airflow user. I have tried setting the run_as_user in DAG definition as both None or airflow, but same error.
Airflow user creation from Docker file:
RUN chown -R airflow: /usr/local/bin/airflow
RUN chown -R airflow: /root
USER airflow
WORKDIR ${AIRFLOW_HOME}

Airflow Configuration
[scheduler]
job_heartbeat_sec = 30
clean_tis_without_dagrun_interval = 15.0
scheduler_heartbeat_sec = 10
num_runs = -1
processor_poll_interval = 1
min_file_process_interval = 0
dag_dir_list_interval = 300
print_stats_interval = 30
pool_metrics_interval = 5.0
scheduler_health_check_threshold = 295
orphaned_tasks_check_interval = 305.0
child_process_log_directory = /root/airflow/logs/scheduler
scheduler_zombie_task_threshold = 300
catchup_by_default = True
max_tis_per_query = 512
use_row_level_locking = True
max_dagruns_to_create_per_loop = 10
max_dagruns_per_loop_to_schedule = 20
schedule_after_task_execution = False
parsing_processes = 2
file_parsing_sort_mode = modified_time
use_job_schedule = True
allow_trigger_in_future = False
dependency_detector = airflow.serialization.serialized_objects.DependencyDetector
run_duration = -1
max_threads = 2
authenticate = False

[kubernetes]
pod_template_file = /usr/local/airflow/pod_templates/pod_template_file.yaml
worker_container_repository = ************************
worker_container_tag = ****************************
namespace = default
delete_worker_pods = False
delete_worker_pods_on_failure = False
worker_pods_creation_batch_size = 1
multi_namespace_mode = False
in_cluster = True
kube_client_request_args =
delete_option_kwargs =
enable_tcp_keepalive = False
tcp_keep_idle = 120
tcp_keep_intvl = 30
tcp_keep_cnt = 6
verify_ssl = True
worker_pods_pending_timeout = 300
worker_pods_pending_timeout_check_interval = 120
worker_pods_pending_timeout_batch_size = 100
airflow_configmap = airflow-configmap
airflow_local_settings_configmap = airflow-configmap

Comment: Could you solve the issue?
Answer: Bug fix should be coming in 2.2.3
Comment: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answer: One of the reasons this might be happening in any version of Airflow is, because of setting retries on any DAG. The DAG arguments at creation:
'retries': 1,
'retry_delay': timedelta(minutes=5)

means that when the task first instance has ran for 5 minutes and not completed, launch the next instance. (Airflow calls it task attempts)
In my case, I was running a docker container inside the DAG task. The container was a long running job which was failing after 5 minutes(the retry_delay) with error:
[2023-06-02, 13:12:43 UTC] {local_task_job_runner.py:271} WARNING - Recorded pid 2038103 does not match the current pid 2036364
[2023-06-02, 13:12:43 UTC] {process_utils.py:131} INFO - Sending Signals.SIGTERM to group 2036364. PIDs of all processes in the group: [2036364]
[2023-06-02, 13:12:43 UTC] {process_utils.py:86} INFO - Sending the signal Signals.SIGTERM to group 2036364
[2023-06-02, 13:12:43 UTC] {taskinstance.py:1517} ERROR - Received SIGTERM. Terminating subprocesses.
[2023-06-02, 13:12:43 UTC] {docker.py:495} INFO - Stopping docker container
[2023-06-02, 13:12:46 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/airflow/providers/docker/operators/docker.py", line 478, in execute
    return self._run_image()
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/airflow/providers/docker/operators/docker.py", line 339, in _run_image
    return self._run_image_with_mounts(self.mounts + [tmp_mount], add_tmp_variable=True)
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/airflow/providers/docker/operators/docker.py", line 400, in _run_image_with_mounts
    for log_chunk in logstream:
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/docker/types/daemon.py", line 29, in __next__
    return next(self._stream)
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/docker/api/client.py", line 422, in <genexpr>
    gen = (data for (_, data) in gen)
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/docker/utils/socket.py", line 113, in frames_iter_no_tty
    (stream, n) = next_frame_header(socket)
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/docker/utils/socket.py", line 85, in next_frame_header
    data = read_exactly(socket, 8)
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/docker/utils/socket.py", line 70, in read_exactly
    next_data = read(socket, n - len(data))
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/docker/utils/socket.py", line 41, in read
    poll.poll()
  File "/home/ubuntu/work-nisha/airflow-poc/venv/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1519, in signal_handler
    raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2023-06-02, 13:12:46 UTC] {taskinstance.py:1345} INFO - Marking task as FAILED. dag_id=s000_docker1, task_id=docker_command, execution_date=20230602T130758, start_date=20230602T131226, end_date=20230602T131246
[2023-06-02, 13:12:46 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 613 for task docker_command (Task received SIGTERM signal; 2036364)

Comparing the task instance PID for first and second attempt, it was matching.
Recorded pid 2038103 does not match the current pid 2036364
The first attempt PID was 2036364 and second attempt PID was 2038103.
And this next attempt was causing the first attempt long process to fail. And since airflow kills all the processes in the group, so second attempt was failing too.
See the airflow code: https://github.com/apache/airflow/blob/dc5bf3fd02c589578209cb0dd5b7d005b1516ae9/airflow/jobs/local_task_job_runner.py#L270
Solution:
If it is a long running task, do not add retry in DAG, or add the retry_delay to a value within which the task instance is expected to complete.


Calculating the mean of something that lies between an upper and lower bound in google sheets

Question: This is based on the answer to my question on the StackExchange math site: https://math.stackexchange.com/a/1983642/382254
Based on the same example in that question, let is say I have a Google Sheet with the following columns and rows:
<code>$0-$10 | $10-$20 | $20-$30
100    | 150     | 80
</code>
The first row is price range and in those columns in the rows beneath it is the number of products that prices that be any price within the given range.
The question is, how do I apply the calculations in the answer:

in Google Sheets?
Answer: <code>=ArrayFormula(
  SUMPRODUCT(REGEXEXTRACT(A1:C1, "\d+"),
             A2:C2) /
  SUM(A2:C2))
</code>
First we extract the left number of the ranges of the first row with
<code>=ArrayFormula(REGEXEXTRACT(A1:C1, "\d+"))
</code>
Then we sum the products of the extracted numbers with the weights (or counts in this case) to have the sum.
Lastly we divide by the total count.
If you want the right end of the intervals use
<code>=ArrayFormula(
  SUMPRODUCT(REGEXEXTRACT(A1:C1, "(\d+)$"),
             A2:C2) /
  SUM(A2:C2))
</code>
Which just takes extracts the number at the end of each interval string.
Or for the middle of each interval
<code>=ArrayFormula(
  SUMPRODUCT((REGEXEXTRACT(A1:C1, "\d+") + REGEXEXTRACT(A1:C1, "\d+$"))/2,
             A2:C2) /
  SUM(A2:C2))
</code>
Comment: Ok... I am confused... lol :D So, perhaps can you explain first (perhaps even visualise) the error in the calculations in the screenshot?
Comment: Sorry I had an error in my spreadsheet treating the cell as text, that was my mistake
Comment: I was about to comment as I thought I was doing a mistake on my part.
Comment: I'll be in the corner taking care of all that egg in my face
Comment: As in using some notion of mean between the ends of the interval?
Comment: Yes, I mean something like that.
Comment: Like returning both the lower end and the upper end into same cell.
Comment: I just added a formula that does the same for the midpoint of each interval. If you want an upper and lower bound just use the first and second formula and concatenate them with a separator
Comment: Thank you very much. One last question. What do you call this scripting language? Where do I find help/documentation for it in google?
Comment: These are [Google Spreadsheet Formulas](https://support.google.com/docs/table/25273?hl=en), these particular ones used [regular expressions](http://regular-expressions.mobi/index.html). If this helped you please consider accepting the answer. Thanks


Return deleted docs data returns undefined

Question: I am using Google Cloud Function, but since it runs on a older version of Node, I can not use this answer anymore. I want a function that will batch delete all the documents in a collection and returns the data from it. This is my attempt:
function deleteCollectionAndReturnResults(db, collectionRef, batchSize) {
  var query = collectionRef.limit(batchSize);
  return deleteQueryBatch(db, query, batchSize, []);
}

function deleteQueryBatch(db, query, batchSize, results) {
  return query.get().then(snapshot => {
    if (snapshot.size == 0) return 0;
    var batch = db.batch();
    snapshot.docs.forEach(doc => {
      if (doc.exists) {results.push(doc);}
      batch.delete(doc.ref);
    });
    return batch.commit().then(() => snapshot.size);
  }).then(function(numDeleted) {
    if (numDeleted >= batchSize) {
      return deleteQueryBatch(db, query, batchSize, results);
    }else{
      return results
    }
  });
}

But when I run it like this:
exports.tester = functions.firestore.document('x/{x}').onCreate(event => {
      deleteCollectionAndReturnResults(db, db.collection("x"), 100).then(docs => {
        console.log(docs)
      })
})

This is my output:

Is there something wrong why I do get the 'function returned undefined'?
Answer: Your tester function doesn't return anything.  Instead, it should return a promise that's resolved when all the work is complete.  It looks like you've simply forgotten to return the promise returned by deleteCollectionAndReturnResults:
exports.tester = functions.firestore.document('x/{x}').onCreate(event => {
    return deleteCollectionAndReturnResults(db, db.collection("x"), 100).then(docs => {
        console.log(docs)
    })
})


arbitrary order of nodequeues in a view

Question: What is the best way to arbitrary order nodequeues in a view?

Drupal 6.26
Nodequeue 2.11
Views 2.16
Comment: You can select sort order as Global:random in view sort settings ..
Comment: arbitrary != random
Answer: ended up using subqueues, was not quite what I wanted.

arbitrary order of nodequeues in a view


experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

Question: I have an experiment with 3 groups:

group 1 got no email
group 2 got an email with 10 dollars incentive
group 3 got an email with 20 dollars incentive

to convert to a specific product.
Conversion rates are as follows:

group 1 - 1.2%
group 2 - 1.3%
group 3 - 1.6%

I am looking at pairwise significance tests, and I get the following results:

group 1 vs group 2: p-value 0.6
group 2 vs group 3: p-value 0.2
group 1 vs group 3: p-value 0.009 - significant.

I am struggling to interpret these results.
It looks like sending an email with 20 dollars is more effective than sending no email, but it is not more effective than sending the 10 dollars email, which in turn is not more effective than sending no email. Contradiction?
I understand p-values are probabilities, so for the first comparison there is a 60% chance to observe those values at random, so I think it's something about probability of intersection of events - but I'm missing something to get the complete picture for this.
Thank you
Answer: The thing that you need to understand/remember is that these tests are used to rule out certain cases, but the failure to rule something out does not make it true.
Consider the following claims (hypotheses) about a coin:

The coin is 2-headed therefore $p(Heads) = 1.0$.
The coin is fair therefore $p(Heads) = 0.5$.
The coin is 2-tailed therefore $p(Heads) = 0.0$
The coin is biased and $p(Heads) = 0.9$

If I flip the coin and observe Heads, then I cannot rule out option 1, but that does not mean that option 1 is the truth, in fact, with a single flip showing heads I can only rule out option 3.  Options 1, 2, and 4 are still possible even though at most 1 of the 3 can be the "Truth".  If I flip the coin a total of 10 times and see Heads each time then I can rule out option 2 at standard significance levels (but there it is still possible that 2 is correct, just unlikely).  But 10 heads still leaves options 1 and 4 as possibilities.
In your case you have ruled out the possibility that groups 1 and 3 are the same (at a given significance level), but you have not ruled out that 1 and 2 are the same (but that does not prove that they are the same) and you have not ruled out that 2 and 3 are the same (but that does not prove that they are), you have also not ruled out the possibility that 2 is different from both 1 and 3.  So from your evidence you are reasonably certain that 1 and 3 are different, but 2 could be the same as either 1 or 3 (but not both at the same time since they are different) or could be different from both, you just don't have enough evidence to rule any of those out.  No contradiction unless you make the mistake of treating possibility as certainty.
Answer: So, there are a couple things going on here.
First, the phrase 'there is a 60% chance to observe those values at random' is a bit loose to understand whether you are in fact interpreting the results correctly.
When you complete a t-test you assume the null hypothesis to be true as well as some other underlying assumptions to be roughly true. Under these assumptions, the p-value tells you how often you would observe a value as extreme or more extreme than the observed value given repeated sampling. When this value is very low, this suggests that the observed sample value has a small probability of occuring by chance.
So what you can conclude from your tests is that the difference between Group 1 and Group 3 does not seem to have occurred as a result of your particular sample and may represent a true difference between the groups.
However, you cannot conclude there is no difference between Group 1 and Group 2. Rather, you have a lack of evidence for a difference between them. You can't prove the null by failing to reject it.  This blog post goes into further detail that might be useful for you on this topic.


What is a good way to unit test templated c++ methods/classes/functions?

Question: I'm looking for a good way to unit test my templated methods, classes, and functions in c++.
I have a feeling that it is not enough to just test using with a single type, and it doesn't feel right to just duplicate the unit tests and replace the types.
To be more specific, I'm working on a Matrix class and using Boost's unit testing framework. The Matrix is to work with different types. It will mainly be used for basic types, but I would like it to support concrete types too.
Note, I doing this for learning purposes, which is why I'm not using a existing matrix implementation.
Comment: "it doesn't feel right to just duplicate the unit tests and replace the types." why? if you want to test that it works with other types too, you cant do it without testing it with other types.
Answer: Boost test has a macro BOOST_AUTO_TEST_CASE_TEMPLATE that runs a test on a template for each type in a boost::mpl::list.
<code>template<typename T>
T Add(T lh, T rh)
{
   return lh + rh;
}

typedef boost::mpl::list<int, float> test_types;

BOOST_AUTO_TEST_CASE_TEMPLATE(Add_1Plus2_Is3, T, test_types)
{
    T result = Add(T(1), T(2));
    BOOST_CHECK_EQUAL(T(3), result);
}
</code>
http://www.boost.org/doc/libs/1_48_0/libs/test/doc/html/utf/user-guide/test-organization/auto-test-case-template.html
Answer: I agree with josuegomes and VJovic. I think, one way to minimize duplication of code is to make a templated function with your unit test logic and then call this with the different type arguments. This way, you are able to place all of your Matrix unit testing logic one place, while calling the templated unit test function once per each desired type.
However, it might be overkill to do so. I think it depends on the amount of logic in your tests.
Answer: The whole point of unit testing is to cover every branch of your code. If testing with different types covers different branches then it makes sense to add type specific unit tests, otherwise it is just copy-n-paste.
Answer: If you are unit testing a template, then you need to unit test the template class with all possible types, or until you think is enough. There are no ways around.
Your class might work with int, but fail for float.
If the template is not meant to be used with certain types, then you should add some static_assert in your code.


Reporting results of a Quade test

Question: I have been scouring the internet to find out how to report the results of a Quade test (non-parametric equivalent of ANCOVA).
Does someone have an example that explains this in plain terms? I'm not sure if I report an F test statistic, df etc.
Answer: The Quade-Test is basically an ANCOVA with ranks instead of the actual data. The test statistic is calculated in a comparable way (comparison of the total sum of squares and treatment sum of squares) and is F-distributed.
You can report the Quade-Test the same way as you would report an ANCOVA. An example in APA-style (maybe add an effect size too):

F(2, 14) = 10.620, p = .002

Reporting results of a Quade test


Read every 5th line until the end of the text file

Question: Hi could someone please give me an example of how to read through a text file to the end and read every 5th line as a string? I know how to read a specific line in the text file using
line = System.IO.file.readAllLines(filepath)(linenum)
and also using streamreader to read each line etc..
But I want to go through the whole text file and pick out every certain number of lines. Pretty sure it's got something to do with a loop but I'm not too clued up yet.
Answer: As files are not line based, you would need to read all the lines and pick out the ones that you want.
You can use the Where method with the overload that gives you the index of the item, and the Mod operator to determine where every fifth line is:
Dim lines As String() = _
  System.IO.File.ReadLines(filepath).Where(Function(line, i) i Mod 5 = 0).ToArray()

The number that you compare the expression to determines which lines you get. i Mod 5 = 0 starts at the first line and then every fifth from there, while i Mod 5 = 4 starts at the fifth line and then every fifth from there.
(The ReadLines method is better than the ReadAllLines for this, as it doesn't read all the lines into memory first, but returns an enumerator so that you can process the lines as they are read.)
Comment: Since the array would be zero indexed, shouldn't it be something like `(i + 1) Mod 5 = 0`?
Comment: @roryap: That depends on the interpretation of "every 5th". I just added an explanation of how that works.
Comment: @Guffa Your addition regarding `ReadLines` vs `ReadAllLines` is true but totally defeated by the use of `ToArray` which will process the operation eagerly "on the spot"
Comment: @Sehnsucht: Not at all, having the `ToArray` at the end only realises the final result as an array, it doesn't cause all lines from the file to be read into memory before processing them.


Failure of Maximum Likelihood Estimation

Question: Given $N<\infty$, $0<q<1$ (arbitrarily close to 1) and $\epsilon>0$ (arbitrarily small)
one can construct an example of a one parameter family of distributions
$P_\theta$, $\theta\in[0,1)$, on the unit circle (viewed as a group, namely the
unit interval with addition modulo 1) such that
$$
   P_\theta=\theta+P_0
$$
and such that if a sample of size $n\leq N$ is drawn from any $P_\theta$, then with probability $q$, the maximum likelihood estimate $\hat\theta$ of the parameter
$\theta$ will be the worst possible distribution in this family for which the data likelihood is still nonzero.
Here "worst possible" is understood as $|\theta-\hat\theta|$ is maximal
(and this probably implies that various distance measures like total variation distance, Kullback-Leibler distance are also maximized although I
have not checked this).
In particular it satisfies $P_{\hat\theta}(support(P_\theta))<\epsilon$ and
$P_{\theta}(support(P_{\hat\theta}))<\epsilon$.
The distribution $P_0$ depends on $N,q$ and $\epsilon$ and is pretty simple
but the density has one thin spike and the maximum likelihood estimate is
obtained by shifting $P_\theta$ so that the support of the spike contains as many points of the sample as possible while maintaining a nonzero data likelihood.  Details will be provided if requested.
Does this bother you?  If not, why can we still trust Maximum Likelihood estimation?  Where are the conditions spelled out under which it can be trusted?
Some details: $P_0$ is a convex combination of uniform distributions
$$
   P=\alpha U_{[0,\delta]}+\beta U_{(\delta,1/4)}+\gamma U_{[1/4,1/2]}
$$
supported on $[0,1/2]$. The idea is that $\delta$ is chosen very small
so that the piece wise constant density has a large spike at the left end of the support of $P_0$.
By translation invariance of the family $P_\theta$ we may assume that we are sampling from the probability $P_0$.
The claimed properties of the maximum likelihood estimate will be verified for all samples $X=(x_j)\subseteq[1/4,1/2]$.
The parameters $\alpha$, $\beta$, $\gamma$ and $\delta$ are chosen depending on $N,q,\epsilon$. Note that a random draw $x$ from $P_0$ will come from $U_{[1/4,1/2]}$, equivalently $x\in[1/4,1/2]$, with probability
$\gamma$.
First you choose $\gamma<1$ so close to one that any sample $X$ of size $n\leq N$ satisfies $X\subseteq[1/4,1/2]$
with probability at least $q$, i.e. $\gamma^N\geq q$.
Given $\alpha,\beta,\gamma$ we note that the density $f_0$ of $P_0$ satisfies $0<c<f_0<C$ on $(\delta,1/2]$, where the constants $c,C$ are independent of $\delta$, while $f_0\uparrow\infty$
on the support $S=[0,\delta]$ of the spike of the density $f_0$, for all
$0<\delta<1/8$ (explicit calculation).
By translation the same inequalities hold true for the density $f_\theta$
of $P_\theta$ on $\theta+(\delta,1/2]$ respectively the support $S(\theta)=\theta+[0,\delta]$ of the spike of the density $f_\theta$.
Now choose $\delta>0$ so small that $S(\theta)$ can contain at most one
sample point $x_j$ and the density $f_0$ of $P_0$ on the interval $[0,\delta]$ is very large. For $\theta>\min x_j$, the data likelihood is zero, since the sample point $\min x_j$ is not in the support of $P_\theta$.
Thus any likelihood maximizing $\theta$ must satisfy $\theta\leq\min x_j$
and so $S(\theta)$ contains no sample point or exactly one sample point,
namely the point $\min x_j$.
The density $f_0$ on the interval $[1/4,1/2]$ is bigger than on the interval $(\delta,1/4)$. From this it follows that
$\hat\theta=\min x_j-\delta$
maximizes the likelihood (if $\delta$ is chosen small enough so that the spike is large enough), since this $\theta$ shifts the point $\min x_j$
into the support of the spike while maintaining as many points as possible
in the interval $\theta+[1/4,1/2]$.
Thus $\hat\theta=\min x_j-\delta$ is a likelihood maximizer but may not be the only one. This gets us close to the claims.
To make $\hat\theta=\min x_j$ the unique maximizer make the density of $P_0$
on the interval $[0,\delta]$ sloping down with $f_0(0)>>f_0(\delta)$ and
$f_0(\delta)$ still very large. Then $f_\theta(\theta)>>f_\theta(\theta+\delta)$ so that shifting the point $min x_j$ to the locus of the maximum of the density $f_\theta$ ensures likelihood maximization.
Obviously some detailed calculations are necessary to verify that
$\hat\theta=\min x_j$ really does maximize the likelihood.
Comment: While this is an interesting result, it doesn't bother me as ML theory states their results only asysmptotically and one has to expect that in some occasions finite sample performance might be pretty bad. However, can you give a more specific example of such a family of distribution $P_\theta$?
Comment: Yeah, I can't make much of this without knowing how $θ$ relates to the actual distribution.
Comment: Can you give the details?
Comment: @chRrr I have added some details.
Comment: Do you have a reference for this construction,or is it yours?
Comment: @kjetil b halvorson No, I came up with it myself but as was writing this post noticed some inaccuracies, so that I actually do not have all the details worked out. I believe the argument goes through though and will check it soon.
Comment: As stated the argument is flawed since we cannot choose $\delta$ dependent on the sample. This can be fixed at the cost of some complications.
Comment: After checking all the details (and replacing the circle group with the real line as parameter space to get rid of some more problems) only the first of the two support inequalities turns out to be true.
Answer: No, this is not bothersome!  The short story: you have explicitly constructed a model that with very high probability will be a bad fit to data generated from the model, so the high spike close to zero in $P_0$ is artificial and not representative of the data.
The way this works is that with high probability the very high spike will not contribute any observations, but the spike will dominate contributions to the log likelihood function so that the spike in the estimated distribution will be placed above the minimum value of the data.  That will with high probability be the wrong place.
I did some calculations to illustrate this, I will not give all details. I started with choosing $q=0.99$, $N=15$ (but thinking of simulating a sample with $n=10$, say).  That gives $\gamma=0.9994$ and then distributing the missing probability equally gives $\alpha=\beta=0.0003$.  Then securing a dominating loglikelihood contribution from the spike, $\delta=10^{-13}$ will do.  That will give an insanely high spike, in a region with no data.  In short, as an approximate representation of the data (with $n=10$ observations) this model is nonsense.
A more formal analysis is difficult, but hardly necessary.  One reason is that computing the Fisher information for this model is "difficult": the derivative of the loglikelihood (with respect to the parameter $\theta$) is mostly zero, with delta function spikes at the borders between the three intervals defining the model.  Maybe I come back with some simulations.  If you replaced the three uniform parts of the model with three gaussian parts, you would get similar results, but a model easier to analyze.
This is a general phenomenon, it is always possible to construct "models" which will misbehave.  So in addition to inference principles like maximum likelihood, we need principles for helping model construction, to get sensible models.  Tukey said that models should be "bland" or "hornfree", meaning that the model itself should not import information (or too much of it) into the analysis.  The information in formal statistical analysis always come from two sources: the data, and the model.  Bayesians are not alone in importing non-sample information into their models!  A book-length treatment is this.
It is the statisticians responsibility to make models which do not import too much information into the analysis.  Your model decidedly has a very big horn!
Comment: This https://www2.warwick.ac.uk/fac/sci/statistics/crism/workshops/non-likelihood/davies_warwick.pdf  is a relevant web presentation which among others contain the Tukey citations.
Comment: @ kjetil b halvorsen A spiky family of densities would occur naturally if you suspect that the data generating distribution has an atom but you don't know where and approximate the atom with a spike in the density. Can we reject this as unnatural?
Comment: Can you propose a natural example? More to the point, approximating the atom with a spike works bad with maxlik methods, it would be better to use a discrete atom! That is done naturally, say, with rainfall modelling and an atom at zero. Using a spike its width and height is largely arbitrary, but makes a huge difference for maxlik. Better thgen use a discrete atom.


extract the date from the file name

Question: I have this code :
import sys
import os
import re
import fnmatch
import csv
import pandas as pd
import numpy as np
from dateutil.parser import parse as parseDate
from datetime import datetime, time, timedelta

file = open("0556_16-09-20114725356354 --- 16-12-2016 07-00-00.csv", 'r')
filename = os.path.splitext(os.path.basename(file))[0]
timestamp = filename[-19:]
print timestamp
date = timestamp[:10] + " " + timestamp[-8:].replace("-",":")
print date
newdatefile = parseDate(date, yearfirst=True)
print newdatefile

My objective is to extract the date from the file name and parse it and save it in a variable. What I want to get is something like this : 16-12-2016 07-00-00
This is the error I get :
Traceback (most recent call last):
  File "t.py", line 13, in <module>
    filename = os.path.splitext(os.path.basename(file))[0]
  File "/usr/lib/python2.7/posixpath.py", line 114, in basename
    i = p.rfind('/') + 1
AttributeError: 'file' object has no attribute 'rfind'

Any ideas about what could that be ?
Comment: You're passing a file object to `os.path.basename` rather than the filename string.
Answer: os.path.basename(file) you're passing a file object here, while it expects a string
Answer: You don't need to have os.path do it for you. Why not simply take the filename directly from the parameter you're passing to open:
filename = "0556_16-09-20114725356354 --- 16-12-2016 07-00-00.csv"
file = open(filename, 'r')

Comment: and read your file content =  file.read()

extract the date from the file name


Stepwise regression - what are the steps in STATA?

Question: This question will probably seem very stupid, but hey, econometrics and statistics were never really my strongest features!
For my BA, my professor adviced me to perform stepwise regression. My dependent variable is Hiv Prevalence (expressed between 0 and 1), whereas my independent variables include GDP per capita, school enrollment, unemployment, urban population rate, population growth, HCI, spending on healthcare. Everything should be estimated on mean. My question is, whether someone would be able to tell me:
1. Is there anything else that I should do after performing both forward and backward procedures in STATA? Will it be enough for me to look at the p-values and then create a regression only with does that turned out to be significant?
2. When it comes to diagnostics tests, which ones should I perform? Linearity, homoscedasticity, normality and something else?
3. What if my results turn out to be heteroscedastic, not linear etc?
What can I do?
I will be very greatful for all the answers!
Comment: As you can learn from https://stats.stackexchange.com/questions/20836/algorithms-for-automatic-model-selection stepwise regression *does not really work*, and the p-values, confidence intervals, and many of the other statistics that would come from such model would be incorrect. So the best thing you could do, is actually not use stepwise regression.
Comment: LASSO regression is the popular, modern alternative https://en.wikipedia.org/wiki/Lasso_(statistics) the linked thread also  discusses this.
Comment: Depending on the nature of your data, predictor selection might not even be necessary. How many observations are there? How many potential independent variables? What's the range of prevalence values? Do you have actual population and HIV numbers, not just prevalence? (In that case something other than standard linear regression might be better.) Are these data from a particular point in time, or are there multiple observations of the same location over time? Please provide that information by editing the question rather than leaving it in a comment, as it's then easier for others to find.
Comment: And what would you recommend me to use instead?
Comment: Is there any specific tutorial on how to apply it?
Comment: In my database, there's a total of 137 countries and there are 8 independent variables (GDP per capita, heathcare expenditure, school enrollment, HCI, Tax revenue, unemployment, urban population and population growth. The dependent variable meaning HIV prevalence is expressed between 0 and 1.
Comment: Do I need to do those typical diagnostic tests after lasso?
Answer: With 137 data points and only 8 predictors, there should be no need to do any predictor selection at all. A rough rule of thumb for ordinary least-squares regression is that you need about 10-20 observations per predictor to avoid overfitting. If your model doesn't include interactions among the predictors then you seem fine in that regard.
A danger in cutting down on the number of predictors is omitted-variable bias. If you omit a predictor that is associated both with outcome and with the included predictors in a linear regression, the coefficient estimates for the included predictors will be biased. It seems likely that most of your predictors are correlated with each other, so that would seem to be a serious risk in your case.
As a comment from @Tim rightly points out, if you do need to cut down on the number of predictors then stepwise regression is not a good choice. LASSO is a more principled approach, in which you penalize the magnitudes of the regression coefficients to help trade off against the overfitting that predictor selection entails. As this answer points out, even though LASSO typically performs better than other predictor-selection techniques, in situations like yours the best model will include all predictors provided that you avoid overfitting.
To learn more about LASSO, see An Introduction to Statistical Learning for a helpful introduction to that and many other techniques. LASSO is implemented in STATA and their website evidently links to video tutorials.
Be aware, however, that LASSO's choice of a set of predictors can be highly dependent on the particular data set at hand. When predictors are correlated with each other, a new data sample might well lead to a different set of chosen predictors. So be wary of jumping to a conclusion that any particular set of chosen predictors (whether by LASSO or other approaches) constitute the "really important ones." At best they are the important ones in your particular data sample.
In terms of diagnostic tests and what to do if they fail, see this page for a start. LASSO is a linear modeling technique so linearity is important to document. Resolving linearity problems typically requires some transformation of predictors or the outcome variable. As usually implemented LASSO doesn't provide p-values anyway, so normality of residuals isn't critical.
Comment: This is all good advice. If you do decide to try the LASSO route and your goal is inference (rather than model selection and prediction), you should use the second type of LASSO in Stata.