Expressing a hazard function with a Cox proportional hazards model

Question: I'm trying to wrap my head around Cox models as well as how to express them in the form of hazard functions when you have both categorical AND numeric covariates.  I need some assistance with this question below.
In a study, the following covariates were collected on a number of subjects.
• z1: Treatment status: 0 if control, 1 if treatment
• z2: Sex: 0 if male, 1 if female
• z3: Age in years
• z4: Height in inches
A Cox proportional hazards model will be constructed to estimate the effect of treatment on the hazard. It is thought that the effect of treatment may be different for men and women. It is further thought that hazard increases with age, but that the effect of treatment does not vary with age. However, the treatment group is slightly older, on the average, than the control group. It is not thought that the hazard function varies with height, but the control group is slightly taller, on average, than the treatment group, and men are somewhat taller, on average, than women.
Express the hazard in terms of the covariates and parameters.
How do you express the hazard in the form below when you have so many different covariates? I know that I need to include the baseline hazard but I'm unsure has to how many beta parameters I need, or what covariates need to be multiplied by those beta parameters.
Comment: This sounds like it might be for a homework problem or some type of self study. If so, please read the [info](https://stats.stackexchange.com/tags/self-study/info) for the tag, add that tag to the question, and say more about where you are stuck. For example, could you answer this question if the outcome were a continuous outcome in linear regression and not a Cox regression for survival times as outcomes?
Comment: I made the edit. Unfortunately, I'm still very much learning how to do this. I've seen a rather unclear example of how this might be  done via linear regression but I wasn't really able to understand it.  I know the form of the equation that answers this problem, but that's about it. I'm just not sure what to label my beta parameters or my covariates as.
Answer: This question apparently expects you to include all predictors that are reasonably related to hazard into the Cox model.
Remember that the inner product ${\bf z}^T {\bf \beta}$ between predictors and their corresponding coefficients (called the linear predictor) can also be written in the form:
$$ \sum_{i=1}^p \beta_i z_i$$
where $z_i$ is a predictor, $\beta_i$ is its corresponding regression coefficient, and $p$ is the total number of predictors. So there is no problem in including multiple predictors in your model, including interactions among your covariates as predictors, and so on. Just add more predictors on to that sum.*
This isn't really different from the situation in a standard linear regression, except that in a linear regression you usually include the intercept in the linear predictor. The baseline hazard $\lambda_0(t)$ takes the place of an intercept in a Cox regression, which you perhaps can see better by taking the logarithm of your equation for the hazard:
$$\log\lambda_{\bf z}(t) = \log\lambda_0(t) + {\bf z}^T{\bf \beta}.$$
With respect to continuous versus categorical predictors, think about what happens with a continuous predictor when it has a value of 0 versus when it has a value of 1. If a continuous predictor $z_i$ has a value of 0 it will add 0 to the linear predictor. If $z_i$ has a value of 1, it will add $\beta_i$ to the linear predictor.
Is there anything different from this if you instead have a 2-level categorical covariate $z_j$, with one level coded as 0 and the other as 1? At the reference level (coded 0) it adds nothing to the linear predictor, and at the other level (coded 1) it adds $\beta_j$.**
Try putting that information together into your answer. Your decision then is which of your covariates are expected to be associated with hazard and thus should be included in your model. You also should think about whether you might need to add any interactions between covariates as predictors, as you would if the effect of one predictor depends on the value of another predictor. (Note that including an interaction can provide a test of assumptions like "the effect of treatment does not vary with age.")

*In practice you can overfit a model if the ratio of cases to predictors is too small, but that doesn't seem to be an issue here.
**Things get a bit trickier if a categorical covariate has more than 2 levels; then you have to add an additional predictor to the model for each level beyond the second.
Comment: @Caladin00 you do need 1 $\beta$ for each covariate. So if you include all of age, sex, treatment, and height (with no interactions) you will need 4 of them. For a categorical covariate it doesn't matter which level gets coded 0 or 1, just affects interpretation of the coefficient and the baseline hazard. In your example, with treatment coded 0 for control and 1 for treated and a coefficient $\beta_t$, treatment adds 0 to the linear predictor for controls and $\beta_t$ for treated. Reversing the coding would just change both the baseline hazard and the meaning of $\beta_t$ accordingly.
Comment: @Caladin00 for which covariates to include, just go down the list and ask: is this covariate expected to affect hazard? If so, include it. The statement of the question seems to include that information for each of them. For interactions, ask: is the effect of covariate A on hazard thought to depend on the value of covariate B? If so, include an interaction between them. For example, if "the effect of treatment does not vary with age" then you don't need an interaction between them.
Comment: Gotcha! I think  I have an understanding of where to go with this now.
Comment: So I have the following so far..
```λ0(t)e^( β1Z1 + β2Z2 + β3Z3 + β4Z4 + β5Z1Z2)```
How does this look?
Comment: @Caladin00 two things to consider, as you handled the treatment:sex interaction nicely. First, think about why you are including height in the model. (If this is a homework problem you should consider the wording and assumptions of the problem as stated.) Second, note that the baseline hazard $\lambda_0(t)$ will mean different things depending on the baseline categories of factors and 0 levels of continuous predictors. Spell out exactly the situation that $\lambda_0(t)$ represents in the model as you wrote it. Then post your result, with explanation of your logic, as a new answer on this page.
Comment: I'll give it another shot and report back what I get. The wording  here is really what's throwing me off. _It is further thought that hazard increases with age, but that the effect of treatment does not vary with age. However, the treatment group is slightly older, on the average, than the control group. It is not thought that the hazard function varies with height, but the control group is slightly taller, on average, than the treatment group, and men are somewhat taller, on average, than women._
It sounds like they're telling me that there MIGHT be a connection....
Comment: But by the same rules I've been going with, if there is no reason to believe the hazard will be affected... I shouldn't add it to the model. How do I handle these ambiguous latter cases?
Comment: Yeah sorry I'm still very confused. I think I understand what you're trying to say but I have no clue how to put this together. I'm still confused how to tie in numerical and categorical covariates.
Comment: How does one determine if a categorical covariate is 0 or 1? And what about interactions between the covariates? I know I'm going to be adding quite a few beta parameters but I'm not sure what covariates to mix and match.
Answer: So here's what I came up with....
$\lambda_z(t)= \lambda_0(t) \exp(\beta_1z_1 + \beta_{_2}z_{_2} + \beta_{_3}z_{_3} + \beta_{_4}z_{_1}z_{_2} + \beta_{_5}z_{_1}z_{_3} + \beta_{_6}z_{_1}z_{_4} + \beta_{_7}z_{_2}z_{_4})$
I believe that I've accounted for every single condition and interaction between covariates accounted for in the paragraph below.
It is thought that the effect of treatment may be different for men and women. It is further thought that hazard increases with age, but that the effect of treatment does not vary with age. However, the treatment group is slightly older, on the average, than the control group. It is not thought that the hazard function varies with height, but the control group is slightly taller, on average, than the treatment group, and men are somewhat taller, on average, than women.
I'm still unsure as to whether I've listed my interactions correctly but this is a start.

Age ($z_{_3}$) was believed to affect the hazard so its included with its own beta parameter and is also used as an interaction on $\beta_{_5}$ with  treatment due to the treatment group being older than the control group.
Height ($z_{_4}$) was believed to not affect the hazard by itself so I've left it out but I included it in an interaction with    treatment($z_{_1}$) and sex ($z_{_2}$) on $\beta_{_6} $ and $\beta_{_7}$ respectively...
Comment: @EdM what do you think?
Comment: First, if you include a covariate in an interaction you should usually include it by itself. I read the question differently from you. When I read "It is not thought that the hazard function varies with height," I take that to mean it doesn't do so even via interactions with other covariates. If your model includes sex as a covariate and height doesn't matter, do you really care that men are taller than women? And if "the effect of treatment does not vary with age" is there any reason to include the $z_1z_3$ interaction term?
Comment: Part of the difficulty here is the difference between a real-life problem and an answer to a question for a course. It's often best practice to include interactions to start, for example to test the claim that "the effect of treatment does not vary with age." But answering a question for a course depends on following the stated assumptions, which in this case only seem to support the one interaction term that you correctly identified in a comment on my answer. Remember that instructors will sometimes include extraneous information to test your understanding of what's really important.
Comment: Gotcha. Thanks for your assistance with this. Major help! +1
Comment: To help others who might find this page, please update your own answer with your final version and how your thinking changed as you pursued it further. Also, consider specifying exactly what the baseline hazard $\lambda_0(t)$ represents in the model the way that you finally specify it. For example, unless age values have been otherwise centered, it represents (among other factors) the hazard for a newborn with age=0! That's OK and would give expected predictions for adult ages, but things can get particularly weird that way when there are interactions with continuous covariates.


how would you convert the following into yii2

Question: I am learning yii2 and wanted to know how would you convert the following into yii2 and still be responsive in I was view in a phone, tablet or pc.
This is the following code.

 CONCIERGE
SERVICES

I wanted to see how to convert it to the yii2 coding for example using the HTML::a() or the HTML::submitButton() tags found in the yii\helpers\Html etc.
Comment: This is something to do with html and css. Not yii i'm afraid
Comment: I was taking about like   `'Find Doctor/Health Professional', 'class'=>'img-responsive center-block']),
           ['/site/docsearch']);`
Comment: can u please edit your question so that it is very clear
Comment: I have updated the Question to try to be more clear.
Answer: for the anchor you can do this way

<?= Html::a('CONCIERGE
SERVICES
', ['consearch.php']) ?>


Update object value for view only

Question: So i have an array of objects that have a role. each role has underscores for spaces. Now i know the best way in most cases would be to simply change the value in the database to the correct format but i need it to have spaces.
So instead of the value to look like this:
role: this_is_my_role

I need it to look like this:
role: this is my role

but only for my view. I've tried the following but it does not work.
def view_info(request):
    device = Device.objects
    for item in device:
        item.role = item.role.replace('_', ' ')

    return render(request, 'table_view.html', {'device': device})

please note, i do not want to change the actual value in the database. I only want to change the value before rendering the template.
Comment: What is rendering now?
Comment: It just shows the value as if the loop wasn't there.
Comment: Did actually loop executed? Try print statement or debug
Comment: well it must do right?
Answer: This isn't really the right approach. You have data in one format and you want to display it in another format. You can either do this in the template itself, with a custom template filter, or provide a property on the model which outputs the converted data. The second is probably the easiest:
class Device(models.Model):
    ...
    @property
    def role_output(self):
        return self.role.replace('_', ' ')

Now you can simply pass the queryset to the template, and there do {{ device.role_output }} to get your converted value.

Update object value for view only


Hausdorff space which is not Urysohn space.

Question: Let $X=\mathbb{R}\cup \{\infty\}$ and for every $p\in \mathbb{R}-\{0\}$, $p$ has the usual neighborhoods in $\mathbb{R}-\{0\}$. Basic neighborhoods of $0$ is the sets of the form $\{[0, \frac{1}{n}):n\in\mathbb{N}\}$ while basic neighborhoods of $\infty$ are the sets of the form $\{(-\frac{1}{n}, 0)\cup\{\infty\}: n\in\mathbb{N}\}$.
I think $X$, endowed with this topology, is a Hausdorff space.
Can We say that  for every closed neighborhood $N$ of diagonal $X$,  $\Delta_X$ , in $X\times X$,  $(0, \infty)\in N$?
Comment: Welcome to MSE. Your question is phrased as an isolated problem, without any further information or context. This does not match many users' quality standards, so it may attract downvotes, or closed. To prevent that, please edit the question. This will help you recognise and resolve the issues. Concretely: please provide context, and include your work and thoughts on the problem. These changes can help in formulating more appropriate answers.
Comment: @SolubleFish, Ok, thanks for it.
Comment: why the uniform spaces tag?
Answer: It's clear that $\overline{U} \cap \overline{V} \neq \emptyset$ whenever $U$ is an open neighbourhood of $0$ and $V$ one for $\infty$. This implies the property you ask about. The above first fact is already a direct proof that $X$ is not Urysohn.


Masters in Numerical Analysis?

Question: My interests in CS are fairly narrow; I'm mainly interested in numerical analysis and numerical methods for solving differential equations. However, during my undergrad I mainly focused on other, more traditional aspects of CS. I don't have as strong a preparation in the applied math side required for a PhD in Applied Math/Numerical Analysis. A class I took my senior year (second semester!) got me really interested in numerics and I've spent much of my free time since then self-studying the material.
Are there any good CS programs where I can get a good background in applied math/numerical analysis and largely avoid topics like graph theory and combinatorics? With my (relatively) weak preparation in formal math, I think a good math masters is not in the cards.
Comment: Have you considered an advanced degree in "Computational Science"?
Comment: From my research it seems that very few schools offer such a specialized degree.
Comment: But if that is what you want to do, why not move there?
Comment: You have a good point, and I'm willing to make the move if need be. My eventual goal is to do a PhD in the field and I'm not sure how well these programs place. Many of them seem geared for industry-bound engineering types.
Comment: You should ask a trusted professor about this. They will have more information about your preparation and abilities, so they will be able to advise you much better than anonymous people on the internet ever could.
Comment: I am working on a posteriori error estimation in differential form finite elements for Maxwell's equations, I happen to know some interdisciplinary PhD programs under the computational science centers. Many of them are doing researches in how to solve large scale partial differential equation problems (electromagnetics, fluid dynamics, or even multiphysics simulation). Are you interested in this?
Comment: Shuhao, sounds potentially interesting. Do you see that most computational science PhDs work on applied numerical analysis rather than theory? Is theoretical NA a mostly complete field?
Comment: Theoretical numerical analysis is sure a completely field, as well as the simulation/implementation part. Recent trends include but not limited to: designing robust model for multiphysics, large scale linear system solvers like LAPACK but more problem oriented (e.g. Maxwell, Navier-Stokes), parallel/high-performance/gpu computing, adaptive mesh generation/automated PDE solution, inverse problems.
Answer: Many schools offer masters in programs that are not specific to just Cs and math. If you're interest in CS is "fairly narrow" as you described it, I wouldn't look into a masters in CS, I would look into a masters in data computing/statistics or even financial Mathematics. Most of these topics do not require combinatorics and graph theory as pre-reqs or requirements.
Answer: There is a program offered by UCSD which is a specialization of Computer Science and Applied Mathematics. It is supposed to be a really good program.

Masters in Numerical Analysis?


C# Entity - Delete Items by ID - ID to delete are in an other list

Question: I need to delete items in a collection managed by Entity 6. The items that have to be deleted are in an other list
Currently I do this :
 idToDelete = model.Courses.Where(x => x.Deleted).Select(x => x.Id);

 entity.Courses
        .Where(ent => idToDelete.Contains(ent.Id))
        .ToList()
        .ForEach(ent =>
        _contexte.Entry(ent).State = EntityState.Deleted);

This code works fine.
How to perform the same in only one linq instruction?
Answer: entity.Courses.RemoveRange(entity.Courses.Where(e => e.Deleted));

Actually, reading your question again, it looks like the list of IDs is in a view model, and possibly your entity model doesn't have the Deleted property. So presumably you need something rather more like what you had:
_contexte.Courses.RemoveRange(
    _context.Courses.Where(c => model.Courses.Where(x => x.Deleted).Select(y => y.Id).Contains(c.Id));

...or more readably:
var idsToDelete = model.Courses.Where(c => c.Deleted).Select(e => e.Id);
var entitiesToDelete = _contexte.Courses.Where(c => idsToDelete.Contains(c.Id));
_contexte.Courses.RemoveRange(entitiesToDelete);

Answer: You can use let to declare your idToDelete within the query:
(from ent in in entity.Courses
 let idsToDelete = model.Courses.Where(x => x.Deleted).Select(x => x.Id)
 where idsToDelete.Contains(ent.Id)
 select ent).ToList()
 .ForEach(ent => _contexte.Entry(ent).State = EntityState.Deleted);


Take screenshot element in selenium on ubuntu wrong position

Question: When I change the environment from Windows 10 to Ubuntu and some function is not working good. Like that function take a screenshot
Please help me find the solution take a screenshot on Ubuntu. When take a screenshot on Ubuntu the picture I think is the wrong position of the element. Thank you
Working for Windows 10
<code> public void captureElementScreenshot(WebElement element, String fileName) {
    //Capture entire page screenshot as buffer.
    //Used TakesScreenshot, OutputType Interface of selenium and File class of java to capture screenshot of entire page.
    File screen = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);

    //Used selenium getSize() method to get height and width of element.
    //Retrieve width of element.
    int ImageWidth = element.getSize().getWidth();
    //Retrieve height of element.
    int ImageHeight = element.getSize().getHeight();

    //Used selenium Point class to get x y coordinates of Image element.
    //get location(x y coordinates) of the element.
    Point point = element.getLocation();
    int xcord = point.getX();
    int ycord = point.getY();

    //Reading full image screenshot.
    BufferedImage img = null;
    try {
        img = ImageIO.read(screen);
        BufferedImage dest = img.getSubimage(xcord, ycord, ImageWidth, ImageHeight);
        ImageIO.write(dest, "png", screen);

        //Used FileUtils class of apache.commons.io.
        //save Image screenshot In D: drive.
        FileUtils.copyFile(screen, new File(System.getProperty("user.dir") + "//src//test//screenshot//" + fileName + ".png"));
        System.out.println("Success save file " + fileName);
    } catch (IOException e) {
        e.printStackTrace();
    }

    //cut Image using height, width and x y coordinates parameters.

}
</code>
And this config browser
<code> System.setProperty("webdriver.chrome.driver", System.getProperty("user.dir") + "//src//test//browser//chromedriver");
 DesiredCapabilities capabilities = DesiredCapabilities.chrome();
 ChromeOptions options = new ChromeOptions();
 options.addArguments("force-device-scale-factor=0.75");
 options.addArguments("high-dpi-support=0.75");
 capabilities.setCapability(ChromeOptions.CAPABILITY, options);
 driver = new ChromeDriver(capabilities);
 driver.manage().window().maximize();
</code>
Answer: It's about two line code. I have commented and It's working
<code>    options.addArguments("force-device-scale-factor=0.75");
 options.addArguments("high-dpi-support=0.75");
</code>


Como baixar um BD pelo console?

Question: Preciso baixar um banco de dados na minha máquina, mas ela não aguenta o MySQL Workbench.
Existe algum outro programa mais leve que o Workbench ou quais os comandos pra eu baixar o BD que eu quero?
Comment: Uma boa opção é o "antigo" MySQL Query Browser, caso vc queira uma GUI.
Answer: Use o MySQLdump:
mysqldump [dbname] -h [host] -u [user] -p > [endereço onde salvar o dump]
pressione enter
digite a senha e pronto

Como baixar um BD pelo console?


Help understanding definition of Darboux integral $U(f)$.

Question: My book defines the upper and lower Darboux sums $U(f,P)$ and $L(f,P)$ respectively then follows up with a confusing definition of the upper and lower Darboux integrals $U(f)$ and $L(f)$ respectively. Before I ask my question I should define some notation:
Here we partition the interval $[a,b]$ by the subset $P$ having the form $P = \{ a=t_0<t_1<\cdots <t_n = b\}$. Also, $M(f,S) = \sup\{ f(x):x\in S\}$ and $m(f,S)=\inf\{ f(x):x\in S\}$.
The definition of the upper Darboux sum is :

$U(f,P) = \sum_{k=1}^{n}M(f,[t_{k-1}, t_k])\  (t_k-t_{k-1})$

The definition of the lower Darboux sum is :

$L(f,P) = \sum_{k=1}^{n}m(f,[t_{k-1}, t_k])\  (t_k-t_{k-1})$

Now, here comes my question. When the book defines the upper Darboux integral it says:
$U(f) = \inf\{ U(f,P) : P$ is a partition of $[a,b]\}$.
Why does it switch to say inf instead of sup like in the previous definition of the sum? How can I conceptually/intuitively understand the difference between the sum and integral? Do you guys have any other literature I could read up on to make me feel more comfortable about this topic?
Thank you!
Comment: All the upper sums $U(f,P)$ are larger than the intended integral.  So to get something that should be called the integral, take their infimum.  Work out some examples.  Say $f(x) = x$ on $[0,1]$.
Answer: Infimum of the set of upper sums decreses as you add partitions and you want the smallest upper "approximation". Further the integral is a real number which is obtained as inf/sup of a sets of sums, it is therefore is is not a sum.


JOOQ use Record7 to encapsulate fields from different tables

Question: I'm using JOOQ to retrieve records from 2 tables, i created custom object NearByPlaceof returned columns, i read about Record7and see that it's much better to use it rather than creating new object as below, i tried it but it failed in compilation saying that can't convert from Org.JOOQ.Record7 to record of below aliases, any idea of how to do that?
public List<NearByPlace> getNearBy(NearByRequestWrapper wrapper) {

return db().select(
    PLACE.NAME.as("placeName"),
    PLACE.TYPE.as("placeType"),
    PLACE.LAT.as("lat"),
    PLACE.LON.as("lon"),
    EVENT.NAME.as("eventName"),
    DSL.field("earth_distance(ll_to_earth(" + wrapper.lat() + "," + wrapper.lon() + "),ll_to_earth(place.lat, place.lon))* 0.000621371192").as("distanceToReach"),
    DSL.field(EVENT.TYPE).as("eventType"))
    .from(PLACE)
    .leftJoin(EVENT)
    .on(PLACE.ID.eq(EVENT.LOCATION_ID))
    .where(PLACE.ID.eq(1))
    .fetchInto(NearByPlace.class);

}

Comment: I can't see where you're trying to use Record7.  It looks like you're trying to load the result into a List of NearByPlace.  as long the object is able to construct it you should be fine.  You can see examples here:  https://www.jooq.org/doc/3.10/manual/sql-execution/fetching/pojos.

You can also just do a .fetch() get a list of Record7 and manually construct the list as you see fit.  If that makes sense?
Comment: yes, i want to use Record7 along with .fetch(), any idea of how to do that ?
Answer: if you want to work with jOOQ's native RecordX types, simply invoke fetch() rather than fetchInto(Class).
in this case, that should yield an org.jooq.Result<Record7<...>>
UPDATE
updated sample code as requested.
(note: this example assumes what the type parameters for Record7 should be based on the names of columns and what i think they mean. if i've guessed wrong, compilation will fail, but this should be a straightforward problem to fix.)
        final org.joog.Result<Record7<String,String,Double,Double,String,String,String>> result = db().select(
                    PLACE.NAME.as("placeName"),
                    PLACE.TYPE.as("placeType"),
                    PLACE.LAT.as("lat"),
                    PLACE.LON.as("lon"),
                    EVENT.NAME.as("eventName"),
                    DSL.field("earth_distance(ll_to_earth(" + wrapper.lat() + "," + wrapper.lon() + "),ll_to_earth(place.lat, place.lon))* 0.000621371192").as("distanceToReach"),
                    DSL.field(EVENT.TYPE).as("eventType"))
                .from(PLACE)
                    .leftJoin(EVENT)
                        .on(PLACE.ID.eq(EVENT.LOCATION_ID))
                .where(PLACE.ID.eq(1))
                .fetch();

        return result;

Comment: could you please edit the sample above to reflect your thoughts ?


What is a "Bourbaki Talk"?

Question: I've seen lately many people using this concept. I have searched the web, but there's no answer.
Anyone knows what does a "Bourbaki Talk" mean?
Note: I am knowledgeable about the Bourbaki group. Nevertheless, I have never heard of the concept "Bourbaki Talk".
EDIT (after answer): A recent example is here: https://valuevar.wordpress.com/2017/01/14/bourbaki-talk/
Comment: Are your referring to the "Séminaires Bourbaki" ? http://www.bourbaki.ens.fr/
Comment: (yes, Bourbaki seminars are held every year)
Comment: You can see [these explanations](https://en.wikipedia.org/wiki/Séminaire_Nicolas_Bourbaki). The 'Séminaire  Bourbaki' happens four times a year,  at the Institut Henri Poincaré in Paris (january, march, june and november, on saturdays). The talks are not necessarily in French.
Comment: I'm not the best person to talk about "Séminaires Bourbaki" but i) Bourbaki is a fictitious mathematician, see Will Jagy's link, ii) he gave his name to seminars, created in 1952 and mostly in French. I guess that saying "I've given a Bourbaki talk" means "I have given a talk in Séminaire Bourbaki", which is very prestigious (I guess).
Comment: I don't know them. These are announcements on the web!
Comment: Can you place this into context?
Comment: https://en.wikipedia.org/wiki/Nicolas_Bourbaki
Comment: I don't believe this has any specific context. People who announce their seminar talk or whatever, simply describe it as "I'm giving a Bourbaki talk". Thus, I'm quite certain that for these people this is a self-evident concept. Of course they are completely wrong, but it is usual that people often assume that what they know is universal and well-known.
Comment: Thanks Jagy. But how does this explains it? I have read even a book about the Bourbaki group, and still I have no idea what this concept means. I can guess it might mean some very rigorous talk, but this is a guess.
Comment: See @anderstood's comment.
Comment: Thanks anderstood. I don't read french, sorry. (And haven't seen it written in french or by french people). What do you mean?
Comment: No, this is not the meaning. People are saying this NOW, at THIS moment. They announce a future "Bourbaki Talk". Is this seminar series still going on?
Comment: But why don't you ask them what they mean when they say it?!?!
Comment: @MarianoSuárez-Álvarez, ah, you are correct! They mean the specific french seminar!!!!
Comment: You should provide context. As it is, your question is unanswerable: you are asking what some people who you do not mention mean when they say something somewhere on the web. Do you have a link, a name, something? Do you see how this is somewhat absurd?
Comment: Provided the context now.
Comment: That link you included quite clearly explains what a Bourbaki talk is...
Comment: Hence, I explain it was discovered after the answer. The initial announcments did not include explanations of the term.
Answer: Due to the comments I found the answer (the problem was that on the web it is referred to always in French, and not English, hence it is virtually undiscoverable, except for French speakers) wikipedia:
"The Séminaire Nicolas Bourbaki (Bourbaki Seminar) is a series of seminars (in fact public lectures with printed notes distributed) that has been held in Paris since 1948. It is one of the major institutions of contemporary mathematics, and a barometer of mathematical achievement, fashion, and reputation. It is named after Nicolas Bourbaki, a group of French and other mathematicians of variable membership."
A Bourbaki talk is always given by a speaker about other people's work. It is accompanied by an expository paper explaining difficult recent material.
Comment: "A Bourbaki talk is always given by a speaker about other people's work." Well, there were well-known exceptions to this in the past...

What is a "Bourbaki Talk"?


Why can Wordpress ship with latest version of jQuery but Drupal cant?

Question: Just a general question to as what holds Drupal back from being able to use the latest version of jQuery? I am guessing it is something technical but would like some details.
Even jquery update module only updates a few versions for Drupal 7 it goes for 1.4.4 to 1.5.2 while Wordpress ships with 1.7.1

Drupal 6 to jQuery 1.3.2
Drupal 7 to jQuery 1.5.2 and jQuery UI 1.8.11
Comment: You can add one more jQuery copy for your Drupal pages, if needed. For example, core will work with old version by you will write javascript code and use plugins for jQuery 1.7.1 on pages you need.
Comment: @kalabro -- I'm not a big fan of this method, it can often cause pretty big script errors.
Answer: My understanding is that the version of jQuery gets locked down with the version of Drupal during development to ensure stability. It is all part of the testing process. The Drupal developers fully test the system before releasing a new major version of the CMS, including the JavaScript functionality. The incremental changes that go on throughout the year do not provide the time to test at this level so the jQuery version stays the same.
I am not very familiar with Wordpress, but my understanding is that they make considerably more regular major upgrades to the versions of their CMS. So jQuery gets updated more frequently. However, from what I've read little testing takes place.
While getting stuck with an old version of jQuery can be frustrating, I'd rather go for a stable well tested system.
Comment: That link came via http://john.albin.net/drupal/dont-get-cocky-kid which is also a good read.
Answer: The project page for JQuery update module is not quite up to date. Have a look at this issue / patch:  -- It's still be worked on so use with care but it looks promising and it looks like it's getting attention and recent as well.
In your tags you did not specify what version of Drupal you are using though, the issue above is for Drupal 7. (That's one of the gotchas of Drupal, sometimes lurking around in the issue queues can yeield some nice finds and solutions that may not be evident yet on a module/s project page.


How to convert csv file data into NSMutabledictionary

Question: I want to parse the csv data with live link and convert the same in NSMutabledictionary.Here is the link that i am using for csv file.
http://ichart.yahoo.com/table.csv?s=BAS.DE&d=2&e=30&c=2012
Please help me out for the same.
Thanks in advance.
Comment: http://stackoverflow.com/questions/3344628/where-can-i-find-a-csv-to-nsarray-parser-for-objective-c/3380724#3380724..visit this post
Answer: This link should have what you need. But, do you download that link as file into your app directory? If so, after importing parseCSV, it is fairly easy to go through the lines of the csv file:
CSVParser *parser = [CSVParser new];
NSString *csvFilePath = [[NSBundle mainBundle] pathForResource:@"myCsvFile" ofType:@"csv"];
[parser openFile:csvFilePath];
NSArray *csvContentArray = [parser parseFile];

for (int i=0; i<csvContentArray.count; i++)
    NSLog(@"parsed %d %@", i, [csvContentArray objectAtIndex:i]);

[parser closeFile];


Check if my time series is forecastable using Shannon entropy

Question: According to this answer: https://datascience.stackexchange.com/a/95232/141037, is possible to verify the forecastability of a time series using the Shannon entropy, the lower the Shannon entropy value, the more forecastable it is. My question is: the time series I use for forecasting tests (regression problem) is windowed (shape (videos, frames, frame height, frame width), I applied a sliding window to it). Can I use this windowed time series to calculate shannon entropy?
Comment: For example, you calculate entropy and you get 42.0.... now what?
Comment: It is the case in many applied predictive situations that 'ground truth' for the model either isn't available or, more simply, does not exist. In such cases model validation is obtained from the usefulness of the results: does it predict? are greater (lesser) values of a metric helpful or insightful? And so on. Wrt time series information theoretic metrics such as *permutation entropy* may be more useful than Shannon entropy. See Brandmaier's papers on PE https://www.jstatsoft.org/article/view/v067i05
Comment: Sure you can. What are you going to do with that entropy? Entropy here is a statistic that is derived from your data. What you need to understand is what values of this statistic are important for you, and whether this statistic is sensitive enough to things you care about. My advice would be to generate some synthetic data which is either completely noise, or time-correlated. Process that data as you wish (windowing etc), calculate entropy, and then decide whether the obtained signal is useful
Comment: @Cryo I'm thinking about analyzing different sections of my windowed time series to see which are the most forecastable, so that I can then test these most forecastable parts via machine learning. Thanks.
Comment: Makes sense. It feels to me like you will still need to know what meaningful difference in entropy is. So it might be a good idea to simulate what you are planning to do, to understand what entropies to expect
Comment: This will answer your 'can I' question as well
Comment: Cool. What do you mean by to simulate what I'm planning to do, to understand what entropies to expect?
Answer: You can apply any technique you want to your data, the trouble will always be with interpreting the results. Say you compute entropy, how do you know which entropy is significant? How do you know one signal's entropy is substantially larger/smaller than another?
Besides analytical derivations, here one can take an empirical approach. What type of time-series are you expecting to forecast? Say something similar to ARMA with one independent variable:
$$
\begin{align}
y_t-\phi \cdot y_{t-1}&=\epsilon_t-\theta \cdot \epsilon_{t-1} + \eta\cdot x_t \\
\left(\begin{array} \\
1 & -\phi & 0 & 0 & \dots & 0 & 0 & 0\\
0 & 1 & -\phi & 0 & 0 & \dots  & 0 & 0\\
\vdots & \ddots & \ddots & \ddots & \ddots & \dots & 0 & 0\\
\vdots & \ddots & \ddots & \ddots & \ddots & \dots & 0 & 0\\
\vdots & \ddots & \ddots & \ddots & \ddots & 1 &-\phi & 0\\
0 & 0 & 0 & \ddots & \ddots & \dots & 1 & -\phi\\
0 & 0 & 0 & 0 & 0 & \dots & 0 &1 \\
\end{array} \right)
\left(\begin{array} \\
y_t \\
y_{t-1} \\
\vdots
\\
\vdots
\\
\vdots
\\
\vdots
\\
y_2
\\
y_1
\end{array} \right)&=
\left(\begin{array} \\
\epsilon_t-\theta\cdot \epsilon_{t-1}+\eta\cdot x_t \\
\epsilon_{t-1}-\theta\cdot \epsilon_{t-2}+\eta\cdot x_{t-1} \\
\vdots
\\
\vdots
\\
\vdots
\\
\vdots
\\
\epsilon_2-\theta\cdot y_1+\eta\cdot x_2
\\
\epsilon_1+\eta\cdot x_1
\end{array} \right)
\end{align}
$$
Where $\epsilon_{\dots}$ is your noise, $x_{\dots}$ is the independent variable, $y_{\dots}$ is dependent (and is zero for $t\le 0$). One can re-arrange the above to:
$$
\left(\begin{array} \\
y_t \\
y_{t-1} \\
\vdots
\\
\vdots
\\
\vdots
\\
\vdots
\\
y_2
\\
y_1
\end{array} \right)=
\left(\begin{array} \\
1 & \phi & \phi^2 & \phi^3 & \dots & \phi^{n-1} & \phi^{n-2} & \phi^{n-1}\\
0 & 1 & \phi & \phi^2 & \phi^3 & \dots & \phi^{n-1} & \phi^{n-2} \\
\vdots & \ddots & \ddots & \ddots & \ddots & \dots & \dots & \dots\\
\vdots & \ddots & \ddots & \ddots & \ddots & \dots & \dots & \dots\\
\vdots & \ddots & \ddots & \ddots & \ddots & 1 &\phi & \phi^2\\
0 & 0 & 0 & \ddots & \ddots & \dots & 1 & \phi\\
0 & 0 & 0 & 0 & 0 & \dots & 0 &1 \\
\end{array} \right)
\left(\begin{array} \\
\epsilon_t-\theta\cdot \epsilon_{t-1}+\eta\cdot x_t \\
\epsilon_{t-1}-\theta\cdot \epsilon_{t-2}+\eta\cdot x_{t-1} \\
\vdots
\\
\vdots
\\
\vdots
\\
\vdots
\\
\epsilon_2-\theta\cdot y_1+\eta\cdot x_2
\\
\epsilon_1+\eta\cdot x_1
\end{array} \right)
$$
Where we assume $n$ time-steps. So all you need to do now is generate random $\epsilon$, choose $\theta,\phi$ and $x_t$. Then plug everything into the vector on the right, multiply it by the matrix (formally the inverse of a Lag Operator matrix). And you will get time-series that is correlated as per your spec. You can then apply all the procedures you want on it, and test to see whether those procedures will allow you to distinguish between predictable (mostly controlled by large $\eta$ and $\phi$) and un-predictable time-series. You can even get an empirical distribution of your entropy (by repeating this process many times.)
This is just one example of course. There are other ways you can generate time series. This one is nice because it is analytically tractable
Comment: Quick link on the inverse of a Lag operator, in case it helps https://ealdrich.github.io/Teaching/Econ211C/LectureNotes/Unit1-ARMA/lagOperators.html#inverse-of-lag-operator


Find the derivative of the polar curve $r = 4\theta + \sin\theta$

Question: I have been trying to solve this equation: Find the derivative $\frac{dy}{dx}$ of the polar curve $r = 4\theta + \sin\theta$ and I'm not sure where to go with this. Anyone mind giving a hand?
Answer: HINT:
$x = r \cos \theta = (4 \theta + \sin \theta) \cos \theta$
$y = r \sin \theta = (4 \theta + \sin \theta) \sin \theta$
$\displaystyle \frac{dy}{dx} = \frac{dy}{d\theta} / \frac{dx}{d\theta}$
Can you take it from here?
Answer: By the Chain Rule, we know that:
$$\frac{dy}{dx}\frac{dx}{d\theta} = \frac{dy}{d\theta}$$
Thus:
$$\frac{dy}{dx} = \frac{\frac{dy}{d\theta}}{\frac{dx}{d\theta}}$$
However, we know that $x = r\cos(\theta)$ and $y = r\sin(\theta)$. Thus, we have:
$$\frac{dy}{dx} = \frac{\frac{d}{d\theta}(r\sin(\theta))}{\frac{d}{d\theta}(r\cos(\theta))}$$
Doing some straightforward product and chain rules (remember that $r$ is also a function of $\theta$):
$$\frac{dy}{dx} = \frac{\frac{dr}{d\theta}\sin(\theta) + r\cos(\theta)}{\frac{dr}{d\theta}\cos(\theta) - r\sin(\theta)}$$
Can you take it from here? If you're still stuck:

 Taking derivatives, we find:

$$\frac{dr}{d\theta} = 4 + \cos(\theta)$$

 Thus, we have:

 $$\frac{dy}{dx} = \frac{(4 + \cos(\theta))\sin(\theta) + (4\theta + \sin(\theta))\cos(\theta)}{(4 + \cos(\theta))\cos(\theta) - (4\theta + \sin(\theta))\sin(\theta)}$$
$$\frac{dy}{dx} = \frac{4\sin(\theta) + 4\sin(\theta)\cos(\theta) + 4\theta\cos(\theta) + \sin(\theta)\cos(\theta)}{4\cos(\theta) + \cos^{2}(\theta) - 4\theta\sin(\theta) - \sin^{2}(\theta)}$$
$$\frac{dy}{dx} = \boxed{\frac{8(\sin(\theta) + \theta\cos(\theta)) + 5\sin(2\theta)}{8(\cos(\theta) - \theta\sin(\theta)) + 2\cos(2\theta)}}$$
Comment: Thank you, I got it down! I appreciate the help


How to mix probability estimators of the same phenomenon?

Question: Also posted here and here.
I have the following problem:
I have N models that give me an estimation of the probability distribution function p(x) of a certain phenomenon x. Let's call them: $p_1(x),...,p_N(x)$. They come from different sources of information, so they can give different values, but they all refere to the same observable fact. Is there a formal and valid way to combine them into a single formula?
I have read about mixture distributions, are they applicable to my case?
Suppose that my models can be more or less reliable, depending on the source of information they are based on, can I also combine them giving more weight to one model rather than another?
One possible solution I have thought is to make a weighted average of all the PDFs: $p(x) = w_1*p_1(x) + ... + w_N * p_N(x)$ where $\sum\limits_{i=0}^N w_i = 1$, does it make sense?
Thanks very much for your suggestions!
PS: to be more concrete my models give me the probability distribution that a certain person is in the position (x,y) and they rely on different sources of information like the power of a received signal or some other observable fact.
Answer: I think Bayesian model comparison might be what you are looking for. See for example Bishop, Chapter 3.4.
Generally speaking, given a set of $N$ models, you choose your weights to correspond to the posterior probability of each model.
$$ p(M_i | D) \propto p(M_i)p(D | M_i) $$
where $p(M_i)$ is the prior of model importance, you can assume this to be uniform, and $D$ is your data. Hence $p(D | M_i)$ is simply the likelihood of model $i$ given data.
The predictive probability for a new value $y^*$ and explanatory values $\mathbf{x}$ is then:
$$ p(y^* | \mathbf{x}, D) = \sum_{i=1}^N p(y^*|\mathbf{x},D,M_i)p(M_i | D)$$
where we use the posteriors as weighting between models.
That's the theory. Now in practice, unless you happen to be dealing with conjugated probabilities, you won't get a closed form solution for those posteriors and the above is pretty much useless.
When people fit a mixture of distributions, they usually use a mixture of Gaussians and in rare occasions a mixture of t distributions. I think the reason is simply that the mixture is fit using the EM algorithm (again, see Bishop) and Gaussians are particularly useful since their posterior is again a Gaussian and you can get all the required updates for the EM algorithm in closed form solution. And when I say "fit", they don't fit them individually, but learn the best parameters for all mixture components from the data, which is not what you are doing.
Don't worry about that for now and simply check whether you can get the posteriors for your model, or whether you don't want to fit your mixture using the EM and some well known distribution, such as the Gaussian, or t distribution in case of many outliers.
EDIT (to your comment):
So first of all, my notation: $y$ is the quantity you are trying to model, $x$ is a vector of data used to model $y$.
The data $D$ is just a tuple of vectors $x$, used to model a tuple of $y$'s. (basically think any normal dataset with a dependent variable and multiple independent ones)
OK. now, if I understand correctly you are trying to fit a complex distribution and you somehow already know the models that explain the data (perhaps because you know the generating mechanism) so you only need the mixing proportions.
You could try the following, which is guaranteed to work for Gaussians, and I don't see why it shouldn't work for other distributions too. (though there is a big caveat here!)
Calculate for each point x the likelihood that the point was generated by model $f1$ and $f2$, where the parameters are known. You end up with a matrix of likelihoods $2 x N$ where N is the number of your points. sum each row and divide by N. You should get the responsibility of each of the models for generating the data.
Use that to weight your mixture. You should also check if the resulting density integrates to 1, and if not normalize appropriately.
Comment: Thanks for the answer! I am still puzzled by the meaning of x,y, and D though. Let me put a concrete example: $f_1(x)$ is an estimation of the PDF that does not change over time, it's sort of a priori information I have, and $f_2(x)$ is an estimation that employs some data of the observable world that change over time. Supposing that none of the two is a perfect estimator, that is, they can have some error, or they can be quite uniform, how can I combine them? $f=(f_1 + f_2)/2$ ?


Python - sumif from csv

Question: There is a column in my csv called cost, which I want to sum based on another column, called factory, to basically create a breakdown of cost by factory. I have rows such as the following, where there are multiple costs for each factory:
Factory,Cost,Cost_Type
Bali,23,0
Sydney,21,1
Sydney,4,2
Denver,8,1
Bali,9,1

I'd like to be able to quickly sum the cost per factory, and save these values to a variable. I think one way to do this is by looping through a list of factories, which then loops through the csv. Here is where I've got to:
factories= ['Bali', 'Sydney', 'Denver']
totalcost = 0
balicost = 0
sydneycost = 0
denvercost = 0

for factory in factories:
    for row in csv.reader(costcsv):
        if row[0] == factory:

Where I'm stuck is that I don't know how to change the variable which is being added to for the different factories, balicost, sydneycost and denvercost. The simplified version, where I'm just getting the total of the cost column was as follows:
for row in csv.reader(costcsv):
        totalcost += float(row[1])

I'm more than open to different approaches than this (I believe dictionaries could come into it), and appreciate any points in the right direction.
Answer: [Community wiki, because it's a little tangential.]
When you're processing tabular data in Python, you should consider the pandas library.  The operation you want to perform is a groupby sum, and that's easily done in two lines:
df = pd.read_csv("factories.csv")
by_factory = df.groupby("Factory")["Cost"].sum()

which produces a Series object you can index into like a dictionary:
>>> by_factory
Factory
Bali       32
Denver      8
Sydney     25
Name: Cost, dtype: int64
>>> by_factory["Bali"]
32

Update, using the updated data-- if you also want to handle Cost_Type, you have several options.  One is to select only the rows with Cost_Type == 1:
>>> df[df.Cost_Type == 1]
  Factory  Cost  Cost_Type
1  Sydney    21          1
3  Denver     8          1
4    Bali     9          1

[3 rows x 3 columns]
>>> df[df.Cost_Type == 1].groupby("Factory")["Cost"].sum()
Factory
Bali        9
Denver      8
Sydney     21
Name: Cost, dtype: int64

or you can expand the groupby and group on both Factory and Cost_Type simultaneously:
>>> df.groupby(["Cost_Type", "Factory"])["Cost"].sum()
Cost_Type  Factory
0          Bali       23
1          Bali        9
           Denver      8
           Sydney     21
2          Sydney      4
Name: Cost, dtype: int64

Comment: Thank you DSM, this looks to be a very quick way to get these numbers. There is one extension that would make this the ideal solution. I have a third column in this dataset called "Cost Type" (added to q), and only want to sum the items which have a specific value in this column, in this case, the value is 1. Is there some way to build this filtering into the pandas series?
Answer: The easiest way is to use a dictionary to hold the count for each factories:
factoriescost = {}
for row in cvs.reader(costcsv):
    factory = row[0]
    if factory not in ('Bali', 'Sydney', 'Denver'):
        continue
    factorycost = factoriescost.get(factory, 0)
    factoriescost[factory] = factorycost + float(row[1])
totalcost = sum(factoriescost.itervalues())

Then you can use factoriescost to get the total for a given factory:
>>> print totalcost, factoriescost
65.0 {'Denver': 8.0, 'Sydney': 25.0, 'Bali': 32.0}
>>> print factoriescost['Bali']
32.0

Answer: You can use a dictionary as shown below. The code uses a try loop to sum the cost of the factories in the dictionary, if the factory is not already inside the dictionary then a KeyError will be thrown and so the factory is simply added.
a = [['Bali', 23],
     ['Sydney', 21],
     ['Sydney', 4],
     ['Denver', 8],
     ['Bali', 9]]

factories = dict()

for factory, cost in a:
    try:
        factories[factory] += cost
    except KeyError:
        factories[factory] = cost

print(factories)
# {'Denver': 8, 'Sydney': 25, 'Bali': 32}

In your example case you would replace the for loop with an appropriate one for csv.reader() along the lines of:
for factory, cost in csv.reader(costcsv):
    try:
        ...

Answer: Rather than having separate variables, consider a dictionary or, easier, collections.defaultdict:
from collections import defaultdict

costs = defaultdict(float)

for line in csv.reader(costcsv):
    if len(line) == 2:
        factory, costs = line
        costs[factory] += float(cost)

This will give you an output where you can select any factory (not just the three you currently hard-code) and get the total cost
cost["denver"] == 8.0

Comment: Thank you Jon, I particularly like that this will allow more factories, as in reality there are more than 3 that need to be assessed.
Comment: (Whoops, no enter in comments) Thank you Jon, I particularly like that this will allow more factories, as in reality there are more than 3 that need to be assessed. Unfortunately I am now getting an error that reads "ValueError: too many values to unpack", which I think could be due to empty files in the input file, or the other columns in the csv. Any idea of a way around this?
Comment: I have added a fix for this - any line of unexpected length will be skipped
Answer: Your csv should be:
Factory,Cost
Bali,23
Sydney,21
Sydney,4
Denver,8
Bali,9

And in python you can:
import csv

factories= ['Bali', 'Sydney', 'Denver']
totalcost = 0

sums = {}

with open('file.csv', 'rb') as f:
    f.next()                        # Jump to second row -> first : header
    reader = csv.reader(f)
    for row in reader:
        if row[0] not in sums:
            sums[row[0]] = int(row[1])
        else:
            sums[row[0]] += int(row[1])

for key,value in sums.items():
    totalcost = totalcost  + int(value)

The result look like:
print sums
>{'Denver': 8, 'Sydney': 25, 'Bali': 32}
print totalcost
>65

Python - sumif from csv


How does Fisher calculate his $p$-value?

Question: After reading a lot of great answers on the topic of Fisherian versus Neyman & Pearson, I still cannot understand how Fisher carries out his test.
Here is my understanding of his workflow:

Ask a question.
Propose a null hypothesis based on the question.
Do some experiments, and collect some data.
Assuming the null is true, calculate the $p$-value.
Report the exact $p$-value without comparing it with any explicit criteria.
Take a look at your $p$:

If you subjectively think it's too small, discard the data at your hand and go to the outermost (2) by proposing another null
  hypothesis.
If you subjectively think large it's enough, discard the data at your hand, do another experiment, collect some new data, and find
  better ways to test your theory on them.

(From my understanding, the data used in the NHST stage cannot be used
  in further steps, regardless of whether we are following Fisher or
  Neyman & Pearson.)

I'm not sure if I'm correct, but there is a problem even if I am. Conventionally, the $p$-value is defined as

the probability of obtaining a test statistic at least as extreme as the actual sample value obtained given that the null hypothesis is true.

OK, but how do you define extreme without an alternative hypothesis? Everyone is a bit loose with their language when it comes to this issue, so while illustrative examples don't hurt, please don't post answers containing only examples. I am essentially asking for a high-level description of how Fisher chooses his critical region.
Note that this is not a problem for the Neyman-Pearson approach, because they didn't even mention the $p$-value in their 1933 paper that proposes the Neyman–Pearson lemma, so we can define Neyman-Pearson hypothesis testing in terms of critical regions, and then establish a bijection between the $p$-value and the critical regions. On the other hand, Fisher doesn't seem to have a single paper this clearly documents his way, and I'm a little confused.
Comment: @Glen_b. I'm not sure Fisher did typically use the likelihood to index extremeness. Yates (1984), JRSS A, **147**, "Tests of Significance for 2x2 Contingency Tables", p. 444, quotes Fisher's reply to a letter from D.J Finney asking about two-tailed tests for FET (1946): "I believe I can defend the simple solution of doubling the total probability, not because it corresponds to any discrete subdivision of cases of the other tail, but because it corresponds with halving the probability, supposedly chosen in advance, with which the one observed is to be compared. [...] How does this strike you?".
Comment: @Scortchi I wouldn't for a moment suggest that it was something he was always and everywhere consistently insisting on - especially across about six decades - but even so there's still a suggestion of it in that quote as what might otherwise have been done (specifically, the "discrete subdivision of cases of the other tail"). He's saying that alternatively to that you can make an argument that you could argue to halve the significance level and look in the observed tail.
Comment: @Glen_b: Sure, but - & I should've said this - I'm not aware of Fisher's having commented on the issue *at all* except in this letter. It's Finney who brings up the matter of correspondence to "discrete subdivision of cases of the other tail" as a criticism of the double-the-one-tailed-p-value approach, though he doesn't suggest any particular alternative approach.
Comment: Yes, certainly it's necessary to provide something like a quote to support my belief (rather than secondary sources making the same claim I did) -- which is why this isn't an answer.
Comment: @Glen_b: Perhaps not necessary: if there's a modern Fisherian approach - one drawing heavily on Fisher's ideas, but rejecting some (e.g. fiducial inference) & extending or formalizing others - it'll only be detailed in secondary sources.
Comment: Fisher typically\*  uses the likelihood to denote what's more extreme -- lower likelihood is more extreme. \*(at least where he doesn't have an explicit test statistic which makes the ordering obvious)... e.g. consider the two tailed version of what's usually called the Fisher exact test (and its extension to $r\times c$ tables), where the tables are unambiguously ordered by their likelihood.
Comment: Actually, this is explicit in the question [P-value: Fisherian vs. contemporary frequentist definitions](https://stats.stackexchange.com/questions/386369/p-value-fisherian-vs-contemporary-frequentist-definitions). (This question was even in the "Related" questions list, which you can presently see in the right hand sidebar -- always a good thing to check.)  $\,$ Using likelihood leads to what I see as the central difference between Fisher's approach and the Neyman-Pearson approach: Typically, a Fisher test is an "omnibus" test in the sense that every alternative that lowers likelihood ... ctd
Comment: ctd ... will tend to lead to rejection, while a Neyman-Pearson test is designed to have power against a specific alternative (or, more generally, against some specific sequence of alternatives). To me that doesn't make them especially competing notions of testing at all, but tools designed for somewhat different situations (i.e. as Alecos mentions [here](https://stats.stackexchange.com/a/112786/805) when discussing work by Spanos, *complementary*), each good at what they're trying to do, and one may quite reasonably choose one or the other depending on the circumstances.
Comment: (This is not presently an answer for two reasons; (i) I am debating whether this should be considered a duplicate, and (ii) if this isn't a duplicate and my comment ('it's based on the likelihood') were to be expanded into an answer, I should like to quote Fisher directly - though I don't expect he will mention the word likelihood specifically in this context.)
Comment: 'how do you define extreme without an alternative hypothesis?' is answered in your question 'as extreme as the actual sample value obtained'
Comment: @nalzok  seems you have a clear understanding  of  Fisher p test. The calculation is apparently based on simulations and observed t statistics.
Comment: @Glen_b Can you point me to a typical Fisherian test? I think the lady-tasting-tea experiment misses something: its null hypothesis is "the lady gives random guesses about if the milk went in first", but rejecting it only implies "the lady does not give random guesses", rather than "the lady's guesses are always correct".
Comment: The lady tasting tea won't do. If you do the 4 vs 4 cups version, there's only one probability to deal with (no ''more extreme" to deal with), and if you extend the number of cups (which Fisher does discuss) and don't require the taster to be perfect (but simply to beat chance) then it's a one-tailed test (so 'more extreme' is otherwise obvious). You need to either go to the two tailed version or the $r\times c$ table version (mentioned in my initial comment above), where some method of deciding more extreme is required. It's there that you see that ordering by likelihood under the null occurs
Comment: At http://math.arizona.edu/~piegorsch/571A/TR194.pdf: "Defined simply, a P-value is a data-based measure that helps indicate departure from a specified null hypothesis, Ho, in the direction of a specified alternative Ha. Formally, it is the probability of recovering a response as extreme as or more extreme than that actually observed, when Ho is true. (Note that 'more extreme' is defined in the context of Ha. For example, when testing Ho:␪ = ␪o vs. Ha:␪ > ␪o, 'more extreme' corresponds to values of the test statistic supporting ␪ > ␪o .)"
Answer: Fisher's approach, in a fully parametric framework, was to reduce the data $X$ to a (one-dimensional) statistic sufficient, or conditionally sufficient, for the parameter of interest $\theta$, & to base inference on its distribution under the null hypothesis $\theta=\theta_0$. Typically he used the (or a) maximum-likelihood estimate $\hat\theta(X)$ (in any case the MLE, when unique, will be a one-to-one function of any one-dimensional sufficient statistic when there is one); though I don't recall any explicit discussion, viewing the maximum-likelihood estimate $\hat\theta(X_1)$ as more extreme than $\hat\theta(X_2)$ because it's further away from $\theta_0$ in the same direction follows naturally enough. See Fisher (1934), Proc. Royal Soc. Lond. A, 144 ,"Two New Properties of Mathematical Likelihood", § 2.6, for his emphasis on the connection between (maximum-likelihood) estimation & significance testing.
He doesn't seem to have given a great deal of thought to the calculation of p-values for two-tailed tests (at least for test statistics having discrete distributions). Yates (1984), JRSS A, 147, "Tests of Significance for 2x2 Contingency Tables", p. 444, quotes Fisher's  (1946) reply to a letter from D.J Finney asking about two-tailed p-values for Fisher's Exact Test:

I believe I can defend the simple solution of doubling the total
  probability, not because it corresponds to any discrete subdivision of
  cases of the other tail, but because it corresponds with halving the
  probability, supposedly chosen in advance, with which the one observed
  is to be compared. [...] How does this strike you?

On the face of it this argument belongs more to the Neyman – Pearson approach.
Fisher (1973), Statistical Methods & Scientific Inference, pp 49 – 50, draws a distinction between testing a "general hypothesis"—a model—as a whole, & testing for a particular value of one of its parameters. In the latter case he reiterates the approach above; in the former his advice is this:

In choosing the grounds upon which a general hypothesis should be
  rejected, personal judgement may & should properly be exercised. The
  experimenter will rightly consider all points on which, in the light
  of current knowledge, the hypothesis may be imperfectly accurate, &
  will select tests, so far as possible, sensitive to these possible
  faults, rather than to others.

Which doesn't seem poles apart from the approach of stipulating an alternative hypothesis precisely & basing your choice of test statistic on considerations of power.
Comment: @naizok: There are lots of test statistics following a chi-squared distribution - can you give a context/reference? But anyway, I was referring to "the probability, supposedly chosen in advance, with which the one observed is to be compared" - Fisher is, it seems to me, justifying the p-value $2\min[\Pr(X \leq x), \Pr(X \geq x)]$ by its correspondence with an N-P style accept/reject test having that as its pre-specified significance level - controlling the Type I error.
Comment: (+1) In the Neyman–Pearson framework, a test on a $\chi^2$ statistic is usually one-tailed, but Fisher considered it as a two-tailed test. Why do you think it belongs more to the Neyman–Pearson approach?
Comment: I was referring to the goodness of fit test since you talked about 2x2 contingency tables. Thanks for the clarification. I can see your point now.


Issue in calculating distance from particular lat long to multiple lat longs

Question: import pandas as p
from cmath import sin, cos, atan, sqrt
import math
import numpy as np
import cmath

d = p.read_excel("codeTest.xlsx")

def distance():

    lat1 = d["Latitude"]
    lat1.astype("float64")
    lat1.dtype
    lon1 = d["Longitude"]
    lon1.astype("float64")
    lon1.dtype

    lat2, lon2 = 28.6225833 , 77.2127222
    radius = 6371 # km
    dlat = np.radians(lat2-lat1)
    dlon = math.radians(lon2-lon1)
    a = math.sin(dlat/2) * math.sin(dlat/2) + math.cos(math.radians(lat1)) \
        * math.cos(math.radians(lat2)) * math.sin(dlon/2) * math.sin(dlon/2)
    c = 2 * math.atan2(math.sqrt(a), math.sqrt(1-a))
    e = radius * c
    print(e)
distance()

For the above code I am getting below error:
M/Desktop/Python/Python code/test1.py"
Traceback (most recent call last):
  File "c:\Users\Shaloo.M\Desktop\Python\Python code\test1.py", line 28, in <module>
    distance()
  File "c:\Users\Shaloo.M\Desktop\Python\Python code\test1.py", line 22, in distance
    dlon = math.radians(lon2-lon1)
  File "C:\Users\Shaloo.M\AppData\Local\Programs\Python\Python310\lib\site-packages\pandas\core\series.py", line 191, in wrapper
    raise TypeError(f"cannot convert the series to {converter}")
TypeError: cannot convert the series to <class 'float'>
PS C:\Users\Shaloo.M\Desktop\Python\Python code>`

Comment: You can try replacing `math` functions like `math.atan2` with `numpy` functions like `np.arctan2`. The builtin math library expects to operate on single numbers, hence your error that tries to convert a series to a float.
Answer: Not sure what the issue is, However...
Can I suggest geopandas and shapely?
import geopandas as gp
from shapely.geometry import Point

gdf = gp.GeoDataFrame(df, geometry=geopandas.points_from_xy(df.Longitude, df.Latitude))

gdf = gdf.to_crs("EPSG:4326") # Specify that we're working with Lat/Long

gdf['distance'] = gdf.distance(Point(28.6225833, 77.2127222).buffer(6371))

GeoPandas
Shapely
Comment: You're probably right, I removed the part where I answered something I don't fully understand...


Automatically defer certain messages in Postfix

Question: Recently we've had a spate of "spear-phishing" attacks that many of our users have fallen for, and as a result our automated scripts are running at full steam locking accounts that have been compromised and are sending out spam.
This morning I spent 2 hours dealing with one in particular that seemed to still be sending mail even after the account had been locked and then authentication entirely (and deliberately) broken altogether. Turns out the problem was a huge backlog of deferred messages that Postfix was diligently trying to re-send. As a result I will be updating our script to purge the queues of accounts that it locks to hopefully cut down on the amount of spam that gets sent out before they're detected.
What I'd like to do now is to configure Postfix to automatically defer messages based on certain criteria, such as the number of intended recipients. If it's a legit message it will go out after a delay, but if the account is seen to be sending spam we've got a better chance of catching and deleting it before it gets sent.
Is there a way that I can configure Postfix to defer any message that arrives and exceeds a certain threshold of intended recipients (or other criteria)?
Answer: This is not possible with vanilla Postfix. It is designed to send mail. Not to do analyze outgoing mails before sending.
If you need something like that, then you have to introduce external policy filters. One of it is policyd (http://wiki.policyd.org/features) which provides exactly what you want: Quotas, Accounting and ACLs.
Comment: @Kromey A strange but acceptable work around is to log entries to a central syslog server. There you analyze the flow of mails. If it gets suspicious, disable the login for that user. Or apply firewall rules (as long as you are allowed to do that on the appliance).
Comment: Well, not the answer I was hoping for; since the server's actually a turnkey appliance, adding software to it isn't really an option, and even reconfiguring Postfix was really pushing it. Oh well.
Comment: That's pretty much what our existing script does, it analyzes syslog for (very specific) suspicious activity. In practice it quite often locks an account within 5-10 minutes of the first spam message, but it's shocking how much can go out in so short a time! I was hoping to be able to delay what does go out long enough to analyze the behavior and thus be able to delete them from the queue before they're sent.


What enables Cloudflare to disable direct IP address access?

Question: When I visit a website that has Cloudflare, using the website's IP address, I get this message:
Error 1003
Direct IP access not allowed

What happened?

You've requested an IP address that is part of the Cloudflare network.
A valid Host header must be supplied to reach the desired website.

I am a student.

What does allow Cloudflare to block direct IP address access?
Is not DNS a layer above the IP address? If yes, being Cloudflare a DNS service, why does Cloudflare have the capacity to block IP addresses?
Comment: I suggest you also go look at cloudflare's services offered page to understand what they are offering in a more complete way.
Comment: @jonathanjo Direct ip address access is blocked because CloudFlare has an HTTP Server Proxy?
Comment: DNS is not a layer. It is a service. This question is likely listed on the Systems Admin stack however I can't easily look that up from the mobile interface.
Comment: If you are talking about layers, IP addresses should "never" be used on the application layer
Answer: There's nothing special in the cloudflare setup.  This is just a property of HTTP.
When a client opens a URL, there are three important steps:

If required, it makes a DNS (or other resolution method) to turn a hostname into an IP address.  If the URL specifies an IP address for the host, use that.
It makes a connection to that IP address on a well-known port number, normally 80 (unless it's overridden in the URL)
It asks the server for the page, including the desired hostname.

A classical example looks like this:
GET /pub/WWW/TheProject.html HTTP/1.1
Host: www.w3.org

Consider a large host with many web sites on it.  For simplicity let's say it has a single IP address.  Hundreds of domain names resolve to this address.  How does the server decide which pages to deliver?  It uses the host detail given by the client in the HTTP request.  If you ask for something it doesn't have or want to give you, it will give you an error response.
In your case, the request contains an IP address for the host specifier.
GET /whatever HTTP/1.1
Host: a.b.c.d

Very many hosts decide not to give out pages when the host is specified by IP address.  There's nothing special about Cloudflare here, nor is it to do with DNS.  It's about how the server responds to requests for the host specified by IP address, and you can see that this error message specifies that A valid Host header must be supplied.
Here's an answer which describes how to configure a server in this way: https://serverfault.com/a/607222
You can easily verify this kind of behaviour by using telnet to connect to a server and issue the HTTP request manually.
PS.  The same general answer applies to an HTTPS request, but using Server Name Indication in the setup.  It's worth noting that Host came in with HTTP 1.1 (1997).  Prior to that, the mechanism described here didn't exist, and a server had no way to reliably tell if the client had asked for a name which legitimately resolved to its IP address, or had asked for the host by IP address directly.  As this was an important development for the explosive growth in web sites, many older clients were updated to send Host. [Thanks commenters for picking up on details.]
Comment: Not necessarily.  It could be anything at the receiving end.  In cloudflare's case there will be large infrastructure, and it is likely they filter out as early as possible.  Perhaps there's a load balancer, perhaps there's a reverse proxy, perhaps there's a whole server dedicated to "hostname misses".
Comment: @sleske thanks for mentioning that, I've added it to the answer.
Comment: So, I can say that this behaviour is because of Cloudflare's HTTP server proxy?
Comment: Thank you @jonathanjo
Comment: The (PS) is misleading, HTTPS works differently. It uses SNI (Server Name Indication) to set up an encrypted TLS connection. With a different hostname you'd get a different encrypted connection. Yes, there's a HTTP connection inside the encrypted connection, but the encryption means you can't see the hostname in there. That's why SNI is needed.
Comment: @MSalters Thanks for picking me up on that, I've edited the answer.
Comment: It may be helpful to note that _originally_ the "Host:" header did not exist (it only became mandatory with HTTP 1.1). So  in the (very) early days of HTTP the server indeed did not know whether users connected by hostname or by IP address. But nowadays the Host: header is mandatory.
Comment: It's also worth mentioning that although the "Host:" header wasn't introduced until HTTP 1.1, many HTTP 1.0 clients were updated to include it, since it was fairly trivial to implement.
Comment: @cjm thanks for comment, updated answer.  Certainly it was a big deal in the ISP I worked in, as we were gaining web hosting clients at a rapid pace and it was burning through our address allocation very wastefully.
Answer: The system you are trying to contact has multiple sites listed in its httpd config.
The only way the HTTP server can differentiate the various sites hosted on it are with direct URLs.
Since those sites are hosted on CloudFlare's hardware, they can provide whatever error the wish.  CDN's (Content Delivery Networks) shift the connection to the closest cached copy across their network based on several factors.
If you look at what CloudFlare offers as a service, you will see that part of the setup process you repoint your domains DNS records to their dns servers which then utilize only their CDN systems. As a developer, If you are using a CDN, you still know where your actual hosts are located and can contact them directly, customers no longer have that access.
That also helps isolate your hosts from DDos Attacks. The CDN still contacts your host for content unless your entire infrastructure is hosted within the CDNs network.


Qual a melhor maneira de se fazer uma função de um arquivo "Node" ser chamada no meu site HTML5?

Question: Estou fazendo um projeto de um player de música simples utilizando o ElectronJS.
Para isso estou usando uma biblioteca que reproduz arquivos MP3(play-sound), como posso chamar essa função dentro do meu HTML? Eu preciso criar outro arquivo apenas para reprodução de sons ou posso jogar essa função dentro do arquivo principal?
CÓDIGO: index.js
console.log('Main process working');

const electron = require('electron');
const app = electron.app;
const BrowserWindow = electron.BrowserWindow;
const path = require('path');
const url = require('url');
const player = require('play-sound');

let win;

function createWindow(){
  win = new BrowserWindow({width: 300, height: 200, resizable: false,
  minimizable: false, fullscreen: false, center: true});
  win.loadURL(`file://${__dirname}/app/public/index.html`);

  win.on('closed', () =>{
    win = null;
  });
}

app.on('ready', createWindow);

O código para tocar sons é:
var audio = player.play('nomedoarquivo.mp3', {/*efeitos*/}, (err) => {
    if (err) throw err
});

E para parar:
audio.kill();

Como eu posso chamar esta função dentro do meu arquivo HTML?
Aceito sugestões de bibliotecas importantes para todos projetos!
Comment: Não querendo parecer "pedante", mesmo que a questão já tenha sido resolvida eu recomendo reformular seu titulo porque: um app Electron não pode ser definido como um site. Isso ajuda a comunidade.
Answer: Diferentemente de NW.js que compartilha o contexto do navegador e de node no mesmo processo o que permite executar pacotes npm e acessar API's do navegador no mesmo arquivo .js, Electron faz diferente, em Electron cada contexto é executado em um processo isolado, a maneira como node e o código de "front-end" se comunicam é através de mensagens.
Mensagens:
Tanto o código no processo principal como no processo de renderização utilizam uma instância de EventEmitter para trocar mensagens ... no processo principal utiliza-se o módule ipcMan ou a propriedade "webContents" de "BrowserWindow":
// use ipcMain no processo principal (sync/async) ----------------
const {ipcMain} = require('electron')
// ouvinte, resposta assíncrona
ipcMain.on('iniciar-player', (event, arg) => {
    // sua lógica aqui ...
    event.sender.send('player-reply', 'started-async')
})

// ouvinte, resposta síncrona
ipcMain.on('iniciar-player', (event, arg) => {
    // sua lógica aqui ...
    event.returnValue = 'started-sync'
})

// no processo de renderização (página web) ----------------------
const {ipcRenderer} = require('electron')

// ouvinte assíncrono (ouvinte antes do envio)
ipcRenderer.on('player-reply', (event, arg) => {
    console.log(arg) // "started-async"
})
// enviar
ipcRenderer.send('iniciar-player', 'argumento-ou-configuração')

// ouvinte síncrono
let response = ipcRenderer.sendSync('iniciar-player', 'argumento-ou-configuração')
console.log(response) // "started-sync"

No processo principal execute a lógica de seu módulo. O próximo exemplo usa "webContents" no processo principal (assincronamente):
// no processo principal (async) ---------------------------------
const {app, BrowserWindow} = require('electron')
let win = null

app.on('ready', () => {
    win = new BrowserWindow({width: 800, height: 600})
    win.loadURL(`file://${__dirname}/app/public/index.html`)
    // ouvinte
    win.webContents.on('iniciar-player', (/*parametros serializados em JSON por padrão*/) => {
        // sua lógica ...
        win.webContents.send('player-reply', 'started-async') // responda
    })
})

// no processo de renderização (página web) ----------------------
const {ipcRenderer} = require('electron')

// ouvinte assíncrono (ouvinte antes do envio)
ipcRenderer.on('player-reply', (event, arg) => {
    console.log(arg) // "started-async"
})
// enviar
ipcRenderer.send('iniciar-player', 'argumento-ou-configuração')

Isto é o básico sobre troca de mensagens entre o código executado na página (render) e o processo principal (main).

Nota: você pode ainda se abstrair destas mensagens usando o módulo remote caso deseje apenas acessar o módulo sem passar pelo processo principal

Remote:
Você pode acessar no processo de renderização um módulo ou arquivo (.js) em especifico através do módulo remote:
// no processo de renderização (página web)
const player = require('electron').remote.require('play-sound')

// abrir player
const audio = player.play('nomedoarquivo.mp3', {/*efeitos*/}, (err) => {
    if (err) throw err
});

// simulando a iteração do usuário para encerrar o player
document.getElementById('btn-fechar-player').addEventListener('click', () => {
    audio.kill()
}, false)

Usar remote parece bem mais simples más, leia a documentação ... "algumas coisas" não são recomendadas fazer ao usar remote.

Fontes:

Electron API

ipcMain
webContents
remote


Why are hidden Markov models (HMM) also called mixture models?

Question: Why are hidden Markov models (HMM) called mixture models?
What does it mix?
Answer: Mixture models are generic probability density functions which are the weighted sums of independent processes that add to a total density function with a total area of 1, which area is common to all probability density functions. Consider, for example that two people are cutting pencils on an assembly line. The first cuts a fraction $0<p<1$ of the pencils with an average pencil length of $\mu_1$ with a standard deviation of $\sigma_1$. A second person is cutting $1-p$ of the pencils with average pencil length of $\mu_2$ with a standard deviation of $\sigma_2$. Then the mixture distribution (normal distribution assumption) of pencils coming off of the assembly line is $MD(p,\mu_1,\sigma_1,\mu_2,\sigma_2)=pN(\mu_1,\sigma_1)+(1-p)N(\mu_2,\sigma_2)$.
In a hidden Markov model, the state (pencil cutters) is not directly visible, but the output (e.g., assembly line output), dependent on the state, is visible. Each state has a probability distribution over the possible output tokens ($p$ and $1-p$ in our case). Now a hidden Markov model does not have to be a mixture model, for example, it can be unimodal, but the mixture model type of hidden Markov model is simple to solve.
To better explore if, as claimed in Wikipedia, a hidden Markov model can be considered a generalization of a mixture model or whether that is just too narrow a view, I posed this as a separate question; Are there any examples of hidden Markov models that are not mixture models? And as it turns out convolutions can be HMM as well, and most people would consider convolution to be a different operation from mixture addition.
It would seem that HMM are not only useful for mixture models, but for convolution models and possibly others.
Comment: And to avoid any confusion: "Why are hidden Markov models (HMM) also called mixture models?" HMMs are NOT called mixture models. Mixture models and HMMs are 2 different models but as explained above, HMMs can embed mixture models as emission distributions.
Comment: @Eskapp,thanks ,as I read on wiki- [link] (https://en.wikipedia.org/wiki/Mixture_model) , mixture models are exactly what Carl explained and yes, it's only a part of HMM as u rightly said.
Comment: @BrianCannard More generally, density functions do not have to be probability density functions (pdf). In general, density functions add by convolution, and a mixture model is the scaled ordinary addition of density functions, which is not convolution. Convolution of density functions is not ordinary addition, mixture models are ordinary addition. It may be hard to grasp what convolution is, but trust me on this, it is not mixture modeling.
Comment: Convolution is exactly the sum law of probability distributions. https://en.m.wikipedia.org/wiki/Convolution_of_probability_distributions#:~:text=The%20convolution%2Fsum%20of%20probability,linear%20combinations%20of%20random%20variables.

Does it mean it's exactly the same as the mixture addition?
Comment: @BrianCannard Convolution of probability distributions, i.e., probability density functions, is equal to their convolution integral, e.g., see [infinite support; Fourier transforms](https://mathworld.wolfram.com/Convolution.html) for one type of convolution integral. There are two others, real space convolution and Laplace transforms, the latter for semi-infinite support densities.
Comment: Excellent intuition! Convolutions feel more natural and general, actually. What is the justification for addition of scaled densities in mixture models?
Comment: @BrianCannard The physics. Suppose for two machines one cuts pipes slightly shorter than the other but both dump pipes onto the same conveyor belt. The conveyor belt then goes to a sorting room. In the sorting room they want to know how many pipes are more than or less than a specific length. They plot a histogram and discover that the pipe lengths are distributed as a mixture model with two distinct peaks for pipe lengths.
Answer: (This answer would be better as a comment to build on @Eskapp's comment)
I think it is important to give the general and simple formula $$p(Y) = \sum_{X} p(X,Y) = \sum_{X} p(X)p(Y|X)$$ (also appearing on Wikipedia). This clearly shows that in HMM, it is the observation process ($Y$) which is modeled as a mixture.
However, as already noted, HMM are not called mixtures models.


Clearing local tasks cache

Question: Using a deriver, I'm able to produce local tasks for each entity of my custom configuration entity type. So if I have two configuration entity labeled "Foo" and "Bar", I can have similarly labeled tabs on a report page.
However, when adding or removing one of my configuration entity, the local tasks are not updated (ie. the tab for the deleted entity is still visible). This is not a surprise as these things get cached.
I've been able to fix the issues using implementations of hook_ENTITY_TYPE_insert() and hook_ENTITY_TYPE_delete() calling drupal_flush_all_caches(). But clearing all caches in order to only get the tabs on a single page updated seems a bit overkill. I already tried to call both menu_cache_clear_all() and Drupal::cache('discovery')->invalidateAll() without success
The tabs are build using the following code:
MODULE.tasks.yml:
MODULE.settings.webhooks_list:
  deriver: 'Drupal\MODULE\Plugin\Menu\LocalTask\ReportLocalTasks'
  base_route: MODULE.settings.list

src/Plugin/Menu/LocalTasks/ReportLocalTasks.php:
namespace Drupal\MODULE\Plugin\Menu\LocalTask;

use Drupal\Component\Plugin\Derivative\DeriverBase;
use Drupal\Core\Entity\EntityStorageInterface;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Plugin\Discovery\ContainerDeriverInterface;
use Drupal\Core\StringTranslation\StringTranslationTrait;
use Drupal\Core\StringTranslation\TranslationInterface;
use Drupal\MODULE\CustomEntityTypeInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

class ReportLocalTasks extends DeriverBase implements ContainerDeriverInterface {
  use StringTranslationTrait;

  protected $entityManager;

  /**
   * Constructor.
   *
   * @param $entityManager
   * @param $stringTranslation;
   */
  public function __construct(EntityTypeManagerInterface $entityManager, TranslationInterface $stringTranslation) {
    $this->entityManager = $entityManager;
    $this->stringTranslation = $stringTranslation;
  }

  public static function create(ContainerInterface $container, $base_plugin_id) {
    return new static(
      $container->get('entity_type.manager'),
      $container->get('string_translation')
    );
  }

  public function getDerivativeDefinitions($base_plugin_definition) {
    $this->derivatives = [];

    /** @var EntityStorageInterface $entityStorage */
    $entityStorage = $this->entityManager->getStorage('custom_type');

    /** @var CustomEntityTypeInterface $connection */
    foreach ($entityStorage->loadMultiple() as $entity) {
      $this->derivatives["MODULE.settings.list.{$entity->id()}"] = [
        'title' => $entity->label(),
        'route_name' => 'MODULE_ecom.settings.list',
        'route_parameters' => [
          'connection' => $entity->id(),
        ],
      ];
    }

    foreach ($this->derivatives as &$entry) {
      $entry += $base_plugin_definition;
    }

    return $this->derivatives;
  }
}

In order to have up to date tabs when adding or removing custom entity, I'm using the following code. What I'm trying to do is to have something more specific than the call the drupal_flush_all_caches().
MODULE.module
/**
 * Implements hook_ENTITY_TYPE_insert();
 */
function MODULE_CUSTOM_ENTITY_ETYPE_insert(\Drupal\MODULE\CustomEntityTypeInterface $connection) {
  drupal_flush_all_caches();
}

/**
 * Implements hook_ENTITY_TYPE_delete();
 */
function lMODULE_CUSTOM_ENTITY_ETYPE_delete(\Drupal\MODULE\CustomEntityTypeInterface $connection) {
  drupal_flush_all_caches();
}

Comment: Core issue to fix this is here https://www.drupal.org/project/drupal/issues/3001284
Comment: Probably you could invalidate only the rendered cache instead of all cache. Something like `\Drupal\Core\Cache\Cache::invalidateTags(['rendered']);` But with this vague context is hard to answer. Please can you share you code?
Answer: First of all it might be that you found a bug in core (a caching issue), so it would be great if you could report it to the Drupal core issue queue as well. There is extensive test coverage for local tasks, but it might be that there is none for adding new entities and that those in core just work due to side effects.
To solve your immediate issue I would try to invalidate just the render cache of the local task blocks.
To find out the cacheability metadata of the local tasks the easiest is to install the renderviz module:
https://www.drupal.org/project/renderviz/
and apply the patch from this comment
https://www.drupal.org/node/2640280#comment-11286381
Once you have that installed and you go to the page with your local tasks, you will see a list in the JS console of all cache tags present.
For the bartik theme you will find (e.g. tested with core search pages, which have similar code to yours):
config:block.block.bartik_local_tasks
If you put into the console:
  renderviz('tags', 'config:block.block.bartik_local_tasks');

Then you will see that indeed the primary local tasks will be highlighted.
So in your hook_entity_insert() / hook_entity_update(), you can just do:
 Drupal\Core\Cache\Cache::invalidateTags('config:block.block.bartik_local_tasks');

And that will invalidate just the local tasks block and solve your issue for now.
What should have happened however is that the local tasks block should show both the cache tags for the list [entity-type] and the entities itself. (like a normal listing of entities)
e.g. if you want to change the title of any of your entities, core should based on the deriver update the local tasks block.
And that we don't do that is likely a core bug in regards to cacheability metadata, so please file it. Unless I miss something and there is some special code that makes this work for core's derivers out-of-the box.
Edit:
We correctly add the cache tags of the config entities, however there is no 'list' cache tag, which means that whenever you add or delete a config entity the local tasks don't get invalidated.
As you have a plugin, you can likely just add your own custom cache tag to the plugin definition.
e.g.
$base_plugin_definition['cache_tags'][] = 'local-tasks-my-entity-type-list-cache-tag';

Then just invalidate that list cache tag in your insert / update.
But that definitely is also an issue in Drupal core as I can reproduce it purely with core search pages.
I missed in your question at first that caches fail to get invalidated only when adding or deleting new entities.
That however was the key clue.
Edit 2:
Core should be invalidating list cache tags automatically, so the easiest is to just use the pre-defined list cache tags.
This is the diff that fixes the bug for me in core:
diff --git a/core/modules/search/src/Plugin/Derivative/SearchLocalTask.php b/core/modules/search/src/Plugin/Derivative/SearchLocal
--- a/core/modules/search/src/Plugin/Derivative/SearchLocalTask.php
+++ b/core/modules/search/src/Plugin/Derivative/SearchLocalTask.php
@@ -52,7 +52,9 @@ public function getDerivativeDefinitions($base_plugin_definition) {
           'route_name' => 'search.view_' . $entity_id,
           'base_route' => 'search.plugins:' . $default,
           'weight' => $entity->getWeight(),
 -        );
 +        ) + $base_plugin_definition;
 +
 +        $this->derivatives[$entity_id] += ['cache_tags' => []];
 +        $this->derivatives[$entity_id]['cache_tags'] += $entity->getEntityType()->getListCacheTags();
       }
     }
     return $this->derivatives;

That fixes the bug for me for core search pages.
Comment: I don't think that would be a core bug. Core does not known about my custom deriver. It cannot known it has to clear/cache something related to local tasks when a custom config entity is inserted/delete.
I don't have to test ATM, but I'm pretty sure this is not a render cache issue. My current assumption is that multiple cache are involved, the render cache may be one but I guess plugin and menu cache are involved.
Comment: The core bug / limitation here is that you have no way to tell core the cacheability metadata that you need.

It should really be associated with the LocalTask item itself.

e.g. you want to show the list and entity cache tags when displaying a list of entities.

I am just unclear how e.g. search pages get around that issue (if they do).

And yes it might well be that the discovery cache needs to be cleared as well.
Comment: Hmmm, maybe you need to add "class: \Drupal\Core\Menu\LocalTaskDefault" to your *.task.yml?
Comment: It _is_ a core bug and a very nasty one for that. I can reproduce it with normal core search pages starting from the third search page.

I get the curious issue that one one page I have 2 tabs on the new page I have three tabs.

Clearly cached by page_cache and dynamic_page_cache.
Comment: I edited my answer to provide the final solution, still need to file a core issue though.
Comment: I reported the core bug here: https://www.drupal.org/node/2797745

Could you accept the answer or is there still something missing?
Comment: With the added code, the answer make sense. Adding the entity type `cache_tags` to the derivatives clear the needed cache on entity insertion/delete.
Answer: Based on LionsAd answer, there is no need to manually clear any cache in implementations of hook_ENTITY_TYPE_insert and hook_ENTITY_TYPE_delete. Instead, the plugin derivative should be marked with the entity type list cache tags, so Drupal can clear the appropriate caches when needed. The code for my derivative now looks like this:
namespace Drupal\MODULE\Plugin\Menu\LocalTask;

use Drupal\Component\Plugin\Derivative\DeriverBase;
use Drupal\Core\Entity\EntityStorageInterface;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Plugin\Discovery\ContainerDeriverInterface;
use Drupal\Core\StringTranslation\StringTranslationTrait;
use Drupal\Core\StringTranslation\TranslationInterface;
use Drupal\MODULE\CustomEntityTypeInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

class ReportLocalTasks extends DeriverBase implements ContainerDeriverInterface {
  use StringTranslationTrait;

  protected $entityManager;

  /**
   * Constructor.
   *
   * @param $entityManager
   * @param $stringTranslation;
   */
  public function __construct(EntityTypeManagerInterface $entityManager, TranslationInterface $stringTranslation) {
    $this->entityManager = $entityManager;
    $this->stringTranslation = $stringTranslation;
  }

  public static function create(ContainerInterface $container, $base_plugin_id) {
    return new static(
      $container->get('entity_type.manager'),
      $container->get('string_translation')
    );
  }

  public function getDerivativeDefinitions($base_plugin_definition) {
    $this->derivatives = [];

    /** @var EntityStorageInterface $entityStorage */
    $entityStorage = $this->entityManager->getStorage('custom_type');

    $entityType = $this-> entityManager->getDefinition('lightspeed_ecom_connection');

    /** @var CustomEntityTypeInterface $connection */
    foreach ($entityStorage->loadMultiple() as $entity) {
      $this->derivatives["MODULE.settings.list.{$entity->id()}"] = [
        'title' => $entity->label(),
        'route_name' => 'MODULE_ecom.settings.list',
        'route_parameters' => [
          'connection' => $entity->id(),
        ],
      ];
    }

    foreach ($this->derivatives as &$entry) {
      $entry += $base_plugin_definition;
      $entry  += ['cache_tags' => []];
      $entry['cache_tags'] += $entityType->getListCacheTags();
    }

    return $this->derivatives;
  }
}

Answer: Knowing that drupal_flush_all_caches() solves your problem, all you need to do is to execute a subset of that function's code. The docs for drupal_flush_all_caches() is well documented so you can create a variant of that function that is specific for your needs.
For example:
function _MODULE_flush_partial_cache() {

  // Empty all cache bins & let modules take action on it
  // It possible that you might get by with emptying a subset of cache bins
  // (e.g. cache.menu & cache.entity?) but lets knock them all out.
  $module_handler = \Drupal::moduleHandler();
  $module_handler->invokeAll('cache_flush');
  foreach (Cache::getBins() as $service_id => $cache_backend) {
    $cache_backend->deleteAll();
  }

  // Ignore CSS, JS, static, Twig, module, & theme
  // rebuilding that's not needed in this context.

  // Obviously, your routes need to be rebuilt.
  \Drupal::service('router.builder')->rebuild();
}

/**
 * Implements hook_ENTITY_TYPE_insert();
 */
function MODULE_CUSTOM_ENTITY_ETYPE_insert(\Drupal\MODULE\CustomEntityTypeInterface $connection) {
  _MODULE_flush_partial_cache();
}

/**
 * Implements hook_ENTITY_TYPE_delete();
 */
function MODULE_CUSTOM_ENTITY_ETYPE_delete(\Drupal\MODULE\CustomEntityTypeInterface $connection) {
  _MODULE_flush_partial_cache();
}

Answer: I found a simple solution for this case:

there is a local task defined in my_module.links.task.yml
Route has _custom_access  - visibility of Tab depends on Node-Values
I add HOOK_menu_local_tasks_alter with this code

    $data['cacheability']->addCacheTags($node->getCacheTags());

Now the local-Tasks will be build again if node is saved.

Clearing local tasks cache