How to merge two dataframes based on time?

Question: Trying to merge to dataframes. First is big, second is small. They have a datetime set as index.
I want the datetime value of the second one (and the row) merged in between the datetime values of the first one, sorted by time.
df1:
df1 = pd.read_csv(left_inputfile_to_read, decimal=".",sep=';', parse_dates = True, low_memory=False)
df1.columns = ['FLIGHT_ID','X', 'Y','MODE_C', 'SPEED', 'HEADING', 'TRK_ROCD', 'TIJD']
df1['datetime'] = pd.to_datetime(df1['TIJD'], infer_datetime_format = True, format="%Y-%M-%D %H:%M:%S")
df1.set_index(['datetime'], inplace=True)
print(df1)

FLIGHT_ID   X   Y   MODE_C  SPEED   HEADING TRK_ROCD    TIJD
datetime
2019-01-28 00:26:56 20034026    -13345  -1923   230.0   414 88  NaN 28-1-2019 00:26:56
2019-01-28 00:27:00 20034026    -13275  -1923   230.0   414 88  NaN 28-1-2019 00:27:00
2019-01-28 00:27:05 20034026    -13204  -1923   230.0   414 88  NaN 28-1-2019 00:27:05
2019-01-28 00:27:10 20034026    -13134  -1923   230.0   414 88  NaN 28-1-2019 00:27:10
2019-01-28 00:27:15 20034026    -13064  -1923   230.0   414 88  NaN 28-1-2019 00:27:15
... ... ... ... ... ... ... ... ...
2019-01-29 00:08:32 20035925    13443   -531    230.0   257 85  NaN 29-1-2019 00:08:32
2019-01-29 00:08:37 20035925    13487   -526    230.0   257 85  NaN 29-1-2019 00:08:37
2019-01-29 00:08:42 20035925    13530   -520    230.0   257 85  NaN 29-1-2019 00:08:42
2019-01-29 00:08:46 20035925    13574   -516    230.0   257 85  NaN 29-1-2019 00:08:46
2019-01-29 00:08:51 20035925    13617   -510    230.0   257 85  NaN 29-1-2019 00:08:51
551446 rows × 8 columns

df2:
df2 = pd.read_csv(right_inputfile_to_read, decimal=".",sep=';', parse_dates = True, low_memory=False)
df2['datetime'] = pd.to_datetime(df2['T_START'], infer_datetime_format = True, format="%Y-%M-%D %H:%M:%S" , dayfirst=True)
df2.set_index(['datetime'], inplace=True)
df2.drop(columns=['T_START', 'T_END', 'AIRFIELD'], inplace=True)
print(df2)

                      QNH  MODE_C_CORRECTION
datetime
2019-01-28 02:14:00  1022                235
2019-01-28 02:14:00  1022                235
2019-01-28 02:16:00  1019                155
2019-01-28 02:21:00  1019                155
2019-01-28 02:36:00  1019                155
...                   ...                ...
2019-01-28 21:56:00  1014                 21
2019-01-28 22:56:00  1014                 21
2019-01-28 23:26:00  1014                 21
2019-01-28 23:29:00  1014                 21
2019-01-28 23:52:00  1014                 21

[69 rows x 2 columns]

The idea is that the first row of df2 should be inserted somewhere at 2019-01-28 02:14:00.
I have spent hours on Stackoverflow and pandas documentation (merge, join, concat) but cannot find the right solution.
The next step would be to interpolate the values in column 'QNH' to the rows that are in df1, based on that time.
Any help greatly appreciated!
Answer: Just concatenate two DataFrames and sort by date:
df = pd.concat([df1,df2]).sort_values(by='datetime')

For the next step you can use pandas.DataFrame.interpolate.
Comment: OMG. Easy as py. Thank you.


MDX: Count number of days excluding weekends

Question: I already had an MDX query that returns number of days in a Month Hierarchy using a calculated member. What I needed to do is to add new calculated member that returns number of days excluding weekends (Saturdays and Sundays).
Below is my existing MDX query using Adventure Works DW 2008R2 SE
<code>WITH
  MEMBER [Measures].[Num of Days] AS
    COUNT (
      Descendants ( [Date].[Calendar],  [Date].[Calendar].[Date])
    )

SELECT
  { [Measures].[Num of Days] } ON COLUMNS,
  [Date].[Calendar].[Month].ALLMEMBERS ON ROWS
FROM [Adventure Works]
</code>
sample existing output:
<code>+----------------+-------------+
|   Month Year   | Num of Days |
+----------------+-------------+
| July 2005      |          31 |
| August 2005    |          31 |
| September 2005 |          30 |
+----------------+-------------+
</code>
sample expected output:
<code>+----------------+-------------+--------------------------------+
|   Month Year   | Num of Days | Num of Days Excluding Weekends |
+----------------+-------------+--------------------------------+
| July 2005      |          31 |                             21 |
| August 2005    |          31 |                             23 |
| September 2005 |          30 |                             22 |
+----------------+-------------+--------------------------------+
</code>
Answer: The standard solution for this is to have a hierarchy in your time dimension with the days of the week ( Monday,... Sunday). Once you for this you can filter directly using those members. You can even do a hierarchy with just weekend/no weekend. The best solution depends on the measures you've in your schema, the best is always using standard MDX aggregations and avoiding using SUM/COUNT on a set of members.
Other solutions might depend on how you implemented your hierarchy, you can use the Weekday and filter function.

Honestly something like, I haven't check it
<code> Count(  FILTER( Descendants ( [Date].[Calendar],  [Date].[Calendar].[Date]) ,  WeekDay( [Date].[Calendar].currentmember.properties("KEY"), 2) <= 5 ) )
</code>
Comment: This is the current structure of the Date dimension http://goo.gl/NjTJQ9. It has already [Date].[Day Name] attribute which has members Sunday to Saturday. Can I use this to achieve the expected output?
Comment: Thank you so much for your help.
I modified your code a bit and this solves my problem :) . `Count(  FILTER( Descendants ( [Date].[Calendar],  [Date].[Calendar].[Date]) ,  WeekDay( [Date].[Calendar].CurrentMember.Member_CAPTION, 2) <= 5 ) )`


How can I get Autocomplete behave as in most of the sites out there?

Question: I have a form built with FORM Api. It has a text field, with Autocomplete.
I also have another form, with several fields, some of them with Autocomplete.
It works well, but I have to press Enter Key twice to send the form.
The first Enter key is to select an item.
The second Enter key sends the selected item for process.
The behaviour that I have seen outside, in the wild, and which is customary in thousands of millions of sites is this:

If only one field form (search form, for example), once you click or press enter with the suggested term highlighted, the form will be sent (let's call this behaviour One-Clik). Note: there are no multiple selections option, if not, this behaviour would have to be different.
In several fields form, you just select and press ENTER, and can continue filling other fields. If you want to submit the form, you press the Go button or just click Enter in the autocomplete field if that is the last one you used. This is called Two-Clicks, except that: you can write some words in the autosuggest field, and not choose anything. The desired behaviour would be that just clicking ONCE the Enter key, you would submit the form. So this behaviour is really One-Click, with the only exception that selecting an item by pressing Enter key does not qualify as a click.

In order to have it working, I had to rewrite the autocomplete.js. Is there another solution?
I cannot understand how Drupal cannot manage by default such an easy behaviour, seem in millions of sites. And I can assure that for normal users this is very frustrating and irritating and very counterintuitive.
Comment: Yep that's how it's written, and yep you'll need to replace/augment that yourself if you want different features (just like anything). Personally I agree with requiring two return presses to submit the form. More intuitive in my opinion, and will produce fewer problems for mobile users.
Comment: Google does not require two return presses. Most of my users criticize and say that it is not intuitive. At least there could be some flexibility with some config screen and two slightly versions of the code (it does not require too much re-coding). My 2-cents thought, anyway.
Comment: Yeah just a matter of opinion really @Cesar. jQuery UI autocomplete requires two button presses, for example, and that's used a lot more widely than Google's. Left-brain/right-brain thing probably
Comment: Ok. It is simply that my boss and client are very demanding on lots of things that should behave like they want because they have seen elsewhere. I try to say that Drupal is extremely customizable but some behaviours are less customizables than other, but it seems they won't listen to me.
Answer: I think this is a JQuery problem rather than a Drupal problem.
You should be able to add a JavaScript function, something like the following:
$("#searchField").select(function(event, ui) {
    $("#searchField").val(ui.item.label);
    $("#searchForm").submit();
});

It's necessary to assign the value before submitting the form because if the user clicked on a list entry with the mouse, the value won't have been assigned before the select event occurs.
I haven't actually tested this, but hopefully this will send you in the right direction.


Adjusting Sample with Propensity Score Weighting and ATT

Question: I have a retrospective sample that contains a treatment and non-treatment group with >10 covariates comprised of both categorical and continuous variables. I used the chi-squared and Mann-Whitney U tests (most of the covariates were non-parametric) to compare the treatment and non-treatment groups. There were significant differences between the groups with some of the covariates, so I was employing propensity scores (PS) to estimate the average treatment effect for treated (ATT) to then weight the non-treated samples to form more similar groups.
I'm relatively new to PS adjustments, but I'm able to calculate the PS using logistic regression and then estimate the ATT using
$$ATT_i = T_i + (1 - T_i) * \frac{PS_i}{(1 - PSi)}$$
where i corresponds to one observation.
My question is what is the proper way to weight the samples to reperform the group analysis to show they are more similar?
For the categorical variables this seemed straightforward as multiplying the weights by the value for the category (0 or 1) and then the group size would be the sum of all weights. Is this the correct way of going about this weighting?
I'm not entirely sure how to adjust the continuous variables with the weights and then recalculate the difference score. To my knowledge, there's no weighted Mann-Whitney U test to accomplish this, and simple multiplication by weights (or normalized weights) doesn't seem correct. What is the proper way to reperform this analysis?
Apologies if this question is basic or if I am incorrect in my assumptions so far.
For the analysis I'm using python with scipy, numpy, and statsmodels (with Jupyter).
Thanks for your help!
Answer: First, many methodologists agree that hypothesis testing is inappropriate for balance testing. That is, you should not be using chi-square and MWU tests to determine whether your groups differ in their covariate distributions. The argument is the balance is a property of your sample, not of a hypothetical population. With this in mind, computing sample statistics like standardized mean differences, differences in proportion, variance ratios, and KS statistics are more appropriate. This is what I would recommend you do before and after weighting. All of these statistics have weighted versions that can be applied to your weighted data.
The weighted mean is the sum of the product of the weight and the value for each unit divided by the sum of the weights. This can be used for standardized mean difference (by dividing by the unweighted standard deviation of the treatment group) and the difference in proportion. Weighted variance can be computed using the "reliability weights" formula. Typically you want standardized mean differences and differences in proportion less than 0.1 (i.e., close to 0) after weighting, and variance ratios close to 1 after weighting.


iptables: block specific port by country

Question: How could I modify this script:
http://www.cyberciti.biz/faq/block-entier-country-using-iptables/
to block by country for a specific port?
Is it just a matter of changing this:
<code># Drop everything
$IPT -I INPUT -j $SPAMLIST
$IPT -I OUTPUT -j $SPAMLIST
$IPT -I FORWARD -j $SPAMLIST
</code>
to:
<code># Drop everything
$IPT -I INPUT --dport XXX -j $SPAMLIST
$IPT -I OUTPUT --dport XXX  -j $SPAMLIST
$IPT -I FORWARD --dport XXX  -j $SPAMLIST
</code>
Answer: You need add protocol(tcp or udp), for tcp:
<code>$IPT -I INPUT -p tcp --dport XXX -j $SPAMLIST
$IPT -I OUTPUT -p tcp --dport XXX  -j $SPAMLIST
$IPT -I FORWARD -p tcp --dport XXX  -j $SPAMLIST
</code>
or(for udp):
<code>$IPT -I INPUT -p udp --dport XXX -j $SPAMLIST
$IPT -I OUTPUT -p udp --dport XXX  -j $SPAMLIST
$IPT -I FORWARD -p udp --dport XXX  -j $SPAMLIST
</code>
must be replaced dport to sport in OUTPUT chain.
Comment: ahhhhhhhhh right.  Is there any point of blocking UDP as well? or rather can I combine them into one line?  I assume the answers to that are yes, and no.
Comment: Omitting the `-p tcp` option will block all protocols.  If you don't have the port open for upd it doesn't help, but won't cost much.  I don't know how the port will map to ICMP protocol so you my want to match the protocol using `-p !icmp` to match any protocol other than ICMP


Não consigo enviar array JSON para pagina PHP

Question: Boa noite,
Estou tentando enviar um array de codigos simples (2,5,7,etc) em formato JSon para uma página php atraves de uma requisicao Ajax mas não consigo.
Meu JS:

O cod_jogador é o vetor com os valores.
Meu HTML:

    <button type="button" name="" value="" class="button" onclick="enviArray()">Teste</button>

Na pagina selecao.php eu tenho a variavel criada para receber o JSON:
$array = json_decode($_GET["x"], false);
echo "$array";

Porém sempre que eu dou echo em qualquer variavel no GET (ja tentei com POST tb), ele sempre me retorna indefinido. O que estou fazendo de errado?
Answer: Nesse exemplo, é utilizado o metodo POST.
HTML

    <button type="button" name="" value="" class="button" onclick="enviArray()">Teste</button>

JS

PHP
<?php
$obj = json_decode($_POST["x"]);
echo json_encode($obj);
exit();

Comment: Oliveira, muito obrigado kra. Funcionou aqui. Estava ha dias buscando essa solucao, me salvou. Grande abraco e que Deus lhe pague
Answer: Como você está enviando informações para o servidor, utilize os verbos HTTP corretos para a requisição que em seu caso é o verbo POST. Além de especificar o verbo HTTP, você precisa "dizer" ao servidor qual o tipo de conteúdo que está enviado, para isso é necessário utilizar o cabeçalho (header) Content-Type.
function enviArray() {

    cod_jogador.sort();
    var meuJson = JSON.stringify({codigos: cod_jogador});

    var httpRequest = new XMLHttpRequest();
    httpRequest.onreadystatechange = function() {
        if (this.readyState == 4 && this.status == 200) {
            document.getElementById("demo").innerHTML = this.responseText;
        }
    };

    httpRequest.open("POST", "selecao.php");
    httpRequest.setRequestHeader("Content-Type", "application/json");
    httpRequest.send(meuJson);
};

O seu arquivo php deve ficar da seguinte forma:
$array = json_decode($_POST["codigos"], true);
print_r($array); //Não é possível utilizar o "echo" em um array, somente em seus valores: echo $array[0].


Add hop in tracert

Question: Basically, I'm interested how to make one of my servers hop in tracert.
My first server is located within DC in France
My second server if located within DC in Russia
I want my 1st server to be in traceroute behind my 2nd server
Current path to my server (let's imagine its like that):
1. Hop #1 (your IP)
2. Hop #2 (my 2nd server)

How I want it to be:
1. Hop #1 (your IP)
1. Hop #2 (my 1st server)
2. Hop #3 (my 2nd server)

Reason: I want all of the traffic to filter through my firewall within 1st server, and only afterwards go to the 2nd server. Both of the servers are not in local network, they're based within two different hosting providers and its impossible to make them in local network. Is it possible, to start routing all of the traffic through the 1st server before it reaches 2nd server without making 1st server a UDP/TCP proxy and giving users its IP to connect and than route traffic to my back-end basically (which is being my 2nd server) and drop all traffic coming except from 1st server?
If I need network access: I do understand that I may need network access directly, but what access I really need? Is it router access within the DC where the 2nd server is located? Or I need access to routers in both DC's? How can I really configure it?
Why I'm asking here?: I tried to google this issue and find explanation for more than a week now, and I still had no clear explanation except people saying "Just do TCP/UDP proxy from your 1st server".
Why I don't want to do proxy?: Because I think ping is going to increase too much to the second server if using first server as proxy, however if I do put in traceroute I think ping is not going to be "seriously increased" since this server can sometimes act just as a switch and route traffic, if I'm wrong here - please, say I'm wrong and ping is going to be the same as traffic still will need to go through this server anyway and TCP/UDP proxy is my easiest solution.
Comment: Do you control the endpoints? Building your own WAN is a somewhat different problem than services for the public Internet.
Comment: @JohnMahowald what do you mean by "do you control endpoints"? I do control both of the servers for sure if that's a question.
Answer: Impossible. You say you want to convince the entire world to change its routing (i.e. to route in a sub-optimal way). In fact all the Internet routers try instead to always figure out an optimal route to each IP.
To play with this, you need to own at least an entire public /24 subnet, although I'm not sure what regulations come to play between these two countries.
So, unless you are a big player, to direct the entire world to Internet provider 1 you need to use IP that belongs to that provider. You could use DNAT&SNAT at the first server, not necessarily a "heavy" proxy.
Comment: Thanks for your reply, basically what I'm trying to achieve - is DDoS Protection, by routing all of the traffic through 1st server (with ddos protection) to 2nd server (which is not ddos protected, but 1st server will filter all of the DDoS attacks)
Comment: @dnleiman I see. Then you need to keep a secret about the unprotected IP, because if attacker figures it out, you're toasted. The DNAT/SNAT approach will work, it's faster and simpler than VPN, but has no encryption and unprotected_IP logs will see all traffic as coming from protected_IP. (google "hairpin NAT" - it's the same idea)
Comment: I will just limit all the traffic coming to it from any IP address except the one which is protected. I understand about logs.
What I want to achieve is: 1st server to my users = 70 ms ping, 2nd server to my users = 5 ms ping. However 2nd server is not protected from DDoS attacks, my goal is to reach minimum increase in ms and protect second server (and primarily - to keep second IP, so users won't connect to IP which is linked to first server, but will connect to IP which is linked to second server and their connection will flow through first server anyway).
Comment: @dnleiman DDOS saturates your entire link, so no filtering on your side can prevent it. That's why you always need ISP protection. About your users knowing the unprotected IP: impossible. This is what I explained in my answer.
Comment: That's actually a bit wrong. I'm protecting 1 gbps link on the 2nd server, my 1st server is also 1 gbps link. If 1st server will detect an attack it will filter it, 1st server is able to handle large DDoS attacks (due to the capacity of network being more than 500 gbps (it's OVH if it matters)), therefore any malicious traffic isn't going to reach 2nd server and so it will be protected.
My target is to allow users to connect to my 2nd server IP straight away, but make all the traffic flow through firewalls on 1st server, and my second target is to minimize ping increase from such action.
Answer: Create a VPN containing both servers.  Set up the second server to route between the Internet and the VPN.  Ensure that the first server does not accept traffic directly from the Internet.
Comment: That's exactly what I don't want. Since it will massively increase ping to my 2nd server and it would be just easier to host everything on 1st server from the start.
What I want to achieve is:
1st server to my users = 70 ms ping
2nd server to my users = 5 ms ping
However 2nd server is not protected from DDoS attacks, my goal is to reach minimum increase in ms and protect second server
Comment: Well, you can't change the topology or routing metrics of other people's networks, except by paying them to do it.
Comment: As for ping times, compare distances of the hosts with speed-of-light travel time.  The speed of light is a hard limit.
Comment: That's true. And that's actually what I'm trying to do somehow, since both of my servers are not in the same local network and I cannot manipulate them in that way, but I'm pretty sure there's good solution to setup such network conditions. As I've written in the comment below: "My target is to allow users to connect to my 2nd server IP straight away, but make all the traffic flow through firewalls on 1st server, and my second target is to minimize ping increase from such action ".

Thanks for your help btw, I appreciate it a lot

Add hop in tracert


How to change options on Android GoogleMap from Java?

Question: I'm trying to add options to my xml-defined google map in my (first!) Android App. From Java I now want to change some options. I try to do this as follows:
GoogleMap mMap = null;
if (mMap == null) {
    mMap = ((MapFragment) getFragmentManager().findFragmentById(R.id.map)).getMap();
    if (mMap != null) {
        Log.wtf("DE MAP IS", "GECREEERD!!!");
        GoogleMapOptions options = new GoogleMapOptions();
        options.mapType(GoogleMap.MAP_TYPE_SATELLITE)
                .compassEnabled(false)
                .rotateGesturesEnabled(false)
                .tiltGesturesEnabled(false);
        mMap.newInstance(options);
    }
}

Unfortunately I get an error at mMap.newInstance(options); saying "The method newInstance(GoogleMapOptions) is undefined for the type GoogleMap".
Does anybody know how I can fix this? All tips are welcome!
Comment: Very late... but still relevant: https://stackoverflow.com/questions/22068451/setting-googlemapoptions-programatically
Answer: use this to select options
int mapType =  GoogleMap.MAP_TYPE_SATELLITE;

FragmentManager manager = getSupportFragmentManager();
SupportMapFragment mapFragment = (SupportMapFragment) manager.findFragmentById(R.id.map);
map = mapFragment.getMap();
map.setMapType(mapType);


Google Maps Geolocation API

Question: Estou querendo verificar o estado em que meu cliente está para mostrar os estabelecimentos próximos aquela região.
Como eu faço para conseguir essas informações através do Google Maps Geolocation API?
Existe outra maneira melhor?
O site em que venho trabalhando é o PiscouAchou.com.br
Answer: Antes de tudo, para conseguir a localização do usuário, ele deve permitir que o seu site tenha acesso a localização dele. Porém nem sempre é uma boa prática pedir a localização dele logo de cara, é necessário um "bom motivo" para tal ação. Você pode estar lendo mais sobre isso aqui: User Location.
Outro ponto importante, a API de Geolocalização da Google, tem um limite para usuários FREE, ou seja, faça um teste com seus usuários, veja se irá passar desse limite diariamente, pois se passar, você terá que ver se é viável pagar a API ou mesmo encontrar outra para suprir o site. Exemplo aqui: Google Maps Geolocation API.

Google Maps Geolocation API


How to Include HTML inside If Statement

Question: Why when I have certain HTML in my C# statements does Razor complain about invalid expressions and terms?
And how can I fix this?
For example; in my _layout.cshtml:
@if (Team.Id == ViewBag.TeamId)
{

}

Comment: As @RowanFreeman has indicated, escaping the HTML with `@:` is one way around this. If you are able to provide more context about what you're doing (what is your data, what is your desired markup result), I'd bet someone could come up with a solution that didn't involve having to worry about this.
Comment: Another option would be to connect the end div to the start div, and vice versa. The code excerpt given is a bit anti pattern from where I come from as you are splitting a div at a at seemingly random place.
Answer: Razor actually tries to understand your HTML. Unfortunately it can't be highly intelligent and recognise dynamic code or code outside of the expression.
Since

is invalid HTML, it complains.
You can avoid this problem by using @:
This forces razor to bypass its "HTML recognition", and simply print whatever you're asking for.
E.g.
@if (Team.Id == ViewBag.TeamId)
{
    @:

    @:

    @:
}

Answer: You can also use @Html.Raw("YOUR HTML"); inside an if statement to add HTML code
Answer: i use This syntax :
@if (Team.Id == ViewBag.TeamId)
{
  <text>

  </text>
}

How to Include HTML inside If Statement


Is it a bad idea to have more than a custom entity defined in a module?

Question: Fact #1
Drush module exposes two commands drush generate module-content-entity and drush generate module-configuration-entity, both generating new custom modules with entity classes defined. There is no separate command to generate custom entity code for existing module.
Fact #2
I could not find the module in Drupal's core, where a single module defines more than one custom entity (Maybe there are some and I have not seen all). There are modules that register content and configuration entities, but not multiple contents or multiple configuration entity (or block or any other).

I have a feeling that, even though it's not forbidden, it's not a good idea to have multiple entities in a single module. Maybe it's an architectural pattern.
Can anyone elaborate on this or share some content (docs, articles) where this is discussed?
Comment: It's entirely fine to have multiple entity types defined in a single module. I've done it on multiple occasions and never run into a single issue on any of the systems by doing so.
Comment: I guess it was simply much easier to code to generate new files all the time than doing all the diffs/merges required to update an existing module...
Comment: The Commerce module does it: https://git.drupalcode.org/project/commerce/-/tree/8.x-2.x/modules/payment/src/Entity
Comment: I also, can't see any issues with this. Can you share any core or well-known contributed modules doing so?
Comment: And, any ideas why Drush is forcing us to generate a new module with new entity?
Comment: What hudri said.
Comment: I see, but updating is not important. I mean the ability to add a new entity to an existing module.
Comment: Same answer....
Comment: In retrospect, Taxonomy in core does it as well, declaring `taxonomy_vocabulary` and `taxonomy_term` in the same module.
Answer: About Fact #1
You can generate multiple entities for a single module directly with the Chi-teck's Drupal Code Generator.
Require the package:
composer require chi-teck/drupal-code-generator --dev

If you have the latest Drush version this package is already installed.
Run the command ./vendor/bin/dcg to display the main navigation:
./vendor/bin/dcg
 Select generator
  [ 0] ..
  [ 1] Console
  [ 2] Entity
  [ 3] Form
  [ 4] Miscellaneous
  [ 5] Plugin
  [ 6] Service
  [ 7] Test
  [ 8] Yaml
  [ 9] composer.json
  [10] Controller
  [11] Field
  [12] Hook
  [13] Install file
  [14] Javascript
  [15] Layout
  [16] Module
  [17] Module file
  [18] PhpStorm metadata
  [19] Plugin manager
  [20] Render element
  [21] Service provider
  [22] Template
  [23] Theme
  [24] Theme file
  [25] Theme settings

Choose [16] to start a new module and then run [ 2] twice.
Edit:
The current Drush version 11 supports also generating entities separately:
drush generate entity:content

Comment: Yes, drush initiates a chi-teck dcg Application and enhances it with additional generators. So this is a matter of which drush version depends on which dcg. dcg works stand-alone without drush if you don't want this version restriction.
Comment: Thank you. Now I found that Drush after 11 version also supports chi-teck dcg and you have this functionality there. I was using 10.6.
Comment: I think all of chi teck's code generator went into Drush, didn't it?
Comment: @4uk4 actually, if you are using Drush under 11, you can't install dcg 2+ version as Drush requires old dcg
Comment: I would uninstall the old Drush temporarily to install dcg 2+. Especially for entities it's important not to use an outdated code generator.
Comment: yes that is exactly what I did
Answer: While Drupal core modules normally implement a single content entity, that isn't a Drupal limit; it is rather the modules that focus on implementing a single functionality and a single entity.
The Aggregator module implements two content entities: the Aggregator feed entity and the Aggregator feed item entity, for example.
/**
 * @ContentEntityType(
 *   id = "aggregator_feed",
 *   label = @Translation("Aggregator feed"),
 *   label_collection = @Translation("Aggregator feeds"),
 *   label_singular = @Translation("aggregator feed"),
 *   label_plural = @Translation("aggregator feeds"),
 *   label_count = @PluralTranslation(
 *     singular = "@count aggregator feed",
 *     plural = "@count aggregator feeds",
 *   ),
 *   handlers = {
 *     "storage" = "Drupal\aggregator\FeedStorage",
 *     "storage_schema" = "Drupal\aggregator\FeedStorageSchema",
 *     "view_builder" = "Drupal\aggregator\FeedViewBuilder",
 *     "access" = "Drupal\aggregator\FeedAccessControlHandler",
 *     "views_data" = "Drupal\aggregator\AggregatorFeedViewsData",
 *     "form" = {
 *       "default" = "Drupal\aggregator\FeedForm",
 *       "delete" = "Drupal\aggregator\Form\FeedDeleteForm",
 *       "delete_items" = "Drupal\aggregator\Form\FeedItemsDeleteForm",
 *     },
 *     "route_provider" = {
 *       "html" = "Drupal\aggregator\FeedHtmlRouteProvider",
 *     },
 *   },
 *   links = {
 *     "canonical" = "/aggregator/sources/{aggregator_feed}",
 *     "edit-form" = "/aggregator/sources/{aggregator_feed}/configure",
 *     "delete-form" = "/aggregator/sources/{aggregator_feed}/delete",
 *   },
 *   field_ui_base_route = "aggregator.admin_overview",
 *   base_table = "aggregator_feed",
 *   render_cache = FALSE,
 *   entity_keys = {
 *     "id" = "fid",
 *     "label" = "title",
 *     "langcode" = "langcode",
 *     "uuid" = "uuid",
 *   }
 * )
 */

/**
 * @ContentEntityType(
 *   id = "aggregator_item",
 *   label = @Translation("Aggregator feed item"),
 *   label_collection = @Translation("Aggregator feed items"),
 *   label_singular = @Translation("aggregator feed item"),
 *   label_plural = @Translation("aggregator feed items"),
 *   label_count = @PluralTranslation(
 *     singular = "@count aggregator feed item",
 *     plural = "@count aggregator feed items",
 *   ),
 *   handlers = {
 *     "storage" = "Drupal\aggregator\ItemStorage",
 *     "storage_schema" = "Drupal\aggregator\ItemStorageSchema",
 *     "view_builder" = "Drupal\aggregator\ItemViewBuilder",
 *     "access" = "Drupal\aggregator\FeedAccessControlHandler",
 *     "views_data" = "Drupal\aggregator\AggregatorItemViewsData"
 *   },
 *   uri_callback = "Drupal\aggregator\Entity\Item::buildUri",
 *   base_table = "aggregator_item",
 *   render_cache = FALSE,
 *   list_cache_tags = { "aggregator_feed_list" },
 *   entity_keys = {
 *     "id" = "iid",
 *     "label" = "title",
 *     "langcode" = "langcode",
 *   }
 * )
 */

It's not a bad idea, implementing more than a content entity in a module. I would only avoid implementing content entities that aren't necessary.
As for why Drush creates a new module for each content/configuration entity, as other users said, that is probably to keep the Drush code simple and because usually that's what is done.


Using Zentyal to control several servers

Question: I am currently in the process of creating a home server, made up of several virtual machines, each running a different type of server (i.e. a file server, a multimedia server, a firewall and a print server as of now).
Today I discovered this new software, Zentyal, which looks interesting for system and network administration. However, I haven't understood one thing: can it be used only with its preconfigured packages or can it also be used to control other servers running other programs (in my case, the four virtual machines)?.
Also, would you suggest to use such a program or do you think it's better to control each server on its own?
Comment: Your question is off topic for Serverfault because it doesn't appear to relate to servers/networking or desktop infrastructure in a professional environment. It may be on topic for [Superuser](http://superuser.com) but please [search](http://superuser.com/search) their site for similar questions that may already have the answer you're looking for.
Comment: @Iain the question can easily be extended to a professional environment, i.e. changing the word 'home' with 'enterprise' :)
Answer: You could use Zentyal for all your needs without using other virtual machines. It depends on which services you need, but Zentyal already ships many different services.
To reduce administrative work, I'd recommend to use as few servers as necessary. If you go for different virtual machines, Zentyal can be the Hypervisor for your needs. But it probably won't help you with administering those virtual machines.
If you're just looking for a Hypervisor to host your VMs, have a look at proxmox (http://pve.proxmox.com/).
Comment: Basically, I am only interested in having a web interface from which to administer my virtual machines, and I didn't know if Zentyal could do that...
Comment: yes, it can do that. But for that job, I'd recommend using Proxmox.
Comment: Thanks. Any particular reasons why I should use Proxmox and not Zentyal?
Comment: First, Zentyal has many different services, a GNOME Shell and a web interface while Proxmox only ships a Debian Squeeze with a custom kernel and and custom Web GUI. Due to the amount of services and integrations in Zentyal, upgrading a machine will be much harder. Not upgrading a system isn't really an option. Second, Zentyal Community Edition does not have a proper upgrade path (only Professional and Enterprise editions have), but Proxmox has because of the underlying Debian and the few services running on the hypervisor.
Answer: I'm new to sysadmin and this site, but I'll try my best to answer this based on my experience with that distro.
You don't specify what you're using it for, and how many boxes you have to work with (physical or virtual) so we'll say 2 possibilities. First, you just want to run some servers at home and this isn't live and essential to some venture.  Then just use one box, with Zentyal, and redundancy and survival is not an issue.  Second, this is for some business or operational function and therefore necessary.  Then do not host all servers on one box for standard practice reasons.
As to whether to use Zentyal or not overall, it's a decent distro but I wouldn't use the free version for a massive network. The commercial version is much more robust with better support.  Having all services on one box is not typically allowed in production networks but from home, it's not as big of a deal and you make the call.
It sounds like you're asking whether other servers can be used on a Zentyal installation. It's still just Ubuntu Linux so you can put anything else on there. But if you want access by way of the web interface, then I believe you'll have to code that into the files (it's written in PHP I think) to allow those additional servers to be shown.  Otherwise just access them by CLI and save time.
Comment: I have 4 virtual machines running on one machine in my home network. I was thinking about the VMs to separate duties and reduce the risk of problems (if there is a problem with my print server VM, the other server could continue to work) and I was asking myself if Zentyal could be used to have a single web interface from which to administer all VMs.
Comment: OK so it sounds like you could be using a cloud solution for that. Cloud management (sounds complex but not really) software can be used like a front-end to manage various services on different boxes virtual or physical. Zentyal isn't cloud, and all you can do is setup servers.  If you want management capability, check out a management web interface called Puppet http://www.puppetlabs.com/puppet/what-is-puppet/


User authentication from domain joined Windows 10 Enterprise 2019 LTSC on isolated network computer takes 2 or 3 minutes

Question: Machines of concern are on a VLAN isolated from external sites via firewall rule. These machines can communicate with the domain controllers. User authentication from Windows 10 Enterprise 2019 LTSC machines takes two or three minutes. Machines running 1709 and 1803 on the same VLAN do not exhibit the problem. The long authentication problem is observed on initial interactive login, remote desktop connection, and when using "Run as different user" to launch an application. After 2 or 3 minutes the logon succeeds.
Used packet capture to troubleshoot. Packet capture shows what I think is a perfectly normal kerberos exchange.  The domain controller grants a ticket to the workstation that appears to be valid for the authenticating user. Then the machine will query DNS for login.microsoftonline.com.  It will spend the next two or three minutes attempting to connect to addresses for this site which are intentionally blocked by a perimeter firewall. And finally, after 2 or 3 minutes, there is some RPC traffic between the workstation and the domain controller which seems to complete the logon.
I think we should be able to authenticate without external internet access. Is there a simple configuration option that I'm missing?
EDIT: Mapped login.microsoftonline.com to 127.0.0.1 in hosts file. Logon completes successfully and immediately! I wouldn't consider it a solution, just additional information.
Comment: Are the affected systems using an OEM image?
Comment: Is this a hybrid joined system? An on-prem only domain joined system will never reach out to login.microsoftonline.com (consequently, hybrid does). Also, verify if this appears on 2004 as well because a similar bug was fixed in this release that is supposed to reduce the blocking time.
Comment: @Davidw no, sir - not OEM.  Installed from a bootable USB created from an installation ISO for Windows 10 Enterprise LTSC v1809.
Comment: @Steve yes, I think it may be a hybrid environment. Does that mean authentication to local domain controllers is insufficient?  I will set up a test using v2004.
Comment: Hybrid is hybrid. It requires access to local DCs and AAD. The observed behavior is potentially a bug, but fundamentally a hybrid-joined device requires access to AAD. If you don't want it to access AAD you need to disable the hybrid state.
Answer: Got a resolution through Microsoft support. Perhaps this will be of use to someone else someday.
Since these machines did have internet access when they were joined to the domain, they became joined with Azure AD. Once they were moved to the isolated network and could no longer reach the cloud, we observed the problem. To leave Azure AD but remain joined to the on-premises domain, use dsregcmd /leave. Docs for dsregcmd.


64-bit web browsers for Windows?

Question:
Notice: This post was relevant to a specific time period. Most of the answers contained herein are outdated, for many the situation described has simply changed. Caveat Emptor

What 64-bit web browsers exist for Microsoft Windows (64-bit
XP/Vista/7)? For example, 64 bit versions of Opera, Firefox or
Safari?
If such a browser exist, do you have any experience with
opening very large HTML files (opening times, stability,
memory consumption per HTML input byte).
I am asking because we have a server application that generates
very large HTML files (the record so far is 900 MB, but a
more typical size is 100-200 MB) and users often run out of
memory when trying to open the HTML files. The current
practical limit is about 130 MB. The memory consumption is
about 10 times the input HTML file size (but this may of
course depend on the kind of HTML).
Comment: Maybe you should be talking to the developers of that application to try and get them to output in a more appropriate format.
Answer: A 64-bit version of Internet Explorer is available for 64-bit versions of Windows Vista/7.  Nobody uses it, primarily because plug-in developers cough Adobe, specifically Flash cough will not bother writing a 64-bit version of their plug-ins, and who will browse the net if YouTube doesn't work?  ;)
It would work fine for opening large HTML files however.
Start --> All Programs, and it is listed right above Internet Explorer as Internet Explorer (64-bit).
Comment: Just a remark: This is not limited to the 64 bit Linux Flash Player ;)
Comment: There's 64bit Linux flash player! problem is it's badly optimized and suffers from poor performance.
Comment: Adobe and Java both have 64-bit plug-ins now, and both are becoming their standard download. We're not *there* yet, but on the way.
Answer: There is a 64-bit version of Firefox available. The browser itself works fine, however there is a severe limit of plugins available, no Flash for example, although I believe they have just released a 64-bit Java plugin.
64-bit versions of Windows come with a 64-bit version of Internet Explorer, but again you meet similar limitations with Flash and Java.
So, using these versions may assist in your application, but it's going to restrict users in what else they can do and will end up going back to their 32-bit version for other browsing.
Answer: I do not have much experience with the 64 bit versions of web browsers. I have tried using a few of the major ones before on a couple 64 bit OS installs, but I can confirm that the plugins not being 64bit or not working optimaly are major issues. Haha, I like what Sean said about "who will browse the net if YouTube doesn't work?"
Answer: 64-bit Flash is here =D (it's a prerelease called Flash square 64).
The prerelease of Firefox is now down the pan as an update the other day made it crash far too often to contemplate using it =(
I despise Internet Explorer and go as far as uninstalling it the second I
reinstall Windows. iI find I then have a lot fewer errors in Windows itself, whether I am using it or not. I find just having Internet Explorer in the file system interferes with Windows far too much, so I resorted to using Safari. It rocks, I mean it really rocks =D
There was once upon a time a 64-bit version of Chrome, but now I can't find any existence of it on Internet. I am now searching for a reliable 64-bit browser. I will let you know my results on everything I try. As far as your huge HTML scripts it sounds like you're running out of virtual memory. Different browsers will not help with this, but would speed up the process.
Think getting your users to up their virtual memory will sort this out I am sure, but having it on a 64-bit only system is not the best answer as it forces your users to use 64-bit. Most won't know the difference and would have to reinstall Windows.
I hope this helps you all.

64-bit web browsers for Windows?


How to ignore content within  tags when using pandas.read_html?

Question: How to ignore content within  when using pandas.read_html method?
An an example the table data 12 December 20181	 gets converted to 12 December 20181.
How to ignore content from within  tags?
Comment: The documentation clearly states: `Expect to do some cleanup after you call this function` . So I would just iron these issues out with some code after having read the data.
Comment: @PankajJoshi Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. - Jamie Zawinski
Answer: The problem with Pandas advice

Expect to do some cleanup after you call this function.

is that it removes the markup, so the needed information to do a proper cleanup is lost.
The only solution I see is to cleanup the HTML as a preprocessing step. You can use lxml.html.clean.Cleaner for that purpose. As a starting point you can try something like:
from lxml.html.clean import Cleaner

def clean_html(raw_html):
    cleaner = Cleaner(remove_tags=["sup"])
    return cleaner.clean_html(raw_html).decode("utf-8")

And then use the cleaned HTML in pd.read_html. You may have to play around with the other settings of the cleaner to preserve/clean exactly the content you need.


Probability of the score of two rolls

Question: I have a probability problem and this is the request:
What is the probability that the score of the first roll is higher than the score of a second roll. The rolls are equilibrated and independent. (I think one per time)
I have solved this problem with this method: $P(X>Y)=P(X-Y>0)= \frac{15}{36}$ with the help of a grid with all the possible $X-Y$ scores.
Is this a good way of resolution? If not, help me with a good method.
Thank you
Comment: Yes, for a problem with a 'low' amount of possible outcomes, gridding them out should work. Your answer is correct. Independence means the outcome of the second roll does not depend on the first roll (and vice versa), which should be the case if you are rolling dice.
Comment: Nothing wrong with this method.  If you want another (that scales up better with the number of cases):  The probability of a tie is $\frac 16$.  If we aren't in a tie then $P(X>Y)=P(Y>X)$ by symmetry.  Thus $P(X>Y)=\frac 12\times \left( 1-\frac 16\right)=\frac 5{12}$, just as you got.  Note that you could use this method to easily handle the case of an $N$ sided die.
Answer: There is no problem with your solution; enumeration is a terrific technique and sometimes it's all there is.  If you want a method that can handle a large number of cases better:
The probability of a tie is $\frac 16$.  If we aren't in a tie then $P(X>Y)=P(Y>X)$ by symmetry.  Thus $$P(X>Y)=\frac 12\times \left( 1-\frac 16\right)=\frac 5{12}$$ just as you got.  Note that you could use this method to easily handle the case of an $N$ sided die.
Comment: thanks for your help, can you give me an explaination about $1/2$ in your formula? Because we have two rolls?
Comment: No.  It's because you have two cases.  $X>Y$ and $Y>X$.  As I say, these must be equally probable.  They don't add to $1$ because it is possible that there is a tie.  Note that for a very large $N$ the probability of a tie is negligible so you should expect $P(X>Y)\to \frac 12$.  You could confirm that this is true analytically.
Comment: everything it's ok, thanks. I have voted your answer

Probability of the score of two rolls


Support vector machine for text classification

Question: I am currently having a data set, class 1 with about 8000 short text files and class 2 with about 3000 short text files. I applied LibSVM and tried a couple of parameter combinations in the cross-validation experiment.
Generally the class 1 precision falls into the range of (85%, 90%); the class 2 precision falls into the range of (70% , 75%); the recall of both class 1 and class 2 fall into the range of (80% , 85%).
For the text classification purposes, I built text feature space following the common approaches, tokening the document, filtering the stopwords and building the word vector using tf-idf or binary frequency, etc. I also tried n-gram model to build the feature space. But these approaches did not improve the performance a lot. I would like to know are there any other ways  that may help tune the LibSVM to improve the performance. LibSVM provides grid search for parameter setting up, but it runs pretty slow.
Answer: If you haven't done so, you can usually gain efficiency by stemming the words, as I explain on this post.
Stemming will replace words by their stem, so that for instance 'sky' and 'skies', or 'hope', 'hopes' and 'hoped', will be recognized as identical. If your texts are in English, you will most likely end up using Porter's stemming algorithm of which you can find many implementations, one of the most famous being that included in NLTK (in Python). The post mentioned above contains a code snippet that shows how to use it.
Another idea that I develop in this post is to use co-inertia analysis (CIA), a forgotten variant of canonical analysis to reduce the dimension of the data, while keeping maximal correlation with your classes. You would end up with only two or three scores instead of the few hundreds that you must have in your corpus, which might give a better bias-variance deal.
Answer: A simple thing to try, if you haven't done so already, is to normalize each document vector so the magnitude is 1.
SVMs tend to perform better if the magnitude of each training vector is similar.
Answer: LibSVM hasn't been getting reliable performance for me, of late. Have you tried using SVMLight ever?
You might also try looking at which features are showing the most predictive power in your model, and adding some sort of enriched-type feature. For example, if I were classifying documents on whether they contain information related to protein-protein interaction, I wouldn't really care about the specific names of the proteins as predictive features. I would pre-process my documents and normalize all protein mentions with some common term that wouldn't normally occur in my documents, like "THISWORDUSEDTOBEAPROTEIN". Previous research (sorry, I can't think of any citations off the top of my head except my own paper Ambert & Cohen, 2012) has shown that this can lead to improved classifier performance by preventing the classifier from getting distracted by common genes (e.g., ADH1A) and ignoring rare ones, instead combining the predictive power of all genes into a single feature you could think of as "GeneMentioned".
Answer: You can clean up your tekst documents using Soundex, a phonetic algorithm for indexing words by sound. This Soundex methods reduces the number of spelling mistakes.
There is a good implementation of soundex in SAS, namely the function Soundex()


Interpreting zero odds ratio from Fisher's exact test

Question: I did fisher's exact test and I got an odds ratio equal to zero. Can you please clarify what it means? I want to plot the odds ratio but don't know how to deal with the zero and INF values.
below are the data and the test:
z <- matrix(c(69,13,33,0),nrow=2)
f <- fisher.test(z)
f

This is the output:
    Fisher's Exact Test for Count Data

data:  z
p-value = 0.01845
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 0.000000 0.742875
sample estimates:
odds ratio
         0

Any help is very appreciated.
Answer: If in a 2-by-2 table, we let $n_{11}$ be the counts in (1,1) cell, $n_{12}$ the counts in the (1,2) cell, $n_{21}$ those of cell (2,1) cell and $n_{22}$ those of cell (2,2), then the sample odds ratio is defined by
$$
\hat\theta = \frac{n_{11}n_{22}}{n_{12}n_{21}}.
$$
If any of $n_{ij}$ is zero, then $\hat\theta$ equals 0 or $\infty$ (it is undefined if both entries in a row or in a column are zero). Because these outcomes have positive probability, then the expected value and variance of $\hat\theta$ are not defined. A better well-behaved estimator is
$$
\tilde\theta = \frac{(n_{11}+0.5)(n_{22}+0.5)}{(n_{12}+0.5)(n_{21}+0.5)}
$$
Both $\hat\theta$ and $\tilde\theta$ have the same asymptotic normal distribution around the true odds-ratio $\theta$. Unless the sample size is quite large, however, the distributions of both $\tilde\theta$ and $\hat\theta$ are highly skewed. The log transform, having an additive rather than multiplicative structure, converges more rapidly to normality. An estimated standard error for $\log\hat\theta$ is
$$
\hat{\rm{se}}(\log\hat\theta) = \left(\frac{1}{n_{11}}+\frac{1}{n_{12}}+\frac{1}{n_{21}}+\frac{1}{n_{22}}\right)^{1/2},
$$
and thus by the large sample normality of $\log\hat\theta$
$$
\log\hat\theta \pm z_{\alpha/2}\hat{\rm{se}}(\log\hat\theta).
$$
provides an approximate confidence interval for $\log\hat\theta$. The confidence interval for $\hat\theta$ can be obtained by exponentiating the limits of the above intervals. A similar interval can be obtained using $\tilde\theta$ in place of $\hat\theta$.
However, when $\hat\theta = 0$ or $\hat\theta = \infty$, this interval cannot be applied. When $\hat\theta = 0$, one should take 0 as the lower limit and when $\hat\theta = \infty$ one should take $\infty$ as the upper limit. For the other bound you can use the above formula following some adjustment, such as replacing $n_{ij}$ by $n_{ij}+0.5$ in the estimator and standard error. Other methods are also possible, such as methods based on score function or conditional maximum likelihood estimation.
As for the odd-ratio estimate provided by fisher.test, the help page (?fisher.test) says

estimate   an estimate of the odds ratio. Note that the conditional
Maximum Likelihood Estimate (MLE) rather than the unconditional MLE
(the sample odds ratio) is used. Only present in the 2 x 2 case.

This means that the odd-ratio provided by fisher.test is derived through an alternative estimator (here conditional maximum likelihood, which is a kind of maximum likelihood estimator) which need not coincide with the sample odds ratio $\hat\theta$ or $\tilde\theta$.
Here is a simple example that supports this claim.
fisher.test(matrix(c(69,13,33,23), ncol=2))

    Fisher's Exact Test for Count Data

data:  matrix(c(69, 13, 33, 23), ncol = 2)
p-value = 0.001425
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
 1.556881 8.947718
sample estimates:
odds ratio
   3.66139

# sample odds-ratio estimate
> 69*23/(13*33)
[1] 3.699301

As you can see, the two estimates are not exactly the same although they are pretty close.
If you want to dive deeper into this subject I suggest having a look at Agresti's Categorical Data Analysis book.
Comment: @MarwahAl-kaabi , look up the Haldane-Anscombe correction for odds ratio, which adds 0.5 to each cell when there are 0 or low counts. There are other methods as well.
Comment: Is there any way to correct for those odd ratios with zero? like adding 1 to all of the cells ? is this correct way to solve this problem?


Correct Analysis for determining impact of input parameters on data

Question: I work for a manufacturing company that collects times (man-hours) for tasks.  The tasks are similar in concept from one product to another, but the characteristics of the product can vary (like size, voltage, etc).
It has been something like ten years since I've had to do any heavy lifting in terms of data analysis... I'm trying to take our dataset (about four years' worth of data) coupled with some parameters about each of the products and use it to ultimately be able to input a product's characteristics and be able to predict the output in hours.
Can anyone help me get started with this?  I'm an EE by background and have worked with neural networks and linear regression in the past, but it's been something like 10 years, so I've forgotten it all.
Thanks so very much,
Timothy
Comment: A little more information about the data you have would be welcome. For instance, your response to @IrishStat's answer suggests your data might be time series, but nothing in this question suggests that this is the case--and in fact it is difficult to see how such data could usefully be construed as, or modeled as, time series at all.  It would be especially helpful to have some clearer idea of what a product's "output" is and how you expect that to be related to the other characteristics.
Comment: Hi, my dataset will include man-hours charged to specific operations each day over a course of several years.  The operations have the same name from product to product (for example, one of them is "connect").  However, the amount of time "connecting" takes varies dramatically from one product to another.  The thought is that I can do some sort analysis comparing time to connect vs. some characteristics of the product (like how many connections), and find the most critical ones as well as predict/estimate times for future product builds.  Let me know if that doesn't address your comment.
Answer: The method is called Transfer Function or Dynamic Regression. It is a super-set of linear regression that takes into account leads and lags of the user-specified predictor series. It also allows the incorporation of lagged effects of the error term. Additionally it allows the identification and incorporation of level shift/seasonal pulses and local time trends. Furthermore one time pulses are adjusted for. All of this while considering parameters and error variance that possibly change over time. A good on line reference for introductory time series material is available at http://www.autobox.com/cms/index.php/afs-university/intro-to-forecasting.
Comment: Perfect - thank you.  I will read up and follow up.


jquery plugin method return value

Question: I am writing a plugin for some common tasks for the drop down. Selected Index method needs to return a value to me. How can i accomplish this inside a plugin where some methods may or may not return a value? For methods that do not return a value, i want to maintain chainability.
jQuery.fn.dropdownHelper = function(method) {
var methods = {
    init: function(){ },
    empty: function(){
        if (this.tagName == 'SELECT')
            this.options.length = 0;
    },
    selectedIndex: function(){
        var index = this.selectedIndex;
        if(index == 'undefined')
            index = -1;
        alert (index);
        return index;
    }
};
return this.each(function() {
    // Method calling logic
    if ( methods[method] )
        return methods[ method ].apply( this, Array.prototype.slice.call( arguments, 1 ));
    else if ( typeof method === 'object' || ! method )
        return methods.init.apply( this, arguments );
    else
        $.error( 'Method ' +  method + ' does not exist' );
});
};

Answer: "Unless you're returning an intrinsic value from your plugin, always have your plugin's function return the this keyword to maintain chainability." -Plugins/Authoring on docs.jquery.com
This means that you return index (intrinsic value) from your selectedIndex function, like you are currently are. Then for all other functions for which you are not currently specifying any return value, you return the this keyword to maintain chainability. For example, to maintain chainability when calling .dropdownHelper('empty'), try the following.
empty: function(){
    if (this.tagName == 'SELECT')
        this.options.length = 0;

    return this;
},

If that doesn't work, try returning $(this) instead.

jquery plugin method return value


Setting up a Honeypot in an Enterprise Environment

Question: I'm interested to see if anyone here who administers a large environment (200-500 Servers), and has a very large public customer base (100,000+), has set up (or has at least considered setting up) a honeypot?  I'm specially interested for those that provide services to nasty/evil/hostile networks.
If you have set one up, can you elaborate on your experience?  In fact, please comment if you don't consider your environment to be large, even a small environment that contains some hostile networks is perfect!
I'm planning to set one up where I work, but naturally that will start with a few battles from management, naturally.  There are risks - the biggest risk would be that things are not setup correctly, and your production servers join your honeypot "cluster", or simply that information about your network leaks out (any information is too much information).

Production Honeypots
A production honeypot is used to assist an organization in protecting its internal IT infrastructure whereas a research honeypot is used to accumulate evidence and information in order to study hackers' or the blackhat criminal attack patterns and motives.
Production honeypots are valuable to the organization especially commercial, as it helps to reduce or mitigate risk that a specific organization faces. Production honeypots secure the
organization by policing its IT environment to identify attacks. These production honeypots are useful in catching hackers with criminal intentions. The implementation and deployment of production honeypots are relatively easier than research honeypots.
Answer: You want Honeyd - Honeyd is a small daemon that creates virtual hosts on a network. The hosts can be configured to run arbitrary services, and their personality can be adapted so that they appear to be running certain operating systems. Honeyd enables a single host to claim multiple addresses - I have tested up to 65536 - on a LAN for network simulation. Honeyd improves cyber security by providing mechanisms for threat detection and assessment. It also deters adversaries by hiding real systems in the middle of virtual systems.
Answer: Honeypots are extremally useful for any environment and they don't need to be anything fancy or crazy.
Examples that we do:
-Create a fake account on or mailserver (say userX) with a few fake links in his mailbox (things like user directory link, payment link, etc all pointing to an internal server). Now we monitor any access to these pages via the logs and we know that if we ever see access to them is because someone is reading someone's else email or broke to the system.
-Add a non-published system with non-used IP to our network. Any access to them is probably caused by a scan or maybe bad-configured system (yes, it happens).
And many other things... These little "honeypots" are so easy to setup and the benefits are amazing. We even had a system admin fired for looking at a fake payroll link.
Comment: That's a great story - thankyou - I'm going to use all these ideas when writing up my proposal :)
Answer: I have to be blunt and say if I was your manager I would shut down that idea before it even started. There are too many risks and zero benefits associated with setting up a honeypot within a large IT environment.
Could you elaborate on why you would want to do this? Aren't honeypots largely confined to security researchers? What are you trying to achieve?
Comment: I think you need to study honeypots again.  Firstly, a honeypot exhibits as much risk to your environment, as your currently running servers and services do - unless you don't know what you're doing.  Your services are already at "risk" by virtue of being accessible.  Secondly, the concept behind a honeypot is that the only traffic that will *ever* get there is guaranteed to be bad traffic, because there is no reason (if set up correctly) for good traffic to ever make it there.  What this does, is give you a view of who your assailants are - Just by ignoring them doesn't mean they don't exist.
Comment: +1 for production honeypots as intrusion detection mechanisms.  It can be as simple as adding a directory with a juicy name to a production system, coupled with thorough logging/auditing on that directory.
Comment: You're right, I was only familiar with the latter. I'll do some more reading. :)
Comment: Neobyte - Have a look at "Production Honeypot" vs "Research Honeypot", perhaps you've only heard of the latter.
Comment: Various consultancies offer honeynets as a service to their large corporate clients. And they charge quite a bit of money for the privilege. Trust me - the benefits can be rather high.


label of non-classified examples in machine learning

Question: What happens if a positive example in neither classified as positive nor negative? Will you treat such an example as False Negative (FN)?
I am using a tool (machine learning) that predicts an example (which is A) as either "A" or "B". However, this tool returns no class for the example. Should you classify the example as FN? I am using supervised classification.
Comment: This sounds like a problem with the tool. What tool is it?
Comment: Can you refrase your question? I couldn't understand
Comment: Please edit to add the clarifying information into the question itself.
Comment: How is it that the tool doesn't return a class? Is it unsure? Has it not seen the pattern? Is it a 3 class classifier?
Answer: A binary classifier has only two possible outputs: TRUE and FALSE, if your classifier has a third option of DONT_KNOW, you have a multi-class classifier.
You might want to have a look at this question regarding multi-classifier:
Evaluating accuracy of a multi classifier
I would suggest using a measure of belief (a.k.a posterior distribution), that would measure for each specimen $P(y=T)$.
The accuracy can then be measured using logloss for example
Comment: First, you should figure out why the tools throws an error. But talking statistics, I would remove the items that cause the error, and talking into account you might be introducing a bias into the sample.
Comment: Thanks Goren. Am using a tool that predicts an example as either "A" or "B". However this tool returns an error, with no classification? Should you then classify this example it as False Negative?


Drupal 7 - Ajax form in modal window

Question: I try to create modal window with form (from page /node/add/page).
Try this
<code>hook_menu() {items['mypath']=...} + callback (where echo drupal_render($form); exit();)
</code>
//form has many elements from different modules (node + inline register)
Next

get rendered form via ajax (form has action 'mypath')
plug jquery.form.js to the form

Submit
Server returns full page and i'm sure that new page material doesn't save even if i change action to '/node/add/page' instead of 'mypath'
What i do wrong?
Answer: It's hard to tell what you doing from your description. A better description could possibly yield better answers.
One possible problem could be that the form needs to be loaded on the page the it is submitted on. This is required for the validate and submit handlers to be processed.
Comment: the question is how send form via jquery.form?

Drupal 7 - Ajax form in modal window


Pipe-symbol meaning in the context of the task on field homomorphisms.

Question: This question is from Rotman's Group Theory. This is more of a "If I knew what that symbol/word/phrase meant I might understand the whole Idea" question and less of a "I am so lost I cannot even understand the question" question.

Assume $F$ is a subfield of $C$ and $\{\alpha_1,...,\alpha_n\}\subset C.$ If $\sigma_i:F(\alpha_1,...,\alpha_n)\to C,i=1,2,$ are field maps with $\sigma_1|F=\sigma_2|F$ and $\forall i \  \sigma_1(\alpha_i)=\sigma_2(\alpha_i)$, then $\sigma_1=\sigma_2$

What does $\sigma_1|F$ mean? My guess is it means either extended image or restricted image.
Comment: You probably mean field homomorphisms rather than just field maps.
Answer: It means the same map as $\sigma_1$ but now with domain only $F$ rather than the original (bigger) domain $F(\alpha_1,\ldots,\alpha_n)$.
This is usually called the restriction to $F$.


Is it possible to recover original numbers of 2x2 table from odds ratio with given 95% confidence interval?

Question: Odds ratio is usually calculated from 2x2 table and looks to have quite unique combination of the resulting statistics and confidence interval values for each number combination (or number ratios) in 2x2 table.
Is it possible to recover original numbers of 2x2 table from odds ratio with given 95% confidence interval?
i.e.: OR=2.252; 95% CI = 0.932 -- 5.439
Is it possible to get number of observations from this data?
Is it possible to recover original numbers WITH given total number of observations?
i.e. 133 for the case above
P.S. Original table:
<code>                positive    negative
exposed         14          39
not exposed     11          69
</code>
Is it possible to do it in R?
Answer: Short answer no, different margins can produce the same odd's ratio and confidence interval. Some examples to follow.

Here is a brief sketch of how to find the minimum possible N for the table. Note that as per your linked site, the standard error can be related to the cell contents by:
$$\text{SE} = \sqrt{\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d}}$$
Ignoring the actual odds ratio, this value is minimized when $a = b = c = d$. This then produces the relationship:
$$\text{SE}^2 = \frac{1}{n/4} + \frac{1}{n/4} + \frac{1}{n/4} + \frac{1}{n/4} = \frac{4}{n/4} = \frac{16}{n}$$
So subsequently we can estimate the minimum possible N for a table as $16/\text{SE}^2$. In your example, the standard error can be recovered by the 95% confidence interval:
$$[\log(\text{High}) -  \log(\text{Low})]/[2 \cdot 1.96]$$
Which is just over $0.45$, and so the minimum possible N a table can have with that standard error is $16/0.45^2 = 80$ (taking the ceiling of this value).

This logic though won't help with finding the maximum. Let's say one row of the table, $c$ and $d$, have really large values. Thus ($1/c + 1/d) \approx 0$, and so we just have:
$$\text{SE}^2 = \frac{1}{a} + \frac{1}{b}$$
Let's say that $c/d = 4/9$, so to get an odds ratio of 2.25 we just need $a = b$. So for this example $\text{SE}^2 = 2/a$, and so $a \approx 10$. So the table below:
<code>                positive    negative
exposed         10          10
not exposed     4e6         9e6
</code>
Produces an odd's ratio of 2.25 and a 95% CI of 0.9365 to 5.4058.

Finally, even if you had the total N for the table, there is a symmetry in the standard error, you can simply swap the row totals and recalculate the cells to have the same odd's ratios. In many situations this will produce approximately the same standard error.
So we could rewrite your original table:
<code>                positive    negative
exposed         14          39
not exposed     11          69
</code>
As:
<code>                positive    negative
exposed         23          57
not exposed      8          45
</code>
Which produces an odds ratio of 2.2697 and a 95% confidence interval of 0.9280 to 5.5516. Not exactly the same, but to a certain extent this identification is dependent on the amount of rounding in the reporting, so I would be hesitant to rely on it, and with larger row totals it will be progressively harder to find the exact N.
If you know the N, you could always do a grid search, but I do not believe it will always result in a unique solution.

I've even forgot the most obvious symmetry, you can simply flip the numbers on the diagonals of the table and still obtain the exact same odds ratio and standard error. E.g.
<code>                positive    negative
exposed         69          11
not exposed     39          14
</code>
results in the same summary statistics as your original table.


Can you delete the missing values before the analysis of a time series dataset?

Question: The question is: Can I delete all the rows of a certain id because of its missing data, before the data analysis? To be more specific I will give the context and the specific problem:
Context
I have a daily time series dataframe from 1960 to 2018, where each observation is a measurement of precipitation, temperature, and level related to 133 rivers (the ids are the rivers). So, for each river, I will have all the measurements needed, for a given day, in a given year, between 1960 and 2018. I need to predict the river level.
I did this missing value plot to get some insights (the lower the percentage, the lower is the data for that flow in that year):

The inputs
There are three insights from this plot that concern me in order to delete or not delete missing values (and when):

There is an increasing number of distinct rivers per year since 1960.
2018 has a lot of missing data (which is because all the data stops at March of that year).
You have some rivers without data for the last 35 years (83 to 87 rivers in the plot, for example). And you have some rivers without data for the last 10, 5, or 2 years (you can see a lot of rivers with 2017 and 2018 data missing in the graph).

The problem
Should I consider all the rivers data for prediction even of those rivers that don't have data for the last X years? Because it doesn't make sense to predict the next year without the data of the last X years (but I do have the data of 1980 to 2000, for example). If I should consider it, what kind of methodology should I use to predict, if is there any?. If I shouldn't consider it, should I delete it? If I should delete it, when should I delete it? Before or after the data analysis for prediction? For example, before or after analyzing the distributions? (in order to group the rivers to impute the missing values as they have different distributions, but I could find a way to group similar distributions). It makes sense to me to delete them before the analysis, as the noise for prediction will be lower as the data for the analysis will be consistent with the data for prediction.
However, if I don't delete them at all I will have more data if I want to see some overall analysis (i.e. the trend from 1960 to 2018 will be a more "representative trend"). Deleting after, in terms of prediction (and insights for feature selection) it will increase the noise as the data for analysis will be different to the data for prediction (as I will have deleted complete data for some rivers).
Besides, I'm thinking of making the assumption that if the river doesn't have data from the last 3 years, the station measuring the river is not operative. But again, if the station is not operative, should I consider its data for overall analysis/insights?.
Answer: On the one hand, you should not be deleting anything. Information is power. On the other hand, I am highly skeptical about your ability to perform missing data imputation for the missing river-days. It is not clear how different rivers are related, and missing data imputation adds value only if different variables (rivers) are highly co-dependent. What you can do, however, is writing a parametric model for all the rivers. In this model

different rivers are allowed to be correlated,
any particular river is described by, say, an ARIMA process with unique parameters.

Then, you can estimate the model using the version of Full Information Maximum Likelihood (FIML) for time series. You would need to look up a standard reference on FIML and read it carefully. In a nutshell, FIML does the following.

The log-likelihood function has only terms corresponding to non-missing values.
All non-missing values are used in at least one term (of the log-likelihood).
Suppose, in time series $X_j$, values $X_j(t)$ and $X_j(t+3)$ are marked but values $X_j(t+1)$ and $X_j(t+2)$ are missing. Then the log-likelihood function contains term

$$
... + \log(f_{M,3}(X(t+3) | X(t))) + ...
$$
where $f_{M,3}(X(t+3) | X(t))$ is the model-implied conditional density of $X_j(t+3)$ given $X_j(t)$. You see that we are skipping missing cells $X_j(t+1)$ and $X_j(t+2)$.
Comment: Sounds sensible. *Kalman filter* and *expectation-maximization (EM) algorithm* may be relevant keywords (in addition to FIML).
Comment: @RichardHardy Thank you. Agree: EM algorithm and filtering are other interesting directions.
Comment: Thanks. I do not think this is all that distinct from what you are suggesting. FIML, ARIMA, state-space models (another relevant keyword) and Kalman filter are all related.
Answer: I would leave all of the data in, but then I would dig through to make sure there were not patterns misleading my data (e.g., that all of the new rivers are smaller because the state has been adding monitoring sites but already had the major rivers covered).
The other caveat is that you may need to use precipitation in the preceding X days as a predictor. In that case, you would need to delete the first X days each time observations start or be able to determine the precipitation from some other source (e.g., if that is from a local weather station that was recording, even though the river monitor was not).


Django F expression and ArrayField position

Question: I have a problem with Django and would like to ask for some advice:
One of my models contains specific indicators with variable values. This is why I am using an Arrayfield for those.
For example one Indicator has 3 values, another only 2 and some only have 1 value.
Now I want to compare values from within the Arrayfield with each other.
So I know there are those fancy F Expressions and I tried to use them, but here I am stuck, because it seems like F doesn't allow an positional-lookup from inside ArrayField.
Does anyone know if i can use F for positional lookup of arrayfield? Or do I have to make a direct SQL-Call from Django or change my db somehow to make one value per field?
Here is the Model:
class Indicators(models.Model):
 datetime = models.DateTimeField(default=datetime.now)
 name = models.TextField(default="none")
 values = ArrayField(models.FloatField(default=0))

This is what I want to achieve:
indicator_compareable = Indicators.objects.filter(name='compareable',
                   values__0=F('values__1')).values_list('values')

And it raises an Exception..

operator does not exist: double precision = double precision[]
  LINE 1: ...areable' AND "indicators"."values"[1] = ("...

As Mikhail suggested (in first comment to my post), here is the translated SQL from my Django filter:
SELECT "indicators"."values"[1] FROM "indicators"
 WHERE ("indicators"."name" = 'compareable'
  AND "indicators"."values"[1] = ("indicators"."values"))

From the Exception it seems like I am not allowed to give ArrayField position to F Expression (or that Django ignores the position..). And the translated sql-query shows, that it seems like F doesn't handle ArrayField-position...
In postgres a correct query for my needs would be:
SELECT values FROM indicators WHERE name='compareable' and
values[1]=values[2];

During my search I found those links here, but they didn't help..:
Django F field iteration
https://code.djangoproject.com/ticket/24709
Comment: Looks like Django generates an invalid SQL query. Can you post the whole query here?
Comment: Hi Mikhail, I edited my post.. seems like F does not allow using ArrayFied positions :-(
Comment: Looking at [this Django issue](https://code.djangoproject.com/ticket/24709), it doesn't seem like this case is supported :(
Comment: I wasn't shure if this issue is still related to my problem because it's one and a half years old. But it seems like.
Comment: Thanks Michail. I will use sql-raw query for my issue
Answer: So actually after investing the translated postgres query it's clear that django's F Expression is (still) not offering positional lookup (and might not be implemented).
I'll leave this as the answer, maybe it helps someone out there who is stuck at the same point.
I decided to use Django's SQL-RAW for my specific needs. But in general the rest of my code uses Django ORM whenever possible.
Here there has been a Django-issue to this topic, but by now it's 1 and a half years old: https://code.djangoproject.com/ticket/24709
by the way, my django version: Django 2.1   (the mentioned django-issue was with v1.8)
Thanks again to Mikhail for his suggestions.


Calculating 90% confidence intervals in regression to identify outlying data

Question: I have been searching for the answer for a problem with confidence intervals for a long time now, so I hope someone can help!
The two psychological papers below performed the same analysis, regressing irregular word reading onto nonword reading for controls, and plotting these regressions with 90% confidence intervals, before superimposing dyslexic participants to find outliers. However, one paper has straight line intervals, the other curved intervals. Why? (I have provided links to the papers so you can observe the plots).
I have scoured both textbooks and online resources, and the only descriptions of confidence intervals I have found are for confidence intervals of the regression line, which estimate an area in which there is a given probability that the regression line will fall. This is not what the studies below used, however. They used what may, I think, be termed 'prediction intervals' and describe an area in which a given percentage of the data will fall (which are then, in this kind of research, used to denote an area of 'normal' performance on a task). Can anyone help with these distinctions, and/or provide a calculation for this type of interval?
The curious thing is that SPSS/PASW will plot these confidence intervals for me, but offers no explanation of where they come from, nor can I edit the provided plot.
Any help, no matter how small, will be greatly appreciated!
References:
Manis, F. R., Seidenberg, M. S., Doi, L. M., McBride-Chang, C. & Petersen, A. (1996). On the bases of two subtypes of developmental dyslexia. Cognition: 58, 157-195. LINK: http://lcnl.wisc.edu/publications/archive/45.pdf
Stanovich, K.E., Siegel, L.S. & Gottardo, A. (1997). Converging evidence for phonological and surface subtypes of dyslexia. Journal of Educational Psychology: 89, 114-127. LINK: http://web.mac.com/kstanovich/Site/Research_on_Reading_files/RdJEdPsy97.pdf
Answer: The formula for the prediction interval can be found in many textbooks on regression.  Here is one set of notes on the web that goes into the mathematical details (other results from a web search show the concept, but not all show the math).
The basic idea is that you add the estimated variance for the y-values around the line to the estimated variance used for the confidence interval and use that for creating a new interval.  You have 2 sources of uncertainty, the line itself (as seen in the confidence bands) and the variation of the points around the line.  Plotting the prediction bands will generally show a curve, but depending on the data and the range plotted it may be difficult to tell the difference between the curve and a straight line.  So the one plot may be showing curves that are nearly linear, or they could be using a different method that finds straight lines for the bounds.
Note that assumptions of normality and equal variances are much more important for prediction intervals than for the other inferences on regression models, so if these assumptions don't hold for your case then the prediction intervals are going to be less useful (or possibly even meaningless).  Also note that if you are using these for outlier detection then you also have the problem of multiple comparisons.
Comment: +1 Good description and commentary.  In the cases I have seen, the parallel straight lines are merely set one or two standard errors above and below the fitted line: crude and misleading, but better than nothing at all.
Answer: Greg Snow's description is good, but I would add that you should probably use a least products type of regression rather than the normal least squares because both axes are subject to (I assume) an equivalent degree of error. The standard least squares regression privileges the x axis variable and often leads to the regression line having a lower slope than it should, a problem that can be seen in the figure 1 of the second paper that you have linked.
John Ludbrook has written about this a couple of times: http://www.cytel.com/papers/Ludbrook-Special-2002.pdf


CSS centering on desktop but not mobile

Question: I have been hunting around and haven't found a solution that works yet.
Here is my site which does what is needed on a desktop but when it loads in mobile the upper icons are no longer centered but the lower ones are
To supply all the code here would be long so I'm sure someone will knock me for that but to me it seems easy to use the inspector since i'm not sure if css else where could be effecting this.
I will gladly paste all of the code here if really needed.
As you can see on the desktop they are in a row but on a mobile I would like to have them centered and 2 in a row on portrait and all in a line on landscape but still be centered in the screen.
My Site Desktop
My Site in an iPhone 5 view
Comment: Does the class `.centermiddle` have a max width?
Comment: @Fraggy .centermiddle has no set width.
Comment: @chdtest answer should do the trick
Answer: Within your div .centermiddle you have divs named .view that have the style float: left;
Instead of floating your elements you should set them as display: inline-block; which is the correct way of achieving what you want to do.
Doing so will center those buttons for you.

CSS centering on desktop but not mobile


Ubercart Discounts (Alternative) needs Content module

Question: I have installed Ubercart in Drupal 7. My client needs to be able to have special pricing for members, so I added Ubercart Discounts (Alternative). I also installed CCK as a per-requisite. However, the installed module says it needs the Content module. I am not able to find this module.
Answer: Content is the core module of CCK. You can find it under the CCK section of your module install page.
Comment: It wasn't there. So, I've deleted CCK and I'm trying to reinstall it.
Comment: Make sure to uninstall the modules before replacing the code files.
Comment: It doesn't want to go away. It shows as disabled in the module list, but it doesn't show in the uninstall list.
Comment: No worries then, if it doesn't show up in the uninstall list the data schema is now gone. You should be ok to try a clean install of CCK's Content and other needed modules.
Comment: Assessing the current situation, it does not seem to match your answer precisely.
1. I have cleared the cache in Configuration more than once
2. The CCK group still appears in the Module list (not enabled)
3. CCK does not appear in the Uninstall list
4. The files are still on the server
Why would a module that is not enabled not show up on the Uninstall list?
Comment: From what it sounds like, the data schema was never installed. If it's not showing up in the uninstall list, you should be safe to enabled/install now.
Comment: Should I only be seeing one table for CCK: cck_field_settings? Should that table have any records after enabling?
Comment: I reinstalled both CCK and uc_discounts_alt module with the same problems. I don't think that Content is supposed to show at all in Drupal 7, but the Discounts module Codeless Discounts Field submodule doesn't seem to recognize that. I may have to start over with Drupal Commerce.
Comment: If this is a new project that doesn't need UC for legacy/requirement reasons, I highly recommend using Commerce in favor of UC. I'll leave that as the last word; comments aren't discussion threads on DA and this discussion is offtopic from your original question.


XGBoost CV GridSearch vs Early Stopping

Question: I am using the XGBoost Gradient Boosting Algorithm for a sales prediction dataset. I am planning to tune the parameters regularly with CVGridSearch. Now I am wondering if it makes sense to still specify the Early Stopping Parameter if I regularly tune the algorithm. From my understanding, the Early Stopping option does not provide such an extensive cross validation than the CVGridSearch method would. Does anyone have any suggestions or recommendations from a similar implementation?
Answer: It makes perfect sense to use early stopping when tuning our algorithm. We are not a faced with a "GridSearch vs Early Stopping" but rather with a "GridSearch and Early Stopping" situation. :)
We can readily combine CVGridSearch with early stopping. We can go forward and pass relevant parameters in the fit function of CVGridSearch; the SO post here gives an exact worked example. Notice that we can define a cross-validation generator (i.e. a cross-validation procedure) in our CVGridSearch.
Using early stopping when performing hyper-parameter tuning saves us time and allows us to explore a more diverse set of parameters. We need to be a bit careful to pull the relevant parameters from our classifier object (i.e. get the best_iteration directly from the fitted object instead of relying on the parameter grid values because we might have hit the early stopping beforehand) but aside that, everything should be fine.

XGBoost CV GridSearch vs Early Stopping


What is gained from a scale-location plot?

Question: The plot function in R provides four diagnostic plots for linear regression:

It seems like the residuals vs fitted plot and the scale-location plot are basically providing the same exact information. So why provide this seemingly redundant information? What is the scale-location plot adding?
Comment: See also https://stats.stackexchange.com/questions/121490/interpretation-of-plot-glm-model
Answer: In the example which you show the two plots, the residuals versus fitted and the scale versus location clearly give the same message. The scale location plot however is superior when the points are rather unevenly distributed along the $x$-axis. In that case it can be hard to distinguish in the residual versus fitted whether the apparent increase in spread is because there are more points in that part of the space or because there is a genuine increase.
Answer: The scale-location plot makes it easier to diagnose heteroscedasticity.  In the present case this is also evident from the residual plot, but the scale-location plot makes it clearer, since it also allows you to fit a smoothed line showing the change in the root-absolute residuals with respect to the fitted values.


This keyword in c#

Question: Hey I am new in C# and ASP.NET.
I tried to learn ASP.NET in this Site. In section of Session State, there is an example code (shown below). But I still don't get it, what function of "this" keyword on that code.
Can someone explain to me, what "this" keyword function in that code?, because I already tried to remove the "this" keyword, but the result still same.
<code>public partial class _Default : System.Web.UI.Page
{
   String mystr;

   protected void Page_Load(object sender, EventArgs e)
   {
      this.lblshstr.Text = this.mystr;
      this.lblsession.Text = (String)this.Session["str"];
   }

   protected void btnstr_Click(object sender, EventArgs e)
   {
      this.mystr = this.txtstr.Text;
      this.Session["str"] = this.txtstr.Text;
      this.lblshstr.Text = this.mystr;
      this.lblsession.Text = (String)this.Session["str"];
   }
}
</code>
Comment: first search this simple keyword and ask again
Comment: Yes, I already search that keyword.
This keyword means refer to this class/object. In javascript I get the function of this keyword, but I still dont get it what function of "this" keyword in that example code.
Comment: It's redundant in that particular code but some people prefer to use that style.
Comment: People sometimes type `this` to trigger IntelliSense in the code editor, and leave it there afterwards although it is not needed.
Comment: @ConnorsFan Thanks for the answer Connor. Btw I want to ask you something. Better if I learn c# first or asp.net first if I want to make web apps using asp.net web forms?
Comment: You will need to have a basic knowledge of the language to understand the code examples when reading about ASP.NET. Here is a post on this question: http://stackoverflow.com/questions/1179433/awesome-asp-net-and-c-sharp-tutorials-for-beginners
Comment: @ConnorsFan Thanks Connor for the answer. In that post contain many links to learn ASP.NET. I will try to uncover each site and decide which is best suite for me.
Answer: 'this' always refers to the object that has called the member function. We can access instance variables of the current object by using this. It increases readability of the code plus if we have instance variable and the local variable with the same name then it is used to refer instance variable instead of the local variables of the function.

This keyword in c#


Commerce Shipping - arbitrary calculations

Question: I'm using Drupal Commerce to create a simple shop and I wish to add shipping cost calculations based on the weight (in grams) of items in the cart. I have set a weight field on each product so I can work out what the total is, but the client has specified arbitrary prices depending on weight and destination:
<code>UK
weight = 0 price = 0.00
weight < 250 price = 1.00
weight < 500 price = 1.50
weight < 750 price = 2.00
weight < 1000 price = 3.00
weight < 2000 price = 5.00

Europe
weight = 0 price = 0.00
weight < 250 price = 2.50
weight < 500 price = 4.00
weight < 750 price = 5.00
weight < 1000 price = 6.25
weight < 1250 price = 7.50
weight < 1500 price = 8.75
weight < 1750 price = 10.00
weight < 2000 price = 11.25

Other
weight = 0 price = 0.00
weight < 250 price = 4.00
weight < 500 price = 6.50
weight < 750 price = 8.75
weight < 1000 price = 11.25
weight < 1250 price = 13.75
weight < 1500 price = 16.25
weight < 1750 price = 18.75
weight < 2000 price = 21.25
</code>
Would I need to add a shipping calculation rule for each possible scenario or is there a way to configure rules to look up a table?
Answer: I found a module that does exactly what I want: Commerce Shipping Weight Tariff.
I set up three services, one for each destination, and used rules based on the shipping address to work out which service to use. Then I added the weight tariffs for each service.
Answer: It wouldn't be a bad idea to create a lookup table with a single Rules action that knows how to read the data from the table and set the shipping rate accordingly. However, if you aren't a developer, you can configure this entirely through Rules.
Shipping rates are passed through Rules similarly to product prices, so you can work with 1 or multiple flat rates (from https://drupal.org/project/commerce_flat_rate) and use multiple rate calculation rules per flat rate to adjust the price based on the destination and the total order weight. I can't recall if we got Rules integration for total order weight into Commerce Physical Products or not, so you might start by looking there.


Why do we need the regularization term for NMF but not for SVD?

Question: In non-negative matrix factorization (NMF) one minimizes the Frobenius norm plus a regularization term. However SVD simply minimizes the Frobenius norm. Why do we need the regularization term for NMF but not for SVD?
Answer: NMF does not always include regularization -- for example, see the first two cost functions here. But, regularized NMF can be useful:

If you're willing to add extra constraints beyond nonnegativity, you can produce highly interpretable structures, including the K-means centroids.
If you want to fit an NMF so that $X = LR$ where $L$ is wider than $X$, the model is useless (trivial, ill-posed, non-identifiable) because you could just set $L$ equal to a zero-padded version of $X$ and set $R$ to a zero-padded identity matrix. In this case, an L1 penalty on each factor matrix will help out by encouraging both factors to contribute. For a low-dimensional example, a pair of 3's incurs a lower L1 penalty than a 9 and a 1.

Another possible answer is that SVD is a rigid term and NMF is not. If you added regularization to an SVD, it would no longer be called an SVD, but the term NMF is less prescriptive.
Comment: NMF is quite often used with the objective function only defined for observed values. That is with a large number of missing values in the matrix.  Is it related to that ?
Comment: Amazingly, the low-rank structure [works pretty well](https://arxiv.org/abs/1212.0467) on its own.


Why does Rust need to know the code in module belong to whom?

Question: I encountered some problems reading the Rust documentation:

In this example, we have three modules again: client, network, and network::client. Following the same steps we did earlier for extracting modules into files, we would create src/client.rs for the client module. For the network module, we would create src/network.rs. But we wouldn't be able to extract the network::client module into a src/client.rs file because that already exists for the top-level client module! If we could put the code for both the client and network::client modules in the src/client.rs file, Rust wouldn't have any way to know whether the code was for client or for network::client

Why does Rust need to know the code in client.rs belongs to client or network::client? Can it belong to both?
Answer: The compiler has rules about where the source file for an external module can be. Those rules ensure that there aren't two modules that use the same source file.
If you really want to, you can override the rules with a #[path] attribute:
mod client; // defaults to client.rs relative to the current file

mod network {
    #[path="client.rs"] // reads the same source as the outer `mod client;`
    mod client;
}

However, doing so would lead to duplicate code, i.e. the code in client.rs would be compiled twice, and everything that's defined in client.rs would be defined twice, in two separate modules. It's as if you made network/client.rs an exact copy of client.rs and didn't write the #[path] attribute.
Another thing you can do is provide an alias for a module by reexporting it elsewhere. This can be useful when building a library: it enables you to present an external module hierarchy that is different from the internal module hierarchy.
mod client; // not accessible externally

pub mod network {
    pub use client; // network::client::* will refer to the same definitions as client::*
}

For example, with the above, the client module is defined in client.rs, but clients use it through my_crate::network::client.


How to store date-time to optimize search in DB

Question: How to store date-time to optimize search in DB and reduce data duplication?
Comment: I'm thinking in separate date and time and put index to each. since the time is manage hh:mm:ss and date is like MM/DD/YYYY
Comment: Why not just use a datetime datatype? That is what they are for.
Answer: Searching through integers are very efficient in SQL. You could store the date as a UNIX timestamp in an int column.  Depending on what language you are accessing the database with, there are probably many functions you can use to convert a human readable date or a date object to a Unix timestamp which you can use in your query. More info here .
Comment: There are a number of challenges with this type of approach. You have to constantly convert dates to the UNIX time which is a pain. The datetime datatype is really the best option in sql server.
Comment: Sean Lange is not wrong.  It all depends on which kind of database and which language is accessing the database.
Comment: Well this is tagged with sql server so we have to assume that is correct. The biggest challenge with the UNIX timestamp is doing datemath is painful and ends up being nonSARGable because you end up wrapping the column in functions to turn it back into a datetime value.


Disabling Mic before exporting from DAW

Question: Sorry if this not the right place to ask this question..but i was just wondering how necessary is it to disable the microphone before exporting recorded audio from DAW to wav,mp3,midi,
Answer: It's not at all necessary. Usually you want to take tracks out of record mode before mixing since any microphones connected to tracks that are still record armed might feedback if and when you turn up the volume on the monitors.

Disabling Mic before exporting from DAW